Analysis of Patient Safety Event Report to Understand the Contribution of Health IT to Diagnostic Error

Abstract

Background:

Diagnostic errors are one of the most common and costly medical errors. Most diagnostic errors are due to provider cognitive processes and biases. With the widespread adoption of electronic health records (EHRs), and other health information technology (health IT), EHRs are now the central repository for clinical information and its design and use affect the diagnostic process and diagnostic errors. The goal of this study was to analyze patient safety event reports to determine health IT contributions to diagnostic errors. Understanding how the health IT contributes to diagnostic error will help direct improvement efforts.

Methods:

From a data set consisting of 1,110,029 reports entered between 2015 and 2021, from 195 unique health care organizations across the United States, 2618 likely diagnostic error reports were retrieved. A sample of these reports were reviewed and those that were diagnostic related were coded by subject matter experts for whether the diagnostic error was preventable, the stage of the diagnostic process in which the error occurred, the type of error, how much health IT contributed to the error, what health IT system was responsible for the error, whether health IT was directly or indirectly responsible for the error, the type of health IT issue, whether copy and paste was mentioned and contributed to the error, whether the health IT contribution was preventable, the outcome of the error, and the severity of the error.

Results:

There were 2618 reports with a general event type category that suggested a diagnostic error. Of these, 119 reports explicitly mentioned health IT and were found to have strong or moderate evidence of health IT contributing to the error. From the remaining 2499 reports, 250 were randomly sampled and 93 (37.2% of 250) had strong or moderate evidence of a health IT contribution. Further analysis of these 212 reports showed EHRs were the most commonly described type of health IT associated with diagnostic errors (58.5%) and most diagnostic errors occurred in the test phase of the diagnostic process (74.5%). Most reports that had health IT as a contributor to the diagnostic error were associated with patient harm (74.5%). There was a trend towards a higher degree of harm when the errors were health IT-related compared with when there was little evidence of health IT contribution.

Conclusions:

Health IT, and specifically the EHR, is a contributor to diagnostic errors. To address these issues, improved reporting taxonomies and improvements in health IT system design are needed.

Diagnostic error, responsible for almost 80,000 deaths per year in the United States, is defined by The National Academy of Medicine as “The failure to establish an accurate and timely explanation of the patient’s health problem(s) or communicate that explanation to the patient.”^1,2 It is a major problem in public health and is associated with more harm to patients than any other type of error.³ Most Americans will experience at least one diagnostic error in their lifetime and a recent study performed in outpatient settings estimated that about 5%, or ~12 million US adult patients will experience a diagnostic error every year.^4,5 Studies have also shown that diagnostic error is the leading contributor to settled malpractice claims and has cost the health care system more than $100 billion per year.^6,7

Diagnostic errors are complex and have multiple contributing factors, with cognitive biases being the most common cause.^8,9 Cognitive biases—systematic patterns of deviation from norm or rationality—can influence various stages of the diagnostic process. These errors can affect decision-making and have been identified in every step of the diagnostic process, including physicians’ ability to find and process relevant information, reason with this information, and develop a diagnosis.^10,11. To address diagnostic error, it is critical to understand providers’ clinical reasoning and cognition in the diagnostic process.³

Advancements in electronic health records (EHR) and health information technology (health IT) have played a key role in provider’s clinical decision-making process.¹² Over the last decade, health care providers have rapidly adopted EHR’s with 96% of all non-federal acute care hospitals in the United States adopting a federally certified EHR in 2021 compared with 9% in 2008.¹³ In addition, as of 2021, about 4 in 5 (78%) office-based physicians in the United States adopted an EHR compared with 17% of in 2008. Several other countries have also transitioned from paper to EHRs in an effort to digitize medicine.^14,15 The adoption of EHR’s has been beneficial in certain regards for both patients and health care providers as EHRs serve as the primary repository for clinical information and have impacted every step of the diagnostic process.^16,17

Although EHRs have improved certain aspects of the quality and efficiency of care delivered, they have also been recognized as a significant contributor to patient harm-related events.^18–20 Specifically, software malfunctions, interoperability challenges, and usability issues may arise within the EHR that can affect a provider’s ability to make an accurate diagnosis, leading to diagnostic error.^21,22. Because the EHR serves as the primary repository for the vast majority of information related to a patient’s medical condition, it is integral to all 3 stages of cognitive processing in the diagnostic process including information gathering, working diagnosis, and information integration and interpretation.¹ Thus, problems with EHR use or design can disrupt diagnostic processing at any one of these stages and negatively affect the provider’s ability to make an accurate diagnosis.²³ Human factors is a scientific discipline that aims to understand human capabilities to design work environments that meet these capabilities and enable optimal human performance.²⁴ A human factors approach can help in understanding, evaluating, and supporting diagnostic reasoning and decision processes, specifically by identifying EHR issues that need to be addressed to reduce the likelihood of diagnostic error.²⁵

Little is known about the association between EHR integration in the clinical environment, clinicians’ cognitive process, and in which part of the diagnostic process errors occur.¹ A previous study using closed ambulatory diagnosis-related legal claims showed the EHR was considered a potential contributor in 61.3% of these claims.²¹ Although use of medical malpractice claims can provide rich and important information on diagnostic errors, the selection bias in their occurrence combined with the time lag between occurrence and settlement, limits their utility for organizations interested in more real-time information for system improvements.

Conversely, patient safety event reporting is a complementary method used by health care facilities for reporting of patient safety issues.^1,26 To support analysis and insights from patient safety event reports, patient safety organizations (PSOs) have been established as “safe harbors” to collect patient safety event report data from different health care facilities. The aggregated data housed by PSOs provides the opportunity to develop a more comprehensive understanding of patient safety issues across health care facilities and to identify key patterns and trends. The goal of this study is to analyze data from a PSO to identify EHR contributions to diagnostic events. Patient safety reports provide a timelier description of these types of issues compared with legal claims and these reports provide a different lens on diagnostic safety given they are often reported by frontline clinicians rather than patients. Understanding how the EHR contributes to diagnostic error will help direct future research to deploy a suite of solutions to improve software, user, and system.

METHODS

Patient safety event reports, which are voluntarily reported free-text descriptions of safety issues and contributing factors, were analyzed from the Collaborative Healthcare Patient Safety Organization. The data set consisted of 1,110,029 reports entered between September 18, 2015, and February 6, 2021, from 195 unique health care organizations across the United States. The organizations were comprised of primarily hospitals spanning large academic centers to smaller community hospitals in rural settings. The patient population is diverse in terms of race/ethnicity, sex, age, and health condition resulting in a generalizable data set. The free-text in these reports were de-identified and structured fields were not available.

Each report consisted of a free-text description and a general event type category. To identify reports that had both a potential diagnostic error and an EHR component we looked at the general event type categories provided by PSO and chose those that were likely diagnostic error-related based on their general event type categories selected by the reporter. General event type categories that were considered included, ‘Delayed Diagnosis, ‘Diagnosis- Incorrect’, and ‘Diagnosis Issue’. A complete list of the categories upon which our search was conducted is included in Appendix A, Supplemental Digital Content 1, http://links.lww.com/JPS/A762. Reports were de-identified and any reports without a free-text description were removed resulting in 2,618 reports for analysis.

These reports underwent a preliminary review by subject matter experts with expertise in medicine, patient safety, and human factors to identify those that were diagnostic error-related and explicitly mentioned health IT resulting in identification of 119 reports.

The events that involved health IT were analyzed for the following topics:

• Types of devices or systems involved

• At which point in the process the error occurred

• Type of information technology (IT) hazard

• Level of preventability

• Level of severity of harm.

To ascertain the sensitivity of clinician direct attribution of health IT in their reporting (eg, a report did not explicitly mention a health IT system or CPOE, but the reviewer could assume an order was placed electronically and health IT was involved), aka the false negative rate of the initial labeling of Health IT contribution), an additional 250 reports, a 10% sample of the remaining 2499 reports that did not explicitly mention health IT were annotated resulting in 369 reports total in the final data set for this analysis (Fig. 1).

FIGURE 1.

Process to identify diagnostic error-related reports with health IT contributions.

Using the de-identified free-text, a subject matter expert reviewed the 369 PSE reports using a taxonomy developed by a group of patient safety experts. We modified a taxonomy originally developed by Schiff et al²⁷ and used in an analysis of legal claims.²¹ Annotating each report included determining¹ the presence of a diagnostic error,² whether the diagnostic error was preventable,³ the stage of the diagnostic process in which the error occurred,⁴ the type of error,⁵ how much health IT contributed to the error (on a scale from 1 to 6 with 1 being ‘little to no evidence of causation’ and 6 being ‘virtually certain evidence for causation’),⁶ what health IT system was responsible for the error,⁷ whether health IT was directly or indirectly responsible for the error,⁸ the type of health IT issue,⁹ whether copy and paste was mentioned and contributed to the error,¹⁰ whether the health IT contribution was preventable,¹¹ the outcome of the error,¹² and the severity of the error. Fifty reports were reviewed by a team of 3 physicians and one nurse to establish the coding process and definitions. The nurse then independently reviewed the remaining reports. χ² tests were used for statistical analysis using GraphPad Prism Software. See Appendix A, Supplemental Digital Content 1, http://links.lww.com/JPS/A762 for the overall coding taxonomy.

RESULTS

All the 119 reports that explicitly mentioned health IT had strong or moderate evidence of health IT contributing to the diagnostic error upon full review of the text. Of the 250 reports that were randomly sampled from the subset that did not explicitly mention health IT, 93 of 250 reports (37.2%) described strong or moderate evidence of health IT contributing to the diagnostic issue. Combining these 2 samples, of the 369 diagnostic error-related reports extracted from the superset of 2618 reports, 25.7% (n=95 of 369) described strong evidence (levels 4, 5, or 6) of health IT contributing to the error, 31.8% (n=117 of 369) described moderate evidence (levels 2 or 3), and 42.5% (157 of 369) had little to no evidence (level 1). Importantly, all of the 119 reports that explicitly mentioned health IT had strong or moderate evidence of health IT contributing to the diagnostic issue. Of the 250 reports that were randomly sampled from the subset that needed detailed review, 93 reports of 250 reports (37.2%) described strong or moderate evidence of health IT contributing to the diagnostic issue.

Type of Health IT and Issues Contributing to the Diagnostic Error

The types of health IT devices or systems identified in the 212 reports were electronic health record (EHR) (n=124 of 212, 58.5%), picture archiving and communication system (PACS) (n=82 of 212, 38.7%), and other (ie, workflow or undetermined) (n=36 of 212, 17.0%). Within these reports, issues such as use/usability, design, and environmental/workflow challenges contributed to the errors. Usability contributed to over half of the health IT diagnostic issues (n=118 of 212, 55.7%), followed by unclear/other (n=51 of 212, 24.1%) and design (n=43 of 212, 20.2%).

Diagnostic Process Analysis

The diagnostic process was analyzed to determine where the error occurred in the diagnostic process. About three-quarters of the errors (n=158 of 212, 74.5%) were identified in the testing phase of the diagnostic process. This involved ordering, interpreting, and reporting of tests. Assessments (n=37 of 212, 17.5%), access and referrals (n=13 of 212, 6.1%), and other (n=4 of 212, 1.9%), made up the remainder of the cases including errors in eliciting a critical piece of patient medical history or failing to provide close follow-up of patients.

Type of Health IT Hazard

Execution (n=53 of 212, 25%) and Interpretation (n=53 of 212, 25%) of tests were the most frequent hazard types followed by documentation (16.0%, n=34 of 212), notification (14.2%, n=30 of 212), order (9.9%, n=21 of 212), communication (7.6%, n=16 of 212), and other/unclear/no hazard (2.3%, n=5 of 212) (Fig. 2).

FIGURE 2.

Distribution of types of health IT hazards.

Preventability Analysis

Most errors were considered preventable (n=105 of 212, 49.5%), followed by probably preventable (n=86 of 212, 40.6%), and then possibly preventable (n=18 of 212, 8.5%) and not preventable (n=3 of 212, 1.4%).

Patient Harm Analysis

Overall, 74.5% of cases with health IT contribution had some degree of patient harm, compared with only 38% of cases without health IT involvement [Fig. 3A; P<0.0001; RR: 2 (1.6-2.4)]. When we further subdivided cases by severity, a trend towards a higher degree of harm was seen when errors were health IT-related. Specifically, there was a higher percentage of high harm events [significant (n=74 of 212, 34.9%), serious (n=69 of 212, 32.5%), life-threatening (n=14 of 212, 6.6%), or fatal (n=1 of 212, 0.1%)] when the health IT contribution to the diagnostic error was considered strong (levels 4, 5, and 6) as opposed to when the health IT contribution was moderate or little to none (levels 1, 2, and 3) (42.1% and 32.5%, respectively) [Fig. 3B; P=0.09; RR: 1.3 (0.96-1.7)]. Finally, EHR contribution was associated with a likelihood of harm (74.5%) when compared with non-EHR sources of health IT [74.5% versus 51.5%; P<0.001; RR: 1.4 (1.2-1.7)] (Fig. 3C).

FIGURE 3.

Health IT and harm severity: patient safety events were codes for both degree of Health IT contribution and Harm Severity. A, Overall association of presence or absence of any contribution of health IT or patient harm. B, Cases were further subdivided to either high versus low likelihood of health IT versus high versus low degree of patient harm. C, Overall association of presence or absence of any contribution of EHR or patient harm.

DISCUSSION

Of the diagnostic error reports reviewed, we found 57.5% (n=212) were health IT-related and described a moderate to strong contribution of health IT to the error (levels 2, 3, 4, 5, and 6). All of the reports that explicitly mentioned health IT contributed to diagnostic error. Furthermore, of the 250 reports randomly sampled that did not explicitly mention health IT, 37.2% were identified as having strong or moderate evidence of health IT contributing to the error. If we use this information to extrapolate to the 2499 total reports that did not explicitly mention health IT ~930 reports would have a health IT contribution to diagnostic error. Our analyses suggest reporters may be unaware that health IT is contributing to their events or that reporters don’t have a comprehensive or easy way to capture health IT events, leading to an underestimation of how much health IT may be contributing to safety events. A more robust taxonomy and an easier process for reporting may enable reporters to better identify, capture, and classify safety events with a health IT contribution. Notably, when health IT was identified as being strongly related to the diagnostic issue, these safety reports were associated with higher harm events compared with when health IT was identified as having moderate or little to no contribution to the report.

When the EHR was the health IT system that contributed to a diagnostic event, the largest percentage of those events were due to use and usability issues as opposed to malfunctions or other technical issues. EHR usability issues include test-ordering processes that are not intuitive for the user or issues with user access. Recent studies point to user-related issues including incorrect or missing information, alert fatigue, various workarounds, and other user-related difficulties such as access data in hybrid record systems.²⁸ Design challenges involving the EHR such as clinical documentation/charting not being accessible or the lack of or poorly designed clinical decision support tools for the prompting of specific order have been shown to lead to diagnostic errors such as missed or delayed diagnosis. These usability issues impact clinician’s ability to recognize, process, and make decisions with information and can exacerbate certain cognitive biases that may promote diagnostic error. Knowing that most errors occur in the test phase, developers or usability experts could focus design improvements to help ensure ordering of tests is more seamless and mistake-proofed. With several countries actively transitioning from paper to EHRs, or planning on transitioning in the near future, ensuring rigorous usability testing during implementation to mitigate technology-related diagnostic errors is imperative. It should be noted that all of the analyzed reports are voluntarily reported and may contain bias that could inflate the number of usability issues, although there is an extensive body of literature highlighting usability challenges with EHRs.

There are several health IT aspects that should be a focus of improvement efforts including:

• Designing ordering processes that are intuitive with ordering menu structures that are free of clutter to enable ordering of the correct test.

• Results displays that show current results and historical results in a display manner that enables easy comparisons.

• Show results graphically when it will promote understanding and ensure the graphs have clear axis labels to enable correct interpretation.

• Improvements in training with the system, which recapitulates workflow.²⁹

The PACS system was also found to be a health IT system that contributed to several diagnostic issues. This may be due to a common issue with PACS, which should be further investigated, or a reporting bias.

One challenge with reviewing patient safety event reports for health IT contributions to diagnostic error is that many of the report free-text descriptions do not explicitly detail how the health IT system contributed to the diagnostic error. Without knowing this information improvements efforts will be difficult. Currently, there is no agreed-upon structured taxonomy that assists in the reporting, categorizing and analyzing health IT-related diagnostic errors. Developing a structured taxonomy for the reporting of health IT-related diagnostic issues, similar to the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP), will be essential for health care facilities to capture the full scope of the problem.³⁰ Any additional taxonomy would need to be designed to integrate with other taxonomies currently used to capture patient safety events. In addition to a structured taxonomy, developing ways of capturing health IT system metadata to be included as part of the patient safety event report would be beneficial. Finally, trigger tools and other mechanisms, aside from patient safety event reports, could be developed to capture health IT contributions to diagnostic error.

In the sample of reports analyzed as part of this study, 57.5% of diagnostic error-related patient safety event reports had a health IT contributing factor suggesting that health IT may be a major contributor to diagnostic errors. This analysis provides a unique lens on the relationship between diagnostic error and health IT compared with a recent analysis of legal claims data which focused specially on diagnostic error legal claims that were EHR related.²¹ There are differences in the methods used in both studies, which makes it difficult to make a direct comparison. The legal claims analysis started with legal claims initially flagged as diagnosis and EHR related and found that 61.3% of these claims involved the EHR as a contributing factor to the safety issue; in the remaining the EHR was noted to have contributed an increased risk for the claim unrelated to safety The analysis presented here started with a sample of diagnostic errors that were not initially filtered to those that may be health IT-related and, therefore, demonstrates the potential for greater prevalence of health IT as a contributing factor to diagnostic errors. There is some alignment in the 2 studies. Within the boarder category of health IT, the PSE analysis found that EHR was the most prevalent health IT system. Also, most of the PSE reports that were diagnosis and EHR related occurred during the testing stage of the diagnostic process, which is similar to the findings in the legal claims study. Together, these 2 studies highlight the contribution of health IT systems, and commonly the EHR, to diagnostic error.

Limitations

PSE report narratives are retrospective accounts and may be incomplete and contain bias from the reporter. They are typically first-hand observations of potential safety hazards or adverse events written in free-text leading to variability in the quality and amount of information in the reports. It has also been shown that there is under-reporting of events as reporters may lack clarity on what needs to be reported, limited perceived benefit of reporting, and barriers to reporting immediately.³¹ In addition, information gathered from risk management or leadership reviews conducted after the reporting, is not available in the event report data. The data we analyzed were from September, 2015 to February, 2021 and may not represent the most current health IT and diagnostic issues given evolving technologies. However, these data do provide insights on the types of health IT safety issues facilities may face. Our sampling approach of including reports that explicitly mention health IT may introduce some bias. In addition, it is possible some hospitals reported more data than others and reported more data with certain characteristics, which could skew the results.

CONCLUSIONS

Health IT, and specifically the EHR, is a contributor to diagnostic errors. Although the EHR is used for several different clinical functions, EHR issues with testing were a prominent driver of diagnostic error. To address health IT contributions to diagnostic error, improved reporting taxonomies and other methods that support detailed descriptions of health IT involvement in these errors are needed.

Supplementary Material

pts-22-162-s001.docx ^{(20.4KB, docx)}