System Related Interventions to Reduce Diagnostic Error: A Narrative Review

Abstract

Background

Diagnostic errors (missed, delayed, or wrong diagnosis) have gained recent attention and are associated with significant preventable morbidity and mortality. We reviewed the recent literature to identify interventions that have been, or could be, implemented to address systems-related factors that contribute directly to diagnostic error.

Methods

We conducted a comprehensive search using multiple search strategies. We first identified candidate articles in English between 2000 and 2009 from a PubMed search that exclusively evaluated for articles related to diagnostic error or delay. We then sought additional papers from references in the initial dataset, searches of additional databases, and subject matter experts. Articles were included if they formally evaluated an intervention to prevent or reduce diagnostic error; however, we also included papers if interventions were suggested and not tested in order to inform the state-of-the science on the topic. We categorized interventions according to the step in the diagnostic process they targeted: patient-provider encounter, performance and interpretation of diagnostic tests, follow-up and tracking of diagnostic information, subspecialty and referral-related; and patient-specific.

Results

We identified 43 articles for full review, of which 6 reported tested interventions and 37 contained suggestions for possible interventions. Empirical studies, though somewhat positive, were non-experimental or quasi-experimental and included a small number of clinicians or health care sites. Outcome measures in general were underdeveloped and varied markedly between studies, depending on the setting or step in the diagnostic process involved.

Conclusions

Despite a number of suggested interventions in the literature, few empirical studies have tested interventions to reduce diagnostic error in the last decade. Advancing the science of diagnostic error prevention will require more robust study designs and rigorous definitions of diagnostic processes and outcomes to measure intervention effects.

INTRODUCTION

A growing body of evidence identifies diagnostic error (missed, delayed, or incorrect diagnosis) as an important patient safety issue. (1-5) Although not all of these errors translate into harm and many go undetected, a substantial number are associated with preventable morbidity and mortality. (6, 7) Diagnostic errors affect every medical discipline and all types of patients. However, the focus on diagnostic errors has lagged behind the rest of the patient safety movement. (8, 9)

Studies have begun to identify the root causes that contribute to diagnostic error. (10-12) These causes include either one or more of the following: clinician cognitive factors, systems factors, and patient factors. Cognitive factors include perceptual and thought processes, which are in turn influenced by differences in clinician training and experience, predisposition to cognitive and affective biases, fatigue, stress, and a variety of other elements. System factors refer to organizational vulnerability to diagnostic error and may include faulty communication practices, inadequate coordination of care, inadequate supervision, poorly designed technology and work environment, reduced availability of resources or personnel, inadequate feedback, and a culture that does not necessarily promote effective learning from error. (10) Patient-related factors include variability in communication styles and practices, heterogeneity in patients’ clinical presentation to providers, and differential access to personal health information. (13)

Reducing the likelihood of diagnostic error and error-related harm is a critical priority. Recent insights about diagnostic error etiology have stimulated ideas about potential solutions. However, to our knowledge, strategies to reduce diagnostic error have not been compiled or comprehensively reviewed since the release of the IOM report To Err is Human. We therefore conducted a literature review to identify key interventions to reduce or prevent diagnostic errors over the past decade. Our aim was to identify interventions that have been, or could be, implemented to address systems-related factors that contribute directly to diagnostic error. For the purposes of the review, we broadly categorized patient factors along with systems factors. This paper examines systems-related interventions, while a companion paper reviews cognitive interventions (including decision support) to improve the reliability of clinical reasoning.

METHODS

Search Strategy

We used multiple search strategies to identify candidate articles that described interventions to prevent, reduce, or mitigate diagnostic errors. Although many advances and interventions in healthcare may be intended to improve diagnosis (e.g., new diagnostic tests or screening methods), we focused specifically on system-level interventions to prevent or mitigate medical error in the diagnostic process. To avoid searching potentially hundreds of thousands of indexed papers, we used more restrictive than inclusive strategies to select for diagnostic errors. Thus, we conducted a search of the PubMed database that combined the major Medical Subject Headings (MeSH) “Diagnostic Errors” or “Delayed Diagnosis” AND one or more relevant MeSH terms and keywords to capture both systems and cognitive interventions (see Table 1 for complete list). We also focused our initial search on the time period after the release of the IOM report To Err is Human (14), 2000-2009, to focus on more recent literature. We limited our search to English language publications that focused on humans and had abstracts that could be used for initial screening. This strategy yielded 949 articles.

Table 1. Medical Subject Headings (MeSH) Terms and Keywords Used as Qualifiers for Major MeSH Terms Diagnostic Error or Delayed Diagnosis, in alphabetical order.

MeSH terms:	Keywords:
Affect	“bias”
Clinical Competence	“cognitive error”
Communication	“metacognition”
Continuity of Patient Care
Decision Making
Decision Making, Organizational
Decision Support Systems, Clinical
Decision Support Techniques
Feedback
Forms and Records Control/standards
Guidelines as Topic Knowledge Bases (includes  heuristics)
Health Knowledge, Attitudes, Practice Knowledge of  Results (Psychology)
Health Literacy
Health Records, Personal
Human Engineering
Judgment
Medical Informatics
Medical Records Systems, Computerized
Mental Recall
Organizational Culture
Patient Access to Records
Patient Participation (includes patient involvement)
Physician Patient Relations
Physician’s Practice Patterns
Problem Solving
Professional-Patient Relations
Reminder Systems
Systems Analysis
Time Factors
Truth Disclosure

We employed several secondary strategies to locate additional relevant articles for review. First, we manually reviewed the references of the articles we identified through PubMed as described above. Second, we used an internet-based search engine (Google) and searched topic-specific research databases (AHRQ’s PSNet, PsycINFO, and the Air University Library Index to Military Periodicals) with a subset of search terms listed in Table 1. Third, we solicited additional recommended references from several authorities in the field of diagnostic error and decision-making sciences. Finally, we identified relevant articles released in 2010 after the cut-off date of the formal review but relevant to the topic. Together, these secondary search methods yielded an additional 160 articles.

Selection Strategy

Because the field is nascent and evolving, we also reviewed the literature for intervention concepts that have been suggested by expert commentators, usually based on studies examining the epidemiology or etiology of these errors. Thus, we considered two broad classes of articles: 1) “actual” interventions that were tested to reduce error or harm in medical diagnostic settings, and 2) “suggested” interventions, i.e., those that had not been tested. The latter category was included to help refine our search for tested interventions, inform the state-of-the-science, and highlight potential areas of future research. Articles that tested actual interventions discussed measurable changes in either patient behaviors or in organizational services, processes, systems, structures, or products in order to prevent or mitigate diagnostic error. We included all study designs, including review papers in the case of suggested interventions.

We excluded studies that described inter-rater or observer variation in the absence of an intervention; validations of screening instruments or tests; single case reports; assessments limited to provider satisfaction, preference, or acceptance of interventions; and techniques to enhance diagnosis involving screening instruments, specific tests, or technologies (e.g., a newer generation CT scan). We also excluded studies of the development of risk models and reports of diagnostic error frequency or etiology.

Abstracts were reviewed by three health services researchers and categorized as “included,” “excluded,” or “unsure.” To improve reliability and consistency of categorization, the three reviewers first independently reviewed 20 abstracts, compared categorization, and refined their categorization criteria. Two physicians with expertise in diagnostic error research further validated the inclusion/exclusion process by reviewing a random sample of excluded articles, and all articles categorized as “unsure” and “include,” a strategy that helped achieve better inter-rater agreement. Disagreements in categorization were resolved by team consensus.

Data Extraction

We extracted data using structured data collection forms. For the studies reporting actual interventions, we documented study design, content and duration of intervention, type of intervention subjects, scope of intervention or “reach” (i.e., by number of participants), outcome measures, and intervention effectiveness. For suggested interventions, we documented the focus (disease, condition, etc.) and methods for diagnostic error prevention. All data collection forms were checked for completeness. All team members, including expert physicians, participated in the review and interpretation of results.

Categorization

Diagnosis is a multistep process that depends on the functioning of the provider, patient, and health system. Accordingly, we categorized systems-based interventions according to five previously described, interactive steps (13) of the diagnostic process: 1) the patient-provider encounter that involves clinician decision-making and test/referral ordering based on details of patient presentation; 2) performance and interpretation of diagnostic tests; 3) follow-up and tracking of diagnostic information over time; 4) subspecialty and referral-specific issues; and 5) patient-related care-seeking and adherence processes. Several interventions were not specific to a particular step in the diagnostic process and were categorized as “general interventions.” (Table 2)

Table 2. Taxonomy of Diagnostic Error Dimensions.

Process dimension	Description	Example
Provider-patient encounter	Problems with history, physical exam, or ordering diagnostic tests for further work-up	Significant symptoms are not noted or acted upon at the time of the encounter
Diagnostic tests	Problems with ordered tests either not performed or performed/interpreted incorrectly	Incorrect laboratory test result due to mislabeled specimen
Follow-up and tracking	Problems with follow-up of abnormal diagnostic test results or scheduling of follow-up visits	No follow-up of an abnormal X-ray despite suspicious finding
Referrals	Lack of appropriate actions on requested consultation or communication breakdown from consultant to referring provider	Consultant requests additional information from referring provider, but referring provider does not respond to the inquiry
Patient related	Delay in follow-up appointments, uncertainty over how to react to abnormal test results, low adherence, failure to provide critical history information	Patient does not show up for a scheduled diagnostic test

RESULTS

We identified 43 articles on systems-related interventions that met criteria for full review. The majority of articles did not describe empirical studies, but rather provided suggestions for interventions based on the origins of diagnostic errors, observational research of system/patient factors, or promising results from studies of related topics (e.g., patient satisfaction with test notification). (Table 3) Six articles reported empirical outcomes of actual systems interventions. (Table 4) These 6 studies were non-experimental or quasi-experimental and measured outcomes before and/or after an intervention among a small number of clinicians or health care sites. Measures of diagnostic error varied markedly between studies, depending on the setting and type of intervention and the diagnostic process involved. In the sections below, we summarize all of the 43 selected studies according to the five interactive steps of the diagnostic process (13).

Table 3. Proposed System-Related Intervention Ideas to Address Multiple Dimensions of Diagnostic Error.

Intervention (Suggested or Tested)	Suggested	Tested
Patient-provider Encounter [2]
Trauma response team		Perno et al.(15)
Comprehensive reexamination in ED		Howard et al. (16)
Diagnostic Test Performance and Interpretation [1]
Availability of electronic systems for results delivery		Weatherburn et al. (17).
Follow-up and Tracking [15]
Explicit criteria for communication of abnormal test results	Gandhi (21), Hanna et al. (28)
Planned follow-up to any test	Berner and Graber (1), Schiff (27)
Test-tracking system for ordering providers (electronic or not)	Gandhi (21), Singh et al. (26), Schiff and Bates (22); Casalino et al. (23);	Singh et al. (18); Poon et al. (19), Piva et al. (20)
Improved standardization of the steps involved in the flow of test result information	Wahls and Cram (24)
Improve the management and presentation of test result data	Wahls and Cram (24)
Use the emergency department manager to monitor radiology test results reporting	Emergency Department Manager (57)
Report discrepancies in radiology reports to emergency department	Emergency Department Manager (57)
Establish back-up processes so that any information about test results can be easily retrieved again	Emergency Department Manager (25)
Establish highly structured hand-offs that are performed systematically	Emergency Department Manager (25)
Systematic tracking of diagnostic error in organization	Colgan (44); Schiff et al. (2)
Referral Related [1]
Ensure availability of appropriate expertise	Emergency Department Manager (25)
Patient Related [18]
Address patient preferences for receiving test results	Leekha et al. (29); Meza and Webster (30); Dolan et al. (31); , Karnieli- Miller (32); Keren et al. (33)
Communicate normal test results	Baldwin et al. (34); Keren et al. (33)
Use automated test results management tool	Matheny et al. (35)
Use online portal to access test results	Wald et al. (58); Ross et al. (59)
Provide access to entire medical record	Ross and Lin (60)
Consider cognitive limitations when taking patient history	Redelmeier et al. (39)
Consider communication strategies to optimize patient understanding of medical information	Michie et al. (61); Lavin et al. (40)
Enhance patient engagement in health care	Schwappach (36); Longtin et al. (37)
Greater involvement of patients to ensure the follow-up of test results	Wahls and Cram (24), Emergency Department Manager (57)
Patient navigator	Singh et al. (26)
General interventions [12]
Provide education on error- producing conditions like fatigue	Caldwell (62); Campbell et al. (63); Singh et al. (64); Beach et al. (65); Jones and Endsley (66) Borrell-Carrio and Epstein (67)
Provide opportunity to correct last response	Fleck and Mitroff (68)
Address environmental conditions that could produce boredom, time pressure, etc.	Tachakra (69); Zwaan et al. (43)
Use of information technology	Becich et al. (45); Singh et al. (46); Schiff and Bates (22)

Table 4. Studies that Tested System Interventions to Address Dimensions of Diagnostic Error.

Provider-patient Encounter
Perno, J. F., et al (2005). Significant reduction in delayed diagnosis of injury with implementation of a pediatric trauma service. Pediatr Emerg Care, 21(6), 367-371.	UBA	Designated pediatric trauma response team	48 months	Care team	Unknown care teams; A total of 3265 patients were included; no patients were excluded.	Incidence of delayed diagnosis of injury (DDI) among pediatric trauma patients.	Y; speculated reasons included team dedicated only to trauma
Howard, J., et al. (2006). Reducing missed injuries at a level II trauma center. J Trauma Nurs, 13(3), 89-95.	Post- test only	Comprehensive reevaluation (i.e., tertiary examination) of trauma patients within 24 hours of admission	6 months	A trauma clinical nurse specialist, 2 emergency physicians, and the trauma medical director	4 healthcare providers, 90 patients	Incidence of missed injuries	Y; tertiary “repeat” examination and review of all lab and radiology studies
Diagnostic Tests
Weatherburn, et al (2000). The effect of a picture archiving and communications system (PACS) on diagnostic performance in the accident and emergency department. (J Accid Emerg Med, 17(3), 180-184.)	CBA	Implementation of Picture Archiving and Communications System (PACS), which acquires, transports, and stores radiographic images electronically, with accident and emergency (A&E) clinicians	Pre-PACS data collection period based on conventional film images: 3/ 31/92 to 10/30/92; Post- PACS data collection period: 4/1/96 to 10/30/96	Accident and emergency (A&E) department.	# of A&E attenders: 14,256 (film), 17,071 (PACS)	Misdiagnosis (false negative) rates for adults and children	Y; Speculated reasons include 1) clinicians could manipulate soft copy images in PACS 2) potential for images to be viewed simultaneously in A&E and Radiology.pro mpting more consultations
Follow-up and Tracking
Singh, H., et al. (2009). Improving follow-up of abnormal cancer screens using electronic health records(BMC Med Inform Decis Mak, 9, 49)	UBA	Added a code to the software configuration that links patients to their PCP for tests ordered by others.	10 months	Primary care physicians	One large urban facility and satellite clinics; 490 alerts	Rates of timely follow- up of positive FOBTs pre- and post- intervention	Y; improved electronic communication of abnormal test results
Poon, E. G., et al (2002). Real-time notification of laboratory data requested by users through alphanumeric pagers. (J Am Med Inform Assoc, 9(3), 217-222).	Post- test only	Implementation of Result Notification via Alphanumeric Pagers (ReNAP), an application that notifies clinicians of patient laboratory results via an alphanumeric pager once results are filed into the patient database.	12 months	Inpatient and clinic physicians	During the 12 month period between Feb 2000 and Jan 2001, 780 different clinicians used ReNAP; a total of 22,775 requests were made during this time period.	# of laboratory notification requests made, user satisfaction scores	Y; improved electronic communication of test results
Piva, E., et al. (2009). Evaluation of effectiveness of a computerized notification system for reporting critical values. (Am J Clin Pathol, 131(3), 432- 441.)	UBA	Implementation of a computerized notification system for critical lab values (email, text message, video alert)	2 months	Clinicians	14 Departments (including Emergency Department) in one large hospital	Percentage of successful notifications (acknowledge d within 1 hour), time to notification.	Y; improved electronic communication of abnormal test results

Key (study design):

• ■UBA (Uncontrolled before and after study)
• ■CBA (Controlled before and after study)
• ■Post-test only (measures only taken after intervention was implemented)

The patient-provider encounter

Two studies were related to diagnostic error during this step. In both studies, the goal of the intervention was to avoid delayed or missed diagnosis of traumatic injuries through changes in care processes. Perno et al. (15) described the implementation of a pediatric trauma response team, whereas Howard et al. (16) implemented a comprehensive reevaluation of trauma patients within 24 hours of admission. Both interventions produced positive results: implementing the pediatric response team significantly reduced delayed diagnosis of injury (15), and tertiary examination of trauma patients identified significantly more previously missed injuries. (16)

Diagnostic test performance and interpretation

One study tested an intervention to prevent diagnostic errors related to diagnostic test performance and interpretation. This trial, conducted in the Emergency Room (ER) setting, focused on the implementation of a Picture Archiving and Communications System (PACS), which electronically acquires, transfers, and stores radiographic images. (17) Using PACS improved diagnostic performance by reducing the overall misdiagnosis rate, although the rate of serious misdiagnosis did not change.

Follow-up and tracking

A number of papers focused on timely follow-up of test results, modes for follow-up, and outcomes. Of these, three described actual interventions: Singh et al. (18) examined the reliability of electronically communicating positive fecal occult blood test results in a system where over a third of results were not followed-up. After identifying and correcting a software misconfiguration in the electronic health record that prevented communication of test results to primary care providers, they found that timely follow-up increased significantly and was sustained at month four following the intervention. Poon and colleagues designed Result Notification via Alphanumeric Pagers (ReNAP), an application that enables clinicians to indicate preferences for notification of patient-specific laboratory test results via an alphanumeric pager. (19) ReNAP was well received, with 780 different clinicians using the feature within a 12-month period and usage averaging 2,300 times per month. Piva and colleagues reported that a computerized test result notification system improved communication of critical laboratory values. Computerized notification was both faster (average of 11 minutes) and more successful (90% notification success rate within 1 hour) as compared to standard telephone notification only (average time of 30 minutes to notification, with less than 50% success within 1 hour). (20) Although these usage statistics implied an improvement in delivery of test results, in the latter two studies no actual data on follow-up of the delivered information was provided. Taken together, the studies illustrated the potential of technology and monitoring to improve transmission of important diagnostic information to clinicians, although no evidence of reduced diagnostic delays was provided.

Many articles suggested potential strategies to prevent test results from being lost to follow-up. Suggested system interventions included both processes to facilitate appropriate follow-up (e.g., explicit communication policies for test results, highly structured hand-off procedures, and pre-planned follow-up for any diagnostic test) and structural changes such as use of electronic tracking systems and patient navigation programs. (1, 21-27) Hanna et al., (28) for instance, described a broad intervention intended to facilitate improvement in communication of test results across multiple hospitals within Massachusetts. The Massachusetts Coalition for the Prevention of Medical Errors and the Massachusetts Hospital Association created a consensus group to identify the tests and the abnormal test results that should be considered critical and communicated in a timely manner, and the groups distributed this “starter set” to hospitals statewide.

Referral related

Although we did not find any tested interventions in this category, strategies to ensure availability of appropriate expertise have been suggested as interventions to reduce diagnostic error (e.g., when there is no radiologist to read films overnight from the ER) (25).

Patient related

The literature suggested several strategies to reduce error by better engaging and communicating with patients, although none of these were tested. Seven studies measured patient satisfaction and preferences with various methods of test result notification. (29-35) Although not focused on diagnostic errors, two recently published literature reviews summarized the effectiveness and feasibility of patient engagement as a potential intervention for error prevention. Schwappach (36) identified several key forces that promote patient engagement, including beliefs about self-efficacy, behavioral control, and the perceived ability to help prevent adverse incidents. Longtin et al. (37) concluded that, while patient engagement has been well documented in studies of decision making and chronic disease management, patient participation in error prevention has not been explored. The authors provided a conceptual model of factors that influence patient participation in preventing errors.

Finally, other suggested interventions have focused on improving patient education and communication between patients and providers to reduce errors. Two articles emphasized the need to anticipate patients’ potentially faulty interpretations or reasoning during the diagnostic process. (38, 39) Another study suggested that better adherence to future care for abnormal Papanicolaou smears might result when adolescents visit a clinic prior to their follow-up colposcopy appointments. (40) This literature demonstrates a need to consider the patient’s perspective in designing interventions to reduce diagnostic errors.

General interventions (not specific to any step)

A number of articles suggested possible strategies to ameliorate “error-producing conditions” that contribute to diagnostic error. Although we found no studies of actual interventions with these aims, many suggested interventions included structural and/or process changes to complement or improve providers’ diagnostic performance. These included specific strategies such as second readings of key diagnostic tests, clinical decision support, and feedback to clinicians on their diagnoses; many of these are discussed in detail in the companion paper, (1, 27, 41) as is general re-design of the working environment to produce better decision-making. (42) Zwaan et al. (43) suggested methods to evaluate such interventions by measuring both the “suboptimal cognitive acts” that could lead to diagnostic error (e.g., not ordering a recommended test) and physicians’ workload and fatigue at the time that they made the diagnosis.

Other publications suggested interventions to reduce diagnostic errors by learning from errors encountered locally. Schiff et al. (2) described the use of physician reports to identify and analyze diagnostic errors and suggested that organizations could identify potential preventive strategies through a similar process. Similarly, Colgan (44) discussed the potential value in mandatory disclosure and review of all diagnostic errors encountered in a cohort of surgical and cytopathology cases. Articles also discussed the potential application of information systems to reduce diagnostic errors. Becich et al. (45) reviewed the opportunities for pathology informatics to enhance patient safety. Singh et al. (46) examined the range of potential communication breakdowns during the diagnostic process that can lead to error and identified opportunities for information technology to reduce these breakdowns. Finally, Schiff and Bates (22) focused on multiple ways in which electronic health records can aide in the prevention of diagnostic errors, provided they are designed and used appropriately.

DISCUSSION

Our literature review of systems-related interventions to reduce diagnostic error published in the last decade yielded very few empirical outcome studies. Because system-based interventions are favored by many as the preferred approach for addressing diagnostic error, the results of our review are rather surprising. (14) Our findings highlight a large gap between suggested interventions and those that have been operationalized and evaluated empirically. Many interventions suggested were already close to implementation, if not already underway, but lacked data to support their effectiveness in reducing diagnostic error. For instance, systems-based interventions based on electronic health records and health information technology have received a great deal of attention, but compelling studies are relatively few. Nevertheless, a handful of system interventions that were tested (e.g., an electronic system to acquire, transfer, and store radiographic images (17) and process-of-care changes in emergency settings) demonstrated some degree of effectiveness in reducing diagnostic error. (15, 16) Interventions to promote more “patient centered” care (e.g., empowering patients in their diagnostic process) represent another concept which, though broadly accepted, has not been tested as a means of reducing error.

Although patients constitute an important and largely neglected resource for improving outcomes related to diagnostic error, no empirical study found during our review examined the direct effect of patient-related interventions. For example, directly notifying patients of abnormal test results has been suggested as a reliable back-up process to help ensure that important results are not missed, but this has not been formally tested. Another interesting example of how patients can be engaged in this context is the now-mandatory reporting of all mammography reports directly to the patient (47).

Our review has several implications for future interventions to reduce diagnostic error. Despite the high volume of care delivered in the primary care setting, few intervention studies directly addressed the primary care work-system. The dearth of such studies was surprising because several intervention ideas for the primary care setting had been well conceptualized in the literature. These promising but as-yet untested strategies include improving follow-up and tracking of abnormal or critical test results, improving hand-off processes, systematic tracking of diagnostic errors and implementing rapid patient follow-up on certain high-risk initial diagnoses. Many of these are ripe for testing and implementation.

Advances in other areas of patient safety over the last decade have not been systematically applied to the science of diagnostic error reduction. One area we particularly found largely absent from the literature was the science of human factors. (48) To reduce mismatches between system-based interventions and the capabilities of providers and patients who interact with them, human factors principles should be applied to the design and development of future interventions. For instance, rapid prototyping techniques could be used to identify awkward and confusing interfaces, while testing the interventions in simulated or actual clinical settings might help identify unintended consequences. (49, 50) Better designs could help ensure that once an error occurs, it does not cascade through the entire multi-stage diagnostic process. Design of other health IT-based applications could also benefit from these same principles. For example, EHR-based intervention design must take into account not only the technology (software, hardware, content of data, information, and knowledge, user interface) but also the workflow in which it will be implemented, the people who use and implement it, the organization in which it will be implemented, and the external legal and regulatory influences in play. (51) Taking into account this interactive “socio-technical” perspective will allow development of concomitant strategies to build resilience into the EHR work-system and mitigate harm, if it occurs. (52) Thus, the fields of cognitive science, informatics, human factors and engineering must come together to design some of these health IT interventions.

Testing and implementation of interventions other than IT to reduce diagnostic error in real-world practice will also need to take into account contextual factors that might affect their success. (53) Factors such as policies and procedures, safety culture, organizational and teamwork related factors could have a substantial impact on effectiveness of systems-based interventions to reduce diagnostic errors. For example, implementing and testing a diagnostic error reporting system for physicians requires significant institutional commitment, (54) and this might not be possible to obtain in many institutions. Recent evidence suggests that most of these contextual factors are generally not reported. (53) Measurement and analysis of contextual factors that affect testing or implementing these interventions might be challenging, but it would provide others useful information for applying these interventions to their own settings. (53)

Our review also highlights some of the main challenges in designing future interventions and studies to measure their impact. First, because of the multifaceted nature of these errors, and the fact that there are many other complex variables involved, the actual intervention effect might be difficult to demonstrate. Second, the impact of interventions on improving patient safety might be difficult to estimate because most studies did not specifically link errors or outcomes (such as delays) to adverse events. Although some robust methods to capture specific aspects of diagnostic error, such as timeliness of diagnosis, have been used, these “process measures” might not always link to reliable clinical outcomes. The few studies that did measure outcomes in terms of an actual diagnostic error rate focused on very specific clinical scenarios (e.g., missed trauma injuries), measurement of which does not generalize broadly across care settings or disease conditions. (15, 16) In general, measurement science (definitions and rigorous process/outcome measures) in this area remains underdeveloped. Third, observational studies were most commonly used to measure outcomes before and after an intervention, with a small number of clinicians or health care sites, without a control group. Controlled study designs are desirable, but not always called for or practical. Fourth, although we categorized interventions in one of five process steps to account for systems-related diagnostic processes, design, and implementation of interventions to reduce diagnostic error in real-world practice should also account for potential interaction between two or more of these steps. (55) As evident in several studies that we could not categorize (general category), it’s not always possible to categorize interventions according to these steps.

Our review had several limitations. Although distinguishing system-related from cognitive interventions facilitates understanding of diagnostic errors and discussion of possible interventions (Graber et al; companion paper), we acknowledge that most diagnostic errors involve complex etiologies that are related to both system and cognitive performance. (56) We could not delineate how the systems-based interventions impacted providers’ cognitive and perceptual capabilities. System-based interventions to facilitate clinical decision-making (e.g., implementation of electronic clinical decision support systems) fall into this category and are discussed in detail in the companion paper. We also used restrictive search criteria to identify literature specific to diagnostic errors or delays. As a result, we likely missed several key papers, especially when interventions were suggested in contexts that were not directly related to diagnostic error. Lastly, we focused largely on studies after the year 2000 in an attempt to capture progress made in the field in the last decade, but in doing so may have excluded earlier important work from our review.

In conclusion, our review summarizes the state of the science in the design of future interventions to reduce diagnostic errors in health care. In light of the gaps in knowledge demonstrated in the recent literature, we believe that future studies should be multifaceted, focus on real-world clinical practice, and aim to measure the direct effects of interventions on rates of errors in diagnosis. Advancing the science of diagnostic error prevention will require more robust study designs and rigorous definitions of diagnostic processes and outcomes to measure intervention effects.