Contributing Factors for Pediatric Ambulatory Diagnostic Process Errors: Project RedDE

Supplemental Digital Content is available in the text.

Abstract

Background:

Pediatric ambulatory diagnostic errors (DEs) occur frequently. We used root cause analyses (RCAs) to identify their failure points and contributing factors.

Methods:

Thirty-one practices were enrolled in a national QI collaborative to reduce 3 DEs occurring at different stages of the diagnostic process: missed adolescent depression, missed elevated blood pressure (BP), and missed actionable laboratory values. Practices were encouraged to perform monthly “mini-RCAs” to identify failure points and prioritize interventions. Information related to process steps involved, specific contributing factors, and recommended interventions were reported monthly. Data were analyzed using descriptive statistics and Pareto charts.

Results:

Twenty-eight (90%) practices submitted 184 mini-RCAs. The median number of mini-RCAs submitted was 6 (interquartile range, 2–9). For missed adolescent depression, the process step most commonly identified was the failure to screen (68%). For missed elevated BP, it was the failure to recognize (36%) and act on (28%) abnormal BP. For missed actionable laboratories, failure to notify families (23%) and document actions on (19%) abnormal results were the process steps most commonly identified. Top contributing factors to missed adolescent depression included patient volume (16%) and inadequate staffing (13%). Top contributing factors to missed elevated BP included patient volume (12%), clinic milieu (9%), and electronic health records (EHRs) (8%). Top contributing factors to missed actionable laboratories included written communication (13%), EHR (9%), and provider knowledge (8%). Recommended interventions were similar across errors.

Conclusions:

EHR-based interventions, standardization of processes, and cross-training may help decrease DEs in the pediatric ambulatory setting. Mini-RCAs are useful tools to identify their contributing factors and interventions.

INTRODUCTION

Diagnostic errors (DE), defined by the National Academy of Science, Engineering, and Medicine as “the failure to establish an accurate and timely explanation of the patient’s health problem(s) or communicate that explanation to the patient,” are increasingly recognized as a critical threat to patient safety.^1–3 The National Academy of Medicine and the World Health Organization call for urgent systems change to address DEs in the primary care setting.^1,4 Despite this growing attention, minimal research identifies wherein the diagnostic process errors occur or the contributing factors of DEs in the pediatric population.

Studies suggest that DEs are complex and multifactorial events, which are the result of a combination of cognitive and systems factors.^5–8 A study of DE in internal medicine found that in 46% of cases analyzed, there were both systems and cognitive factors that contributed to the error, and on average, there were 5.9 contributing factors per case.⁶ Schiff et al⁵ found that major DEs occurred most often in the laboratory and during radiology testing phase and clinician assessment.

DEs frequently occur in the pediatric ambulatory setting. Studies have shown that 35%–54% of pediatric ambulatory providers report making a DE at least monthly and 33%–45% report making a DE that harms a patient at least annually.^9,10 In our previous epidemiologic study of 25 pediatric practices, DEs were found to occur in 54% of patients with elevated blood pressure (BP), 11% of patients with abnormal laboratory values, and 62% of adolescents with an opportunity to evaluate for depression.¹¹

Root cause analysis (RCA) is a methodology that provides a structured approach to analyze and learn from systems issues and human factors that contribute to adverse events.^12,13 RCAs can identify steps where a process has failed, create targets for improvement, and drive change in pediatric quality improvement collaboratives (QICs), such as those aimed at preventing central line–associated bloodstream infections and promoting best practices for asthma care.^14–16 Analysis of DEs in the adult primary care setting suggests that most commonly, they involve conditions that are frequently encountered in ambulatory practice.⁷ The process failures most often included cognitive errors during the patient–practitioner clinical encounter and issues with care coordination and tracking.^7,8 However, many gaps remain in our understanding of DEs. In particular, the failure points of pediatric ambulatory DE have not been well characterized, nor is it clear what interventions will have the highest yield to prevent DEs in this practice setting.

A greater understanding of the causes of DEs in the pediatric ambulatory setting is necessary to develop effective interventions. We had 3 primary aims for our analysis. First, we aimed to use the mini-RCA tool to characterize diagnostic process errors (DPEs), including failure points and contributing factors, for 3 ambulatory pediatric DEs across 31 clinics during 2 years of error reduction work.¹⁷ The DPEs represented failures at different stages of the diagnostic process to provide a model for understanding other errors in pediatric primary care. Second, we sought to elicit pediatric providers’ recommendations for potential interventions to prevent these DPEs in the future. These data provide information to clinicians, ambulatory system leaders, and researchers, highlighting possible intervention points for future DPE reduction work. Finally, from an implementation science standpoint, we also sought to understand how a diverse set of practices would use the mini-RCA tool during a QIC. This work adds to our previously published manuscripts from the Reducing Diagnostic Errors in Pediatric Primary Care (Project RedDE) QIC by providing an in-depth analysis of our mini-RCA data across the DPEs.

METHODS

The parent study, Project RedDE, sought to reduce 3 DPEs in primary care pediatric practices through participation in a multi-institutional QIC.^18–20 We previously published the complete methods used in Project RedDE.²¹ Practices were cluster randomized to receive a QIC intervention aimed at reducing each of these errors sequentially, in randomly assigned order during 1 of the 3 action periods. Participating practices worked from October 2015 to September 2017 to reduce these errors.

As described previously, we recruited 34 pediatric ambulatory practices in March 2015, and 9 practices dropped out after randomization. One of these practices left the study after 8 months due to their lead physician leaving the practice. Nine additional practices were recruited and randomized in December 2015; 2 of these practices dropped out and 2 practices from a single care network combined into 1 team to increase their sample size. Therefore, 31 practices had the opportunity to submit data for inclusion in this analysis.

As one component of the larger QIC intervention,²¹ practices were encouraged to perform monthly “mini-Root Cause Analyses” related to the error they were actively working to reduce. The goal of this tool was to help practices develop a “preoccupation with failure” and to learn from every error by identifying failure points and potential interventions to prevent reoccurrence.^22–24 Participants were taught how to use this tool, and they were given time to practice during day-long videoconference learning sessions. Practices chose the case that they thought was most relevant and important to the clinic each month and identified all failure points and contributing factors associated with that case. Although practices were encouraged to complete a mini-RCA monthly by project leadership and quality improvement coaches, they were not required to do so. Data from these mini-RCAs were shared during learning sessions and monthly videoconferences.

The mini-RCA tool, based on the Agency for Healthcare Research and Quality’s Learn from Defects Tool, was a structured form with questions designed to prompt team discussion and highlight root causes.²⁵ As the Learn from Defects Tool is focused on the inpatient setting, modifications were made to adapt the tool to the ambulatory environment, and the expert group selected changes to the potential contributing factors. Teams were asked to (1) describe the error, (2) identify the process step where the error occurred, (3) identify specific contributing factors to the error, (4) identify the 3 factors that contributed the most to the error, and then (5) identify interventions to reduce future risk of this error. Teams were selected from checkboxes with potential process step failure points and contributing factors; “other” was also an option, and they were given an opportunity to provide free-text responses. Specific contributing factors were split into 3 domains: patient/family, staff, and system (Table 1). Teams were provided the “Strength of Interventions” chart as a reference when recommending interventions.²⁵ De-identified data were entered monthly into an online survey platform (SurveyMonkey, San Mateo, CA). The complete mini-RCA form is available online (see Appendix 1, Supplemental Digital Content, available at http://links.lww.com/PQ9/A182).

Table 1.

Mini-RCA Categories for Contributing Factors

Error Descriptions

Missed Diagnosis of Adolescent Depression

Research suggests that 1 in 10 adolescents is depressed, and that many depressed adolescents are not diagnosed by primary care providers.^11,26–30 Data support that patients with documented signs and symptoms of depression in their medical record are typically referred for mental health treatment, but that this data collection methodology likely under-identifies the true prevalence of adolescent depression.³⁰ Therefore the missed opportunity for diagnosis of adolescent depression measure was identified as the improvement in the percentage of patients with a diagnosis of depression in a given clinic comparing the change in depression diagnosis rates between control and intervention clinics. Practices also tracked when a provider failed to document concerns for depression or exclude concerns for depression during a health supervision visit in patients 11 years or older, as recommended by the American Academy of Pediatrics.³¹ Failure to document or exclude concerns for depression was the primary trigger for a mini-RCA for this DPE.

Missed Elevated Blood Pressure

Studies suggest that providers fail to recognize elevated BP in 58% of patients.³² Participants reviewed charts to identify patients 3 years and older with an elevated systolic or diastolic BP recorded at their health supervision visit. We defined an elevated BP as ≥90% for age, height, and sex or ≥120 mm Hg systolic or 80 mm Hg diastolic at any age.³³ For this study, missed elevated BP occurred when a provider failed to document an appropriate action for a patient with an elevated BP. “Appropriate action” was defined broadly and included repeating the BP, planning to recheck BP at a subsequent visit, referral to a specialist, or additional BP-related testing and evaluation.

Missed or Delayed Response to Abnormal Laboratory Values

Primary care providers often order laboratory tests during patient visits, and missed abnormal results can lead to patient harm.^34–41 Missed or delayed response to abnormal laboratory values was limited to patients with specific abnormal results that are often received by pediatric practices but can cause harm if missed.²¹ These abnormal results include hemoglobin <11 g/dL and mean corpuscular volume <75 fL (ie, microcytic anemia) in 1- and 2-year-olds; lead >5 μg/dL in 1-, 2-, and 3-year-olds; any positive Neisseria gonorrhoeae, Chlamydia trachomatis, Treponema pallidum, or human immunodeficiency virus test in patients older than 10 years; positive group A Streptococcus throat culture with a negative rapid test in patients older than 1 year; and thyroid-stimulating hormone (TSH) <0.5 or >4.5 μIU/mL in patients aged 1 year and older. An error occurred when a provider was delayed in documenting an appropriate action in a patient with any of these abnormal results. We defined a delay as no appropriate action recorded within 30 days for microcytic anemia and elevated lead levels within 7 days for the other laboratory results. “Appropriate action” was defined broadly and specifically for each laboratory test, including starting treatment, family discussion of options, sending additional laboratory studies, or referral to a relevant specialist.

Analysis

We analyzed data using descriptive statistics, including medians and interquartile ranges and Pareto charts. Pareto charts are quality improvement tools that provide a graphic representation of the absolute count of data in each category, as well as the cumulative percentage that it contributes to the underlying problem. Pareto charts are used to identify the most common contributing factors to a problem to target the most high-yield interventions. Qualitative data submitted by practices on recommended interventions were analyzed thematically according to major categories. Themes were coded and tallied by DPE and compared across DPEs. We conducted analyses using Microsoft Excel Version 15 (Microsoft Corp., Redman, Wash.) and STATA 13 (StataCorp LLC, College Station, Tex.). The Institutional Review Boards of American Academy of Pediatrics and the Albert Einstein College of Medicine approved this study.

RESULTS

Practice Characteristics

Data from 31 practices were eligible for analysis. Eighteen (58%) practices were university affiliated. In 17 (55%) practices, 80% or more of their patients were publicly insured.

Mini-RCA Utilization

Twenty-eight (90%) practices submitted a total of 184 mini-RCAs. Practices provided between 0 and 15 mini-RCAs (Table 2). The median number of mini-RCAs submitted was 6 (interquartile range, 2–9). Missed adolescent depression was the most common DPE for which participants submitted an RCA (Table 3). Practices sent the most mini-RCAs in action period 1 when compared with action periods 2 and 3 (Table 3).

Table 2.

Frequency of Mini-RCAs Submitted

Table 3.

Frequency of Mini-RCAs by Diagnostic Error Type and Action Period

Missed Adolescent Depression

For missed adolescent depression, the process step most commonly identified as a failure point was the failure to screen (68%) (Fig. 1A). Top factors that contributed to errors included high patient volume (16%), inadequate staffing (13%), and language barriers (9%). Practices submitted 83 recommended improvement ideas resulting from 73 mini-RCAs for missed adolescent depression. Interventions focused on education, standardizing processes, and creating new workflows around the screening process and cross-training of staff (Table 4). Six (7%) of the suggested interventions addressed new workflows for screening adolescents with language barriers.

Fig. 1.

Pareto charts demonstrating process steps identified as failure points for each diagnostic error. A, Failure points for missed adolescent depression. B, Failure points for missed elevated blood pressure. C, Failure points for missed actionable laboratories.

Table 4.

Recommended Interventions by Diagnostic Error Type

Missed Elevated BP

For missed elevated BP, practices indicated that the process steps involved in the DPE were the failure to recognize (36%) and to act on (28%) abnormal BP (Fig. 1B). Top contributing factors included high patient volume (12%), clinic milieu (9%), and electronic health record (EHR) issues (8%) (Table 4). Practices submitted 77 recommended interventions on 53 mini-RCAs for missed elevated BP. Common themes, in addition to education/training, included modifying the EHR system; visual reminders or cognitive aids not in the EHR system such as stickers or cards on the providers’ workstation to flag that a patient has an abnormal BP; standardized processes and improving communication (Table 4). Specific types of EHR interventions centered around the inclusion of BP percentiles in vital signs and/or in note template, changing alert thresholds, creating hard stops in the EHR system for elevated BP, and changing note templates.

Missed Actionable Laboratories

For missed or delayed actionable laboratories, the process steps most commonly identified as failure modes were a failure to notify families of abnormal results (23%) and failure to document actions on abnormal results (19%) (Fig. 1C). Top factors that contributed to errors of missed actionable laboratories included written communication (13%), EHR issues (9%), and provider knowledge (8%). Practices submitted 62 recommended interventions derived from 58 mini-RCAs. Top intervention categories, in addition to training and education, included standardization and creation of new workflows, cross-training of staff, and EHR interventions (Table 4). EHR concerns seemed particularly frustrating for some providers due to difficulty in achieving resolution, one of whom free-texted “…will talk to [name of EHR] vendor; we are at their mercy.” EHR interventions focused on processes around laboratory inbox creation or coverage, incorporation of information technology–based reminders or hard stops, and creation of smart phrases to insert structured text and data into notes to facilitate documentation and prompt appropriate actions. One suggested intervention explicitly focused on human factors engineering and reducing the number of clicks and physical movement in the process of laboratory follow-up and family notification. Providers noted difficulty with patient enrollment in patient portals and viewed portals as a potential solution for family notification if able to increase participation. Adolescent and parent engagement was suggested in 4 recommended interventions for missed actionable laboratories.

DISCUSSION

Our previous work demonstrates that pediatric ambulatory DEs occur frequently.^9,11 This study builds upon that literature, showing that ambulatory pediatric DPEs at different stages of the diagnostic process have unique failure points. Furthermore, staff and providers at ambulatory pediatric clinics were able to successfully utilize mini-RCA tools as a method of identifying causal factors of DEs and suggesting potential interventions. In a convenience sample survey at the end of the project, clinics cited the mini-RCA as 1 of the 5 tools they found to be most useful to reduce BP errors.⁴²

Consistent with the adult DE literature, we found that the root causes of pediatric DPEs in the ambulatory setting were multifactorial.^5–8 Previous work by Schiff et al⁵ in using the Diagnostic Error Evaluation and Research (DEER) taxonomy of DE classification found that in major DEs, there was most often a failure of the clinician to consider a diagnosis followed by a failure to order needed tests. Although our study did identify that some failure points were related to cognitive factors, particularly in missed elevated BP where the failure to recognize and act on abnormal values was most prevalent, practices pointed to systems issues as critical components of the errors. Due to the substantial overlap between cognitive and system failures, some authors have advocated that this distinction is spurious and the 2 should not be treated as separated entities.⁴³ As a contribution beyond our prior work, our study also identified potential interventions to address specific DEs. Interventions suggested by practices included cognitive aids and human factors engineering approaches incorporated in the EHR to address provider analytic errors as essential mechanisms to reduce DE. These types of interventions make it easier for providers to “do the right thing.” Other interventions, particularly for depression and missed actionable laboratories, targeted patient-specific factors such as new workflows for patients with language barriers or developing methods for engaging adolescents and their families. Patient engagement in the diagnostic process is a concept that has also been supported by the National Academy of Medicine and the Society for Improving Diagnosis in Medicine.⁴⁴ Our data suggest that while there are commonalities across the different DEs, there are also unique aspects that require tailored interventions. Ongoing evaluation of errors and clinic processes with a tool such as the mini-RCA is helpful in continuous process improvement.

The Department of Veteran Affairs is a model health system in which in-depth RCA are conducted in the ambulatory setting for adverse events and close calls related to DEs.⁸ This process includes detailed interviews and meetings to identify the root causes and develop action plans. However, these full investigations can be time-consuming and resource-intensive and may not be feasible in smaller practice environments. Therefore, a scaled-down version, such as the mini-RCA format used in our study, may provide an effective and efficient process for enabling teams to identify causal factors and develop interventions without a significant time commitment. The majority of practices involved in our QIC submitted mini-RCAs, with over half of the practices submitting more than 6, suggesting that mini-RCAs were feasible and that participants found value in the process. There were more mini-RCAs provided in the first action period than in the subsequent 2 action periods, which may reflect the greater engagement of practices earlier in the collaborative or may suggest that interventions were more self-evident as they practiced quality improvement methodologies.

Our study findings must be considered within the context of several limitations. Some of the mini-RCA comments lacked specificity, limiting how we could interpret them and describe them in this article. We are unable to link specific causative factors to proposed interventions, and this could be a potential target for future work. We also could not assess how many teams carried out the interventions that they recommended as a result of their RCA. Studies have suggested that failure to follow through on action plans and to share lessons learned across the organization renders RCAs ineffective.⁴⁵ We were also unable to track the total amount of time teams spent on conducting each mini-RCA.

CONCLUSIONS

Ambulatory pediatric DPEs at different stages of the diagnostic process have unique failure points that can inform targeted interventions. Interventions identified from this group of practices to reduce ambulatory pediatric DPEs include standardizing processes, creating new workflows, and developing changes to the EHR system. Mini-RCAs are an effective and efficient tool to identify contributing factors to ambulatory DEs and help teams develop interventions to mitigate them.

DISCLOSURE

The authors have no financial interest to declare in relation to the content of this article.

ACKNOWLEDGMENTS

The authors gratefully acknowledge the contribution of all pediatric primary care sites who participated in this work and Drs. Brady, O’Donnell, Stein, Lehmann, Kairys, and Lilienfeld and Ms. Norton, as well as the American Academy of Pediatrics’ Quality Improvement and Innovations Network, and Drs. Orringer and Helms.