Abstract
Objective:
Diagnostic errors (DEs) can harm critically ill children. However, we know little about their prevalence in pediatric intensive care units (PICUs) and factors associated with error. The objective of this pilot study was to determine feasibility of record review to identify patient, provider, and work system factors associated with DEs during the first 12 hours after PICU admission.
Design:
Pilot retrospective cohort study with structured record review using a structured tool (Safer Dx instrument) to identify DE.
Setting:
Academic tertiary referral PICU.
Participants:
Patients 0-17 years old admitted non-electively to the PICU.
Measurements and Main Results:
Four of 50 (8%) patients had DEs in the first 12 hours after admission. The Safer Dx instrument helped identify delayed diagnoses of chronic ear infection, increased intracranial pressure (2 cases), and Bartonella encephalitis. We calculated that 610 PICU admissions are needed to achieve 80% power (α=0.05) to detect significant associations with error.
Conclusions:
Our pilot study found 4 patients with DE out of 50 children admitted non-electively to a PICU. Retrospective record review using a structured tool to identify DEs is feasible in this population. Pilot data are being used to inform a larger and more definitive multi-center study.
Keywords: pediatrics, critical care, diagnosis, diagnostic error, misdiagnosis, patient safety
INTRODUCTION
Diagnostic errors (DEs) are prevalent and can be harmful (1). Diagnosing critical illness that leads children to be admitted to pediatric intensive care units (PICUs) is challenging because children vary widely in age and development, cannot fully articulate symptoms, and are acutely ill from a broad range of problems (2). Major diagnoses affecting survival were missed in 20% of autopsied patients who died in PICUs (3). Harmful DEs affected 21% of patients discussed at a PICU morbidity and mortality conference with > 50% of DEs occurring within the first 12 hours of admission (4). In a survey, 81% of pediatric cardiac ICU clinicians reported that they witnessed DEs harm children more than 5 times each year (5).
Despite these significant consequences, little is known about the prevalence of DEs and patient, provider, and work system factors associated with error. Our long-term goal is to perform a definitive study investigating the epidemiology of PICU DEs as a necessary step towards designing effective surveillance and interventions. The objective of this pilot study is to determine feasibility of record review to identify patient, provider, and work system factors associated with DEs during the first 12 hours after PICU admission.
MATERIALS AND METHODS
We conducted a pilot retrospective cohort study by reviewing electronic health records (EHRs) of patients non-electively admitted to an academic tertiary referral PICU. This work was approved by the local Institutional Review Board.
We included 50 patients 0-17 years old admitted consecutively to the PICU over one month. We excluded scheduled admissions (lower DE risk than patients with acute illness) (6) and patients admitted by the clinician reviewer (to minimize bias in review).
DEs were defined as missed opportunities to make a correct diagnosis based on evidence available during the first 12 hours after PICU admission, regardless of patient harm (7, 8). Missed opportunities in diagnosis (MODs) consist of preventable breakdowns within the diagnostic process (given information available at particular time points) and may be missed by clinicians, the system, and/or patients/family members (9). Diagnostic process breakdowns were identified for each DE.
To determine occurrence of DEs, a single clinician reviewer (CC) used the Safer Dx instrument to conduct structured reviews of patients’ EHRs (Supplemental Digital Content 1). This instrument was validated in an adult outpatient cohort (71% sensitivity, 90% specificity) (10) and has been used in the PICU with an inter-rater reliability of κ=0.72 (6). Clinician reviewers familiar with the setting are needed for accurate determinations of error (11). On average, our reviewer required 15 minutes (10-20 minutes) to review one record with no DE and 35 minutes (20-45 minutes) for one with a DE. We separately collected data on patient, provider, and work system factors potentially associated with DE.
We reported descriptive statistics and performed statistical comparisons of factors between patients with/without DE. We calculated the sample size needed to determine associations of factors with DE (80% power, α=0.05). Statistical analyses were performed using SAS 9.4 ©2017 (Cary, NC, 2017).
RESULTS
Fifty of 97 PICU admissions were included. We excluded 37 scheduled admissions, 4 patients ≥18 years old, and 6 patients admitted by the clinician reviewer.
Four (8%) patients had DEs. Chronic otitis media was missed in an infant while addressing respiratory failure. Recognition of increased intracranial pressure (ICP) was delayed when asymmetric pupils were attributed to cranial nerve inflammation in a patient with ischemic stroke. Recognition of increased ICP was delayed when it was thought unlikely given a pre-existing ventriculo-peritoneal shunt in a patient with head trauma. Bartonella encephalitis was not considered in a seizing patient with distinctive historical features. The first three (75%) patients were diagnosed correctly >24 hours from admission (range 24-72 hours). Eleven diagnostic process breakdowns were identified leading to DEs. Six of 11 (55%) breakdowns involved suboptimal actions based on information documented in the history or suboptimal diagnostic evaluation given available clinical information (Table 1).
Table 1.
Diagnostic Process Breakdowns Contributing to Diagnostic Error in the First 12 Hours after Pediatric Intensive Care Unit Admission
| Diagnostic Process Breakdowns (Items from Safer Dx Instrument) | PICU Admissions with Diagnostic Errora | |||
|---|---|---|---|---|
| Missed chronic ear infection | Delayed diagnosis of elevated ICP | Delayed diagnosis of elevated ICP | Delayed diagnosis of infectious encephalitis | |
| Item #1: Based on patient history that was documented during the first 12 hours after PICU admission, an opportunity to make the subsequent final diagnosis was missed. | X | X | X | |
| Item #2: Based on patient physical exam that was documented during the first 12 hours after PICU admission, an opportunity to make the subsequent final diagnosis was missed. | X | |||
| Item #3: Based on diagnostic data (laboratory, radiology, pathology or other results) that were available or documented during the first 12 hours after PICU admission, an opportunity to make the subsequent final diagnosis was missed. | X | |||
| Item #4: The diagnostic process during the first 12 hours after PICU admission was affected by incomplete or incorrect clinical information given to the care team by patient or caregiver. | ||||
| Item #5: The clinical information (i.e., history, physical exam and diagnostic data) present during the first 12 hours after PICU admission should have prompted additional diagnostic evaluation through tests or consults. | X | X | X | |
| Item #7: Alarm symptoms or “red flags” (i.e., features in the clinical presentation that are considered to predict serious disease) were not acted upon during the first 12 hours after PICU admission. | X | |||
| Item #8: Diagnostic data (laboratory, radiology, pathology or other results) available or documented during the first 12 hours after PICU admission were misinterpreted in relation to the subsequent final diagnosis. | ||||
| Item #9: The differential diagnosis documented during the first 12 hours after PICU admission did not include the subsequent final diagnosis. | X | X | ||
PICU - pediatric intensive care unit; ICP - intracranial pressure
An “X” denotes that the particular diagnostic process breakdown was present for that case with diagnostic error.
All 4 patients with DE were admitted on the nightshift compared with 17 of 46 (37%) patients without DE (p=0.026). Two (50%) patients with DE had discrepancies between admission and discharge diagnoses (chronic ear infection and Bartonella encephalitis), while 1 (2%) without DE had a discrepancy (cardiac arrest from arrhythmia later diagnosed with cardiomyopathy) (p=0.049) (Table 2). We did not identify other statistically significant differences likely because of low power.
Table 2.
Patient, Provider, and Work System Factors and Diagnostic Error in the First 12 Hours after Pediatric Intensive Care Unit Admission
| Factors | No Diagnostic Error (n=46) | With Diagnostic Error (n=4) | p valuea |
|---|---|---|---|
| Patient Factors | |||
| Age, years, median (IQR) | 3.3 (0.7-11.2) | 6.1 (1.0-11.4) | 0.94 |
| Sex, female, n (%) | 21 (46) | 2 (50) | 1.00 |
| PRISM III score-based risk of mortality, %, median (IQR) | 0.5 (0.3-0.6) | 0.7 (0.6-0.9) | 0.06 |
| Top chief complaint categories, n (%) | |||
| Respiratory | 14 (30) | 1 (25) | 0.41 |
| Neurologic | 9 (20) | 3 (75) | |
| Cardiovascular | 3 (7) | 0 | |
| Required life support interventions/emergency procedures in 1st 12 hours, n (%) | |||
| Mechanical ventilationb | 10 (22) | 2 (50) | 0.24 |
| Vasoactive infusions | 1 (2) | 1 (25) | 0.16 |
| Emergency proceduresc | 7 (15) | 0 | 1.00 |
| Top discharge diagnosis categories, n (%) | |||
| Respiratory | 14 (31) | 1 (25) | 0.35 |
| Neurologic | 9 (20) | 2 (50) | |
| Toxicologic/Trauma | 8 (17) | 0 | |
| Infectious | 3 (7) | 1 (25) | |
| Length of PICU stay, days, median (IQR) | 1 (1-3) | 6.5 (2-12) | 0.09 |
| Length of hospital stay, days, median (IQR) | 4 (2-11) | 13.5 (8.5-16) | 0.09 |
| Death in-hospital, n (%) | 2 (4) | 0 | 1.00 |
| Provider Factors | |||
| Years in practice of admitting attending physician, median (IQR) | 3 (1, 5) | 4 (2, 13) | 0.64 |
| Number of subspecialty consultations requested on admission, mean (SD) | 0.6 (0.9) | 0 | 0.20 |
| Work System Factors | |||
| Admission source, n (%) | |||
| UIHC emergency department | 16 (35) | 0 | 0.29 |
| UIHC-affiliated clinic | 2 (4) | 0 | |
| UIHC inpatient unit | 8 (17) | 0 | |
| Other institution | 20 (44) | 4 (100) | |
| Day of admission, weekend, n (%) | 21 (46) | 2 (50) | 1.00 |
| Time of admission, night shift, n (%) | 17 (37) | 4 (100) | 0.026 |
| Comparison of PICU admission diagnosis vs. discharge diagnosis, n (%)d | |||
| Identical | 37 (80) | 2 (50) | 0.049 |
| Difference in specificity | 5 (11) | 0 | |
| Difference in hierarchy | 3 (7) | 0 | |
| Diagnostically different | 1 (2) | 2 (50) |
PICU - pediatric intensive care unit, IQR - interquartile range, PRISM - Pediatric Risk of Mortality, SD - standard deviation, UIHC - University of Iowa Hospitals and Clinics
Statistical comparisons performed using t test, Wilcoxon rank-sum test, and Fisher’s exact test.
Invasive positive pressure ventilation; excludes patients on chronic mechanical ventilation via tracheostomy
Includes endotracheal intubation, chest tube placement, dialysis catheter placement, and other emergency bedside/operative procedures; excludes central/arterial line placement
Identical - the two diagnoses are either verbatim or medically identical; Difference in specificity - the discharge diagnosis is more specific than the admission diagnosis, but otherwise identical; Difference in hierarchy - the admission diagnosis is listed among the discharge diagnoses, but is not the primary listed diagnosis; Diagnostically different - the admission diagnosis is not listed among the discharge diagnoses or the two diagnoses are medically different (15)
We calculated the minimum sample size needed (80% power, α=0.05) to detect a significant association with DE for each predictor variable while accounting for clustering of DE by provider. We determined that 610 PICU admissions will provide sufficient power for all variables (Supplemental Digital Content 2).
DISCUSSION
Within the limitations of a pilot, we found 8% of patients had a DE. This is comparable to a 7% DE rate found in an adult ICU (12). One study found a 12% DE rate in a selected high-risk PICU subpopulation (6). We identified errors in diagnosing neurologic and infectious conditions similar to common misdiagnoses in adult (12, 13) and pediatric ICUs (3, 4). None of our patients with DEs died, emphasizing the importance of including all PICU patients (not just autopsied ones) in DE research (14).
DEs on PICU admission are not rare, thus we should be able to identify associated factors by reviewing a feasible number of records. Nightshift admissions may play a role, consistent with studies finding worse outcomes with after-hours admission (15, 16). Prior studies also found associations between DE and patient age (17, 18), longer hospital stays (19), and severe illness (6, 12). Our pilot study lacked power to confirm these associations.
Half of patients with DE had discrepancies between admission and discharge diagnoses, which highlights a limitation of using these discrepancies as a flag for DE (19, 20). In the PICU, important missed diagnoses may be complications of the main diagnoses (e.g., elevated ICP in stroke), which do not change from admission to discharge. Alternatively, we noted a patient without DE but with a diagnostic discrepancy (arrhythmia later diagnosed with cardiomyopathy). Diagnoses can evolve; therefore, discrepancies between admission and discharge diagnoses may not be accurate in DE screening.
Finally, we confirmed that the Safer Dx instrument is relatively easy to apply in a PICU setting. Assuming 10% DEs, we estimate approximately 173 hours will be needed to review 610 records. Pilot data are being used to inform a definitive multi-center study (4 PICUs with at least 2 clinician reviewers per site reviewing a total of 1000 records). Because record reviews are resource-intensive, our goal with this foundational work is to determine common PICU DEs and identify potentially modifiable factors associated with DE so we can develop targeted surveillance systems and interventions. Although there are automated tools for identifying adverse events (21, 22), these do not currently screen for DEs. Our work can inform the development of electronic triggers to identify a select group of records to review for DEs (23).
Limitations include the retrospective design, which relies on documentation accuracy. We were unable to precisely quantify diagnostic delays because events were not documented in real time; however, a delay of > 24 hours would generally be considered an opportunity for improvement. The Safer Dx instrument, although validated in an adult outpatient cohort, has not been fully validated in the PICU, which may have affected results. We are unable to fully determine why DEs occurred without observing workflow and interviewing staff. Some factors contributing to DE (e.g., teamwork) are not captured in records and will be missed. Despite these limitations, this work contributes to the growing body of literature on critical care DEs.
CONCLUSIONS
Our pilot study found 4 patients with DE out of 50 children admitted non-electively to the PICU. Record reviews using a structured tool to identify DEs is feasible in the PICU. Pilot data are being used to inform a definitive multi-center study, which is a foundational step towards designing effective DE surveillance and interventions.
Supplementary Material
Supplemental Digital Content 1. Safer Dx Instrument
Details of the adapted Safer Dx Instrument are presented as they appeared in data collection forms used for the study.
Supplemental Digital Content 2. Sample Size Calculations
Details of sample size calculations are presented.
ACKNOWLEDGMENTS
The authors thank Kiera Deal for her assistance in identifying eligible patients and collecting demographic and clinical data.
Conflicts of Interest and Source of Funding:
Dr. Cifra is supported by the National Institutes of Health (NIH) through an Institutional K12 grant (HD027748) and an internal start-up grant from the University of Iowa Carver College of Medicine Department of Pediatrics. Dr. Singh is supported by the VA Health Services Research and Development Service (Presidential Early Career Award for Scientists and Engineers USA 14-274), the VA National Center for Patient Safety, the Agency for Health Care Research and Quality (R01HS022087), the Gordon and Betty Moore Foundation, and the Houston VA HSR&D Center for Innovations in Quality, Effectiveness and Safety (CIN 13–413). Support for data collection (REDCap®) and statistical analysis was provided through an NIH Clinical and Translational Science Award (UL1TR002537). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The remaining authors have no conflicts of interest.
Copyright Form Disclosure:
Dr. Cifra’s institution received funding from the National Institutes of Health (NIH)/NICHD and NIH/NCATS. Drs. Cifra, Eyck, and Reisinger received support for article research from the NIH. Dr. Eyck’s institution received funding from the NIH. Dr. Singh disclosed government work. Dr. Herwaldt’s institution received funding from 3M, PDI, and AHRQ. Dr. Dawson disclosed that he does not have any potential conflicts of interest.
Footnotes
This work was performed at the University of Iowa Stead Family Children’s Hospital Pediatric Intensive Care Unit.
Reprints will not be ordered.
REFERENCES
- 1.National Academies of Sciences, Engineering, and Medicine: Summary In: Improving Diagnosis in Health Care. Washington, D.C.: The National Academies Press; 2015. p. 1–18. [Google Scholar]
- 2.Roland D, Snelson E: “So why didn’t you think this baby was ill?” Decision-making in acute paediatrics. Arch Dis Child 2019; 104:43–48 [DOI] [PubMed] [Google Scholar]
- 3.Custer JW, Winters BD, Goode V, et al. : Diagnostic Errors in the Pediatric and Neonatal ICU: A Systematic Review*. Pediatr Crit Care Med 2015; 16:29–36 [DOI] [PubMed] [Google Scholar]
- 4.Cifra CL, Jones KL, Ascenzi JA, et al. : Diagnostic Errors in a PICU: Insights From the Morbidity and Mortality Conference. Pediatr Crit Care Med 2015; 16:468–476 [DOI] [PubMed] [Google Scholar]
- 5.Bhat PN, Costello JM, Aiyagari R, et al. : Diagnostic errors in paediatric cardiac intensive care. Cardiol Young 2018; 28:675–682 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Davalos MC, Samuels K, Meyer AND, et al. : Finding Diagnostic Errors in Children Admitted to the PICU. Pediatr Crit Care Med 2017; 18:265–271 [DOI] [PubMed] [Google Scholar]
- 7.Singh H, Schiff GD, Graber ML, et al. : The global burden of diagnostic errors in primary care. BMJ Qual Saf 2017; 26:484–494 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Medford-Davis L, Park E, Shlamovitz G, et al. : Diagnostic errors related to acute abdominal pain in the emergency department. Emerg Med J 2016; 33:253–259 [DOI] [PubMed] [Google Scholar]
- 9.Singh H: Editorial: Helping health care organizations to define diagnostic errors as missed opportunities in diagnosis. Jt Comm J Qual Patient Saf 2014; 40:99–101 [DOI] [PubMed] [Google Scholar]
- 10.Al-Mutairi A, Meyer AND, Thomas EJ, et al. : Accuracy of the Safer Dx Instrument to Identify Diagnostic Errors in Primary Care. J Gen Intern Med 2016; 31:602–608 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Singh H, Khanna A, Spitzmueller C, et al. : Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis 2019; [DOI] [PubMed] [Google Scholar]
- 12.Bergl PA, Taneja A, El-Kareh R, et al. : Frequency, Risk Factors, Causes, and Consequences of Diagnostic Errors in Critically Ill Medical Patients: A Retrospective Cohort Study. Crit Care Med 2019; [DOI] [PubMed] [Google Scholar]
- 13.Winters B, Custer J, Galvagno SM Jr, et al. : Diagnostic errors in the intensive care unit: a systematic review of autopsy studies. BMJ Qual Saf 2012; 21:894–902 [DOI] [PubMed] [Google Scholar]
- 14.Zwaan L, Singh H: The challenges in defining and measuring diagnostic error. Diagnosis 2015; 2:97–103 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bhonagiri D, Pilcher DV, Bailey MJ: Increased mortality associated with after-hours and weekend admission to the intensive care unit: a retrospective analysis. Med J Aust 2011; 194:287–292 [DOI] [PubMed] [Google Scholar]
- 16.Abella A, Hermosa C, Enciso V, et al. : Effect of the timing of admission upon patient prognosis in the Intensive Care Unit: On-hours versus off-hours. Med Intensiva 2016; 40:26–32 [DOI] [PubMed] [Google Scholar]
- 17.Mattsson B, Ertman D, Exadaktylos AK, et al. : Now you see me: a pragmatic cohort study comparing first and final radiological diagnoses in the emergency department. BMJ Open 2018; 8:e020230. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Newman-Toker DE, Moy E, Valente E, et al. : Missed diagnosis of stroke in the emergency department: a cross-sectional analysis of a large population-based sample. Diagnosis 2014; 1:155–166 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hautz WE, Kämmer JE, Hautz SC, et al. : Diagnostic error increases mortality and length of hospital stay in patients presenting through the emergency room. Scand J Trauma Resusc Emerg Med 2019; 27:54. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Wogan JM: ED follow-up: A comparison of admission and discharge diagnoses. Am J Emerg Med 2001; 19:249–251 [DOI] [PubMed] [Google Scholar]
- 21.Landrigan CP, Stockwell D, Toomey SL, et al. : Performance of the Global Assessment of Pediatric Patient Safety (GAPPS) Tool. Pediatrics 2016; 137. [DOI] [PubMed] [Google Scholar]
- 22.Stockwell DC, Landrigan CP, Toomey SL, et al. : Adverse Events in Hospitalized Pediatric Patients. Pediatrics 2018; 142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Murphy DR, Meyer AN, Sittig DF, et al. : Application of electronic trigger tools to identify targets for improving diagnostic safety. BMJ Qual Saf 2019; 28:151–159 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental Digital Content 1. Safer Dx Instrument
Details of the adapted Safer Dx Instrument are presented as they appeared in data collection forms used for the study.
Supplemental Digital Content 2. Sample Size Calculations
Details of sample size calculations are presented.