Abstract
Background:
Timely diagnosis of pediatric sepsis remains elusive. We estimated the risk of potentially missed pediatric sepsis in US emergency departments (EDs) and determined factors associated with its occurrence.
Methods:
In a retrospective study of linked inpatient and ED records from four states using administrative data (excluding 40% with missing identifiers), we identified children admitted with severe sepsis and/or septic shock who had at least 1 ED treat-and-release visit in the 7 days prior to sepsis admission. An expert panel rated the likelihood of each ED visit being related to subsequent sepsis admission. We used multivariable regression to identify associations with potentially missed sepsis.
Results:
Out of 1,945 patients admitted with severe sepsis/septic shock, 158 (8.1%, 95% CI: 6.9, 9.4%) had potentially missed sepsis during an antecedent treat-and-release ED visit. The odds of potentially missed sepsis were lower for each additional comorbid chronic condition (OR 0.86, 95% CI: 0.80, 0.92) and higher in California (OR 2.26, 95% CI:1.34, 3.82), Florida (OR 3.33, 95% CI:1.95, 5.70), and Massachusetts (OR 2.87, 95% CI:1.35, 6.09), compared to New York.
Conclusions:
Administrative data can be used to screen large populations for potentially missed sepsis and identify cases that warrant detailed record review.
Keywords: pediatrics, sepsis, diagnostic error, misdiagnosis, emergency department
Pediatric severe sepsis is the 10th leading cause of death among children (1). In 2005, there were 75,255 pediatric hospitalizations for severe sepsis in the US, costing $4.8 billion (2). Prompt sepsis diagnosis and timely interventions generally lead to favorable outcomes, while missed sepsis diagnosis contributes to increased mortality and length of hospital stay (3).
Sepsis is challenging to diagnose, as it occurs on a spectrum with no gold standard for early diagnosis. Over the years, efforts to improve pediatric outcomes by implementing “sepsis bundles” in the ED encouraged early recognition and aggressive management of suspected sepsis, increased adherence to management guidelines, and decreased mortality (4). It is unknown, however, whether these programs prevent sepsis misdiagnosis because there are no current population-based estimates of missed sepsis. Thus, our objective is to estimate the risk of potentially missed pediatric sepsis in US EDs and to identify patient and institutional factors associated with its occurrence.
We performed a retrospective cohort study of children 0–19 years admitted to an inpatient unit with severe sepsis and/or septic shock. We used a modified version of the Symptom-Disease Pair Analysis of Diagnostic Error (SPADE) method to identify misdiagnosis using administrative databases (5). Details of methods and results are included in an online Supplement. The local Institutional Review Board determined that this study was exempt from human subjects research oversight.
We used administrative data from four states (CA, FL, MA, NY) in 2010–2011 from the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project (HCUP) (6). We linked patient visits in both inpatient and ED settings using synthetic patient identifiers. Institutional characteristics were captured by merging data from the American Hospital Association’s Annual Survey in 2011 (7).
The primary outcome was potentially missed pediatric sepsis in the ED. Using International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) codes, we identified inpatients admitted with severe sepsis (995.92) and/or septic shock (785.52). This case-finding method has been validated with a sensitivity of 73%, specificity of 92%, and positive predictive value of 79% (8). We excluded records with missing unique identifiers, duplicate records, and subsequent inpatient sepsis admissions for the same patient. From this cohort, we then identified patients who had at least one ED treat-and-release visit in the 7 days prior to sepsis admission (preventable pediatric sepsis deaths often have up to 7 days of preceding symptoms (9)). We excluded ED patients transferred to an acute hospital for inpatient admission (not considered ED treat-and-release visits) and those with ICD-9 codes consistent with correctly diagnosed sepsis (see flowchart in Supplement).
To identify potentially missed sepsis cases, we convened a panel of 3 board-certified physicians (general pediatrician, pediatric emergency medicine physician, and pediatric intensivist). Given only the patient’s age and list of ED ICD-9 diagnoses (no other clinical data available), two panel members were asked to rate the likelihood that each patient’s ED visit was related to his/her subsequent sepsis admission. For discrepant ratings, a third panel member made the final adjudication. Interrater reliability (kappa statistic) was calculated. Patients with an ED visit that was rated as likely related to their sepsis admission were categorized as potentially missed sepsis cases. We performed analyses to assess the validity of our missed sepsis construct (see Supplement).
Patient and institutional characteristics of cases with unlikely missed sepsis vs. potentially missed sepsis were compared using Student’s t-test/Wilcoxon rank-sum test for continuous variables and Chi-square test/Fisher’s exact test for categorical variables (unadjusted analysis). We constructed generalized linear mixed models (GLMM) with random effects at the hospital level to examine the odds of potentially missed sepsis in the ED, controlling for patient and institutional characteristics (adjusted analysis). We used multiple imputation by chained equations (MICE) procedures to impute values for missing variables (10), in order to avoid dropping 331 (17%) cases from the analysis. We performed a complete case analysis to assess the sensitivity of our results to assumptions regarding missing data. For the final model, 23 records with missing hospital identifiers were dropped as we were unable to impute missing values due to lack of all hospital-level data (see Supplement).
Using these methods, we determined that out of 3,467,908 pediatric inpatient admissions, 3,464 patients were admitted with severe sepsis/septic shock (visit-level data). Of these sepsis admissions, 1,945 (56.1%) met all inclusion criteria (patient-level data). Of these patients, 198 (10.2%) had at least one ED treat-and-release visit (total of 219 ED visits) in the preceding 7 days prior to sepsis admission.
The most common diagnoses at prior ED treat-and-release visits were fever of unknown origin (30.6%), viral infection (13.7%), and other lower respiratory disease (13.2%) (full list in Supplement). Out of 219 ED visits, the expert panel identified 172 ED visits (in 158 unique patients) likely related to subsequent inpatient sepsis admission (potentially missed sepsis cases) with moderate interrater reliability (κ=0.61, p<0.001). This results in an estimated 8.1% (158/1,945, 95% CI: 6.9, 9.4) risk of potentially missed pediatric sepsis in the ED.
Table 1 shows patient and hospital characteristics by occurrence of potentially missed sepsis (unadjusted). Of the patient-level characteristics in multivariable models, only the presence of comorbid chronic conditions was significantly associated with a reduced odds of potentially missed sepsis (OR 0.86, 95% CI: 0.80, 0.92, p<0.001) (Table 2). Of the institution-level characteristics, only the geographic location of the ED was significantly associated with potentially missed sepsis. Compared to New York state, ED patients seen in California (OR 2.26, 95% CI: 1.34, 3.82, p=0.002), Florida (OR 3.33, 95% CI: 1.95, 5.70, p<0.001), and Massachusetts (OR 2.87, 95% CI: 1.35, 6.09, p=0.006) had significantly higher odds of potentially missed sepsis. A complete case analysis showed similar results.
Table 1.
Characteristics of pediatric patients admitted with severe sepsis/septic shock stratified by potentially missed diagnosis of sepsis in an emergency department visit within 7 days before inpatient admission (patient-level unadjusted analysis)
| Characteristic | Overall | Unlikely missed sepsis in EDa | Potentially missed sepsis in ED | p valueb |
|---|---|---|---|---|
| n = 1,945 | n = 1,787 | n = 158 | ||
| Patient Characteristics | ||||
| Sex, female (%) | 910 (47.1) | 831 (46.8) | 79 (50.6) | 0.352 |
| Age, years, mean (95% CI) | 8.2 (7.9, 8.5) | 8.2 (7.9, 8.5) | 8.4 (7.4, 9.3) | 0.680 |
| Age categories, years (%) | ||||
| Less than 1 | 430 (22.1) | 411 (23.0) | 19 (12.0) | 0.005 |
| 1–4 | 310 (15.9) | 276 (15.4) | 34 (21.5) | |
| 5–9 | 298 (15.3) | 265 (14.8) | 33 (20.9) | |
| 10–14 | 397 (20.4) | 364 (20.4) | 33 (20.9) | |
| 15 or more | 510 (26.2) | 471 (26.4) | 39 (24.6) | |
| Race/ethnicity (%) | ||||
| White | 651 (35.2) | 604 (35.6) | 47 (31.3) | 0.460 |
| Black | 330 (17.9) | 301 (17.7) | 29 (19.3) | |
| Hispanic | 599 (32.4) | 548 (32.3) | 51 (34.0) | |
| Asian/Pacific Islander | 106 (5.7) | 94 (5.5) | 12 (8.0) | |
| Native American | 6 (0.3) | 5 (0.3) | 1 (0.7) | |
| Other | 156 (8.4) | 146 (8.6) | 10 (6.7) | |
| Payer (%) | ||||
| Medicaid | 1,117 (57.4) | 1,032 (57.8) | 85 (53.8) | 0.515 |
| Private insurance | 669 (34.4) | 605 (33.9) | 64 (40.5) | |
| Uninsured | 42 (2.2) | 40 (2.2) | 2 (1.3) | |
| Other | 117 (6.0) | 110 (6.2) | 7 (4.4) | |
| Number of comorbid chronic conditions, mean (95% CI) | 3.7 (3.6, 3.9) | 3.8 (3.7, 3.9) | 3.0 (2.6, 3.4) | <0.001 |
| Weekend admission (%) | 507 (26.1) | 456 (25.5) | 51 (32.3) | 0.064 |
| Hospital Characteristicsc | ||||
| State (%) | ||||
| California | 757 (38.9) | 687 (38.4) | 70 (44.3) | 0.001 |
| New York | 783 (40.3) | 742 (41.5) | 41 (26.0) | |
| Massachusetts | 103 (5.3) | 90 (5.0) | 13 (8.2) | |
| Florida | 302 (15.5) | 268 (15.0) | 34 (21.5) | |
| Type of hospital ownership | ||||
| Public | 315 (16.4) | 286 (16.2) | 29 (18.7) | <0.001 |
| Private, non-profit | 1,517 (79.0) | 1,409 (79.8) | 108 (69.7) | |
| Private, for-profit | 89 (4.6) | 71 (4.0) | 18 (11.6) | |
| Children’s hospital (%) | 309 (17.3) | 300 (18.2) | 9 (6.7) | 0.001 |
| Teaching hospital (%) | 1,550 (80.7) | 1,472 (83.4) | 78 (50.3) | <0.001 |
| Has fully implemented electronic records (%) | 805 (50.7) | 744 (50.1) | 61 (59.8) | 0.159 |
| Has a pediatric ED (%) | 1,286 (72.2) | 1,220 (74.0) | 66 (49.3) | <0.001 |
| Has a pediatric ICU (%) | 1,451 (81.4) | 1,392 (84.5) | 59 (44.0) | <0.001 |
| ED volume (total number of ED visits in a year), median (IQR) | 72,386 (56906, 121686) | 73,747 (57499, 124858) | 57,499 (34511, 95617) | <0.001 |
| ED volume categories (%) | ||||
| <20,000 | 137 (7.1) | 116 (6.6) | 21 (13.6) | <0.001 |
| 20,000–60,000 | 581 (30.2) | 517 (29.3) | 64 (41.3) | |
| 60,001–100,000 | 597 (31.1) | 559 (31.7) | 38 (24.5) | |
| >100,000 | 606 (31.6) | 574 (32.5) | 32 (20.7) |
CI - confidence interval, ED - emergency department, ICU - intensive care unit, IQR - interquartile range,
This group includes patients categorized by an expert panel as unlikely missed sepsis in a prior ED visit and patients without any identified ED visits 7 days before inpatient sepsis admission.
Unlikely missed sepsis vs. potentially missed sepsis groups were compared using Student’s t-test or Wilcoxon rank-sum test for continuous variables and Chi-square test or Fisher’s exact test for categorical variables (unadjusted analysis).
Hospital characteristics are those of the institution diagnosing the patient upon presentation with sepsis (whether correctly diagnosed or potentially missed).
Table 2.
Odds ratios of patient and institutional characteristics associated with potentially missed diagnosis of pediatric sepsis (n=1922, patient-level adjusted analysis)a
| Variablesb | Categories/Value | Odds Ratio | Odds Ratio 95% CI | p value | |
|---|---|---|---|---|---|
| LCL | UCL | ||||
| Patient Characteristics | |||||
| Sex | Female | 1.11 | 0.79 | 1.56 | 0.53 |
| Male (ref) | |||||
| Age, years | 1.00 | 0.98 | 1.03 | 0.84 | |
| Race/ethnicity | Black | 1.46 | 0.87 | 2.44 | 0.15 |
| Hispanic | 1.19 | 0.75 | 1.88 | 0.46 | |
| Asian/Pacific Islander | 1.73 | 0.85 | 3.53 | 0.13 | |
| Native American | 2.33 | 0.25 | 22.02 | 0.46 | |
| Other | 1.29 | 0.61 | 2.74 | 0.50 | |
| White (ref) | |||||
| Payer (%) | Medicaid | 0.80 | 0.54 | 1.18 | 0.26 |
| Uninsured | 0.37 | 0.08 | 1.61 | 0.19 | |
| Other | 0.46 | 0.19 | 1.12 | 0.09 | |
| Private insurance (ref) | |||||
| Number of comorbid chronic conditions | 0.86 | 0.80 | 0.92 | <0.001 | |
| Weekend admission | Weekend Sat-Sun | 1.37 | 0.96 | 1.96 | 0.09 |
| Weekday Mon-Fri (ref) | |||||
| Hospital Characteristics | |||||
| State | California | 2.26 | 1.34 | 3.82 | 0.002 |
| Florida | 3.33 | 1.95 | 5.70 | <0.001 | |
| Massachusetts | 2.87 | 1.35 | 6.09 | 0.006 | |
| New York (ref) | |||||
| Type of hospital ownership | Private, non-profit | 1.39 | 0.81 | 2.38 | 0.23 |
| Private, for-profit | 0.81 | 0.24 | 2.79 | 0.74 | |
| Public (ref) | |||||
| Children’s hospital | Yes | 1.22 | 0.73 | 2.05 | 0.45 |
| No (ref) | |||||
| Teaching hospital | Yes | 0.81 | 0.41 | 1.58 | 0.53 |
| No (ref) | |||||
| Has fully implemented electronic records | Yes | 0.91 | 0.62 | 1.35 | 0.64 |
| No (ref) | |||||
| Has a pediatric ED | Yes | 1.20 | 0.72 | 2.00 | 0.49 |
| No (ref) | |||||
| Has a pediatric intensive care unit | Yes | 0.85 | 0.42 | 1.76 | 0.67 |
| No (ref) | |||||
| Total number of ED visits in a year | 1.00 | 1.00 | 1.00 | 0.62 | |
CI - confidence interval, ED - emergency department, LCL - lower confidence limit, UCL - upper confidence limit, ref - reference category for categorical variables
Generalized linear mixed models with random effects at the hospital level were constructed.
Multiple imputation was used to address missing data (final n=1922). Twenty-three records were missing all hospital-level variables and were dropped from this analysis.
This is the first study to estimate the risk of potentially missed pediatric sepsis in a large population using administrative data (8.1%). Published estimates of missed pediatric sepsis vary because of differences in study populations, settings, and methods. One study in a single health system with an active sepsis quality improvement program had an overall missed pediatric sepsis rate of 2.9% in the ED (11). In contrast, another study showed that ED clinicians missed 27% of patients with severe sepsis/septic shock (12), which may be an overestimate as the initiation of a sepsis order set was used as a surrogate marker of sepsis diagnosis. An analysis of administrative data determined that 20% of children admitted for sepsis/meningitis had a prior ED treat-and-release visit before their admission (13); however, it is unclear if these visits represent sepsis misdiagnoses.
We found a significant reduction in the odds of being potentially misdiagnosed among patients with comorbid chronic conditions, probably because clinicians are more vigilant when evaluating these children. Similar associations were observed in adults with chronic disease who presented with stroke (14) and with acute myocardial infarction (15).
Patients in CA, FL, and MA had increased odds of potentially missed sepsis compared to those in NY. It is unclear if this represents a true difference since there is considerable missing data across states, with NY having more complete data available (95% vs. 39%−56% complete). We used HCUP data 2–3 years before NY implemented a state-wide mandate for ED sepsis protocols; thus, this mandate would not have affected our results (3).
Our study had several limitations. Administrative data lack clinical details, including vital signs, to make definitive judgments of sepsis misdiagnosis. Data were from 2010–2011, prior to institution of ED sepsis protocols; thus, our findings may not reflect current practice. Patient identifiers were missing in 40% of cases, making it impossible to link records; however, in a model restricted to NY (least missing data), we did not identify any additional variables associated with potentially missed sepsis. Using the SPADE method, an ED diagnosis that is rarely associated with the diagnosis of interest can serve as a negative control to help discriminate between unrelated vs. related prior ED visits (5). Because sepsis presentations vary, it is difficult to identify a negative control. Thus, an expert panel was used; however, judgments can be inaccurate despite good interrater reliability. Our estimate of potentially missed sepsis may include cases representing disease progression rather than misdiagnosis. Sepsis can also progress rapidly, making it difficult to diagnose at early stages. We did not identify patients with potentially missed sepsis seen in non-ED settings, who sought care in another state, or who died at home.
Analysis of large databases to estimate missed sepsis remains imperfect and requires patient-level validation. Nonetheless, this work is important as it will allow investigators to track rates of missed sepsis over time, permitting one to quickly evaluate whether interventions including ED sepsis protocols reduce misdiagnosis. This strategy can also be used to screen for cases that warrant detailed medical record review.
Overall, our study shows that in a large multi-state cohort, 8.1% of children admitted for severe sepsis/septic shock had potentially missed sepsis during a prior ED visit. Administrative data can be used to screen large numbers of patients for potentially missed sepsis and identify cases that warrant detailed review. Further work is needed to validate and refine this method.
Supplementary Material
Acknowledgments:
The authors thank the University of Iowa Institute for Clinical and Translational Science Bioinformatics Core (Heather Davis, James Schappet, and Nicholas Smith) for their assistance in downloading, preparing, and organizing data for this study. We also thank the members of our expert panel, Dr. Susan Feigelman, Dr. Charles Jennissen, and Dr. Veerajalandhar Allareddy, for lending their time and expertise. Finally, we thank Dr. David Newman-Toker for his valuable input into the overall design of this study.
Research funding: All phases of this study were supported by the Children’s Miracle Network - University of Iowa Department of Pediatrics Research Grant. Support for informatics and statistical analysis was provided through a National Institutes of Health (NIH) Clinical and Translational Science Award #UL1TR002537. Additionally, Dr. Cifra is supported by an NIH Institutional K12 grant #HD027748 and Dr. Mohr is supported by an Agency for Healthcare Research and Quality K08 grant #HS025753. The content of this work is solely the responsibility of the authors and does not necessarily represent the official views of these funding agencies.
Footnotes
Publisher's Disclaimer: Disclaimer: The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
Employment or leadership: No conflicts declared.
Competing Interests: No conflicts declared.
Honorarium: None.
References
- 1.Murphy SL, Mathews TJ, Martin JA, Minkovitz CS, Strobino DM. Annual Summary of Vital Statistics: 2013–2014. Pediatrics. 2017. June;139(6). [DOI] [PubMed] [Google Scholar]
- 2.Hartman ME, Linde-Zwirble WT, Angus DC, Watson RS. Trends in the epidemiology of pediatric severe sepsis*. Pediatr Crit Care Med. 2013. September;14(7):686–93. [DOI] [PubMed] [Google Scholar]
- 3.Seymour CW, Gesten F, Prescott HC, Friedrich ME, Iwashyna TJ, Phillips GS, et al. Time to Treatment and Mortality during Mandated Emergency Care for Sepsis. N Engl J Med. 2017. 08;376(23):2235–44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Damiani E, Donati A, Serafini G, Rinaldi L, Adrario E, Pelaia P, et al. Effect of performance improvement programs on compliance with sepsis bundles and mortality: a systematic review and meta-analysis of observational studies. PloS One. 2015;10(5):e0125827. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Liberman AL, Newman-Toker DE. Symptom-Disease Pair Analysis of Diagnostic Error (SPADE): a conceptual framework and methodological approach for unearthing misdiagnosis-related harms using big data. BMJ Qual Saf. 2018. July;27(7):557–66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Healthcare Cost and Utilization Project (HCUP) [Internet]. [cited 2015 Aug 10]. Available from: http://www.ahrq.gov/research/data/hcup/
- 7.Access comprehensive hospital data with AHA Annual Survey Database [Internet]. AHA Data. [cited 2019 Feb 14]. Available from: http://www.ahadata.com/aha-annual-survey-database-asdb/
- 8.Balamuth F, Weiss SL, Hall M, Neuman MI, Scott H, Brady PW, et al. Identifying Pediatric Severe Sepsis and Septic Shock: Accuracy of Diagnosis Codes. J Pediatr. 2015. December;167(6):1295–1300.e4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Bamber AR, Mifsud W, Wolfe I, Cass H, Pryce J, Malone M, et al. Potentially preventable infant and child deaths identified at autopsy; findings and implications. Forensic Sci Med Pathol. 2015. September;11(3):358–64. [DOI] [PubMed] [Google Scholar]
- 10.Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009. June 29;338:b2393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Scott HF, Greenwald EE, Bajaj L, Deakyne Davies SJ, Brou L, Kempe A. The Sensitivity of Clinician Diagnosis of Sepsis in Tertiary and Community-Based Emergency Settings. J Pediatr. 2018;195:220–227.e1. [DOI] [PubMed] [Google Scholar]
- 12.Balamuth F, Alpern ER, Grundmeier RW, Chilutti M, Weiss SL, Fitzgerald JC, et al. Comparison of Two Sepsis Recognition Methods in a Pediatric Emergency Department. Acad Emerg Med. 2015. November;22(11):1298–306. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Vaillancourt S, Guttmann A, Li Q, Chan IYM, Vermeulen MJ, Schull MJ. Repeated emergency department visits among children admitted with meningitis or septicemia: a population-based study. Ann Emerg Med. 2015. June;65(6):625–632.e3. [DOI] [PubMed] [Google Scholar]
- 14.Newman-Toker DE, Moy E, Valente E, Coffey R, Hines AL. Missed diagnosis of stroke in the emergency department: a cross-sectional analysis of a large population-based sample. Diagnosis. 2014;1:155–66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Moy E, Barrette M, Coffey R, Hines AL, Newman-Toker DE. Missed diagnoses of acute myocardial infarction in the emergency department: variation by patient and facility characteristics. Diagnosis. 2015;2(1):29–40. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.