Abstract
OBJECTIVE
To compare the performance of three methods of identifying children with severe sepsis and septic shock from the Virtual Pediatric Systems (VPS) database to prospective screening using consensus criteria.
DESIGN
Observational cohort study.
SETTING
Single-center pediatric intensive care unit (PICU).
PATIENTS
Children admitted to the PICU in the period between 3/1/2012 and 3/31/2014.
INTERVENTIONS
None.
MEASUREMENTS AND MAIN RESULTS
During the study period, all PICU patients were prospectively screened daily for sepsis, and those meeting consensus criteria for severe sepsis or septic shock on manual chart review were entered into the sepsis registry. Of 7459 patients admitted to the PICU during the study period, 401 met consensus criteria for severe sepsis or septic shock (reference standard cohort). Within VPS, patients identified using “Martin” (n=970, κ=0.43, PPV=34%, F1=0.48) and “Angus” ICD-9-CM codes (n=1387, κ=0.28, PPV=22%, F1=0.34) showed limited agreement with the reference standard cohort. By comparison, explicit ICD-9-CM codes for severe sepsis (995.92) and septic shock (785.52) identified a smaller, more accurate cohort of children (n=515, κ=0.61, PPV=57%, F1=0.64). PICU mortality was 8% in the reference standard cohort and the cohort identified by explicit codes; age, illness severity scores, and resource utilization did not differ between groups. Analysis of discrepancies between the reference standard and VPS explicit codes revealed that prospective screening missed 66 patients with severe sepsis or septic shock. After including these patients in the reference standard cohort as an exploratory analysis, agreement between the cohort of patients identified by VPS explicit codes and the reference standard cohort improved (κ=0.73, PPV=70%, F1=0.75).
CONCLUSIONS
Children with severe sepsis and septic shock are best identified in the VPS database using explicit diagnosis codes for severe sepsis and septic shock. The accuracy of these codes and level of clinical detail available in the VPS database allow for sophisticated epidemiologic studies of pediatric severe sepsis and septic shock in this large, multicenter database.
Keywords: severe sepsis, septic shock, pediatrics, epidemiology, electronic health record, critical care
INTRODUCTION
Advances in early detection and intensive care management have led to improved survival among children with severe sepsis and septic shock (1), but it remains a leading cause of morbidity and mortality among critically ill children worldwide (2, 3). Compared to adult sepsis, pediatric sepsis possesses a distinct epidemiology with lower mortality rates (4–6), lower rates of progressive organ failure (4, 5), and different determinants of poor outcomes (6). Multicenter databases have the potential to yield important insights into the management of pediatric severe sepsis that would otherwise be impossible to obtain, however their utility in sepsis research has been somewhat limited due to difficulty of retrospectively identifying patients with sepsis (7).
Due to its size and level of clinical detail, the Virtual Pediatric Systems (VPS) multicenter database is an ideal data source for pediatric severe sepsis and septic shock epidemiology research (8). VPS is a pediatric critical care registry with limited protected health information in which prospective data is collected by trained individuals with a clinical background in pediatric critical care using standardized clinical data definitions and rigorous data quality control procedures. In addition to standard demographic data, trained coders assign relevant diagnosis codes and abstract a large amount of detailed clinical data for every pediatric intensive care unit (PICU) admission at participating sites (e.g. illness severity scores and procedures, see eTable 1 for details). VPS currently includes data from >135 PICUs and >1 million hospitalizations; participating sites represent a great diversity of pediatric critical care facilities, from small community hospitals to large, academic quaternary care PICUs. VPS performs initial and quarterly inter-rater reliability testing, and inter-rater reliability concordance in the VPS database is consistently >95% (9). Although utilization of ICD-9-CM diagnosis codes is a well-described limitation of administrative databases, we expect the accuracy of diagnosis codes in VPS to exceed that of other administrative databases because of the clinical experience and continual reliability testing required for data entry.
In our analysis, we assessed the performance of three published methods (14–16) to retrospectively identify children with severe sepsis and septic shock from an electronic database using coded diagnosis data in VPS. We compared the performance of these algorithms to a manual approach of identifying children with severe sepsis and septic shock identified through prospective screening and medical record review process. We hypothesized that children with severe sepsis and septic shock could be accurately identified in VPS using diagnosis codes.
MATERIALS AND METHODS
Study Design
After IRB approval, we constructed an observational cohort study using data from our single center available in the VPS database. We also utilized data from the Children’s Hospital of Philadelphia (CHOP) Sepsis Registry, a prospective, single-center observational registry of patients with sepsis which has been described previously (10, 11). During the study period (3/1/2012 to 3/31/2014), all CHOP PICU patients were screened each day for the presence of sepsis based on consensus criteria (5), through a paper-based sepsis screening checklist which was integrated into morning teaching rounds. A chart review was completed for each patient who screened positive to determine sepsis severity and identify the date of onset; relevant clinical data were also abstracted from the electronic medical record, including laboratory results, resource utilization, and clinical outcomes.
Case Ascertainment
Data from all admissions to the CHOP PICU during the study period were extracted from the VPS database, including demographic information, source of admission, coded diagnoses, severity of illness data [Paediatric Index of Mortality (PIM)-2 (12), Pediatric Risk of Mortality (PRISM)-III (13)], PICU interventions, length of stay, and PICU outcome. PICU patients with severe sepsis or septic shock in the CHOP sepsis registry were matched to their corresponding VPS record using medical record number, date of birth, and date of PICU admission. All records were successfully matched, and sepsis severity was subsequently extracted from the CHOP sepsis registry.
Three methods of retrospectively identifying severe sepsis and septic shock from an electronic database were utilized in this study: 1) “Explicit codes”: an ICD-9-CM code for severe sepsis (995.92) or septic shock (785.52) as proposed by Balamuth et al (14); 2) “Martin codes”: ICD-9-CM codes for sepsis as proposed by Martin et al (15); and 3) “Angus codes”: ICD-9-CM codes for infection and organ dysfunction as proposed by Angus et al (16). A listing of all ICD-9-CM codes used in our analysis are available in eTable 2. VPS assigns diagnoses at the encounter level; therefore, these search algorithms identify the presence of severe sepsis or septic shock at any point during the PICU encounter. A secondary chart review was completed to independently determine presence of severe sepsis or septic shock for patients in whom prospective screening and VPS screening algorithms yielded discrepant results.
Outcomes
The primary outcomes were the diagnostic test characteristics of each ICD-9-CM identification algorithm compared to a manually defined cohort of cases identified by prospective screening using consensus criteria (the reference standard cohort). Secondary outcomes included differences in patient characteristics, resource utilization, and clinical outcomes among the tested sepsis identification algorithms.
Statistical Analysis
Level of agreement between each retrospective ICD-9-CM sepsis identification algorithm and the reference standard cohort was assessed by sensitivity, specificity, percentage agreement, Cohen’s κ, positive predictive value (PPV), and F1 score, and a confusion matrix was constructed for each comparison. Details regarding each calculation are included in the Supplemental Digital Content. Comparisons of secondary outcomes were performed using the Wilcoxon rank-sum test or Kruskal-Wallis test as appropriate for continuous variables and the χ2 test for categorical variables.
RESULTS
Prevalence of Severe Sepsis and Septic Shock by Prospective Screening
During the study period, 7459 patients were admitted to the PICU. Prospective screening during this time identified 401 patients with severe sepsis and septic shock who were enrolled in the CHOP sepsis registry (the reference standard cohort) and successfully matched with VPS. Details from VPS of demographics, resource utilization, and clinical outcomes for these patients compared to all other PICU admissions during the study period are shown in Table 1.
Table 1.
Demographics, Resource Utilization, and Clinical Outcomes of Patients with Severe Sepsis and Septic Shock Identified by Prospective Screening, Compared to All Other PICU Admissions
| Patient Characteristic | CHOP Sepsis Registry Patients (n=401) | All Other CHOP VPS Patients (n=7058) | pa |
|---|---|---|---|
| Age in years, median (IQR) | 7 (2–14) | 5 (1–12) | 0.001* |
| Weight in kg, median (IQR) | 24 (12–45) | 18 (11–41) | 0.047* |
| Male gender, n (%) | 220 (55) | 3957 (56) | 0.637 |
| Race, n (%) | 0.190 | ||
| White | 178 (44) | 3433 (49) | |
| Black | 109 (27) | 1937 (27) | |
| Hispanic / Latino | 37 (9) | 382 (5) | |
| Asian | 16 (4) | 251 (4) | |
| Other / Mixed | 61 (16) | 1055 (15) | |
| Admission source, n (%) | <0.001* | ||
| Emergency department | 236 (59) | 3053 (43) | |
| Inpatient ward | 99 (25) | 961 (14) | |
| Operating room | 19 (5) | 2333 (33) | |
| Another hospital’s ICU | 30 (7) | 178 (3) | |
| Other | 17 (4) | 533 (8) | |
| PIM-2 risk of mortality, median (IQR) | 2.9 (1.0–4.7) | 0.8 (0.2–1.2) | <0.001* |
| PRISM-III score, median (IQR) | 8 (4–14) | 0 (0–3) | <0.001* |
| Cardiac arrest prior to admission, n (%) | 10 (3) | 79 (1) | 0.014* |
| Oncologic diagnosis, n (%) | 73 (18) | 587 (8) | <0.001* |
| Conventional mechanical ventilation, n (%) | 258 (64) | 1799 (24) | <0.001* |
| Advanced mode of ventilationb, n (%) | 72 (18) | 84 (1) | <0.001* |
| Hemodialysis, n (%) | 14 (3) | 56 (1) | <0.001* |
| Extracorporeal support, n (%) | 12 (3) | 9 (0.1) | <0.001* |
| PICU LOS in days, median (IQR) | 9 (3–19) | 2 (1–4) | <0.001* |
| PICU mortality, n (%) | 34 (8) | 79 (1) | <0.001* |
Wilcoxon rank-sum test for continuous variables, Chi-squared test for categorical variables
Advanced modes of ventilation include high frequency oscillatory ventilation, volume diffusive respirator, and airway pressure release ventilation.
Significance at the α = 0.05 level
Abbreviations:
IQR = interquartile range; ICU = intensive care unit; PIM = paediatric index of mortality; PRISM = pediatric risk of mortality; PICU = pediatric intensive care unit; LOS = length of stay
Performance of Each Retrospective Identification Algorithm
The performance of each diagnosis code algorithm was evaluated as compared to the reference standard cohort. Performance characteristics of each coding algorithm are shown in Table 2. Utilizing explicit codes for severe sepsis and septic shock, 515 patients were retrospectively identified in the VPS database. This identification algorithm demonstrated good agreement when compared to the reference standard cohort, with a 95% agreement across all patients, a Cohen’s κ of 0.61, a PPV of 57%, and an F1 score of 0.64. Utilizing the Martin methodology, 970 patients were retrospectively identified in the VPS database. This identification algorithm demonstrated fair agreement when compared to the reference standard cohort, with a 90% agreement across all patients, a Cohen’s κ of 0.43, a PPV of 34%, and an F1 score of 0.48. Utilizing the Angus methodology, 1387 patients were retrospectively identified in the VPS database. This identification algorithm demonstrated poor agreement when compared to the reference standard cohort, with an 84% agreement across all patients, a Cohen’s κ of 0.28, a PPV of 22%, and an F1 score of 0.34. The confusion matrix for each algorithm is shown in eFigure 1.
Table 2.
Performance Characteristics of Each VPS Screening Algorithm, as Compared to Reference Standard Cohort
| Performance Characteristic | Explicit Codes | Martin Codes | Angus Codes |
|---|---|---|---|
| Sensitivity (95% CI) | 72.8% (68.2–77.1) | 82.0% (77.9–85.7) | 76.6% (72.1–80.6) |
| Specificity (95% CI) | 96.8% (96.4–97.2) | 90.9% (90.2–91.6) | 84.7% (83.8–85.5) |
| Cohen’s κ (95% CI) | 0.61 (0.59–0.63) | 0.43 (0.41–0.45) | 0.28 (0.26–0.30) |
| PPV (95% CI) | 56.7% (52.4–61.0) | 33.9% (30.9–36.9) | 22.1% (19.9–24.3) |
| F1 score (95% CI) | 0.64 (0.60–0.68) | 0.48 (0.45–0.51) | 0.34 (0.32–0.37) |
Abbreviations:
CI = confidence interval; PPV = positive predictive value
Overlapping but distinct cohorts of patients were identified through each identification algorithm. In total, 1755 patients were identified by at least one method. Figure 1 depicts the overlapping cohorts of patients identified by all three VPS identification algorithms, and the overlap between patients identified by explicit codes and patients prospectively identified in the reference standard cohort.
Figure 1.
Venn diagrams of the overlapping cohorts of patients identified by the VPS search algorithms (panel A), and the overlap between patients identified by VPS explicit codes and patients prospectively identified in the reference standard cohort (panel B). The area of the outer box represents the total population, and circles represent the proportional area attributable to each subgroup. The proportional sizes of circles are consistent across panels to allow for comparison to the reference standard cohort.
Patient Characteristics, Resource Utilization, and Clinical Outcomes
The prospectively defined patients with severe sepsis or septic shock in the reference standard cohort had higher admission illness severity, resource utilization, and PICU mortality than any of the retrospectively identified cohorts (Table 3). Explicit codes for severe sepsis and septic shock identified the cohort most similar to the reference standard cohort, with a higher prevalence of severe sepsis or septic shock (6.9% vs. 5.4%) but similar age, illness severity scores, resource utilization, and PICU mortality. By comparison, Martin methodology and Angus methodology produced successively larger cohorts of patients with lower illness severity, resource utilization, and PICU mortality.
Table 3.
Characteristics of Patients Identified by Each VPS Screening Algorithm
| Population Characteristic | Sepsis Registry (n=401) | Explicit Codes (n=515) | Martin Codes (n=970) | Angus Codes (n=1387) |
|---|---|---|---|---|
| Prevalence of Severe Sepsis / Septic Shocka | 5.4% | 6.9% | 13.0% | 18.6% |
| Age in years, median (IQR) | 7 (2–14) | 7 (2–14) | 7 (2–14) | 5 (1–12) |
| Admitted from inpatient unit, n (%) | 99 (25) | 140 (27) | 275 (28) | 375 (27) |
| PIM-2 risk of mortality, median (IQR) | 2.9 (1.0–4.7) | 1.8 (1.0–4.9) | 1.3 (0.9–4.4) | 2.8 (0.9–4.2) |
| PRISM-III score, median (IQR) | 8 (4–14) | 8 (3–14) | 5 (2–11) | 3 (0–9) |
| Oncologic diagnosis, n (%) | 73 (18) | 99 (19) | 156 (16) | 128 (9) |
| Conventional mechanical ventilation, n (%) | 258 (64) | 301 (58) | 482 (50) | 826 (59) |
| Advanced mode of ventilationb, n (%) | 72 (18) | 83 (16) | 97 (10) | 127 (9) |
| Extracorporeal support, n (%) | 12 (3) | 14 (3) | 14 (1) | 18 (1) |
| PICU LOS in days, median (IQR) | 9 (3–19) | 7 (2–17) | 5 (2–13) | 6 (2–13) |
| PICU mortality, n (%) | 34 (8) | 46 (8) | 60 (6) | 64 (4) |
Prevalence for each method is calculated as number of patients identified divided by number of patients screened (n=7459).
Advanced modes of ventilation include high frequency oscillatory ventilation, volume diffusive respirator, and airway pressure release ventilation.
Abbreviations:
IQR = interquartile range; PIM = paediatric index of mortality; PRISM = pediatric risk of mortality; PICU = pediatric intensive care unit; LOS = length of stay
Analysis of Discrepancies between VPS Explicit Codes and Reference Standard Cohorts
Because explicit codes for severe sepsis and septic shock performed better than the other identification algorithms in our cohort, we evaluated discrepancies between patients identified by explicit codes and those included in the reference standard cohort. Characteristics of patients in either cohort and both cohorts are shown in Table 4. While patient demographics and mortality rates were not different when compared across groups, admission illness severity, rates of respiratory failure, and length of stay differed across groups.
Table 4.
Detailed analysis of characteristics of sepsis episodes identified in the VPS explicit codes cohort, the reference standard cohort, or both.
| Patient characteristicsaracteristic | Both Cohorts Sepsis Registry (+) VPS Search (+) (n=292) | Reference Cohort Only Sepsis Registry (+) VPS Search (–) (n=109) | VPS Cohort Only Sepsis Registry (–) VPS Search (+) (n=223) | pa |
|---|---|---|---|---|
| Age in years, median (IQR) | 8 (2–14) | 4 (1–13) | 7 (2–14) | 0.073 |
| Admitted from inpatient unit, n (%) | 72 (25) | 27 (25) | 68 (30) | 0.292 |
| PIM-2 risk of mortality, median (IQR) | 3.1 (1.1–5.0) | 2.0 (0.9–4.2) | 1.4 (1.0–4.5) | 0.034* |
| PRISM-III score, median (IQR) | 9 (5–15) | 7 (3–13) | 7 (3–12) | 0.013* |
| Oncologic diagnosis, n (%) | 59 (20) | 14 (13) | 40 (18) | 0.247 |
| Conventional mechanical ventilation, n (%) | 187 (64) | 71 (65) | 114 (51) | 0.005* |
| Advanced modes of ventilationb, n (%) | 55 (19) | 17 (16) | 28 (13) | 0.155 |
| Extracorporeal support, n (%) | 10 (3) | 2 (2) | 4 (2) | 0.443 |
| PICU LOS in days among survivors, median (IQR) | 9 (4–19) | 7 (3–19) | 4 (2–11) | <0.001* |
| PICU mortality, n (%) | 28 (10) | 6 (6) | 18 (8) | 0.414 |
Comparisons across all three groups assessed by Kruskal-Wallis test for continuous variables and Chi-squared test for categorical variables.
Advanced modes of ventilation include high frequency oscillatory ventilation, volume diffusive respirator, and airway pressure release ventilation.
Significance at the α = 0.05 level
Abbreviations:
PIM = pediatric index of mortality; PRISM = pediatric risk of mortality; PICU = pediatric intensive care unit; LOS = length of stay
We performed a secondary chart review to independently assess sepsis severity by consensus criteria in the subsets of patients identified by either prospective screening or explicit codes, but not both; results from this chart review are summarized in eTable 3. Among the 109 patients identified in the reference standard cohort but not identified by the VPS explicit codes (i.e. false negatives), 85/109 patients (78%) were confirmed to have severe sepsis or septic shock, while 19/109 patients (17%) had sepsis but did not definitively meet consensus criteria for organ dysfunction. Among the 223 patients identified by the VPS explicit codes but not identified in the sepsis registry (i.e. false positives), 131/223 patients (59%) had sepsis but did not meet consensus criteria for severe sepsis or septic shock, and 66/223 patients (30%) were confirmed to have severe sepsis or septic shock. Of these 66 false positive patients found to have severe sepsis or septic shock on supplemental chart review, none had been screened for inclusion in the CHOP sepsis registry despite its rigorous universal screening process, conducted each day during morning teaching rounds. Based on this supplemental chart review, the false omission rate for the reference standard cohort was only 0.94% (66/7016). We additionally reviewed the charts of all patients identified by the Angus and Martin methodologies but did not have severe sepsis or septic shock by explicit codes or the reference standard; none of the 188 patients in this group had severe sepsis or septic shock on supplemental chart review.
After updating our reference standard cohort by adding the 66 newly identify cases of severe sepsis and septic shock as an exploratory analysis, the VPS explicit codes demonstrate excellent agreement with the adjusted reference standard cohort, with 97% agreement across all patients, a Cohen’s κ of 0.73, a PPV of 70%, and an F1 score of 0.75. Detailed analysis of the explicit codes compared to the original and adjusted reference standard cohorts are shown in eFigure 2.
DISCUSSION
This is the first study to evaluate the accuracy of diagnosis codes to identify children with severe sepsis and septic shock in the VPS database, and one of the only studies of pediatric sepsis to utilize a prospectively defined cohort of patients as a reference standard. We found that explicit codes can accurately identify patients with severe sepsis and septic shock in the VPS database when compared to a cohort identified by prospective screening and medical record review. During our validation study, we discovered that the manual identification process, despite its rigor, missed some cases of severe sepsis and septic shock. After adding these cases to our reference standard cohort, the accuracy of the identification algorithm utilizing explicit codes is higher than that of any previously reported administrative coding strategies in studies of pediatric sepsis.
In our primary analysis, we found that explicit codes for severe sepsis and septic shock, Martin methodology, and Angus methodology identified distinct but overlapping cohorts of patients. The VPS explicit codes demonstrated the highest level of agreement with the reference standard cohort with acceptable sensitivity and specificity. In contrast, the Martin and Angus methodologies performed less well than explicit codes, and also less well than in previous studies of their use (7, 17, 18). We speculate the major reason for their poor performance in the present study is that our target cohort was children with severe sepsis and septic shock, as opposed to children with sepsis of any level of severity. The Martin and Angus methodology were validated to identify patients with sepsis and were not intended to stratify septic patients with regard to severity of illness. Consequently, the κ was much higher for explicit codes, reflecting the higher precision achieved by targeting patients with severe sepsis and septic shock in this identification strategy. While a cohort of sepsis patients with a range of disease severity may be useful in some settings, identification of a less heterogeneous group of patients with high disease severity allows for focused study of higher-risk patients. To that end, only the explicit codes were accurately able to identify patients with severe sepsis and septic shock.
Analysis of our secondary outcomes supports the conclusion that explicit codes can be used to identify a cohort of patients with high illness severity similar to the children with severe sepsis and septic shock identified through prospective screening. Subset analyses of patients identified by manual review, explicit codes, or both methods suggest that patients identified in the explicit code cohort but not included in the reference standard cohort have lower illness severity and lower resource utilization than the patients identified in by manual review. Our chart review of discrepancies found that one-quarter of these patients identified by the explicit code method only had sepsis and some organ dysfunction, but they did not meet the strict criteria for organ dysfunction necessary for a diagnosis of severe sepsis or septic shock.
There are several important strengths of the methodology we employed to identify a cohort of patients with severe sepsis and septic shock from an administrative database. Retrospective identification of patients through VPS for epidemiologic purposes is efficient, relatively inexpensive, and could allow for multicenter analysis with relative ease. By contrast, the CHOP sepsis registry is a local research database developed through prospective, universal screening, an expensive and labor-intensive process that would be difficult to translate to a multicenter cohort. Additionally, despite the use of a rigorous methodology in screening and enrolling patients in the CHOP sepsis registry, the highly sensitive VPS explicit codes methodology identified 66 patients with severe sepsis or septic shock who were missed by the daily prospective screening process, which represents 15% of the total cohort of patients with severe sepsis or septic shock during the study period. Therefore, screening patients for severe sepsis and septic shock using explicit codes in VPS is a reasonable alternative to prospective screening and medical record review.
When compared to our adjusted reference standard cohort, the patients identified by VPS explicit codes for severe sepsis and septic shock were among the most accurate cohort of retrospectively identified patients with sepsis ever reported in the literature. Previous efforts to use administrative data to identify pediatric patients with sepsis in single-center studies (7) or large databases (19) have yielded cohorts with higher prevalence and lower mortality than well designed, prospective trials designed to identify patients with sepsis (4). Conversely, our cohort of patients with severe sepsis and septic shock identified through VPS explicit codes identifies an accurate cohort of patients with sepsis and high disease severity and resource utilization, which may serve as a large, robust cohort for future pediatric sepsis research. Data in VPS is extracted by expert, trained coders at each site utilizing standard data definitions, and we believe the reliability of the data in the present report reflects the strength of these coding strategies. While many compelling research questions can be answered directly from the data available in single-center or multicenter VPS, this work could potentially be expanded through planned or existing mergers of VPS with other large pediatric registries, including the Pediatric Health Information System (PHIS) database (20), the PEDSnet clinical data research network (21), and the Center for International Blood and Marrow Transplant Research (CIBMTR) registry (22), among others.
Our study has several important limitations. First, our results are limited to a single, quaternary care PICU with a dedicated pediatric sepsis program. While VPS operates with standardized clinical data definitions, there is likely a variance in coding practice across all VPS sites, and PICU admission criteria may also vary by site. Second, our study period is limited to the 25-month time period when the CHOP sepsis registry was actively enrolling patients by prospective screening and medical record review, so we cannot analyze trends in VPS coding of sepsis across longer time periods. We did not directly compare VPS to an administrative dataset, although future comparison of VPS and PHIS could yield important insights into the relative performance of sepsis diagnosis codes in both. We also cannot determine if our results are generalizable to ICD-10 codes, though direct translations of severe sepsis and septic shock codes from ICD-9 to ICD-10 are available. Third, while VPS enrolls patients from a large, diverse collection of PICUs, large quaternary care PICUs care for more patients with high illness severity than smaller, community centers; it is likely that the majority of patients with severe sepsis and septic shock captured in VPS are being cared for in major children’s hospitals. Finally, the definitions of severe sepsis and septic shock used in clinical medicine and documented in the electronic health record may shift as our understanding of pediatric sepsis pathophysiology and epidemiology continues to develop and new clinical definitions are developed (23, 24).
CONCLUSIONS
Children with severe sepsis and septic shock can be identified in the VPS database using diagnosis codes. In comparison to a cohort of patients identified through prospective screening and medical record review, the cohort of patients identified in VPS using ICD-9-CM codes for severe sepsis (995.92) and septic shock (785.52) have similar mortality, illness severity, and resource utilization. The accuracy of these codes and the level of clinical detail available in the VPS database will allow for future epidemiologic studies of pediatric severe sepsis and septic shock, and their use may facilitate reliable identification of patients with severe sepsis and septic shock in other large databases through probabilistic linkages to VPS.
Supplementary Material
eFigure 1. Confusion matrix for comparisons of each retrospective method to identify patients with severe sepsis and septic shock, as compared to the reference standard cohort, in order of performance. The confusion matrix is a two-by-two table that cross tabulates the positive and negative results for each approach. Perfect agreement occurs when there are no numbers on the off-diagonal cells, which correspond to discrepancies between the test algorithm and the reference standard cohort. Retrospective screening using Explicit codes (A) outperformed Martin codes (B), which outperformed Angus codes (C).
eFigure 2. Confusion matrix for the performance of explicit codes in VPS, as compared to (A) the reference standard cohort and (B) the reference standard cohort adjusted based on results of supplemental chart review.
eTable 1. Data elements available in the VPS database.
eTable 2. List of ICD-9-CM codes utilized to identify patients with severe sepsis and septic shock, by identification algorithm.
eTable 3. Sepsis severity among patients in the reference standard cohort or the VPS cohort, but not both
ACKNOWLEDGEMENTS
The authors would like to thank the staff of the CHOP Critical Care Center for Evidence and Outcomes for their efforts in abstracting and coding the CHOP VPS data used to prepare this report. VPS data was provided by Virtual Pediatric Systems, LLC. No endorsement or editorial restriction of the interpretation of these data or opinions of the authors has been implied or stated. This manuscript has been reviewed by the VPS Research Committee.
Conflicts of Interest and Source of Funding:
Financial support was provided by the Endowed Chair, Department of Anesthesia and Critical Care, Division of Emergency Medicine, and the Office of the Chief Medical Office at The Children’s Hospital of Philadelphia, University of Pennsylvania Perelman School of Medicine. Dr. Weiss is also supported by NIGMS K23-GM110496. Dr. Balamuth is also supported by NICHD K23-HD082368. Dr. Nishisaki is also supported by NICHD R21-HD089151. For the remaining authors, no conflicts were declared.
Footnotes
Copyright form disclosure: Dr. Nishisaki’s institution received funding from Agency for Healthcare Research and Quality R18 HS022464–01 and R18 HS024511–01 and the National Institute of Child Health and Human Development 1R21HD089151–01A, and he received support for article research from the National Institutes of Health. Dr. Weiss’ institution received funding from National Institute of General Medical Sciences K23GM110496, and he received funding from Bristol-Myers Squibb Company (consultant) and Medscape (honorarium for lecture). The remaining authors have disclosed that they do not have any potential conflicts of interest.
REFERENCES
- 1.Dellinger RP, Levy MM, Rhodes A, et al. : Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. 2013. p. 580–637. [DOI] [PubMed]
- 2.Watson RS, Carcillo JA: Scope and epidemiology of pediatric sepsis. Pediatric Critical Care Medicine 2005; 6:S3–5 [DOI] [PubMed] [Google Scholar]
- 3.Hartman ME, Linde-Zwirble WT, Angus DC, et al. : Trends in the epidemiology of pediatric severe sepsis*. Pediatric Critical Care Medicine 2013; 14:686–693 [DOI] [PubMed] [Google Scholar]
- 4.Weiss SL, Fitzgerald JC, Pappachan J, et al. : Global epidemiology of pediatric severe sepsis: the sepsis prevalence, outcomes, and therapies study. Am J Respir Crit Care Med 2015; 191:1147–1157 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Goldstein B, Giroir B, Randolph A, et al. : International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. 2005. p. 2–8. [DOI] [PubMed]
- 6.Weiss SL, Fitzgerald JC, Maffei FA, et al. : Discordant identification of pediatric severe sepsis by research and clinical definitions in the SPROUT international point prevalence study. Crit Care 2015; 19:325. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Weiss SL, Parker B, Bullock ME, et al. : Defining pediatric sepsis by different criteria: discrepancies in populations and implications for clinical practice. Pediatric Critical Care Medicine 2012; 13:e219–26 [DOI] [PubMed] [Google Scholar]
- 8.Bennett TD, Spaeder MC, Matos RI, et al. : Existing data analysis in pediatric critical care research. Front Pediatr 2014; 2:79. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Virtual Pediatric Systems, LLC. Available at: www.myvps.org. Accessed April 1, 2018.
- 10.Weiss SL, Fitzgerald JC, Balamuth F, et al. : Delayed antimicrobial therapy increases mortality and organ dysfunction duration in pediatric sepsis. Critical Care Medicine 2014; 42:2409–2417 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Balamuth F, Weiss SL, Fitzgerald JC, et al. : Protocolized Treatment Is Associated With Decreased Organ Dysfunction in Pediatric Severe Sepsis. Pediatric Critical Care Medicine 2016; 17:817–822 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Slater A, Shann F, Pearson G, et al. : PIM2: a revised version of the Paediatric Index of Mortality. Intensive Care Med 2003; 29:278–285 [DOI] [PubMed] [Google Scholar]
- 13.Pollack MM, Patel KM, Ruttimann UE: PRISM III: an updated Pediatric Risk of Mortality score. Critical Care Medicine 1996; 24:743–752 [DOI] [PubMed] [Google Scholar]
- 14.Balamuth F, Weiss SL, Neuman MI, et al. : Pediatric severe sepsis in U.S. children’s hospitals. Pediatric Critical Care Medicine 2014; 15:798–805 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Martin GS, Mannino DM, Eaton S, et al. : The epidemiology of sepsis in the United States from 1979 through 2000. N Engl J Med 2003; 348:1546–1554 [DOI] [PubMed] [Google Scholar]
- 16.Angus DC, Linde-Zwirble WT, Lidicker J, et al. : Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Critical Care Medicine 2001; 29:1303–1310 [DOI] [PubMed] [Google Scholar]
- 17.Balamuth F, Weiss SL, Hall M, et al. : Identifying Pediatric Severe Sepsis and Septic Shock: Accuracy of Diagnosis Codes. J Pediatr 2015; 167:1295–300.e4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Johnson AEW, Aboab J, Raffa JD, et al. : A Comparative Analysis of Sepsis Identification Methods in an Electronic Database. Critical Care Medicine 2018; 46:494–499 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Ruth A, McCracken CE, Fortenberry JD, et al. : Pediatric severe sepsis: current trends and outcomes from the Pediatric Health Information Systems database. Pediatric Critical Care Medicine 2014; 15:828–838 [DOI] [PubMed] [Google Scholar]
- 20.Ruth A, McCracken CE, Fortenberry JD, et al. : Extracorporeal therapies in pediatric severe sepsis: findings from the pediatric health-care information system. Crit Care 2015; 19:397. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Forrest CB, Margolis PA, Bailey LC, et al. : PEDSnet: a National Pediatric Learning Health System. J Am Med Inform Assoc 2014; 21:602–606 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Horowitz M: The role of registries in facilitating clinical research in BMT: examples from the Center for International Blood and Marrow Transplant Research. Bone Marrow Transplant 2008; 42 Suppl 1:S1–S2 [DOI] [PubMed] [Google Scholar]
- 23.Singer M, Deutschman CS, Seymour CW, et al. : The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). 2016. p. 801–810. [DOI] [PMC free article] [PubMed]
- 24.Matics TJ, Sanchez-Pinto LN: Adaptation and Validation of a Pediatric Sequential Organ Failure Assessment Score and Evaluation of the Sepsis-3 Definitions in Critically Ill Children. JAMA Pediatr 2017; 171:e172352. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eFigure 1. Confusion matrix for comparisons of each retrospective method to identify patients with severe sepsis and septic shock, as compared to the reference standard cohort, in order of performance. The confusion matrix is a two-by-two table that cross tabulates the positive and negative results for each approach. Perfect agreement occurs when there are no numbers on the off-diagonal cells, which correspond to discrepancies between the test algorithm and the reference standard cohort. Retrospective screening using Explicit codes (A) outperformed Martin codes (B), which outperformed Angus codes (C).
eFigure 2. Confusion matrix for the performance of explicit codes in VPS, as compared to (A) the reference standard cohort and (B) the reference standard cohort adjusted based on results of supplemental chart review.
eTable 1. Data elements available in the VPS database.
eTable 2. List of ICD-9-CM codes utilized to identify patients with severe sepsis and septic shock, by identification algorithm.
eTable 3. Sepsis severity among patients in the reference standard cohort or the VPS cohort, but not both