Evaluating Pediatric Sepsis Definitions Designed for Electronic Health Record Extraction and Multicenter Quality Improvement

Abstract

Objectives:

To describe the Children’s Hospital Association’s Improving Pediatric Sepsis Outcomes sepsis definitions and the identified patients; evaluate the definition using a published framework for evaluating sepsis definitions.

Design:

Observational cohort.

Setting:

Multicenter quality improvement collaborative of 46 hospitals from January 2017 to December 2018, excluding neonatal ICUs.

Patients:

Improving Pediatric Sepsis Outcomes Sepsis was defined by electronic health record evidence of suspected infection and sepsis treatment or organ dysfunction. A more severely ill subgroup, Improving Pediatric Sepsis Outcomes Critical Sepsis, was defined, approximating septic shock.

Interventions:

Participating hospitals identified patients, extracted data, and transferred de-identified data to a central data warehouse. The definitions were evaluated across domains of reliability, content validity, construct validity, criterion validity, measurement burden, and timeliness.

Measurements and Main Results:

Forty hospitals met data quality criteria across four electronic health record platforms. There were 23,976 cases of Improving Pediatric Sepsis Outcomes Sepsis, including 8,565 with Improving Pediatric Sepsis Outcomes Critical Sepsis. The median age was 5.9 years. There were 10,316 (43.0%) immunosuppressed or immunocompromised patients, 4,135 (20.3%) with central lines, and 2,352 (11.6%) chronically ventilated. Among Improving Pediatric Sepsis Outcomes Sepsis patients, 60.8% were admitted to intensive care, 26.4% had new positive-pressure ventilation, and 19.7% received vasopressors. Median hospital length of stay was 6.0 days (3.0–13.0 d). All-cause 30-day in-hospital mortality was 958 (4.0%) in Improving Pediatric Sepsis Outcomes Sepsis; 541 (6.3%) in Improving Pediatric Sepsis Outcomes Critical Sepsis. The Improving Pediatric Sepsis Outcomes Sepsis definitions demonstrated strengths in content validity, convergent construct validity, and criterion validity; weakness in reliability. Improving Pediatric Sepsis Outcomes Sepsis definitions had significant initial measurement burden (median time from case completion to submission: 15 mo [interquartile range, 13–18 mo]); timeliness improved once data capture was established (median, 26 d; interquartile range, 23–56 d).

Conclusions:

The Improving Pediatric Sepsis Outcomes Sepsis definitions demonstrated feasibility for large-scale data abstraction. The patients identified provide important information about children treated for sepsis. When operationalized, these definitions enabled multicenter identification and data aggregation, indicating practical utility for quality improvement.

Pediatric sepsis was formally defined by a consensus conference in 2005 that used the framework of Systemic Inflammatory Response Syndrome to define sepsis, severe sepsis, and septic shock (1). Although alternative definitions for organ dysfunction in sepsis have been proposed since, and efforts are underway to create new definitions for pediatric sepsis, this remains the current definition of record (2–5). Originally developed for trial enrollment, these definitions are precise, but laborious to operationalize, particularly for benchmarking and quality improvement (QI).

Definitions for sepsis vary and are deployed for purposes ranging from clinical identification to research to administrative coding (6). Angus et al (7) have suggested a framework for evaluating sepsis definitions that uses traditional measures of reliability and validity to evaluate definitions on their strengths relative to their intended use (8).

The Children’s Hospital Association’s (CHA) Improving Pediatric Sepsis Outcomes (IPSO) Collaborative created novel, functional sepsis definitions designed to facilitate large-scale automated identification via electronic health record (EHR) extraction. In the first 2 years of a multiyear QI campaign, 46 hospitals of varied size and type used these definitions to ascertain cases and submit data to a central data warehouse. The objectives of this report are as follows: 1) to describe the frequency, demographics, and outcomes of pediatric sepsis cases ascertained using the IPSO Sepsis definitions and 2) to use the Angus framework to retrospectively evaluate the performance of the IPSO Sepsis definitions within the domains of reliability, validity, measurement burden, and timeliness.

MATERIALS AND METHODS

Study Design

This was an observational cohort study.

Setting and Patients

This study included 46 hospitals participating in the first wave of the IPSO collaborative from January 1, 2017, to December 31, 2018. Teams began collecting data in January 2017, and the central data warehouse opened to accept data submissions in March 2017. Hospitals ranged from 43 to 604 inpatient beds and included university-affiliated hospitals, freestanding and non-freestanding pediatric hospitals, and urban and suburban hospitals. Hospitals included pediatric patients following their institutional practices for pediatric care, without age restriction. Neonatal ICUs were excluded from the initial phase of the IPSO collaborative. Patients were included using the sepsis definitions described below. The aims of the IPSO collaborative were to decrease sepsis-attributable mortality and hospital-onset critical sepsis through structured QI. The collaborative centralized expertise to create a key driver diagram and implementation bundles emphasizing early diagnosis and treatment, and facilitated collaborative learning, data collection, and analysis through statistical process control (9).

Definitions

Two sets of sepsis definitions were created by members of a national expert advisory committee and CHA staff members, including physicians, nurses, families, and hospital executives with expertise in pediatric sepsis, QI, and data quality and analysis. Physicians and nurses represented pediatric subspecialties of critical care, emergency medicine, hospital medicine, infectious diseases, hematology/oncology, and surgery.

IPSO Sepsis was an inclusive definition intended to capture patients with sepsis treatment initiated, not requiring a confirmed diagnosis of infection-related organ dysfunction (Table 1). It was designed to identify patients using fields already existing in the medical record that could be automated for EHR extraction. A more severely ill subgroup, IPSO Critical Sepsis, was defined to approximate septic shock (Table 1).

TABLE 1.

Improving Pediatric Sepsis Outcomes Sepsis Definition

IPSO Sepsis
Stand-alone diagnostic criteria (presence of any of the following):
Severe sepsis, septic shock ICD-10 codes (R65.20, R65.21)
Severe sepsis order set
Positive huddle (bedside diagnosis, locally defined) of sepsis
Treatment criteria	Culture criterion	Additional criterion (any one of the following) within 24 hr of treatment
IV antibiotics and within 6 hr	Blood culture within 72 hr of treatment	Sepsis screen + (locally defined) ICU admission Lactate measured
Two boluses OR (bolus and vasoactive agent)		Vasoactive administered Other sepsis ICD-10 codes Infectious disease order set (locally defined, included order sets such as pneumonia or fever/neutropenia order sets)
IPSO Critical Sepsis: IPSO Sepsis criteria AND IV antibiotic AND ([three boluses within 6 hr] OR [bolus within 6hr and vasoactive agent within 24 hr])

ICD-10 = International Classification of Diseases, 10th Revision, IPSO = Improving Pediatric Sepsis Outcomes.

Patients meeting any stand-alone criteria or who had treatment and blood culture and at least one additional criterion had IPSO Sepsis. IPSO Critical Sepsis patients met this definition, including meeting treatment criteria, and additionally received a third bolus within 6 hr or vasoactive agent within 24 hr of meeting treatment criteria.

Patients met criteria for the IPSO Sepsis definition in several ways. Stand-alone criteria indicated that clinicians in the hospital had identified the patient as either having sepsis or requiring treatment for sepsis, as indicated by the use of a sepsis-specific order set, International Classification of Diseases, 10th Revision (ICD-10) code for severe sepsis or septic shock, or a positive huddle. Huddles were an improvement tool consisting of a structured discussion among the clinical team of a patient in whom there was concern for sepsis, with documentation of a decision to proceed with sepsis treatment or not. Additional patients were identified who met multiple criteria: treatment consistent with sepsis (blood culture, IV antibiotics, ≥ 2 IV fluid boluses, or ≥ 1 bolus and a vasoactive agent) and one additional criterion indicating severe infection (Table 1). IPSO Critical Sepsis was a subgroup of IPSO Sepsis patients who received a third IV fluid bolus or a vasoactive agent in addition to receiving treatment consistent with sepsis. Boluses were required to be greater than or equal to 5 mL/kg each, allowing for patient differences and evolving treatment approaches to fluid volume and timing. In patients with multiple sepsis events during hospitalization, data were collected from the first episode.

Data Source

The transfer of limited datasets from participating hospitals to the IPSO Registry for QI purposes was governed by participation agreements at each hospital. The data transfer was approved by appropriate bodies at each hospital, including Institutional Review Boards at some hospitals but not others, reflecting varied interpretation of the requirements for QI data governance. This secondary analysis study was considered exempt from institutional review board review by the Colorado Multiple Institutions Review Board.

Participating hospitals used a data dictionary (Supplemental Digital Content 1, http://links.lww.com/CCM/F680) to identify patients meeting sepsis definitions, extract data elements and transfer de-identified data to an encrypted, HIPAA-compliant central data warehouse (Prometheus Research, New Haven, CT). Each hospital determined the method of patient identification and data extraction, based on informatics infrastructure, personnel, and resources available. Patient identification and data extraction were performed using queries of the EHR data warehouse, manual chart review, or a combination. Once identified, patient data were entered either via a secure monthly batch upload of all visits or case-by-case manually into a web portal designed for this purpose.

In order to support data gathering and standardization, the IPSO collaborative provided participants with a detailed data dictionary, individual consultations, and weekly office hours with data and QI consultants. Participating hospitals shared data tools via webinars, a collaborative listserv, biannual inperson workshops, and an online repository of tools and documents. Frequently, hospitals with shared EHR vendors shared coding scripts used in data gathering.

Measures

While guidelines for sepsis definitions were specific, variation was allowed in ascertainment of secondary clinical measures, in keeping with the QI focus of the collaborative. A data dictionary was disseminated and hospitals had discretion in implementation based on available resources (Supplemental Digital Content 1, http://links.lww.com/CCM/F680). For example, categories of immunosuppressed and immunocompromised patients were described in guidance documents, but individual hospitals used a variety of means to assess these variables, including manual chart review, EHR problem lists, or ICD-10 codes. Similarly, organ dysfunction could be reported using any of several standard definitions or scores that were locally available.

Analysis

Standard descriptive statistics were used to report the characteristics of patients with IPSO Sepsis and IPSO Critical Sepsis. Analyses were conducted in SAS 9.4 (SAS, Cary, NC).

In order to evaluate the domains of usefulness described in the Angus framework, we chose proxy measures for each domain that could be ascertained in the dataset (Table 2). In order to assess test-retest reliability, we measured the change in frequency of sepsis per 1,000 hospital admissions between 2017 and 2018. We considered a greater change to indicate less reliability. In order to assess inter-rater reliability, we assessed the differences in sepsis frequency between individual hospitals, considering a greater interquartile range to indicate lower inter-rater reliability. To assess convergent validity, we assessed the proportion of patients who had critical illness, as defined by the presence of any of the following variables: new positive-pressure ventilation, lactate greater than or equal to 4 mmol/L, extracorporeal membrane oxygenation, or ICU admission (10, 11). Concurrent criterion validity was assessed by comparing the patients identified by the IPSO definitions with those identified in preexisting institutional sepsis registries. In hospitals where registries existed, hospital sepsis leaders reported to IPSO an estimated percentage of the patients in their independent registries who were identified by IPSO definitions. Predictive criterion validity was assessed by comparing the IPSO Sepsis mortality rate to rates reported in the literature. Timeliness was assessed by measuring the median time to data submission, both initially and after data submission was established. A site was considered established when it had submitted data for up to 80% of months of the collaborative to date.

TABLE 2.

Evaluation of Improving Pediatric Sepsis Outcomes Sepsis Definitions Using the Angus Framework (7) for Development and Interpretation of Sepsis Definitions

Domain	Definition	Proxy Measure of This Domain in IPSO Definition	Result (IPSO Sepsis)	Result (IPSO Critical Sepsis)	Definition Performance (Low, Medium, High)
1) Reliability: Criteria yield stable reproducible results
Test-retest	When tests are repeated	Median change in frequency from 2017 to 2018 (greater change = less reliability)	Median increase 2.44 cases/1,000 admissions (IQR, 0.13–7.74)	Median increase 0.70 cases/1,000 admissions (IQR, −0.68 to 1.87)	Sepsis: Medium Critical Sepsis: Medium to High
Inter-rater	When tests are interpreted across different raters	Among all hospitals, median frequency per 1,000 admissions (proxy for interrater reliability, greater IQR = less reliability)	20.4 cases/1,000 admissions (IQR, 9.1–29.3)	6.4 cases/1,000 admissions (IQR, 3.3–9.1)	Sepsis: Medium Critical Sepsis: Medium to High
2) Content validity
Content validity	Criteria fit with current understanding and knowledge	Qualitative subject matter expert review	This definition implies clinically perceived sepsis based on treatment. This is useful for QI but is subject to limitations of inappropriate treatment; does not reflect all organ dysfunction	This definition is similar to current understanding of shock/cardiovascular dysfunction but is also limited by being subject to the appropriateness of treatment	High
3) Construct validity: Criteria measure what they purport to measure
Convergent	The extent to which two or more aspects that should agree do agree (occur together)	% of patients who have intensive care admission, positive-pressure ventilation, lactate ≥ 4 mmol/L, or extracorporeal membrane oxygenation	74% (see Table 4 for details of each measure)	88%	Medium to high
4) Criterion Validity: New criteria agree with existing standard
Concurrent	Comparison to a current standard available at the same time	Six hospitals with preexisting sepsis registries reported approximate percent of patients in registry also identified by IPSO definitions	80% (Additional detail in Supplemental Digital Content 2, http://links.lww.com/CCM/F681)		Medium to high
Predictive	Comparison to a later outcome believed to be strongly associated with the disease of interest	Overall mortality Sepsis-specific mortality	4.0% 2.5%	6.3% 3.7%	Medium to high
			Similar to existing pediatric sepsis studies from emergency departments with QI (12–15), lower than International Classification of Diseases, 10th Revision coded hospital-wide sepsis mortality (17, 18)
5) Measurement burden: Burden to implement criteria
Lower cost	Financial costs	Qualitative subject matter expert review	There was no direct clinical cost to the patient at bedside. Personnel needs were greatest in establishing data capture and diminished for maintenance. Typically needed at minimum a medical informaticist, a physician, a nurse, and process improvement specialist with a portion of time on this project		Medium
		Self-report of sites using manual abstraction	Sites using manual data abstraction reported case ascertainment using the definition required 3–20 min per case
Greater safety	Side effects, complications to patient	Qualitative subject matter expert review	Little to no direct safety concern. Loss of confidentiality is a theoretical risk, however, no data breaches occurred		High
Lower complexity	Difficulties executing the steps to obtain/interpret the measures	Qualitative subject matter expert review	Definition was complex to initially create in each hospital’s infrastructure, however, easier to routinely report Definition was flexible to each hospital’s internal processes		Medium
6) Timeliness
Timeliness	Speed with which criteria are generated with respect to the course of the disease	Median (IQR) to first case submission Median (IQR) to submission after first submission	15 mo (IQR, 13–18 mo) 44 d (IQR, 24–82 d)		Timeliness for first case identification: Low to medium
		Median (IQR) to submission after 50% submission	31 d (IQR, 23–76 d)
		Median (IQR) to submission after 80% submission	26 d (IQR, 23–56 d)		Timeliness for ongoing submission: Medium

IPSO = Improving Pediatric Sepsis Outcomes, IQR = interquartile range, QI = quality improvement.

Content validity and measurement burden were assessed qualitatively by this author group, composed of IPSO steering committee members and CHA data analysts, with expertise in QI, pediatric sepsis, and statistics. After reviewing the proxy measures of performance in each domain, whether quantitative or qualitative, the authors used majority vote to classify the definitions as having low, low to medium, medium, medium to high, or high performance to aide interpretation.

RESULTS

From January 2017 to December 2018, 40 of 46 participating hospitals with 934,772 total admissions met IPSO data quality criteria for inclusion in the aggregate data presented here. Four different EHR vendors and paper medical records were used. Automatic EHR extraction accounted for data gathering at 26 (56.5%) participating hospitals; the remainder used mostly or entirely manual chart review.

There were 23,976 cases of IPSO Sepsis, representing 2.6% of admissions to participating hospitals, including 8,565 (0.9%) with IPSO Critical Sepsis (Table 3). Many children had complex care needs, including 43.0% with immunosuppression, 20.3% with central lines, and 11.6% chronically ventilated. Critical care was required in a majority of patients, 60.8% were admitted to the ICU, 26.4% experienced new positive-pressure ventilation, and 19.7% received vasopressors.

TABLE 3.

Study Population

			Percentages for These Columns Calculated From Cases in Which the Variable Was Reported, n (%)
Population Characteristic	Hospitals Reporting Variable, n	Cases in Which This Variable Was Reported, n (%)	IPSO Sepsis	IPSO Critical Sepsis
Total	40	23,976 (100.0)	23,976 (100.0)	8,565 (35.7)
Patient characteristics
Age, yr, median (IQR)	38	23,173 (96.7)	5.9 (1.5–13.3)	7.1 (1.9–14.1)
< 18 yr	38	22,793 (95.1)	21,175 (92.9)	7,585 (92.2)
Medically complex	40	23,976 (100.0)	13,520 (56.4)	5,607 (65.5)
Indwelling central venous catheter present on arrival	34	20,357 (84.9)	4,135 (20.3)	2,269 (29.8)
Immunosuppressed or immunocompromised	40	23,976 (100)	10,316 (43.0)	3,986 (46.5)
Chronically ventilated	34	20,234 (84.4)	2,352 (11.6)	909 (12.1)
Transferred from outside hospital	36	20,942 (87.3)	2,621 (12.5)	898 (11.4)
Treatment characteristics
Sepsis episode starting location	40	22,869 (95.4)	22,869 (95.4)	8,293 (96.8)
Emergency department			16,024 (70.1)	5,958 (71.8)
ICU			3,761 (16.4)	1,497 (18.0)
General inpatient floor			2,135 (9.3)	620 (7.5)
Hematology/oncology			606 (2.6)	156 (1.9)
Other			343 (1.5)	62 (0.8)
Total bolus volume in 6 hr surrounding time zero, mL/kg, median (IQR)	38	22,391 (93.4)	40.0 (20.0–46.0)	58.8 (39.6–60.0)
Vasoactive agent during hospitalization	36	21,829 (91.0)	4,307 (19.7)	3,065 (38.6)
Vasoactive days in patients requiring vasoactive, median (IQR)	36	4,257 (98.8)	2.0 (1.0–4.0)	2.0 (1.0–4.0)
New positive-pressure ventilation	30	13,083 (54.6)	3,450 (26.4)	1,529 (32.1)
Ventilation days in patients requiring ventilation, median (IQR)	30	3,450 (100.0)	5.0 (3.0–10.0)	5.0 (3.0–10.0)
ICU admission	38	22,430 (93.6)	13,644 (60.8)	6,115 (73.6)
ICU days associated with sepsis episode in patients with ICU admission, median (IQR)	38	13,637 (99.9)	4.0 (2.0–10.0)	5.0 (2.0–10.0)
Hospital days per sepsis episode (capped at 31), median (IQR)	39	23,507 (98.0)	6.0 (3.0–13.0)	7.0 (4.0–15.0)
Extracorporeal membrane oxygenation	36	21,258 (88.7)	289 (1.4)	162 (2.1)
Laboratory data
Blood culture positive	36	20,860 (87.0)	2,612 (12.5)	1,066 (14.3)
Lactate, mmol/L, median (IQR)	23	8,062 (33.6)	1.9 (1.3–3.1)	2.2 (1.3–3.6)
Outcomes
Hypotension in first 24 hr of sepsis event	36	21,532 (89.8)	10,597 (49.2)	4,771 (61.1)
Organ dysfunction present	28	12,102 (50.5)	5,773 (47.7)	2,612 (58.0)
Sepsis-specific 30-d in-hospital mortality	40	23,976 (100.0)	597 (2.5)	318 (3.7)
Sepsis-specific 30-d in-hospital mortality in medically complex patients	40	13,520	483 (3.6)	269 (4.8)
Sepsis-specific 30-d in-hospital mortality in nonmedically complex patients	40	10,456	114 (1.1)	49 (1.7)
All-cause 30-d in-hospital mortality	40	23,976 (100.0)	958 (4.0)	541 (6.3)
All-cause 30-d in-hospital mortality in medically complex patients	40	13,520	751 (5.6)	439 (7.8)
All-cause 30-d in-hospital mortality in nonmedically complex patients	40	10,456	207 (2.0)	102 (3.4)

IPSO = Improving Pediatric Sepsis Outcomes, IQR = interquartile range.

IPSO Critical Sepsis is a subset of IPSO Sepsis patients.

Hospitals reported secondary variables based on local feasibility (Table 3). Most variables were reported in 80–100% of cases, with a few notable exceptions. Organ dysfunction status was reported in only 50.5% of cases and 28 hospitals, and positive-pressure ventilation status was reported in 54.6% of cases and 30 hospitals; both variables were difficult to extract automatically (Supplemental Digital Content 2, http://links.lww.com/CCM/F681). There was a lag between the time that the central data warehouse opened to accept submissions in the collaborative and submission of the first episodes, shown in Figure 1.

Figure 1.

Completeness of sepsis data submitted to the central data warehouse over time. Percent submitted represents the aggregate number of months of data submitted among all hospitals divided by all participating hospital months of the collaborative that had occurred at that point.

The characteristics of the IPSO Sepsis and IPSO Critical Sepsis definitions using proxy measures for the domains in the Angus framework are presented in Table 2. The strengths of the definitions were identified. There was high content validity, aligning with current concepts of sepsis and septic shock. Convergent construct validity was medium to high, with 74% of IPSO Sepsis patients and 88% of IPSO Critical Sepsis patients demonstrating at least one measure of critical illness (Table 4). Concurrent criterion validity was medium to high, with hospitals reporting 80% overlap between patients identified by IPSO definitions and patients in individual hospital registries (Supplemental Digital Content 3, http://links.lww.com/CCM/F682), and predictive criterion validity was medium to high, with mortality rates similar to those reported in prior U.S. emergency department (ED)-based pediatric sepsis studies (12–15). The definitions were less strong in the domains of reliability, where significant test-retest and interrater variation was noted. Measurement burden was substantial, with medium cost, high safety, and medium complexity. Timeliness had medium performance, initially requiring a median of 15 months to submit the first cases, but accelerating to a median time of 26 days from case frequency to centralized data submission (Table 2).

TABLE 4.

Convergent Construct Validity, Restricted to the 18 Hospitals That Reported the Proxy Measures for Organ Dysfunction Below in at Least 75% of Cases

Critical Care Data Element (Proxy Measure for Organ Dysfunction)	IPSO Sepsis Episodes With This Element Reported, n	Element Present in n (%) IPSO Sepsis Episodes	IPSO Critical Sepsis Episodes With This Element Reported, n	Element Present in n (%) IPSO Critical Sepsis Episodes
ICU admission	5,214	3,728 (71.5)	1,931	1,662 (86.1)
New positive-pressure ventilation	4,443	1,539 (34.6)	1,625	711 (43.7)
Lactate ≥ 4 mmol/L	4,456	733 (16.4)	1,797	433 (24.1)
Extracorporeal membrane oxygenation	5,193	119 (2.3)	1,923	66 (3.4)
Any of the above categories	5,219	3,850 (73.8)	1,934	1,701 (87.9)

IPSO = Improving Pediatric Sepsis Outcomes.

DISCUSSION

The IPSO Sepsis definitions are pragmatic definitions that were operationalized for large-scale data abstraction in a multicenter quality initiative. The patients identified with these definitions provide an important picture of children in whom sepsis treatment was initiated. With a primary goal of identifying patients to drive QI, these definitions captured not only patients who developed sepsis with organ dysfunction, but also those in whom early sepsis was treated with organ dysfunction potentially averted. This cohort was more heterogeneous, less severely ill than described previously in studies requiring organ dysfunction or ICU admission for inclusion. This cohort represents a fuller spectrum of children treated for sepsis, and thus the children in whom it is important to measure processes of care for QI.

There was some imprecision inherent in the definitions. While the definitions ensured that identified patients had sepsis treatment initiated, it was not possible to ensure through automated parameters that sepsis treatment was completed for an illness ultimately determined to be caused by infection. The definition focused on patients in whom infection was being treated or critical care required, but it was possible for patients with other underlying etiologies to meet criteria. While this would be unacceptable for some purposes, such as a clinical trial, for this purpose, an automated definition promoted sustainable case ascertainment and quality measurement. Since the goal of QI is to enhance outcomes by reliably improving care delivery processes, identifying patients in whom clinicians initially intended to treat sepsis was in keeping with this goal, even if an alternate final diagnosis was retrospectively identified. The definition was subject to the limitation that the definition could not distinguish whether a patient truly required sepsis treatment or was over-treated. From an epidemiologic standpoint, the IPSO Sepsis cohort is expected to differ from patients in whom severe sepsis is confirmed through labor-intensive clinical criteria such as organ dysfunction, or the recently proposed pediatric sepsis surveillance definition in which specificity and reproducibility are higher priorities (5).

IPSO Sepsis cases were most often first identified in the ED, and the patient characteristics are similar to those reported in previous ED studies (12–15). The prevalence of ED cases reflects a spectrum bias of these definitions, which used the first sepsis episode to identify patients, and focused on treatment initiation as inclusion criteria, rather than ICU admission or organ dysfunction. Notably, approximately 8% of patients cared for in these pediatric settings were 18 years or greater. The care of adult patients in pediatric settings may be an important area of future study. Similar to prior studies of pediatric sepsis, a substantial proportion of patients had chronic illness.

The proxy measures of test-retest and inter-rater reliability have limitations. Although we considered variation in sepsis frequency from year to year and between hospitals to indicate less reliability, this variation could instead reflect true differences in frequency, due to infectious outbreaks or differences in patient population. The increase in IPSO Sepsis frequency from the first to the second year of the collaborative was likely due to increased sepsis recognition and treatment promoted by the collaborative, a desired QI effect. The definition might function less well if practice patterns in pediatric sepsis changed and might require updating; for example, if recommendations for fluid therapy change, the definitions could be modified accordingly.

Measurement burden and timeliness shared the requirement of a significant initial investment to establish data capture and transfer, and improvement over time. Once data gathering structures and procedures were established, the measurement burden decreased and timeliness increased, as seen in the decreasing time to data submission as hospitals submitted increasing amounts of data (Table 2). While the measurement burden for this definition cannot be considered low given the personnel time required to create local infrastructure, once established, it provided a sustainable, lower-burden data source for monitoring care over time. Once hospitals had established a data pipeline and “caught up” on data submission (as indicated by having submitted data for at least 80% of prior collaborative months), the median time from the end of a month to data submission for that month was 26 days.

In the most recent pediatric septic shock clinical guidelines from the American College of Critical Care Medicine (ACCM), the “performance bundle” was introduced. It indicated that to improve sepsis outcomes, measurement of performance is needed (16). To measure performance, a sepsis definition is required, ideally one that facilitates reliable and logistically feasible ascertainment of similar patients over time. Existing studies of the epidemiology of pediatric sepsis have ascertained cases from large administrative databases, primarily using International Classification of Diseases, 9th Revision/ICD-10 codes (17, 18). There are important limitations to these codes, and significantly for QI, these codes do not capture patients in whom early, appropriate treatment prevented deterioration. Definitions for sepsis comprised of information available in the EHR, and ascertained reproducibly over time, are important in order to improve quality in accordance with the ACCM guideline performance bundle.

Organ dysfunction is an important part of current concepts of sepsis, however, formal definitions are time-consuming to abstract manually and difficult to extract automatically from charts or the EHR. The challenge of collecting these variables was demonstrated with sites able to report organ dysfunction status for only half of IPSO Sepsis patients (Table 3). They are difficult for many hospitals to implement routinely and the IPSO Sepsis definitions are a useful alternative.

Since it was first described in 2016, the Angus framework for assessing sepsis definitions has been considered in the analysis of several tools or definitions (7, 8). Applications ranged from evaluation of ICU admission criteria to assessments of various sepsis definitions in adults (19, 20). We were limited in our ability to assess quantitatively every domain, and thus the measures reported were the best available proxies. This framework is an important acknowledgment that no single sepsis definition will suit all purposes, from prospective clinical identification to retrospective epidemiology to research trial enrollment. As research seeks to identify meaningful endotypes of sepsis patients who might benefit from precision therapeutics, there remains a need to also identify patients in the broader umbrella group of sepsis syndrome, who benefit from shared universal therapies and processes. When developing definitions, particularly for the heterogeneous entity of sepsis, this framework is a useful tool to prioritize the domains most important for the given purpose of the definition.

An additional limitation is that only the first sepsis episode in each hospitalization was recorded. This study is limited by potential reporting bias in measures which relied on self-report and the subjectivity of the authors’ post hoc rating of the performance of the definition in each domain. Finally, this study assessed these definitions during this particular QI initiative and the performance may reflect not only characteristics of the definitions, but also fidelity of implementation at the participating sites.

CONCLUSIONS

The IPSO Sepsis definitions demonstrated strengths in content validity, convergent construct validity, and criterion validity. The definitions were weaker in reliability, although IPSO Critical Sepsis, which identified a more severe subgroup of patients, had stronger reliability than IPSO Sepsis. The definitions had a significant initial measurement burden to establish data capture processes, but once established, became timely and sustainable.

The patients identified using these definitions represent one of the largest cohorts of sepsis patients identified using a shared set of definitions for QI across multiple centers reported to date to our knowledge. Despite some known limitations, these definitions identified patients in whom clinicians initially intended to treat for sepsis, and thus measuring the quality of the processes of care in these patients is an appropriate approach to sepsis QI. When operationalized, these definitions enabled multicenter case identification and data aggregation, indicating feasibility and practical utility for QI.

APPENDIX 1: THE IMPROVING PEDIATRIC SEPSIS OUTCOMES COLLABORATIVE INVESTIGATORS

Jeffery J. Auletta, MD (Hematology/Oncology/BMT and Infectious Diseases, Nationwide Children’s Hospital, Columbus, OH and Department of Pediatrics, The Ohio State University College of Medicine, Columbus, OH); Audrey H. Barnett, MSN, RN, RNC-NIC, CPHQ (Department of Quality & Safety, Children’s Memorial Hermann Hospital, Houston, TX); Sopnil Neil Bhattarai, CPHQ (Department of Performance Improvement, Children’s National Hospital, Washington, DC); David G. Bundy, MD, MPH (Department of Pediatrics, Medical University of South Carolina, Charleston, SC); Deborah R. Campbell, RN, BC, MSN, CPHQ, CCRN alumna, T-CHEST (Department of Infection Prevention and Quality Department, Kentucky Hospital Association; Louisville, KY); Patricia M. Conlon, APRN, MS, CNS, CNP (Department of Nursing, Mayo Clinic Children’s Center, Mayo Clinic, Rochester, MN); Jason W. Custer, MD (Department of Pediatrics, University of Maryland School of Medicine, Baltimore, MD); Emily C. Dawson, MD (Department of Pediatric Critical Care, Advocate Children’s Hospital, Oak Lawn, IL); Susan J. Duffy, MD, MPH (Departments of Emergency Medicine and Pediatrics, Alpert Medical School, Brown University, Providence, RI and Department of Pediatric Emergency Medicine, Hasbro Children’s Hospital, Providence, RI); Jill B. Dykstra-Nykanen, RN, MSN, CPHQ (Department of Quality and Patient Safety, Orlando Health Arnold Palmer Hospital for Children, Orlando, FL); Akinbode S. Egbelakin, MD, FAAP, MBBS, MPH (Department of Pediatrics, MercyOne Children’s Hospital, Des Moines, IA); Beth L. Emerson, MD (Department of Pediatrics, Yale University School of Medicine; New Haven, CT); Julie C. Fitzgerald, MD, PhD, MSCE (Department of Anesthesiology and Critical Care, The University of Pennsylvania Perelman School of Medicine, Philadelphia, PA and Department of Anesthesiology and Critical Care, Children’s Hospital of Philadelphia, Philadelphia, PA); Meg Frizzola, DO, FAAP (Department of Pediatrics, Nemours AIDHC, Wilmington, DE and Department of Pediatrics, Thomas Jefferson University, Wilmington, DE); Christopher M. Horvat, MD, MHA (Department of Critical Care Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA and Department of Pediatrics, UPMC Children’s Hospital of Pittsburgh, Pittsburgh, PA); Leslie A. Hueschen, MD, FAAP (Department of Pediatrics, Division of Emergency Medicine, Children’s Mercy Hospital, Kansas City, MO and University of Missouri-Kansas City, Kansas City, MO); Sarah B. Kandil, MD (Department of Pediatrics, Critical Care Medicine, Yale University School of Medicine, New Haven, CT and Yale New Haven Children’s Hospital, New Haven, CT); Raed M. Khoury, MA, MPH, CIC, MT(ASCP), CLS(CA), ARM, CPHQ, CHSP, CJCP, CSSBB, FAPIC (Department of Quality & Patient Safety, Valley Childrens Hospital, Madera, CA); V. Matt Laurich, MD (Department of Pediatrics, University of Connecticut School of Medicine, Hartford, CT and Division of Emergency Medicine, Connecticut Children’s Medical Center, Hartford, CT); Stephanie M. Lavin, MSN, RN, CPN (Clinical Collaborative Department, Cook Children’s Medical Center, Ft Worth, TX); Jeremy M. Loberger, MD (Department of Pediatrics, Division of Pediatric Critical Care, University of Alabama at Birmingham, Birmingham, AL); Justin Lockwood, MD, MSCS (Department of Pediatrics, University of Colorado School of Medicine, Aurora, CO); Elizabeth H. Mack, MD, MS (Department of Pediatrics, Medical University of South Carolina, Charleston, SC); Himi Mathur, BDS, MPH (Program for Patient Safety and Quality, Boston Children’s Hopsital, Boston, MA); Vanessa C. McFadden, MD, PhD (Department of Pediatrics, Medical College of Wisconsin, Milwaukee, WI); Erica A. Michiels, MD (Pediatric Emergency Department, Helen DeVos Children’s Hospital, Grand Rapids, MI and Department of Pediatrics, Michigan State University College of Human Medicine, Grand Rapids, MI); Kristina J. Murphy, DO (Department of Pediatric Critical Care, Cohen Children’s Hospital, Queens, NY); Nikhil Patankar, MD, MBA (Pediatric Intensive Care Unit, Beacon Children’s Hospital, South Bend, IN and Department of Pediatrics, Indiana University School of Medicine, South Bend, IN); Linda J. Perron, MSN, APRN, PCNS-BC, CCRN-K (Pediatric Intensive Care Unit, The Children’s Hospital at OU Medicine, Oklahoma City, OK); Eduardo Pino, MD (Cabell Huntington Hospital/Hoops Children’s Hospital, Huntington, WV and Department of Pediatrics, Joan C Edwards School of Medicine at Marshall University, Huntington, WV); Gregory P. Priebe, MD (Department of Anesthesiology, Critical Care and Pain Medicine, Boston Children’s Hospital, Boston, MA and Department of Anaesthesia, Harvard Medical School, Boston, MA); Sandhya Ramachandran, MPH (Quality Improvement Services, Nationwide Childrens Hospital, Columbus, OH); Faisal S. Razzaqi, MD (Department of Pediatric Hematology and Oncology, Valley Children’s Hospital, Madera, CA); Jay F. Rilinger, MD (Department of Pediatrics, University of Missouri-Kansas City, Kansas City, MO); Lori Rutman, MD, MPH (Division of Pediatric Emergency Medicine, University of Washington, Seattle Children’s, Seattle, WA); Melissa Schafer, MD (Department of Pediatrics, Upstate Medical University; Upstate Golisano Children’s Hospital, Syracuse, NY); Jerry Schwartz, RN, BSN, MHHA, NE-BC (Pediatric Intensive Care Unit, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH); Amanda M. Sebring, MD (Department of Pediatric Critical Care Medicine, Levine Children’s, Charlotte, NC); Mona D. Shah, MD, MS, MBA (Department of Pediatrics, Baylor College of Medicine, Texas Children’s Hospital, Houston, TX); Maeve Sheehan, MD, MBA (Department of Pediatrics, UT Southwestern Medical Center, Dallas, TX); Peter Silver, MD, MBA (Cohen Children’s Medical Center of New York, Queens, NY and Department of Pediatrics, Zucker School of Medicine at Hofstra/Northwell, Queens, NY); Jonathan A. Silverman, MD, MPH (Department of Emergency Medicine, Virginia Commonwealth University School of Medicine, Richmond, VA and Children’s Hospital of Richmond at VCU, Richmond, VA); Lawrence D. Spack, MD (Pediatric Critical Care Department, Arnold Palmer Hospital for Children, Orlando, FL and Orlando Health, Orlando, FL); Erika L. Stalets, MD, MS (Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, OH and Division of Critical Care Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH); Rebecca J. Stephen, MD, MS (Department of Pediatrics, Division of Hospital Based Medicine, Northwestern Feinberg School of Medicine, Chicago, IL); Hannah R. Stinson, MD (Department of Anesthesiology and Critical Care Medicine, Children’s Hospital of Philadelphia, Philadelphia, PA); Zebulon J. Timmons, MD, FAAP (Department of Emergency Medicine, Phoenix Children’s Hospital, Phoenix, AZ); Beth Wathen, MSN RN CCRN-K (Pediatric Intesive Care Unit, Children’s Hospital Colorado, Aurora, CO); Michele A. Wilson, MS, RN, NP, CCNS, CCRN-K (Pediatric Intensive Care Unit, Loma Linda University Children’s Hospital, Loma Linda, CA); and Jennifer K. Workman, MD, MSCI (Department of Pediatrics, University of Utah School of Medicine, Salt Lake City, UT and Primary Children’s Hospital, Salt Lake City, UT).