TABLE 2.

Evaluation of Improving Pediatric Sepsis Outcomes Sepsis Definitions Using the Angus Framework (7) for Development and Interpretation of Sepsis Definitions

Domain	Definition	Proxy Measure of This Domain in IPSO Definition	Result (IPSO Sepsis)	Result (IPSO Critical Sepsis)	Definition Performance (Low, Medium, High)
1) Reliability: Criteria yield stable reproducible results
Test-retest	When tests are repeated	Median change in frequency from 2017 to 2018 (greater change = less reliability)	Median increase 2.44 cases/1,000 admissions (IQR, 0.13–7.74)	Median increase 0.70 cases/1,000 admissions (IQR, −0.68 to 1.87)	Sepsis: Medium Critical Sepsis: Medium to High
Inter-rater	When tests are interpreted across different raters	Among all hospitals, median frequency per 1,000 admissions (proxy for interrater reliability, greater IQR = less reliability)	20.4 cases/1,000 admissions (IQR, 9.1–29.3)	6.4 cases/1,000 admissions (IQR, 3.3–9.1)	Sepsis: Medium Critical Sepsis: Medium to High
2) Content validity
Content validity	Criteria fit with current understanding and knowledge	Qualitative subject matter expert review	This definition implies clinically perceived sepsis based on treatment. This is useful for QI but is subject to limitations of inappropriate treatment; does not reflect all organ dysfunction	This definition is similar to current understanding of shock/cardiovascular dysfunction but is also limited by being subject to the appropriateness of treatment	High
3) Construct validity: Criteria measure what they purport to measure
Convergent	The extent to which two or more aspects that should agree do agree (occur together)	% of patients who have intensive care admission, positive-pressure ventilation, lactate ≥ 4 mmol/L, or extracorporeal membrane oxygenation	74% (see Table 4 for details of each measure)	88%	Medium to high
4) Criterion Validity: New criteria agree with existing standard
Concurrent	Comparison to a current standard available at the same time	Six hospitals with preexisting sepsis registries reported approximate percent of patients in registry also identified by IPSO definitions	80% (Additional detail in Supplemental Digital Content 2, http://links.lww.com/CCM/F681)		Medium to high
Predictive	Comparison to a later outcome believed to be strongly associated with the disease of interest	Overall mortality Sepsis-specific mortality	4.0% 2.5%	6.3% 3.7%	Medium to high
			Similar to existing pediatric sepsis studies from emergency departments with QI (12–15), lower than International Classification of Diseases, 10th Revision coded hospital-wide sepsis mortality (17, 18)
5) Measurement burden: Burden to implement criteria
Lower cost	Financial costs	Qualitative subject matter expert review	There was no direct clinical cost to the patient at bedside. Personnel needs were greatest in establishing data capture and diminished for maintenance. Typically needed at minimum a medical informaticist, a physician, a nurse, and process improvement specialist with a portion of time on this project		Medium
		Self-report of sites using manual abstraction	Sites using manual data abstraction reported case ascertainment using the definition required 3–20 min per case
Greater safety	Side effects, complications to patient	Qualitative subject matter expert review	Little to no direct safety concern. Loss of confidentiality is a theoretical risk, however, no data breaches occurred		High
Lower complexity	Difficulties executing the steps to obtain/interpret the measures	Qualitative subject matter expert review	Definition was complex to initially create in each hospital’s infrastructure, however, easier to routinely report Definition was flexible to each hospital’s internal processes		Medium
6) Timeliness
Timeliness	Speed with which criteria are generated with respect to the course of the disease	Median (IQR) to first case submission Median (IQR) to submission after first submission	15 mo (IQR, 13–18 mo) 44 d (IQR, 24–82 d)		Timeliness for first case identification: Low to medium
		Median (IQR) to submission after 50% submission	31 d (IQR, 23–76 d)
		Median (IQR) to submission after 80% submission	26 d (IQR, 23–56 d)		Timeliness for ongoing submission: Medium

IPSO = Improving Pediatric Sepsis Outcomes, IQR = interquartile range, QI = quality improvement.