Reliability of the Identification of the Systemic Inflammatory Response Syndrome in Critically Ill Infants and Children

Abstract

Objective

To assess inter-observer reliability of the identification of episodes of the systemic inflammatory response syndrome (SIRS) in critically ill hospitalized infants and children.

Design

Retrospective cross-sectional study of the application of the 2005 consensus definition of SIRS in infants and children by two independent trained reviewers using information in the electronic medical record.

Setting

18-bed pediatric multidisciplinary medical/surgical pediatric intensive care unit (PICU).

Patients

A randomly selected sample of children admitted consecutively to the PICU between May 1 and September 30, 2009.

Measurements and Main Results

60 infants and children were selected from a total of 343 admitted patients. Their median age was 3.9 years (interquartile range [IQR], 1.5–12.7), 57% were female, and 68% were Caucasian. 19 (32%) children were identified by both reviewers as having an episode of SIRS (88% agreement, 95% confidence interval [CI] 78–94; κ = 0.75, 95% CI, 0.59–0.92). Amongst those 19 children, agreement between the reviewers for individual SIRS criteria was: temperature (84%, 95% CI 60–97); white blood cell count (89%, 95% CI 67–99); respiratory rate (84%, 95% CI 60–97); heart rate (68%, 95% CI 33–87).

Conclusions

Episodes of SIRS in critically ill infants and children can be identified reproducibly using the consensus definition.

INTRODUCTION

Sepsis is among the top ten causes of death in the United States, with the highest incidences in infants and elderly adults.(1) Among infants and children, an estimated 42,000 episodes of severe sepsis occur in the United States each year, resulting in ~4,300 deaths and an annual treatment cost of ~$1.8 billion. (2)

In 1991, a consensus conference defined sepsis in adults as the systemic inflammatory response syndrome (SIRS) in association with a confirmed or suspected infection. (3, 4) SIRS was defined by abnormalities of temperature, heart rate, respiratory rate, and peripheral white blood cell count. In adults, this definition demonstrated good inter-observer reliability (κ = 0.56) that improved to excellent (κ = 0.91) with additional training and experience of the observers. (5) In 2002, a separate consensus conference defined SIRS in infants and children. (6) The pediatric SIRS definition (Table 1) is more detailed, including six sets of age-specific criteria for abnormalities of heart rate, respiratory rate, and white blood cell count, as well as more complex, including qualifying statements to enhance the specificity of the definition by excluding abnormalities not due to infection (e.g., bradycardia due to administration of a beta-blocker, leukopenia due to administration of cytotoxic chemotherapy). These definitions were intended to standardize the identification of SIRS in children for both clinical and research purposes, but inter-observer reliability in the identification of episodes of SIRS using this definition has not been reported.

Table 1.

Consensus Conference Definition of the Systemic Inflammatory Response Syndrome in Infants and Children^a The systemic inflammatory response syndrome is defined as the presence of at least two of the following four criteria, one of which must be an abnormal core temperature or an abnormal leukocyte count:

Criteria	Definition
Core temperatureb	>38.5°C or <36°C
Heart rate	tachycardia defined as mean heart rate >2 SD above normal for age (see below) in the absence of external    stimulus, chronic drugs, or painful stimuli or otherwise unexplained persistent elevation over a 0.5- to 4-hr    time period
	0 days – 1 month: >180 beats/minute
	1 month – 1 year: >180 beats/minute
	1 year – 5 years: >140 beats/minute
	5 – 12 years: >130 beats/minute
	12 – 17 years: >110 beats/minute
	or bradycardia (children <1 year only) defined as a mean heart rate <10th percentile for age (see below) in the    absence of external vagal stimulus, beta-blocker drugs, or congenital heart disease; or otherwise unexplained    persistent depression over a 0.5-hr time period
	0 days – 1 month: 100 bpm
	1 month – 1 year: <90 bpm
Respiratory rate	tachypnea, defined as mean respiratory rate >2 SD above normal for age or mechanical ventilation for an acute    process not related to underlying neuromuscular disease or the receipt of general anesthesia
	0 days – 1 week: >50 breaths/minute
	1 week – 1 month: >40 breaths/minute
	1 month – 1 year: >34 breaths/minute
	1 year – 5 years: >22 breaths/minute
	5 – 12 years: >18 breaths/minute
	12 – 17 years: >14 breaths/minute
Leukocyte count	elevated for age (see below)
	0 days - 1 week: >34,000/mm3
	1 week – 1 month: >19,500/mm3
	1 month – 1 year: >17,500/mm3
	1 year – 5 years: >15,500/mm3
	5 – 12 years: >13,500/mm3
	12 – 17 years: >11,000/mm3
	or depressed for age (see below) not secondary to chemotherapy-induced leukopenia
	0 days – 1 week: not applicable
	1 week – 5 years: <5,000/mm3
	5 – 17 years: <4,500/mm3
	or >10% immature neutrophils

C, centrigrade; SD, standard deviation; bpm, beats per minute

^aModified from reference 6

^brectal, bladder, oral, or central catheter measurement

The objective of this study was to assess inter-observer reliability of the identification of episodes of SIRS in critically ill hospitalized infants and children using information in the electronic medical record.

MATERIALS AND METHODS

Children admitted consecutively to the Mayo Eugenio Litta Children Hospital’s pediatric medical/surgical intensive care unit (PICU) in Rochester, Minnesota between May 1, 2009 and September 30, 2009 were identified retrospectively. This PICU provides care to patients with medical conditions or who have undergone multidisciplinary surgical procedures (excluding cardiovascular surgery) or transplantation (liver, kidney, hematopoietic stem cell). As required by Minnesota statute, patients whose parents refused to give research authorization were excluded. A subset of 60 of these subjects was selected using a random number generator since 60 subjects would estimate 90% agreement with a precision (one-sided confidence interval width) of 7.6% using a two-sided type I error of 0.05. If a child had more than one PICU visit during the study period, one visit was selected randomly for review. The study was approved by the Mayo Clinic Institutional Review Board with a waiver of informed consent.

Two trained reviewers examined data in the electronic medical record of the selected PICU visits retrospectively and independently. Reviewers identified abnormalities of temperature, heart rate, respiratory, and peripheral white blood cell count that occurred during the PICU visit and determined if they were sufficient to satisfy the pediatric SIRS definition criteria considering also any exclusionary conditions (Table 1). (6) Clinical diagnoses were not considered. If the reviewer determined that the SIRS criteria were satisfied, he/she recorded the date and time of the onset of the SIRS episode and the specific criteria that were satisfied. If multiple episodes of SIRS were identified, only the first episode was recorded.

Study data were recorded and managed using the Research Electronic Data Capture (REDCap) system. (7) Statistical analyses were performed using JMP version 8 (Cary, NC). Inter-observer reliability of the SIRS definition was assessed using percent agreement and the kappa statistic. Qualitative interpretations of kappa statistics were made using published definitions. (8) For those subjects whom both reviewers identified an episode of SIRS, the percent agreement and kappa statistics were assessed for individual SIRS criteria. In these subjects, agreement on the date and time of the first SIRS episode was compared using Wilcoxon Signed Rank test. 95% confidence intervals (CI) were calculated using the exact or approximate binomial method as appropriate. A p-value < 0.05 was considered statistically significant.

RESULTS

Between May 1, 2009 and September 30, 2009, 343 children were admitted to the PICU whose parents had given research authorization. These 343 children had a median age of 6.7 years (IQR, 2.0 – 13.5), 48% were female, and 76% were Caucasian. From these 343 children, 60 (18%) were randomly selected for review of their electronic medical records; these children had a median age of 3.9 years (IQR, 1.5 – 12.7), 57% were female, and 68% were Caucasian.

Among these 60 children, 19 (32%) were identified by both reviewers as having an episode of SIRS (Table 2; 88% agreement, 95% CI 78–94; κ = 0.75 [very good]; 95% CI, 0.59–0.92). Among these 19 children, the median difference in the time of onset of the SIRS episode identified by the reviewers was 0 hours (IQR, −1.75-0; p = 0.275).

Table 2.

Agreement between Two Independent Expert Reviewers in Identifying Episodes of SIRS in Pediatric Intensive Care Unit Patients

	Reviewer 1
	No	Yes
Reviewer 2
SIRS episodea
No	34	1b
Yes	6b	19
Criteria associated with SIRS episodesc
Temperature
No	6	1
Yes	2	10
Peripheral white blood cell count
No	9	0
Yes	2	8
Heart rate
No	8	2
Yes	4	5
Respiratory rate
No	1	1
Yes	2	15

SIRS, Systemic Inflammatory Response Syndrome

^aAgreement between Reviewers 1 and 2 on identification of SIRS episodes in 60 randomly selected subjects from an overall cohort of 343 consecutively admitted subjects; SIRS definition requires satisfaction of at least 2 individual SIRS definition criteria, at least 1 which must be temperature or peripheral blood white blood cell count (6)

^bDisagreements were due to a missed vital sign (heart rate or respiratory rate), misapplication of age-specific criteria, and lack of recognition of chemotherapy-induced leukopenia

^cAgreement between Reviewers 1 and 2 on satisfaction of individual SIRS definition criteria in the 19 patients with SIRS episodes identified by both reviewers

Among these 19 children, agreement between reviewers on individual SIRS definition criteria (Table 2) was higher for the major criteria of temperature (84%, 95% CI 60–97; κ = 0.67 [good], 95% CI = 0.31–1) and white blood cell count (89%, 95% CI 67–99; κ = 0.79 [very good], 95% CI 0.52–1) and the minor criterion of respiratory rate (84%, 95% CI 60–97; κ = 0.31 [slight], 95% CI 0–0.90) than for the minor criterion of heart rate (68%, 95% CI 33–87; κ = 0.36 [slight], 95% CI 0–0.77).

DISCUSSION

This study is the first to report the reproducibility of identifying episodes of SIRS in infants and children using the published consensus pediatric SIRS definition. Despite the detailed and complex nature of the definition, we found good to very good inter-observer reliability (as determined by kappa statistics) in the identification of SIRS episodes between two trained reviewers who examined relevant data in the electronic medical records of PICU patients retrospectively. These findings compare favorably with those reported previously in adults using the simpler adult SIRS definition. (5)

Moreover, we found no difference in the time of onset of the SIRS episode identified by the two reviewers, good to very good reliability with respect to identifying major SIRS criteria (temperature, white blood cell count) satisfied by specific episodes of SIRS, and a high level of agreement for all SIRS criteria except heart rate (84–89% versus 68%). However, the reliability of identifying minor criteria (respiratory rate, heart rate) was not as good, as reflected by the substantially lower kappa statistics. Since abnormalities of the respiratory and heart rates alone cannot satisfy the pediatric SIRS definition, the impact of the lower reproducibility of these minor criteria may be less important.

Nonetheless, the 12% disagreement between observers documented in this study has important implications for research studies that use the presence of SIRS as a component of their eligibility criteria or outcome measures, especially multicenter studies in which variability in data sources, case finding methods, and observer interpretation of the SIRS criteria may increase the frequency of disagreement. As has been demonstrated in adult studies, training on the application of the definition, particularly with respect to the minor criteria of heart and respiratory rate, may reduce disagreement. (5)

This study has limitations. The reviewers examined data in the electronic medical record of PICU patients retrospectively; consequently, the reproducibility of the identification of SIRS episodes we report cannot be extrapolated to prospective studies, such as the identification of SIRS in children in real-time, or in other settings, such as the emergency room or outpatient settings. The study was conducted in a single center over a short period with a consistent electronic medical record system, so the impact of variability in the characteristics of the electronic medical record on the reproducibility of the identification of SIRS episodes could not be assessed. Finally, the limited number of subjects limits the precision of our estimates of the percent agreement and kappa statistic.

CONCLUSIONS

These findings indicate that the definition of SIRS in children, which serves as a foundation for the definitions of sepsis, severe sepsis and septic shock in children, can be applied reproducibly by different individuals using data in the electronic medical record. Based on these findings, we conclude that the pediatric SIRS definition is sufficiently robust for use in clinical and translational research, assuming that adequate training of observers is provided and reproducibility is confirmed under the particular circumstances of individual studies.