Weiwei et al. comment on the low kappa statistics and wide confidence intervals for the heart rate and respiratory rate criteria of the pediatric SIRS definition reported in our publication (1, 2).
The participants in the International Conference that defined the systemic inflammatory response syndrome (SIRS) in pediatric patients recognized that heart rate and respiratory rate can vary substantially, even over short periods of time, and may be affected by multiple other factors (3). For this reason, they categorized abnormal temperature or peripheral white blood as “major” criteria, at least one of which must be present, and abnormal heart rate and respiratory rate criteria as “minor” criteria.
Nonetheless, the comments by Weiwei et al. point out a peculiarity in the measurements of reliability—the kappa statistic and the percent agreement in—our report. That is, the kappa statistic may be low even when the percent agreement is high. This paradox was addressed in two papers by Feinstein and Cicchetti (4, 5). To illustrate how this may occur using our data, the table shows the kappa statistic and the percent agreement overall, which were reported in our publication, and the percent positive agreement (agreement on the presence of the criterion) and the negative agreement (agreement on the absence of the criterion), which we have calculated using the method of Cicchetti (5) and the numbers in Table 2 of our publication. The kappa statistic for respiratory rate is low (0.31) despite a fairly high percent agreement overall (84%)—a percent agreement equivalent to that for temperature and peripheral white blood cell count that both had relatively high kappa statistics. This finding is due to the confluence of two factors in our data. First, per the SIRS definition (3), the use of mechanical ventilation satisfies the respiratory rate criterion. Consequently, in our data, the prevalence of a positive (present) respiratory criterion was high (Table) and, conversely, the prevalence of a negative (absent) respiratory criterion was low. Second, the percent negative agreement (agreement between reviewers on the absence of the criterion) was only 40% (Table). In contrast, the kappa statistic for heart rate is low because all of the measures of agreement—the percent agreement overall, the percent positive agreement, and the percent negative agreement—were all relatively low.
Table.
Measures of Reliability in the Assessment of Individual SIRS Criteria
| SIRS Criteriona | Prevalenceb | Kappa statistic | Percent Agreementc | Percent Positive Agreementd | Percent Negative Agreementd |
|---|---|---|---|---|---|
| Temperature | 53% | 0.67 | 84% | 87% | 80% |
| White Blood Cell Count | 42% | 0.79 | 89% | 89% | 90% |
| Respiratory Rate | 79% | 0.31 | 84% | 91% | 40% |
| Heart Rate | 26% | 0.36 | 68% | 63% | 73% |
SIRS, systemic inflammatory response syndrome
As defined by the Internation Pediatric Sepsis Consensus Conference (3)
Number of cases in which both reviewers agreed the SIRS criterion was present divided by total number of SIRS cases
Number of cases in which both reviewers agreed the SIRS criterion was either present or absent divided by total number of SIRS cases
Calculated the method of Cicchetti (5)
These findings illustrate the wisdom of the International Conference participants in relegating the respiratory rate and respiratory criteria to “minor” status. Indeed, the application of the SIRS definition overall was quite reliable in our study because the “major” criteria—temperature or white blood cell count—demonstrated good or very good kappa statistics and overall percent agreement (Table).
Our estimates of the kappa statistic for individual criteria would have been more precise (i.e., the confidence limits would be narrower) if we had included more subjects in the study; however, determining the reliability of individual criteria was a secondary goal of our study.
Weiwei et al. also expressed concern that the reviewers were not blinded from the clinical diagnoses during their review of individual cases and, therefore, may have been biased in their assessments of the SIRS criteria. Since it was not feasible to blind the reviewers to portions of the electronic medical record, we instructed the reviewers to first assess the presence or absence of SIRS criteria using vital sign and laboratory reports. They were instructed to then access other portions of the electronic medical record, including clinical notes that listed diagnoses, as needed to determine if qualifying factors were present, as described in the SIRS definition (3). Consequently, the effect of such a bias was likely to be minimal.
Acknowledgments
The authors received funding from the National Institutes of Health (NIH).
References
- 1.Weiwei J, Zhengdong K, Min L. Evaluation of respiration and heart rate decreases reliability of the pediatric systemic inflammatory response syndrome definition. Pediatr Crit Care Med. doi: 10.1097/PCC.0b013e31823c9909. (in press) [DOI] [PubMed] [Google Scholar]
- 2.Juskewitch JE, Prasad S, Salas CF, et al. Reliability of the identification of the systemic inflammatory response syndrome in critically ill infants and children. Pediatr Crit Care Med. doi: 10.1097/PCC.0b013e31822f177a. (in press) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Goldstein B, Giroir B, Randolph A, et al. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatric Critical Care Medicine. 2005;6(1):2–8. doi: 10.1097/01.PCC.0000149131.72248.E6. [DOI] [PubMed] [Google Scholar]
- 4.Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543–549. doi: 10.1016/0895-4356(90)90158-l. [DOI] [PubMed] [Google Scholar]
- 5.Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43(6):551–558. doi: 10.1016/0895-4356(90)90159-m. [DOI] [PubMed] [Google Scholar]