To the Editor
Dr Seymour and colleagues1 assessed the predictive validity of various clinical criteria to identify patients with sepsis. However, their conclusion that the Sequential [Sepsis-related] Organ Failure Assessment (SOFA) and quick SOFA (qSOFA) scores are more clinically useful than the systemic inflammatory response syndrome (SIRS) criteria was primarily based on differences in the area under the receiver operating curve (AUROC), which has several limitations.2
First, differences in the AUROC may be of minimal clinical relevance. For example, although the American College of Cardiology/American Heart Association (ACC/AHA) and the Adult Treatment Panel III cardiovascular risk prediction models have similar AUROCs, the ACC/AHA model recommends statin therapy for more adults and better reclassifies patients according to their true risk (net reclassification index [NRI], 0.332).3,4 To better assess for clinically meaningful differences in the utility of different sepsis diagnostic criteria, the authors should report the overall categorical NRI (for ≥2 vs <2 points), event and nonevent NRIs, and risk reclassification tables.
Second, Seymour and colleagues proposed implementing both the SOFA and qSOFA scores in practice by defining patients with scores of 2 points or greater as having “sepsis” and those with less than 2 points as not having sepsis. However, the reported AUROCs were calculated using continuous scores. Assessing the performance of each set of clinical criteria using the dichotomous categorization and reporting the positive and negative predictive values and corresponding likelihood ratios compared with the SIRS criteria would be more clinically relevant than reporting the differences in the AUROC or the relative difference in outcomes across deciles of baseline risk.
Most important, it is unclear that the lower AUROC reflects inferior predictive validity of the SIRS criteria. Rather, the lower AUROC(and relative fold difference in outcomes)may reflect the influence of incorporation bias. The current standard of care for clinicians during the period of this study was to initiate early goal-directed therapy for patients with suspected sepsis based on SIRS criteria. Thus, the lower performance of SIRS for predicting mortality may simply be an artifact— patients diagnosed as having sepsis by 2 or more SIRS criteria were more likely to have received early goal-directed therapy and less likely to die as a consequence.5
Before the current diagnostic criteria for sepsis are replaced, we believe that more robust analyses are needed, including a prospective study of the utility of these various clinical criteria that would eliminate the influence of incorporation bias.
Footnotes
Conflict of Interest Disclosures: The authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported.
References
- 1.Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis. JAMA. 2016;315(8):762–774. doi: 10.1001/jama.2016.0288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928–935. doi: 10.1161/CIRCULATIONAHA.106.672402. [DOI] [PubMed] [Google Scholar]
- 3.Kavousi M, Leening MJ, Nanchen D, et al. Comparison of application of the ACC/AHA guidelines, Adult Treatment Panel III guidelines, and European Society of Cardiology guidelines for cardiovascular disease prevention in a European cohort. JAMA. 2014;311(14):1416–1423. doi: 10.1001/jama.2014.2632. [DOI] [PubMed] [Google Scholar]
- 4.Paixao AR, Ayers CR, Berry JD, de Lemos JA, Khera A. Atherosclerotic cardiovascular disease prevention. Circ Cardiovasc Qual Outcomes. 2014;7(5):778–779. doi: 10.1161/CIRCOUTCOMES.114.001139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Mouncey PR, Osborn TM, Power GS, et al. Trial of early, goal-directed resuscitation for septic shock. N Engl J Med. 2015;372(14):1301–1311. doi: 10.1056/NEJMoa1500896. [DOI] [PubMed] [Google Scholar]