Reply to Adelman et al

To the Editor—We appreciate Adelman and colleagues’ [1] thoughtful response to our study [2]. We agree that sepsis diagnosis is complex and that it is important to consider whether our adjudication protocol yielded reproducible and accurate results. One potential limitation noted is that the final presence of infection was adjudicated by more than 1 person for only a subset of our cohort. However, interrater agreement was high and comparable to past studies [3, 4]—κ = 0.69 (95% confidence interval [CI]: .60–.78) for the binary determination of infection presence and κ = 0.83 (95% CI: .80–.86) for infection presence and source [5] — and there was not a systematic pattern of between-rater disagreement (Figure 1). As such, it seems unlikely that additional two-reviewer adjudications would have substantially altered our findings.

Figure 1.

Alluvial diagram illustrating the final presence and source of infection for 812 emergency department patients who met Sepsis-3 criteria adjudicated by independent reviewers using structured medical record review. Block sizes are proportional to the number of patients with each infection status/source during each round of adjudication. “Streams” between initial abstraction and validation results depict patterns and proportions of interrater agreement and disagreement for each infection status/source. Abbreviation: GI, gastrointestinal.

We employed a panel of trained research assistants and medical students for most chart adjudication and validation. While this pragmatic strategy allowed us to evaluate a cohort substantially larger than the cited studies that used multiphysician adjudication (n = 211 [6], n = 447 [7], and n = 2579 [8]), we concur that the use of nonclinician adjudicators is a possible limitation of our study. However, as noted by Adelman and colleagues, even physician adjudication is not perfect, with the determination of the presence of sepsis varying substantially between physicians. In one study, clinicians who were given a series of case vignettes but no structured adjudication criteria exhibited poor agreement regarding the presence or absence of sepsis (κ only 0.18) [9]. By contrast, a study that used structured adjudication criteria had interrater agreement similar to the agreement in our study (κ = 0.79) between a two-physician panel with all available discharge information and a “gold standard” external 3-physician panel [3]. Taken together, these data suggest that expert judgment is insufficient for reliable and reproducible sepsis adjudication, while reproducible adjudication criteria like those used in our study are critical. It is also worth considering whether physicians can bring systematic biases to the adjudication task. A recent study found that infectious diseases specialists had higher thresholds for recognizing and treating infection than critical care and emergency medicine physicians [10], suggesting that the training background of physician adjudicators could influence study findings.

Overall, we believe that structured sepsis adjudication by comprehensively trained nonclinician personnel using objective criteria formulated by a panel of expert infectious diseases, emergency department, and critical care physicians provided an unbiased and reproducible estimate of false-positive presumptive infection diagnosis rates among emergency department patients meeting sepsis criteria.

Notes

Financial support. This work was funded by grants from the Intermountain Research and Medical Foundation and the National Institute of General Medical Sciences (K23GM129661 to I. D. P.) and the National Heart, Lung, and Blood Institute (T35HL007744 to G. A. H.).