Sepsis Prediction Models are Trained on Labels that Diverge from Clinician-Recommended Treatment Times

Abstract

Many sepsis prediction models use the Sepsis-3 definition or its variants as a training label. However, among the few sepsis models ever deployed in practice, there is scant evidence that they offer clinically meaningful decision support at the bedside. As a potential mechanism to explain this limitation, we hypothesized that clinician-recommended treatment times for sepsis would diverge from onset time defined by Sepsis-3. We conducted an electronic survey that was completed by 153 clinicians at three large and geographically diverse medical centers using vignettes derived from eight real cases of sepsis. After reviewing these vignettes, participants suggested antibiotic treatment to start an average of 7.0 hours (95% confidence interval 5.3 to 8.8) before the Sepsis-3 definition onset. Thus, predicting Sepsis-3 onset as a treatment prompt could lead to inappropriate and delayed treatment recommendations. Building predictive decision support systems that identify outcomes aligned with bedside decisions would increase their clinical utility.

Introduction

Sepsis is among the most common clinical syndromes for which researchers have trained machine learning and artificial intelligence models to support early warning systems (EWSs)^1,2. In 2019, over 100 teams from academia and industry participated in a sepsis prediction data science competition.³ This massive effort is proportional to the epidemiologic burden of sepsis which contributes to significant mortality and resource utilization in the United States each year^4,5. A sepsis EWS is a promising tool to improve patient care because early recognition and treatment are infrequently achieved yet are the mainstays of therapy^6–8. Indeed, many sepsis models exhibit strong predictive performance when evaluated in retrospective datasets.’ However, very few sepsis prediction models have ever been deployed in clinical workflows. Among those that have been implemented prospectively, most exhibit poor predictive performance, may alter testing or treatment decisions but not patient outcomes, and/or have been reported by physicians and nurses as not useful at the bedside^9–3.

There is no clear evidence to suggest a primary cause for this consistent discrepancy between strong retrospective predictive performance and minimal bedside usefulness for sepsis prediction models. Emerging data from the clinical informatics literature suggests “label bias” — the divergence of a training label from its intended real-world target — as an important cause of racial bias in clinical prediction models^14,15. Label bias may also lead to discrepancies between a model’s predictions and its intended clinical use, thereby undermining the clinical utility of its predictions when used at the bedside. In the case of sepsis, training labels are often derived from Sepsis-3 or Centers for Disease Control and Prevention (CDC) Adult Sepsis Event (ASE) criteria^16,17, definitions that were not primarily intended to guide bedside treatment decisions^18–20. Instead, they were developed primarily to standardize clinical trial enrollment, epidemiologic surveillance, and reporting of quality measures^16,21. Thus, using such labels for sepsis EWS risks undermining the usefulness of such models through clinical label bias.

Identifying a discordance between clinician judgement about bedside treatment decisions — the task for which sepsis EWSs are intended to inform — and commonly used training labels would highlight an opportunity to improve the clinical relevance of sepsis-focused prediction models. We hypothesized that clinician judgement about when to initiate antibiotics for patients suspected of sepsis would precede the timing of onset according to definitions commonly used as training labels for such models. Therefore, we conducted a cross-sectional survey of practicing clinicians across three diverse hospital systems to estimate the relationship between a clinician’s judgement about treatment timing and the time of sepsis onset according to criteria commonly used for training sepsis prediction models.

Methods

Participants and recruitment. We emailed an electronic survey instrument to curated lists of clinicians at the University of Pennsylvania Health System in Philadelphia, Pennsylvania, the University of Michigan Health System in Ann Arbor, Michigan, and Atrium Health in Charlotte, North Carolina. Mailing lists were generated from preexisting faculty and staff lists when available and obtained by the investigators at their home institutions or curated based on publicly available information on organizational web sites. Recruitment efforts targeted clinicians in critical care, emergency medicine, infectious diseases, and hospital medicine, all of whom care for sepsis patients. Specifically, all attending physicians, advanced practice providers (nurse practitioners and physician assistants), and subspecialty fellows (in pulmonary/critical care and infectious diseases programs), when their contact information was available, were included in recruitment emails sent between August and November 2020. Two reminder emails were sent after the initial email. Participants documented electronic consent and were given the option to enter into a raffle for one of four $250 gift cards allocated to each of the three study sites. Response rates were calculated according to the American Association for Public Opinion Research²².

Sample size estimation. We planned to recruit 120 participants based on simulations to achieve 80% power with α = 0.05 to detect a 3-hour difference between clinician-recommended treatment time and Sepsis-3 onset across 8 vignettes. These calculations were robust to an intra-class correlation ranging from 0.1 to 0.3.

Clinical vignettes and instrument development. Clinical vignettes were generated from eight randomly sampled, real cases of sepsis that occurred in the University of Pennsylvania Health System between March 2017 and December 2019. A ninth clinical vignette in which the patient did not develop sepsis or any organ dysfunction was included as an internal control. Each vignette was presented with the patient’s age, admitting diagnosis, and temporal trends in heart rate, systolic blood pressure, temperature, respiratory rate, serum lactic acid, peripheral oxygen saturation, serum creatinine, white blood cell count, and Glasgow Coma Scale score (GCS) over a period of about 100 hours during the hospitalization (Figure 1). Each window was approximately centered around the time the patient first met Sepsis-3 criteria. The choice of variables and data presented in the vignettes was based on feedback from two pilot studies with 34 clinicians from internal medicine, critical care medicine, infectious diseases, and other medical specialties. Because appropriateness of antibiotic initiation may be influenced by factors not recorded in the electronic health record (EHR) or presented in the vignette, we chose to include vignettes from patients with different observed antibiotic delays. Specifically, two vignettes were randomly sampled from each of the following strata based on the observed time in hours from Sepsis-3 onset to antibiotic initiation: (0, 6], (6, 12], (12, 24], and [24, 48], for a total of eight vignettes. Each participant was presented with all 9 vignettes in a randomized order. Participants were asked to review the vignettes to assess, in retrospect, if and when would have been a clinically optimal time to initiate treatment.

Figure 1:

An educational module explaining the structure and content of each vignette was presented to all study participants.

Defining sepsis onset. We identified sepsis onset using a modified Sepsis-3 approach^16,23. This definition was implemented using data from the EHR to identify (1) a blood culture draw, (2) new broad-spectrum antibiotic initiation within 2 days of the blood culture, and (3) an increase in Sequential Organ Failure Assessment (SOFA) score by at least two points from baseline within the same window. The hour of sepsis onset was identified as the time that the earliest of these criteria was met in accordance with prior approaches²⁴. Given known variation in sepsis onset time when using different sepsis definitions^24,25, we conducted a sensitivity analysis using the CDC ASE criteria to identify the hour of sepsis onset¹⁷. Although the CDC ASE definition was developed for epidemiologic surveillance to be used at the calendar-day rather than at the hour level²⁶, it is still sometimes used for time-sensitive benchmarking and comparisons of predictive models^9,25.

Statistical analysis. We fit a mixed-effects linear regression model adjusted for clinical specialty with vignette fixed effects and nested site and participant random effects for the continuous outcome of the difference in hours between Sepsis-3 onset and suggested antibiotic start time. Cases where the participant did not recommend any antibiotics were excluded from the primary analysis. To assess mean differences between these times we calculated marginal estimates standardized across vignettes. The same modeling approach was used in the sensitivity analysis that identified sepsis onset using the CDC ASE criteria.

In a secondary analysis, we compared the differences between the suggested and observed antibiotic start times as documented in the EHR. In another secondary analysis, we included all participant responses and fit a Cox proportional hazards model to estimate the time to antibiotic initiation in which those that did not suggest antibiotic initiation were considered right-censored. In this model, a participant identifier was included as a frailty term.

In a sensitivity analysis, we refit the primary model with adjustment for the category of observed antibiotic delay (as described above) as a check against the antibiotic initiation criterion in the definition of sepsis onset introducing an artifact in a potentially circular assessment of time differences.

We reported the standard deviation of suggested start times overall and by vignette as a measure of agreement among participants. All analyses were conducted with the R language for statistical computing version 4.0 (R Foundation for Statistical Computing, Vienna, Austria) with additional supporting libraries for data cleaning, analysis, and visualization^27–36. We adhered to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for reporting cross-sectional studies^37,38. All of the de-identified data included in the vignettes and the de-identified participant responses from the survey were made publicly available for transparency and reproducibility in an online data repository³⁹.

Manual comparison of vignettes and annotations. After completing the primary study, we conducted a post hoc analysis to identify patterns in the clinical data that may have influenced participants’ responses. Five authors who are practicing physicians (GEW, SDH, SPT, RK, and JCG) reviewed each vignette to identify clinically relevant patterns in the presented data that coincided with the median annotated antibiotic start time. For example, in vignette 4, the median suggested antibiotic start time coincides perfectly with the time of the first measurement of lactic acid, even though that and subsequent values were all within normal limits. Each reviewer identified up to five relevant patterns in the data potentially associated with each median suggested start time. In cases where there was an obvious bimodal distribution of suggested antibiotic times for a single vignette, clinical features were reported for the median of each peak. Consensus was defined as a data point being identified by at least four of the reviewers as contributing to a participant’s choice to initiate antibiotics. Identified patterns were aggregated for each vignette and discrepancies were resolved by group discussion and consensus. Because some responses appeared to have been based on the presence of new data points (e.g. the first time a lactic acid value was revealed) rather than the values themselves (e.g. in cases where the lactic acid was normal), we also conducted a post hoc secondary analysis in which we repeated the primary analysis but excluded responses in which the suggested antibiotic start time was within 2 hours of such newly presented (but not necessarily abnormal) values.

Results

Of 923 eligible clinicians across the three sites, 228 (25%) initiated and 153 (17%) completed the survey (Figure 2). Among these, 139 (91%) were physicians, 52 (34%) worked in critical care, and 150 (98%) had treated a patient suspected of sepsis in the preceding 12 months (Table 1).

Figure 2:

Flow diagram showing the process of participant recruitment and enrollment.

Table 1:

Characteristics of study participants. Years since completing school refers to the number of years since graduating from the last clinical program,including medical school or nursing school. Respondents who indicated an “Other” specialty included: “Pulmonary”, “Outpatient Internal Medicine ” and “Pediatric Emergency Medicine”.

Participant characteristics, N (%)	N = 153
Role
Physician	139 (91%)
Nurse practitioner	10 (7%)
Physician Assistant	4 (3%)
Specialty
Critical Care	52 (34%)
Hospital/Internal Medicine	45 (29%)
Emergency Medicine	33 (22%)
Infectious Diseases	19 (12%)
Other	4 (3%)
Years since completing school, Median (Interquartile range)	14 (6 to 20)
Treated any patients with suspected sepsis in past 12 months	150 (98%)
Site
University of Pennsylvania Health System	61 (40%)
University of Michigan Health System	48 (31%)
Atrium Health	44 (29%)

Only 4 respondents (3%) suggested starting antibiotics in the control vignette (number 9) without sepsis or organ dysfunction. Among the vignettes with sepsis, the modal response with regard to whether antibiotics should be initiated was “yes” in all eight, and in six of these, clear majorities (range 66% to 86%) of participants recommended antibiotic initiation (Figure 3).

Figure 3:

Categorical responses to the question, “In retrospect, would you have initiated broad­spectrum antibiotics for this patient in this time window?” Vignettes 1 through 8 are based on real patients with hospital­onset sepsis identified by Sepsis­3 criteria. Vignette 9 was used as a control vignette in a patient who did not meet Sepsis­3 criteria and did not experience any organ dysfunction.

This amounted to 1,224 vignettes of patients with sepsis with suggested antibiotic treatment times in the final analytic sample. Among the responses in which treatment was recommended (N=854, 70%), participants suggested starting antibiotics an average of 7.0 hours (95% CI 5.3 to 8.8) before onset as defined by Sepsis-3 (Figure 4). However, in vignettes 3 and 6, Sepsis-3 onset preceded the median suggested antibiotic start time by 17 and 19 hours, respectively. The observed antibiotic delay was not significantly associated with differences between Sepsis-3 onset and suggested antibiotic start times.

Figure 4:

Distribution of recommended antibiotic start times compared to sepsis onset by Sepsis­3 criteria.

Critical care clinicians suggested antibiotic start times 6.4 hours (95% confidence interval [CI] 1.2 to 11.6) and 10.6 hours (95% CI 3.8 to 17.5) earlier than hospital medicine and infectious disease specialists, respectively (Figure 5). The vignette-level standard deviation in start times was 25 hours overall and ranged from 11 to 27 across vignettes with substantial variability among vignettes and clinicians. There was no significant difference in suggested antibiotic times by study site when modeled as a fixed or random effect.

Figure 5:

Differences in suggested antibiotic start times by clinical specialty.

The manual review of the eight clinical vignettes with sepsis revealed at least one consensus data point that likely informed treatment decisions in each vignette. Vignette 1 had the most consensus data points identified (lactate, GCS, and white blood cell count were identified by all five reviewers), while the first peak of recommended treatment times in Vignette 5 had the fewest identified data points (all reviewers identified white blood cell count and one reviewer also identified GCS). One or two reviewers identified the new measurement of some variable (as opposed to the value of the data point itself) as a potential explanatory factor in vignettes 1 through 6. When we excluded 178 responses that appeared potentially motivated by such “meta data” we found that the difference between suggested antibiotic start times and Sepsis-3 onset increased to 21.4 hours (95% CI 18.4 to 24.3).

Based on the observed treatment patterns from the actual cases from which the vignettes were derived as a point of reference, study participants suggested treatment initiation 22.8 hours (95% CI 21.0 to 24.5) before the first broad-spectrum antibiotic was actually given. Only five of eight vignettes (numbers 13, 7, 8) met criteria for the CDC ASE sepsis definition. Among these five, the mean suggested antibiotic start time preceded CDC ASE onset by 12.9 hours (95% CI 11.3 to 14.6).

Using the Cox proportional hazards model with right-censoring for those participants who did not suggest starting antibiotics, we estimated the conditional probability of antibiotic initiation at each hour in aggregate and stratified by medical specialty (Figure 6). Using this approach, only the infectious disease specialty was significantly associated with a reduced hazard of antibiotic initiation (hazard ratio 0.62, 95% CI 0.43 to 0.90) compared to critical care physicians.

Figure 6:

Cumulative risk of recommending antimicrobial initiation over time by clinical specialty.

Discussion

In this study of practicing clinicians reviewing vignettes from real patients with sepsis, we found that clinicians strongly supported antibiotic use among all patients who met Sepsis-3 criteria. But clinicians indicated antibiotic start times that preceded fulfillment of Sepsis-3 criteria by an average of 7 hours. There was substantial heterogeneity in suggested treatment times among vignettes, with some being either much earlier or much later than the Sepsis-3 onset time. Additionally, we identified substantial differences in mean antibiotic start times among participants from different clinical specialties and no differences among hospitals. These results were robust in several sensitivity analyses and the main findings also held when using an alternative definition of sepsis onset.

Our findings have several important implications for how the Sepsis-3 definition might be used as a training label for sepsis EWSs. First, the lack of a consistent relationship between the timing of meeting Sepsis-3 criteria and the recommended timing of antibiotic initiation indicates that the Sepsis-3 definition may not be a useful construct to guide time-sensitive bedside treatment decisions. The difference of 7 hours between suggested antibiotic initiation and Sepsis-3 onset indicates that, on average, using Sepsis-3 as a treatment trigger could potentially lead to patient harm based on prior studies of the effects of such treatment delays of this magnitude^40–42,6,7. However, the wide variation in this time difference across vignettes highlights the difficulties in producing a reliable estimate of the time at which treatment should be initiated. These findings extend previous work in the ICU that showed similar variability and discordance between physician judgement and formal sepsis criteria⁴³.

Second, the high variability in the relationship between the recommended antibiotic start time and Sepsis-3 onset among vignettes suggests that antibiotic decisions depend, at least in part, on factors not captured by the Sepsis-3 definition. The median suggested antibiotic times, estimated as vignette-level fixed effects, ranged from almost 30 hours before to 10 hours after Sepsis-3 onset. Three of the eight sepsis vignettes had suggested start times after Sepsis-3 onset. Further work to identify the full range of factors that guide antibiotic initiation may therefore yield a more clinically appropriate training label that is useful for prompting antibiotic initiation.

Third, the high variability in suggested treatment times highlights a major challenge to researchers in choosing a clinically meaningful training label for sepsis prediction models. This uncertainty reflects ongoing scientific uncertainty and disagreement among medical specialties about sepsis treatment guidelines^8,24,25,44. The observed specialty-level differences could be due to availability and base-rate heuristics⁴⁵, true clinical differences in sepsis presentation across different clinical settings, or specialty-specific educational or cultural differences. Notably, we did not observe differences by study site, suggesting that these specialty-level differences persist across a geographically diverse range of hospitals.

Fourth, the large discrepancy between suggested and observed antibiotic start times highlights an opportunity for improvements in care processes for patients with sepsis. Retrospectively suggested antibiotic start times were, on average, nearly 24 hours prior to the actual delivery of a first broad-spectrum antibiotic. Some of this large lead time may be due to the benefit of hindsight in which annotators had access to future information after the onset of sepsis. However, the vignettes did not include information beyond 48 hours after Sepsis-3 onset so only a short future time horizon was available for review. These findings are consistent with prior work describing significant treatment delays among patients with hospital-onset sepsis and highlight opportunities to improve care delivery^46–50. Thus, clinically meaningful sepsis prediction models remain elusive but with significant potential to improve care.

Fifth, the large difference in suggested antibiotics start times after excluding responses that potentially relied on meta data highlight the difficulty in achieving unbiased retrospective evaluations of appropriateness of antibiotic initiation. Absent a precise biochemical definition of sepsis onset or even infection, the optimal treatment timing can only be defined through clinical judgement, readily available clinical data, or a combination of the two. Future studies should test alternative strategies such as real-time, interactive simulations to identify best practices for retrospective determinations of optimal treatment timing.

The study has important strengths. First, it provides novel evidence that prevailing definitions of sepsis do not align with when clinicians deem it appropriate to initiate antibiotics. Second, the breadth of clinicians included in the study from different specialties, different training levels, and across three diverse health systems augments the generalizability of the results.

This study has several limitations. First, although each vignette contained clinical information that was selected based on pilot testing with practicing clinicians, the available data did not include sufficient information to calculate all elements of the Sepsis-3 definition, nor did it include the full breadth of data available in the EHR such as clinical notes, imaging studies, or medications. These additional data sources may have biased respondent decisions about antibiotic initiation in uncertain ways, including toward earlier or later treatment recommendations. Second, we did not elicit the reasoning behind the respondents’ choices so we are unable to assess how participants interpreted the data or their degree of certainty around their selections, including how they may have considered the risk of antibiotic over-treatment. Third, we presented only eight vignettes with sepsis from a single health system, thus capturing only a fraction of the myriad possible presentations of hospital-onset sepsis. These vignettes were limited to cases where there were observed delays and did not include cases where observed antibiotic timing preceded Sepsis-3 onset, a situation that may have prompted a different pattern of recommendations. Fourth, these findings likely underestimate the discrepancy between suggested antibiotic start times and fulfillment of formal sepsis criteria. Since the definitions of sepsis used here are implemented retrospectively, relying on the identification of the earliest of several criteria, using such criteria prospectively, when all necessary criteria would need to be met, would result in identification up to 48 hours later. Fifth, with a low response rate, sampling bias may have contributed to participation that favored clinicians with more confidence or knowledge of sepsis. Finally, the definition of Sepsis-3 onset used in this study relied on the timing of antibiotic administration and blood culture draws, both of which are already markers of a clinician’s suspicion for infection. While this approach is aligned with current practices for EHR-based implementations of Sepsis-3^25,51,24, it will be impossible to disentangle the identification of sepsis onset from a clinician’s treatment and diagnostic decisions until there is a more objective definition of the sepsis syndrome.

Conclusion

Our study provides novel evidence that prevailing definitions of sepsis do not align with when clinicians deem it appropriate to initiate antibiotics, and this result held across clinicians from different specialties, levels of training, and diverse health systems. Thus, while potentially helpful for clinical trial enrollment and epidemiologic surveillance, Sepsis-3 may not be able to achieve its stated goal of guiding time-sensitive, treatment decisions. Future work on building sepsis EWSs should focus on obtaining a training label that aligns with the treatment decisions about antibiotic initiation that clinicians face at the bedside. The feasibility of developing such a training label for this purpose absent a more robust biochemical definition of sepsis onset is unknown.