Learning a Severity Score for Sepsis: A Novel Approach based on Clinical Comparisons

Abstract

Sepsis is one of the leading causes of death in the United States. Early administration of treatment has been shown to decrease sepsis-related mortality and morbidity. Existing scoring systems such as the Acute Physiology and Chronic Health Evaluation (APACHE II) and Sequential Organ Failure Assessment scores (SOFA) achieve poor sensitivity in distinguishing between the different stages of sepsis. Recently, we proposed the Disease Severity Score Learning (DSSL) framework that automatically derives a severity score from data based on clinical comparisons – pairs of disease states ordered by their severity. In this paper, we test the feasibility of using DSSL to develop a sepsis severity score. We show that the learned score significantly outperforms APACHE-II and SOFA in distinguishing between the different stages of sepsis. Additionally, the learned score is sensitive to changes in severity leading up to septic shock and post treatment administration.

Introduction

Sepsis — a whole-body inflammatory response to infection — is one of the leading causes of death in the inpatient setting and is associated with significantly higher costs of care¹. The risk of sepsis-related adverse outcomes can be reduced by early treatment². Thus, changes in a patient’s health status is regularly assessed by the caregiver (clinicians and nurses) to plan timely interventions. This requires the caregiver to interpret a diverse array of markers (e.g., heart rate, respiratory rate, blood counts, and serum measurements) that measure the underlying physiologic and metabolic state. Such clinical assessments are time-consuming, require extensive experience, and more importantly, are prone to missing signs of decline that may be only subtly visible in the observed markers.

In this paper, we address the problem of quantifying (scoring) the latent sepsis severity of a patient at a given time. That is, we derive a mapping from the high-dimensional observed marker data to a numeric score that tracks changes in sepsis severity over time — as health worsens, the score increases, and as the individual’s health improves, the score declines. More generally, a means for accurate estimation and tracking of an individual’s health status can enable clinicians to detect critical decline such as decompensations, and acute adverse events in a timely manner. Additional potential benefits include assessment for whether an individual is being responsive to therapy, stratification of patients for resource management, and risk adjustment for clinical research³.

Background

Qualitatively, the concept of disease severity score has been described as the total effect of disease on the body; the irreversible effect is referred to as damage, while the reversible component is referred to as activity⁴. The precise interpretation of concepts of damage and activity are typically based on the application at hand. Desirable properties of a severity scale include: 1) face and content validity i.e., the variables included are important and clinically credible, and 2) construct validity i.e., the scoring system parallels an independently ascertained severity measurement⁴.

Historically, severity scores have been defined and derived in a number of different ways⁵. One approach is to have clinical experts fully specify the score. Namely, using the existing clinical literature, a panel of experts identifies factors that are most indicative of severity of the target disease. These factors are weighted by their relative contribution to the severity and summed together to yield the total resulting score. For example, the Acute Physiology And Chronic Health conditions score⁶ (APACHE II), which assesses the overall health state in an inpatient setting, uses factors that are most predictive of mortality. For instance, a heart rate between 110 and 139 beats per minute adds 2 points to the final score while a heart rate higher than 180 beats per minute adds 4 points. Similarly, mean arterial blood pressure between 70 and 109 mm Hg adds no points while a value between 50 and 69 mm Hg adds 2 points. A number of additional widely used scoring systems have been designed in this way, including the Multiple Organ Dysfunction Score⁷ (MODS), the Sequential Organ Failure Assessment⁸ (SOFA), and the Medsger’s scoring system⁴.

A second approach commonly taken is to characterize severity as a predictive score that is associated with the risk of experiencing a target downstream event. That is, high severity states are more likely to be associated with adverse events and higher mortality rates. To train such a score, supervised learning is used with the presence or absence of the downstream adverse event (e.g., septic shock or motality) as the labels. For example, the pneumonia severity index (PSI) uses this approach to combine 19 factors including age, vitals and laboratory test results, to calculate severity as the probability of morbidity and mortality among patients with community acquired pneumonia⁹. The relative weight of each factor in the resulting score was derived by training a logistic regression function that predicts risk of death in the following 30-hour window. Others have similarly used downstream adverse events such as development of c.diff infection¹⁰, septic shock¹¹, morbidity¹², and mortality¹³ as surrogates for training severity scores. Yet other approaches use probabilistic state estimation techniques^14–16.

However, the above-mentioned approaches for the derivation of disease severity scores have their limitations. The expert-based approach captures known clinical expertise well, but it does not extend to populations where current clinical knowledge is incomplete. The approach of regressing against the risk of downstream adverse event yields scores that suffer from bias due to treatment-related censoring^17,18. Specifically, scores learned from data assuming one standard of care, when used under a different standard of care may lead to erroneous conclusions about the individuals disease severity state. Say, we have data from a unit where children with temperature of 102°F are frequently prescribed treatment for the flu. This intervention subsequently cures the patients and they do not experience any associated adverse events (e.g., sepsis and septic shock) or death. However, it is also known that a child with temperature of 102°F, if left untreated, is likely to die. Thus, a temperature of 102°F is clinically considered to be a high severity state which in this unit is frequently treated in a timely manner. Since the learning algorithm uses the presence of a downstream adverse event as a surrogate marker for severity, rare occurrence of adverse events following 102°F causes the learning algorithm to score the temperature of 102°F as benign. This is problematic for two reasons. First, it is not consistent with how severity is interpreted clinically. Moreover, if such a score were to be used to guide interventions, it might erroneously cause the clinician to undertreat children with temperature of 102°F, thus likely worsening outcomes. See Dyagilev and Saria¹⁸ for a numerical example.

In recent work¹⁸, we proposed an alternative Disease Severity Score Learning (DSSL) framework for learning severity scores. The DSSL framework leverages this key observation that, while requesting experts to quantify disease severity at a given time is challenging, acquiring clinical comparisons – clinical assessments that order the disease severity at two different times – is often easy. From these clinical comparisons, DSSL learns a function that maps the patients observed feature vectors to a scalar severity score. With some abuse of terminology, we refer to this mapping function as the DSS function. We showed empirically¹⁸ that the generalization performance of DSSL is less sensitive to variations in treatment administration patterns compared to the supervised learning approach described above.

In this paper, we test the feasibility of using DSSL to develop a sepsis severity score. We use the MIMIC-II dataset¹⁹ containing electronic health record data from patients admitted to the Beth Israel Deaconess Intensive Care Units between 2001 and 2008. Clinical comparisons required for training the score are generated automatically using the Surviving Sepsis Campaign (SSC) guidelines²⁰. We show that the learned score significantly outperforms APACHE-II and SOFA in distinguishing between the different stages of sepsis. Additionally, the learned score is sensitive to changes in severity leading up to septic shock and post treatment administration. We also show that only a small number of clinical comparisons are needed to obtain a high quality score. Thus, by not having to rely on needing a large number of clinical comparisons, the use of DSSL in domains where guidelines are unavailable for generating clinical comparisons in an automated manner (as we do in this paper) becomes feasible.

Methods

In this section, we describe the DSSL framework and the L-DSS method for learning linear DSS functions.

General DSSL Framework

We consider longitudinal data routinely collected in a hospital setting. These include covariates obtained at the time of admission such as age, gender, and clinical history; time-varying measurements such as heart rate and respiratory rate; and text notes summarizing the patients evolving health status. These data are processed and transformed into tuples $<x_i^p,t_i^p>$ where $x_i^p\in {ℛ}^d$ is a d-dimensional feature vector associated with patient p ∈ P at time $t_i^p$ for i ∈ {1, …, T^p} and T^p is the total number of tuples for patient p. A feature vector $x_i^p$ contains raw measurements (e.g., last measured heart rate or last measured white blood cell count) and features derived from one or more measurements (e.g., the mean and variance of the measured heart rate over the last six hours or the total urine output in the last six hours per kilogram of weight). Let D denote the set of tuples across all patients in the study. The problem of learning a DSS function is defined by the sets O and S of pairs of tuples from the set D of all tuples, and by the set G of permissible DSS functions. The set O contains pairs of tuples $(<x_i^p,t_i^p>,<x_j^q,t_j^q>)$ that are ordered by severity based on clinical assessments, i.e., $x_i^p$ corresponds to a more severe state than $x_j^q$. We refer to each of these paired tuples as a clinical comparison and the set O as the set of all available clinical comparisons. These clinical comparisons can be obtained by presenting clinicians with data $x_i^p$ for patient p ∈ P at time $t_i^p$ and data $x_j^q$ for patient q ∈ P at time $t_j^q$. For each such pair of feature vectors, the clinical expert identifies which of these correspond to a more severe health state; the expert can choose not to provide a comparison for a pair where the severity ordering is ambiguous. These pairs can also be generated in an automated fashion by leveraging existing clinical guidelines. In the Experimental Methods section, we describe how we use an existing guideline in the task of learning sepsis severity score. The set S contains pairs of tuples $(<x_i^p,t_i^p>,<x_{i+1}^p,t_{i+1}^p>)$ that correspond to feature vectors that are taken from the same patient p at consecutive time steps $t_i^p$ and $t_{i+1}^p$. These pairs are used to impose smoothness constrains on the learned severity scores. We thus refer to the pairs in S as the smoothness pairs.

Finally, the set G contains a parameterized family of candidate DSS functions g that map feature vectors x to a scalar severity score. In this paper we focus on linear DSS function alone, however, the DSSL framework extends to non-linear DSS functions as well.

In the DSSL framework, our goal is to learn parameters of the function g ∈ G that quantifies the severity of the disease state represented by a feature vector x. This is done using an empirical risk minimization approach, i.e., an objective function is constructed that maps functions g ∈ G to their empirical risk. This function objective contains two key terms. The first term penalizes g for pairs of tuples $(<x_i^p,t_i^p>,<x_j^q,t_j^q>)\in O$ for which the severity ordering induced by g on vectors $x_i^p$ and $x_j^q$ is inconsistent with the ground truth clinical assessment. The second term imposes penalty of high rates of temporal changes of the DSS value, thus encouraging selection of a temporally smooth DSS function. The smoothness requirement allows to learn severity scores that mimic the natural inertia exhibited by biological systems. In what follows we present the formal definition of this objective function for learning a linear DSS.

Learning a Linear DSS

Below, we describe the L-DSS algorithm for learning linear DSS functions. This algorithm builds on the widely-used soft max-margin training²¹ technique that seeks to maximize the distance between the pairs that are at different severity levels. We briefly review the key concepts of soft max-margin ranking before we describe these are adapted to the task of learning a linear DSS function.

Soft Max-Margin Ranking

Consider the toy example shown in Figure 1. In this example, we assume that all feature vectors are taken on same patient and the time at which these feature vectors are taken is irrelevant. We simplify the notation accordingly. Let D contain the three feature vectors $\left\{x_1,x_2,x_3\right\}$ where $x_i\in {ℛ}^2$, and O contain the pairs (x₂, x₁) and (x₃, x₂), i.e., feature vectors x₂ and x₃ have higher disease severity x₁ and x₂ respectively.

Figure 1:

We show the projections of x₁, x₂ and x₃ on vectors ${\overline{w}}_1$ and ${\overline{w}}_2$ representing two candidate ranking functions; vectors drawn in red, green and blue identify projections of x₁, x₂ and x₃ respectively. Ranking is induced by the differences in the projections—for example, ${\overline{w}}_2$ induces the ordering $g_w{}_{{}_2}\left(x_1\right)>g_w{}_{{}_2}\left(x_2\right)$ because $g_w{}_{{}_2}(x_1)-g_w{}_{{}_2}(x_2)>0$.

Max-margin ranking seeks to find a vector w such that the margin between pairs of different severity levels is maximized. In our example, we show parameter vectors w₁, w₂ and w₃ for three candidate ranking functions in Figure 1. For each feature vector x, the assigned (severity) score for a given ranking function parameter w_i is computed as the projection, g_wi(x), of x on w_i. The induced ranking between two vectors x₁ and x₂ is computed based on the margin which is defined as the difference in their projections. In the example shown, the rankings induced by both g_w1 and $g_w{}_{{}_3}$ correctly order all pairs in O, i.e., $g_w{}_{{}_1}(x_3)>g_w{}_{{}_1}(x_2)>g_w{}_{{}_1}(x_1)$ and $g_w{}_{{}_3}(x_3)>g_w{}_{{}_3}(x_2)>g_w{}_{{}_3}(x_1)$, while the rankings induced by w₂ do not. Furthermore, w₃ also induces an ordering with a larger margin between the pairs in O. Margin-maximization leads to an ordering that is more robust with respect to noise in x.

More formally, for each pair of feature vectors (x_i, x_j) ∈ O, we define the margin of their separation by the function $g_w(\cdot )$ as ${¼}_{i,j}^w=g_w(x_i)-g_w(x_j)$. The maximum-margin approach suggests that we can improve generalization and robustness of the learned separator by selecting w that maximizes the number of tuples that are ordered correctly (i.e., ${\mu }_{i,j}^w>0$) while simultaneously maximizing the minimal normalized margin ${\mu }_{i,j}^w/\Vert w\Vert$. Using the standard soft max-margin framework, the SVMRank algorithm²¹ approximates the above-mentioned problem as the following convex optimization program:

$$\begin{array}{l}\underset{w,{\varsigma }_{\text{ord}}^{i,j}}{\min }\left[\frac{1}{2}{\Vert w\Vert }^2+\frac{{\mathrm{\lambda }}_{\mathrm{O}}}{\left|O\right|}{\displaystyle \sum _{(x_i,x_j)\in O}{\varsigma }_{\text{ord}}^{i,j}}\right],\text{subject to the}\mathbf{\text{ordering contraints}}:\\ \forall (x_i,x_j)\in O:g_w(x_i)-g_w(x_j)=w^T(x_i-x_j)\ge 1-{\varsigma }_{\text{ord}}^{i,j}\text{and}{\varsigma }_{\text{ord}}^{i,j}\ge 0\end{array}$$ (1)

Joachims et al.²¹ solved this optimization program in the dual formulation²¹. Chapelle et al.^22,23 proposed to transform the problem in (1) into a twice differentiable unconstrained convex problem and solve it in its primal form. To this end, we observe that for every value of w, the optimal values of ${\varsigma }_{\text{ord}}^{i,j}$ are given by ${\varsigma }_{\text{ord}}^{i,j}=\max \left\{0,1-w^T(x_i-x_j)\right\}$. Substituting to (1), we obtain the following unconstrained convex optimization problem:

$$\underset{w}{\min }\left[\frac{1}{2}{\Vert w\Vert }^2+\frac{{\mathrm{\lambda }}_{\mathrm{O}}}{\left|O\right|}{\displaystyle \sum _{(x_i,x_j)\in O}\max \left\{0,1-w^T(x_i-x_j)\right\}}\right].$$

The terms of the form max{0, a}, also called the hinge loss, are not differentiable at a = 0. We approximate these terms with the Huber loss L_h for 0 < h < 1 given by

$$L_h(a)=\{\begin{array}{ll}0\hfill & ,\text{if}a<-h\hfill \\ {(a+h)}^2/(4h)\hfill & ,\text{if}\left|a\right|\le h\hfill \\ a\hfill & ,\text{if}a>h\hfill \end{array}$$

It can be seen that L_h(a) is identical to hinge loss for a < −h and a > h. This approximation yields the following unconstrained, convex, twice-differentiable optimization problem:

$$\underset{w}{\min }\left[\frac{1}{2}{\Vert w\Vert }^2+\frac{{\mathrm{\lambda }}_{\mathrm{O}}}{\left|O\right|}{\displaystyle \sum _{(x_i,x_j)\in O}{L}_h(1-w^T(x_i-x_j))}\right].$$ (2)

The L-DSS Objective and Optimization Algorithm

We now describe the L-DSS algorithm for learning linear DSS functions. We return to our original setting, where we are given sets O and S which contain feature vectors belong to more than one patients at varying times.

The L-DSS objective function is obtained by augmenting the Eq. (2) with the following term:

$${{\displaystyle \sum _{(<x_i^p,t_i^p>,<x_{i+1}^p,t_{i+1}^p>)\in S}\left[\frac{w^t(x_{i+1}^p-x_i^p)}{t_{i+1}^p-t_i^p}\right]}}^2.$$

This term penalizes DSS functions that exhibit large changes in the severity score over short durations, hence encouraging selection of temporally smooth DSS functions. Substitution of this term yields the following L-DSS objection function:

$${}_{\text{Objective}}^{\text{L-DSS}}:\begin{array}{|c|}\hline \begin{array}{r}{}_w^{\text{min}}\frac{1}{2}{\Vert w\Vert }^2+\frac{{\mathrm{\lambda }}_{\mathrm{O}}}{O}{\displaystyle \sum _{(<x_i^p,t_i^p>,<x_j^p,t_j^p>)\in O}{L}_h(1-w^T(x_i^p-x_j^p))}\\ +\frac{{\mathrm{\lambda }}_{\mathrm{s}}}{\left|S\right|}{{\displaystyle \sum _{(<x_i^p,t_i^p>,<x_{i+1}^p,t_{i+1}^p>)\in S}\left[\frac{w^T(x_{i+1}^p-x_i^p)}{t_{i+1}^p-t_i^p}\right]}}^2\end{array}\\ \hline\end{array}$$ (3)

This optimization program is solved using the Newton-Raphson algorithm.

Experimental Methods

In this section we derive a sepsis severity score by applying the L-DSS method to the large scale MIMIC-II dataset¹⁹. With some abuse of terminology, we refer to the resulting sepsis severity score as the L-DSS-Sepsis score. We assess the quality of the obtained L-DSS-Sepsis score by testing its sensitivity to severity changes of different granularity. In our first experiment, we test whether the L-DSS-Sepsis score is sensitive to significant changes severity. Specifically, we test whether the L-DSS-Sepsis score can accurately distinguish and order correctly by their severity different stages of sepsis. We show that the L-DSS-Sepsis score outperforms on this task two clinical scores that are widely used in the ICU setting. In two additional experiments, we analyze the sensitivity of the L-DSS-Sepsis score to a finer-grained changes in sepsis severity. In particular, in the second experiment, we show that the L-DSS-Sepsis score is sensitive to changes in severity state leading up to septic shock. This suggests that the L-DSS-Sepsis score can potentially be used for early detection of this adverse event. In the third experiment, we show that the L-DSS-Sepsis score is sensitive to post-therapy changes in severity and hence could potentially be used to assess patient’s treatment response.

The rest of this section proceeds as follows. We begin with the brief overview of MIMIC-II, the dataset used for learning the sepsis severity score. We then describe the clinical guideline for coarse grading of sepsis severity and show how this coarse grading can be used for automatic generation of clinical comparison pairs. Next, we describe how to choose the values of free parameters for the L-DSS algorithm. We conclude with the detailed description of the three experiments that are used to assess the quality of the learned L-DSS-Sepsis score.

Dataset

We use MIMIC-II, a publicly available dataset containing electronic health record data from patients admitted to the ICUs at the Beth Israel Deaconess Medical Center from 2001 to 2008¹⁹. We only include adults (> 15 years old) in our study (N = 16, 234). We compute 42 different features that are derived from vital sign measurements, clinical history variables, and laboratory test results.

Sepsis Severity Grading and Automatic Generation of the Clinical Comparison Pairs

One of the inputs required by the L-DSS algorithm is the set O of clinical comparisons. For sepsis, these pairs can be created automatically by leveraging the coarse severity grading of sepsis proposed in the Surviving Sepsis Campaign (SSC)²⁰. SSC classifies sepsis development into the following four stages with decreasing levels of severity: septic shock, severe sepsis, SIRS, and ”none” (i.e., none of the above). For each of these stages, the guideline defines criteria as combination of 1) thresholds for individual measurements that capture deviation of this measurement from its normal range, and 2) presence of specific diagnosis codes or diagnoses noted in their clinical notes. However, not all measurements needed for implementing the SSC criteria may be available at the given time. We thus relax the definition of the SSC guidelines so it uses measurements made within a short time window prior to the given time. For instance, for fast changing signals (e.g., blood pressure) we use a two hour window, and for slower changing signals (e.g., creatinine), we use an eight hour window. If no measurements were taken in the designated time window, we consider sepsis stage to be unknown. We implement these criteria and automatically identify tuples in the dataset where these SSC definitions are met.

We leverage the SSC grading of sepsis severity to automatically generate clinical comparisons in the following manner. First, we randomly assign the 16, 234 patients in the MIMIC dataset to training (60%) and testing sets (40%). Within the training set, we assign two thirds of the patients to the development set and the remaining third to the validation set. For each of the development, validation and testing sets of patients we generate a separate set O of clinical comparison pairs. This is done by randomly sampling an equal number of pairs of feature vectors ( $x_i^p$, $x_j^q$) for each of six combinations of different sepsis stages according to SSC, i.e., shock-severe, shock-SIRS, shock-none, severe-SIRS, severe-none, and SIRS-none. These pairs include feature vectors that are taken from the same patient (i.e., p = q) or from different patients (i.e., p ≠ q). Overall, we extracted 30, 000 clinical comparison pairs for the validation set, and 60, 000 clinical comparison pairs for the testing set. For the development set, we extracted several sets O of different sizes, ranging from 120 pairs to 60, 000 pairs. We construct the set S of smoothness pairs to contain 80, 000 randomly sampled pairs ( $x_i^p$, $x_{i+1}^p$) of consecutive feature vectors.

When generating clinical comparisons by automating clinical rules, a natural question that one might ask is whether the learned scores simply recover these clinical rules and thus yield no generalization beyond the SSC grading. This is not the case for two reasons. First, the guideline criteria rely on information captured in notes and diagnosis codes that are not available to the L-DSS method. Second, the temporal smoothness term constrains the learned scores to be smooth and to generalize beyond grading for the coarse severity stages. This issue is covered in greater detail in Section 3.1 of Dyagilev and Saria¹⁸.

Selection of Free Parameters

In addition to sets O and S, the required input of the L-DSS algorithm includes the values of free parameters λ_O and λ_S. To specify these values, we follow the two-step procedure proposed in Dyagilev and Saria¹⁸. First, with λ_S set to 0, we select the value of λ_O that maximizes the accuracy of ordering of clinical comparison pairs for the validation set of patients. That is, we count the fraction of clinical comparison pairs in the set O that are ordered by L-DSS in concordance with the ordering prescribed by the SSC. We refer to this quantity as the severity ordering accuracy or SOA. For a given λ_O, we choose the value of λ_S that maximizes the SOA of the resulting score. When there are several such values, we pick to the largest λ_S among these maximizers. Namely, when there are multiple scores that achieve the maximal SOA, we give preference to the smoothest one.

Missing data imputation

Following Hug²⁴, we used a modified Last Observation Carried Forward (LOCF) method to impute missing data in given feature vectors. Specifically, within a given time window w, the last value of the observed signal was used. The window length w for each signal was set to be the typical sampling frequency of this signal. Specifically, the value of w was determined by combining clinical knowledge regarding signal’s typical rate of change and manual inspection of the distribution of duration between consecutive measurements of this signal. If no measurements were taken within the specified window, the missing value was replaced with the population mean.

Experiment 1: Distinguishing between the Severity Stages of Sepsis

In our first experiment, we evaluate whether L-DSS-Sepsis can distinguish and correctly order the different stages of sepsis severity. We consider L-DSS-Sepsis scores learned from a different number of training clinical comparison pairs. For each size of the training set, we report the SOA on the testing set of the obtained L-DSS-Sepsis score. As a baseline for our assessment, we use the severity ordering accuracy of existing clinical scores that are widely used in ICU³. There exists several general purpose severity scores that have been validated to assess illness severity and risk of mortality among septic patients^5,25–27. In this paper we focus on two of them. The first one is the Acute Physiology and Chronic Health Evaluation⁶ (APACHE II) score, which is a widely used scoring system for assessing general (not necessarily sepsis-related) disease severity in hospitalized individuals. The second one is based on the Sequential Organ Failure Assessment (SOFA) score⁸, which was originally designed to assess per-organ sepsis-related damage severity. Specifically, we use the total SOFA score which is is the sum of SOFA scores of all organ systems.

Experiment 2: Is L-DSS-Sepsis sensitive to changes in severity leading up to adverse events?

In the second experiment, we assess whether L-DSS-Sepsis is sensitive enough to capture changes in severity that can occur over the time period leading up to an adverse event. To this end, we examine the behavior of L-DSS-Sepsis in the 18 hour period prior to septic shock. We consider all patients with septic shock in our data with at least 18 hours of data prior to septic shock onset (N = 684). On these patients, we define three time intervals of interest: 1) 6 hours prior to the onset of septic shock; 2) 6 − 12 hours prior to the onset of the septic shock; 3) 12 – 18 hours prior to the onset of septic shock. We denote the average values of the learned scores in these intervals by ${\overline{s}}_{0–6}$, ${\overline{s}}_{6–12}$ and ${\overline{s}}_{12–18}$, respectively. We calculate values of ${\mathrm{\Delta }}_1={\overline{s}}_{0-6}-{\overline{s}}_{6-12}$ and ${\mathrm{\Delta }}_2=({\overline{s}}_{0-6}-{\overline{s}}_{6-12})-({\overline{s}}_{6-12}-{\overline{s}}_{12-18})$ for each patient. Using the standard one-tailed t-test, we assess the p-value p_trend-up for whether the obtained ∆₁ can be observed by chance under the null hypothesis that ∆₁ are drawn from a zero mean distribution. Similarly, we assess the p-value p_{rate acceleration} for whether the obtained ∆₂ can be observed by chance under the null hypothesis that ∆₂ are drawn from a zero mean distribution.

Experiment 3: Is L-DSS-Sepsis sensitive to post-therapy changes in severity?

We now evaluate whether L-DSS-Sepsis is sensitive to changes in severity state due to administration of fluid bolus – a treatment used for septic shock²⁰. To wards this, we use the self-controlled case series method. We compare trends exhibited by L-DSS-Sepsis values over the five hour intervals prior to and post the administration of fluid bolus. We refer to the trends over these intervals as ∆_prior and ∆_post. The value of ∆_prior is computed as the difference between the value of the L-DSS-Sepsis at the time of treatment administration and the mean value of L-DSS-Sepsis over the five hour interval prior to treatment administration. Similarly, the value of ∆_post is calculated as the difference between the mean value of L-DSS-Sepsis over the five hour interval after treatment administration and the value of L-DSS-Sepsis at the moment of treatment administration. If the patient is responsive to fluid therapy, then ∆_treat = ∆_post ∆_prior < 0, that is, if the L-DSS-Sepsis was trending up prior to treatment administration, we expect this trend to be attenuated or even reversed by the treatment.

We identify cases of fluid administration events related to sepsis using the following criteria: 1) the patient is experiencing SIRS, severe sepsis or septic shock at the time of treatment administration, and 2) the patient is hypotensive (has systolic blood pressure below 100 mm Hg), a commonly used criteria for prescribing fluids in sepsis. To avoid confounding due to multiple administration of fluids, we restrict our attention to treatment administrations that were not preceded or followed by another fluid bolus administration within a five hour window. This yielded a total of 81 fluid bolus administration events. Employing the one-tailed t-test, we assess the p-value p_{treatment response} for whether the observed values of ∆_treat = ∆_post −∆_prior can be observed by chance under the null hypothesis that ∆_treat are drawn from a zero mean distribution.

Results and Discussion

In our first experiment we assess the ability of the L-DSS-Sepsis score to distinguish and correctly order the different stages of sepsis severity. Specifically, we train the L-DSS-Sepsis score on clinical comparisons sets O of different sizes and measure the corresponding values of severity ordering accuracy (SOA). We present the results of this experiment in Figure 2(a). As expected, the SOA is an increasing function of the number of training clinical comparisons. We note that the L-DSS algorithm yields highly accurate scores even with relatively small amount of training examples. In particular, the maximal SOA equals 0.855 and is obtained by L-DSS-Sepsis scores trained on 30, 000 clinical comparisons or more. The SOA of 0.847, which is 99% of the maximal SOA, is obtained for 1200 clinical comparisons or more. The routinely used clinical scores — APACHE-II and Total SOFA — yield SOA of 0.654 and 0.601, respectively. They are thus outperformed by all L-DSS-Sepsis scores that are trained on 120 clinical comparisons or more. This performance achieved by the L-DSS-Sepsis is significant from a clinical standpoint as it orders severity states more accurately than the two widely used clinical scores. Moreover, it suggests that the L-DSS algorithm yields high quality severity scores even when trained on a small number of clinical comparisons. Hence, it can be applied to diseases where clinical comparisons need to be manually created by experts, thus their number should be minimal.

Figure 2:

(a) Severity ordering accuracy (SOA) of the L-DSS-Sepsis scores on the testing set for number of training clinical comparisons. Vertical line marks 1200 training clinical comparisons. (b) Distribution of the L-DSS-Sepsis values for different stages of sepsis. (c) Temporal trajectory of a severity score over the time period leading up to septic shock. The vertical line marks the onset of septic shock. (d) Temporal trajectory of a severity score before and after administration of fluid bolus. The solid vertical line marks the administration of fluid bolus.

Hereafter, we use the L-DSS-Sepsis score trained on 1200 clinical comparisons. In Figure 2(b) we plot the probability density of the L-DSS values at different sepsis severity stages. We observe that, on average, the value of L-DSS score during septic shock is higher than in other less severe stages of sepsis. Since the L-DSS score is temporally smooth, this suggest that the L-DSS trajectory should trend up over the time period leading up to septic shock, i.e, should be sensitive to changes in severity leading up to adverse events. In our second experiment, we verify this hypothesis. Overall, over 67% (95% confidence interval: 63% – 70%) out of 684 observed values of ∆₁ and 57% (95% confidence interval: 53%–60%) out of 684 observed values of ∆₂ were positive. The obtained p-values p_trend-up < 10⁻²¹ and p_{rate acceleration} < 2.25 × 10⁻² rule out the null hypothesis in favor of the stated hypothesis, that is, the learned scores trend up significantly prior to a septic shock and this trend accelerates over time. As an illustration, in Figure 2(c), we show the L-DSS trajectory on an example patient leading up to the onset of shock. It can be seen that the severity score of this patient exhibits a clear upward trend prior to the septic shock. Early identification of this trend can potentially alert the caregiver about a need for a medical intervention and prevent the impending adverse event.

In our last experiment, we addressed the question of whether the L-DSS score is sensitive to post-therapy changes in severity. Overall, the change of trend ∆_treat is negative in at least 77% (95% confidence interval: 68% – 86%) out of 81 recorded values of ∆_treat. The obtained p-value p_{treatment response} < 5 × 10⁻⁸ rules out the null hypothesis in favor of the stated hypothesis, that is, that is L-DSS shows significant response to therapy. As an illustration, in Figure 2(d), we show the L-DSS trajectory on an example patient 10 hours before and after administration of fluid bolus. It can be seen that the severity score of this patient trends up prior to treatment and trends down post-treatment.

Conclusion

In this paper we evaluated the feasibility of automatically learning a score (L-DSS-Sepsis) that tracks the severity of sepsis over time. We validated the learned sepsis severity score using electronic health record data obtained from patients admitted to four different ICUs at an academic medical center over a period of eight years. Compared to existing illness severity scores of APACHE-II and SOFA, the L-DSS-Sepsis score was significantly more accurate in distinguishing between the different stages of sepsis. The L-DSS-Sepsis score also showed face validity: its trajectories were temporally smooth and tended to trend upwards in individuals that progressed to septic shock. Furthermore, the L-DSS-Sepsis score also behaved as expected after fluid administration–as the patient’s health improved, the value of the L-DSS-Sepsis score decreased. These experiments suggest that an automated severity tracking score such as L-DSS-Sepsis may enable early interventions and monitor a patient’s responsiveness to therapy.

Our study has several limitations. While the L-DSS-Sepsis score showed desirable behavior in tracking changes in illness severity, additional studies are needed to validate whether it identifies at-risk patients before the caregiver has identified them as such. Similarly, whether the L-DSS-Sepsis score can be used to develop a tool for measuring fluid responsive remains to be studied. Finally, the key advantage of the DSSL framework is that it is less prone to practice pattern variations. Data from more than hospital using on a similar population of patients is needed to further test this hypothesis.

Overall, the discussed results are promising. The DSSL framework is general purpose and can be applied to other disease domains and populations. Furthermore, we are encouraged by the fact that the DSSL framework yields high quality sepsis severity scores even when given as few as 120 clinical comparisons. This suggests that it can be successfully applied to diseases for which coarse severity grading guidelines are not available and thus clinical comparisons need to be obtained from asking experts to annotate.