Validation of Test Performance and Clinical Time Zero for an Electronic Health Record Embedded Severe Sepsis Alert

Summary

Bachground

Increasing use of EHRs has generated interest in the potential of computerized clinical decision support to improve treatment of sepsis. Electronic sepsis alerts have had mixed results due to poor test characteristics, the inability to detect sepsis in a timely fashion and the use of outside software limiting widespread adoption. We describe the development, evaluation and validation of an accurate and timely severe sepsis alert with the potential to impact sepsis management.

Objective

To develop, evaluate, and validate an accurate and timely severe sepsis alert embedded in a commercial EHR.

Methods

The sepsis alert was developed by identifying the most common severe sepsis criteria among a cohort of patients with ICD 9 codes indicating a diagnosis of sepsis. This alert requires criteria in three categories: indicators of a systemic inflammatory response, evidence of suspected infection from physician orders, and markers of organ dysfunction. Chart review was used to evaluate test performance and the ability to detect clinical time zero, the point in time when a patient develops severe sepsis.

Results

Two physicians reviewed 100 positive cases and 75 negative cases. Based on this review, sensitivity was 74.5%, specificity was 86.0%, the positive predictive value was 50.3%, and the negative predictive value was 94.7%. The most common source of end-organ dysfunction was MAP less than 70 mm/Hg (59%). The alert was triggered at clinical time zero in 41% of cases and within three hours in 53.6% of cases. 96% of alerts triggered before a manual nurse screen.

Conclusion

We are the first to report the time between a sepsis alert and physician chart-review clinical time zero. Incorporating physician orders in the alert criteria improves specificity while maintaining sensitivity, which is important to reduce alert fatigue. By leveraging standard EHR functionality, this alert could be implemented by other healthcare systems.

1. Background

Sepsis is the most common cause of death in hospitalized patients and costs the U.S. health care system $20 billion annually [1–2]. Similar to other acute medical conditions such as acute myocardial infarction or stroke, early diagnosis and treatment of sepsis is critical to positive treatment outcomes. For every hour delay in initiating antibiotics after the development of severe sepsis, expected patient mortality increases significantly [3–5]. Though timely administration of antibiotics and fluids is associated with improved patient outcomes, barriers to early recognition and treatment of sepsis continue to present significant challenges. Early signs of the progression from simple sepsis to sepsis with end-organ dysfunction (severe sepsis) may be subtle and non-specific. For that reason, treatment often fails to adhere to guidelines despite generally consistent evidence supporting standardized treatment [6].

The recent widespread adoption of electronic health record systems (EHRs), combined with increasingly sophisticated clinical decision support, has generated interest in the potential of computerized alerts to improve early recognition and treatment of sepsis. Several published studies have described the results of efforts to develop electronic best practice alerts (BPAs) for the detection of sepsis in pediatric populations, emergency departments, intensive care units, and medical wards [7–15].

Although alerts offer the advantage of continuous, automated screening, published alerts for severe sepsis typically demonstrate either poor sensitivity or poor specificity [7,11,13,16]. Furthermore, sepsis alerts have shown only a modest impact on outcomes. Alerts have demonstrated a modest impact on target process metrics, such as antibiotic administration, without any clear impact on mortality or length of stay [17]. The test performance of these alerts may provide one reason for their limited success. Using physician orders in the emergency department may improve specificity of an EHR alert system, but physician orders have not been utilized to represent infection in published sepsis alerts [18–19].

Perhaps more challenging than accurately diagnosing sepsis is recognizing the onset of sepsis in a timely manner Despite the wealth of data captured from usual clinical care within the EHR, most electronic sepsis alerts have had difficulty in the timeliness of diagnosis. Evidence suggests that early detection can improve outcomes [20]. Existing guidelines that define time zero as the time of triage in the Emergency Department does not apply well to the inpatient setting, In hospitalized patients, the period between onset of sepsis and identification by the physician may provide the best opportunity to improve treatment outcomes [21]. Yet, despite the importance of timely detection, no published studies have evaluated an alert’s ability to detect sepsis or severe sepsis in a timely manner, as close as possible to the onset of sepsis. Here, we describe the development and validation of a computerized alert for severe sepsis in hospital inpatients that identifies clinical time zero with reasonable accuracy and is embedded in a common commercial EHR (Epic Systems, Verona, WI) [22]. Although the alert may be applicable to a wide range of patient care settings, here it was designed and validated for hospitalized patients. Identification and treatment of sepsis in hospitalized patients may be a particular challenge. Patients may not receive the same level of intensive monitoring as patients presenting to emergency departments, leading to underdiagnosis [23]. The objective was to develop an alert that combined robust test characteristics with timely detection of the onset of severe sepsis.

2. Methods

2.1 Development and Overview of Validation Steps

2.1.1 Development of the Severe Sepsis Alert

The computerized sepsis alert was developed from Surviving Sepsis campaign guidelines by a multidisciplinary team of physicians, nurses, quality improvement specialists, and informaticists at Stanford Hospital [24]. The alert was based on criteria from the Surviving Sepsis Campaign’s 2012 guidelines, which define sepsis as “the presence (probable or documented) of infection together with systemic manifestations of infection” and severe sepsis is defined as “sepsis plus sepsis-induced organ dysfunction or tissue hypoperfusion [24].” The alert is triggered when a combination of criteria in three categories are met: systemic manifestations of sepsis (M criteria), suspicion of infection (S criteria), and evidence of end-organ damage (OD criteria). Following campaign guidelines, systemic manifestations (M criteria) include vital sign changes, hemodynamic changes, and abnormal laboratory values (see Table 1 in the 2012 guidelines). Markers of end-organ dysfunction were also derived from campaign guidelines (see Table 2). Modifications to criteria were made as necessary to accommodate Epic capabilities, such as omitted organ dysfunction criteria like “ileus” that could not be captured in structured data. To improve specificity, the sepsis alert relies on physician orders suggesting concern for infection such as an order for blood cultures or serum lactate. A serum lactate is a component of the three-hour sepsis bundle. Since physicians often order a lactate as part of the evaluation of sepsis, a lactate order suggests suspicion of infection. The alert described here is triggered when there are at least three M criteria, at least one S criterion, and at least 1 OD criterion within any 24-hour period. Some criteria overlap between categories and may thus serve as a criterion in more than one category at the same time (e.g. hypotension can fulfill an M criterion and the OD criterion). Following guidelines, a lactate > 1 mmol/L was considered a systemic manifestation and a lactate greater than the institution’s upper limit of normal was considered evidence of organ dysfunction [24]. A schematic of the alert is depicted in ► Figure 1. The alert criteria were chosen using guidelines from the Surviving Sepsis campaign.

Table 1.

Encounters selected for validation. Table 1 shows characteristics of the alert positive and alert negative encounters selected for validation. An encounter is considered “positive” if the alert fired during the encounter. The principal problem represents the ICD-9 code selected by the physician as the patient’s primary problem for the encounter.

	Alert Positive	Alert Negative
Total Encounters	100	75
Mean Age	64.9 (1.7)	55.9 (2.2)
Principal Problem
Sepsis or severe sepsis	9 (9%)	1 (1%)
Other infection	28 (28%)	12 (16%)
Surgery or surgical complication	14 (14%)	18 (24%)
Hematologic or oncologic disease	15 (15%)	12 (16%)
ACS, heart failure, or vascular disease	18 (18%)	12 (16%)
Other	16 (16%)	20 (27%)
Primary Service at Firing
General medicine	23 (23%)	22 (29%)
Hematologic or oncologic service	24 (24%)	7 (9%)
General surgery/trauma/surgical ICU	11 (11%)	6 (8%)
Other surgical service	39 (38%)	31 (41%)
Other medical service	3 (4%)	9 (12%)
Positive cultures for admission
Blood cultures	14 (14%)	2 (2.6%)
Other cultures	28 (28%)	5 (6.7%)
Died during Admission	17 (17%)	0 (0%)
Mean time from Admission to Firing (days) (SD)	3.4 (0.47)	N/A

Table 2a-c.

Test performance. Alert performance by physician chart review (N = 175).

Combined (a)
Sensitivity	74.5%	(57.1% – 91.8%)
Specificity	86.0%	(80.9% – 91.1%)
PPV	50.3%	(38.4% – 62.1%)
NPV	94.7%	(89.0% – 100%)
Accuracy	84.1%	(77.7% – 90.6%)
Prevalence	16%	(11.1% – 21.3%)
Positive Likelihood Ratio	5.33	(3.12 – 7.53)
Negative Likelihood Ratio	0.29	(0.08 – 0.51)
Medical Services (b)
Sensitivity	73.6%	(54.6% – 92.7%)
Specificity	84.6%	(75.9% – 94.0%)
PPV	47.6%	(33.9% – 61.2%)
NPV	94.4%	(87.8% – 100%)
Accuracy	82.8%	(72.6% – 93.2%)
Prevalence	16%	(9.0% – 22.8%)
Positive Likelihood Ratio	4.79	(1.92 – 7.67)
Negative Likelihood Ratio	0.31	(0.12 – 0.60)
Surgical Services (c)
Sensitivity	75.2%	(52.8% – 97.5%)
Specificity	91.5%	(83.2% – 99.9%)
PPV	53.1%	(32.9% – 73.2%)
NPV	94.9%	(88.2% – 100%)
Accuracy	85.4%	(78.9% – 91.9%)
Prevalence	16%	(8.4% – 23.5%)
Positive Likelihood Ratio	8.84	(5.1 – 12.58)
Negative Likelihood Ratio	0.27	(0.05 – 0.51)

Fig. 1.

Schematic representation of an electronic algorithm for the detection of severe sepsis embedded in the Epic EHR platform. The algorithm requires the presence of criteria in three categories: systematic manifestations, suspicion of infection, and markers of end-organ dysfunction, over a 24-hour period.

The alert was implemented using the “Best Practice Alert” functionality of the Epic Systems platform. The Epic platform integrates physician notes, laboratory studies, imaging studies, and order entry into a single system. All clinical documentation and clinical data from inpatient and outpatient encounters appears in the same system. Order entry is performed in this system as well. When triggered, the alert is capable of generating an automatic page to the treating physician,. The text of this page is:

“Auto EMR alert. first name, last name MRN, Bed, screens positive severe sepsis. Last filed CMAP: __ mmHg BP: ___ mmHg P: __ R: __. Assess pt/discuss with RN.”

The alert was initially validated using ICD-9 diagnosis codes. ICD-9 codes have been in the past to track rates of severe sepsis and validate computerized sepsis alerts [1,25]. A true positive was defined by the presence of a ICD-9 code for severe sepsis (995.82) or septic shock (785.52, 998.02) listed as a primary or secondary diagnostic code for that encounter. Billing codes confirmed the high sensitivity and specificity of the alert. The alert criteria were applied retrospectively to all inpatient adult encounters at Stanford Hospital from November 2, 2013 – February 1, 2014 (supplementary appendix). However, a sepsis alert can promote timely diagnosis and treatment only if it accurately detects severe sepsis at or near its onset. Due to the limitations of coding data, we thus undertook a validation of the alert’s test performance and ability to detect sepsis in a timely fashion using physician chart review.

2.1.2 Study Population for Chart Validation

We used a stratified random sample of encounters from the coding validation (i.e. those patients who screened positive for severe sepsis with 3 or more M criteria, 1 or more S criteria and 1 or more OD criteria). First, we selected at random 100 severe sepsis screen-positive adult patients hospitalized at our institution from August 1, 2013 – February 1, 2014 on a medical/surgical ward, intermediate ICU, or surgical ICU, excluding patients with a “comfort care” code status. The initial intention was to exclude ICU patients as outside the alert’s intended scope, but due to the small number of surgical patients with a billing code for severe sepsis, surgical ICU patients were included. The sample was stratified to include 50 patients whose primary service was general medicine or a medical subspecialty and 50 patients whose primary service was general surgery/trauma or a surgical subspecialty. The paging alert was not clinically utilized during this period (i.e. no automated page was sent to the treatment team).

Second, in order to determine the alert’s sensitivity, we also randomly sampled 75 patient encounters that did not meet criteria for a positive sepsis screen/alert according to their M, S and OD criterion during the admission. This sample population was drawn from the same medical/surgical wards, intermediate intensive care units and surgical intensive care units during the same time period as the severe sepsis-positive population. Since the alert was designed for inpatient use, patients still physically located in the emergency department were excluded as were any patients with a code status of ‘comfort care’.

2.1.3 Definition of Severe Sepsis and Clinical Time Zero

For the purpose of the physician chart review, we utilized the definition of severe sepsis from the Surviving Sepsis campaign [24]. Severe sepsis was defined as a systemic inflammatory response or other manifestations with suspected infection and end-organ dysfunction attributed to sepsis. The evidentiary threshold employed was “sufficient concern to justify treatment”: an encounter was coded as positive if, in the judgment of the reviewing physician, severe sepsis was a sufficiently high concern that treatment was indicated at the time of firing. That is, that severe sepsis was a sufficient concern that they would treat the patient were they the treating physician at that time. The physician reviewers screened all vital signs, laboratory data, and notes to make this determination. Clinical notes were utilized to provide information on the patient’s signs and symptoms. Physicians could use any information from clinical notes regarding the patient’s subjective reported information, observed symptoms and exam (e.g. the patient complains of productive cough and shortness of breath, new disorientationetc.) Physicians underwent a training session prior to chart review to ensure a uniform approach. During training, residents reviewed the Surviving Sepsis campaign guidelines for the definition of severe sepsis and the criteria for determining whether an encounter qualified for severe sepsis or not. Specific example encounters were reviewed and discussed with an attending hospitalist. Training occurred in the context of the broader initiative to develop a severe sepsis alert. During this process, residents met regularly with a sepsis alert committee to review specific cases.

For patients deemed by the reviewing physician to have severe sepsis, a ‘clinical time zero’ was identified as the first moment the patient exhibited at least one marker of a systematic inflammatory responses (attributed to infection rather than another cause), suspected infection, and evidence of end-organ dysfunction, according to Surviving Sepsis campaign guidelines. Suspected infection was derived from the reviewing physician’s clinical judgment, based on documented history, signs and symptoms, laboratory, radiologic, and microbiologic evidence, such as radiographic and clinical evidence of pneumonia, positive blood cultures, or a history consistent with urinary tract infection. End-organ dysfunction was defined as the presence of at least one source of end-organ dysfunction reasonably attributable to infection. For example, the presence of thrombocytopenia might be excluded as a criterion if the patient has a malignancy or was receiving chemotherapy known to cause thrombocytopenia, especially if thrombocytopenia predated other evidence of sepsis. The evidentiary threshold was again “sufficient concern to justify treatment.”

During some patients’ hospitalization, the alert would have fired more than once. For the purposes of all validations in this study, the first alert firing was used. As noted, the alert was not active at the time: no physicians or nurses received the alert. The purpose of the data was to determine the alert’s performance prior to implementing the alert for active use.

2.2 Validation of Alert Performance

2.2.1 Validation of Test Performance and Clinical Time Zero by Physician Chart-review

All 175 alert positive and negative encounters were reviewed by one of two third-year internal medicine residents to determine if the patient had severe sepsis during admission. The reviewers were not blinded to the status of the alert (positive vs. negative). All data from physician notes, laboratory data, imaging, and procedures were reviewed beginning with the patient’s admission. Using criteria specified above, the reviewer determined if severe sepsis was present and, if so, the date and time of clinical time zero. A random sample of ten cases was reviewed by both physicians to determine inter-rater variability for the presence of severe sepsis. Performance endpoints included sensitivity, specificity, positive predictive value, negative predictive value, and accuracy (ratio of the sum of true positives and true negatives to the sum of true positives, true negatives, false positives, and false negatives).

In addition, alert firing was compared to the institution’s current sepsis screening standard care, a bedside nursing screen. This nursing screen is performed on all units outside the intensive care units at each change of shift and with any change in the patient’s clinical status (such as vital sign disturbances or new subjective complaints). When the screen is performed by the nurse at the beginning of a shift, the nurse is expected to review laboratory values and vital signs, The nurse also determines if infection is suspected based on physician notes or the nurse’s own assessment. The nurse proceeds through a EHR checklist for this screen. If the screen is positive for simple sepsis, the nurse then proceeds with organ dysfunction screening to screen for severe sepsis.

2.2.2 Comparison to Physician Diagnosis and Treatment of Sepsis

In addition to early identification of patients with severe sepsis, a sepsis alert might improve outcomes by identifying cases of severe sepsis that are undertreated. Undertreatment can occur because sepsis is not diagnosed or diagnosed late, because sepsis is not treated appropriately, or because treatment is not appropriately escalated with a change in clinical condition. To assess diagnosis and treatment, three physicians reviewed all true positives: patients who were deemed to be severely septic based on chart review at the time the alert fired. The three physicians included one attending hospitalist, one hematology/oncology fellow, and one medical intern. Review focused on recording if 1) sepsis or severe sepsis was documented in either the admission history and physical or the daily progress note on the day the alert would have fired, 2) whether the patient was treated with intravenous fluids and antibiotics according to Surviving Sepsis guidelines, and 3) for alerts that fired at clinical time zero, whether the three hour sepsis bundle was completed. For the purpose of this analysis, intravenous antibiotics were defined as any intravenous antibiotics administered in the 24 hours prior to alert activation. A change in antibiotic therapy was defined as any new intravenous antibiotic within three hours of the alert, including the substitution of one antibiotic for another one or the addition of a new antibiotic to the regimen in place prior to the alert. Intravenous fluids were defined as isotonic crystalloid solutions or colloid solutions. The three hour bundle also included blood cultures and measurement of a serum lactate.

2.3 Sampling Method and Statistical Analysis

The sample was stratified by alert status and service (medical or surgical) to ensure an adequate number of alert-positive encounters. Encounters were weighted using the inverse of the probability of selection to generate population-representative estimates from the stratified sample. The population size was determined using institutional data from the period of selection on alert firing and patient service location. A similar method has been used in prior studies with stratified chart reviews [26–27]. Confidence intervals were bootstrapped using case resampling for chart validations to produce 95% confidence intervals. For administration validation, a parametric bootstrap was performed using a Poisson distribution. Cohen’s Kappa was used to calculate inter-rater agreement for determination of severe sepsis.

All analysis was performed using Stata 13 [28].

3. Results

3.1 Test Performance

The sample of alert positive and negative encounters is shown in ► Table 1. The mean time from admission to firing was 3.4 days. In the alert group, 14% of patients had positive blood cultures and 28% had other positive cultures or microbiologic testing (e.g. respiratory PCR for influenza). 17% of patients died during admission. In the negative group, 2.6% patients had positive blood cultures and another 6.7% patients had other positive cultures. We excluded one patient with one out of four bottles positive for a common blood contaminant. All other cultures were negative for the patient’s admission.

► Table 2 shows the overall performance of the alert. Sensitivity was 74.5% (57.1% – 91.8%) and specificity was 86.0% (80.9% – 91.1%). The positive predictive value was 50.3% (38.4% – 62.1%) with a negative predictive value of 94.7% (89.0% – 100%). The positive likelihood ratio was 5.33 (3.12 – 7.53) and the negative likelihood ratio was 0.29 (0.08 – 0.51). Two-by-two tables of alert performance are included in the Supplementary Materials. It performed well for patients on medical and surgical services. Kappa for the presence of severe sepsis was 1.00: both reviewers agreed on all ten cases.

The most common source of end-organ dysfunction was a MAP less than 70 mm/Hg (59%), the second most common was a systolic blood pressure less than 90 mm/Hg (32%). Hypotension in the form of a low MAP or systolic blood pressure was present in 72% of cases. Hypotension or hyperlactemia (lactate >= 3) was present in 77% of cases. The least common source of documented end-organ dysfunction was delayed capillary refill (2%).

3.2 Detection of Clinical Time Zero

The alert was triggered at clinical time zero in 41% of encounters (► Table 3). The mean lag time between clinical time zero and the alert was 7.6 hours. However, after excluding six outliers in which the alert was triggered more than 24 hours after time zero, the mean lag time decreased to 2.6 hours. In one case, the alert fired before clinical time zero. The median lag time was 2.7 hours. The 25% percentile was 0 hours (triggered at time zero) and the 75% percentile was 7.4 hours. The alert was then compared to the bedside nursing screen. A positive screen for severe sepsis was present in 67% of cases of validated severe sepsis. In all but two cases (96%), the alert triggered before a positive nursing sepsis screen. The nursing screen was delayed by a mean 10.8 hours (3.65 – 18.1 hours), even after excluding eight cases in which the nursing screen lagged the alert by more than 48 hours.

Table 3.

Estimation of clinical time zero, defined as the first moment a patient exhibited evidence of severe sepsis by physician chart review. The table includes all encounters. When six encounters with an alert lag ≥ 24 hours were excluded, the mean lag time decreased to 2.6 hours. Nursing screen data excludes surgical ICU encounters because the nursing screen is not performed in the intensive care units.

Relationship between physician chart-review clinical time zero, alert time, and a nurse-administered screen time (N = 63)
Alert triggered at clinical time zero	41.1% (23.3% – 58.9%)
Alert triggered at time zero or within three hours	53.6% (40.6% – 66.7%)
Mean lag time (hours)	7.6 (3.6 – 11.5)
Positive nursing screen for severe sepsis	67% (48.6% – 85.6%)
Positive nursing screen occurred after alert	96% (90.6% – 100%)

The presence of a billing code for severe sepsis or septic shock had a high positive predictive value of 88.6% (79.5% – 97.7) and the absence of a billing code had a negative predictive value of 88.2% (79.3% – 93.8%). However, due to the small percentage of encounters coded for severe sepsis, coding only captured 30% of all encounters determined to involve severe sepsis by chart review.

3.3 Physician Treatment of Sepsis

Given that the alert relies on a physician order, it is possible that the alert only detects cases of severe sepsis that have already been identified by clinicians and thus does not add value in early identification. To investigate, we analyzed physician treatment of sepsis for all true positive firings (► Table 4). Results show that although sepsis was recognized and treated in many cases, it also went untreated in other cases. Sepsis was documented in physician notes in less than 50% of cases (42.1%) identified by the alert. In 23.4% of cases, the patient had no orders for intravenous antibiotics at the time the alert fired.

Table 4a-b.

Identification and treatment of sepsis in the absence of a computerized sepsis alert. Table 4a-b shows the mean percentages with 95% confidence intervals. Table 4a shows antibiotics and fluids administered in the 24 hours prior to the alert or within three hours after the alert for vasopressors. Table 4b shows interventions performed within three hours after the alert for alerts triggers precisely at clinical time zero.

All True Positives (a) (N = 63)
Sepsis documented in physician note	42.1%	(31.5% – 52.9%)
Intravenous antibiotics prior to sepsis alert	76.6%	(66.9% – 86.1%)
Intravenous fluids prior to sepsis alert	59.3%	(49.0% – 69.7%)
Started on vasopressors before or after alert	5%	(0% – 10%)
Alert Activated at Time Zero (b) (N = 28)
Change in antibiotic therapy in three hours	42.9%	(26.9% – 58.7%)
Received intravenous fluids in three hours	50.0%	(31.5% – 65.4%)
Did not receive fluids or changed antibiotics	28.6%	(11.3% – 45.7%)
Mean intravenous fluids in three hours (L)	0.5	(0.25 – 0.75)

A subgroup analysis was performed for cases in which the alert was triggered at clinical time zero. 42.9% received a change in antibiotic therapy, defined as initiating intravenous antibiotics or changing therapy if the patient was already on antibiotics. 50% of patients received intravenous fluids within three hours of the time the alert would have fired. Mean fluid administration was 0.5 L. 24% of patients received more than 1 L intravenous fluids, after excluding one patient noted to have volume overload at clinical time zero. There were no changes to either antibiotics or intravenous fluid orders in 28.6% of patients

4. Discussion

Despite numerous published attempts to design an accurate, timely sepsis alert, published alerts have had limited success. We developed a severe sepsis alert that uses standard Epic EHR functionality to accurately identify patients with severe sepsis with reasonable timeliness. The alert is capable of generating an automatic page to the physician, thus eliminating the need for the physician to open the chart to be notified. Using physician chart review as the gold standard, we present the first description of the time between an electronic sepsis alert and the actual onset of severe sepsis. Overall, this alert performed well. In 54% of all cases, the alert was triggered within three hours of time zero. In more than 40% of cases, it was triggered at clinical time zero. In a small number of cases, the alert lagged behind clinical time zero by more than 24 hours, usually due to delays in physician orders for cultures or lactates. This data demonstrated that the alert has the capability to facilitate early antibiotics and completion of three and six hour bundle goals. By outperforming the current standard care of a manual bedside nursing screen, it suggests the potential to augment, if not eventually replace, more time-intensive manual screening, in which a nurse must run through a check list of sepsis markers, with continuous, automated, EHR-based evaluation.

We employed a dual method of validation—first with administrative data as a silver standard, and then with physician chart-review as the gold standard. The use of administrative data permitted us to validate thousands of encounters in a short period. However, our subsequent analysis revealed that ICD-9 coding is specific but not sensitive for the presence of severe sepsis. Our findings are consistent with prior results showing that administrative data identifies only the most severe cases of sepsis [29]. Importantly, the timing of onset of sepsis cannot be determined via administrative data and therefore, cannot determine if an alert provides timely detection. Our results show that, though labor-intensive, clinical chart review should supplement administrative data in validating detection algorithms for sepsis. We noted that the majority of cases of severe sepsis identified by physician chart-review were not documented as such – consistent again with literature demonstrating a vast underestimation of the prevalence of severe sepsis in the inpatient setting [23].

Although our alert performs well, to maintain specificity it depends, in part, on physician orders suggestive of clinical concern for infection such as an order for blood cultures or serum lactate. This decision imposes a cost: the alert misses some cases of severe sepsis. Furthermore, by missing cases that lack recent physician orders, the alert may overlook the cases in which severe sepsis is most likely to go undetected. Physicians need to maintain clinical vigilance to minimize the risk of missing sepsis. This choice thus reflects an important trade off.

Nevertheless, we believe the benefits outweigh costs. Despite this limitation, the alert was activated at or within three hours of clinical time zero in 53.6% of cases—that is, the alert fired in close proximity to the earliest time a physician would be able to identify severe sepsis. In the vast majority of cases, severe sepsis does not develop absent any clinical concern for infection. Rather, failures to intervene appropriately occur because of delays in diagnosis, a failure to conceptualize a change in condition as the onset of sepsis, or failure to initiate prompt treatment after diagnosis. Thus, an accurate, timely alert can still facilitate improved treatment, even in the presence of independent clinical concern, by flagging the presence of severe sepsis and directing additional resources in the form of a specialized crisis nurse. In addition, as a practical matter, we have found that low specificity can prevent staff buy-in. A sepsis alert is only effective if it can change behavior. In our own case, we sought to change behavior by protocolizing a response to the alert on the part of a crisis nurse, who evaluates the patient and communicates with the physician. High specificity helps ensure staff adherence to a protocol that diverts a busy nurse from other important tasks. That said, the alert is designed only to supplement other institutional initiatives to promote sepsis recognition, ranging from traditional methods, which include vital sign notification parameters, to sepsis awareness campaigns for physicians and nurses.

The positive predictive value was in the range of 45–50%, suggesting that approximately half of all alerts were false positives. It should be noted that 33% of the “false positives” were patients who still had evidence of simple sepsis, that is to say, suspected or documented infection with systematic manifestations but without evidence of organ dysfunction. Furthermore, as should be clear, the positive and negative predictive values are affected by the prevalence of sepsis in the applicable patient population. To provide a measure of the alert’s performance unaffected by the institutional prevalence of sepsis, we also report positive and negative likelihood ratios. The positive likelihood ratio was 5.33 and the negative likelihood ratio was 0.29. Ultimately, the alert’s diagnostic usefulness will depend on the prevalence of sepsis in the population to which it is applied.

Our study design also had several limitations. Due to the labor-intensive nature of chart review, our validation had a small sample size, resulting in our large confidence intervals. Since only a fraction of encounters possess severe sepsis, our calculated sensitivity reflects an even smaller sample size. In addition, as noted, due to the stratified nature of our sample, a small absolute number of false negative encounters assumed a significant weight in the analysis. In addition, the physician reviewing the chart was not blinded to whether the alert had been triggered for that encounter. Although our inter-rater agreement was high and both reviewers were trained to use a standardized evaluation process, knowledge of the alert may have biased both reviewers in the same direction, artificially inflating our test characteristics. In addition, a small number of cases were used to calculate inter-rate agreement. Finally, our reviewing physicians were third-year internal medicine residents, not attendings. To mitigate this limitation, the residents were trained by an attending and inter-rater agreement was calculated for the presence of severe sepsis. Inter-rater agreement for clinical time zero was not calculated.

Currently, we are evaluating the impact of our sepsis alert and the associated clinical workflow downstream of the alert in a randomized control study. The impact of a best practice alert for severe sepsis depends not only on accurate detection of severe sepsis, but critically on the downstream clinical workflow. The variable performance of sepsis alerts in prior studies may reflect differences in effective clinical workflow emphasizing the importance of the clinical workflow in place to respond to a positive alert. Additional current efforts include further refinement of the alert criteria, and the development of separate alerts for different patient populations. As the capabilities of EHRs evolve, we expect alerts to develop increasing ability to monitor changes in measures of organ dysfunction, such as selected changes in serum creatinine, permitting more dynamic predictive models that rely on changes from a base state. Ideally, alerts will also develop the ability to use natural language processing, such as by extracting text like “infiltrate” from a chest radiograph. These changes may lead to increasingly accurate ability of alerts to identify—and even predict—the onset of sepsis [30].

5. Conclusion

We created a severe sepsis alert utilizing standard commercial EHR functionality. We are the first to our knowledge to describe the time between an electronic sepsis alert and the actual time of onset of severe sepsis through extensive chart review. 41% of alerts were triggered at time zero, and 96% of alerts were triggered before a manual nurse screen. Using physician orders in alert criteria can improve specificity while maintaining reasonable sensitivity. Standard EHR functionality can provide robust alerting functionality while maintaining generalizability to other health systems interested in adopting our approach.