Summary
Summary
To conduct an independent secondary analysis of a multi-focal intervention for early detection of sepsis that included implementation of change management strategies, electronic surveillance for sepsis, and evidence based point of care alerting using the POC AdvisorTM application.
Methods
Propensity score matching was used to select subsets of the cohorts with balanced covariates. Bootstrapping was performed to build distributions of the measured difference in rates/means. The effect of the sepsis intervention was evaluated for all patients, and High and Low Risk subgroups for illness severity. A separate analysis was performed patients on the intervention and non-intervention units (without the electronic surveillance). Sensitivity, specificity, and the positive predictive values were calculated to evaluate the accuracy of the alerting system for detecting sepsis or severe sepsis/ septic shock.
Results
There was positive effect on the intervention units with sepsis electronic surveillance with an adjusted mortality rate of –6.6%. Mortality rates for non-intervention units also improved, but at a lower rate of –2.9%. Additional outcomes improved for patients on both intervention and non-intervention units for home discharge (7.5% vs 1.1%), total length of hospital stay (-0.9% vs –0.3%), and 30 day readmissions (-6.6% vs –1.6%). Patients on the intervention units showed better outcomes compared with non-intervention unit patients, and even more so for High Risk patients. The sensitivity was 95.2%, specificity of 82.0% and PPV of 50.6% for the electronic surveillance alerts.
Conclusion
There was improvement over time across the hospital for patients on the intervention and non-intervention units with more improvement for sicker patients. Patients on intervention units with electronic surveillance have better outcomes; however, due to differences in exclusion criteria and types of units, further study is needed to draw a direct relationship between the electronic surveillance system and outcomes.
Keywords: Sepsis, clinical decision support, electronic surveillance, sepsis outcomes, informatics
1. Background and Significance
Sepsis continues to escalate in incidence, costs and poor outcomes regardless of international consensus about the best strategies for early identification and treatment of sepsis [1–3]. Mortality rates for severe sepsis range from 10 to 30%, and quality of life for patients that survive is diminished, including significant organ dysfunction resulting in chronic health problems, decreased functional status, cognitive impairment, and loss of independence [4]. While evidence-based guidelines exist for early detection and treatment of sepsis, compliance with guideline recommendations has been a challenge [5, 6].
1.1 Summary of Multi-focal Interventions to Detect Sepsis and Improve Outcomes
The Surviving Sepsis Campaign (SSC) guidelines represent international consensus for early detection and treatment leading to improved outcomes for sepsis and severe sepsis [7]. In general, there is agreement across studies that multi-focal interventions improve compliance with SSC guidelines and/or demonstrate improvement in patient outcomes [8–11]. Levy and colleagues provided sepsis resources to 218 community, academic and tertiary care hospitals in the United States, South America, and Europe who then developed their own sepsis protocols. In addition, an electronic bimonthly newsletter was sent out to share strategies, success stories, and events. These investigators found that compliance with the 6-hour sepsis bundle increased from 10.9% in the first site quarter to 31.3% by the end of 2 years. Also mortality was lowered with an increase in compliance. Another study conducted in intensive care units (ICUs) [11], implemented sepsis education and the use of a software tool, Protocol Watch, to evaluate compliance with sepsis management and resuscitation bundles. Protocol Watch automated detection of sepsis and provided a screen alert. There was a significant difference in compliance with the resuscitation bundle but not the management bundle after implementing Protocol Watch. Palleschi et al. [10] combined education in multiple formats about sepsis, posters and badges reminding providers about sepsis care guidelines, and electronic health record (EHR) alerting sent to pagers. They found that compliance and timing of the 3-hour sepsis bundle improved, though all findings were not significant. Sensitivity and specificity of the alerts were not reported. When comparing a paper vs computerized protocol for detection and management of sepsis, investigators found that compliance with the sepsis bundle increased and mortality decreased [12]. In addition to the computerized protocol, significant team consensus was built for development of the protocol and daily rounds of patient progress were included. Some studies implemented screening tools [13, 14] while others included alerts based on patient parameters, order sets, and recommended interventions [10–12, 14]. The sensitivity and specificity of alerts varied considerably across studies [14–16]. The higher rate of accurately detecting sepsis is associated with settings such as the emergency department or ICUs.
What is unknown is accurate detection and improved outcomes with a multi-focal intervention, which includes use of an electronic surveillance system that does early detection and alerting, combined with quality improvement processes on general medical surgical floors. The purpose of our study was to conduct an independent analysis of data from a multi-focal intervention for early detection of sepsis that included implementation of change management strategies, electronic surveillance for sepsis, and evidence based point of care alerting using the POC AdvisorTM application [17]. Details of the original study are explained next.
1.2 Evaluating a Multi-focal Intervention and Electronic Surveillance System
A recent study [17] sought to address the problem of electronic sepsis surveillance, the POC A dvisorTM Application (http://www.pocadvisor.com/), as part of a multi-focal intervention using a quasi-experimental study design for early detection of sepsis and improvement of outcomes (mortality, readmissions, and length of stay). One of the major components of the POC Advisor was to develop an electronic surveillance system that had the highest accuracy possible for detecting sepsis and providing advice via a mobile device to nurses based on the 3- and 6-hour bundles from the Society for Critical Care’s Surviving Sepsis Campaign [7].
1.2.1 Setting and Sample
The study was conducted at a community owned regional hospital that includes 941 beds between three campuses with 42,000 annual inpatient discharges and 164,000 emergency department visits. The three units were located on two floors that included 58 beds in two respiratory care units and one medical surgical unit. The multi-focal intervention was implemented over a 4-month period from October 1, 2013 – February 28, 2014 (implementation phase). Subsequently, real-time electronic surveillance began in March, 2014 with data collected over a 10 month period from March 1, 2014 – December 31, 2014 (the post-implementation phase). The data from the study units was retrospectively compared to data from January 1, 2011 through September 30, 2013 (pre-implementation phase) for the incidence of sepsis and quality indicators. Patients were excluded from calculating outcomes on the study units 1) if they were sent directly to the ICU from the emergency department or received care for more than 4 days in the ICU before admission to the study units, and 2) if they had limited care in which appropriate sepsis care was withheld such as patients receiving palliative or end-of-life care (documented as comfort measures only). The reason for exclusions was that patients could not fully benefit from the electronic surveillance system as ICU patients would have received the bulk of their care before moving to the study units and patients with comfort care only were excluded as not all interventions would be applied.
1.2.2 Multi-focal Intervention Description
The intervention was composed of education, change management, and electronic surveillance (only on study units). Hospital-wide, a variety of interventions for early detection and prevention of sepsis were implemented: creating an interdisciplinary governance committee; creating a sepsis screening instrument, order sets and protocols; modifying nursing documentation to include discrete data for the sepsis algorithm; educating staff about the sepsis protocols; conducting grand rounds; and, creating excitement and awareness for adoption of the changes.
The three study units received education about and implemented the POC Adviser for electronic surveillance. The POC Adviser was designed as an expandable set of rules whose goal was to create very high sensitivity and specificity. The algorithms were based on the 2012 Surviving Sepsis Campaign and the Society of Critical Care Surviving Sepsis Campaign Guidelines [7, 18] and consisted of over 100 rules for early detection of sepsis and alerting clinicians to take appropriate action or dismiss alerts with a rationale. The rules were disease and medication specific to account for abnormal labs and vital sign abnormalities that resulted due to other comorbid conditions such as abnormal test results due to chronic liver disease, chronic renal disease, heart failure, alcohol withdrawal, malignancy, emphysema, etc. Another example is that lactate tests are usually elevated in patients with cirrhosis, and so interpretation of lactate values was different in patients with liver disease compared to healthy patients. A flow diagram of the sepsis rules, order set, and lactate protocol are available in the supplementary Appendix from the primary publication of the study [17].
Prior to transferring to the study units, patients received all components of the multi-focal sepsis intervention provided throughout the hospital; the POC Adviser then was added for electronic surveillance once the patient was admitted to one of the study units. The nurses performed their usual documentation which included assessing the patient at the time of a new admission to floor, vitals assessed approximately every 4 hours and as needed with a minimum of once every 12 hours. The system was designed to push evidence-based advice directly to the point of care via smart devices carried by nursing staff with four types of advice individualized to the patient and based on the protocols drafted by the change management process. The four types of alerts were: 1.) Informational prompts, like isolated tachycardia, isolated hypothermia, etc.; 2.) Diagnostic alerts that informed nurses about new positive sepsis screening results or signs of worsening sepsis in patients who previously screened as positive for sepsis; 3.) Advice alerts containing narratives for evidence-based care for sepsis, such as intravenous fluids, antibiotics, lactate testing, and recommended site-specific sepsis order sets, communication protocols, etc. (the advice mirrored and supported the protocols created during the change management effort); and 4.) Reminder alerts, which ensured that all alerts were acknowledged and that staff were complying with the recommended treatment plans [17, p.2]. Nurses implemented the order set for sepsis and the lactate protocol as well as contacting the provider. Concurrently, the charge nurses received alerts for severe sepsis and septic shock.
A subset of patients during 3 months of the post-implementation phase on the study units were evaluated comparing the accuracy of sepsis diagnosis alerts from the POC Advisor with comprehensive chart reviews (with full access to the EHR) to determine sensitivity, specificity, and positive predictive values (PPV). Sepsis diagnoses were obtained from the billing data; this method has been found to have high specificity [20]. During chart review, the presence of two or more of the SIRS criteria and infection were used to define sepsis. Severe sepsis included these criteria plus the presence of organ dysfunction due to sepsis while septic shock also included hypotension that was persistent after an initial fluid challenge or a serum lactate > mmol/L due to sepsis. Since the presence of infection is necessary for diagnosing sepsis, a certified medical coder first screened the International Classification of Diseases Ninth Revision (ICD9) billing codes for cases with infection to filter the cases that needed chart review. Chart review was done for all cases that fired any type of alert within the POC Advisor AND for all patients that had any Angus ICD9 code for infection [21]. Patients that neither fired any alerts nor had any codes for infection or sepsis were assumed to not have sepsis and were marked as true negatives. Overall, determination of sepsis was made by two internists experienced in treating sepsis and with informatics training who conducted the chart reviews using the latest evidence based guidelines, „Sepsis syndromes in adults: Epidemiology, definitions, clinical presentation, diagnosis, and prognosis” [18]. The physicians reviewed all vital sign, labs, and clinical notes to determine the diagnosis of sepsis. The physicians were not blinded to the sepsis diagnosis or chart alert during the review. All disagreements were reviewed and adjudicated. Both physicians reviewed a subset of each other’s cases to ensure internal validity. An external physician reviewed 10% random sample of all cases. To allow for a conservative evaluation of performance, the following three categories were used for chart review results: 1.) No Sepsis (no sepsis or possible sepsis), 2.) Sepsis (sepsis possible severe sepsis), and, 3.) Severe Sepsis or Septic Shock. The software audit trail was used to determine if patients had a sepsis alert (yes/no).
Overall, there were no significant differences in age, sex, or comorbid conditions between the pre implementation and post implementation periods. The study units with the POC Alerts had a 53% decrease in mortality and a decrease of 13% in hospital readmission, but no significant change in length of hospital stay. A Kappa statistic to evaluate agreement between an independent physician knowledgeable about sepsis care and the 2 original reviewers was conducted with a Kappa of 0.67 (95% CI, 0.41 – 0.92) for a 10% random sample of charts. The sensitivity of alerts was 95%, specificity of 82%, and PPV of 50.21 (43.64, 56.78) compared to a gold standard of physician chart review. Because the original study was conducted by the developers of the POC Adviser, we were asked to do an independent analysis of the data using methods we deemed appropriate.
2. Method
2.1 Purpose and Design
The purpose of this study was to conduct an independent evaluation of the data from a multi-focal intervention for early detection of sepsis that included implementation of change management strategies, electronic surveillance for sepsis, and evidence based point of care alerting using the POC A dvisorTM application. Specific aims were to evaluate the influence of the multi-focal intervention approach on 1.) reduction of sepsis mortality, 2.) decreased hospital length of stay, 3.) increased discharge to self-care, and 4.) decreased 30 day hospital readmission. Additionally the sensitivity, specificity, and PPV of the alerting system were evaluated. The research design for this study was a two group pre-post quasi-experimental quality improvement study. Adult patients on three hospital (intervention) units who received the electronic sepsis surveillance along with multi-focal interventions were compared before and after implementation of the electronic surveillance system. Patients in the remainder of the hospital (non-intervention units) were analyzed independently to determine if there was a maturation effect for changes in sepsis and sepsis outcomes. Data were not compared statistically between the intervention units and the rest of the hospital due to differences in the exclusion criteria applied to patients on the intervention units but not the remainder of the hospital.
2.2 Setting and Sample Selection
The setting was described in the background section of this paper. In addition to analyzing outcomes pre- and post-implementation of the electronic surveillance system, we also evaluated patients on the non-intervention units. ▶ Figure 1 shows the three phases of the original study with dates and numbers of patients for intervention and non-intervention units (the remainder of the hospital). The exclusion criteria previously described only applied to intervention units since the primary purpose of the original study was to determine the impact of the POC Adviser on outcomes. In the non-intervention units, 45.7% were admitted to the ICU during the pre-implementation phase and 41.8% during the post-implementation phase. No data was available about comfort care only for non-intervention unit patients.
Fig. 1.
Study phases
2.3 Data Source
Three files were obtained that contained de-identified data from the original study. One file included coded patient IDs, demographics (age, race, sex, and payer), intervention criteria (intervention or non-intervention unit, and exclusion criteria for patients on the intervention unit of prior ICU admission or limited care), sepsis status (identified as 995.91 sepsis, 995.92 severe sepsis, and 785.52 septic shock) and outcome data (expired, hospital length of stay, discharge location, and 30 day readmission). Data were not available for hospital length of stay for non-intervention patients since that was not the focus in the original study. A second file included all ICD9 codes used to calculate the Charlson Index of Comorbidity (CIC). Information in the third file was used to calculate accuracy of the POC AdvisorTM alerts (chart review notes; whether the patient had sepsis, severe sepsis, or septic shock based on the chart review; and if an alert was fired).
2.4 Intervention
The intervention conducted in the original study is described in the background section of this paper.
3. Analysis
3.1 Evaluating the Effect of the Intervention on Outcomes
The intervention‘s effect on outcomes was evaluated by comparing outcomes between the pre- and the post-implementation phases for adult patients on the intervention units and adult patients on non-intervention units in the remainder of the hospital. In order to control for potentially confounding covariates, the ICD-9 Deyo version of the Charlson Index of Comorbidity (CIC) was calculated using the disease weights in Charlson et al. [22] and the coding algorithms in Quan et al. [23]. Propensity score matching (PSM) was then used to select subsets of the cohorts with balanced covariates [24]. PSM is a statistical matching technique that attempts to estimate the effect of a treatment, policy, or other intervention by accounting for the covariates that predict group membership in either the treatment or non-treatment groups. In this case, the treatment groups are pre- and post-implementation phases.
A propensity model was built via logistic regression in which the group membership (i.e. pre-implementation phase vs. post-implementation phase) was the dependent variable and age, sex, insurance provider, sepsis diagnosis codes, and CIC were the predictors. This resulted in a propensity score for each patient that represents the probability of the patient‘s inclusion in the post-implementation phase with the given values of the predictor variables. A subset of the cohort with equal counts of patients from the pre-implementation phase and from the post-implementation phase and with approximately matching propensity scores were then selected using a nearest neighbor matching method [24]. From this matched set, the difference in rates between the matched pre-implementation phase patients and the matched post-implementation phase patients was measured for all binary outcomes (mortality rate, home discharge rate, and 30 day readmission rate), and the difference in means was measured for total hospital length of stay. Note that only a subset of the cohort had length of stay information available, and thus analysis for that outcome variable is used for only that subset. For the home discharge rate outcome, patients were excluded that had missing discharge information, died during hospitalization, left against medical advice, or that were discharged to court/law enforcement.
Bootstrapping was performed to build distributions of the measured difference in rates/means. Specifically, for each outcome, the original data set was randomly sampled with replacement, propensity score matching was done on each random sample, and the difference in rates/means was calculated. The distributions of these results were used to construct the 95% confidence intervals and boxplots of expected difference in rates/means. The 95% confidence intervals that were non-overlapping with zero were then considered to be a significant effect. Additionally, the entire analysis process was repeated for subsets of patients for High and Low Risk for illness severity, which was based on the severity of sepsis (sepsis vs. severe sepsis/ septic shock) or the CIC (<3 vs. ≥3). Since these partitions for severity of sepsis or the CIC alone would result in too small of a sample size for some groups of patients (▶ Table 1 as an example), we chose to group together all patients with either severe sepsis, septic shock, or a CIC ≥3 as a High Risk group, and all other patients as a Low Risk group. This resulted in a more balanced split and thus larger sample sizes for the intervention group (358 Low Risk patients vs. 420 High Risk patients) than either the regular sepsis vs. severe sepsis split (605 vs. 173), or the low comorbidity vs. high comorbidity split (435 vs. 343). All computations and statistical analyses were done using R v3.1.1 [25] and the MatchIt package [26].
Table 1.
Example of Patients in the Intervention Group by CIC (< 3 vs. ≥ 3) and by Sepsis Severity*
| Regular Sepsis Only | Severe Sepsis/Septic Shock | Total | |
|---|---|---|---|
| Low CIC (< 3) | 358 (LR) | 77 (HR) | 435 |
| High CIC (> 3) | 247 (HR) | 96 (HR) | 343 |
*Entries marked with (LR) indicate membership in the “Low Risk” patient subgroup, while entries marked with (HR) indicate membership in the “High Risk” patient subgroup.
3.2 Determining Accuracy of the Sepsis Alerts
The notes from the chart audit provided to investigators were reviewed for agreement on sepsis diagnoses. The sensitivity, specificity and PPV for sepsis alerts were calculated for all patients during the post-implementation phase on intervention units as well as sensitivity and specificity for Low and High Risk patients. The PPV was not included for subgroups because the POC Adviser alert was meant to indicate sepsis in general, not a certain severity of sepsis.
4. Results
▶ Table 2 shows the sample size, baseline characteristics, most common sources of infection using Angus codes [21], and outcome information for the four patient subgroups (i.e. intervention group during the pre- and post-implementation phases and non-intervention group pre- and post-implementation phases). Demographics and baseline characteristics were fairly consistent across subgroups with the exception that there were more noticeable differences between subgroups for comorbidity and for sepsis diagnosis. For example, the proportion of patients with either a severe sepsis or septic shock diagnosis ranged from 20% (for the intervention group in the pre-implementation phase) to 42% (for the non-intervention group in the post-implementation phase).
Table 2.
Comparison between Non-intervention and Intervention Patients and Between the Pre- and Post- Implementation Study Phases
| Pre-Implementation Phase | Post-Implementation Phase | |||||||
|---|---|---|---|---|---|---|---|---|
| Non-Intervention Units (n, %) | Intervention Units (n, %) | Non-Intervention Units (n, %) | Intervention Units (n, %) | |||||
| Sample size (n) | 4244 | 566 | 1509 | 212 | ||||
| Age | ||||||||
| Range | 18 – “> 90” | 19 – “> 90” | 18 – “> 90” | 19 – “> 90” | ||||
| Mean ± SD | 64.1 ± 16.4 | 64.1 ± 16.7 | 62.7 ± 17.4 | 62.5 ± 17.1 | ||||
| (25th %, Median, 75th %) | (54.0, 66.0, 77.0) | (53.0, 67.0, 76.0) | (52.8, 65.0, 76.0) | (51.0, 65.5, 76.0) | ||||
| Sex (% male) | 52% | 2,210 | 55% | 313 | 51% | 764 | 115 | 54% |
| Sepsis group (%) | ||||||||
| Sepsis ONLY | 63% | 2,661 | 80% | 451 | 58% | 872 | 73% | 154 |
| Severe Sepsis OR Septic Shock | 37% | 1,583 | 20% | 115 | 42% | 627 | 27% | 58 |
| Source of Infection Codes (%) | ||||||||
| 038 Septecemia | 99% | 4,184 | 98% | 554 | 98% | 1,481 | 98% | 208 |
| 486 Pneumonia, organ unspecified | 17% | 724 | 27% | 155 | 19% | 291 | 30% | 64 |
| 041 Bacterial infection in conditions classified elsewhere and unspecified site | 13% | 539 | 12% | 70 | 12% | 184 | 14% | 29 |
| 682 Other cellulitis and abscess | 11% | 444 | 10% | 57 | 11% | 163 | 13% | 27 |
| Payer (%) | ||||||||
| Medicare | 66% | 2802 | 71% | 403 | 63% | 957 | 66% | 140 |
| Medicaid | 5% | 201 | 4% | 22 | 5% | 75 | 7% | 14 |
| Other | 29% | 1241 | 25% | 141 | 32% | 477 | 27% | 58 |
| Charlson Index of Comorbidity (%) | ||||||||
| 0 | 15% | 651 | 16% | 91 | 19% | 291 | 17% | 35 |
| 1 – 2 | 37% | 1,557 | 40% | 228 | 32% | 487 | 38% | 81 |
| 3 – 4 | 25% | 1069 | 29% | 163 | 27% | 402 | 27% | 57 |
| > 5 | 23% | 967 | 15% | 84 | 22% | 329 | 18% | 39 |
| Mortality | 20% | 842 | 9% | 51 | 17% | 253 | 4% | 9 |
| Discharge Disposition (%) | ||||||||
| Home (Self-Care) | 34% | 1436 | 41% | 233 | 38% | 571 | 53% | 112 |
| Home (Health Care) | 22% | 918 | 23% | 132 | 22% | 336 | 21% | 45 |
| SNF-Medicare Approved | 17% | 716 | 19% | 109 | 15% | 223 | 14% | 29 |
| Expired | 20% | 842 | 9% | 51 | 17% | 253 | 4% | 9 |
| Other | 8% | 332 | 7% | 41 | 8% | 126 | 8% | 17 |
| Hospital LOS | ||||||||
| Range | - | 1 – 61 | - | 2 – 60 | ||||
| Mean ± SD | - | 8.8 ± 8.2 | - | 8.2 ± 7.4 | ||||
| (25th %, Median, 75th %) | - | (4.0, 6.0, 10.0) | - | (4.0, 6.0, 10.0) | ||||
| 30 Day readmission (%) | 19% | 789 | 19% | 108 | 16% | 224 | 13% | 28 |
4.1 Controlling for Confounders
Propensity score matching and bootstrapping were applied to account for differences in baseline characteristics and minimize their ability to act as confounders. After matching on propensity scores, the absolute value of standardized difference in means between post-implementation phase and pre-implementation phase was less than 0.1 for all covariates in both the intervention and non-intervention groups, with the exception of sex in the intervention group (0.104). This has been used as a threshold to indicate a negligible difference between treatment groups [27]. In the raw data set, the maximum absolute standardized difference in mean among all covariates was 0.161 (for severe sepsis diagnosis) for the intervention group and 0.100 (for sepsis diagnosis) for the non-intervention group. After matching, the maximum absolute standardized difference in mean was reduced to 0.104 (for sex) for the intervention group and 0.060 (for sex) for the non-intervention group.
4.2 Intervention Effect on Outcomes
The overall effect of the multi-focal sepsis intervention with electronic surveillance on various outcomes was evaluated for all patients as well as those in the High Risk and Low Risk subgroups for illness severity. The results of our bootstrapped PSM analyses are shown for outcomes of mortality rate (▶ Figure 2), home discharge rate (▶ Figure 3), 30 day readmission rate (▶ Figure 4), and total hospital length of stay (▶ Figure 5). ▶ Table 3 provides more detail for each of the figures. There is a positive effect over time, indicating that outcomes improved in the post-implementation phase over the pre-implementation phase for patients on both intervention and non-intervention units. The rates of home discharge and 30 day readmissions both have modest signals of improvement in the post-implementation phase, each with only one quartile of the bootstrap results overlapping with a difference in rates of 0. The average hospital length of stay is the least likely to be improved in the post-implementation phase for the non-intervention group. The 95% CI for improvement in 30 day readmission rate in the post-implementation phase for the intervention group is non-overlapping with zero (-15.5%, –0.4%), and the improvement appears to be particularly focused on the High Risk group of patients. Less improvement is observed in the Low Risk group, which experienced a similar change in readmission rates compared to what was observed in the non-intervention group. For both the home discharge rate and total hospital length of stay, the intervention group experienced more positive changes during the post-implementation phase than did the non-intervention group both overall and for the High Risk group.
Fig. 2.
Comparison in percentage point difference in mortality rates
Fig. 3.
Comparison in percentage point difference in home discharge rates
Fig. 4.
Comparison in percentage point difference 30 day readmission rates
Fig. 5.
Comparison of difference in means for total hospital length of stay
Table 3.
Summary of Results Measuring Outcome Improvements between Pre- and Post-Implementation Phases
| Intervention Units | Non-intervention Units | |||||||
|---|---|---|---|---|---|---|---|---|
| Mortality Rate | Home Discharge Rate | Thirty Day Readmission Rate | Total Hospital Length of Stay | Mortality Rate | Home Discharge Rate | Thirty Day Readmission Rate | Total Hospital Length of Stay | |
| All Patients | ||||||||
| Rates/Means (Raw) | ||||||||
| Post-Implementation | 4.2% | 78.5% | 13.2% | 8.2 | 16.8% | 73.1% | 16.2% | 10.2 |
| Pre-Implementation | 9.0% | 71.3% | 19.1% | 8.8 | 19.8% | 69.7% | 18.6% | 10.3 |
| Rates/Means (w/ PSM) | ||||||||
| Post-Implementation | 4.2% | 78.5% | 13.2% | 8.2 | 16.8% | 73.1% | 16.2% | 10.2 |
| Pre-Implementation | 10.8% | 71.0% | 19.8% | 9.1 | 19.7% | 72.0% | 17.8% | 10.4 |
| Difference of Means / Percentage Point Difference of Rates (w/ PSM) | ||||||||
| Difference | -6.6% | 7.5% | -6.6% | -0.9 | -2.9% | 1.1% | -1.6% | -0.3 |
| 95% Confidence Interval | (-10.8%,-1.4%) | (-1.5%, 15.0%) | (-15.5%,-0.4%) | (-2.3,0.9) | (-6.0%, -0.6%) | (-1.3%, 6.1%) | (-4.2%, 1.0%) | (-1.1,0.7) |
| High Risk Patients | ||||||||
| Rates/Means (Raw) | ||||||||
| Post-Implementation | 7.5% | 76.6% | 12.5% | 9.3 | 23.4% | 67.8% | 18.5% | 11.4 |
| Pre-Implementation | 14.0% | 64.5% | 20.3% | 10.4 | 27.0% | 63.9% | 20.8% | 11.4 |
| Rates/Means (w/ PSM) | ||||||||
| Post-Implementation | 7.5% | 76.6% | 12.5% | 9.3 | 23.4% | 67.8% | 18.5% | 11.4 |
| Pre-Implementation | 14.2% | 66.7% | 22.5% | 10.6 | 26.6% | 63.2% | 19.3% | 11.4 |
| Difference of Means / Percentage Point Difference of Rates (w/ PSM) | ||||||||
| Difference | -6.7% | 9.9% | -10.0% | -1.3 | -3.3% | 4.6% | -0.8% | -0.03 |
| 95% Confidence Interval | (-15.8%, 0.9%) | (-0.9%, 23.0%) | (-18.8%, 0.0%) | (-3.4,1.3) | (-7.0%, 0.7%) | (-0.6%, 8.7%) | (-4.9%, 2.0%) | (-1.1,1.2) |
| Low Risk Patients | ||||||||
| Rates/Means (Raw) | ||||||||
| Post-Implementation | 0.0% | 80.9% | 14.1% | 6.7 | 3.6% | 81.7% | 11.5% | 7.6 |
| Pre-Implementation | 3.4% | 78.1% | 17.7% | 7.0 | 6.7% | 78.0% | 14.6% | 8.2 |
| Rates/Means (w/ PSM) | ||||||||
| Post-Implementation | 0.0% | 80.9% | 14.1% | 6.7 | 3.6% | 81.7% | 11.5% | 7.6 |
| Pre-Implementation | 4.3% | 84.3% | 22.8% | 5.3 | 5.6% | 81.3% | 14.1% | 8.4 |
| Difference of Means / Percentage Point Difference of Rates (w/ PSM) | ||||||||
| Difference | -4.3% | -3.4% | -8.7% | 1.4 | -2.0% | 0.4% | -2.6% | -0.8 |
| 95% Confidence Interval | (-7.4%, 0.0%) | (-12.0%, 10.6%) | (-17.7%, 5.6%) | (-1.9,1.7) | (-4.8%, 0.4%) | (-4.4%, 5.4%) | (-6.7%, 1.6%) | (-2.1,0.6) |
Both the intervention and non-intervention groups demonstrate improvements in mortality rate in the post-implementation phase, with 95% confidence intervals (CI) that are non-overlapping with zero. In the intervention group, there is a slightly larger positive effect on mortality rate than was observed in the non-intervention group (-6.6% difference in rate as opposed to -2.9%). The High Risk patient subgroup appears to be particularly benefitted. For these patients, there is a difference in mortality rate of -6.7% between post-implementation phase and pre-implementation phase for the intervention group. In contrast, over the same period for the non-intervention group there was only a -3.3% difference in mortality rate.
4.3 Performance of Sepsis Alert System
To determine performance of the sepsis alert system, medical charts from 771 patients in the intervention group during the post-implementation phase were manually reviewed to determine the presence of sepsis and its severity (▶ Table 4). There was an overall agreement of 96% for the sepsis diagnosis and after discussion with the original investigators; there were only three diagnoses that were changed. Overall sensitivity for the sepsis alert system was 95.20%, with specificity of 82.04% and PPV of 50.64%. Notably, the only false negatives were for patients that were categorized as “sepsis” or “possible severe sepsis.” In other words, an alert was issued for all cases of “severe sepsis” and “septic shock,” giving a sensitivity of 100% for such cases.
Table 4.
Sensitivity Analysis for All Patients and by Sepsis Severity
| Chart Review | ||||
| WK Alert | No Sepsis | Sepsis | Severe Sepsis/Septic Shock | Total |
| No | 530 | 6 | 0 | 536 |
| Yes | 116 | 96 | 23 | 235 |
| 646 | 102 | 23 | 771 | |
| All Patients | ||||
| Numerator | Denominator | Value | ||
| Sensitivity/Recall | 119 | 125 | 95.2% | |
| Specificity | 530 | 646 | 82.0% | |
| PPV / Precision* | 119 | 235 | 50.6% | |
| Sepsis Only Patients | ||||
| Numerator | Denominator | Value | ||
| Sensitivity/Recall | 96 | 102 | 94.1% | |
| Specificity | 530 | 646 | 82.0% | |
| Severe Sepsis/Septic Shock Patients | ||||
| Numerator | Denominator | Value | ||
| Sensitivity/Recall | 23 | 23 | 100.0% | |
| Specificity | 530 | 646 | 82.0% | |
5. Discussion
The purpose of our study was to conduct an independent analysis of data from a multi-focal intervention for early detection of sepsis that included implementation of change management strategies, electronic surveillance for sepsis, and evidence based point of care alerting using the POC AdvisorTM application [17]. The percentage point differences in rates or means for outcomes between the pre- and post-implementation phases were compared for intervention and non-intervention patients. In our study, we used PSM in conjunction with bootstrapping to balance covariates and develop confidence intervals, thus attributing differences in rates/ means for outcomes to the intervention which included the electronic sepsis surveillance. Both the intervention and non-intervention group of patients implemented a multi-focal approach for early detection and treatment of sepsis; however, the intervention units also included an electronic surveillance system. Both intervention and non-intervention units showed a positive effect over time for outcomes, however, intervention patients improved more than non-intervention patients. The improvement in both groups is likely due to a number of factors: all patients received the multi-focal sepsis intervention except for electronic sepsis surveillance and better education and awareness for all providers likely occurred over time.
We compared mortality rates between the pre- and post-implementation phases in several ways. The absolute rates of mortality for all patients on the intervention units were 9.0% in the pre-implementation phase vs. 4.2% in the post-implementation phase, for a difference in rate of –4.8%. When using PSM to control confounders the difference in rates was –6.6%. Patients in the High Risk intervention group had the greatest improvement in death rates (-6.7%) but not those in the Low Risk intervention group. It is likely that the intervention is most useful for High Risk patients. This finding of sicker patients benefiting more from the intervention is similar to a conclusion in a meta-analysis of observational studies that implementing a protocolized intervention may benefit severely ill patients the most [6]. We also compared non-intervention patients between the pre-implementation and post-implementation periods. For patients in the non-intervention group, there was a decrease in mortality during the post- vs pre-implementation period (-2.9%). This possibly is due to the multi-focal sepsis intervention with the exception of the electronic surveillance, demonstrating a maturation effect over time. However, the change in non-intervention patients is less than that of the intervention group. One might argue that if the non-intervention units had also excluded patients in ICUs and those with limited care that mortality might be even lower. However, the opposite could also be posited that patients receiving care in the ICU likely benefited from close surveillance resulting in fewer deaths. More research is needed to address this issue. Additionally, there could be other factors not measured that account for differences in outcomes.
We evaluated additional outcomes of hospital length of stay, 30 day readmission, and discharge to home. There was a trend toward a decrease in total hospital length of stay, though, this was minimal with a decrease of .9 days. There was little difference for High and Low Risk patients and confidence intervals crossed 0. The non-intervention units also had a slight decrease in length of stay, but less than the intervention units. Likely other factors affected length of stay, such as reimbursement models, availability of support after discharge, or provider preference. There were inconsistent results reported in previous studies when examining length of stay [28, 29]. When using PSM analyses, we found that the 30 day readmission rate was lower in the post- vs the pre-implementation phase for intervention patients (-6.6%). Furthermore, patients in the High Risk group benefited the most (-10.0%). We also analyzed discharge to home with a 7.5% increase in home discharge from pre- to post-implementation for the intervention group, and a 1.1% increase for the non-intervention group. The confidence intervals for improvement in home discharge rates and hospital length of stay overlap zero for both the High Risk and Low Risk groups for both the intervention and non-intervention groups. There is limited research regarding discharge to home and with the emphasis on newer methods of bundled payment to save costs and improve outcomes, this is a fruitful area for future research.
The sensitivity and specificity of alerts has been a challenge when implementing an electronic reminding system for early detection of sepsis [14–16]. A unique feature of this study was examining the accuracy of alerts on medical-surgical units with patients at risk for varying levels of sepsis severity vs. previous studies conducted in the emergency department or ICUs or just with patients experiencing severe sepsis/ septic shock [14–16]. We found that sensitivity was 95.1% and specificity was 82% with the highest sensitivity for severely ill patients (100%). The PPV was 50.64%, indicating that approximately half of all alerts issued reflect a true positive, which is an improvement over previous sepsis alert systems. While the sensitivity and specificity are excellent, the PPV rate could be improved. For example, timely documentation can improve the ability of the surveillance system to provide increasingly timely alerts. Additionally, while there are more than 100 rules to improve alerts based on co-morbid conditions and medications, continuous investigation of over alerting and missed alerts can enhance the overall PPV rate. Overall, use of computers to remind, alert, and implement interventions for early detection and management is consistently associated with better compliance and outcomes.
We compared results of our secondary analysis with the primary analysis for evaluating a multi-focal intervention, including the POC Advisor, for early detection of sepsis and improvement of outcomes [17]. The direction of outcome improvement and sensitivity/ specificity of the alerting system in both studies are similar, but there are differences in how outcome rates were calculated, how the data were analyzed, and results reported. In the original study, mortality rates were calculated as the number of deaths per 1000 cases and comparing the post- versus the pre-implementation period yielded a mortality rate reduction of 53% (P = 0.03). In our study we calculated mortality as the percentage point difference of rates; after using PSM and bootstrapping to adjust for confounders, the difference in percentage points for mortality was -6.6% (95% CI = (-10.8%, –1.4%)). If we used a metric similar to the original study (number of deaths per 1000 cases), we found a greater difference in the PSM adjusted death rate of 61% rather than the reported 53%. Hospital length of stay in the original analysis was defined as the time on the study unit which was considered as the time in which the intervention could influence the outcomes compared to our study which used the total length of stay. In the original analysis, there was a decrease of 6.72 to 6.68 days (a difference of 0.04 days, P > 0.05), whereas we found a difference of -0.896 days. The 30 day readmission was reduced by 5.87% (P = 0.05) compared with our findings of 6.6% (95% CI = (-15.5%, –0.4%)). Both studies had sensitivity rates of 95% with a specificity of 82% for detecting sepsis in the intervention group during the study phase. We went beyond the original study by also including additional analyses; we compared non-intervention units pre- and post-implementation as well as a sub-analysis of High Risk and Low Risk groups.
6. Limitations
There are a number of limitations to this secondary analysis of data from a previous study. One limitation is a small sample size from a single institution. The lack of comparability between intervention and non-intervention unit patients limited the analysis. A comparison was made between pre-and post-implementation phases for intervention and non-intervention patients; however, the period of the study phase was only ten months long. Levy and colleagues [8] noted that the longer interventions are implemented, the more likely outcomes will improve. Additionally, we recommend a future analysis be conducted for a longer time period and include more than one pre- and post-intervention time point. A major limitation in this study was the lack of more granular data for PSM analyses; future research using more granular data could provide further insights about development of rules for the surveillance system and targeting of interventions to subpopulations. While future studies should include both additional institutions and longer study periods, the analysis presented indicates that the intervention shows promise for improving patient outcomes.
7. Conclusion
In summary, the study analyzed in this paper was a multi-focal intervention for early detection of sepsis that included implementation of change management strategies, electronic surveillance for sepsis, and evidence based point of care alerting using the POC AdvisorTM Application. Our analyses shows improved outcomes with an adjusted outcome of almost 7 percentage point for mortality and 30 day readmission. The High Risk sepsis patients benefitted more from the intervention with weaker support for Low Risk patients. There was improvement over time across the hospital for patients on the intervention and non-intervention units with more improvement for sicker patients. Patients on intervention units with electronic surveillance has better outcomes; however, due to differences in exclusion criteria and types of units, further study is needed to draw a direct relationship between the electronic surveillance system and outcomes.
Funding Statement
This study was funded by Wolters Kluwer
Footnotes
Clinical Relevance
Sepsis continues to escalate and affects outcomes, particularly for severely ill patients. The use of an evidence-based electronic surveillance and alerting system, combined with change management strategies resulted in a trend toward improved outcomes compared with pre-implementation subjects. A high sensitivity and specificity for alerts prevents alert fatigue and can contribute to more prompt action by staff.
Conflict of Interest
None of the listed authors have any financial or personal relationships with other people or organizations that may inappropriately influence or bias the objectivity of submitted content and /or its acceptance of publication in this journal.
Human Subjects Protections
Both the Huntsville Hospital and the University of Minnesota’s Institutional Review Board (IRB) determined that approval was not required since this was a quality improvement project and the data were de-identified.
References
- 1.Singer M, Deutschman CS, Seymour C W, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche JD, Coopersmith CM, Hotchkiss RS, Levy MM, Marshall JC, Martin GS, Opal SM, Rubenfeld GD, van der Poll T, Vincent JL, Angus DC. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016; 315(8): 801-810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A. Inpatient care for septicemia or sepsis: A challenge for patients and hospitals. NCHS Data Brief 2011; 62: 1–8. [PubMed] [Google Scholar]
- 3.Fleischmann C, Thomas-Rueddel DO, Hartmann M, Hartog CS, Welte T, Heublein S, Heublein S, Dennler U, Reinhart K. Hospital incidence and mortality rates of sepsis. Dtsch Arztebl Int 2016; 113(10): 159-166. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Yende S, Austin S, Rhodes A, Finfer S, Opal S, Thompson T, Bozza FA, LaRosa SP, Ranieri VM, Angus DC. Long-term quality of life among survivors of severe sepsis: Analyses of two international trials. Crit Care Med 2016; 44(8): 1461-1467. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Shiramizo SC, Marra AR, Durao MS, Paes AT, Edmond MB, Pavaodos Santos OF. Decreasing mortality in severe sepsis and septic shock patients by implementing a sepsis bundle in a hospital setting. PLoS One 2011; 6(11): e26790. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Damiani E, Donati A, Serafini G, Rinaldi L, Adrario E, Pelaia P, Busani S, Girardis M. Effect of performance improvement programs on compliance with sepsis bundles and mortality: A systematic review and meta-analysis of observational studies. PLoS One 2015; 10(5): e0125827. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, Sevransky JE, Sprung CL, Douglas IS, Jaeschke R, Osborn TM, Nunnally ME, Townsend SR, Reinhart K, Kleinpell RM, Angus DC, Deutschman CS, Machado FR, Rubenfeld GD, Webb S, Beale RJ, Vincent JL, Moreno R, Surviving Sepsis Campaign Guidelines Committee including The Pediatric Subgroup Surviving sepsis campaign: International guidelines for management of severe sepsis and septic shock, 2012. Intensive Care Med 2013; 39(2):165–228. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Levy MM, Rhodes A, Phillips GS, Townsend SR, Schorr CA, Beale R, Osborn T, Lemeshow S, Chiche JD, Artigas A, Dellinger R P. Surviving sepsis campaign: Association between performance metrics and outcomes in a 7.5-year study. Crit Care Med 2015; 43(1):3–12. [DOI] [PubMed] [Google Scholar]
- 9.Levy MM, Dellinger RP, Townsend SR, Linde-Zwirble WT, Marshall JC, Bion J, Schorr C, Artigas A, Ramsay G, Beale R, Parker MM, Gerlach H, Reinhart K, Silva E, Harvey M, Regan S, Angus DC. The surviving sepsis campaign: Results of an international guideline-based performance improvement program targeting severe sepsis. Intensive Care Med 2010; 36(2): 222-231. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Palleschi MT, Sirianni S, O‘Connor N, Dunn D, Hasenau SM. An interprofessional process to improve early identification and treatment for sepsis. J Healthc Qual 2014; 36(4):23–31. [DOI] [PubMed] [Google Scholar]
- 11.Giuliano KK, Lecardo M, Staul L. Impact of protocol watch on compliance with the surviving sepsis campaign. Am J Crit Care 2011; 20(4): 313-321. [DOI] [PubMed] [Google Scholar]
- 12.McKinley BA, Moore LJ, Sucher JF, Todd SR, Turner KL, Valdivia A, Sailors RM, Moore FA. Computer protocol facilitates evidence-based care of sepsis in the surgical intensive care unit. J Trauma 2011; 70(5): 1153–1166; discussion 1166-1167. [DOI] [PubMed] [Google Scholar]
- 13.Powell KK, Fowler RJ. Driving sepsis mortality down: Emergency department and critical care partnerships. Crit Care Nurs Clin North Am 2014; 26(4): 487-498. [DOI] [PubMed] [Google Scholar]
- 14.Herasevich V, Pieper MS, Pulido J, Gajic O. Enrollment into a time sensitive clinical study in the critical care setting: Results from computerized septic shock sniffer implementation. J Am Med Inform Assoc 2011; 18(5): 639-644. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Umscheid CA, Betesh J, VanZandbergen C, Hanish A, Tait G, Mikkelsen ME, French B, Fuchs BD. Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015; 10(1):26–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Alsolamy S, Al Salamah M, Al Thagafi M, Al-Dorzi HM, Marini AM, Aljerian N, Al-Enezi F, Al-Hunaidi F, Mahmoud AM, Alamry A, Arabi YM. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak 2014; 14: 105; doi:10.1186/s12911–014–0105–7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. JAMIA 2016; ocw056. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Neviere R, Parsons PE, Finlay T. Sepsis and the systemic inflammatory response syndrome: definitions, epidemiology, and prognosis. Parsons PE, Ed. UpToDate. September, 19, 2016. www.uptodate.com, Last accessed 9/27/2016 [Google Scholar]
- 19.Manaktala S, Claypool SR. Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. Journal of the American Medical Informatics Association. 2016. May 25:ocw056 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Iwashyna TJ, Odden A, Rohde J, Bonham C, Kuhn L, Malani P, Chen L, Flanders S. Identifying patients with severe sepsis using administrative claims: Patient-level validation of the angus implementation of the international consensus conference definition of severe sepsis. Med Care 2014; 52(6): e39-e43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the united states: Analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001; 29(7): 1303-1310. [DOI] [PubMed] [Google Scholar]
- 22.Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation 1987; 40(5): 373-383. [DOI] [PubMed] [Google Scholar]
- 23.Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, Saunders LD, Beck CA, Feasby TE, Ghali WA. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005; 43(11): 1130-1139. [DOI] [PubMed] [Google Scholar]
- 24.Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46(3):399–424. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.R Development Core Team (2011), R: A Language and Environment for Statistical Computing. Vienna, Austria: : the R Foundation for Statistical Computing; ISBN: 3–900051–07–0. Available online at http://www.R-project.org/. Last accessed 9/27/16 [Google Scholar]
- 26.Ho DE, Imai K, King G, Stuart EA. MatchIt: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software 2011; 42(8):1–21. [Google Scholar]
- 27.Normand ST, Landrum MB, Guadagnoli E, Ayanian JZ, Ryan TJ, Cleary PD, McNeil BJ. Validating recommendations for coronary angiography following acute myocardial infarction in the elderly: A matched analysis using propensity scores. J Clin Epidemiol 2001; 54(4): 387-398. [DOI] [PubMed] [Google Scholar]
- 28.Angus DC, Barnato AE, Bell D, Bellomo R, Chong CR, Coats TJ, Davies A, Delaney A, Harrison DA, Holdgate A, Howe B, Huang DT, Iwashyna T, Kellum JA, Peake SL, Pike F, Reade MC, Rowan KM, Singer M, Webb SA, Weissfeld LA, Yealy DM, Young JD. A systematic review and meta-analysis of early goal-directed therapy for septic shock: The ARISE, ProCESS and ProMISe investigators. Intensive Care Med 2015; 41(9): 1549-1560. [DOI] [PubMed] [Google Scholar]
- 29.ARISE Investigators ANZICS Clinical Trials Group. Peake SL, Delaney A, Bailey M, Bellomo R, Cameron PA, Cooper DJ, Higgins AM, Holdgate A, Howe BD, Webb SA, Williams P. Goal-directed resuscitation for patients with early septic shock. N Engl J Med 2014; 371(16): 1496-1506. [DOI] [PubMed] [Google Scholar]