A Nurse-Driven Screening Tool for the Early Identification of Sepsis in an Intermediate Care Unit Setting

Abstract

BACKGROUND

Use of a screening tool as a decision support mechanism for early detection of sepsis has been widely advocated, yet studies validating tool performance are scarce, especially in non–intensive care unit settings.

METHODS

For this pilot study we prospectively screened consecutive patients admitted to a medical/surgical intermediate care unit at an academic medical center over a 1-month period and retrospectively analyzed their clinical data. Patients were screened with a 3-tiered, paper-based, nurse-driven sepsis assessment tool every 8 hours. For patients screening positive for sepsis or severe sepsis, the primary treatment team was notified and the team’s clinical actions were recorded. Results of the screening test were then compared to patient International Classification of Diseases, Ninth Revision (ICD-9) codes for sepsis, severe sepsis, and septic shock identified during the study time period, and performance of the screening test was assessed.

RESULTS

A total of 2143 screening tests were completed in 245 patients (169 surgical, 76 medical). ICD-9 codes confirmed sepsis incidence was 9%. Of the 39 patients who screened positive, 51% were positive for sepsis, and 49% screened positive for severe sepsis. Screening tool sensitivity and specificity were 95% and 92%, respectively. Negative predictive value was 99% and positive predictive value was 54%. Overall test accuracy was 92%. There was no statistically significant difference in tool performance between medical and surgical patients.

CONCLUSIONS

A simple screening tool for sepsis utilized as part of nursing assessment may be a useful way of identifying early sepsis in both medical and surgical patients in an intermediate care unit setting.

Sepsis remains a significant healthcare burden and is the sixth most common reason for hospitalization in the United States. For patients presenting with severe sepsis, mortality rates are approximately 30%,^1,2 and sepsis remains the most expensive reason for hospitalization. In 2009, septicemia accounted for nearly $15.4 billion in aggregate hospital costs.²

Early identification of sepsis and the timely implementation of goal-directed therapy significantly decrease sepsis-related mortality and are cost-effective,^3–5 highlighting the need for new clinical strategies to aid in early diagnosis. To date, most studies have focused on the screening and management of sepsis in the emergency department and intensive care unit (ICU),^6,7 and less is known about the benefits of screening in non-ICU settings. In the non-ICU setting, conditions may go unrecognized and treatments delayed. Evidence suggests that patients diagnosed with severe sepsis in the non-ICU setting are almost twice as likely to die as those diagnosed in an emergency department.^8,9

Application of a sepsis screening tool to both medical and surgical patients poses an additional challenge that may impact the screen’s performance. The specificity may be compromised by noninfectious causes of systemic inflammatory response syndrome (SIRS) commonly seen in the postsurgical patient. For example, the tachycardia and fever often seen in the postoperative patient are sufficient to qualify for SIRS, making the diagnosis of sepsis more challenging. The purpose of this study was to examine the performance of a nurse-driven, simple sepsis screening tool in a mixed medical and surgical non-ICU setting.

METHODS

Setting

This was an observational pilot study of prospectively screened patients admitted to a 26-bed medical/surgical intermediate care unit with telemetry monitoring in a 613-bed university tertiary referral hospital over a 1-month time period. The surgical patient population of this floor consisted of cardiothoracic (50%), general (24%), and vascular surgery (17%) patients as well as a small number of trauma (7%) patients. The medical patient population admitted to this unit included pretransplant and complex medical patients requiring telemetry monitoring. Though the incidence of sepsis specific to this unit was unknown prior to the study, after an analysis of discharges the study team surmised there would be sufficient volume for testing of a nurse-based screening tool.

Nurse Education

Registered nurses (RNs) working on the study unit had an average of 5 to 7 years of experience. The all-RN unit was staffed predominantly at a 1:3 RN to patient ratio. RNs were supported by a clinical nurse specialist (CNS) and clinical educator (CE) RN who provided regular ongoing education about infection prevention and identification of common conditions that are seen on the unit.

In the 6 months prior to our sepsis screening initiative, nursing staff had been given more than 8 hours of education on infection- and sepsis-related topics in 15- to 20-minute blocks of time. This dedicated education took place during the nurses’ shift in groups of 2 to 3, and was run by the CNS, assistant nurse manager, and CE RN. Nurses were also encouraged to attend an optional 8-hour sepsis continuing medical education (CME) program. Approximately 20% of the nurses on the study unit attended.

Just prior to the pilot study, nursing staff completed a 1-hour refresher self-study module on severe sepsis stressing the importance of early identification. There was also a “training month” prior to the actual data collection time frame, where unit core trainers (RNs) or “champions” who had attended the optional 8-hour sepsis CME conducted 1:1 follow-up with each RN, reviewing at least 1 of their screens to validate understanding of screening concepts. Each RN was checked off after correctly completing a screen. During the study, unit educators and the CNS provided additional on-unit in-service training with screening tool completion instructions and advice on how to incorporate the tool into the RN’s current assessment workflow. In addition, the charge nurses were asked to review the screens collected each shift and validate any that may have seemed inconsistent with the RN’s verbal report of the patient’s status.

The university’s institutional review board notice of determination waived review for this study because it was classified as quality improvement.

Screening Tool

A sepsis screening tool was developed as part of a broader initiative to improve sepsis-related morbidity and mortality at our hospital. The screening tool was adapted from the severe sepsis screening tool created by the Surviving Sepsis Campaign and Institute for Healthcare Improvement,¹⁰ and consisted of a simple 3-tiered paper-based screening assessment that was to be completed by the bedside RN (Figure 1). RNs on the pilot medical/surgical intermediate care unit performed the screening assessment with their regular patient assessment at the beginning of each shift.

FIG. 1.

Paper-based sepsis screening tool. Adapted from Evaluation for Severe Sepsis Screening Tool from the Surviving Sepsis Campaign and Institute for Healthcare.¹⁰ Abbreviations: RN, Registered Nurse; Temp, Temperature; HR, Heart Rate; BPM, beats per minute; RR, respiratory rate; PaCO2, partial pressure of carbon dioxide; WBC, White Blood Cells; SIRS, systemic inflammatory response; MAP, mean arterial blood pressure; UO, urine output; INR, international normalized ratio; PTT, Partial Thromboplastin Time.

The first tier of the tool screened for the presence of SIRS. Positive parameters included heart rate >90, temperature >38°C or <36°C, white blood cell count >12,000 or <4000 or >10% bands, and/or respiratory rate >20 or partial pressure of carbon dioxide (PaCO₂) <32 mm Hg. To decrease the number of false-positive screens in patients whose abnormal vitals could already be attributed to a condition other than sepsis, these symptoms were only scored if they had emerged within the previous 8 hours.

If patients met ≥2 SIRS criteria, the nurse would move to the second tier of the tool, which involved consideration of possible infection as a contributor to a patient’s clinical condition as well as a source of infection. If infection was not suspected, further screening was terminated. If infection was suspected, the patient then met criteria for a positive sepsis screen, and a third tier of screening involving assessment of organ dysfunction was initiated.

If the patient screened positive for sepsis (≥2 SIRS and suspicion for new infection) or severe sepsis (sepsis with end-organ dysfunction), nurses were instructed to document this in the patient’s electronic medical record (EMR) and call the primary team to initiate actions following the hospital-wide sepsis guidelines. Any subsequent actions were recorded in the patient’s EMR.

Data Collection

Completed sepsis screening forms during the month of October 2010 were reviewed by the authors (E.G., L.S., and P.M.). Data including age, gender, International Classification of Diseases, Ninth Revision (ICD-9) admission and discharge diagnoses, vital signs, lab results, clinical interventions, and documented clinical decision processes by healthcare staff were collected on patients with a positive screen or those who did not screen positive but had an ICD-9 code for sepsis, severe sepsis, or septic shock during their hospitalization or at discharge. We also collected demographic and clinical data for a random sample of patients who consistently screened negative for sepsis.

Performance Measurement

The sensitivity and specificity of the screening tool was determined by identifying true-positive, false-positive, true-negative, and false-negative results and calculating accordingly using a 2 × 2 contingency table. True positives were defined as cases where patients screened positive for sepsis and had a documented diagnosis of sepsis in their EMR within 24 hours of the positive screening or had an ICD-9 billing code for sepsis. False-positive cases were those in which patients screened positive for sepsis but did not have a diagnosis of sepsis by manual chart review nor was there an ICD-9 code for sepsis for their hospital stay. True-negative cases were those where patients screened negative and did not have an ICD-9 code for sepsis. False negatives were cases where patients consistently screened negative for sepsis but had an ICD-9 code for sepsis.

Clinical Activities

To examine the impact of a positive sepsis screen on subsequent clinical action, we assessed the frequency with which a treatment or diagnostic workup was initiated after a positive screen and compared this to clinical activity initiated after a negative screen. Specifically, the patient’s EMR was reviewed for actions including measurement of lactate, blood cultures, administration of broad spectrum antibiotics, administration of fluid boluses, or consultation with or transfer to the ICU. These actions were chosen because they are part of the Surviving Sepsis Bundle, which has been demonstrated to improve mortality rates after diagnosis of severe sepsis or septic shock,^11,12 and can be done outside of an ICU setting. Because screening was done every 8 hours, clinical activity was only attributed to a positive or negative sepsis screen if it occurred within 8 hours of the screening result. Patients were excluded if there were missing data points that precluded full analysis of their clinical course.

Statistical Analysis

To compare the performance of the screening tool between surgical and medical patients, we calculated 95% confidence intervals of screening test sensitivity and specificity. To test if performance was significantly different between these groups, we performed a nonparametric, 2-sided, 2-sample test of proportions. Though similar to a χ² test, the 2-sided test of proportions allowed us to determine if there was a directional difference in test performance (ie, Does the screening tool perform better or worse in a certain patient group?). We also used the test of proportions to compare differences in the proportion of patients receiving sepsis-related interventions before and after a positive or negative screening result. For comparisons of demographic variables we used nonparametric tests including the χ² test for categorical variables and the Kruskal-Wallis test for continuous variables. We used SAS 9.3 (SAS Institute Inc., Cary, NC) to perform our analyses.

RESULTS

Over a 1-month time period, 2143 screens were completed on 245 patients (169 surgical, 76 medical). The overall incidence of sepsis on the treatment unit during this time period was 9%. Surgical patients had an 8.9% incidence of sepsis, and medical patients had an incidence of 9.2%.

Screening tool performance is presented in Table 1. The screening tool had 95.5% sensitivity and 91.9% specificity, with no significant differences in performance between surgical and medical patients. The overall negative predictive value was 99.5%, also with comparable performance in both surgical and medical patients (P = 0.89). The overall positive predictive value (PPV) was 70% in medical patients and 48% in surgical patients (P = 0.12). Screening tool accuracy for medical and surgical patients was 92%.

TABLE 1.

Comparison of Screening Tool Performance in Surgical and Medical Patients

	Overall, N = 245 (95% CI)	Surgery, N = 169 (95% CI)	Medicine, N = 76 (95% CI)	P Value*
Sensitivity	95.5% (75%–99.7%)	93% (66%–99.6%)	100% (56%–100%)	0.17
Specificity	91.9% (87%–95%)	90% (84%–94%)	95% (87%–99%)	0.48
NPV	99.5% (81%–100%)	99.3% (71%–100%)	100% (67%–100%)	0.89
PPV	53.8% (39%–70%)	48% (23%–73%)	70% (30%–100%)	0.12
LR+	11.8	9.3	20
LR−	0.05	0.08	0
Confirmed patient diagnosis, overall†
	Sepsis		No sepsis
Screen positive	21 (TP)		18 (FP)
Screen negative	1 (FN)		205 (TN)
Confirmed patient diagnosis, medicine†
	Sepsis		No sepsis
Screen positive	7 (TP)		3 (FP)
Screen negative	0 (FN)		66 (TN)
Confirmed patient diagnosis, surgery†
	Sepsis		No sepsis
Screen positive	14 (TP)		15 (FP)
Screen negative	1 (FN)		139 (TN)

NOTE: Abbreviations: CI, confidence interval; FN, false negative; FP, false positive; LR+, positive likelihood ratio; LR−, negative likelihood ratio; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive.

^*Comparing medicine to surgery patient test performance.

^†Confirmed by International Classification of Diseases, Ninth Revision code and/or medical record documentation.

Clinical Activities

Of the 39 patients who screened positive for sepsis, nurses classified 20 with sepsis and 19 with severe sepsis. Of these 39 patients, 33 were included in our descriptive analysis of the effect of positive screening results on clinical activity (3 were excluded for admission for sepsis and 3 for missing data). As a comparison, we randomly selected 30 patients of the 206 patients who screened negative for sepsis to evaluate clinical activity before and after a negative screen.

Characteristics of patients screening positive and negative for sepsis are reported in Table 2. We found no statistically significant differences in age, sex, length of hospital stay, or mortality amongst all groups.

TABLE 2.

Patient Characteristics of 33 Patients With a Positive Sepsis Screen and 30 Randomly Selected Patients With Negative Sepsis Screens

Patient Characteristics	Surgery (Positive)	Medicine (Positive)	Surgery (Negative)	Medicine (Negative)	P Value
No.	26	7	20	10
Age, y, mean	57.8 (± 16.5)	72.4 (± 16.8)	64.6 (± 19.4)	63.6 (± 16.8)	0.25
% Male (no.)	50% (13)	57% (4)	60% (12)	60% (6)	0.27
Length of stay, d, median (IQR)	9 (7–16.7)	7 (5.5–11.5)	11 (7.7–22)	8 (4–21)	0.38
No. of PODs until first positive screen, d, median (IQR)	2 (1–3)	N/A	N/A	N/A	—
% Mortality (no.)	0%	14% (1)	5% (1)	10% (1)	0.19

NOTE: Abbreviations: IQR, interquartile range; N/A, not applicable; PODs, postoperative days.

Figure 2 illustrates differences in the proportion of patients receiving a clinical action before and after a negative or positive screening test result. In the cohort of 33 patients screening positive for sepsis, clinical action after a positive screen was taken in 4 of the 7 (50%) medical patients and 11 of 26 (42%) surgical patients. In patients screening negative for sepsis we found only 1 incident in which a sepsis-related action was taken after a negative screen. In this case the patient was admitted to the ICU within 8 hours of a negative screen, though there was no explicit documentation that sepsis was the reason for this admission.

FIG. 2.

Proportion of patients receiving a sepsis-related clinical action before and after a positive or negative screening test result (N = 30 negative patients, N = 33 positive patients). Abbreviations: ICU, intensive care unit.

We compared the proportion of patients receiving sepsis-related treatment before either a negative or positive screen and found no significant difference (Table 3). We then compared the proportion of patients receiving sepsis-related actions after a positive or negative screening test result and found that the proportion of patients receiving antibiotics, blood cultures, and lactate measurement was significantly higher for patients with a positive sepsis screening result compared to those with a negative screening result (Table 3).

TABLE 3.

Comparison of the Proportion of Patients Receiving Sepsis-Related Clinical Actions Before and After a Positive or Negative Screen

Intervention and Group	Proportion	P Value
Before screening test
Antibiotics		0.066
Positive screen	45%
Negative screen	23%
Lactate		0.837
Positive screen	15%
Negative screen	13%
Blood culture		0.181
Positive screen	18%
Negative screen	17%
Fluid administration		0.564
Positive screen	6%
Negative screen	10%
ICU transfer/consult		0.337
Positive screen	3%
Negative screen	0%
After screening test
Antibiotics		0.006
Positive screen	58%
Negative screen	23%
Lactate		0.018
Positive screen	36%
Negative screen	13%
Blood Culture		0.002
Positive screen	24%
Negative screen	17%
Fluid administration		0.112
Positive screen	24%
Negative screen	10%
ICU transfer/consult		0.175
Positive screen	9%
Negative screen	3%

NOTE: Abbreviations: ICU, intensive care unit.

DISCUSSION

Improving recognition and time to treatment of sepsis in a non-ICU setting is an important step toward decreasing sepsis-related mortality. Lundberg and colleagues found that mortality rates for patients diagnosed with septic shock on a general ward were higher than for patients diagnosed in the ICU, even though ward patients were younger and healthier at baseline.⁸ For ward patients, treatment delays were most profound in initiating vasoactive therapies, and minor delays were encountered in initiating fluid resuscitation. In their international study on the impact of early goal-directed therapy guidelines, Levy and colleagues found that patients diagnosed with severe sepsis on the wards were almost twice as likely to die as patients diagnosed with sepsis in the emergency department.⁹

We are the first to report about an accurate nurse-driven SIRS-based sepsis screening protocol that is effective in the early identification of sepsis in both medical and surgical patients in an intermediate care setting. We found no significant difference in the screening tool performance between the medical and surgical cohorts. This is an important comparison given that SIRS criteria alone can be nonspecific in the postoperative population, where it is common to have hemodynamic changes, elevation of inflammatory markers, and fevers from noninfectious sources.

Our sepsis screening tool was designed in 3 tiers to improve its specificity. The first tier was based strictly on SIRS criteria (eg, tachycardia or fever), whereas the second and third tiers served to increase the specificity of the screening tool by instructing the evaluator to assess possible sources of infection and assess for objective signs of organ dysfunction. We relied heavily on the nursing staff to assess for the presence or absence of infection and believe that the educational component prior to initiating the screening protocol was vital.

EMR-based screening tools that rely purely on physiologic data have been considered for the early detection and management of sepsis, although they lack the specificity gained through the incorporation of clinical judgment.¹³ Sawyer and colleagues report using a real-time EMR-based method for early sepsis detection in non-ICU patients that is based solely on objective measures; however, their PPV was only 19.5%. The model we describe in this study is one that incorporates real-time physiologic data available from an EMR coupled with the clinical judgment of a bedside registered nurse. As our data suggest, this provides a screen that is both sensitive and specific.

It is interesting to note that in our assessment of clinical action taken 8 hours after a positive screening test (the interval after which a new screening test was performed), the rate of diagnostic workup and/or treatment for sepsis was relatively low. One reason for this could have been that the treating team had suspicion for sepsis prior to a positive screen and had already initiated clinical action. Of the 51 recorded clinical actions taken around the time of a positive screen, the majority (67%) occurred before the screening result. It is also possible that clinical action was not pursued because the treatment team disagreed with a diagnosis of sepsis. Of all the false positive screening cases, manual chart review confirmed that these patients did not have sepsis, nor did they develop sepsis during their index hospital stay. Last, we only recorded clinical actions taken within 8 hours of the first positive screen for sepsis and measured 5 very specific actions. Thus, our analysis may have missed actions taken after 8 hours or actions that differed from the 5 we chose to assess.

Even with the apparently low levels of new clinical activity after a positive screen, when compared to patients who screened negative for sepsis, a significantly higher number of patients who had a positive screen received antibiotics, had lactate measured, and had blood cultures drawn. We did not find a significant difference in the proportion of patients receiving a sepsis-related clinical action before a screening result (positive or negative), which suggests that a positive screening test may have led to increased clinical action.

A limitation of our study is its small size and that it was conducted in 1 pilot unit. Additionally, our retrospective analysis of clinical care inhibited our ability to fully understand a patient’s clinical course or retrieve missing data points. A related limitation is that we could not ascertain how often the screening tool did not identify a case of sepsis before it was clinically diagnosed. Assessing the temporal performance of our screening tool is of great interest and may be more easily performed using an electronic version of the screening tool, which is currently in development.

Using ICD-9 codes to determine the true-negative cohort is another limitation of our study. It is well documented that use of administrative data can lead to inaccurate classification of patients.¹⁴ To address this, we performed random audits of 30 test-negative patients. In doing so we did not find any errors in classification.

Although we did not find a significant difference in screening tool performance between surgical and medical patients, the PPV of the tool was lower in the surgical population (48%) compared to the medical population (70%). The lower PPV observed in surgical patients could be attributable to an overall lower incidence of sepsis in this cohort as well as possible errors in initial assessment of infection, which can be difficult in postsurgical patients. Our retrospective analysis included data from the early months of the screening protocol, a time in which nursing staff was still developing clinical acumen in identifying sepsis. However, this could have led nurses to either overestimate or underestimate the presence of infection in either patient group.

Suspicion for infection is the cornerstone definition of sepsis, and in our screening protocol nurses were charged with making this decision based on their knowledge of the patient’s clinical course and current status. Issues concerning nurses’ recognition of infection symptoms are an area of opportunity for further research and education and could aid in improving PPV. Clinical judgment could be further bolstered by adding promising laboratory tests such as C-reactive protein or procalcitonin as objective adjuncts to an initial assessment for sepsis,¹⁵ which could potentially increase screening test PPV.

CONCLUSIONS

A simple screening tool for sepsis performed by the bedside nurse can provide a means to successfully identify sepsis early and lead to more timely diagnostics and treatment in both medical and surgical patients in an intermediate care setting.