Sepsis patients in the emergency department: stratification using the Clinical Impression Score, Predisposition, Infection, Response and Organ dysfunction score or quick Sequential Organ Failure Assessment score?

Abstract

Objective

The aim of this study was to compare the stratification of sepsis patients in the emergency department (ED) for ICU admission and mortality using the Predisposition, Infection, Response and Organ dysfunction (PIRO) and quick Sequential Organ Failure Assessment (qSOFA) scores with clinical judgement assessed by the ED staff.

Patients and methods

This was a prospective observational study in the ED of a tertiary care teaching hospital. Adult nontrauma patients with suspected infection and at least two Systemic Inflammatory Response Syndrome criteria were included. The primary outcome was direct ED to ICU admission. The secondary outcomes were in-hospital, 28-day and 6-month mortality, indirect ICU admission and length of stay. Clinical judgement was recorded using the Clinical Impression Scores (CIS), appraised by a nurse and the attending physician. The PIRO and qSOFA scores were calculated from medical records.

Results

We included 193 patients: 103 presented with sepsis, 81 with severe sepsis and nine with septic shock. Fifteen patients required direct ICU admission. The CIS scores of nurse [area under the curve (AUC)=0.896] and the attending physician (AUC=0.861), in conjunction with PIRO (AUC=0.876) and qSOFA scores (AUC=0.849), predicted direct ICU admission. The CIS scores did not predict any of the mortality endpoints. The PIRO score predicted in-hospital (AUC=0.764), 28-day (AUC=0.784) and 6-month mortality (AUC=0.695). The qSOFA score also predicted in-hospital (AUC=0.823), 28-day (AUC=0.848) and 6-month mortality (AUC=0.620).

Conclusion

Clinical judgement is a fast and reliable method to stratify between ICU and general ward admission in ED patients with sepsis. The PIRO and qSOFA scores do not add value to this stratification, but perform better on the prediction of mortality. In sepsis patients, therefore, the principle of ‘treat first what kills first’ can be supplemented with ‘judge first and calculate later’.

Introduction

Time is of the essence in the treatment of sepsis; early and aggressive treatment is important to reduce mortality as indicated in recent studies 1,2. In 30–50% of patients, sepsis treatment is initiated in the emergency department (ED) 3,4. Considering that ICU capacity is limited and that not all sepsis patients will benefit from ICU admission, the main challenge that ED physicians face is to effectively stratify patients between patients requiring ICU treatment and patients who can be treated on the general ward. Incorrect stratification may result in increased morbidity and mortality, and increased length of stay 5,6.

There are multiple ways to stratify patients with sepsis, such as stratification by using scoring systems, stratification on the basis of the clinical judgement of the nurse and attending physician, or stratification on the basis of the sepsis categories defined by the Surviving Sepsis Campaign 2. However, stratification on the basis of sepsis categories is not as accurate as clinical judgement or an adequate scoring system 7. Numerous scoring systems for patients with sepsis exist, which predict sepsis-related mortality as well as sepsis severity. These include the Predisposition, Infection, Response and Organ dysfunction (PIRO) score 8, the Mortality in Emergency Department Sepsis (MEDS) score 9, the Mortality In Severe Sepsis in the Emergency Department (MISSED) score 10, the Sequential Organ Failure Assessment (SOFA) score 11 and the recent quick SOFA (qSOFA) score introduced with the new Sepsis-3 definitions 6. Of these scoring systems, the PIRO score is one of the most comprehensive scoring systems, while at the same time requiring data routinely available in the ED. Moreover, it was developed as both a staging system and to predict mortality, and is well known to stratify patients on the basis of the severity of disease and risk of mortality before ICU admission 8,12. However, it may not be the most practical bedside scoring system for the ED as it requires information that is not readily available on admission, such as biomarker levels, patient history and living situation. Thus, effective stratification can be delayed by having to wait for these details. This is a cause of concern as early ICU transfer may lead to improved patient outcomes 13. The recent qSOFA score is a simple score that could be calculated at the bedside. However, this score has not yet been validated for patients with sepsis in the ED 6. The clinical judgement of nurses and attending physicians are available at the bedside during ED admission and do not lead to the aforementioned delays. With this in mind, our aim was to compare the stratification for ICU admission and mortality using the PIRO and qSOFA scoring systems with clinical judgement assessed by the ED staff. We hypothesized that clinical judgement assessed by the ED staff would be as accurate for predicting ICU admission and mortality as the PIRO and qSOFA scoring systems for patients with sepsis in the ED.

Patients and methods

Study design and setting

A prospective observational study was carried out in the ED of the University Medical Center Groningen, a tertiary care teaching hospital with over 34 000 ED visits annually. The study was approved by the Medical Ethical Committee of the University Medical Center Groningen, the Netherlands (METc 2013/297; METc 2012.177). Written informed consent was obtained from all patients included in the study.

Study population and protocol

Adult nontrauma patients visiting the ED between 8 a.m. and 6 p.m. with suspected infection or sepsis were screened for inclusion. Inclusion criteria included patients of 18 years and older age, suspected or confirmed infection and two or more Systemic Inflammatory Response Syndrome criteria as defined by the International Sepsis Definitions Conference 14. Patients were included from August 2012 until April 2014. Because of changes in research staffing, no patients were included between June 2013 and October 2013.

Vital parameters of patients were measured by a nurse upon arrival to the ED. All vital parameter measurements were performed using a bedside patient monitor (IntelliVue MP30 System with Multi-Measurement Module; Philips, Eindhoven, the Netherlands). Temperature was measured using an electronic tympanic ear thermometer (Genius 2; Mountainside Medical Equipment, Marcy, New York, USA). After briefly assessing the patient and immediately after the vital signs were available, the nurse and attending physician were asked for their clinical impression of the patient. The nurse and physician were asked for their impression separately to ensure adequate blinding. Their clinical impression was recorded using the Clinical Impression Score (CIS). The CIS score, a singular integer, ranges from 1, indicating not ill, to 10, indicating extreme illness 15.

For each patient, the PIRO score was calculated. The PIRO score takes several factors into account: the (P)redisposition, for example, age and patient history, the type of (I)nfection, the (R)esponse to treatment and factors that indicate (O)rgan failure, for example, lactate and systolic blood pressure 8. The PIRO score was calculated using the results from routine blood analysis, sociodemographic information gathered during admission and the patient’s electronic medical record. These medical records were subsequently monitored to allow for follow-up and to collect demographic data and patient history.

In light of the new Sepsis-3 consensus definitions, a post-hoc analysis on our data was carried out to calculate the qSOFA score. The qSOFA score is based on three items: altered mental status, respiratory frequency and systolic blood pressure. For each item, one point is scored. Patients with a qSOFA score of at least 2 are considered to have an increased mortality risk 6. The qSOFA score was calculated using the initial vital parameters measured during admission to the ED.

All patients received hospital treatment for sepsis as per our hospital’s standardized protocol. This protocol included intravenous antibiotics, fluid resuscitation and oxygen supplementation 16. The protocol did not change during the inclusion period.

Endpoints and definitions

The primary endpoint for this study was direct admission to the ICU. The secondary endpoints were in-hospital, 28-day and 6-month mortality, indirect admission to the ICU and length of stay. For transfer to the ICU, we distinguished between direct and indirect admission. Direct admission was immediate transfer from ED to ICU. Indirect admission was transfer of a patient from ED to first a general ward and thereafter during the patient’s stay in the hospital to the ICU for any reason. In-hospital, 28-day and 6-month mortalities were defined as all-cause mortality during the patient’s stay in the hospital and within the respective times from the day of admittance. Length of stay was defined as the number of days in the hospital; any amount of time spent in-hospital during a 24- h period was considered a full day. Low oxygen saturation was defined as a peripheral SaO₂ of less than 90% on room air or less than 95% with at least 2 l of oxygen supplementation per minute. Patients were categorized into sepsis severity groups using the definitions of the Surviving Sepsis Campaign 2.

Statistical methods

For normally distributed data, the mean and SD were calculated. The Shapiro–Wilk test for normality was used to test for normality. For binominal variables, frequency and percentage of cases were calculated. For non-normally distributed data, the median and interquartile ranges (IQR) were calculated. To compare the variance between sepsis severity groups, the nonparametric Jonckheere trend test was performed. Receiver operator characteristic (ROC) curves and the area under the ROC curve [area under the curve (AUC)] were calculated to determine the relationship between clinical score and endpoint. All AUCs were tested against the null hypothesis (AUC=0.5) using the Wilcoxon signed rank test with continuity correction. For each combination of clinical score and outcome parameter with a significant AUC, we calculated cut-off point, sensitivity, specificity and positive/negative likelihood ratios. Cut-off points were chosen for maximum sensitivity and specificity, that is, closest to the upper-left corner of the ROC. Missing data were excluded from the analysis. All statistical analyses were carried out using IBM SPSS Statistics for Windows, version 20.0 (IBM Corp., Armonk, New York, USA), except for a comparison of AUCs, which was performed using MedCalc, version 14.12.0 (MedCalc, MedCalc Software bvba, Ostend, Belgium). A P-value of 0.05 or less was considered significant; all tests were two tailed.

Results

Patient characteristics

Of the 193 patients enrolled in this study, 103 patients presented with sepsis, 81 presented with severe sepsis and nine patients presented with septic shock (Table 1). The most frequently suspected foci of sepsis were respiratory and urogenital. The CIS, PIRO and qSOFA scores increased, respectively, with sepsis severity (P=0.001, <0.001, and 0.002). However, although the PIRO score differentiated between sepsis categories, we did not observe a significant difference in CIS or qSOFA scores with respect to these categories.

Table 1.

Demographics and distribution of covariates of the study population

In the septic shock group, four (44%) patients had low oxygen saturation on admission. Although the respiratory rates of all patients were available to the attending physician and nurse, they were not recorded in ten cases. In one case, the peripheral oxygen saturation and the rate of oxygen supplementation were not recorded.

The length of stay increased significantly with sepsis severity (P=0.002). The median length of stay of patients presenting with septic shock (13 days, IQR=6–19) was more than twice as long as that of patients presenting with sepsis (6 days, IQR=4–10).

ICU admission

Twenty-one of the 193 (10.9%) patients were admitted to the ICU, of whom 15 (7.8%) were admitted directly from the ED, whereas the remaining six (3.1%) were admitted to the ICU indirectly from a nursing ward (Table 2). One of the 21 patients presented with sepsis, but developed respiratory insufficiency in the ED and needed transfer to the ICU. Six of the 21 patients presented with severe sepsis and were directly admitted to the ICU. Three of these patients had respiratory insufficiency and required mechanical ventilation, and three were transferred to the ICU as a result of persistent haemodynamic instability, despite treatment in the ED. Of the nine (4.7%) patients presenting with septic shock, eight required direct ICU admission, seven because of persistent haemodynamic instability, requiring inotropic support and one because of respiratory insufficiency. One of the nine septic shock patients was sufficiently stabilized in the ED so that general ward admission was possible. The chances of direct ICU admission increased with greater sepsis severity (P<0.001).

Table 2.

Results on the primary and secondary endpoints and clinical scores, including their distribution over the sepsis severity categories

We assessed the accuracy of the CIS, PIRO and qSOFA scores in predicting overall and direct ICU admission. All three scores predicted overall and direct ICU admission (Tables 3 and 4). For direct ICU admission, there was no significant difference between AUCs for the CIS scores of the nurse (AUC=0.896) and the attending physician (AUC=0.861). Furthermore, there was no significant difference between AUCs of the CIS scores and the PIRO (AUC=0.876) and the qSOFA scores (AUC=0.849). When assessing for sensitivity and the positive likelihood ratio of direct ICU admission, the CIS score scored higher than the PIRO score (Table 4). The CIS score of the attending physician for direct ICU admission also showed a good negative likelihood ratio (0.1). However, the qSOFA score showed the highest specificity for direct ICU admission at the cost of a low sensitivity (66.7%). The optimal cut-off point for direct ICU admission of the CIS scores of both the nurse and the attending physician was 8. Fifty-three of the 193 patients had a CIS score of the nurse above the cut-off point, of whom 22.6% were directly admitted to the ICU. Fifty-nine patients had a CIS score of the attending physician above the cut-off point, of whom 23.7% were directly admitted to the ICU. The optimal cut-off point of the PIRO score was 13. Forty-two patients had a PIRO score above this cut-off point, of whom 28.6% were directly admitted to the ICU, and 32 patients had a qSOFA score above the predefined cut-off point of two; 29.4% of these patients were directly admitted to the ICU.

Table 3.

Area under curve for the primary and secondary endpoints

Table 4.

Cut-off points, sensitivity, specificity and likelihood ratios for the Clinical Impression, PIRO and qSOFA scores

Six of the 193 (3.1%) patients required an indirect transfer to the ICU (Table 2), five patients as a result of respiratory insufficiency and one because of haemodynamic instability. Notably, only the CIS score obtained from the nurse at the ED significantly predicted indirect ICU admission (AUC: 0.741; Tables 3 and 4).

Mortality

We assessed the CIS, PIRO and qSOFA scores for in-hospital, 28-day and 6-month mortality (Tables 2–4). Eight of the 193 patients died during their stay in the hospital (Table 2). One of these eight patients died after more than 28 days in the hospital. The CIS scores did not significantly predict any of the assessed mortality endpoints (Table 3). Conversely, the PIRO and qSOFA scores were predictors for mortality. For all three mortality endpoints, the PIRO score showed the highest sensitivity and the qSOFA score showed the highest specificity (Table 4).

Discussion

Time is of the essence in the treatment of sepsis; early and aggressive treatment is imported to reduce mortality 1,2. The main challenge that ED physicians face is effectively stratifying patients on the basis of the need for ICU treatment. Especially, considering that ICU capacity is limited, not all patients may benefit from ICU admission and incorrect stratification may result in increased morbidity, mortality and length of stay 5,6. In this study, we compared stratification for ICU admission and mortality using the PIRO and qSOFA scoring systems with clinical judgement assessed by the ED nurse and attending physician. We hypothesized that clinical judgement would be as accurate to predict ICU admission and mortality as these scoring systems. The clinical judgement of the ED staff was recorded using the CIS. We found that clinical judgement and the PIRO and qSOFA scoring systems performed equally well as predictors of direct ICU admission. Furthermore, we found that the PIRO and qSOFA scores predicted for the mortality endpoints, whereas clinical judgement did not.

All three scores performed equally well as predictors of ICU admission. However, it must be noted that the qSOFA score had a low sensitivity (66.7%), suggesting that about a third of patients requiring ICU treatment will be missed by this score. De Groot et al. 17 also found that the PIRO and clinical judgement scores performed equally on the stratification between ICU and general ward admission for an ED population with only severe sepsis and septic shock. In our study, more than half of the population included patients with (uncomplicated) sepsis. Comparing our results with the results of De Groot, our results suggest that using clinical judgement and the PIRO score to stratify patients in a population including sepsis does not affect the accuracy of the scores. The qSOFA score was not reported in the study by De Groot and can therefore not be compared. Tsai et al. 18 found that the PIRO score predicted ICU admission with an AUC of 0.889, in patients unexpectedly transferred from the ED to the ICU. This AUC is comparable with the AUC for direct ICU admission in our results (AUC=0.876). Our results show that the PIRO and qSOFA scores compared with clinical judgement (measured using the CIS score) are equally good predictors of ICU admission. Considering that clinical judgement can be easily determined bedside within the first 15 min after the patient’s arrival at the ED, this suggests that clinical judgement is an important asset early in the ED stratification process.

It should be noted that the relationship between clinical judgement and ICU admission is not independent; in everyday practice, a patient judged by the treating physician as critically ill or requiring a critical intervention (e.g. ventilation, inotropes) may be more readily admitted to the ICU. This dependency might have introduced a bias that causes an overestimation of the performance of the CIS scores. This bias is potentially limited by the fact that our hospital’s ICU functions as a closed-format ICU. This entails that an ED physician first needs to consult the ICU physician for admission to the ICU. Furthermore, neither CIS, PIRO nor qSOFA scores were communicated to the ICU physician when requesting a transfer; the scores, therefore, did not influence the decision to admit a patient to the ICU.

Perhaps even more interesting than the prediction of direct ICU admission is the prediction of indirect ICU admission as it may provide an opportunity to preventively admit a patient to the ICU to prevent organ failure or mortality. Our results suggest that both clinical judgement and the PIRO and qSOFA scores are poor at predicting indirect ICU admittance. The exception was the CIS score of the nurse (AUC=0.741). However, it should be noted that only six of the 193 patients were indirectly admitted to the ICU. This makes our study underpowered to be conclusive on the prediction of indirect ICU admission. A larger study designed to compare clinical judgement with the PIRO and qSOFA scores is required to be conclusive on predicting indirect ICU admission. However, we speculate that changes over time in scores or vital signs might be more accurate at predicting indirect ICU admission (or patient deterioration) than scores or measurements on a single point in time 16. Therefore, we plan further studies to assess changes in scores and vital signs over time.

The new Sepsis-3 definitions place more emphasis on organ dysfunction 6. Our results suggest that being alert for organ dysfunction (included as elements in the PIRO and qSOFA scores) may aid in predicting mortality. However, to stratify between ICU and general ward admission, the use of a scoring system that includes indicators of organ failure does not add value. Furthermore, the Sepsis-3 definitions no longer include the group of patients with (uncomplicated) sepsis. According to the Sepsis-3 definitions, the patients in our study population with sepsis would have been designated as patients with infection (but not sepsis) without organ failure 6. However, our study results show that ICU admission was 4% and mortality was 3% in the sepsis group. Therefore, further studies should focus on the best way to treat patients in this group (i.e. with infection without organ failure) and on how to best detect early signs of organ failure.

We compared clinical judgement with the PIRO and qSOFA scores for the mortality endpoints. Clinical judgement did not significantly predict any of the mortality endpoints, whereas the PIRO and qSOFA scores did predict mortality. The PIRO score had the best sensitivity and the qSOFA score had the best specificity for the in-hospital and 28-day mortality endpoints. Mortality in our study was lower than expected on the basis of the existing literature 3,7,19,20. Especially, the in-hospital mortality rate of 4.9% in the severe sepsis category and the 28-day and 6-month mortality rates were lower than those in previous studies. These low mortality rates may be partially explained by the fact that we introduced a novel sepsis bundle in our ED in 2008. The aims of this bundle include earlier recognition of septic patients, immediate nurse/physician contact at admission, administration of antibiotics within 60 min and routine fluid resuscitation during the first 2 h (for as long as required) 16. Furthermore, it should be noted that the recent literature reports considerably lower mortality rates compared with earlier publications. This suggests an increased global awareness of sepsis in addition to an early and initially more aggressive treatment 1,21,22. Although low mortality is a positive outcome for our patients, the small number of events limits the power of our results. However, the AUCs for in-hospital and 28-day mortality of PIRO and CIS scores in our study agree with the results of previous studies 7,18. The AUC for the qSOFA score for non-ICU patients as set out by the Third International Consensus Definitions for Sepsis and Septic Shock is not (yet) known, and can therefore not be compared with our results 6.

Our study has several limitations: first, the low mortality rates in all groups, as described above. Second, this study was carried out in a single tertiary centre with an established sepsis protocol, which may lead to higher overall sepsis awareness and therefore a stronger performing CIS score. Third, we did not correct for different experience levels of the nursing staff or the attending physicians; their experience levels ranged from junior to decades of experience in emergency medicine. We will consider correcting for experience levels in future studies as this correction may further improve the predictive accuracy of the CIS score. Fourth, this study was not designed to measure or correct for nurse or physician fatigue. We do not expect that this considerably affected our results as all staff work maximum 9-h shifts, with a maximum of 48 h/week. In other hospitals, shifts may be longer and the effects of fatigue larger. Fifth, daytime-only inclusion of patients introduced a selection bias. Our visit logs show that 65% of admissible patients visited our ED within this timeframe. However, previous clinical evaluations in our department (unpublished data) showed that patients visiting our ED outside this timeframe did not have more severe sepsis. Hence, whether daytime-only inclusion introduced a significant bias in our results is questionable. Finally, treatment limitations dictated by individual patient wishes or by severe comorbidity (Table 1) introduced a bias that led us to underestimate the performance of CIS, PIRO and qSOFA scores as patients who might have required ICU treatment were not transferred to the ICU.

As with any scoring system or biomarker used in medicine, they can only be used as a tool to guide the treating physician. The stratification of patients into different (risk-)groups is a sensible and effective way to triage and to communicate with other physicians in a standardized way. However, scoring systems on their own merits are not a substitute for individual decision-making. Therefore, scoring systems should always be evaluated within the context of the individual patient and the patient’s wishes, and not as a hard criterion for ICU admission.

Conclusion

This study shows that clinical judgement is both fast and reliable in stratifying sepsis patients between the ICU and the general ward. Furthermore, our results show that the PIRO and qSOFA scores do not add value to this stratification process. Therefore, compared with clinical judgement, the PIRO and qSOFA scores perform better as predictors of mortality. In patients with sepsis, we therefore conclude that the principle ‘treat first what kills first’ can be supplemented with ‘judge first and calculate later’.