Performance of the Mortality in emergency department Sepsis score for predicting hospital mortality among patients with severe sepsis and septic shock

Abstract

Objective

The aim of the study was to test if the Mortality in Emergency Department Sepsis (MEDS) score accurately predicts death among emergency department (ED) patients with severe sepsis and septic shock.

Methods

This study was a preplanned secondary analysis of a before-and-after interventional study conducted at a large urban ED. Inclusion criteria were suspected infection, 2 or more criteria for systemic inflammation, and either systolic blood pressure of less than 90 mm Hg after a fluid bolus or lactate 4 mmol/L or higher. Exclusion criteria were: age of less than 18 years, no aggressive care desired, or need for immediate surgery. Clinical and outcomes data were prospectively collected on consecutive eligible patients for 1 year before and 1 year after implementing early goal-directed therapy (EGDT). The MEDS scores and probabilities of in-hospital death were calculated. The main outcome was in-hospital mortality. The area under the receiver operating characteristic curve was used to evaluate score performance.

Results

One hundred forty-three patients, 79 pre-EGDT and 64 post-EGDT, were included. The mean age was 58 ± 17 years, and pneumonia was the source of infection in 37%. The in-hospital mortality rate was 23%. The area under the receiver operating characteristic curve for MEDS to predict mortality was 0.61 (95% confidence interval [CI], 0.50–0.72) overall, 0.69 (95% CI, 0.56–0.82) in pre-EGDT patients, and 0.53 (95% CI, 0.33–0.74) in post-EGDT patients.

Conclusions

The MEDS score performed with poor accuracy for predicting mortality in ED patients with sepsis. These results suggest the need for further validation of the MEDS score before widespread clinical use.

1. Introduction

Seven hundred fifty thousand persons per year in the United States develop severe sepsis with a mortality rate of 30%, equating to approximately 215000 deaths annually [1]. Recent estimates indicate that the rate of severe sepsis hospitalizations doubled during the last decade and age-adjusted population based mortality is increasing [2]. Inasmuch as nearly half of sepsis hospitalizations originate in the emergency department (ED), it is important to develop accurate and reliable methods to assess severity of illness and risk-stratify patients to ensure proper allocation of limited hospital resources [3,4].

Logistic regression–derived scoring systems are commonly used to determine the severity of illness and assess prognosis among critically ill patients. The Mortality in Emergency Department Sepsis (MEDS) score was specifically developed to predict 28-day mortality among ED patients with suspected infection [5]. The MEDS score assigns various points to 9 components (age >65 years, nursing home resident, rapid terminal comorbid illness, lower respiratory tract infection, bands >5%, tachypnea or hypoxemia, septic shock, platelet count <150000, altered mental status) that are readily available at the time of ED evaluation. A recent report indicates it is also efficient at predicting mortality at 1-year after the index hospitalization for infection [6]. Its performance in specific populations with infection, such as those with severe sepsis and septic shock and those undergoing aggressive resuscitation and therapeutic intervention, has not been studied. The aims of this study were to (1) determine if the MEDS score, when applied to a cohort of ED patients with severe sepsis and septic shock, will perform with high accuracy for predicting hospital mortality, defined as an area under the receiver operating characteristic (ROC) curve of more than 0.80 [7] and (2) determine if the high accuracy of the score will be maintained in both patients receiving and not receiving early goal-directed therapy (EGDT) [8].

2. Materials and methods

This study was a preplanned secondary analysis of patients enrolled in a prospective before and after interventional trial investigating the clinical effectiveness of implementing EGDT as a standard-of-care resuscitation protocol in ED patients with severe sepsis and septic shock [9]. This study protocol was reviewed and approved by the institutional review board for the execution of human research before enrollment of patients.

Subjects were enrolled from August 2004 through November 2006 in the ED at Carolinas Medical Center (Charlotte, NC), an 800-bed teaching hospital with more than 100000 patient visits per year. Explicit criteria for enrollment included (1) age of more than 17 years; (2) suspected or confirmed infection; (3) 2 or more systemic inflammatory response syndrome criteria [10]: heart rate >90 beats per minute, respiratory rate >20 breaths per minute, temperature >38 or <36°C, white blood cell count >12000 or <4000 cells/mm³ or >10% bands; (4) systolic blood pressure <90 mm Hg or mean arterial pressure <65 mm Hg after a isotonic fluid bolus, or anticipated need for ICU care and a serum lactate concentration ≥4.0 mmol/L. Exclusion criteria included (1) age of less than 18 years; (2) need for immediate surgery; and (3) absolute contraindication for a chest central venous catheter.

In both phases, eligible subjects were identified by board-certified emergency physicians. In the before phase, patients with severe sepsis and septic shock were managed at the discretion of the attending physician without any written management protocol. In the after phase, patients were treated in the ED and ICU with an institution approved EGDT protocol that has been previously described [9]. All data elements required to calculate the MEDS score as well as the hospital outcomes were prospectively collected on standardized forms and entered into a database for later analysis. The definitions of MEDS score variables used in this study were identical to those reported in the original publication by Shapiro et al [5]. All scores were calculated post hoc, and there were no missing data points.

2.1. Data analysis

We defined the primary dependent variable for statistical analysis as in-hospital mortality. The predictive ability of the MEDS score was evaluated using ROC curve analysis generated on standard statistical software (StatsDirect v 2.5.7, Cheshire, UK). The area under the curve (AUC) was used to compare the discriminatory power of the scoring system, with an AUC 1.0 considered perfect discrimination and 0.5 considered equal to chance [7]. In addition, the actual vs predicted mortalities by MEDS score risk group was examined by graphic analysis.

3. Results

A total of 143 patients, 79 pre-EGDT and 64 post-EGDT, were analyzed. Table 1 summarizes the demographics and initial clinical characteristics of the entire study population and the subpopulations that either received or did not received EGDT. The average MEDS score was 10 ± 3 points in the entire population, 10 ± 3 points in pre-EGDT patients, and 11 ± 3 points in post-EGDT patients. The in-hospital mortality rate of the entire population was 23% (33/143).

Table 1.

Demographics and initial clinical characteristics

Variable	All patients (N = 143)	Pre-EGDT (n = 79)	Post-EGDT (n = 64)
Age (y)	57.8 ± 17	57.6 ± 18	58.1 ± 15
Lowest O2 sat (%)	92.8 ± 7	92.8 ± 7	92.9 ± 7
Highest RR (breaths/min)	27.5 ± 10	25.6 ± 9	29.7 ± 10
AMS (%)	33	25	42
NH resident (%)	24	23	27
Rapid terminal condition (%)	1	1	1
Platelet count (cell/mm3)	243 ± 119	246 ± 123	240 ± 116
Bands (%)	10 ± 15	9 ± 14	12 ± 17
Suspected pneumonia (%)	37	32	43
In-hospital mortality (%)	23	27	18

RR indicates respiratory rate; O₂ sat, oxygen saturation; NH, nursing home; AMS, altered mental status.

Fig. 1 compares predictive ability for mortality of MEDS score using ROC curve analysis. Neither the entire population nor the population dichotomized as either receiving or not receiving EGDT performed with excellent accuracy. The MEDS score demonstrated AUC suggesting poor accuracy for mortality prediction in the entire population (AUC, 0.61; 95% confidence interval [CI], 0.50–0.72), in the patients not receiving EGDT (AUC, 0.69; 95% CI 0.56–0.82), and in patients receiving EGDT in the ED (AUC, 0.53; 95% CI, 0.33–0.74).

Fig. 1.

Receiver operating characteristic curves of MEDS for (1) all patients, (2) pre-EGDT, and (3) post-EGDT for predicting hospital mortality.

Fig. 2 shows a plot of the actual mortalities vs the MEDS score predicted mortalities in the entire population. Among the moderate-risk groups (5–7, 8–12, and 13–15), the MEDS score consistently underestimated mortality by 14% in all patients, 16% in pre-EGDT patients, and 12% in the post-EGDT patients.

Fig. 2.

Graphic analysis of the predicted vs observed mortality MEDS score groups for (A) all patients; (B) pre-EGDT; and (C) post-EGDT. The N denotes the number of subjects analyzed in each of the corresponding MEDS score groups.

4. Discussion

The results of this study suggest that the MEDS scoring system functions with poor accuracy for predicting in-hospital mortality when applied to ED patients with severe sepsis and septic shock. Furthermore, the score consistently underestimated mortality risk in patients resuscitated with EGDT. Thus, the routine use of the MEDS score in the risk stratification of ED patients with severe sepsis and septic shock cannot be recommended without further investigation.

The MEDS score was derived and validated in a heterogeneous group of ED patients at risk for infection evidence by the treating clinician's decision to order a blood culture [5]. This broad inclusion is one explanation for the findings of the inferior performance of the MEDS score in the present study, given that we tested the performance of the score in a population of much more acutely ill patients. In addition, the original data were collected before the introduction of aggressive therapeutic interventions for severe sepsis such as EGDT. Thus, our finding of the worst performance of the score in patients treated with EGDT may be partially explained by the improved survival of patients who would have died before early and aggressive resuscitation [8].

The MEDS score consistently underestimated mortality in the moderate risk groups (5–7, 8–12, 13–15) in this study. This point is of critical importance if the MEDS score were to be used to make decisions regarding disposition from the ED or potential resource use, as the underestimation of acuity of illness has the potential to result in poorer patient outcomes. Our findings suggest that alternative approaches to risk stratification may be necessary for use in ED patients with severe sepsis and septic shock.

We are aware of only one previous full-length publication that has evaluated the external validity of the MEDS score for predicting mortality [11]. In a similar population to the present study (severe sepsis), Chen et al found the MEDS score to have an area under the ROC curve of 0.74 (no CIs given) for predicting 28-day mortality. These results suggest only fair or moderate performance of the scoring system and the point estimate reported by Chen et al (0.74) is not contained within the 95% CIs found in the present study for overall MEDS performance (0.50–0.72). Although the exact reason for these discrepant findings is unknown, one explanation is that the Chen et al study is a retrospective chart review and thus subject to the biases imparted by that study design. We collected the variables for the MEDS score prospectively, thus eliminating some of the variability in data collection and abstraction found in retrospective studies. Nonetheless, these discrepant findings highlight the need for further investigation before applying this score at the bedside.

It is important to point out that the MEDS score does have several desirable characteristics as an ED-based scoring system because it is easy to calculate at the bedside and includes vital signs and laboratory data that are routinely available in the ED. Although our results presented here show less than desirable performance characteristics of the MEDS score, as other investigators have suggested [11], we would suggest that other studies should evaluate its performance in similar populations at other centers.

4.1. Limitations

This report has several limitations to be considered. First, all scores were calculated post hoc and not applied in real time. It is possible that if the scores had been calculated and applied prospectively, they would have performed with different accuracy because of their potential impact on disposition decisions. Another limitation of our study is the size of the sample studied. It is possible that a larger sample would have resulted in a more precise estimation of the accuracy of the scoring system. In addition, this was a single-center study with a select patient population, which limits the generalizabiltiy of the results to other centers and patient populations.

5. Conclusions

The MEDS score demonstrated poor accuracy for predicting in-hospital mortality among ED patients with severe sepsis and septic shock. The score consistently underestimated mortality in the moderate-risk groups. These results suggest the need for further validation of the MEDS score before widespread clinical use.