Comparison of the Reverse Shock Index Multiplied by Glasgow Coma Scale Score, MEWS, and qSOFA as Sepsis Screening Tools for Predicting Short-Term Outcomes

Abstract

Background

A simple screening tool is needed for resource-limited settings because rapid treatment is crucial in sepsis. We investigated whether a simplified score, the reverse shock index multiplied by the Glasgow Coma Scale score (rSIG), could replace the Modified Early Warning Score (MEWS) or the quick Sequential Organ Failure Assessment (qSOFA) for sepsis screening.

Methods

We used data from a Japanese multicenter prospective observational study. This dataset included patients with suspected infection who were admitted from 35 emergency departments (cohort 1) and patients with suspected infection who were admitted to 22 intensive care units (cohort 2). The primary outcome was 28-day mortality. Secondary outcomes were ICU admission or death within 28 days and mechanical ventilation or death within 28 days in cohort 1 and diagnosis of sepsis, need for invasive support (composite of vasopressor use, mechanical ventilation, or death before day 4) in cohort 2.

Results

In cohort 1, the AUROC for rSIG was significantly higher for 28-day mortality than for MEWS but not significantly different from that of qSOFA (0.69 [95% CI 0.64–0.74] vs 0.64 [0.59–0.69] vs 0.68 [0.63–0.72]). In cohort 2, the AUROC of rSIG for 28-day mortality was similar to that of MEWS and qSOFA (0.62 [0.56–0.68] vs 0.58 [0.52–0.64] vs 0.62 [0.56–0.67]). The AUROCs for diagnosis of sepsis, ICU admission or 28-day mortality, and mechanical ventilation or 28-day mortality were similar. The AUROC for need of invasive support was significantly higher for rSIG than for MEWS. For most outcomes, rSIG ≥15 had higher sensitivity than a qSOFA ≥2 or a MEWS total ≥5 or any variable ≥3.

Conclusion

Although there are limitations in the data, rSIG predicted short-term outcomes in patients with suspected infections as well as or better than MEWS and qSOFA.

Background

Sepsis is a life-threatening condition, particularly in low- and middle-income countries.¹ Because rapid initial treatments such as administration of antibiotics and fluid resuscitation is a key of treatments in sepsis, screening tools is important of the sepsis management. Current international guidelines recommend implementing a performance improvement program for sepsis, including screening for high-risk patients.²

Early warning scores, including the Modified Early Warning Score (MEWS), are now widely used to screen high-risk patients with sepsis. However, early warning scores are complex and can sometimes result in misclassification. Previous studies have showed that early warning scores were inaccurate in 18.9% to 28.6% of cases.^3–6 Furthermore, it has been suggested that the manual measure of respiratory rate, which is required for early warning scores, may contribute to under-triage.⁴ The implementation of early warning scores may also be challenging in resource-limited settings. Therefore, it is desirable for sepsis screening tools to be not only accurate but also simple.

The reverse shock index multiplied by Glasgow Coma Scale (GCS) score (rSIG; GCS score × systolic blood pressure/heart rate) requires minimal information and resources and has been confirmed to predict mortality and the need for an invasive procedure in patients with severe trauma.^7–22 We have noticed that rSIG can also recognize high-risk patients with suspected infection because rSIG requires fewer variables than an early warning score and does not require use of a scoring table except for the GCS. Our previous study showed that the rSIG was superior to both MEWS and the quick Sequential Organ Failure Assessment (qSOFA) in predicting a composite outcome of vasopressor use, need for mechanical ventilation, or death within 72 hours among 724 adult patients with infection admitted from our emergency department (AUROC for each outcome [95% CI]: rSIG 0.84 [0.78–0.88]; MEWS 0.78 [0.71–0.84]; qSOFA 0.72 [0.65–0.79]; SI 0.80 [0.74–0.85]).²³ However, this study was performed at a single center, and the extent to which its findings were generalizable to other institutions was unclear.

We hypothesized that rSIG would be superior to MEWS and qSOFA in predicting 28-day mortality and the need for vasopressor or mechanical ventilation. The aim of this study was to determine whether rSIG is superior to MEWS and qSOFA in predicting short-term outcomes in patients with suspected infection using data from a multicenter observational study.

Materials and Methods

Ethical Approval and Informed Consent

This study was a post-hoc analysis of data from a study conducted by the Japanese Association for Acute Medicine Sepsis Prognostication in Intensive Care Unit and Emergency Room (JAAM SPICE-ER and SPICE-ICU). The study complied with the Declaration of Helsinki and was approved by the research ethics committees of all participating institutions in the JAAM SPICE Study Group (supplemental data), including our hospital, National Center for Global Health and Medicine (approval number for our hospital: NCGM-G-002247-00). Informed consent was obtained via the opt-out route using the hospital website.

Design, Setting, and Participants

The study had two cohorts of patients: one including patients with suspected infection admitted to any of 35 hospitals from the emergency department between December 2017 and February 2018 (cohort 1) and the other including patients with suspected infection admitted to any of 22 ICUs from the emergency department or general ward between December 2017 and May 2018 (cohort 2). We screened all patients in cohort 1 and all patients in cohort 2 with the exception of those who had already been admitted to an ICU for suspected infection. Data for patients with cardiac arrest at initial triage, those with missing information on 28-day mortality, and those with incomplete recording of vital signs were excluded.

Collection of Data

We collected information on demographics, comorbidities, severity scores, vital signs, need for mechanical ventilation or vasopressor agents, and 28-day mortality. We calculated rSIG, MEWS, and qSOFA scores according to the vital signs recorded at the time of diagnosis. The AVPU (alert, verbal, pain, unresponsive) score was used to estimate MEWS in accordance with a previous study²⁴ because the GCS score was used rather than the AVPU score in the original cohort.

The primary outcome was 28-day mortality. The secondary outcomes were diagnosis of sepsis (cohort 1), need for invasive support (a composite of vasopressor use, mechanical ventilation or death within 4 days) (cohort 2), ICU admission or death within 28 days (cohort 1), and need for mechanical ventilation or death within 28 days (cohort 1). We compared the prediction performance of each screening tool using the area under the receiver-operating characteristic curve (AUROC). We calculated the optimal cut-off values of rSIG and MEWS for each endpoint, namely, the value that maximized the AUROC for each score. We also analyzed the primary endpoint when the cut-off value was used as an alert trigger. The trigger for MEWS was defined as ≥5 points in total or ≥3 points for any single variable and a positive qSOFA result was defined by a score ≥2.²⁵ The cutoff value for rSIG was ≤15 based on a previous study.²³

Statistical Analysis

The data are summarized as the median [interquartile range] or as the number (percentage) as appropriate. Fisher’s exact test was used to compare categorical variables and the Mann–Whitney U-test to compare continuous variables. Receiver-operating characteristic curves were created to show the impact of shifting the positive cutoff value on the true-positive rate (sensitivity) and false-positive rate (1 − specificity) rates. The AUROCs were compared using the technique described by DeLong et al.²⁶ Statistical significance was set at p<0.05 for all analyses. We compared the AUROCs between rSIG, MEWS, and qSOFA for each endpoint. We performed a sensitivity analysis in the subgroup of elderly patients (age ≥75 years), considering the possibility that the prediction of early warning scores may have high heterogeneity according to age.²⁷ All statistical analyses were performed using R version 3.4.1 (R Foundation for Statistical Computing, Vienna, Australia) and JMP Pro version 15 (SAS Institute Inc., Cary, NC, USA).

Results

Of 1060 patients in cohort 1, 151 were excluded for cardiac arrest (n=3), missing data for 28-day outcomes (n=139), or incomplete data for vital signs (n=9). Of 652 patients in cohort 2, 79 were excluded for having already been admitted to an ICU with suspected infection (n=34), missing data for 28-day outcomes (n=41), or incomplete recording of vital signs (n=4). Finally, data were analyzed for 910 patients in cohort 1 and 573 in cohort 2 (Figure 1). In the original cohort, data were missing for rSIG, MEWS, and qSOFA in 3, 13, and 2 patients, respectively.

Figure 1.

Flow diagram showing the patient selection process.

Abbreviations: ED, emergency department; ICU, intensive care unit.

Table 1 shows the patient demographics and clinical characteristics. Mortality at 28 days was 14% (126/910) in cohort 1 and 18% (103/573) in cohort 2.

Table 1.

Patient Demographics and Clinical Characteristics

Variable	Cohort 1*, n=910	Cohort 2**, n=573
Age, median [IQR]	79 [68, 85]	74 [63, 82]
Male, n (%)	545 (60)	321 (56)
BMI, median [IQR]	21.2 [18.6, 23.8]	22.0 [19.2, 24.5]
CCI, median [IQR]	2 [0, 3]	4 [3, 6]
Admission from ER, n (%)	910 (100)	338 (59)
Vital signs
Heart rate, beats/min, median [IQR]	99 [84, 114]	106 [89, 123]
Systolic blood pressure, mmHg, median [IQR]	126 [105, 149]	107 [85, 130]
GCS score, median [IQR]	14 [13, 15]	13 [7, 15]
Respiratory rate, breaths/min, median [IQR]	22 [18, 28]	24 [19, 30]
Body temperature, °C, median [IQR]	37.5 [36.7, 38.5]	37.4 [36.5, 38.4]
28-day mortality, n (%)	126 (14)	103 (18)

Notes: *Patients admitted to hospital from the emergency department. **Patients admitted to the intensive care unit from the emergency department or general ward).

Abbreviations: BMI, body mass index; CCI, Charlson Comorbidity Index; GCS, Glasgow Coma Scale; IQR interquartile range.

Primary Outcome

Figure 2 shows the ROC curves for the ability of rSIG, MEWS, and qSOFA to predict 28-day mortality in cohorts 1 and 2. In cohort 1, the AUROC for rSIG was significantly higher than that for MEWS but not significantly different from that for qSOFA (0.69 [95% confidence interval (CI) 0.64–0.74] vs 0.64 [95% CI 0.59–0.69] vs 0.68 [95% CI 0.63–0.72]). The cut-off values when the AUROCs for rSIG and MEWS were maximum were 13.5 and 6, respectively. In cohort 2, the AUROC for rSIG was similar to that for MEWS and qSOFA (0.62 [95% CI 0.56–0.68] vs 0.58 [95% CI 0.52–0.64] vs 0.62 [95% CI 0.56–0.67]). The cut-off values when the AUROCs for rSIG and MEWS were maximum were 10.9 and 7, respectively.

Figure 2.

Ability of the three screening tools to predict the primary outcome of 28-day mortality.

Notes: (a), Cohort 1. (b), Cohort 2. The AUROCs were compared using the technique described by DeLong et al.²⁹

Abbreviations: AUROC, area under the receiver-operating characteristic curve; CI, confidence interval; MEWS, Modified Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment; rSIG, reverse shock index multiplied by Glasgow Coma Scale score.

Secondary Outcomes

Table 2 and Figure 3 show the secondary outcomes. There was no significant difference in the AUROC for a diagnosis of sepsis between the three tools (rSIG 0.77 [95% CI 0.70–0.84] vs MEWS 0.74 [95% CI 0.66–0.81] vs qSOFA 0.75 [95% CI 0.66–0.82]). In terms of need for invasive organ support within 4 days, the AUROC for rSIG was significantly higher than that for MEWS but not significantly different from that for qSOFA (0.71 [95% CI 0.66–0.76] vs 0.65 [95% CI 0.60–0.71] vs 0.67 [95% CI 0.61–0.72]). Furthermore, the AUROC for rSIG was similar to that for MEWS and qSOFA for ICU admission or death within 28 days (0.68 [95% CI 0.64–0.72] vs 0.67 [95% CI 0.63–0.70] vs 0.69 [95% CI 0.66–0.72]) and need for mechanical ventilation or death within 28 days (0.70 [95% CI 0.66–0.74] vs 0.67 [95% CI 0.63–0.70] vs 0.69 [95% CI 0.65–0.73]).

Table 2.

Ability of the Three Sepsis Screening Tools to Predict Secondary Outcomes

Variable	AUROC (95% CI)	Difference (95% CI)	p-value
Diagnosis of sepsis*
rSIG	0.77 (0.70–0.85)
vs MEWS	0.74 (0.67–0.82)	0.030 (−0.036–0.097)	0.52
vs qSOFA	0.75 (0.67–0.83)	0.024 (−0.056–0.105)	0.56
Need for invasive organ support**
rSIG	0.71 (0.66–0.77)
vs MEWS	0.65 (0.60–0.71)	0.062 (0.018–0.105)	0.005
vs qSOFA	0.67 (0.62–0.72)	0.047 (−0.006–0.100)	0.081
ICU admission or death within 28 days
rSIG	0.68 (0.65–0.72)
vs MEWS	0.67 (0.63–0.70)	0.017 (−0.016–0.049)	0.31
vs qSOFA	0.69 (0.66–0.73)	−0.008 (−0.041–0.026)	0.65
Need for mechanical ventilation or death within 28 days
rSIG	0.70 (0.66–0.74)
vs MEWS	0.67 (0.63–0.71)	0.032 (−0.004–0.068)	0.078
vs qSOFA	0.69 (0.65–0.73)	0.010 (−0.054–0.048)	0.58

Notes: *Sepsis was diagnosed according to the Third International Consensus Definitions for Sepsis and Septic Shock criteria. **Defined as a composite of vasopressor use, mechanical ventilation, or death within 4 days.

Abbreviations: AUROC, area under the receiver-operating characteristic curve; CI, confidence interval; MEWS, modified Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment; rSIG, reverse shock index multiplied by Glasgow Coma Scale score.

Figure 3.

Receiver-operating characteristic curves showing the ability of the three screening tools to predict secondary outcomes. (a), Diagnosis of sepsis. (b), Need for invasive organ support. (c), Intensive care unit admission or death. (d), Need for mechanical ventilation or death. Need for invasive organ support was defined by a composite of vasopressor use, mechanical ventilation, or death within 4 days. Sepsis was diagnosed using the Third International Consensus Definitions for Sepsis and Septic Shock criteria.

Abbreviations: MEWS, Modified Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment; rSIG, reverse shock index multiplied by Glasgow Coma Scale score. There were no significant differences in the AUROCs of the three tools, except that rSIG was higher than MEWS for the need for invasive organ support.

Sensitivity and Specificity for Use of the AUROC Cut-Off Value as an Alert Trigger

Table 3 shows the sensitivity and specificity of the AUROCs for rSIG and MEWS when the cut-off value was used as a trigger. The sensitivity of rSIG ≤15 was higher than that of MEWS or qSOFA for most outcomes, including need for invasive organ support and diagnosis of sepsis.

Table 3.

Sensitivity and Specificity of the Three Screening Tools When the AUROC Cut-Off Value Was Used as an Alert Trigger

Variable	Sensitivity (95% CI)	Specificity (95% CI)	Negative Predictive Value (95% CI)	AUROC (95% CI)	p-value
28-day mortality in cohort 1
rSIG ≤15	0.60 (0.51–0.69)	0.69 (0.66–0.73)	0.92 (0.89–0.94)	0.65 (0.60–0.69)
vs MEWS total ≥5 or any variable ≥3	0.60 (0.51–0.69)	0.58 (0.55–0.62)	0.90 (0.87–0.93)	0.59 (0.55–0.64)	0.030
vs qSOFA ≥2	0.61 (0.52–0.70)	0.66 (0.52–0.70)	0.91 (0.89–0.94)	0.63 (0.59–0.68)	0.57
28-day mortality in cohort 2
rSIG ≤15	0.78 (0.68–0.85)	0.33 (0.29–0.38)	0.87 (0.81–0.92)	0.56 (0.51–0.60)
vs MEWS total ≥5 or any variable ≥3	0.74 (0.64–0.82)	0.36 (0.32–0.41)	0.86 (0.81–0.91)	0.55 (0.50–0.60)	0.78
vs qSOFA ≥2	0.72 (0.62–0.80)	0.42 (0.37–0.46)	0.87 (0.82–0.91)	0.57 (0.52–0.62)	0.55
Diagnosis of sepsis*
rSIG ≤15	0.74 (0.68–0.78)	0.68 (0.51–0.83)	0.18 (0.12–0.26)	0.71 (0.63–0.79)
vs MEWS total ≥5 or any variable ≥3	0.69 (0.64–0.73)	0.61 (0.43–0.76)	0.14 (0.09–0.21)	0.65 (0.56–0.73)	0.12
vs qSOFA ≥2	0.63 (0.59–0.68)	0.74 (0.57–0.89)	0.15 (0.10–0.21)	0.69 (0.61–0.76)	0.63
Need for invasive organ support**
rSIG ≤15	0.77 (0.73–0.81)	0.49 (0.40–0.59)	0.36 (0.29–0.45)	0.63 (0.58–0.68)
vs MEWS total ≥5 or any variable ≥3	0.74 (0.69–0.78)	0.52 (0.42–0.61)	0.34 (0.27–0.42)	0.63 (0.58–0.68)	0.84
vs qSOFA ≥2	0.68 (0.63–0.72)	0.59 (0.49–0.68)	0.33 (0.26–0.40)	0.63 (0.58–0.69)	0.95
ICU admission or death within 28 days
rSIG ≤15	0.52 (0.47–0.79)	0.75 (0.71–0.79)	0.27 (0.23–0.30)	0.64 (0.61–0.67)
vs MEWS total ≥5 or any variable ≥3	0.61 (0.56–0.67)	0.65 (0.61–0.69)	0.25 (0.22–0.29)	0.63 (0.60–0.67)	0.83
vs qSOFA ≥2	0.57 (0.51–0.62)	0.73 (0.69–0.76)	0.25 (0.22–0.29)	0.65 (0.61–0.68)	0.57
Need for mechanical ventilation or death within 28 days
rSIG ≤15	0.58 (0.51–0.64)	0.73 (0.70–0.76)	0.17 (0.14–0.20)	0.66 (0.62–0.69)
vs MEWS total ≥5 or any variable ≥3	0.65 (0.58–0.71)	0.63 (0.59–0.66)	0.16 (0.13–0.20)	0.64 (0.60–0.67)	0.37
vs qSOFA ≥2	0.61 (0.54–0.67)	0.70 (0.66–0.73)	0.16 (0.13–0.20)	0.65 (0.62–0.69)	0.80

Notes: *Sepsis was diagnosed according to the Third International Consensus Definitions for Sepsis and Septic Shock criteria. **Need for invasive organ support was defined as a composite of vasopressor use, mechanical ventilation, or death within 4 days.

Abbreviations: AUROC, area under the receiver-operating characteristic curve; CI, confidence interval; MEWS, modified Early Warning Score; qSOFA, quick Sequential Organ Failure Assessment; rSIG, reverse shock index multiplied by Glasgow Coma Scale score.

Sensitivity Analysis

In the sensitivity analysis in the subgroup of elderly patients aged 75 years and older, the results were similar to those in analysis of all patients (in supplemental figure and supplemental table). In Cohort 2, the AUROC of rSIG for 28-day mortality was significantly higher than that of MEWS. On the other hand, no significant difference was observed between rSIG and MEWS in the AUROC for the need for invasive organ support.

Discussion

Our analysis of data from a multicenter study shows that rSIG may be superior to MEWS in predicting 28-day mortality in patients with suspected infection. We also found that rSIG was significantly better able to predict the need for invasive organ support in these patients. Triage tools must be able to not only predict fatal outcomes but also facilitate rapid decision making. rSIG may be useful for identifying high-risk patients with suspected infection who need early intervention. In this study, the prediction performance of rSIG was as good as that of qSOFA. However, owing to its low sensitivity, the current international guidelines do not recommend qSOFA for use as a single screening tool. We also found that qSOFA tended to be less sensitive than rSIG. Although rSIG had a limited ability to predict 28-day mortality in ICU patients, it was superior in predicting mortality in patients in the emergency department and the need for invasive organ support up to day 4. These findings suggest that rSIG may be particularly useful for decision-making in the acute phase, such as during initial triage in the emergency department or other prehospital settings.

Previous studies in trauma patients have reported AUROC cut-off values for rSIG that are in the range of 10.2–18.0.^{8,10,13,15,20} The AUROC cut-off value for rSIG in this study was within that range but different from that in our previous study in patients with infection.²³ This inconsistency may reflect differences in patient characteristics, considering that the data analyzed in the present study were mainly from tertiary hospitals and many patients who were included were at high-risk of mortality. Larger studies may be needed to generalize the optimal cut-off value for rSIG.

The advantage of rSIG is that it includes few variables and does not require a scoring table, so can be calculated quickly. International clinical practice guidelines recommend use of the early sepsis treatment bundle.² However, the target times for initiation and completion of the bundle vary according to the patient’s level of risk. For example, a previous study reported that antibiotic administration within 1 h was associated with mortality in patients who had sepsis with shock but not in those without shock.²⁸ However, adherence to the sepsis bundle for sepsis requiring intensive care can be difficult, as treating more severe sepsis requires more resources. Therefore, to improve adherence to the sepsis bundle, there is a need to selectively focus existing resources on high-risk patients. Although rapid sepsis triage may not be difficult for some specialists, it is often performed by non-experts, such as residents or nurses. Loss of time and misclassification in initial triage make it difficult to implement the sepsis bundle. Therefore, we believe that use of rSIG during triage will help to increase adherence in high-risk patients.

This study had some limitations. First, the data did not include the indication for hospital or ICU admission. Second, the AVPU score used for MEWS was the value predicted by the GCS score. Additionally, since the GCS score is assessed subjectively, there was a possibility of observer bias. Third, although NEWS is often used as well as MEWS,²⁹ we were not able to compare the performance of rSIG with that of NEWS because of lack of data. However, because NEWS requires saturation of percutaneous oxygen and the presence of oxygen therapy, it is likely to be even less feasible than MEWS in low-resource settings. We believe that NEWS and rSIG are non-competitive and are useful in different settings. Fourth, this study included patients admitted to emergency departments and ICUs in Japan. Differences in patient characteristics such as race and in healthcare environments may affect the results. In addition, in Japan, triage nurses and residents who are likely to perform the initial assessment of patients suspected infection are generally trained to measure the GCS score, but in countries where this is not the case, additional training may be required to assess rSIG. Finally, to implement rSIG in clinical practice, it is desirable to conduct prospective studies.

Conclusions

Although there are limitations in the data, rSIG predicted short-term outcomes in patients with suspected infections as well as or better than MEWS and qSOFA. rSIG is easy to use and includes few variables, so it could be an alternative to MEWS as a screening tool for sepsis, especially in low-resource settings.

Funding Statement

This work was supported in part by Grants-in-Aid for Research from the National Center for Global Health and Medicine (Nos. 20A3002, 23A3001).

Abbreviations

AUROC, area under the receiver-operating characteristic curve; AVPU, alert, verbal, pain, unresponsive; EWS, Early Warning Score; GCS, Glasgow Coma Scale; JAAM, Japan Association for Acute Medicine; MEWS, Modified Early Warning Score; NEWS, National Early Warning Score; qSOFA, Quick Sequential Organ Failure Assessment; rSIG, reverse shock index multiplied by Glasgow Coma Scale score; SPICE, Sepsis Prognostication in Intensive Care Units and Emergency Rooms.

Data Sharing Statement

The data that support the findings of this study were supplied by the Japanese Association for Acute Medicine (JAAM). It is necessary to contact JAAM regarding data use and are available upon approval by JAAM (jaam-6@bz04.plala.or.jp).

Ethics Approval and Informed Consent

The study was approved by each Research Ethics Committee of all participating institutions at the JAAM SPICE study group, including our hospital (approval number: NCGM-G-002247-00). Informed consent was obtained using the opt-out method via the hospital website.

Author Contributions

All authors made a significant contribution to the work reported, whether that is in the conception, study design, execution, acquisition of data, analysis and interpretation, or in all these areas; took part in drafting, revising or critically reviewing the article; gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

The authors declare that they have no competing interests.