Predicting mortality in adults with suspected infection in a Rwandan hospital: an evaluation of the adapted MEWS, qSOFA and UVA scores

Abstract

Rationale

Mortality prediction scores are increasingly being evaluated in low and middle income countries (LMICs) for research comparisons, quality improvement and clinical decision-making. The modified early warning score (MEWS), quick Sequential (Sepsis-Related) Organ Failure Assessment (qSOFA), and Universal Vital Assessment (UVA) score use variables that are feasible to obtain, and have demonstrated potential to predict mortality in LMIC cohorts.

Objective

To determine the predictive capacity of adapted MEWS, qSOFA and UVA in a Rwandan hospital.

Design, setting, participants and outcome measures

We prospectively collected data on all adult patients admitted to a tertiary hospital in Rwanda with suspected infection over 7 months. We calculated an adapted MEWS, qSOFA and UVA score for each participant. The predictive capacity of each score was assessed including sensitivity, specificity, positive and negative predictive value, OR, area under the receiver operating curve (AUROC) and performance by underlying risk quartile.

Results

We screened 19 178 patient days, and enrolled 647 unique patients. Median age was 35 years, and in-hospital mortality was 18.1%. The proportion of data missing for each variable ranged from 0% to 11.7%. The sensitivities and specificities of the scores were: adapted MEWS >4, 50.4% and 74.9%, respectively; qSOFA >2, 24.8% and 90.4%, respectively; and UVA >4, 28.2% and 91.1%, respectively. The scores as continuous variables demonstrated the following AUROCs: adapted MEWS 0.69 (95% CI 0.64 to 0.74), qSOFA 0.65 (95% CI 0.60 to 0.70), and UVA 0.71 (95% CI 0.66 to 0.76); there was no statistically significant difference between the discriminative capacities of the scores.

Conclusion

Three scores demonstrated a modest ability to predict mortality in a prospective study of inpatients with suspected infection at a Rwandan tertiary hospital. Careful consideration must be given to their adequacy before using them in research comparisons, quality improvement or clinical decision-making.

Strengths and limitations of this study.

• We evaluated the three severity of illness (SOI) scores in the literature that are most likely to be feasible and predictive in low and middle income countries (LMIC) settings; this includes the first hospital-wide evaluation of Universal Vital Assessment score, the only score that was developed using LMIC cohorts.

• Many SOI scores are developed and tested intheintensive care unit (ICU) populations while our analysis also includes hospitalised patients outside the ICU; this is important because many critically ill patients in LMICs remain outside the ICU due to resource constraints.

• We analysed the predictive capacity of the SOI models as both continuous and dichotomous scores and using multiple metrics, including sensitivity, specificity, positive and negative predictive value, OR, area under the receiver operating curve and performance by underlying risk quartile.

• Vital signs used in the scores were collected at different times in the participants’ hospitalisations, depending on how they met inclusion criteria for the study (time of fever, operation or culture sample retrieval); while this may decrease the predictive capacity of the scores, it also mirrors how the scores might be used in practice.

• The results from this single-centre study among adults with suspected infection may not be generalisable to other populations; this variability in predictive capacity is a known challenge in using SOI scores and the reason it is important to validate a score in a particular site before using it.

Introduction

Multiple mortality prediction models have been developed or validated in low and middle income countries (LMICs) over the last 5 years.^1–11 The proposed uses of these models include identifying patients at acute risk for deterioration in order to trigger increased levels of care,^{3 11–15} more informed allocation of scarce resources,^{13 15} benchmarking for quality assessment and quality improvement¹ and controlling for severity of illness (SOI) in future trials.^{13 16 17} In addition, updates to definitions of critical illness syndromes, most notably sepsis and acute respiratory distress syndrome, have increasingly emphasised definitions that have predictive validity.^{18 19}

The modified early warning score (MEWS) was first reported describing 709 medical patients in a district hospital in the UK in 2001,²⁰ and was based on an early warning score developed and published in an abstract in 1997.²¹ It was created by assigning weighted scores to each vital sign based on severity of the vital sign abnormality, and it has since been tested in multiple LMIC sites.8 12 22 23 The quick Sequential (Sepsis-Related) Organ Failure Assessment (qSOFA) score was developed as part of an international re-defining of sepsis, using high income country (HIC) hospital administrative data¹⁹ and retrospectively tested in nine sites in LMICs; it demonstrated variable predictive capability across these sites.¹⁵ qSOFA was also prospectively tested in a study from an upper middle income country with multiple sites.¹¹ The Universal Vital Assessment (UVA) score was recently developed using linear regression in 15 in-hospital cohorts from 6 African countries, and showed good predictive capability across the entire derivation population, with no reporting on its performance in the individual cohorts.¹³ It has only been assessed in one small emergency department cohort outside the initial derivation population.²³

All three scores use accessible bedside clinical measures and are, therefore, appealing for LMIC settings where laboratory values and detailed comorbidity histories are often not available. All three scores have also been developed for hospital ward patients, which is relevant to LMICs, where critically ill patients often remain in general wards due to the scarcity of intensive care unit (ICU) beds.

We prospectively collected data on all adult hospitalised patients with suspected infection over a 7-month period in a study of antimicrobial resistance patterns in a tertiary referral hospital in Rwanda.²⁴ The current study was planned as part of the original study design, and is a secondary analysis of this data evaluating the predictive capacity of adapted MEWS, qSOFA and UVA scores for in-hospital mortality in this population.

Methods

Study oversight

The Institutional Review Board of the University of Rwanda, College of Medicine and Health Sciences in Kigali, Rwanda and the Committee on Clinical Investigations at Beth Israel Deaconess Medical Center (BIDMC) in Boston, Massachusetts approved the study. Verbal consent for participation was obtained using a script in the participant’s primary language.

Patient and public involvement

This research was performed without explicit patient feedback on the design or implementation. Results will be available to the public through open access publication.

Setting

The study took place at the University Teaching Hospital of Kigali. The hospital is a public academic tertiary referral hospital in Kigali, Rwanda. It is one of three public referral hospitals in a country of approximately 12 million people, with 560 total beds including a 35-bed adult emergency department, a 7-bed ICU, a 4-bed step-down unit and approximately 12 000 admissions each year.

Inclusion criteria and data collection

We prospectively enrolled all hospitalised adult patients (age ≥15 years, the hospital’s cut-off for adult hospital ward admission) with suspected infection between 25 January 2017 and 14 August 2017 as part of a study examining antimicrobial resistance patterns.²⁴ All hospitalised patients were screened for inclusion criteria each day of their hospitalisation. We recorded the number of patients screened each day in each area of the hospital; we did not record the number of unique patients screened over the entire study period. Patients met inclusion criteria if they had temperature ≤35.0°C or ≥38.0°C and suspected infection, underwent surgery for an infectious process or had a positive microbial culture collected by the clinical team. For those who met inclusion criteria and provided consent, demographic and clinical data needed for each of the scores were collected at one time point from each participant’s chart by study research assistants. Vital sign and mental status data to include in the models were collected at the time of fever or hypothermia, the time of surgery or the time of culture sample collection, depending on the inclusion criteria met for each participant. For patients who met more than one inclusion criteria, the time point for clinical data collection was based on the first inclusion criteria met. Participants were followed through hospital discharge to determine length of stay and in-hospital mortality. All coded data were entered into a secure online database, REDCap (Vanderbilt University, Nashville, Tennessee), which was hosted by BIDMC.

Definitions

MEWS includes five variables, with scores between 0 and 3 assigned for each variable²⁰ table 1. It yields a maximum score of 14, with a score >4 considered to be high risk for mortality in prior studies.²⁰ Because we collected altered mental status as a binary variable (present or not), we adapted this variable in the MEWS score to be 0 for normal mental status and 2 for any altered mental status, rather than a range of severity of altered mental statuses from 0 to 3. qSOFA includes three variables, with 1 point given to each abnormal value, a maximum score of 3 and ≥2 considered high risk.¹⁵ UVA includes seven variables, with variable points given for each abnormality. It yields a maximum score of 13, with >4 considered high risk based on its derivation study.¹³

Table 1.

Variables and values in adapted MEWS, qSOFA and UVA scores

	Adapted MEWS*		qSOFA		UVA
	Cut-off	Points	Cut-off	Points	Cut-off	Points
Respiratory rate (breaths/min)	15–20	1	≥22	1	≥30	1
	21–29 or <9	2
	≥30	3
Altered mental status (Glasgow Coma Scale <15)	Present	2	Present	1	Present	4
Systolic blood pressure (mm Hg)	81–100	1	≤100	1	<90	1
	71–80 or ≥200	2
	≤70	3
Temperature (°C)	≥38.5	1			<36	2
	<35	2
Heart rate (beats/min)	101–110 or 41–50	1			≥120	1
	111–129 or <40	2
	≥130	3
Oxygen saturation (%)					<92	2
HIV seropositivity					Present	2

*The adaptation to the MEWS score pertains to the altered mental status score. In the original MEWS, 0 point was assigned for alert patients, 1 if they reacted to voice, 2 if they reacted to pain and 3 if they were unresponsive. In our adapted MEWS, we assign 0 point for an alert patient and 2 for a patient with any altered mental status.

GCS, Glasgow Coma Scale; MEWS, modified early warning score; qSOFA, quick Sequential (Sepsis-Related) Organ Failure Assessment; UVA, Universal Vital Assessment.

To replicate the methods for predictive validity in the original qSOFA and qSOFA LMIC validation studies,^{15 25} we also calculated a baseline risk model to stratify the population, using the same variables used in these studies: age, sex, HIV status and hospital transfer status (whether the patient had been transferred from another facility).

Data analysis

The primary outcome of interest was in-hospital mortality. The sample size was determined based on adequate power for the antimicrobial resistance study from which this cohort was taken, and is described in the methods of that study.²⁴ Adapted MEWS, qSOFA and UVA scores were calculated for all enrolled participants. Missing data were assumed to be within normal range, with no additional points assigned. Data are presented as median (IQR) or frequency (proportion) depending on variable type. Normality was assessed with the Shapiro-Wilk test. Demographic differences between survivors and non-survivors were assessed with a Wilcoxon rank-sum test, χ² or Fisher’s exact test, as appropriate. Sensitivity, specificity, positive and negative predictive values for the previously reported cut-offs for each score are reported. Separate unadjusted logistic regression models were used to generate ORs and 95% CIs for adapted MEWS, qSOFA and UVA. Multivariable logistic regression models using the four variables noted above were calculated for the baseline risk model.

We used the predicted probabilities from our baseline risk model to stratify our results into risk quartiles, presenting ORs and 95% CIs for adapted MEWS, qSOFA and UVA with their previously defined cut-offs separately, as was done in the original LMIC cohort qSOFA study.¹⁵ We calculated the discriminative ability of adapted MEWS, qSOFA and UVA as continuous variables and found the area under the receiver operating characteristic (AUROC) curves for each of these models. We also calculated the discriminative ability of the three scores as continuous variables in models with baseline risk adjustment.

Data analysis was performed using SAS 9.4 (SAS Institute, Cary, North Carolina) with two-sided p values <0.05 considered statistically significant.

Results

We screened every patient in the hospital for suspected infection each day of the study period, for a total of 19 178 patient days screened. We enrolled 647 of the 648 unique patients who met our criteria for suspected infection; the only exclusion was one patient who met study criteria, but declined enrollment. Within this study population, 497 participants (76.8%) had hypo or hyperthermia and suspected infection, 308 participants (47.6%) underwent surgery for an infectious process, and 273 participants (42.2%) had a positive microbial culture (online supplemental figure 1). The median age was 35 years (IQR 27-51) and 347 (53.6%) of participants were male (table 2). Known pre-existing comorbidities were present in 143 (22.1%) of participants, and 68 (10.5%) of participants were known to be HIV positive.

Table 2.

Baseline characteristics of patients admitted with suspected infection

	Total n=647	Survivors n=530	Non-survivors n=117	P value
Demographics
Age, median (IQR)	35.0 (27.0, 51.0)	35.0 (27.0, 51.0)	36.0 (27.0, 56.0)	0.46
Male sex, n (%)	347 (53.6)	273 (51.5)	74 (63.2)	0.02
HIV positive, n (%)	68 (10.5)	52 (9.8)	16 (13.7)	0.22
Other known pre-existing co-morbidity*, n (%)	143 (22.1)	106 (20.0)	37 (31.6)	0.01
Any positive bacterial culture, n (%)	273 (42.2)	223 (42.1)	50 (42.7)	0.90
Transferred from an outside hospital	414 (64.0)	342 (64.5)	72 (61.5)	0.54
Adapted† MEWS components
Respiratory rate, beats/min				0.0002
9–14	72 (11.1)	51 (9.6)	21 (17.9)
15–20	417 (64.4)	361 (68.1)	56 (47.9)
21–29 or <9	122 (18.9)	94 (17.7)	28 (23.9)
≥30	36 (5.6)	24 (4.5)	12 (10.3)
Altered mental status	150 (23.2)	92 (17.4)	58 (49.6)	<0.0001
Systolic blood pressure, mm Hg				0.13
100–199	533 (82.4)	437 (82.4)	96 (82.0)
81–100	97 (15.0)	81 (15.3)	16 (13.7)
71–80 or ≥200	12 (1.8)	10 (1.9)	2 (1.7)
≤70	5 (0.8)	2 (0.4)	3 (2.6)
Temperature				0.002
≥38.5°C	309 (47.8)	238 (44.9)	71 (60.7)
35°C–38.4°C	338 (52.2)	292 (55.1)	46 (39.3)
<35°C	0 (0)	0 (0)	0 (0)
Heart rate, beats/min				<0.0001
51–100	286 (44.2)	257 (48.5)	29 (24.8)
101–110 or 41–50	98 (15.1)	76 (14.3)	22 (18.8)
111–129 or <40	177 (27.4)	136 (25.7)	41 (35.0)
≥130	86 (13.3)	61 (11.5)	25 (21.4)
Adapted MEWS >4	192 (29.7)	133 (25.1)	59 (50.4)	<0.0001
qSOFA components
Altered mental status	150 (23.2)	92 (17.4)	58 (49.6)	<0.0001
Systolic blood pressure ≤100	112 (17.3)	91 (17.2)	21 (17.9)	0.84
Respiratory rate ≥22	147 (22.7)	110 (20.7)	37 (31.6)	0.01
qSOFA ≥2	81 (12.5)	52 (9.8)	29 (24.8)	<0.0001
UVA components
Temperature <36°C	12 (1.8)	12 (2.3)	0 (0)	0.10
Heart rate ≥120	175 (27.0)	129 (24.3)	46 (39.3)	0.001
Respiratory rate ≥30	37 (5.7)	25 (4.7)	12 (10.3)	0.02
Systolic blood pressure <90 mm Hg	37 (5.7)	29 (5.5)	8 (6.8)	0.56
Oxygen saturation <92%	149 (23.0)	118 (22.3)	31 (26.5)	0.33
Altered mental status	150 (23.2)	92 (17.4)	58 (49.6)	<0.0001
HIV positive	68 (10.5)	52 (9.8)	16 (13.7)	0.22
UVA >4	80 (12.4)	47 (8.9)	33 (28.2)	<0.0001

*Includes patients who had any of the following documented co-morbidities: diabetes, hypertension, tuberculosis, cancer and/or severe malnutrition.

†The adaption to the MEWS score pertains to the altered mental status score. In the original MEWS, 0 point was assigned for alert patients, 1 if they reacted to voice, 2 if they reacted to pain and 3 if they were unresponsive. In our adapted MEWS, we assign 0 point for an alert patient and 2 for a patient with any altered mental status.

MEWS, modified early warning score; qSOFA, quick Sequential (Sepsis-Related) Organ Failure Assessment; UVA, Universal Vital Assessment.

In the full cohort, the in-hospital mortality rate was 18.1% (117 of 647 participants). An adapted MEWS score of >4 was present in 29.7% (192/647) of cases, qSOFA score of >2 was present in 12.5% (81/647) of cases, while a UVA score >4 was present in 12.4% (80/647) of cases (table 2). The full distribution for each score is shown in figure 1, with adapted MEWS range 0–10, median 3, IQR 2- 5; qSOFA range 0–3, median 0, IQR 0- 1; and UVA range 0–8, median 2, IQR 0- 4. The proportion of data that was missing for the components of the scores ranged from 0% to 11.7% (online supplemental table 1).

Figure 1.

Distribution of patients (A) and observed mortality (B) with standard errors by adapted modified early warning score (MEWS), quick Sequential (Sepsis-Related) Organ Failure Assessment (qSOFA) score and Universal Vital Assessment (UVA) among patients with suspected infection.

The sensitivity and specificity of the adapted MEWS score with cut-off value >4 to predict in-hospital mortality were 50.4% (59/117) and 74.9% (397/530), respectively (table 3). The sensitivity and specificity of qSOFA with cut-off value ≥2 were 24.8% (29/117) and 90.4% (479/530), respectively. For the UVA score with cut-off value >4, the sensitivity and specificity were 28.2% (33/117) and 91.1% (483/530), respectively. The sensitivity, specificity, and positive and negative predictive values for each score using the full range of possible cut-off values are presented in online supplemental table 2. The unadjusted ORs for adapted MEWS >4, qSOFA ≥2 and UVA >4 were 3.04 (95% CI 2.01 to 4.59), 3.10 (95% CI 1.86 to 5.15) and 4.04 (95% CI 2.44 to 6.67), respectively. The OR for hospital mortality was most often >1 for each binary score within each quartile of baseline risk, though the 95% CI for the OR crossed 1 for qSOFA and UVA in quartile 4, and for adapted MEWS in quartile 1 (online supplemental figure 2).

Table 3.

Predictive capacity of adapted MEWS, qSOFA and UVA scores

	Adapted MEWS* >4	qSOFA ≥2	UVA >4
Unadjusted
Sensitivity	50.4	24.8	28.2
Specificity	74.9	90.4	91.1
Positive predictive value	30.7	36.2	41.2
Negative predictive value	87.2	84.5	85.2
OR (95% CI)	3.04 (2.01 to 4.59)	3.10 (1.86 to 5.15)	4.04 (2.44 to 6.67)

*The adaptation to the MEWS score pertains to the altered mental status score. In the original MEWS, 0 point was assigned for alert patients, 1 if they reacted to voice, 2 if they reacted to pain, and three if they were unresponsive. In our adapted MEWS, we assign 0 point for an alert patient and 2 for a patient with any altered mental status.

MEWS, modified early warning score; qSOFA, quick Sequential (Sepsis-Related) Organ Failure Assessment; UVA, Universal Vital Assessment.

Overall, increasing scores for adapted MEWS, qSOFA and UVA corresponded with increasing mortality, though this was not true for every 1 point increase in adapted MEWS (figure 1). For each 1 point increase in score as a continuous variable, the unadjusted ORs were: adapted MEWS 1.41 (95% CI 1.28 to 1.56), qSOFA 2.20 (95% CI 1.68 to 2.88) and UVA 1.46 (1.32 to 1.61) (online supplemental table 3).

The AUROC for each score as a continuous variable was: adapted MEWS 0.69 (95% CI 0.64 to 0.74), qSOFA 0.65 (95% CI 0.60 to 0.70) and UVA 0.71 (95% CI 0.66 to 0.76) (figure 2, online supplemental table 3). There was no statistically significant difference between the AUROCs for the three scores as pairwise comparisons: UVA versus adapted MEWS p=0.57; UVA versus qSOFA p=0.09; and adapted MEWS versus qSOFA p=0.26.

Figure 2.

Receiver operating characteristic curves for adapted MEWS, qSOFA or UVA criteria as continuous variables. MEWS, modified early warning score; qSOFA, quick Sequential (Sepsis-Related) Organ Failure Assessment; UVA, Universal Vital Assessment.

The AUROC for the baseline risk model was 0.57 (95% CI 0.52 to 0.63). Adding adapted MEWS, qSOFA and UVA as continuous variables to the baseline risk model changed the AUROC to 0.72 (95% CI 0.66 to 0.77), 0.68 (95% CI 0.63 to 0.74), and 0.72 (95% CI 0.66 to 0.77), respectively (online supplemental figure 3, table 4).

Discussion

In a prospective study of 647 patients with suspected infection in a Rwandan tertiary referral hospital, we found that the adapted MEWS, qSOFA and UVA scores had modest ability to predict mortality. Using previously defined cut-offs for each of the scores, adapted MEWS had sensitivity and specificity of 50% and 75%, respectively, while qSOFA and UVA were less sensitive, but had higher specificity (25% and 90%, respectively, for qSOFA and 28% and 91%, respectively, for UVA). AUROCs for the continuous scores ranged from 0.65 to 0.71, with no AUROC for continuous score demonstrating statistically significant superiority to another.

We presented the performance of the three scores using the continuous scores, continuous scores in addition to a baseline risk model, and binary scores using previously defined cut-off values. Depending on the intended use of the scores, any of these might be appropriate in understanding the adequacy of the score. For quality improvement and research comparisons, the AUROC is a useful single value in deciding whether a model can help determine differences in SOI between cohorts.¹³ For determining the predictive validity of a definition of sepsis, assessing mortality risk above baseline risk may be most appropriate.¹⁵ For deciding who needs escalation of care, the sensitivity and specificity with a particular cut-off value is likely to be more important in judging the adequacy of the model.¹¹ Particularly in the latter example, which is the most often cited for scores in LMICs, care must be taken in how the scores are used for individual clinical decision-making since low sensitivity could lead to patients who need additional care being missed and low specificity could lead to attempts at using scarce resources for a relatively large population.^{11 26 27}

Our study has several strengths. We looked at adult patients across the entire hospital rather than the ICU alone,1 2 7 10 16 17 which is particularly important in settings where many critically ill patients remain outside the ICU due to limited ICU capacity.¹³ We also analysed the score performances in multiple ways: as continuous scores, continuous scores added to baseline risk and as dichotomous values. In addition, the retrospective multisite LMIC qSOFA validation included a cohort from the emergency department of our hospital;¹⁵ our cohort and that cohort showed similarly modest predictive capacity for the continuous qSOFA score without baseline model, providing criterion validity to our results (AUROC 0.55 in the multisite study and 0.65 in this study). Finally, other than one small study confined to emergency department patients and with a low (5%) mortality rate,²³ our study is the first to assess the UVA score outside of its LMIC derivation cohort.¹³

Our study also has several limitations. First, we conducted it in a single tertiary care hospital in sub-Saharan Africa, so its results may not be generalisable. Even more complex SOI scores derived from much larger populations, such as the APACHE score for ICU patients in HICs, have quite variable performance, requiring recalibration for different populations and over time in the same population.^{12 28 29} It is reasonable to expect that variations in patient characteristics, management systems, and resources across hospitals would translate to different predictive capacities of scores across hospitals. Of note, in the retrospective study of qSOFA in nine LMIC cohorts, the AUROC for all combined sites without the baseline model was 0.69, but the AUROC range for individual sites was wide, from 0.55 to 0.81.¹⁵ Second, the variables used to calculate the scores for patients in our study were recorded from different time points (time of fever, operation or culture sample retrieval) depending on the inclusion criteria each participant met for the study. This likely simulates how the scores might be used in practice; however, it is certainly possible the scores would perform better with more consistent data collection time points. We may also have a survivor bias of unknown direction since patients who died rapidly after admission to the hospital before they could be screened, or who died before infection was suspected, were not included. Third, oxygen saturation was included as a variable, without oxygen delivery; this was a feature of the UVA score design, but it nonetheless seems likely that oxygen saturation without oxygen delivery will be more limited in its predictive power. Fourth, we had some missing data, up to 11.7% for oxygen saturation, for which we assumed normal values; however, the missingness was relatively low compared with many other LMIC studies^{1 12} and reflects reasonable real world data availability. Fifth, our positive culture rate of 42.2% in this population is likely artificially high given that one of the inclusion criteria for the study was a positive culture. Finally, we were unable to evaluate the original MEWS score since we did not have detailed mental status data. We used an adapted MEWS with a binary version of the mental status variable without prior validation of this adaptation; these scores could have been overestimated or underestimated and, therefore, impacted the score’s capacity to differentiate participants.

Conclusion

Our study found modest predictive power of adjusted MEWS, qSOFA and UVA scores in our cohort of inpatients with suspected infection at a Rwandan tertiary hospital. These modest predictive performances must be acknowledged if these scores are to be considered for use in research comparisons, quality improvement or clinical decision-making.