Comparison of Quick Sequential Organ Failure Assessment and Modified Systemic Inflammatory Response Syndrome Criteria in a Lower Middle Income Setting

Abstract

Introduction

Quick Sequential Organ Failure Assessment (qSOFA) is potentially feasible tool to identify risk of deteriorating in the context of infection for to use in resource limited settings.

Purpose

To compare the discriminative ability of qSOFA and a simplified systemic inflammatory response syndrome (SIRS) score to detect deterioration in patients admitted with infection.

Methods

Observational study conducted at District General Hospital Monaragala, Sri Lanka, utilising bedside available observations extracted from healthcare records. Discrimination was evaluated using area under the receiver operating curve (AUROC). 15,577 consecutive adult ( ≥ 18 years) admissions were considered. Patients classifi ed as having infection per ICD-10 diagnostic coding were included.

Results

Both scores were evaluated for their ability to discriminate patients at risk of death or a composite adverse outcome (death, cardiac arrest, intensive care unit [ICU], admission or critical care transfer). 1844 admissions (11.8%) were due to infections with 20 deaths (1.1%), 29 ICU admissions (1.6%), 30 cardiac arrests and 9 clinical transfers to a tertiary hospital (0.5%). Sixty-seven (3.6%) patients experienced at least one event. Complete datasets were available for qSOFA in 1238 (67.14%) and for simplified SIRS (mSIRS) in 1628 (88.29%) admissions. Mean (SD) qSOFA score and mSIRS score at admission were 0.58 (0.69) and 0.66 (0.79) respectively. Both demonstrated poor discrimination for predicting adverse outcome AUROC = 0.625; 95% CI, 0.56-0.69 and AUROC = 0.615; 95% CI, 0.55-0.69 respectively) with no significant difference (p value = 0.74). Similarly, both systems had poor discrimination for predicting deaths (AUROC = 0.685; 95% CI, 0.55-0.82 and AUROC = 0.629; 95% CI, 0.50-0.76 respectively) with no statistically signifi cant difference (p value = 0.31).

Conclusions

qSOFA at admission had poor discrimination and was not superior to the bedside observations featured in SIRS. Availability of observations, especially for mentation, is poor in these settings and requires strategies to improve reporting.

Introduction

Sepsis is an important cause of morbidity and mortality worldwide and the primary cause of death following infection. [1] The burden of sepsis is higher and outcomes poorer in low and middle income countries (LMIC’s) when compared to high income countries (HIC). [2,3] For instance, mortality attributable to infectious diseases is considerably higher in developing countries (29% in South Asia, 56% in sub- Saharan Africa) compared to the US and UK (6%). [4] Despite ongoing efforts, reliable information on sepsis epidemiology, outcomes and the socio-economic impact from LMIC’s remains scarce. [2,5]

The third International Consensus Definitions Task Force defined sepsis as a “life-threatening organ dysfunction” due to a dysregulated host response to infection. [1] The quick Sequential Organ Failure on Admission (qSOFA) score is advocated for its role in nonintensive care unit (ICU) settings in stratifying patients “at risk” of sepsis or physiological deterioration. [1,6] Early identification of such patients is particularly important outside HIC’s where access to and availability of critical care services are limited and outreach services that may assist ward based teams with detection and escalation to critical care are non-existent. [7,8]

qSOFA is potentially more feasible to use in resource limited settings due to the relative simplicity of information required for scoring. [1,3] However, it was developed retrospectively from patient databases exclusively from adults in HICs, predominantly from the United States and may thus be not applicable to a LMIC setting where different cultures of observation reporting, clinical decision making exist. [9] Whilst the sequale of sepsis is on the whole universal, the causation, stage of pathophysiology at presentation to hospital and the availability of equipment to support clinicians in diagnosis is varied. [2] Patients with infection in LMIC settings often present to hospital late, and can thus exhibit different pathophysiological features, either due to difficulties in accessing health-care or due to seeking indigenous treatments prior to admission. [10] It is increasingly recognised that recommendations from HIC cannot be directly transposed to low income country (LIC) and LMICs. [11] Therefore the feasibility of using such tools and their ability to identify those with infections at risk of adverse outcomes, must be evaluated in resource limited settings. [1,4] This study aims to compare the discriminative ability of qSOFA and a modified systemic inflammatory response syndrome (SIRS) (based on bedside observations) score at the time of admission to detect deterioration in patients with infection at a district general hospital in a LMIC.

Methods

A prospectively collected dataset from an ongoing early warning system (EWS) implementation study at a District General Hospital (Moneragala) in Sri Lanka was used for this comparative evaluation. The cohort was constructed using the dataset from all consecutive adult (age ≥ 18 years) hospital encounters over a six-month period in 2015. Diagnoses were coded using ICD-10 system [12] and only patients suspected of having an infection were included in this analysis. [13,14]

Systolic blood pressure, respiratory rate and mentation (“alert, verbal, pain, unresponsive”, AVPU) were extracted in order to calculate qSOFA. ‘AVPU’ was used to assess altered mentation as Glasgow coma score (GCS) was not in use in this hospital. [9] A value other than “A” was considered to be altered mentation. [15] Laboratory tests such as PaCO₂ and white blood cell count are rarely available at the time of hospital admission in most LMIC’s. [8] Comparison was thus made with a simplified SIRS (mSIRS) consisting of temperature, respiratory rate and heart rate. Unavailability of observations is not limited to the LMIC’s, with similar challenges reported in settings such as the United States. [16]

Observations collected at hospital admission were extracted on a daily basis from patient notes by trained data collectors and recorded using an opensource electronic system. All patients were followed up daily until hospital discharge. The frequency of missing data was determined for each component of qSOFA and mSIRS. In-hospital death, cardiac arrest, emergency admission to ICU and clinical transfer to another ICU were considered as adverse outcomes. The latter three were included as they are associated with high mortality in this setting. [9,17]

The odds ratios for death and any of the 4 adverse events for the individual components of qSOFA, mSIRS and for qSOFA scores of two and above (qSOFA positive vs. negative), and for mSIRS scores of two and above (mSIRS positive vs. negative) were calculated. Discrimination was assessed by the AUROC for all events and for death separately for qSOFA and mSIRS.

All significance tests used a 2-sided P ≤ 05. AUROCs were considered to be poor at 0.6 to 0.7, adequate at 0.7 to 0.8, good at 0.8 to 0.9, and excellent at 0.9 or higher. [18] Missing observations were imputed using three different approaches; as “normal” values with respect to qSOFA and mSIRS cut offs; as “abnormal” values for the same; and with the closest available inpatient observation for that patient. Calibration was assessed using Hosmer-Lemeshow goodness of fit test, and applied where discrimination was shown to be greater than 0.7. [19] All analyses were performed with STATA software, version 13.0. [20]

This dataset was generated, subsequent to ethical review and waiver of individual patient from the Ethical Review Committee of the Faculty of Medicine, University of Colombo, in order to develop and implement an early warning system (EWS) in Moneragala (district general hospital) DGH. This study was supported by the National Science Foundation, Sri Lanka and by the Mahidol Oxford Tropical Medicine Unit, Bangkok.

Results

There were a total of 15,577 encounters with 11.8% (1844) classified with ICD-10 diagnoses attributable to infection. Salient characteristics and outcomes of the patients are described in Table 1. Sixtythree percent of those with infections were male, compared to 38% in the non-infection group. Commonest ICD-10 diagnoses in those with infections are listed in Table 2.

Table 1. Characteristics of study population.

^aSD: standard deviation ^bIQR: inter quartile range ^cmSIRS: Simplified SIRS. ^dqSOFA: The quick Sequential Organ Failure on Admission

Characteristic	Infection	Non-infection
Total encounters (%)	1,844 (11.8)	13,733 (88.2)
Age mean (SDa)	43 (17.19)	43 (17.64)
Hospital LOS median (IQRb)	3 (3)	2 (2)
1 or more events	67 (3.63%)	514 (3.74%)
Death	20 (1.08%)	105 (0.76%)
Male (%)	1,165 (63.3)	5,191 (37.9)
mSIRSc
Mean (SDa)	0.66 (0.79)	0.37 (0.59)
Median (IQRb)	0 (1)	0 (1)
qSOFAd
Mean (SDa)	0.58 (0.69)	0.43 (0.57)
Median (IQRb)	0 (1)	0 (1)

Table 2. Common ICD diagnoses for patients with infection in Sri Lanka.

Diagnosis as per ICD-10	Frequency (%)
Viral infection , (unspecified)	269 (14.59)
Hepatitis A (without hepatic coma)	247 (13.39)
Unspecified acute lower respiratory infection	171 (9.27)
Urinary tract infection, site not specified	149 (8.08)
Acute bronchitis due to other specified organisms	140 (7.59)
Cellulitis of other parts of limb	126 (6.83)
Other gastroenteritis and colitis of infectious and unspecified origin	93 (5.04)
Cutaneous abscess, furuncle and carbuncle of limb	84 (4.56)
Leptospirosis, unspecified	75 (4.07)
Cutaneous abscess, furuncle and carbuncle of buttock	52 (2.82)

Complete datasets were available for qSOFA in 1238 (67.1%) and for mSIRS in 1628 (88.29%) admissions, resulting in 606 (32.9%) and 216 (10.9%) incomplete datasets for qSOFA and mSIRS respectively. For qSOFA, 545 were missing 1 variable, 45 were missing 2 variables, and 16 patients were missing all 3 variables on admission. “AVPU” was the most commonly missed variable in 549 patients (29.8%).

The mean SIRS score was 0.65 (SD 0.78) and mean qSOFA score was 0.57 (SD 0.69) when the missing values were imputed as normal. The mean SIRS score was 0.69 (SD 0.83) and qSOFA was 0.66 (SD 0.80) when imputed as abnormal. When the next available observation for each patient was used, mean mSIRS was 0.66 (SD 0.79) and qSOFA was 0.58 (SD 0.69) when imputed with the next available value for that patient (Table 5). The distributions of mSIRs and qSOFA scores were shown in Figure 1.

Table 5. Mean the quick Sequential Organ Failure on Admission (qSOFA) and simplified SIRS (mSIRS) scores and Standard deviation, based on patients observations on admission. Missing parameters imputated with normal, abnormal and next available parameter imputation.

Score	Mean Scores and standard deviation (SD)
	normal value imputation	abnormal value imputation	next available parameter imputation
qSOFA	0.57 (SD 0.69)	0.66 (SD 0.80)	0.58 (SD 0.69)
mSIRS	0.65 (SD 0.78)	0.69 (SD 0.83)	0.66 (SD 0.79)

Figure 1. Distribution of simplfied SIRS (mSIRS) and the quick Sequential Organ Failure on Admission (qSOFA) scores.

Sixty-seven (3.6%) patients experienced at least one event (Table 1). Odds ratios for qSOFA components for adverse events (when missing values were imputed as normal) are shown in Table 3. Relative to qSOFA scores lower than 2, encounters with 2 or higher were 3.19 (95% CI 1.79, 5.69) times more likely to experience an adverse event and 6.59 (95% CI 2.73, 15.92) times more likely to die. Similarly, for modified SIRS tool, compared to scores lower than 2, encounters with 2 or higher were 3.16 (95% CI 1.87, 5.33) times more likely to experience an adverse event and 4.14 (95% CI 1.72, 9.96) times more likely to die.

Table 3. Odds ratios for the quick Sequential Organ Failure on Admission (qSOFA) cut-offs for deaths and adverse events.

Odds ratio (events)	Odds ratio (death)	Events	Deaths	Component
1.9335 (1.1871 , 3.1498)	2.3400 (0.9935 , 5.5118)	30 (44.8%)	10 (50%)	rr ≥ 22 (qSOFA)
2.2839 (1.3948 , 3.7402)	3.8136 (1.6110 , 9.02606)	28 (41.8%)	11 (55%)	sys ≤ 100 (qSOFA)
4.9909 (2.2987 , 10.8606)	11.78 (4.2872 , 32.5223)	8 (11.9%)	5 (25%)	AVPU ≤ A (qSOFA)
1.0232 (0.5074 , 2.0649)	0.3425 (0 , 2.0228)	9 (13.4%)	1 (5%)	tem < 36 or tem > 38 (SIRS)
2.9808 (1.8253 , 4.8681)	3.5425(1.5020 , 8.3553)	30 (44.8%)	10 (50%)	HR > 90 (SIRS)
1.9336 (1.1871 , 3.1498)	2.3401 (0.9935 , 5.5119)	30 (44.8%)	10 (50%)	RR > 20 (SIRS)

Discriminatory ability of qSOFA and mSIRS calculated using AUROC are described in Table 4. qSOFA and modified SIRS both demonstrated poor discrimination for predicting events with no significant difference between the two systems in all three methods of imputation. Discrimination for predicting deaths for qSOFA and mSIRS were also poor, with no significant difference between the two scores in all three methods of imputation (Table 4). Calibration was not assessed as discrimination was below 0.7 in all instances. [19]

Table 4. Evaluation of discrimination of the quick Sequential Organ Failure on Admission (qSOFA) and simplified SIRS (mSIRS) to predict death and events based upon admission observations.

Score	Outcome	AUROC (CI)
		normal imputed value	abnormal imputed value	next available parameter
qSOFA	events	0.6206 (0.5525 , 0.6887)	0.6224 (0.5533 , 0.6914)	0.6254; 95% CI, 0.5580-0.6928
mSIRS	events	0.6086 (0.5392 , 0.6780)	0.6145 (0.5441 , 0.6850)	0.6161; 95% CI, 0.5468-0.6854
p value	p value	0.67	0.78	0.74
qSOFA	death	0.6863 (0.5540 , 0.8186)	0.6874 (0.5551, 0.8197)	0.6845 (95% CI, 0.5520-0.8171)
mSIRS	death	0.6310 (0.5016 , 0.7604)	0.6506 (0.5156, 0.7856)	0.6285 (95% CI, 0.4989-0.7581)
p value	p value	0.31	0.49	0.31

Discussion

This comparative study between qSOFA and the bedside observations featured by SIRS criteria in a LMIC, demonstrates the poor ability of both systems to discriminate, between patients with infection experiencing an adverse outcome and those who do not, at the time of admission. In addition, the availability of observations, especially measures of mentation, remain limited, highlighting the challenges of validating and implementing such tools in LMIC settings. The sepsis 3.0 recommendations explicitly set out to produce criteria to detect sepsis which would be universally applicable, for example by excluding lactate. The use of SIRS to define sepsis has been criticised for its poor discrimination and concurrent validity. [21] In this study however, qSOFA was also poor in discriminating between those at risk of adverse outcomes and those without, and was no better than mSIRS. Similarly, qSOFA performance remained inadequate even when death was considered as the only adverse outcome. These findings are echoed by a recent comparison of qSOFA, SIRS and EWSs in the US which reported similarly poor discrimination of qSOFA and SIRS, and found aggregate weighted EWSs to have greater discrimination by comparison. [22]

This paper details a single centre evaluation from a DGH in a LMIC with relatively low acuity for this setting. [4,5] Perhaps uniquely, this relatively large electronic dataset collected on a daily basis for the development and implementation of an EWS (more output expected in 2017), allowed assessment of the applicability of the qSOFA tool in this setting, albeit using AVPU instead of GCS. It also provides a snapshot of epidemiology and presentation of sepsis and of clinical observation reporting behaviours in this setting.

Just over ten percent (11.8%) of patients in this dataset had conditions which can be considered as infection per the ICD-10 classification. This proportion is similar to the proportion with infection in the US datasets used in sepsis 3.0. [1] However, the commonest infections in this LMIC are very different from the datasets in the US used for qSOFA development. Casemix, pathophysiology (as demonstrated in Table 2) and stage of physiological sequale at time of presentation to hospital may differ widely compared to HIC’s. The availability of resources, staffing and facilities for diagnosis, monitoring and treatment is vastly different in lower middle income countries compared to the US[4,10] though information for a more formal comparison was not available. No further information on the severity of illness (eg qSOFA), baseline risk (eg Charlson index) were collected, as robust information is unlikely to be available for a large majority of the patients. These differences emphasise the need for validation of tools against setting specific epidemiology prior to advocating local application (Table 2), and provide a timely reminder that untested implementation can be potentially detrimental. [23,24]

Limited availability of robust and accessible healthcare data is a common global problem. [3,22] This is exaggerated outside the ICU, where datasets in both HIC and resource limited settings are often incomplete. In fact, authors of the Sepsis 3 recommendations reported limitations in observation availability for the US dataset. In LMIC settings the problem is even greater; resource availability including staff, healthcare culture, paper based records and clinician bias are reported contributors. [3,25,26] In the current dataset, assessment of mentation (AVPU) was most problematic with only 67.14% patients had all parameters needed for modified qSOFA calculation, in contrast, with 88% for the bedside SIRS criteria. While it may be argued that mentation was not recorded by the clinical staff as it was normal, the extent of missing data complicates interpretation of the applicability of tools such as qSOFA. For the purposes of this study it was possible to mitigate for the missing values by imputing with normal, abnormal and next available observations. [22] The uniformly poor discriminations for all three (effectively categorically) imputed versions of the dataset for both qSOFA and SIRS allowed appropriately robust conclusions regarding their unsuitability to be made in this instance. A research question hinging on the performance of several continuous variables may have led to far greater uncertainty.

AVPU is far more widely available in this setting when compared to GCS and should thus be formally replaced as the measure mentation in LMIC settings, to enable wider testing of qSOFA applicability. [27] The absence of formal observation charts where nurses and doctors jointly record observations and the lack of electronic observation systems contribute to the availability and accessibility of these vital signs in hospitalised patients in this setting. Further exploration of the barriers and challenges in assessing mentation in LMIC, especially by nursing staff, and the impact of any remedial measures including training, are necessary.

The dataset used for this comparison is from an ongoing study aimed at collaboratively developing and implementing a feasible EWS system in a LMIC. [17] Any such system will need to perform adequately in patients with and without infection. As part of the ongoing implementation study, work is underway to increase the availability and reporting of physiological observations and the recognition of patients at risk of sepsis based upon clinical assessment. This study demonstrates, that universal availability of even the simplest bedside observations should not necessarily be assumed in LMIC’s and reinforces the need to improve clinical training for frontline staff, a focus of ongoing efforts by global health groups, including in Sri Lanka, in the detection and management of the “at-risk” patient and in establishing systems for accurate accessible data capture. [28,29]

Limitations

Comparison was limited to the readily available bedside parameters of SIRS and a modified qSOFA due to the unavailability of GCS. Although AVPU was used in place of GCS, AVPU has been shown to be comparable to more complicated neurological assessment tools in determining drop in conscious level. [30] This study is based on a single centre (with relatively low mortality compared to other estimates from LMIC’s [29] and a broader evaluation is underway in Sri Lanka to ascertain wider applicability. The limitation of availability of clinical information in this settings precluded the study team from evaluating feasibility of tools to define sepsis and septic shock. [1]

Conclusion

qSOFA and physiological parameters of SIRS have poor ability to discriminate risk of adverse events or deaths in patients admitted with infection in this LMIC setting. Limited availability of clinical information at the bedside hinders the application of risk stratification and diagnostic tools. Investment is needed to explore the barriers and enablers to observations reporting and is an important step towards improving uptake of or development of tools to aid clinicians in the recognition and management of sepsis in LMIC’s. Tools such as qSOFA which use readily available bedside parameters, though limited availability of a measure of mentation was an acknowledged difficulty in this setting, are likely to be more evaluable for applicability in LMIC settings.

Conflicts of Interest statement

The authors declared no conflict of interest .

Acknowledgments

The authors acknowledge the contribution of staff of Monaragla DGH for data collection.