External validation of the quick Sequential Organ Failure Assessment score for mortality and bacteraemia risk evaluation in Japanese patients undergoing haemodialysis: a retrospective multicentre cohort study

Abstract

Objectives

We aimed to examine the validity of the quick Sequential Organ Failure Assessment (qSOFA) score for mortality and bacteraemia risk assessment in Japanese haemodialysis patients.

Design

This is a retrospective multicentre cohort study.

Setting

The six participating hospitals are tertiary-care institutions that receive patients on an emergency basis and provide primary, secondary and tertiary care. The other participating hospital is a secondary-care institution that receives patients on an emergency basis and provides both primary and secondary care.

Participants

This study included haemodialysis outpatients admitted for bacteraemia suspicion, who had blood drawn for cultures within 48 hours of their initial admission.

Primary and secondary outcome measures

The primary outcome measure was overall in-hospital mortality. Secondary outcomes included 28-day in-hospital mortality and the incidence of bacteraemia diagnosed based on blood culture findings. The discrimination, calibration and test performance of the qSOFA score were assessed. Missing data were handled using multiple imputation.

Results

Among the 507 haemodialysis patients admitted with bacteraemia suspicion between August 2011 and July 2013, the overall in-hospital mortality was 14.6% (74/507), the 28-day in-hospital mortality was 11.1% (56/507) and the incidence of bacteraemia, defined as a positive blood culture, was 13.4% (68/507). For predicting in-hospital mortality among haemodialysis patients, the area under the receiver operating characteristic curve was 0.61 (95% CI 0.56–0.67) for a qSOFA score ≥2. The Hosmer-Lemeshow χ² statistics for the qSOFA score as a predictor of overall and 28-day in-hospital mortality were 5.72 (p=0.02) and 7.40 (p<0.01), respectively.

Conclusion

On external validation, the qSOFA score exhibited low diagnostic accuracy and miscalibration for in-hospital mortality and bacteraemia among haemodialysis patients.

Strengths and limitations of this study.

• This is the first study to assess the diagnostic performance of the quick Sequential Organ Failure Assessment (qSOFA) score for in-hospital mortality and bacteraemia among haemodialysis patients, according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis statement.

• We could not precisely determine the performance of the qSOFA score in haemodialysis patients with symptoms that did not warrant blood culture evaluation because we did not evaluate the reasons blood was drawn for culture.

• We used consecutive data of haemodialysis patients suspected of having bacteraemia, which is expected to increase the generalisability of our findings.

• Our cohort contains patients who used antibiotics during the week leading up to the hospital visit, which could have decreased infection-related mortality and decreased the rate of positive blood cultures.

• Our cohort was geographically and temporally different from the cohort used to derive the qSOFA criteria, which enabled us to perform a true external validation study.

Introduction

Patients undergoing haemodialysis are at high risk for bloodstream infections due to immunocompromised status and daily punctures required for vascular access.¹ Moreover, the morbidity and mortality of bacteraemia are higher among haemodialysis patients than in the general population,^2–10 as is the incidence of Staphylococcus aureus bloodstream infections.¹¹ Therefore, appropriate diagnosis and timely treatment of bacteraemia are of critical importance in haemodialysis patients.

While many risk stratification tools are available for the general population, their diagnostic accuracy is likely to differ when applied in specific populations. Adequate validation of population-specific diagnostic performance is particularly important in high-risk populations such as haemodialysis patients. For example, we previously reported that the Systemic Inflammatory Response Syndrome (SIRS) score has low sensitivity for predicting bloodstream infections in haemodialysis patients (SIRS score ≥2: sensitivity, 71.9%; specificity, 45.2%; positive likelihood ratio, 1.31; negative likelihood ratio, 0.62).¹² These previous findings suggested that the prediction criteria for bacteraemia or sepsis, which are well-established for the general population, might have different diagnostic accuracy among haemodialysis patients.

We also proposed a clinical prediction rule for bacteraemia among haemodialysis outpatients with suspicion of bacteraemia (BAC-HD).¹³ The BAC-HD score takes into account body temperature, heart rate, C-reactive protein levels, alkaline phosphatase levels and use of antibiotics within the week leading up to the assessment. A BAC-HD score ≥2 was useful for predicting bacteraemia in haemodialysis patients (sensitivity, 89.6%; specificity, 51.4%; positive likelihood ratio, 1.8; negative likelihood ratio, 0.2; area under the curve (AUC), 0.76).¹³

The quick Sequential Organ Failure Assessment (qSOFA) score was introduced as a novel risk-stratification tool intended for use outside the intensive care unit (ICU). The qSOFA score is based on three clinical criteria: systolic hypotension, defined as a systolic blood pressure ≤100 mm Hg; tachypnea, defined as a respiratory rate ≥22 breaths/min; and altered mentation.¹⁴ In a previous study, the qSOFA score showed predictive validity (area under the receiver operating characteristic curve, 0.81; 95% CI 0.80 to 0.82) for sepsis in non-ICU patients with suspected infection identified as the combination of antibiotics use and body fluid cultures.¹⁴ Several studies have been conducted to validate the diagnostic performance of the qSOFA score among patients in various settings or with specific comorbidities.^15–21 However, the validity of qSOFA for risk evaluation in haemodialysis patients has not been confirmed to date.

In the present study, we aimed to examine the external validity of qSOFA as an easy-to-use tool for rapid evaluation of the risk of in-hospital death and bacteraemia in patients undergoing haemodialysis.

Materials and methods

Study design and participants

Seven hospitals participated in this multicentre, retrospective cohort study of maintenance haemodialysis patients. The six participating hospitals are tertiary-care institutions that receive patients on an emergency basis and provide primary, secondary and tertiary care. The other participating hospital is a secondary-care institution that receives patients on an emergency basis and provides both primary and secondary care. The study results are reported in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.²²

The present study included consecutive haemodialysis patients with suspected bacteraemia who visited the outpatient department or emergency room between August 2011 and July 2013 and had blood drawn for cultures within 48 hours of their initial arrival at the hospital. The exclusion criteria of this study were as follows: age below 18 years; low frequency of haemodialysis (less than once per week); combination dialysis regimen (peritoneal dialysis and haemodialysis); admission within ≤2 weeks of haemodialysis initiation; and referral from another hospital.

Outcome measures

The primary outcome measure was the overall in-hospital mortality. Considering the findings of previous validation studies, 28-day in-hospital mortality was defined as the secondary outcome. Bacteraemia incidence was another secondary outcome measure in this study. Bacteraemia was diagnosed based on the results of blood cultures at the time of the patient’s visit. Specifically, the diagnosis of bacteraemia was made if the blood cultures were positive for any bacteria and there was no suspicion of contamination. Contamination was considered the most probable cause of positive blood culture results if only one of two sets of culture bottles was positive, or if all detected bacterial species were known to be common contaminants (ie, diphtheroids, Bacillus sp, Propionibacterium sp, micrococci, Corynebacterium sp, and coagulase-negative staphylococci). Finally, an external consensus panel of two physicians well trained in infectious diseases determined whether a culture was contaminated or not, based on the above definitions and their clinical expertise.

Method of measurement

The following data were extracted from the medical records: age; sex; dialysis vintage; cause of end-stage renal disease; vital signs at the time of the first visit, including body temperature, systolic blood pressure, pulse rate, respiratory rate, percutaneous oxygen saturation, Glasgow Coma Scale (GCS) score, and Japan Coma Scale (JCS) score^{23 24}; comorbidities; type of vascular access; history of bacteraemia; medication use including antibiotics use within the week leading up to the hospital visit; and laboratory data at the time of the hospital visit, including white blood cell count, platelet count, serum albumin levels and C-reactive protein levels.

A positive qSOFA result (qSOFA score ≥2) was defined in patients who fulfilled two or more of the following criteria at the same time: systolic blood pressure ≤100 mm Hg, respiratory rate ≥22 breaths/min and altered mentation. The qSOFA score ranges from 0 to 3, with each criterion being worth one point. The initial qSOFA scores were established according to the patients’ vital signs and mental status within 24 hours of arrival. Altered mentation was defined as a GCS score <13. If the JCS score was reported instead of the GCS score, the following equivalence was applied: a JCS score of 0 (alert) was considered to correspond to a GCS score of 15, while a JCS score of 300 (no motor response) was considered to correspond to a GCS score of 3.^{23 24} Converting JCS scores to GCS scores has not been validated. Thus, the other value of the JCS score was considered as missing data and handled using multiple imputation.

Statistical analysis

Data are presented as median values and IQRs for continuous variables, and as frequencies and percentages for categorical variables. The number of patients who had complete data for each qSOFA category is listed.

In the analysis of the discrimination, calibration and performance of the qSOFA, primary imputation was employed to handle missing values for covariates, assuming that data were missing at random. To impute the missing values, we constructed multiple regression models including variables that could potentially explain the missing data, as well as variables correlated with the outcome. The results obtained across 100 imputed data sets were combined by averaging, and SE were adjusted to reflect both within-imputation and between-imputation variability. These estimates and their SE were combined using Rubin’s rules.

For each qSOFA score cut-off (≥1, ≥2, and 3), the discrimination for predicting overall in-hospital mortality, 28-day in-hospital mortality and bacteraemia was assessed as the AUC considering data for all patients. The calibration of the risk score predictions was assessed by plotting observed proportions versus predicted probabilities and by calculating the Hosmer-Lemeshow χ² statistic. Performance was evaluated as sensitivity, specificity, positive and negative likelihood ratio, and positive and negative predictive value.

The minimum required sample size was estimated at 500 patients, based on the TRIPOD statement.²² All analyses were performed using the statistical software programmes Stata V.14.2 (StataCorp) and R V.3.4.1 (The R Foundation for Statistical Computing, https://www.r-project.org). Two-sided significance was set at 0.05.

Patient and public involvement

The public and patients were not involved in the development of the research question and outcome measures, study design or study recruitment. We will disseminate the final results to the study participants after they are published in a peer-reviewed journal.

Results

A total of 507 haemodialysis patients treated during the study period fulfilled the criteria for inclusion in this study. The overall in-hospital mortality in this population was 14.6% (74/507), whereas 28-day in-hospital mortality was 11.1% (56/507) and incidence of positive blood culture was 13.4% (68/507). In-hospital mortality rates were 5.2% among patients with a qSOFA score <2 and 29.6% among those with a qSOFA score ≥2. The corresponding mortality rates among patients with a positive blood culture were 3.9% and 35.3%, respectively. Table 1 provides a summary of the final diagnoses and corresponding mortality rates.

Table 1.

Final diagnoses and corresponding mortality rates

Final diagnosis	Patients (n)	Mortality, n (%)
System with Infection
Heart and vessels	11	5 (45.5)
Musculoskeletal system	18	5 (27.8)
Infectious disease related to the vascular access	24	6 (25.0)
Intra-abdominal	54	12 (22.2)
Respiratory system	99	9 (9.1)
Urinary organ	32	2 (6.3)
Skin	24	1 (4.2)
Other	38	7 (18.4)
Unknown	57	5 (8.8)
Non-infectious disease	150	22 (14.7)

Of the 507 participants (median age, 73 years), 36.5% were women. The most common cause of chronic kidney disease was diabetic nephropathy (40.0%), while the most frequent route of vascular access was arteriovenous fistula (74.0%). The mean haemodialysis vintage was 61 months, and 16.4% of patients had taken antibiotics within the week leading up to the hospital visit (table 2).

Table 2.

Characteristics of haemodialysis outpatients admitted for suspected bacteraemia (n=507)

Characteristic	Value*	Missing data†	Characteristic	Value*	Missing data†
Age, years	73 (66, 81)	0 (0.0%)	Vascular access		44 (8.7%)
Female sex	185 (36.5%)	0 (0.0%)	AV fistula	375 (74.0%)
Dialysis vintage, months	61 (23, 117)	25 (4.9%)	AV graft	59 (11.6%)
Cause of ESRD		14 (2.8%)	Superficial artery	17 (3.4%)
Diabetic nephropathy	203 (40.0%)		Permanent catheter	12 (2.4%)
Nephrosclerosis	100 (19.7%)		History of bacteraemia	50 (9.9%)	4 (0.8%)
Glomerulonephritis	87 (17.2%)		Medication
Other/unknown	103 (20.3%)		Steroids	50 (9.9%)	3 (0.6%)
Vital signs			Immunosuppressants	7 (1.4%)
Body temperature, °C	37.1 (36.6, 38.0)	36 (7.1%)	Antibiotics within 1 week	83 (16.4%)	6 (1.2%)
Systolic BP, mm Hg	136 (113, 159)	30 (5.9%)	Laboratory findings
Systolic hypotension‡	71 (14.0%)	30 (5.9%)	White cell count, 10 9/L	7.9(5.7, 11.2)	12 (2.4%)
Respiratory rate, breaths/min	20 (16, 24)	255 (50.3%)	Platelet count, 10 9/L	153 (107, 209)	12 (2.4%)
Tachypnea§	89 (17.6%)	255 (50.3%)	Albumin, g/dL	3.3 (2.9, 3.7)	53 (10.5%)
Heart rate, beats/min	86 (75, 100)	35 (6.9%)	C reactive protein, mg/dL	5.9 (1.7, 12.6)	18 (3.6%)
SpO2, %	97 (95, 100)	118 (23.3%)
GCS score <13	46 (9.1%)	80 (15.8%)	Positive blood culture	68 (13.4%)	0 (0.0%)
Comorbidities			In-hospital death	74 (14.6%)	0 (0.0%)
Malignancy	61 (12.0%)	1 (0.2%)
Diabetes	222 (43.8%)	1 (0.2%)

*Continuous data are summarised as median (IQR), while categorical data are summarised as frequency and percentage.

†Missing data are summarised as frequency and percentage.

‡Systolic hypotension was defined as systolic BP ≤100 mm Hg.

§Tachypnea was defined as a respiratory rate of ≥22 breaths/min.

AV, arteriovenous; BP, blood pressure; ESRD, end-stage renal disease; GCS, Glasgow Coma Scale

The most frequent pathogen in blood cultures was S. aureus, accounting for 28 cases of all bacteraemia cases (15 cases involving methicillin-sensitive S. aureus infection and 13 cases involving methicillin-resistant S. aureus infection). Klebsiella pneumoniae and Escherichia coli were the causal agent in 11 and 9 cases, respectively. Among the 68 patients with bacteraemia, 5 had polymicrobial infection (table 3).

Table 3.

Pathogens causing bacteraemia in haemodialysis patients

Bacterium	No
Staphylococcus aureus	28
Methicillin-sensitive S. aureus	15
Methicillin-resistant S. aureus	13
Klebsiella pneumoniae	11
Escherichia coli	9
Coagulase-negative Staphylococcus species	5
Enterococcus faecalis	3
Clostridium perfringens	2
Bacteroides species	2
Enterococcus faecium	2
Other	14

Of the 255 patients with complete data, 140 (54.9%), 91 (35.7%), 21 (8.2%) and 3 (1.2%) had qSOFA scores of 0, 1, 2 and 3 on hospital arrival. Among the patients with a qSOFA score of 1, tachypnea (respiratory rate ≥22 breaths/min) was the clinical criterion most commonly fulfilled (61.5%; 56/91). Among the patients with a qSOFA score of 2, the combination of altered mentation and tachypnea was the most common (47.6%; 10/21).

For predicting in-hospital mortality in haemodialysis patients, the areas under the receiver operating characteristic curves were 0.59 (95% CI 0.53 to 0.66) for a qSOFA score ≥1, 0.61 (95% CI 0.56 to 0.67) for a score ≥2 and 0.51 (95% CI 0.49 to 0.53) for a score ≥3 (table 4). A summary of sensitivity, specificity, positive and negative likelihood ratios, and positive and negative predictive values for each qSOFA score cut-off is provided in table 4.

Table 4.

Performance of the qSOFA score for predicting in-hospital mortality and bacteraemia in haemodialysis patients

Cut-off	AUC (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)	LR+ (95% CI)	LR- (95% CI)	PPV (95% CI)	NPV (95% CI)
Predicted outcome: overall in-hospital mortality
≥1	0.59 (0.51 to 0.66)	62.7% (50 to 74.2)	56.2% (50.9 to 61.4)	1.43 (1.15 to 1.78)	0.66 (0.48 to 0.92)	21.2% (15.7 to 27.6)	88.9% (84.0 to 92.7)
≥2	0.61 (0.56 to 0.67)	26.9% (16.8 to 39.1)	95.2% (92.5 to 97.2)	5.63 (3.06 to 10.3)	0.77 (0.66 to 0.89)	51.4% (34.0 to 68.6)	87.4% (83.6 to 90.5)
≥3	0.51 (0.49 to 0.53)	3.0% (0.4 to 10.4)	99.5% (98.4 to 100)	10.6 (0.98 to 116)	0.98 (0.93 to 1.01)	66.7% (9.4 to 99.2)	84.5 (80.7 to 87.8)
Predicted outcome: 28-day in-hospital mortality
≥1	0.59 (0.52 to 0.66)	62.7% (48.1 to 75.9)	55.4% (50.2 to 60.5)	1.41 (1.11 to 1.79)	0.67 (0.47 to 0.97)	16.2% (11.3% to 22%)	91.6% (87.1 to 94.8)
≥2	0.63 (0.57 to 0.70)	31.4% (19.1 to 45.9)	94.9% (92.1 to 96.9)	6.14 (3.38 to 11.2)	0.72 (0.60 to 0.87)	45.7% (28.8 to 63.4)	91.0 (87.7 to 93.6)
≥3	0.52 (0.49 to 0.55)	3.9% (0.5 to 13.5)	99.7% (98.5 to 100)	14.6 (1.35 to 158)	0.96 (0.91 to 1.02)	66.7% (9.4 to 99.2)	88.3% (84.9 to 91.2)
Predicted outcome: bacteraemia
≥1	0.51 (0.49 to 0.54)	57.6% (44.8 to 69.7)	55.2% (49.9 to 60.4)	1.28 (1.01 to 1.63)	0.77 (0.57 to 1.03)	19.2% (14.0 to 25.4)	87.6% (82.5 to 91.6)
≥2	0.56 (0.50 to 0.63)	15.2% (7.5 to 26.1)	93.0% (89.8 to 95.4)	2.16 (1.09 to 4.29)	0.91 (0.82 to 1.01)	28.6% (14.6 to 46.3)	85.6% (81.7 to 88.9)
≥3	0.54 (0.50 to 0.59)	3.0% (0.4 to 10.5)	99.7% (98.4 to 100)	10.8 (1.0 to 118)	0.97 (0.93 to 1.02)	66.7% (9.4 to 99.2)	94.8% (81.0 to 88.1)

AUC, area under the curve; LR+, positive likelihood ratio; LR−, negative likelihood ratio; NPV, negative predictive value; PPV, positive predictive value; qSOFA, quick Sequential Organ Failure Assessment

The Hosmer-Lemeshow χ² statistics for the qSOFA score as a predictor of overall in-hospital mortality and 28-day in-hospital mortality were 5.72 (p=0.02) and 7.40 (p<0.01), respectively. The observed and predicted overall in-hospital mortality and 28-day in-hospital mortality were compared on calibration plots (figure 1). As the number of patients with a qSOFA score of 3 was too small, calibration analysis considered patients with a qSOFA score of 2 or 3 together (figure 1).

Figure 1.

Observed and predicted in-hospital mortality among haemodialysis outpatients admitted for suspected bacteraemia. (A) Overall in-hospital mortality. (B) 28-day in-hospital mortality. qSOFA, quick Sequential Organ Failure Assessment.

Discussion

In this study, we investigated the diagnostic accuracy of qSOFA for predicting in-hospital mortality and bacteraemia incidence in haemodialysis patients who presented to the hospital with suspicion of bacteraemia. Overall, the qSOFA criteria had low accuracy for predicting mortality and bacteraemia incidence among such haemodialysis patients.

qSOFA has several advantages including easy bedside application, reliance on very few variables and no requirement for laboratory tests. However, of the recent studies on the validity of qSOFA in the emergency department setting,^15–20 one reported poor sensitivity for qSOFA-based out-of-hospital identification of severe sepsis and septic shock.²¹ To the best of our knowledge, the present study represents the first investigation of the external validity of qSOFA for risk stratification of haemodialysis patients with suspicion of infection. Our results revealed that qSOFA exhibits low sensitivity and miscalibration for in-hospital mortality and bacteraemia in haemodialysis patients. In particular, the calibration plot revealed that a qSOFA score of 1 overestimated, while qSOFA score of 2 or 3 underestimated both overall and 28-day in-hospital mortality. There may be several reasons for such findings. First, infection with different causal pathogens may have different manifestations. We confirmed previous observations that S. aureus is the most common bacterial pathogen causing bloodstream infection among haemodialysis patients.¹¹ Nevertheless, sepsis may have a different causal agent in haemodialysis patients than in the general population; the qSOFA score may not be able to fully account for different clinical presentations. Second, dialysis patients often present with immune system dysfunction and uraemia, as well as with comorbidities such as diabetes mellitus and connective tissue disorder,²⁵ which may also affect clinical manifestation, further distinguishing haemodialysis patients from the general population and detrimentally affecting the performance of the qSOFA score. In addition, most dialysis patients have hypertension,²⁶ and thus the incidence of hypotension, which is a key qSOFA criterion, may be low in haemodialysis patients with bacteraemia. Third, our baseline data were collected at the time of the hospital visit. One study revealed that a positive qSOFA result (qSOFA score ≥2) at hospital presentation and at 3, 6 and 24 hours after admission had poor sensitivity and specificity for predicting 28-day mortality.²⁰ In other words, the timing of data collection may also affect the performance of the qSOFA score, especially in haemodialysis patients.

Our study has several strong points. First, we included a multicentre cohort of haemodialysis patients, which reduced selection bias. Second, we used multiple imputation, which allowed us to investigate the entire cohort without having to exclude subjects with a relatively mild clinical presentation and thus without a detailed history or laboratory test findings, which would have induced information bias. Third, our cohort was geographically and temporally different from the cohort used to derive the qSOFA criteria, which enabled us to perform a true external validation study.

Several limitations of the present study warrant mention. First, given that we did not evaluate the reasons blood was drawn for culture, we cannot precisely determine the performance of qSOFA in haemodialysis patients with symptoms (eg, fever) that did not warrant blood culture evaluation. However, because it is not possible to predict clinical judgement in such situations, we believe this lack of consideration actually increases the generalisability of our findings, as is the case with the study that developed the clinical prediction rule for bacteraemia in the general population.²⁷ Second, our cohort contains patients who used antibiotics during the week leading up to the hospital visit, which could have affected their vital signs at presentation and decreased infection-related mortality and the rate of positive blood cultures. Third, we could not exclude the possibility that some patients had bacteraemia that was not detected on blood culture examination (ie, blood culture-negative bacteraemia), which is considered a limitation of blood culture. Finally, the exact time from hospital arrival to vital sign collection varied, which may have affected the qSOFA score and its relationship with patient prognosis. Employing routinely collected vital signs (eg, vital signs collected at the dialysis centre) for qSOFA score calculation might have provided a better reflection of bacteraemia status; however, vital sign data from the dialysis centres were not available to us at the time of the study.

To summarise, our validation study revealed that, in haemodialysis patients, the qSOFA score exhibits low diagnostic accuracy and miscalibration for in-hospital mortality and bacteraemia. A new prediction score is needed for mortality risk stratification of haemodialysis patients. For bacteraemia risk stratification, the BAC-HD score may outperform the qSOFA score in terms of predicting value.