External Validation of Bloodstream Infection Mortality Risk Score in a Population-Based Cohort

Abstract

A risk score was recently derived to predict mortality in adult patients with Gram-negative bloodstream infection (BSI). The aim of this study was to provide external validation of the BSI mortality risk score (BSIMRS) in a population-based cohort. All Olmsted County, Minnesota, residents with Escherichia coli and Pseudomonas aeruginosa BSI from 1/1/1998 to 12/31/2007 were identified. Logistic regression was used to examine the association between BSIMRS and mortality. Area under receiver operating characteristic curve (AUC) was calculated to quantify the discriminative ability of the BSIMRS to predict a variety of short- and long-term outcomes. Overall, 424 unique Olmsted County residents with first episodes of E. coli and P. aeruginosa BSI were included in the study. Median age was 68 (range: 0–99) years, 280 (66%) were women, 61 (14%) had cancer and 9 (2%) had liver cirrhosis. The BSIMRS was associated with 28-day mortality (p<0.001) with an AUC of 0.86. There was nearly 56% increase in 28-day mortality for each point increase in BSIMRS (odds ratio 1.56, 95% confidence intervals: 1.40–1.78). A BSIMRS ≥ 5 had a sensitivity of 74% and a specificity of 87% to predict 28-day mortality with a negative predictive value of 97%. The BSIMRS had AUC of 0.85, 0.85 and 0.81 for 7-, 14- and 365-day mortality, respectively. BSIMRS stratified mortality with high discrimination in a population-based cohort that included patients of all age groups who had a relatively low prevalence of cancer and liver cirrhosis.

INTRODUCTION

Bloodstream infection (BSI) is a major cause of morbidity and mortality in Europe and North America with approximately 1.8 million episodes of BSI and 250,000 deaths annually [1]. Gram-negative bacilli account for nearly one-half of the cases of BSI [2]. A risk score was recently derived to predict the prognosis of patients at the time of diagnosis with Gram-negative BSI based on acute severity of illness as summarized in the Pitt bacteremia score, source of infection and two major comorbid medical conditions that were independently associated with mortality (Table 1). The BSI mortality risk score (BSIMRS) had high prognostic ability to predict mortality in adult patients with Gram-negative BSI that were treated at tertiary care medical centers [3].

Table 1.

Bloodstream infection mortality risk score

Variable	Point allocation
Malignancy	3
Liver cirrhosis	4
Source of infection other than urinary tract or central venous catheter Pitt bacteremia score*	4
0–1	0
2–3	2
≥ 4	5

^*Pitt bacteremia score was calculated based on temperature (35.1–36 or 39.0–39.9 C: 1 point, ≤ 35 or ≥ 40 C: 2 points), blood pressure (hypotension: 2 points), mental status (disorientation: 1 point, stupor: 2 points, coma: 4 points), respiratory status (mechanical ventilation: 2 points) and cardiac status (cardiac arrest: 4 points). The worst reading was recorded on the day the first positive blood culture was obtained or the day before for nosocomial bloodstream infections.

The aim of the current study was to provide external validation for the BSIMRS in a population-based cohort in order to account for the potential limitations of the derivation cohort that were related the referral sample, including underrepresentation of patients at the extremes of age and relatively high prevalence of cancer and liver cirrhosis. We used a population-based cohort of patients with BSI due to Escherichia coli and Pseudomonas aeruginosa, the most common Gram-negative bacilli among Enterobacteriaceae and lactose non-fermenters, respectively. The primary aim was to evaluate the ability of the BSIMRS to predict 28-day mortality in this population-based cohort. Secondary aims included examination of other short-term (7- and 14-day mortality) and long-term outcomes (1-year mortality).

METHODS

Setting

Olmsted County is located in southeastern Minnesota and has a population of 124,277 according to the 2000 census (US Census Bureau, Olmsted County QuickFacts [http://quickfacts.census.gov], accessed 1 October, 2012). The population characteristics of Olmsted County residents are similar to those of USA non-Hispanic whites [4, 5]. The Rochester Epidemiology Project (REP) is a unique medical records-linkage system that encompasses healthcare delivered to Olmsted County, Minnesota, residents. The microbiology laboratories at Mayo Medical Center and Olmsted Medical Center are the only two laboratories in Olmsted County. These two medical centers are geographically isolated from other urban centers as previously described [4, 6], which increases the likelihood that residents get their healthcare at the local facilities rather than seeking care at a distant geographic location.

Definitions

The primary source of BSI was defined using the Centers for Disease Control and Prevention criteria [7]. Patients with cancer were defined as those with current diagnosis of malignant tumor, excluding skin basal and squamous cell carcinoma. Liver cirrhosis was defined based on clinical, laboratory, ultrasonography or histopathology results, when available [8].

Case ascertainment

All residents of Olmsted County, Minnesota, were eligible for inclusion as we used complete enumeration of Olmsted County population from 1 January 1998 to 31 December 2007. After the institutional review boards of Mayo Medical Center in Rochester, Minnesota, and Olmsted Medical Center approved the study, we used the microbiology laboratory databases at both institutions to identify all patients with E. coli and P. aeruginosa BSI during the study period. All patients were considered for inclusion in the study, regardless of the status of hospitalization, site of infection acquisition, admission location during hospitalization, or primary admitting service. Using the REP tools, we identified residents of Olmsted County, Minnesota, for inclusion in the study and excluded those who lived outside Olmsted County at the time of BSI. The primary investigator (M.N.A.) reviewed the medical records of all patients to confirm the diagnosis, determine patient residency status and obtain clinical features and outcome. The detailed case ascertainment methods were described elsewhere [9, 10]. Since both the derivation and validation cohorts were located in the same geographic area, we excluded patients who were previously included in the derivation cohort to avoid potential overlap.

Blood cultures were processed using standard microbiology techniques according to the Clinical and Laboratory Standards Institute (CLSI). The detailed blood culture methods used were described elsewhere [2, 11]. The microbiology laboratories at the Mayo Medical Center in Rochester, Minnesota, and Olmsted Medical Center are certified by the College of American Pathologists.

Statistical Analysis

BSIMRS was calculated by adding the allocated points for variables present in each patient (Table 1). Logistic regression was used to examine the association between BSIMRS and 28-day mortality. Odds ratios (OR) with 95% confidence intervals (CI) were calculated to indicate the strength of association between BSIMRS and mortality. The area under a receiver operating characteristic (ROC) curve (AUC), was used to quantify the discriminative ability of the BSIMRS, with a value of 0.5 denoting random predictions and a value of 1.0 denoting perfect predictions. To visually assess calibration, deciles of predicted risk of mortality were plotted from the risk score model by the actual fraction of patients who died within 28 days. Predicted probabilities obtained directly from the scoring model were plotted by risk score values to visualize the estimated risk of mortality. The sensitivity, specificity and negative predictive value (NPV) were calculated from the ROC curve for the best cut point in the BSIMRS for 7-, 14-, 28- and 365-day mortality. Only patients who were followed for the entire duration of the interval (7-, 14-, 28-days and 1-year, respectively) were included in the statistical analysis. Patients who were lost to follow up prior to the end of the interval duration were excluded from the analysis of that particular interval. Death was confirmed by reviewing medical records and the Minnesota death registry database.

JMP (version 10.0, SAS Institute Inc, Cary, NC) was used for statistical analysis. The level of significance for statistical testing was defined as p<0.05 (2-sided) unless otherwise specified.

RESULTS

Overall, 508 unique Olmsted County residents with first episodes of E. coli and P. aeruginosa BSI were identified during the 10-year study period. After exclusion of 84 patients who were previously included in the derivation cohort, 424 individuals with BSI were included in the current study. The mean age was 68 years (range: 0–99) and 280 (66%) were women. The baseline clinical characteristics of patients are shown in Table 2. Overall, 419 (99%) of patients in this cohort were followed for at least 28 days from the time blood cultures were obtained. The overall 28-day mortality following BSI was 9% (38/419).

Table 2.

Clinical characteristics of patients with bloodstream infection

Variable	(N=424)
Age in years, median (range)	68 (0–99)
Female gender, n (%)	280 (66)
White ethnicity, n (%)	366 (88)
Years of education*, median (range)	12 (0–20)
Diabetes mellitus, n (%)	99 (23)
Cancer, n (%)	61 (14)
Liver cirrhosis, n (%)	9 (2)
Primary source of infection, n (%)
Urinary tract	332 (78)
Gastrointestinal tract	25 (6)
Respiratory tract	24 (6)
Central venous catheter	3 (1)
Other, n (%)	8 (2)
Unknown, n (%)	32 (8)
Pitt bacteremia score, n (%)
0–1	326 (77)
2–3	56 (13)
≥ 4	38 (9)
Microbiology, n (%)
Escherichia coli	385 (91)
Pseudomonas aeruginosa	39 (9)
Site of infection acquisition, n (%)
Community-acquired	235 (55)
Healthcare-associated	145 (34)
Nosocomial	44 (10)

^*Total years of education are provided for 346 adults with available data.

There was nearly 56% increase in 28-day mortality for each point increase in BSIMRS (OR 1.56, 95% CI 1.40–1.78). The AUC for the risk score model was 0.86 (Figure 1). Model calibration looked satisfactory since the observed outcomes appeared fairly close to the predictions (Figure 2). The predicted 28-day mortality by BSIMRS is shown in Figure 3.

Figure 1.

Receiver operating characteristic plot of the bloodstream infection mortality risk score model of 28-day mortality

* Area under the curve for the bloodstream infection mortality risk score model = 0.86

Figure 2.

Calibration plot of bloodstream infection mortality risk score model for 28-day mortality

* The observed frequency of 28-day mortality plotted by deciles of predicted probability from the bloodstream infection mortality risk score model (black line). Perfect calibration is represented by the grey Y=X line.

Figure 3.

Predicted probability of 28-day mortality by bloodstream infection mortality risk score

* Size of marker for point estimates is weighted approximately by the relative number of subjects with corresponding risk score.

A BSIMRS ≥ 5 had a sensitivity of 74% and a specificity of 87% to predict 28-day mortality with a NPV of 97%. There was a substantial increase in 28-day mortality as the BSIMRS increased from < 5 (reference) to 5–9 (OR 15.09, 95% CI: 6.66–36.69) to ≥ 10 (OR 217.33, 95% CI: 32.77–54337.71).

The overall 7- and 14- day mortality rates were 6% (24/422) and 7% (31/421), respectively. The BSI risk score performed well in predicting 7- and 14-day mortality with AUC of 0.85 for both. A BSI risk score ≥ 5 had comparable sensitivity, specificity and NPV to predict 7- and 14-day mortality as those for 28-day mortality (Table 3). Nearly 92% (391/424) of patients in this cohort were followed for at least 1-year. The overall 1-year mortality was 23% (89/391). The BSIMRS predicted 1-year mortality with an AUC of 0.81. A BSIMRS ≥ 3 had reasonable sensitivity, specificity and NPV to predict 1-year mortality (Table 3).

Table 3.

Performance of bloodstream infection mortality risk score for various short- and long-term outcomes

Outcome	AUC	Best cut point	Sensitivity (%)	Specificity (%)	NPV (%)	Complete follow-up (%)
7-day mortality	0.85	≥ 5	78	85	99	100
14-day mortality	0.85	≥ 5	77	86	98	99
28-day mortality	0.86	≥ 5	74	87	97	99
1-year mortality	0.81	≥ 3	77	76	92	92

Note: AUC: area under receiver operating characteristic curve, NPV: negative predictive value

DISCUSSION

Despite the notable differences between the derivation and validation cohorts (supplementary Table 4), the BSIMRS estimated mortality with high discrimination in both referral and population-based settings.

The BSIMRS performed well in predicting a variety of short-term outcome measures, ranging from 7- to 28-day mortality. It is conceivable that the best AUC was observed for 28-day mortality, for which the risk score was derived. However, the BSIMRS had higher NPV for 7-day than that for 28-day mortality due to the higher overall survival in the former interval.

Examination of long-term outcome is a unique feature of this population-based study. The BSIMRS had a relatively high discrimination to predict 1-year mortality. However, the AUC was higher for 28-day morality and other short-term outcomes as compared to 1-year mortality. It is not surprising that the best cut point for 1-year mortality was 3 rather than 5 as in short-term outcomes. Malignancy (3 allocated points in the risk score) was likely a contributor to late deaths between 28 and 365 days from BSI. Including major host factors such as cancer and liver cirrhosis distinguished the BSIMRS from other acute severity of illness scores and allowed for prediction of long- in addition to short-term outcome [12, 13].

The BSIMRS demonstrates that BSI is a spectrum of illness. Patients with BSI have tremendous difference in prognosis ranging from an estimated short-term mortality of nearly 1% at one end to over 80% at the other end of the spectrum (Figure 3). The BSIMRS provides a simple tool to predict the outcome of patients at the time of diagnosis of Gram-negative BSI. This wide variation in survival and its predictability has both clinical and research implications.

The simplicity of the BSIMRS and relatively limited number of variables included make it an attractive choice. The Pitt bacteremia score is a measure of the acute severity of illness and is primarily based on readily available clinical variables such as temperature, blood pressure, respiratory, cardiac, and mental status [14]. The primary source of infection is a reflection of the inoculum of infection. BSI is categorized into low inoculum such as urinary or central venous catheter (CVC) infections versus other high inoculum infections [15–22]. To maintain simplicity, the exact source of infection does not need to be established to execute the score as long as urinary and CVC infections are ruled in or out by history, physical examination, urinalysis and time to positivity of blood cultures drawn peripherally and through CVC [3, 23]. The last two components of the BSIMRS, cancer and liver cirrhosis, provide as assessment of the host's ability to survive BSI [24–28].

The BSI risk score enables treating clinicians to better estimate the prognosis of patients with Gram-negative BSI, which could be valuable in clinical management and providing patient/family expectations. For example, a patient with liver cirrhosis and Gram-negative BSI secondary to spontaneous bacterial peritonitis (BSIMRS of 8) may present without high fever, hypothermia, hypotension or respiratory failure. Nevertheless, such a patient has a relatively high-predicted 28-day mortality of nearly 40% as estimated by BSIMRS (Figure 3), which is also consistent with the results of a recent large systematic review [28]. It would be reasonable to closely monitor such patients with high-predicted mortality in an intensive or intermediate care unit in order to allow sooner interventions in case of clinical deterioration. On the other hand, a serious conversation with the family regarding prognosis and expectations may need to be established early in patients with BSIMRS ≥ 10 due to the poor prognosis in this category.

Estimation of mortality in patients with Gram-negative BSI may also have an impact on empirical antimicrobial therapy. The threshold for aggressive antimicrobial therapy may need to be lower in patients with BSMRS ≥ 5 as compared to those with a score < 5 in order to ensure appropriate therapy in patients with relatively high-predicted mortality. Future studies evaluating the impact of inappropriate antimicrobial therapy on outcome of patients with Gram-negative BSI after stratification by BSIMRS would provide valuable information in this regard.

In addition to its clinical utility, the BSIMRS may be a useful tool in clinical research. The overall decline in mortality following Gram-negative BSI over the past two decades has left many small studies, particularly those focused on one microorganism or a specific antimicrobial resistance mechanism, without adequate power to adjust for all individual variables that are known to be associated with mortality in a multivariable model. The BSI risk score provides clinical investigators with an alternative option to adjust for the risk score as one variable as it includes all variables that are independently associated with mortality, after adjustments for age and other relevant variables to the study population. The BSIMRS may also be used as a method of randomization or stratification in clinical trials.

The primary strengths of this study are the population-based design, which decreases the possibility that referral bias affects the results, large sample size, and near-complete 28-day follow-up facilitated by the REP.

The study has some limitations. First, this was a retrospective cohort study. A prospective validation with real time collection of variables would have been ideal. However, the database was near complete due to availability of electronic medical records and timing of included variables was clearly predefined prior to data collection. Second, the population of Olmsted County consists mainly of middle class whites; therefore, our study results may be generalized only to communities with similar population characteristics. Third, our data were derived from one geographic area. The results of studies from multiple geographic locations might provide a more generalizable application. Finally, the validation cohort included patients with BSI predominately due to E. coli and, to a lesser extent, P. aeruginosa, rather than all Gram-negative bacilli as in the derivation cohort. However, this is less likely to influence the results since E. coli and P. aeruginosa are the first and third most common Gram-negative bloodstream isolates and when combined, account for > 60% of all Gram-negative BSI in our population [11]. In summary, The BSIMRS had a high prognostic ability to predict mortality in a population-based cohort that included patients in all age groups who had a relatively low prevalence of cancer and liver cirrhosis. The robust performance of the BSIMRS in both the derivation and validation cohorts provides assurance of the risk score's utility in both clinical settings and research investigations in the future.