Abstract
OBJECTIVE.
To develop and evaluate computer algorithms with high negative predictive values that augment traditional surveillance for central line-associated bloodstream infection (CLABSI).
SETTING.
Barnes-Jewish Hospital, a 1,250-bed tertiary care academic hospital in Saint Louis, Missouri.
METHODS.
We evaluated all adult patients in intensive care units who had blood samples collected during the period from July 1, 2005, to June 30, 2006, that were positive for a recognized pathogen on culture. Each isolate recovered from culture was evaluated using the definitions for nosocomial CLABSI provided by the National Healthcare Safety Network of the Centers for Disease Control and Prevention. Using manual surveillance by infection prevention specialists as the gold standard, we assessed the ability of various combinations of dichotomous rules to determine whether an isolate was associated with a CLABSI. Sensitivity, specificity, and predictive values were calculated.
RESULTS.
Infection prevention specialists identified 67 cases of CLABSI associated with 771 isolates recovered from blood samples. The algorithms excluded approximately 40%−62% of the isolates from consideration as possible causes of CLABSI. The simplest algorithm, with 2 dichotomous rules (ie, the collection of blood samples more than 48 hours after admission and the presence of a central venous catheter within 48 hours before collection of blood samples), had the highest negative predictive value (99.4%) and the lowest specificity (44.2%) for CLABSI. Augmentation of this algorithm with rules for common skin contaminants confirmed by another positive blood culture result yielded in a negative predictive value of 99.2% and a specificity of 68.0%.
CONCLUSIONS.
An automated approach to surveillance for CLABSI that is characterized by a high negative predictive value can accurately identify and exclude positive culture results not representing CLABSI from further manual surveillance.
Central line-associated bloodstream infections (CLABSIs) are associated with excess morbidity, mortality, and hospital costs. Studies performed among intensive care unit (ICU) patients in the United States estimate that hospital lengths of stay attributable to CLABSI range from 8 to 22 days, and that attributable costs range from $11,971 to $56,167.1–3 Each year, an estimated 250,000 cases of CLABSI occur in US hospitals, including approximately 80,000 cases among ICU patients.4 The pooled rates of CLABSI in ICUs of hospitals participating in the National Healthcare Safety Network of the Centers for Disease Control and Prevention range from 1.6 to 6.8 infections per 1,000 central line-days, depending on the type of ICU.5,6
Surveillance for CLABSI in the ICU is traditionally performed by infection prevention specialists using standardized definitions. However, this process is very labor intensive. Automated surveillance systems that utilize existing laboratory, pharmacy, and clinical electronic data have shown promise in identifying patients with bloodstream infection,7–10 but surveillance by infection prevention specialists remains the gold standard in the ICU. The automated surveillance systems evaluated to date tend to have a lower specificity than do traditional manual surveillance systems, sometimes with a considerable number of false-positive results. Given the increased interest in public reporting of healthcare-associated infection rates, it is unlikely that hospitals would be willing to rely on automated surveillance systems alone. In the absence of automated surveillance with perfect sensitivity, an automated approach characterized by a high negative predictive value could reliably identify patients without infection, thereby greatly reducing the number of patients requiring review by an infection prevention specialist. The objective of this study was to develop and evaluate computer algorithms with high negative predictive values that could augment traditional surveillance for CLABSI.
METHODS
The study was conducted at Barnes-Jewish Hospital, a 1,250-bed tertiary care academic hospital affiliated with the School of Medicine at Washington University in St. Louis, Missouri. The Washington University Human Studies Committee approved the study. The hospital serves an adult population exclusively and includes 6 ICUs with a total of 106 beds (range, 10–24 beds). We evaluated all ICU patients who had blood samples collected during the period from July 1, 2005, to June 30, 2006, that were positive for a recognized pathogen. Blood cultures with negative results were excluded from the analysis. Blood cultures with positive results were also excluded if they yielded the same organism that had been recovered from blood samples obtained in the previous 7 days. For patients with common skin contaminants, we were able to determine whether the same organism was isolated in another culture of a sample obtained within the last 5 days, by using the Barnes-Jewish Hospital Medical Informatics database. This database was used to identify all culture-positive blood samples from ICU patients during the study period. Electronic data were collected on patient laboratory results, inpatient medication orders, vital signs, and dates of admission, discharge, and collection of culture samples. Data on central venous catheter (CVC) use were collected on a monthly basis via a report generated from the electronic nursing documentation system; these data were then transferred to the Medical Informatics database used for the study. Each isolate from a blood culture was evaluated by use of 2 surveillance methods, to determine whether it represented nosocomial CLABSI.
Manual Surveillance
Infection prevention specialists reviewed the medical charts, physician summaries, and microbiologic and pharmacy data of each ICU patient who had a culture-positive blood sample. Each isolate was reviewed and classified as being associated with CLABSI or not. Isolates that were associated with a CLABSI were recorded in a standardized database. During the study period, the Barnes-Jewish Hospital employed 7 infection prevention specialists (ie, 6 registered nurses and 1 master of public health). The infection prevention specialists had 1–26 years (median, 5 years) of infection control experience, and 5 were certified in infection control.
Electronic Surveillance
Combinations of dichotomous prediction rules were applied to electronic data on patients to determine the presence of CLABSI. The rules are displayed in Table 1. The presence of an organism at a secondary body site of a patient was enough to exclude that patient’s blood culture isolate from consideration of being associated with CLABSI. Patient data on CVC use, vancomycin therapy, and temperature were evaluated at the time of each collection of culture samples. Positive cultures of samples from secondary sites (wound site, urine, respiratory tract, other sterile site, and other nonsterile site) were evaluated in relation to the time of each culture-positive blood sample (ie, before, after, or ever).
TABLE 1.
Dichotomous Rules Used in Computer Algorithms for Electronic Surveillance of Central Line-Associated Bloodstream Infection in Adult Patients in Intensive Care Units at Barnes-Jewish Hospital
| Rule | Description |
|---|---|
| Nosocomial | Culture samples collected >48 hours after admission |
| CVC | CVC present ≤48 hours before collection of culture samples |
| Non-CSC | Organism is not a CSCa |
| Fever | Feverb present ≤48 hours before or after collection of culture samples |
| Repeat (+) | Organism confirmed by another positive culture result in ≤5 days |
| Vancomycin | Treatment with vancomycin |
| Wound | Organism grown from culture of a wound sample |
| Urine | Organism grown from urine culture |
| Respiratory | Organism grown from culture of a respiratory- tract sample |
| Sterile other | Organism grown from culture of other sterile-site sample |
| Nonsterile other | Organism grown from culture of other nonsterile site sample |
NOTE. CSC, common skin contaminant; CVC, central venous catheter.
Common skin contaminants included diphtheroids, Bacillus species, Propionibacterium species, coagulase-negative staphylococci, and micrococci.
Temperature of >38°C.
Definition of CLABSI
The criteria used in both surveillance methods to identify CLABSI were based on the definition of CLABSI that was in use at the time of the study.5 This definition includes the following patient information: type of organism isolated (recognized pathogen or common skin contaminant), CVC use, type and duration of antibiotic therapy, and clinical characteristics, such as fever (temperature of more than 38°C), chills, or hypotension. Common skin contaminants included diphtheroids, Bacillus species, Propionibacterium species, coagulase-negative staphylococci, and micrococci. CVC use was recorded if the patient had a CVC in place within 48 hours before collection of culture samples. Vancomycin therapy was defined as receipt of at least 1 dose of vancomycin every other day during a 5-day period, initiated within 3 days after an organism was isolated from culture. A patient’s temperature was evaluated for 3 days before and 1 day after the collection of culture samples. Blood samples obtained after the first 48 hours of hospitalization were defined as nosocomial.
Statistical Analysis
The CLABSI status associated with every isolate was determined both by manual surveillance and by each algorithm. Using manual surveillance by infection prevention specialists as the gold standard, we evaluated the ability of various combinations of dichotomous rules to determine whether an isolate was associated with a CLABSI. Sensitivity, specificity, and predictive values were calculated. Discrepancies of case determination were compared between manual surveillance and the best-performing algorithms. Statistical analyses were performed with SPSS for Windows, version 14.0 (SPSS).
RESULTS
During the study period, 771 isolates recovered from 540 patients were evaluated. Of the 694 culture-positive blood samples collected, 625 (90%) yielded 1 organism, 62 (9%) yielded 2 organisms, 6 (1%) yielded 3 organisms, and 1 (0.1%) yielded 4 organisms. The median number of culture-positive blood samples per patient was 1 (range, 1–4). The most common organisms isolated were coagulase-negative staphylococci (45% of isolates), Corynebacterium species (6%), Candida albicans (5%), Staphylococcus aureus (5%), Enterococcus faecalis (5%), Enterococcus faecium (5%), and Klebsiella pneumoniae (3%). During the study period, 26,519 central line-days were accrued. For 62 patients, infection prevention specialists identified a total of 67 isolates that were associated with CLABSI.
Of the 771 isolates analyzed, 516 (67%) were associated with nosocomial CLABSI, and 640 (83%) were recovered from patients whose blood samples were collected within 48 hours of CVC use. More than half (419 [54%]) of the isolates were common skin contaminants, of which 95(23%) were confirmed by another positive culture result for a blood sample obtained from the patient within 5 days. Five hundred forty-seven (71%) of the isolates were recovered from patients who had fever within 48 hours before or after collection of blood samples for culture. For 162 (21%) isolates, the same organisms were detected at secondary sites. Vancomycin therapy was administered within 3 days after organism collection of 162 (21%) isolates.
The algorithms excluded 40%−62% of the isolates from consideration of possibly representing CLABSI (Table 2). Algorithm 1 was the simplest algorithm and was comprised of dichotomous rules for nosocomial infection and CVC use; it excluded 312 (40%) isolates from consideration as representing CLABSI. Algorithm 4 excluded the most isolates from consideration (481 [62%]) and was comprised of dichotomous rules for nosocomial infection, CVC use, and the presence of non-common skin contaminants or common skin contaminants confirmed by another culture-positive sample. Algorithm 6 was the most complex algorithm that excluded isolates from consideration (440 [57%]) and was comprised of dichotomous rules for nosocomial infection, CVC use, and the presence of either non-common skin contaminants or common skin contaminants confirmed by another culture-positive sample (as in algorithm 4) or, alternatively, by the presence of fever plus receipt of vancomycin therapy.
TABLE 2.
Characteristics of the Best-Performing Algorithms for the Surveillance of Central Line–Associated Bloodstream Infection (CLABSI), Compared With Manual Surveillance, and the Screening of 771 Isolates Recovered From 540 Patients at Barnes-Jewish Hospital
| Algorithm | No. (%) of isolates | |||||
|---|---|---|---|---|---|---|
| Classified as representing CLABSI |
Not classified as representing CLABSI |
Comparison with manual surveillance | ||||
| Sens, % | Spec, % | PPV, % | NPV, % | |||
| 1. Nosocomial + CVC | 459 (60) | 312 (40) | 97.1 | 44.2 | 14.8 | 99.4 |
| 2. Nosocomial + CVC + [Non-CSC or (CSC + Fever)] | 415 (54) | 356 (46) | 95.7 | 50.4 | 16.1 | 99.2 |
| 3. Nosocomial + CVC + [Non-CSC or (CSC + Vanc)] | 339 (44) | 432 (56) | 92.9 | 60.9 | 19.2 | 98.8 |
| 4. Nosocomial + CVC + [Non-CSC or (CSC + Repeat ( + ))] | 290 (38) | 481(62) | 94.3 | 68.0 | 22.8 | 99.2 |
| 5. Nosocomial + CVC + {Non-CSC or [CSC + (Vanc or Repeat ( + ))]} | 362 (47) | 409 (53) | 95.7 | 57.9 | 18.5 | 99.3 |
| 6. Nosocomial + CVC + {Non-CSC or [CSC + Fever + (Vanc or Repeat ( + ))]} | 331 (43) | 440 (57) | 94.3 | 62.2 | 19.9 | 99.1 |
NOTE. See Table 1 for descriptions of the algorithms. CSC, common skin contaminant; CVC, central-venous catheter; NPV, negative predictive value; PPV, positive predictive value; Repeat (+), another positive-culture result; Sens, sensitivity; Spec, specificity; Vanc, vancomycin.
The screening characteristics of the best-performing algorithms are shown in Table 2. The simplest algorithm (ie, algorithm 1), with dichotomous rules for nosocomial infection and CVC use, had the highest negative predictive value (99.4%) and the lowest specificity (44.2%) for CLABSI. Augmentation of this algorithm with rules for presence of common skin contaminants that had been confirmed by another culturepositive sample (algorithm 4) resulted in the highest specificity (68.0%) and only a minimal decrease in negative predictive value (99.2%). The further addition of rules for presence of fever and receipt of vancomycin therapy (algorithm 6) produced a similar negative predictive value (99.1%), although the specificity was not as high (62.2%). We attempted to eliminate secondary-site bloodstream infections by including rules that evaluated wound, urine, and respiratory-tract culture results, and this excluded a slightly higher proportion of isolates from consideration (68%) than did the best-performing algorithm (algorithm 4) but reduced the negative predictive value (98%).
Of the 481 isolates excluded from consideration by the best-performing algorithm (ie, algorithm 4), 4 were identified as associated with CLABSI by the infection prevention specialists (ie, by the “gold standard”). Of the 4 isolates, 2 were identified as common skin contaminants but were not confirmed by another culture-positive sample from the patient within 5 days; 1 was excluded by algorithm 4 because, according to the electronic data, the patient did not use a CVC, yet use of a multi-lumen catheter inserted into the groin was documented on the patient’s medical chart; and 1 was excluded by algorithm 4 because, according to the electronic data, the infection was community acquired, which was consistent with the patient’s medical record documenting hospital length of stay to be 47 hours and 25 minutes (ie, less than 48 hours) at the time of collection of blood samples.
DISCUSSION
To our knowledge, this is the first study to date to evaluate the ability of automated surveillance to augment traditional manual surveillance by identifying positive blood culture results that do not need further evaluation for CLABSI. Our automated surveillance system, which utilized existing laboratory, pharmacy, and clinical electronic data, had an excellent negative predictive value and was able to exclude approximately two-thirds of positive blood culture results from consideration as representing CLABSI. These results suggest that computer algorithms can effectively identify patients without infection, thereby reducing the number of ICU patients requiring additional manual surveillance by infection prevention specialists.
Because of the increasing availability of electronic medical-record data, numerous hospitals have implemented automated surveillance systems for nosocomial infections.11–14 To our knowledge, 4 studies have focused exclusively on bloodstream infections, although these studies evaluated the ability of automated surveillance systems to identify patients with infection rather than without.7–10 By altering the focus, our best-performing algorithm (algorithm 4), which included rules for nosocomial infection, CVC use, and detection of common skin contaminants confirmed by another positive culture result, had a higher negative predictive value (99%) than those reported in published studies, and thus we were able to more reliably exclude isolates from consideration as possible sources of CLABSI. A comparison of these studies is limited by the wide variety of definitions of bloodstream infection used and by the wide variablity of the study populations. For example, Yokoe et al.10 evaluated only blood samples positive for common skin contaminants, and they defined bacteremia as the isolation of the same common skin contaminant organism from at least 2 blood cultures within 5 days. The automated system used by Graham et al.8 classified all isolates recovered from blood cultures as sources of hospital-acquired bloodstream infection without assessing the clinical status or antibiotic treatment of each patient. Furthermore, our study utilized a homogenous adult ICU population, whereas previous studies included an assortment of neonatal, pediatric, and adult patients from both ICU and non-ICU patient care areas.
This is the first study to incorporate electronic data regarding the presence or absence of a CVC into computer algorithms for CLABSI surveillance. In the Trick et al. study,9 which lacked electronic data on CVC use, the addition of a rule for manual surveillance for CVC use to the best-performing computer algorithm increased the agreement between investigator review and automated methods from 0.49 to 0.73 and increased the algorithm’s negative predictive value from 87% to 90%. In the absence of electronic data on CVC use, Bellini et al.7 used microbiological data for the automated classification of nosocomial bloodstream infection as catheter related. More specifically, nosocomial bacteremia was classified as a CLABSI if the same organism was isolated from blood and from the catheter tip, regardless of the time to culture positivity between blood and catheter tip cultures. This approach achieved 74% concordance of catheter-related classification between the automated system and manual review but was inconsistent with the definitions for CLABSI provided by the National Healthcare Safety Network. Electronic data on CVC use are valuable for automated CLABSI surveillance. Although CVC use is generally very high in the ICU, 17% of the isolates in our study were not associated with CVC use within 48 hours before collection of blood samples for culture. These isolates were therefore excluded from consideration as possible sources of CLABSI.
The rule for confirmation of detection of a common skin contaminant by another culture-positive blood sample was crucial for the differentiation of patients with and patients without CLABSI, particularly considering the high prevalence of common skin contaminants in our blood culture data. Although the definition for CLABSI provided by the National Healthcare Safety Network does not specify a time window for such confirmation, we used the same 5-daytime window as did Trick et al.9 and Yokoe et al.,10 to allow comparison between computer algorithms. In their evaluation of blood cultures positive for common skin contaminants, Yokoe et al.10 found that a similar rule for another positive blood culture result within a 5-day window agreed well with the definition for CLABSI provided by the National Healthcare Safety Network (κ = 0.91).
The limitations of this study include errors in data quality inherent in large electronic databases. As noted previously, 1 patient who was classified as having CLABSI by infection prevention specialists using manual surveillance had contradictory information regarding CVC use (ie, patient did not use a CVC according to the electronic data, yet use of a multilumen catheter inserted into the groin was documented on the patient’s medical chart), which resulted in discrepant CLABSI classifications and slightly lower performance values for the algorithms. Data on chills were not available electronically, and information on blood pressure was not included in the data set used for analysis. In our patient population, however, no CLABSI cases were missed due to lack of electronic information on chills or hypotension (data not shown). Although microbiological data maybe available electronically at many hospitals, some of the other data available to us, especially on CVC use, may not be as readily available electronically. This would limit the generalizability of these algorithms. However, the potential to improve surveillance for CLABSI by making such information available may encourage hospitals to capture data on CVC use in a way that allows it to be entered and stored electronically.
This study was based on the determination of CLABSI by infection prevention specialists using the definitions provided by the National Healthcare Safety Network that were current at the time, as part of routine ICU surveillance. Since the work for this study was done, the National Healthcare Safety Network has removed from its definition of CLABSI the criterion involving a single culture positive for a common skin contaminant and a record of appropriate therapy. In addition, a time window of 2 days has been defined for the confirmation of a detection of common skin contaminant with another positive culture result. Because the infection prevention specialists did not retrospectively reevaluate cases of CLABSI using the revised definitions, it was not possible to evaluate an updated algorithm that incorporated these changes. However, from the current results, an algorithm that did not include vancomycin use (algorithm 4) was able to exclude an additional 9% of positive blood cultures, even though we used a 5-day time window for confirmation of common skin contaminants. Thus, with the recent revision in the definitions provided by the National Healthcare Safety Network, it seems likely that a computer algorithm with high negative predictive values would be even more useful in excluding positive culture results not associated with CLABSI.
In our study, approximately two-thirds of the positive blood culture results were removed from consideration as representing CLABSI. Augmentation of manual CLABSI surveillance with an automated approach has several advantages, including reducing the time spent by infection control specialists on routine surveillance10 and reallocating it to prevention efforts, and potentially extending surveillance outside of the ICU. In the absence of automated surveillance methods with perfect sensitivity, an automated approach characterized by a high negative predictive value can accurately identify and exclude positive culture results not associated with CLABSI from further manual surveillance.
ACKNOWLEDGMENTS
Financial support. This work was supported by grant 00426-0805-01 from the Barnes-Jewish Hospital Foundation.
Presented in part: 12th International Health (Medical) Informatics Congress, MedInfo; Brisbane, Australia; August 20-24, 2007.
Footnotes
The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Barnes-Jewish Hospital Foundation.
Potential conflicts of interest. All authors report no conflicts of interest relevant to this article.
REFERENCES
- 1.Dimick JB, Pelz RK, Consunji R, Swoboda SM, Hendrix CW, Lipsett PA. Increased resource use associated with catheter-related bloodstream infection in the surgical intensive care unit. Arch Surg 2001;136:229–234. [DOI] [PubMed] [Google Scholar]
- 2.Warren DK, Quadir WW, Hollenbeak CS, Elward AM, Cox MJ, Fraser VJ. Attributable cost ofcatheter-associated bloodstream infections among intensive care patients in a nonteaching hospital. Crit Care Med 2006;34: 2084–2089. [DOI] [PubMed] [Google Scholar]
- 3.Blot SI, Depuydt P, Annemans L, et al. Clinical and economic outcomes in critically ill patients with nosocomial catheter-related bloodstream infections. Clin Infect Dis 2005;41:1591–1598. [DOI] [PubMed] [Google Scholar]
- 4.O’Grady NP, Alexander M, Dellinger EP, et al. Guidelines for the prevention of intravascular catheter-related infections. Centers for Disease Control and Prevention. MMWR Recomm Rep 2002;51(RR-10):1–29. [PubMed] [Google Scholar]
- 5.Centers for Disease Control and Prevention. National Center for Infectious Diseases. The National Healthcare Safety Network (NHSN) Manual. Department of Health and Human Services Publication 5–24-0007. Available at: http://www.cdc.gov/ncidod/dhqp/pdf/nhsn/NHSN_Manual_PatientSafetyProtocol_CURRENT.pdf. >Accessed October 31, 2008.
- 6.Edwards JR, Peterson KD, Andrus ML, et al. National Healthcare Safety Network (NHSN) Report, data summary for 2006, issued June 2007. Am J Infect Control 2007;35:290–301. [DOI] [PubMed] [Google Scholar]
- 7.Bellini C, Petignat C, Francioli P, et al. Comparison of automated strategies for surveillance of nosocomial bacteremia. Infect Control Hosp Epidemiol 2007;28:1030–1035. [DOI] [PubMed] [Google Scholar]
- 8.Graham PL III, San Gabriel P, Lutwick S, Haas J, Saiman L. Validation of a multicenter computer-based surveillance system for hospital-acquired bloodstream infections in neonatal intensive care departments. Am J Infect Control 2004;32:232–234. [DOI] [PubMed] [Google Scholar]
- 9.Trick WE, Zagorski BM, Tokars JI, et al. Computer algorithms to detect bloodstream infections. Emerg Infect Dis 2004;10:1612–1620. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Yokoe DS, Anderson J, Chambers R, et al. Simplified surveillance for nosocomial bloodstream infections. Infect Control Hosp Epidemiol 1998; 19:657–660. [DOI] [PubMed] [Google Scholar]
- 11.Evans RS, Larsen RA, Burke JP, et al. Computer surveillance of hospital-acquired infections and antibiotic use. JAMA 1986;256:1007–1011. [PubMed] [Google Scholar]
- 12.Gastmeier P, Brauer H, Hauer T, Schumacher M, Daschner F, Ruden H. How many nosocomial infections are missed if identification is restricted to patients with either microbiology reports or antibiotic administration? Infect Control Hosp Epidemiol 1999;20:124–127. [DOI] [PubMed] [Google Scholar]
- 13.Platt R, Yokoe DS, Sands KE; CDC Eastern Massachusetts Prevention Epicenter Investigators. Automated methods for surveillance of surgical site infections. Emerg Infect Dis 2001;7:212–216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Pokorny L, Rovira A, Martin-Baranera M, Gimeno C, Alonso-Tarres C, Vilarasau J. Automatic detection ofpatients with nosocomial infection by a computer-based surveillance system: a validation study in a general hospital. Infect Control Hosp Epidemiol 2006;27:500–503. [DOI] [PubMed] [Google Scholar]