Abstract
Objective
Surviving Sepsis guidelines recommend blood cultures before administration of intravenous (IV) antibiotics for patients with sepsis or moderate to high risk of bacteremia. Clinical decision support (CDS) that reminds emergency department (ED) providers to obtain blood cultures when ordering IV antibiotics may lead to improvements in this process measure.
Methods
This was a multicenter causal impact analysis comparing timely blood culture collections prior to IV antibiotics for adult ED patients 1 year before and after a CDS intervention implementation in the electronic health record. A Bayesian structured time-series model compared daily timely blood cultures collected compared to a forecasted synthetic control. Mixed effects models evaluated the impact of the intervention controlling for confounders.
Results
The analysis included 54 538 patients over 2 years. In the baseline phase, 46.1% had blood cultures prior to IV antibiotics, compared to 58.8% after the intervention. Causal impact analysis determined an absolute increase of 13.1% (95% CI 10.4–15.7%) of timely blood culture collections overall, although the difference in patients with a sepsis diagnosis or who met CDC Adult Sepsis Event criteria was not significant, absolute difference 8.0% (95% CI −0.2 to 15.8). Blood culture positivity increased in the intervention phase, and contamination rates were similar in both study phases.
Discussion
CDS improved blood culture collection before IV antibiotics in the ED, without increasing overutilization.
Conclusion
A simple CDS alert increased timely blood culture collections in ED patients for whom concern for infection was high enough to warrant IV antibiotics.
Keywords: clinical decision support, sepsis, emergency medicine, antibiotic stewardship
INTRODUCTION
Blood cultures are an important diagnostic test for many patients who present to the hospital with serious infections. In patients with sepsis, collection of blood cultures before intravenous (IV) antibiotic administration for patients with suspected sepsis is considered a best practice by the Surviving Sepsis guidelines and is an element of the Centers for Medicare and Medicaid Services Severe Sepsis/Septic Shock Early Management Bundle (SEP-1).1 Antibiotic administration prior to blood culture collection reduces culture positivity, which in turn hampers pathogen identification and subsequent narrowing of broad empiric coverage to more targeted therapy.2–4 Proper antibiotic stewardship with de-escalation of antibiotics when appropriate has been associated with decreased mortality and lower costs.3,5,6 Despite these benefits, one large multicenter analysis of the sepsis 6-h care bundle found only 77% compliance with microbiology specimen collection prior to antibiotic administration.7
Clinical decision support (CDS) systems within the electronic health record (EHR) have been implemented to improve various aspects of sepsis care. Most prior efforts have focused on early detection of sepsis or guiding appropriate antibiotic selection.8–11 Previous CDS systems related to blood culture ordering focus primarily on reducing unnecessary ordering.12,13 No prior published literature reports on the use of CDS to improve timely blood culture collection; that is, prior to IV antibiotic administration.
The primary goal of this study was to determine if a CDS intervention that asks providers to consider ordering blood cultures when clinically appropriate can improve the rate of timely blood culture collections in patients for whom concern for infection is high enough to warrant IV antibiotics, including patients ultimately found to have sepsis. The secondary goal was to determine the impact of the CDS intervention on blood culture positivity rates for both contaminants and pathogenic organisms.
METHODS
Study design
A retrospective pre-post, time-series cohort study using a synthetic control was used to determine the causal impact of the CDS intervention on rates of blood culture collection prior to first IV antibiotic administration in the emergency department (ED). The CDS alert was designed as a quality improvement measure and implemented on August 24, 2020. We compared 12-month intervals before and after the deployment of the CDS intervention within the EHR: a baseline phase from August 24, 2019 to August 23, 2020, and an intervention phase from August 24, 2020 to August 24, 2021. The daily proportion of ED encounters where blood cultures were obtained prior to the first IV antibiotic administration was determined. This study was approved by the Mass General Brigham Institutional Review Board. We adhered to the SQUIRE standard for reporting on quality improvement efforts.14
Setting
The study occurred in 9 EDs, including 2 urban tertiary-care academic centers and 7 affiliated community hospitals. The annual adult census of the EDs over the 1-year study period ranged from 6483 to 83 757 encounters per hospital. Both academic centers are affiliated with the Harvard Emergency Medicine Residency. All hospitals share the same Epic EHR (Epic Systems Corporation, Verona, WI), and are part of the same integrated healthcare network in New England (Mass General Brigham). Prior to this study intervention, a standardized sepsis order set was available to providers that included orders for blood cultures and recommended antibiotics according to suspected infection source, however utilization was relatively low (∼5 uses per day).
Selection of participants
All adult patients (≥18 years) who received at least one dose of IV antibiotics in the ED of the 9 study sites between August 2019 to August 2021 were considered eligible for the study. Antibiotics were defined by medications within antibacterial pharmaceutical classes set by First Databank, our medication decision support content vendor. We excluded patients transferred in from other EDs as those patients may have received a first dose of antibiotics from the sending facility that was not documented in our local EHR. We excluded encounters where the blood culture collection timestamp was before the arrival of the patient to the ED, which happened either because of erroneous documentation of the collection time or because the cultures were collected prior to the ED visit in an outpatient setting (eg, from a hemodialysis or an infusion center). We only included the first ED encounter for each patient and excluded subsequent encounters (at any site) to maintain independence of the observations.
Intervention
A CDS alert was designed as a BestPractice Advisory (a pop-up alert) in the Epic EHR to warn ED providers (attendings, fellows, residents, physician assistants, and nurse practitioners) upon ordering a first dose of IV antibiotics if blood cultures were not yet ordered or collected within the prior 7 days. The alert did not fire if the patient was in cardiac arrest. The alert allowed the user to order blood cultures directly or to indicate that blood cultures were not indicated (see Figure 1). The alert was approved by our healthcare system’s emergency medicine EHR governance committee (which includes membership from all study sites) prior to implementation. No specific user training related to this alert was provided before or during the intervention phase. There was no “settle in” period between when the intervention was introduced to users and when data collection began. The logic of the alert is available in Supplementary Table S1 accompanying the online article.
Figure 1.
CDS intervention screenshot.
CDS interventions must be crafted carefully to avoid undue alert burden, which can contribute to alert fatigue.15,16 During the study period, there were 354 other BestPractice Advisories in place in the ED, and on an average 2 unique alerts were shown per encounter. We triggered our alert only on ordering IV antibiotics in the ED, which targets the intervention to the treating provider within their existing workflow. Understanding that users would not appreciate overly aggressive interruptions, the alert was interruptive, but not a hard-stop so it could be dismissed without ordering cultures.
Measurements
Data were extracted from our EHR’s data warehouse using Structured Query Language (SQL). For each patient, we obtained basic demographics including age, gender, ethnic group, and race. Patient gender, ethnic group, and race were self-reported. The patient’s problem list was extracted and transformed into the Charlson comorbidity categories using the comorbidity R package, and the Charlson Comorbidity Index was calculated.17 The patient’s COVID-19 infection status was extracted from the EHR and reflects if the patient was considered actively infected by the SARS-CoV-2 virus at any time during the same hospital encounter, either because they had a recent positive test prior to ED arrival or tested positive during the hospitalization. All admitted patients were tested for SARS-CoV-2 via a PCR test as part of standard infection control policies. We extracted encounter-specific timestamps of ED arrival and departure, first IV antibiotic order and administration, and first blood culture order and collection. Antibiotic administration and blood culture collection timestamps were based on nursing documentation within the EHR. The time between antibiotic administration and blood culture collection was calculated. We extracted the results of the blood cultures that were sent from the ED, including the discrete list of organisms identified per culture. A blood culture was considered a potentially contaminated sample if at least one of the organisms identified included any of the following common skin contaminants: coagulase negative staphylococcus, Bacillus species not anthracis, Corynebacterium, Aerococcus, Micrococcus, and Propionibacterium.18 We extracted which antibiotic was ordered and whether it was ordered for “empiric coverage,” “definite infection,” or “prophylaxis.” These indications as well as the type of infection being treated (ie, “pneumonia,” “urinary tract infection/pyelonephritis,” etc.) are mandatory fields that must be entered by the provider when ordering antibiotics in the ED and inpatient setting. These indications and infection types are routinely reviewed by antibiotic stewardship and infection control teams to monitor antibiotic utilization in our healthcare system. As the data for this study were derived using a SQL query, we reviewed 25 charts manually to ensure accuracy of all of the extracted variables.
Outcomes
The primary outcome was blood culture collection prior to the first IV antibiotic administration in the ED. Not all patients that receive IV antibiotics are suspected to have bacteremia or sepsis, and therefore the secondary outcome was to determine if the intervention affected the blood culture positivity rate, as an increase in unnecessary blood culture orders in patients with minimal risk of having bacteremia or sepsis would be expected to decrease the blood culture positivity rate. We also looked at the intervention’s impact on blood culture contamination rates.
Since patients suspected of having bacteremia or sepsis are primary targets for best practice guidelines and quality measures that recommend blood cultures prior to antibiotics, we conducted a sensitivity analysis focusing on patients who had clinical evidence of sepsis, as defined by either meeting the CDC’s Adult Sepsis Event (ASE) definition or having a sepsis diagnosis as defined by the Center for Medicare Services (CMS) SEP-1 guidelines.19,20 Patients with ASE were identified using the definition and methodology outlined by the Hospital Toolkit for Adult Sepsis Surveillance produced by the CDC.20 Briefly, to meet the ASE sepsis definition a patient must have blood cultures obtained, be treated with antibiotics for at least 4 qualifying days, and have evidence of organ dysfunction. We determined if patient’s met the ASE definition using data extracted via a SQL query of our EHR’s data warehouse and processed with a Python script modeled after the framework provided in the Hospital Toolkit for Adult Sepsis Surveillance (code available on request).20 Sepsis diagnosis was defined by an ED primary diagnosis or principal hospital diagnosis that included any of the following International Classification of Diseases (ICD)-10-CM codes: A02.1, A22.7, A267, A32.7, A40.0, A40.1, A40.3, A40.8, A40.9, A41.01, A41.02, A41.1, A41.2, A41.3, A41.4, A41.50, A41.51, A41.52, A41.53, A41.59, A41.81, A41.89, A41.9, A42.7, A54.86, B37.7, R65.20, R65.21.
Analyses
The overall proportion of blood culture collections before or after IV antibiotic administration was summarized for each study period, and further delineated by patients that met the composite sepsis definition (either meeting CDC ASE sepsis definition or having a sepsis diagnosis), antibiotic indication, and infection source.
A causal impact approach using synthetic controls described by Broderson et al was implemented using a Bayesian structured time series (BSTS) model, and is described briefly as follows.21 A BSTS model was created using the primary outcome (timely blood culture collections) in the baseline period, as well as observations of multiple independent covariates in the baseline and intervention periods. The independent covariates were the number of daily complete blood cell counts and lactic acid collections before IV antibiotic administration, daily patient encounters where IV antibiotics were administered, daily COVID-19 cases, and daily number of IV antibiotics for empiric coverage, definitive infections, and prophylaxis. These covariates were correlated with the primary outcome, but not affected by the CDS intervention (a requirement of BSTS models). The BSTS model accounted for weekly variations, as more patients were seen in the ED on some days of the week compared to others, as well as seasonality and autocorrelation. The BSTS model created predictions of the counterfactual (or synthetic control) of the CDS alert during the intervention study phase, specifically, the predicted number of daily blood cultures collections before IV antibiotic administration if the CDS intervention had not been implemented. The absolute risk reduction (ARR) between the synthetic control and observed timely blood culture collections in the intervention phase was calculated. The 95% confidence intervals of the predicted synthetic controls were used to derive the confidence intervals for the ARR. The number needed to treat (NNT) between the synthetic control and actual observations in the intervention phase was calculated, and reflects the number of patients where the CDS alert was shown to result in one additional blood culture collection before IV antibiotics.
We chose a causal impact framework over the standard interrupted time series approach for several reasons. A traditional interrupted time series analysis uses segmented regression to determine the difference in the “level” and “slope” of the outcome between study phases. This assumes that the baseline (pretest) phase is an adequate proxy for the baseline of the measured outcome in the intervention phase, despite the fact that the effect of the intervention may change over time.21 BSTS models are state-space models that do not rely on this assumption, and can account for time-varying relationships through the incorporation of dynamic covariates.21 BSTS models therefore account for autocorrelation, whereas traditional interrupted time series do not. Auto-Regressive Integrated Moving Average (ARIMA) models account for autocorrelation, but BSTS models have the advantage of being modular, that is, each component of the prediction can be separated, making the model more transparent. BSTS models can also inherently handle different levels of seasonality, whereas this is more complicated in ARIMA models.
To determine if the patient’s demographic characteristics or the practice patterns of the ED provider that received the alert confounded the timing of blood culture collections beyond the CDS intervention, a multivariable mixed-level regression model was performed, and adjusted odds ratios were reported. As the intervention was shown to ED providers and practice patterns may differ at various hospitals, the ED provider and study site were treated as random effects.22,23 Similarly, we performed multivariable regressions to determine if blood culture positivity and contaminant rates were different between study phases, adjusting for basic demographics, the Charlson Comorbidity Index, vital sign abnormalities, first white blood cell count, and COVID-19 infection status. Vital sign abnormalities were defined as follows: hypotension was lowest mean arterial pressure <65 mm Hg, hypoxia was lowest oxygen saturation <92%, fever was highest temperature ≥101.5°F, tachycardia was highest heart rate >100 beats per minute. Leukocytosis was defined as the first serum white blood cell count in the ED encounter >11 K/µL, while leukopenia was a white blood cell count <4 K/µL.
All visualizations and statistics were performed in R (version 4.1.2) using the tidyverse, lme4, ggplot2, and CausalImpact packages.21,24–27 Code available upon request.
RESULTS
Characteristics of study subjects
Between August 2019 and August 2021, there were 76 629 encounters for adult patients who received IV antibiotics in the ED at 1 of the 9 study sites. After excluding patients that did not meet the study criteria, 54 538 patients were included in the final study population, 28 332 in the baseline phase and 26 206 in the intervention phase (see Figure 2). Overall, the alerts were seen by the following unique ED providers: attending physicians (n = 1215), resident physicians (n = 904), physician assistants (n = 458), other providers (such as students) (n = 295), pharmacists (n = 191), and nurse practitioners (n = 91). There were no significant differences in patient demographics, proportion of IV antibiotic administrations, or patients that ultimately met the composite sepsis definition between the study phases (see Table 1).
Figure 2.
Description of study population.
Table 1.
Study group comparison
| Baseline (n = 28 332) | Intervention (n = 26 206) | |
|---|---|---|
| Age (years) | ||
| Median (IQR) | 63 (46–76) | 62 (44–75) |
| Gender | ||
| Female, n (%) | 14 375 (50.7) | 13 307 (50.8) |
| Male | 13 957 (49.3) | 12 899 (49.2) |
| Race | ||
| White, n (%) | 22 141 (78.1) | 20 447 (78.0) |
| Two or more Races | 2249 (7.9] | 2164 (8.3) |
| Black or African American | 2147 (7.6) | 1979 (7.6) |
| Unavailable or declined | 893 (3.2) | 725 (2.8) |
| Asian | 842 (3.0) | 840 (3.2) |
| Other racesa | 60 (0.2) | 51 (0.2) |
| Ethnic group | ||
| Non-Hispanic, n (%) | 24 099 (85.1) | 22 330 (85.2) |
| Hispanic | 3116 (11.0) | 3000 (11.4) |
| Unavailable or declined | 1117 (3.9) | 876 (3.3) |
| COVID-19 infection status | ||
| Not COVID-19, n (%) | 27 006 (95.3) | 24 802 (94.6) |
| COVID-19 | 1326 (4.7) | 1404 (5.4) |
| ED disposition | ||
| Admit to floor, n (%) | 18 606 (65.7) | 16 696 (63.7) |
| Discharge or outside transfer | 7531 (26.6) | 7814 (29.8) |
| Admit to ICU | 2195 (7.7) | 1696 (6.5) |
| ED provider type | ||
| Attending physician, n (%) | 13 231 (46.7) | 10 932 (41.7) |
| Physician assistant | 9876 (34.9) | 9038 (34.5) |
| Resident physician | 3622 (12.8) | 4757 (18.2) |
| Nurse practitioner | 740 (2.6) | 760 (2.9) |
| Pharmacist | 612 (2.2) | 524 (2.0) |
| Other | 251 (0.9) | 195 (0.7) |
| Antibiotic indication | ||
| Empiric coverage, n (%) | 15 253 (53.8) | 14 673 (56) |
| Definitive infection | 12 437 (43.9) | 10 849 (41.4) |
| Prophylaxis | 642 (2.3) | 684 (2.6) |
| Sepsis cohort | ||
| ASE, n (%) | 4168 (14.7) | 3588 (13.7) |
| Sepsis ICD-10 diagnosis | 1287 (4.5) | 1047 (4.0) |
| ASE or sepsis ICD-10 diagnosis | 4879 (17.2) | 4178 (15.9) |
| Blood cultures | ||
| Negative, n (%) | 14 450 (51.0) | 15 283 (58.3) |
| Not collected | 12 345 (43.6) | 9232 (35.2) |
| Positive | 1199 (4.2) | 1250 (4.8) |
| Contaminant | 304 (1.1) | 394 (1.5) |
| Creditedb or duplicate | 34 (0.1) | 47 (0.2) |
IQR: interquartile range; ICU: intensive care unit; ASE: adult sepsis event.
Other Races includes Native American, Alaskan Native, Native Hawaiian, and Pacific Islander.
Credited denotes blood cultures that were submitted to the lab but not processed because lab error, mislabeled samples, or inadequate sample to perform the test.
Main results
In the intervention phase, the CDS alert was shown in 12 781/26 206 (48.8%) of encounters. The alert was overridden 7952 (62.2%) times, blood cultures were ordered directly from the alert 4517 (35.3%) times, and the alert was dismissed without placing the order or entering an override 315 (2.5%) times. When the alert was overridden or dismissed, an order for blood cultures was placed within 10 min of the alert firing 234/8267 (2.8%) of the time—which may reflect reconsideration of the necessity of cultures shortly after the alert fired.
During the baseline phase, blood cultures were collected prior to IV antibiotic administration 46.1% of the time, compared to 58.8% in the intervention phase (see Table 2). Forecasted synthetic controls were calculated using the BSTS model (see Figure 3). The ARR of late (or missed) blood culture collections between the synthetic controls derived from the BSTS model was 13.1% (95% CI 10.4–15.7) and the NNT was 7.6 (95% CI 6.4–9.6). The improvement of blood culture collection prior to IV antibiotics was highest when antibiotics were given for “definitive infection,” ARR of 15.3% (95% CI 11.5–19.2), compared to “empiric coverage” or “prophylaxis.” For patients that ultimately met the composite sepsis outcome, the proportion of timely blood culture collections rose from 82.2.5% in the synthetic control to 90.2% in the intervention phase, however the absolute difference was not statistically significant, ARR of 8.0% (95% CI −0.2 to 15.8).
Table 2.
Summary statistics of timely blood culture collections between baseline, synthetic control, and intervention
| Baseline (n (%)) |
Synthetic control (n (%)) |
Intervention (n (%)) |
||||||
|---|---|---|---|---|---|---|---|---|
| BCx before Abx | BCx after Abx or not collected | BCx before Abx | BCx after Abx or not collected | BCx before Abx | BCx after Abx or not collected | ARR [95% CI] | NNT [95% CI] | |
| Sepsis | ||||||||
| Not sepsis | 9079 (38.7) | 14 374 (61.3) | 8537 (38.8) | 13 491 (61.2) | 11 647 (52.9) | 10 381 (47.1) | 14.1 [11.3–16.8] | 7.1 [5.9–8.8] |
| Sepsis | 3978 (81.5) | 901 (18.5) | 3435 (82.2) | 743 (17.8) | 3768 (90.2) | 410 (9.8) | 8.0 [−0.2 to 15.8] | N/A |
| Antibiotic indication | ||||||||
| Definitive infection | 4444 (35.7) | 7993 (64.3) | 3889 (35.8) | 6960 (64.2) | 5554 (51.2) | 5295 (48.8) | 15.3 [11.5–19.2] | 6.5 [5.2–8.7] |
| Empiric coverage | 8521 (55.9) | 6723 (44.1) | 8056 (54.9) | 6617 (45.1) | 9727 (66.3) | 4946 (33.7) | 11.4 [7.6–14.9] | 8.8 [6.7–13.1] |
| Prophylaxis | 92 (14.3) | 550 (85.7) | 95 (13.9) | 589 (86.1) | 134 (19.6) | 550 (80.4) | 5.7 [−1.3 to 12.9] | N/A |
| Overall | 13 057 (46.1) | 15 275 (53.9) | 11 979 (45.7) | 14 227 (54.3) | 15 415 (58.8) | 10 791 (41.2) | 13.1 [10.4–15.7] | 7.6 [6.4–9.6] |
BCx: blood cultures; Abx: antibiotics; ARR: absolute risk reduction; NNT: number needed to treat (the number of encounters where the alert fires to cause one blood culture collection before IV antibiotic administration). NNT are not meaningful when the ARR is not statistically significant, and are therefore reported as N/A.
Figure 3.
Bayesian structured time series (BSTS) model of daily blood culture (BCx) collections prior to IV antibiotic (Abx) administration. The darker blue solid line represents the difference between the observed number of daily blood culture collections and the predicted number of timely blood culture collections based on the BSTS synthetic control. Gray lines represent confidence intervals.
To determine if the impact of intervention decreased over time, the weekly proportion of blood culture collections relative to first IV antibiotic administration over the study course was plotted and demonstrated sustained increases in blood culture collections prior to IV antibiotics and decreases in cases where cultures were not collected or collected after antibiotics (Figure 4). The overall proportion of patients that never had blood cultures collected fell from 43.6% (95% CI 43.0–44.2) in the baseline phase to 35.2% (95% CI 34.6–35.8) in the intervention phase (see Figure 4). This increase in overall blood culture collection was seen in nearly every cohort aside from cases where IV antibiotics were given for “prophylaxis.” For patients with urinary infection sources, pneumonia, intra-abdominal infections, and skin and soft tissue infections, the intervention lowered the proportion of cases where no blood cultures were collected (see Supplementary Figure S1 accompanying the online article).
Figure 4.
Interrupted time-series of blood culture collection timing relative to first IV antibiotic administration between study phases.
On multivariable analysis, older patients were more likely to have cultures collected before IV antibiotics while women and Hispanic patients were less likely (Table 3). Patients who received antibiotics for “definitive” infection and “empiric coverage” were more likely to have timely blood cultures, as were those given antibiotics with an indication of “sepsis,” OR 13.52 (95% CI 12.25–14.92). Adjusting for demographic factors, antibiotic indication, infection type, COVID-19 infection status, and provider type, patients in the intervention phase were more likely to have timely blood culture collections, OR 2.02 (95% CI 1.94–2.11).
Table 3.
Multivariable Regression of Blood Culture Collection Prior to IV Antibiotics
|
The proportion of positive blood cultures (excluding contaminants) relative to all completed blood cultures was not statistically significant between study phases: 7.5% (95% CI 7.1–7.9) in the baseline phase versus 7.4% (95% CI 7.0–7.8%) in the intervention phase. On multivariable analysis, older patients were more likely to have positive blood cultures (excluding contaminants), as were patients with leukopenia, leukocytosis, and most importantly fever, OR 2.04 (95% CI 1.86–2.25) (Table 4). Blood culture positivity was lower when culture collection was after the administration of IV antibiotics, OR 0.54 (95% CI 0.47–0.62). Blood culture positivity was not statistically significant between study phases on adjusted analysis, OR 1.00 (95% CI 0.93–1.08).
Table 4.
Multivariable Regression of Blood Culture Positivity
|
The proportion of contaminated cultures relative to all completed blood cultures was 1.9% (95% CI 1.7–2.1) in the baseline phase and 2.3% (95% CI 2.1–2.6) in the intervention phase. In the multivariable model, there was a decrease in the odds of having contaminated blood cultures in the intervention phase compared to the baseline, OR 0.84, (95% CI 0.71–0.98) (see Supplementary Figure S2 accompanying the online article).
DISCUSSION
In the baseline phase, blood cultures were collected before first IV antibiotic administration in the ED 46.1% of the time for all patients and 81.5% of the time for patients that ultimately met the composite sepsis definition, a rate slightly higher than that reported in prior literature.7 A CDS alert that reminded ED providers to consider blood cultures when ordering IV antibiotics decreased missed or delayed blood culture collections (ARR 13.1%). Statistical significance was not reached in the composite sepsis cohort, possibly because the smaller sample size resulted in larger confidence intervals in the forecasted synthetic control, as well as the relatively high proportion (81.5%) of timely blood culture collections in the baseline period.
One could argue that rather than shifting the timing of collection (before vs after the first antibiotic administration), the CDS intervention prompted providers to order blood cultures when they were not otherwise clinically indicated. However, the observation that the blood culture positivity rate for true pathogens did not decrease in the postintervention period suggests that clinicians were not necessarily ordering many more blood cultures in patients at low risk for bacteremia. Furthermore, blood cultures are an important diagnostic test in many patients with infectious syndromes where the source of primary infection is not known, which is often the case in the ED in the early phase of the patient’s clinical course.28
Our findings demonstrated the largest increase in blood culture collections for patients being treated for pneumonia. Although blood cultures are not routinely recommended for all admitted patients with community-acquired pneumonia (CAP), the most recent consensus guidelines recommend they be drawn in patients with severe CAP who are receiving empiric antibiotics directed at multidrug resistant organisms, have a history of such organisms, or received IV antibiotics while hospitalized within the prior 90 days—criteria that apply to many patients treated for CAP at our institutions, particularly the academic hospitals.29 For patients with a urinary infection source, the utility of blood cultures is more controversial but may be appropriate when polymicrobial infection or contamination of the urine specimen is likely.30 While most patients with community-acquired intra-abdominal infections do not benefit from blood cultures, for those that are clinically toxic or immunocompromised, the presence of bacteremia may guide the duration of treatment.31 Our study did not directly evaluate the appropriateness of blood culture collection in every case, however the modest increase in blood culture collections we found may reflect the role the CDS intervention had in reminding providers to make a clinical determination of the clinical utility of blood cultures in real time.
Our CDS alert had a 62% override rate, which may seem high at first glance, but is similar to rates reported in the literature.32–34 Ideally, the CDS intervention would be more specific and only display for patients that are likely to benefit from blood culture collection, that is, those with sepsis or moderate to high risk of developing bacteremia. However, prediction of sepsis early in the clinical course is challenging, with prior work showing that SIRS criteria, qSOFA score, and machine learning models have only modest performance.35–37 As early antibiotics improve sepsis outcomes, waiting for laboratory results (ie, to identify lactic acidosis) as a requisite criteria for the CDS intervention to fire would likely have resulted in missed opportunities to collect blood cultures before the administration of antibiotics.38 The placement of the IV antibiotic order therefore seemed the right time in the clinical workflow to ask ED providers to consider whether blood cultures should be considered. The alert can be improved in future iterations by providing clearer guidance on when blood cultures would be clinically appropriate, and with narrower targeting by not firing when the antibiotic indication is for “prophylaxis.”
Limitations
Our study has several limitations. We used a retrospective pre-post time-series design which could have been strengthened by a randomized control trial. This study was conceived after the quality improvement intervention was already in place. Time-series study designs are subject to threats to internal validity. It is possible that another event aside from the CDS intervention occurred at the same time and caused an immediate or sustained effect on the outcome.
There may be minor discrepancies in the blood culture collection time, as it was extracted from the EHR. In our EHR, collection time was defined as when the lab labels were printed by the nurses before specimen collection. Nurses may in fact draw blood cultures during initial IV insertion (in anticipation that they will be ordered). In these cases, the prescribed workflow is to manually record the actual collection time when the lab label is printed and affixed to the specimen. If that does not happen however, the collection time in the EHR would default to when the label was printed versus when it was collected. However, this discrepancy is expected to be similar in both study phases. The use of other timestamps during the lab collection process is not ideal. There may be delays between when a blood culture is ordered versus when it was actually collected, and since blood cultures are collected in glass bottles, they cannot be transported in the pneumatic tube system, and therefore there may be variable delays between culture collection and when the lab indicates the specimen was received or processing was begun. For these reasons, we felt that blood culture collection time reflected the closest time to actual collection compared to other available timestamps such as when cultures were ordered or received by the lab.
We did not assess whether each blood culture order was clinically appropriate. Doing so would have required manual review of thousands of records and was beyond the scope of this study. We also did not do a comprehensive analysis of the free-text override comments beyond our usual CDS monitoring process. Additionally, we did not evaluate the economic impact this intervention had on patient care.
We also recognize that our findings may not be generalizable to other healthcare systems. Our alert is suppressed if a blood culture collection has occurred within 7 days. Healthcare systems that do not have an integrated EHR between their EDs and ambulatory practices may not be able to target the intervention as we did. Additionally, we found that some patient populations had lower proportions of timely blood culture collections than others, namely females, multiracial patients, and those with Hispanic ethnicity. Our healthcare system may not represent the same demographic distribution as other sites.
Finally, we did not perform manual review of every encounter, and therefore this study is subject to potential data inconsistencies within the EHR. For the most part, our SQL queries captured data on discrete events (eg, first blood culture collection, first IV antibiotic administration) recorded automatically within the EHR during standard workflows. We mitigated this risk through manual chart review to ensure the extracted data were accurate.
CONCLUSION
Our study demonstrates that a simple CDS intervention can improve timely blood culture collections when the concern for infection is high enough to warrant IV antibiotics, including in patients with sepsis. Our CDS tool had straightforward underlying logic and was built directly in the EHR. The CDS alert had an immediate and sustained impact and required no end-user training prior to implementation. While prior work has described the role of CDS for early detection of sepsis and to reduce blood culture utilization, we believe this is the first study that demonstrates the effect of CDS to improve timely blood culture collections.
FUNDING
This work was supported in part by the Agency for Healthcare Research and Quality (R01HS027170 to CR, MRF, and SD). The Agency for Healthcare Research and Quality had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data, preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
AUTHOR CONTRIBUTIONS
SD, DSM, and DMR conceived the study and designed the trial. CR, ASD, and MRF provided subject matter expertise. DMR and SD designed and implemented the CDS alert. SD supervised the conduct of the trial and data collection. DSM and SD extracted and managed the data and SD cleaned and analyzed the data. SD drafted the manuscript, and all authors contributed substantially to its revision. SD takes responsibility for the paper as a whole.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.
CONFLICT OF INTEREST STATEMENT
SD, DMR, DSM, and ASD report no conflict of interest. CR has received funding from Pfizer for consulting. CR has received funding from UpToDate for authorship royalties.
Supplementary Material
Contributor Information
Sayon Dutta, Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, USA; Mass General Brigham Digital Health, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA.
Dustin S McEvoy, Mass General Brigham Digital Health, Boston, Massachusetts, USA.
David M Rubins, Mass General Brigham Digital Health, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA; Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA.
Anand S Dighe, Mass General Brigham Digital Health, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA; Department of Pathology, Massachusetts General Hospital, Boston, Massachusetts, USA.
Michael R Filbin, Department of Emergency Medicine, Massachusetts General Hospital, Boston, Massachusetts, USA; Harvard Medical School, Boston, Massachusetts, USA.
Chanu Rhee, Division of Infectious Diseases, Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA; Department of Population Medicine, Harvard Medical School/Harvard Pilgrim Health Care Institute, Boston, Massachusetts, USA.
Data Availability
SQL code and R analysis code available upon request. Study data are unavailable as they contain personal health information and are confidential.
Presented at the Society of Academic Emergency Medicine (SAEM) 2022.
REFERENCES
- 1. Evans L, Rhodes A, Alhazzani W, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021. Crit Care Med 2021; 49 (11): e1063–143. [DOI] [PubMed] [Google Scholar]
- 2. Scheer CS, Fuchs C, Gründling M, et al. Impact of antibiotic administration on blood culture positivity at the beginning of sepsis: a prospective clinical cohort study. Clin Microbiol Infect 2019; 25 (3): 326–31. [DOI] [PubMed] [Google Scholar]
- 3. Pollack LA, Santen KL, van Weiner LM, et al. Antibiotic stewardship programs in U.S. acute care hospitals: findings from the 2014 National Healthcare Safety Network Annual Hospital Survey. Clin Infect Dis 2016; 63 (4): 443–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Cheng MP, Stenstrom R, Paquette K, for the FABLED Investigators, et al. Blood culture results before and after antimicrobial administration in patients with severe manifestations of sepsis: a diagnostic study. Ann Intern Med 2019; 171 (8): 547. [DOI] [PubMed] [Google Scholar]
- 5. Garnacho-Montero J, Gutiérrez-Pizarraya A, Escoresca-Ortega A, et al. De-escalation of empirical therapy is associated with lower mortality in patients with severe sepsis and septic shock. Intensive Care Med 2014; 40 (1): 32–40. [DOI] [PubMed] [Google Scholar]
- 6. Rhodes A, Evans LE, Alhazzani W, et al. Surviving sepsis campaign. Crit Care Med 2017; 45 (3): 486–552. [DOI] [PubMed] [Google Scholar]
- 7. Cardoso T, Carneiro AH, Ribeiro O, Teixeira-Pinto A, Costa-Pereira A.. Reducing mortality in severe sepsis with the implementation of a core 6-hour bundle: results from the Portuguese community-acquired sepsis study (SACiUCI study). Crit Care 2010; 14 (3): R83. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Sendak MP, Ratliff W, Sarro D, et al. Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study. JMIR Med Inform 2020; 8 (7): e15182. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Amland RC, Hahn-Cover KE.. Clinical decision support for early recognition of sepsis. Am J Med Qual 2016; 31 (2): 103–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Downing NL, Rolnick J, Poole SF, et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual Saf 2019; 28 (9): 762–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Dean NC, Jones BE, Jones JP, et al. Impact of an electronic clinical decision support tool for emergency department patients with pneumonia. Ann Emerg Med 2015; 66 (5): 511–20. [DOI] [PubMed] [Google Scholar]
- 12. Jorgensen SCJ, Lagnf AM, Bhatia S, et al. Diagnostic stewardship: a clinical decision rule for blood cultures in community-onset methicillin-resistant Staphylococcus aureus (MRSA) skin and soft tissue infections. Infect Dis Ther 2019; 8 (2): 229–42. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Woods-Hill CZ, Lee L, Xie A, et al. Dissemination of a novel framework to improve blood culture use in pediatric critical care. Pediatric Qual Saf 2018; 3 (5): e112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D.. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf 2016; 25 (12): 986–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Baysari MT, Reckmann MH, Li L, Day RO, Westbrook JI.. Failure to utilize functions of an electronic prescribing system and the subsequent generation of ‘technically preventable’ computerized alerts. J Am Med Inform Assoc 2012; 19 (6): 1003–10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (6): 523–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Gasparini A. comorbidity: an R package for computing comorbidity scores. J Open Source Softw 2018; 3 (23): 648. [Google Scholar]
- 18. Rhee C, Dantes R, Epstein L, et al. ; CDC Prevention Epicenter Program. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. JAMA 2017; 318 (13): 1241–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Buchman TG, Simpson SQ, Sciarretta KL, et al. Sepsis among medicare beneficiaries: 1. The burdens of sepsis, 2012–2018. Crit Care Med 2020; 48 (3): 276–88. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Rhee C, Kadri S, Huang SS, et al. Objective sepsis surveillance using electronic clinical data. Infect Control Hosp Epidemiol 2016; 37 (2): 163–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Brodersen KH, Gallusser F, Koehler J, Remy N, Scott SL.. Inferring causal impact using Bayesian structural time-series models. Ann Appl Stat 2015; 9 (1): 247–74. [Google Scholar]
- 22. Whiting-O’Keefe QE, Henke C, Simborg DW.. Choosing the correct unit of analysis in medical care experiments. Med Care 1984; 22 (12): 1101–14. [DOI] [PubMed] [Google Scholar]
- 23. Divine GW, Brown JT, Frazier LM.. The unit of analysis error in studies about physicians’ patient care behavior. J Gen Intern Med 1992; 7 (6): 623–9. [DOI] [PubMed] [Google Scholar]
- 24. Wickham H, Averick M, Bryan J, et al. Welcome to the Tidyverse. J Open Source Softw 2019; 4 (43): 1686. [Google Scholar]
- 25. Wickham H. ggplot2: Elegant Graphics for Data Analysis. New York: Springer-Verlag; 2016. https://ggplot2.tidyverse.org. [Google Scholar]
- 26. Bates D, Mächler M, Bolker B, Walker S.. Fitting linear mixed-effects models using lme4. J Stat Softw 2015; 67 (1): 1–51. [Google Scholar]
- 27. Sjoberg DD, Whiting K, Curry M, Lavery JA, Larmarange J.. Reproducible summary tables with the gtsummary package. R J 2021; 13 (1): 570. [Google Scholar]
- 28. Fabre V, Sharara SL, Salinas AB, Carroll KC, Desai S, Cosgrove SE.. Does this patient need blood cultures? A scoping review of indications for blood cultures in adult nonneutropenic inpatients. Clin Infect Dis 2020; 71 (5): 1339–47. [DOI] [PubMed] [Google Scholar]
- 29. Metlay JP, Waterer GW, Long AC, et al. Diagnosis and treatment of adults with community-acquired pneumonia. An official clinical practice guideline of the American Thoracic Society and Infectious Diseases Society of America. Am J Respir Crit Care Med 2019; 200 (7): e45–67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Karakonstantis S, Kalemaki D.. Blood culture useful only in selected patients with urinary tract infections—a literature review. Infect Dis-nor 2018; 50 (8): 1–9. [DOI] [PubMed] [Google Scholar]
- 31. Solomkin JS, Mazuski JE, Bradley JS, et al. Diagnosis and management of complicated intra-abdominal infection in adults and children: guidelines by the Surgical Infection Society and the Infectious Diseases Society of America. Clin Infect Dis 2010; 50 (2): 133–64. [DOI] [PubMed] [Google Scholar]
- 32. Jani YH, Barber N, Wong ICK.. Characteristics of clinical decision support alert overrides in an electronic prescribing system at a tertiary care paediatric hospital. Int J Pharm Pract 2011; 19 (5): 363–6. [DOI] [PubMed] [Google Scholar]
- 33. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS.. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21): 2625–31. [DOI] [PubMed] [Google Scholar]
- 34. H van der S, Aarts J, Vulto A, Berg M.. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (2): 138–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Gando S, Shiraishi A, Abe T, et al. ; Japanese Association for Acute Medicine (JAAM) Sepsis Prognostication in Intensive Care Unit and Emergency Room (SPICE) (JAAM SPICE) Study Group. The SIRS criteria have better performance for predicting infection than qSOFA scores in the emergency department. Sci Rep 2020; 10 (1): 8095. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 2021; 181 (8): 1065. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Song JU, Sin CK, Park HK, Shim SR, Lee J.. Performance of the quick Sequential (sepsis-related) Organ Failure Assessment score as a prognostic tool in infected patients outside the intensive care unit: a systematic review and meta-analysis. Crit Care 2018; 22 (1): 28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. Liu VX, Fielding-Singh V, Greene JD, et al. The timing of early antibiotics and hospital mortality in sepsis. Am J Respir Crit Care Med 2017; 196 (7): 856–63. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
SQL code and R analysis code available upon request. Study data are unavailable as they contain personal health information and are confidential.