Abstract
Background
In this pilot trial, we evaluated whether audit-and-feedback was a feasible strategy to improve antimicrobial prescribing in emergency departments (EDs).
Methods
We evaluated an audit-and-feedback intervention using a quasi-experimental interrupted time-series design at 2 intervention and 2 matched-control EDs; there was a 12-month baseline, 1-month implementation, and 11-month intervention period. At intervention sites, clinicians received (1) a single, one-on-one education about antimicrobial prescribing for common infections and (2) individualized feedback on total and condition-specific (uncomplicated acute respiratory infection [ARI]) antimicrobial use with peer-to-peer comparisons at baseline and every quarter. The primary outcome was the total antimicrobial-prescribing rate for all visits and was assessed using generalized linear models. In an exploratory analysis, we measured antimicrobial use for uncomplicated ARI visits and manually reviewed charts to assess guideline-concordant management for 6 common infections.
Results
In the baseline and intervention periods, intervention sites had 28 016 and 23 164 visits compared to 33 077 and 28 835 at control sites. We enrolled 27 of 31 (87.1%) eligible clinicians; they acknowledged receipt of 33.3% of feedback e-mails. Intervention sites compared with control sites had no absolute reduction in their total antimicrobial rate (incidence rate ratio = 0.99; 95% confidence interval, 0.98–1.01). At intervention sites, antimicrobial use for uncomplicated ARIs decreased (68.6% to 42.4%; P < .01) and guideline-concordant management improved (52.1% to 72.5%; P < .01); these improvements were not seen at control sites.
Conclusions
At intervention sites, total antimicrobial use did not decrease, but an exploratory analysis showed reduced antimicrobial prescribing for viral ARIs. Future studies should identify additional targets for condition-specific feedback while exploring ways to make electronic feedback more acceptable.
Keywords: ambulatory care, antimicrobial stewardship, audit-and-feedback
An audit-and-feedback stewardship intervention was implemented at 2 emergency departments (EDs); outcomes were compared with 2 control sites. At intervention sites, total antimicrobial use did not decrease compared with control sites, but antimicrobial-prescribing for viral acute respiratory infections (ARIs) declined.
Antimicrobials are frequently prescribed in both hospitals and ambulatory care, but the majority of antimicrobial prescribing is in outpatient settings [1]. An estimated 30% of all outpatient antimicrobial use is unnecessary [2, 3].
Audit and feedback is an effective strategy to improve antimicrobial prescribing [4]. Audit and feedback involves reviewing a clinician’s antimicrobial-prescribing behavior and providing feedback to help the clinician adjust his/her performance. The use of audit and feedback for outpatient antimicrobial stewardship has been largely studied in primary care clinics [5–9] with limited data in emergency departments (EDs) and urgent care centers [10–13].
Emergency departments are a major provider of medical care in the United States and, in turn, an important partner in efforts to improve antimicrobial prescribing [14]. Prior studies have demonstrated several opportunities to improve antimicrobial prescribing in EDs [15–19].
In this quasi-experimental pilot study, we evaluated whether the use of audit and feedback with peer-to-peer comparisons could reduce unnecessary antimicrobial use at 2 intervention EDs compared with 2 matched-control EDs.
METHODS
Study Design
We performed a quasi-experimental study with an interrupted time-series design and a matched-pair nonequivalent control group to evaluate the effect of a pilot intervention on clinicians at 2 participating EDs versus 2 control EDs [20]. This study design provided a practical approach to explore our research question using a pilot trial. All EDs were affiliated with Veterans Affairs Medical Centers (VAMCs) in the midwestern United States.
Site Selection
First, we contacted VAMCs within 2 midwestern Veterans Integrated Service Networks (VISNs) to assess their interest in participating in this study. These VISNs were chosen because they were among the few VISNs that had not implemented regional initiatives to improve antimicrobial use in ambulatory care. The 2 selected intervention VAMCs each had a 24 hours a day, 7 days a week ED staffed by a mix of full-time and part-time clinicians. Intervention site 1 exclusively had physicians, whereas intervention site 2 also had some advanced practice providers. Some physicians were board-certified in Emergency Medicine, whereas others held board certification in Internal Medicine or Family Practice. There were no moonlighters at either intervention site.
Eligible control sites had a baseline total antimicrobial-prescribing rate (outpatient antimicrobial prescriptions dispensed per total ED patient-visits) that was similar to the intervention sites (see Outcomes). Selection of controls on the preintervention outcome of interest minimized bias and reduced the impact of regression to the mean, which is a key threat when selecting poor performers in nonrandomized trials [20]. To further ensure that intervention and control sites were as similar as possible, control and intervention sites were also matched on their hospital complexity level, as defined by the Veterans Health Administration (VHA).
Study Interventions
The entire study period was 2 years in duration. The baseline period covered 1 year between October 1, 2017 through September 30, 2018, the implementation phase lasted 1 month (October 2018), and the intervention period spanned another 11 months (November 1, 2018 through September 30, 2019). The intervention consisted of an initial one-on-one educational meeting with each willing clinician as well as feedback delivered at baseline and via a quarterly e-mail.
Initial Meeting With Study Participants
In October 2018, the study team had meetings with all willing ED clinicians at the intervention sites. Names of eligible clinicians were provided by each ED’s medical director. At intervention site 1, these meetings were face-to-face and took place over 2 days when D.J.L., an Infectious Disease (ID) physician, was present onsite. At intervention site 2, D.J.L. enrolled ED clinicians by telephone, whereas an onsite ID physician (A.D.) met with clinicians in-person.
These one-on-one meetings with each clinician lasted 10–15 minutes and included a review of the clinician’s individual antimicrobial-prescribing data (see Feedback on Antimicrobial-Prescribing), an anonymous comparison to his/her local peers, and a review of antimicrobial-prescribing guidelines. This guideline review included a page-by-page discussion of a VA quick reference guide on the diagnosis and management of acute respiratory infections (ARIs) with a specific focus on not prescribing antimicrobials for viral ARIs. The guideline review also included a short discussion on the correct interpretation of urine studies, including the importance of not prescribing antimicrobials for patients with abnormal urinalyses but no acute genitourinary symptoms of infection (Supplemental Table 1). All enrolled ED clinicians were given a copy of these educational materials. After this initial meeting, no further education was provided. There was no education at control sites.
Feedback on Antimicrobial Prescribing
Feedback individualized to each clinician was provided at baseline during the initial meeting and via quarterly e-mails. Feedback consisted of 2 metrics: (1) the clinician’s total antimicrobial-prescribing rate (ie, the clinician’s total number of antimicrobial-prescribing visits divided by the clinician’s total number of ED visits) and (2) the clinician’s rate of prescribing antimicrobials for uncomplicated ARIs. The uncomplicated ARI metric captured how often an antibacterial agent (hereafter “antimicrobial”) was prescribed for visits coded as acute bronchitis or an unspecified upper respiratory tract infection (URI), as shown in Supplemental Table 2. It excluded visits for patients who were immunosuppressed, had chronic lung disease, or had a concurrent infection [12].
For the enrollment meeting with clinicians, both metrics were based on the year preceding the initiation of the study. Once the intervention period began, metrics were only drawn from the intervention period. Data on the uncomplicated ARI metric were only provided if the clinician had seen ≥8 eligible ARI cases during the prior quarter; if 8 eligible visits did not occur during the prior quarter, then more than one quarter of data could be included as long as the data were all from the intervention period. Even if a clinician did not qualify for receiving feedback on the uncomplicated ARI metric, feedback was still provided every quarter on the clinician’s total antimicrobial-prescribing rate.
After the initial meeting, feedback was delivered quarterly via e-mail. We believed that, by providing a larger sample size, data aggregated over one quarter would provide a more meaningful assessment than data aggregated over a shorter time increment (eg, monthly).
Peer-to-peer comparisons were leveraged. At enrollment, clinicians were compared to their peers for both metrics. In subsequent e-mail communications, clinicians were compared to their peers and to their own prior performance for the uncomplicated ARI metric. For the total antimicrobial-prescribing rate, e-mails only compared clinicians to their personal baseline performance; the lack of risk adjustment for differences in patient case-mix across clinicians precluded effective peer-to-peer comparisons for this specific metric. The wording used in all feedback reports was based on the MITIGATE toolkit (Supplemental Table 3) [21]. All peer-to-peer comparisons were also demonstrated graphically in an attachment to the e-mail (Supplemental Figure 1).
In accordance with VA regulations, all e-mail feedback was sent to clinicians’ VA e-mail accounts. To assess whether the clinician received the e-mail feedback, we requested a read receipt for all e-mail communication. The percentage of read receipts that were acknowledged was tracked over time.
We did not provide feedback to ED clinicians at the 2 control sites. The control sites were not aware that their antimicrobial prescribing was being monitored.
Other Stewardship Processes
Beginning in late September 2018, a clinical pharmacist was assigned to the ED at intervention site 2 from 10 am to 10 pm every day. One of the clinical pharmacist’s responsibilities was to prospectively monitor medication prescribing, including antimicrobial selection. These pharmacists were not part of this trial, and the study team was only made aware of their presence as the intervention period was about to begin. To account for this, separate analyses were performed for intervention sites 1 and 2. To our knowledge, there were no other ED stewardship interventions at the intervention or control sites during the baseline or intervention periods.
Data Collection
Data on oral antimicrobial prescriptions, demographics, comorbidities, infections, and laboratory data for ED visits during the study period were extracted from the Corporate Data Warehouse using the VA Informatics and Computing Infrastructure. The International Classification of Diseases, Tenth Revision (ICD-10) codes were used to identify existing comorbidities and infection diagnoses linked to the ED visit. Data for the uncomplicated ARI metric were collected from the VA Academic Detailing Service’s internal ARI dashboard; this study’s uncomplicated ARI metric was identical to the dashboard’s metric for uncomplicated bronchitis/URI not otherwise specified.
Outcomes
The primary outcome was the total antimicrobial-prescribing rate, defined as the cumulative frequency of prescribing an outpatient antimicrobial prescription within 24 hours of the ED visit. Emergency department visits that resulted in an inpatient admission within 24 hours after the visit were not eligible for inclusion in the outcome. Antimicrobials were defined as all agents included in the National Healthcare Safety Network’s Antimicrobial Use and Resistance Module [22]. On a monthly basis, the cumulative number of outpatient antimicrobial-prescribing visits by the ED and the total number of eligible ED visits were used as the numerator and denominator, respectively. This outcome was chosen because it would not be influenced by diagnostic shifting [12, 23]. Several exploratory analyses were performed. (1) The uncomplicated ARI metric, as defined above, was captured across all clinicians at the control and intervention sites (see Feedback on Antimicrobial-Prescribing). (2) Guideline-concordant management was assessed using manual chart reviews for 6 conditions: acute bronchitis, acute exacerbations of chronic obstructive pulmonary disease (AE-COPD), acute sinusitis, cystitis, pharyngitis, and URI (see Manual Chart Reviews to Assess Guideline-Concordant Management). These conditions were chosen because their optimal management was reviewed in the one-on-one educational sessions at the onset of the intervention. (3) Several safety outcomes were measured from 24 hours after the ED visit up to 30 days after the visit. These safety outcomes included testing for the presence of Clostridioides difficile, confirmed C difficile infection, return ED visit, late antimicrobial use, and admission to a VHA acute-care facility. Clostridioides difficile linked to a visit was confirmed if the patient had a positive C difficile test and a qualifying antimicrobial (metronidazole, oral vancomycin, or fidaxomicin) was dispensed within 7 days of the test. Safety outcomes, except return ED visits, included all eligible visits in the denominator. A return visit was any visit that occurred from 24 hours up to 30 days after the prior visit, and ED visits that occurred after 30 days from the prior ED visit were considered as independent visits.
Statistical Analysis
An interrupted times-series analysis was used to evaluate the effect of audit and feedback between the intervention and control sites on the primary outcome. The crude differences in patient characteristics between the baseline and intervention periods were assessed for intervention and control sites using the Fisher’s exact test and Wilcoxon rank-sum test. Monthly proportions of antimicrobial use were calculated at each site for the baseline and intervention time periods. Segmented regression analysis was conducted using generalized linear models to estimate change in monthly total antimicrobial prescription rates between baseline and intervention periods for intervention and control sites, respectively. Models were adjusted for continuous variables for time (months before, during, and after implementation). The estimate for time during implementation was used to calculate the immediate change in prescription rates and was reported as incident rate ratios (IRRs) with 95% confidence intervals (CIs). Indicator variables for the 4 seasons were also included in each model to adjust for seasonal trends. An R-side random statement was included in each model to vary the degree of overdispersion for each facility. Finally, a difference-in-difference estimate was reported for the change in monthly total prescription rates between intervention and control sites in the intervention period while controlling for changes in the baseline period.
A Poisson regression model with log link was used to estimate the rate of safety outcomes. Models were adjusted for time as a continuous covariate, an indicator for month of implementation of intervention (October 2018), and included random effects to account for repeated measurements among sites for each month during the study period. An interaction variable for the study period (baseline or intervention) and site (intervention or control) was included in the model to estimate the IRR and 95% CIs.
Manual Chart Reviews to Assess Guideline-Concordant Management
Patient-visits were eligible for manual chart review if the visit met the following criteria: (1) a patient’s first ED visit during the study period; (2) a qualifying infection diagnosis, as defined by ICD-10 codes; and (3) no evidence of immunosuppression. Qualifying diagnoses were acute bronchitis, AE-COPD, acute sinusitis, cystitis, pharyngitis, and URI (Supplemental Table 4). Immunosuppression was defined, as shown in Supplemental Table 4. At each of the 4 EDs, eligible patient-visits were randomly chosen for review. Two independent, blinded reviewers evaluated selected medical records while using defined algorithms (Supplemental Figures 2–7). Reviewers assessed the accuracy of the diagnosis and, if necessary, reclassified the infection type; they also evaluated a decision to prescribe or not prescribe antimicrobials as well as the antimicrobial selection and duration, as informed by guidelines. Reviewers consisted of a fourth-year medical student, an Internal Medicine resident physician, and an Infectious Disease fellow physician. If the initial reviewers’ assessment were discordant, a third reviewer, who was not blinded, reviewed the medical record. The third reviewer (D.J.L.) had to evaluate 128 (26.7%) charts during the baseline period and 135 (29.7%) during the intervention period. Based on available resources, the goal was to include at least 110 adjudicated charts per site for the baseline and intervention periods. Guideline-concordant management was defined as either prescribing an antimicrobial when indicated or not prescribing an antimicrobial when not indicated.
Patient Consent Statement
The clinical trial was registered on ClinicalTrials.gov, NCT03349567. The Institutional Review Boards (IRBs) at the University of Iowa and Indiana University as well as the Research & Development Committee of the Iowa City and Indianapolis VAMCs approved this study and waived written informed consent. All other sites were not engaged in research activities, so local IRB approval was not sought.
RESULTS
Enrollment of and Feedback to Emergency Department Clinicians
In all, 27 of 31 (87.1%) eligible ED clinicians were enrolled, including 8 of 9 (88.9%) at intervention site 1 and 19 of 22 (86.4%) at intervention site 2. All eight initial meetings at site 1 were conducted in-person. At site 2, 6 of 19 (31.6%) initial meetings were done over telephone and 13 of 19 (68.4%) were conducted in-person. Two clinicians from intervention site 2 later withdrew from the study; one did not give a reason for withdrawing, whereas the other expressed frustration at not being evaluated more favorably on the study’s feedback metrics. There were 13 and 29 unique providers at control sites 1 and 2, respectively.
Feedback was provided to all enrolled clinicians at intervention sites every quarter. However, there was insufficient data to provide feedback on the uncomplicated ARI metric for 12 (42.9%) clinicians during quarter 1, 3 (10.7%) during quarter 2, and 1 (3.6%) during quarter 3.
During quarter 1, we received e-mail read receipts from 14 (51.9%) participants. This decreased to 10 (38.5%) read receipts during the second quarter and 2 (7.7%) in the third quarter. In all, only 33.3% of all e-mails were acknowledged; 18 (66.7%) of all clinicians acknowledged receipt of at least 1 feedback e-mail.
Characteristics of Patients Seen in Emergency Departments During Baseline and Intervention Periods
In the baseline and intervention periods, intervention sites had 28 016 and 23 164 visits, respectively, compared to 33 077 and 28 835 visits at control sites. Table 1 summarizes patient-visit characteristics during the baseline and intervention periods.
Table 1.
Characteristics of Patient-Visits at 2 Intervention and 2 Control Emergency Departments, Stratified by Baseline and Intervention Periods
| Intervention Sites | Control Sites | |||
|---|---|---|---|---|
| Characteristics | Baseline | Intervention | Baseline | Intervention |
| Aggregated Frequencies | ||||
| Number of visits | 28 016 | 23 164 | 33 077 | 28 835 |
| Number of visits associated with an antimicrobial prescription, n (%) | 6265 (22.4%) | 4554 (19.7%) | 7164 (21.7%) | 6202 (21.5%) |
| Monthly Frequencies | ||||
| Median number of visits (IQR) | 2198 (2084–2219) | 2105 (2042–2206) | 2556 (2483–2648) | 2666 (2391–2863) |
| Median antimicrobial prescriptions | 21.2% (20.3%–23.7%) | 19.1% (17.8%–20.4%) | 20.9% (19.4%–23.4%) | 21.6% (19.9%–22.4%) |
| Median age (IQR), years | 60 (58–62) | 60 (59–62) | 60 (59–60) | 60 (59–61) |
| Male | 89.5% | 88.8% | 90.2% | 90.7% |
| Race | ||||
| White | 75.2% | 74.2% | 83.7% | 83.1% |
| Black | 15.3% | 15.5% | 9.4% | 9.6% |
| Other | 1.6% | 1.1% | 2.2% | 2.3% |
| Unknown | 6.9% | 7.5% | 5.1% | 5.3% |
| Immunosuppresseda | 5.3% | 4.8% | 5.8% | 5.8% |
| Infections | ||||
| Acute COPD | 1.7% | 1.9% | 1.2% | 1.6% |
| Bronchitis | 3.2% | 2.6% | 3.4% | 3.3% |
| UTI | 2.3% | 2.3% | 2.5% | 2.2% |
| Pharyngitis | 1.1% | 1.2% | 1.2% | 1.5% |
| Pneumonia | 0.6% | 0.9% | 0.8% | 0.8% |
| Sinusitis | 1.6% | 1.4% | 1.8% | 1.9% |
| SSTI | 3.6% | 3.5% | 4.3% | 4.4% |
| URI | 1.4% | 1.9% | 1.9% | 1.8% |
| Diverticulitis | 0.4% | 0.3% | 0.3% | 0.4% |
| Comorbidities | ||||
| Alcohol abuse | 10.5% | 11.7% | 10.3% | 11.9% |
| CHF | 7.8% | 9.5% | 7.7% | 8.8% |
| Dementia | 3.6% | 3.4% | 2.8% | 2.9% |
| Diabetes | 29.9% | 31.9% | 23.8% | 26.5% |
| Drug abuse | 6.7% | 8.2% | 7.3% | 9.2% |
| Liver disease, severe | 0.6% | 0.8% | 0.5% | 0.7% |
| Neurological disorders | 5.2% | 5.3% | 4.6% | 5.6% |
| Paralysis | 0.6% | 0.7% | 0.9% | 1.1% |
| PVD | 9.4% | 9.9% | 6.4% | 8.9% |
Abbreviations: CHF, congestive heart failure; COPD, chronic obstructive pulmonary disease; IQR, interquartile range; PVD, peripheral vascular disease; SSTI, skin and soft tissue infection; URI, upper respiratory tract infection; UTI, urinary tract infection
aImmunosuppression was defined as having a diagnosis of lymphoma, leukemia, human immunodeficiency virus/acquired immune deficiency syndrome, or organ transplantation during the 12 months before presentation or receipt of an immunosuppressive medication, which was defined as follows: prednisone or steroid equivalent at a dose ≥20 mg/day during the 30 days before admission, chemotherapy within the 30 days before presentation, or an antirejection medication, biologic agent, or a disease-modifying antirheumatic drug within the 3 months before presentation.
Changes in Total Antimicrobial Prescription Rates
There was an immediate, nonsignificant reduction in monthly total antimicrobial prescription rates (IRR = 0.77; 95% CI, 0.54–1.09) at intervention site 1 (Figure 1). During the intervention period, there was a nonsignificant absolute reduction of 1% in the monthly total antimicrobial-prescribing rate between intervention sites compared to control sites while controlling for changes in the baseline period (IRR = 0.99; 95% CI, 0.98–1.01; P = .35) (Table 2). As shown in Supplemental Figure 8, there was no graphical difference in total antimicrobial-prescribing rates between clinicians at intervention sites who acknowledged receipt of e-mail feedback and those who did not.
Figure 1.
(a) Comparison of monthly total antimicrobial-prescribing rates between the baseline and intervention periods at 2 intervention emergency departments. (b) Comparison of monthly total antimicrobial-prescribing rates between the baseline and intervention periods at 2 control emergency departments.
Table 2.
Effect of the Intervention on the Monthly Total Antimicrobial Prescribing Rates at 2 Intervention and 2 Control Emergency Departments, Stratified by Baseline and Intervention Periods
| Outcome | Baseline Monthly Prescription Rate | Immediate Change in Prescription Rate IRR (95% CI) | Monthly Prescription Rate | Reduction in Monthly Prescription Rate (Difference-in-Difference) | |
|---|---|---|---|---|---|
| Baseline Trend | Intervention Trend | ||||
| Antimicrobial Use | |||||
| Intervention Site 1 | 17.9% | 0.77 (0.54–1.09) | −0.9% | +1.0% | 0.99% (0.98–1.01) (P = .35) |
| Intervention Site 2 | 21.4% | 1.11 (0.93–1.33) | −0.9% | −0.9% | |
| Control Site 1 | 20.8% | 1.21 (0.94–1.55) | −0.9% | +1.0% | |
| Control Site 2 | 20.9% | 1.13 (0.93–1.38) | −0.9% | +1.0% | |
Abbreviations: CI, confidence interval; IRR, incident rate ratio.
Changes in Antimicrobial-Prescribing for Visits Coded as Uncomplicated Acute Respiratory Infections (Exploratory Analysis)
At baseline, antimicrobial use for uncomplicated ARIs accounted for 15.8% of all antimicrobial-prescribing visits at intervention sites and 8.9% of all antimicrobial-prescribing visits at control sites (Supplemental Table 5). In comparing the baseline and intervention periods, antimicrobial-prescribing for uncomplicated ARIs increased from 56.8% to 62.5% at control sites (P < .01) while decreasing from 68.6% to 42.4% at intervention sites (P < .01). Figure 2 shows time-series data for changes in antimicrobial-prescribing in visits coded as uncomplicated ARIs. Graphically, antimicrobial use for uncomplicated ARIs remained stable at control sites but decreased at intervention sites.
Figure 2.
Antimicrobial prescribing in emergency department (ED) visits associated with a diagnostic code for an uncomplicated acute respiratory tract infection at intervention and control sites during October 2017–September 2018 (baseline period) and October 2018–September 2019 (intervention period).
Guideline-Concordant Management, Based on Manual Chart Reviews (Exploratory Analysis)
Supplemental Tables 6 and 7 provide an overview of how many patient-visits were eligible for review and how many were excluded. In all, full-chart adjudication was performed on 480 visits during the baseline period and 455 during the intervention period.
In comparing the baseline and intervention periods, guideline-concordant management improved from 52.1% to 72.5% (P < .01) at intervention sites compared to 51.3% to 58.2% (P = .13) at control sites (Table 3). If cases reviewed by the nonblinded third reviewer are excluded, guideline-concordant management still improved at the intervention sites (88 of 176 [50.0%] to 133 of 170 [78.2%], P < .001) but not at control sites (94 of 176 [53.4%] to 93 of 150 [62.0%], P = .12). The improvements in guideline-concordant management at intervention sites were driven by declines in prescribing antimicrobials for viral ARIs (Supplemental Table 8). Table 3 also shows findings on antimicrobial selection and duration in cases in which an antimicrobial was indicated.
Table 3.
Guideline-Concordant Management Based on Findings From Manual Chart Reviews
| Intervention Sites | Control Sites | |||
|---|---|---|---|---|
| Type of Assessment | Baseline Period (n = 240) | Intervention Period (n = 230) | Baseline Period (n = 240) | Intervention Period (n = 225) |
| Coded diagnosis changed by reviewersa | 46 (19.2%) | 13 (5.7%) | 35 (14.6%) | 13 (5.8%) |
| Guideline-concordant managementb,c | 125 (52.1%) | 166 (72.5%) | 123 (51.3%) | 131 (58.2%) |
| Antimicrobial-prescribed when indicated | 55 (22.9%) | 73 (31.7%) | 54 (22.5%) | 69 (30.7%) |
| Antimicrobial not prescribed when not indicated | 70 (29.2%) | 93 (40.4%) | 69 (28.8%) | 62 (27.6%) |
| Guideline-concordant selection | 51/55 (92.7%) | 68/73 (93.2%) | 50/54 (92.6%) | 66/69 (95.7%) |
| Guideline-concordant duration | 35/55 (63.6%) | 51/73 (69.9%) | 34/54 (63.0%) | 44/69 (63.8%) |
aIf there was discrepancy between the diagnostic code and the reviewer’s assessment on either upper respiratory tract infection or acute bronchitis, the consensus diagnosis was considered to be a viral respiratory tract infection and a change in diagnosis was deemed to not have been made.
bWhen management was not guideline-concordant, it was almost always because an antimicrobial was prescribed when not indicated. There were only 1 case across both study periods in which the reviewers thought an antimicrobial was indicated but an antimicrobial had not been prescribed.
cGuideline-concordant management improved in the intervention sites (52.1% to 72.5%, P < .001) but not at the control sites (51.3% to 58.2%, P = .13). If sites reviewed by the third reviewer are excluded, guideline-concordant management improved at the intervention sites (88 of 176 [50.0%] to 133 of 170 [78.2%], P < .001) but not at control sites (94 of 176 [53.4%] to 93 of 150 [62.0%], P = .12).
Safety Outcomes
During the intervention period, there were no statistically significant differences in the IIRs of monthly safety outcomes between intervention and control sites, while controlling for changes in the baseline period (Table 4).
Table 4.
Median Outcomes at 30 Days for Patient-Visits to 2 Intervention and 2 Control Emergency Departments, Stratified by Baseline and Intervention Periods
| Intervention Sites | Control Sites | ||||
|---|---|---|---|---|---|
| Secondary Outcomesa | Baseline % | Intervention % | Baseline % | Intervention % | Adjusted IRR (95% CI) |
| Repeat ED visit | 18.4 | 19.2 | 18.2 | 19.1 | 1.01 (0.85–1.19) |
| Inpatient admission | 5.9 | 5.9 | 5.6 | 5.8 | 1.01 (0.81–1.27) |
| Late antimicrobial use | 31.5 | 31.1 | 29.5 | 28.5 | 1.12 (0.93–1.35) |
| Clostridioides difficile testing | 0.60 | 0.69 | 0.86 | 0.89 | 0.83 (0.53–1.29) |
| C difficile infectionb,c | 0 | 0 | 0 | 2.5 | 1.41 (0.88–2.25) |
| All-cause mortality | 0.56 | 0.60 | 0.61 | 0.59 | 0.88 (0.58–1.34) |
Abbreviations: CI, confidence interval; ED, emergency departments; IRR, incident rate ratio.
aAll secondary outcomes were measured from 24 hours after the ED visit up to 30 days after the visit.
b Clostridioides difficile infection was confirmed if the patient had a positive C difficile test and a qualifying antimicrobial agent (metronidazole, oral vancomycin, or fidaxomicin) was dispensed within 7 days of the test.
cAlthough the median is reported as 0, there were still C difficile infections that occurred during both the baseline and intervention periods. The median absolute count varied from 1–2 infections per month.
DISCUSSION
After the implementation of audit and feedback at 2 EDs, total antimicrobial use did not decrease at intervention sites compared with control sites. However, antimicrobial use for viral ARIs decreased at the intervention sites, based on an exploratory analysis that included manual chart reviews of 6 infection types and an electronic evaluation of visits coded as uncomplicated ARIs (ie, acute bronchitis or URI).
Our feedback to clinicians focused on 2 metrics: (1) the frequency of prescribing antimicrobials for uncomplicated ARIs and (2) each clinician’s frequency of prescribing antimicrobials for all patient-visits. Like prior outpatient behavioral interventions for stewardship, we provided repeated individualized feedback that leveraged peer-to-peer comparisons for the uncomplicated ARI metric [5, 10, 12, 24]. The feedback on antimicrobial use for uncomplicated ARIs was specific and actionable, which are both characteristics of effective feedback; in turn, antimicrobial prescribing for ARIs seems to have declined.
Providing clinicians feedback on their total antimicrobial-prescribing rate was based on the premise that clinicians who more frequently prescribe antimicrobials are more likely to prescribe unnecessary antimicrobials [25, 26]. However, this type of feedback may have been too nonspecific to drive behavior change, because it did not identify the specific situations in which antimicrobials were unnecessarily prescribed. In addition, we chose not to make quarterly peer-to-peer comparisons on this unadjusted metric, and this too may have limited its impact. Based on existing literature, it is unclear whether feedback on total antimicrobial-prescribing volume is effective. In one study, general practitioners in England decreased their antibiotic use after receiving feedback on their practice’s antibiotic-prescribing rate, compared with other practices [27]. However, the feedback came from the country’s Chief Medical Officer, so it may have been more influential than the feedback in our study, which came from the study’s lead physician who was not otherwise known to participants. In line with our study, primary care physicians in Switzerland who received quarterly feedback on the frequency of their antibiotic prescribing did not change their antibiotic use [28].
We encountered several additional challenges with providing feedback. First, we had difficulty confirming that our electronic feedback had actually been received by participants. As the study progressed, fewer and fewer participants acknowledged receipt of our e-mail feedback. It is unclear whether the e-mail content had been read but the read receipt was just not sent. All feedback e-mails were sent to participants’ VA e-mail accounts, which may have been infrequently checked. Such a low and declining rate of acknowledgment suggests that ED clinicians may have found the feedback less acceptable over time and represents a barrier to the feasibility of our intervention. Second, a large proportion of clinicians had insufficient data to qualify for feedback on the uncomplicated ARI metric during quarter 1, which may have limited our ability to influence antimicrobial prescribing for ARIs. Third, we only provided feedback every quarter, because we believed that quarterly feedback would ensure an adequate sample size for each clinician. We acknowledge that providing more frequent feedback may have been more effective at changing behavior. However, some clinicians had few uncomplicated ARI visits, so it may have been challenging to make more frequent peer-to-peer comparisons that were still meaningful. Fourth, our intervention did not address several important barriers to acting on our feedback. For example, we did not address the challenge of diagnostic uncertainty, time constraints, or patients frequently demanding antimicrobial therapy [29, 30].
Without question, there is a need to develop effective antimicrobial stewardship strategies for EDs [31, 32]. To our knowledge, our pilot trial is the third multicenter stewardship study that leveraged audit and feedback to improve antimicrobial prescribing in EDs [10, 12]. Both prior studies, which found audit-and-feedback to be effective, provided their feedback on a metric similar to our uncomplicated ARI metric, and both measured antimicrobial use solely for ARI visits. One of these studies provided feedback on a monthly basis, which was more frequent than our study, whereas the other provided feedback every 2–3 months [10, 12]. Although our study did not show a change in our primary outcome, our intervention seems to have reduced unnecessary antimicrobial use for viral ARIs. It is unclear whether this improvement was the result of our initial education, the quarterly feedback, or simply the result of ED clinicians’ awareness that their antimicrobial prescribing was being monitored. Because uncomplicated ARIs (ie, acute bronchitis and URI) only accounted for a small percentage of all antimicrobial-prescribing visits, providing specific feedback for this condition may have had a limited impact on total antimicrobial use, our primary outcome. Although a metric that included all ARIs would have captured a higher proportion of visits, certain ARIs (eg, sinusitis) sometimes require antimicrobial therapy, so interpretation of such an expanded metric could have been difficult.
Future stewardship studies on audit and feedback in EDs should incorporate additional strategies, such as local champions, to improve the adoption and sustainability of the intervention. Future studies could also evaluate the perceived acceptability of the feedback and optimal methods for delivering feedback, including for infections other than ARIs. To this end, measuring implementation outcomes, as was done in at least 1 prior ED stewardship study, would be helpful [10, 33]. Finally, as done in our study, future studies should include clinical outcomes to ensure safety and measure antimicrobial-prescribing outcomes that are independent of diagnostic codes.
Our study has a few limitations that should be acknowledged. First, because the study was not randomized, we cannot exclude temporal confounding or selection bias. By conducting a time-series analysis, we tried to account for confounders that may change over time, and by including a matched-control group, we tried to minimize the possibility of regression to the mean. Second, we cannot exclude the possibility that sites had any other processes in place that could have influenced our outcomes. In fact, intervention site 2 had assigned 2 dedicated ED pharmacists to their ED the month before our intervention started. However, based on our analysis, the baseline and intervention trends in antimicrobial use did not differ in the 2 intervention sites, which suggests that these ED pharmacists did not have an added effect on the primary outcome. Third, this intervention only lasted for 12 months, so we cannot comment on its sustainability, but the low rate at which our e-mails were acknowledged suggests that sustainability may be difficult.
CONCLUSIONS
In conclusion, the implementation of audit and feedback at 2 EDs raised questions about the acceptability of this type of electronic feedback. Although our pilot intervention did not decrease total antimicrobial use, an exploratory analysis showed that it decreased antimicrobial prescribing for viral ARIs. There is a continued need to improve antimicrobial prescribing in EDs, and audit and feedback remains a promising tool. Future studies should be sufficiently powered and develop strategies to optimize the delivery of electronic feedback. Future studies should also explore the added benefit of using further behavioral interventions and the effectiveness of giving feedback for infections other than ARIs.
Supplementary Data
Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.
Acknowledgments
Author contributions. D. J. L., R. N., and E. N. P. conceived the study, designed the trial, and obtained research funding. D. J. L. and A. D. recruited providers to participate. D. J. L. drafted the manuscript with input from A. D., A. A., B. A., M. W. S., S. K., and E. N. P. All authors contributed substantially to the manuscript’s revision.
Disclaimer. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States Government.
Financial support. This work was funded by the Merck Investigator Studies Programs.
Potential conflicts of interest. All authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.
References
- 1. Suda KJ, Hicks LA, Roberts RM, et al. Antibiotic expenditures by medication, class, and healthcare setting in the United States, 2010–2015. Clin Infect Dis 2018; 66:185–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Fleming-Dutra KE, Hersh AL, Shapiro DJ, et al. Prevalence of inappropriate antibiotic prescriptions among US ambulatory care visits, 2010–2011. JAMA 2016; 315:1864–73. [DOI] [PubMed] [Google Scholar]
- 3. Cosgrove SE, Seo SK, Bolon MK, et al. ; CDC Prevention Epicenter Program. Evaluation of postprescription review and feedback as a method of promoting rational antimicrobial use: a multicenter intervention. Infect Control Hosp Epidemiol 2012; 33:374–80. [DOI] [PubMed] [Google Scholar]
- 4. Barlam TF, Cosgrove SE, Abbo LM, et al. Implementing an antibiotic stewardship program: guidelines by the Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America. Clin Infect Dis 2016; 62:e51–77. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Meeker D, Linder JA, Fox CR, et al. Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA 2016; 315:562–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Gjelstad S, Høye S, Straand J, et al. Improving antibiotic prescribing in acute respiratory tract infections: cluster randomised trial from Norwegian general practice (prescription peer academic detailing (Rx-PAD) study). BMJ 2013; 347:f4403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Vinnard C, Linkin DR, Localio AR, et al. Effectiveness of interventions in reducing antibiotic use for upper respiratory infections in ambulatory care practices. Popul Health Manag 2013; 16:22–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Madridejos-Mora R, Amado-Guirado E, Pérez-Rodríguez MT. Effectiveness of the combination of feedback and educational recommendations for improving drug prescription in general practice. Med Care 2004; 42:643–8. [DOI] [PubMed] [Google Scholar]
- 9. Buehrle DJ, Shively NR, Wagener MM, et al. Sustained reductions in overall and unnecessary antibiotic prescribing at primary care clinics in a veterans affairs healthcare system following a multifaceted stewardship intervention. Clin Infect Dis 2020; 71:e316–22. [DOI] [PubMed] [Google Scholar]
- 10. Yadav K, Meeker D, Mistry RD, et al. A multifaceted intervention improves prescribing for acute respiratory infection for adults and children in emergency department and urgent care settings. Acad Emerg Med 2019; 26:719–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Cummings PL, Alajajian R, May LS, et al. Utilizing behavioral science to improve antibiotic prescribing in rural urgent care settings. Open Forum Infect Dis 2020; 7:ofaa174. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Madaras-Kelly KJ, Hostler C, Townsend M, et al. Impact of implementation of the core elements of outpatient antibiotic stewardship within Veterans Health Administration clinics on antibiotic prescribing and patient outcomes clinical infectious diseases. Clin Infect Dis 2020; doi: 10.1093/cid/ciaa1831 [DOI] [PubMed] [Google Scholar]
- 13. Buehrle DJ, Phulputo RH, Wagener MM, et al. Decreased overall and inappropriate antibiotic prescribing in a Veterans Affairs Hospital Emergency Department following a peer comparison-based stewardship intervention. Antimicrob Agents Chemother 2020; 65:e01660–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Centers for Disease Control and Prevention. National Hospital Ambulatory Medical Care Survey: 2017 Emergency Department Summary tables. Available at: https://www.cdc.gov/nchs/data/nhamcs/web_tables/2017_ed_web_tables-508.pdf. Accessed May 19 2021.
- 15. Meyer HE, Lund BC, Heintz BH, et al. Identifying opportunities to improve guideline-concordant antibiotic prescribing in veterans with acute respiratory infections or cystitis. Infect Control Hosp Epidemiol 2017; 38:724–8. [DOI] [PubMed] [Google Scholar]
- 16. Timbrook TT, Caffrey AR, Ovalle A, et al. Assessments of opportunities to improve antibiotic prescribing in an emergency department: a period prevalence survey. Infect Dis Ther 2017; 6:497–505. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Denny KJ, Gartside JG, Alcorn K, et al. Appropriateness of antibiotic prescribing in the emergency department. J Antimicrob Chemother 2019; 74:515–20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Chua KP, Fischer MA, Linder JA. Appropriateness of outpatient antibiotic prescribing among privately insured US patients: ICD-10-CM based cross sectional study. BMJ 2019; 364:k5092. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Lowery JL, Alexander B, Nair R, et al. Evaluation of antibiotic prescribing in emergency departments and urgent care centers across the Veterans’ Health Administration. Infect Control Hosp Epidemiol 2020: 1–8. [DOI] [PubMed] [Google Scholar]
- 20. Shadish WR, Cook TD, Campbell DT.. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Belmont, CA: Wadsworth Cengage Learning; 2002. [Google Scholar]
- 21. May L, Yadav K, Gaona SD, et al. MITIGATE Antimicrobial Stewardship Toolkit: a guide for practical implementation in adult and pediatric emergency department and urgent care settings. Available at: http://shea-online.org/images/priority-topics/MITIGATE_TOOLKIT_final.pdf. Accessed 18 January 2021.
- 22. National Healthcare Safety Network. Antimicrobial Use and Resistance Module. Atlanta, GA: Centers for Disease Control and Prevention; 2021. [Google Scholar]
- 23. Sanchez GV, Fleming-Dutra KE, Roberts RM, Hicks LA. Core elements of outpatient antibiotic stewardship. MMWR Recomm Rep 2016; 65:1–12. [DOI] [PubMed] [Google Scholar]
- 24. Brehaut JC, Colquhoun HL, Eva KW, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med 2016; 164:435–41. [DOI] [PubMed] [Google Scholar]
- 25. Kitano T, Langford BJ, Brown KA, et al. The association between high and unnecessary antibiotic prescribing: a cohort study using family physician electronic medical records. Clin Infect Dis 2021; 72:e345–51. [DOI] [PubMed] [Google Scholar]
- 26. Livorsi DJ, Linn CM, Alexander B, et al. The value of electronically extracted data for auditing outpatient antimicrobial prescribing. Infect Control Hosp Epidemiol 2018; 39:64–70. [DOI] [PubMed] [Google Scholar]
- 27. Hallsworth M, Chadborn T, Sallis A, et al. Provision of social norm feedback to high prescribers of antibiotics in general practice: a pragmatic national randomised controlled trial. Lancet 2016; 387:1743–52. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Hemkens LG, Saccilotto R, Reyes SL, et al. Personalized prescription feedback using routinely collected data to reduce antibiotic use in primary care: a randomized clinical trial. JAMA Intern Med 2017; 177:176–83. [DOI] [PubMed] [Google Scholar]
- 29. May L, Gudger G, Armstrong P, et al. Multisite exploration of clinical decision making for antibiotic use by emergency medicine providers using quantitative and qualitative methods. Infect Control Hosp Epidemiol 2014; 35:1114–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Goulopoulos A, Rofe O, Kong D, et al. Attitudes and beliefs of Australian emergency department clinicians on antimicrobial stewardship in the emergency department: a qualitative study. Emerg Med Australas 2019; 31:787–96. [DOI] [PubMed] [Google Scholar]
- 31. Morris AM, Calderwood MS, Fridkin SK, et al. Research needs in antibiotic stewardship. Infect Control Hosp Epidemiol 2019; 40:1334–43. [DOI] [PubMed] [Google Scholar]
- 32. Pulia M, Redwood R, May L. Antimicrobial stewardship in the emergency department. Emerg Med Clin North Am 2018; 36:853–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011; 38:65–76. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.