Expert Consensus on Metrics to Assess the Impact of Patient-Level Antimicrobial Stewardship Interventions in Acute-Care Settings

Summary

The Structured Taskforce of Experts Working at Reliable Standards for Stewardship (STEWARDS) panel convened to select metrics relevant for assessing the impact of patient-level antimicrobial stewardship interventions in acute-care settings.

Abstract

Antimicrobial stewardship programs (ASPs) positively impact patient care, but metrics to assess ASP impact are poorly defined. We used a modified Delphi approach to select relevant metrics for assessing patient-level interventions in acute-care settings for the purposes of internal program decision making. An expert panel rated 90 candidate metrics on a 9-point Likert scale for association with 4 criteria: improved antimicrobial prescribing, improved patient care, utility in targeting stewardship efforts, and feasibility in hospitals with electronic health records. Experts further refined, added, or removed metrics during structured teleconferences and re-rated the retained metrics. Six metrics were rated >6 in all criteria: 2 measures of Clostridium difficile incidence, incidence of drug-resistant pathogens, days of therapy over admissions, days of therapy over patient days, and redundant therapy events. Fourteen metrics rated >6 in all criteria except feasibility were identified as targets for future development.

The primary goal of hospital antimicrobial stewardship programs (ASPs) is to improve patient care. Evidence-based strategies involve individualized review of patient-specific clinical data and prescriber-targeted, active interventions to positively impact decisions about antimicrobials (eg, restriction and preauthorization, postprescription audit and feedback) [1, 2]. Metrics to assess the impact of patient-level interventions are poorly defined for hospital ASPs for many reasons. First, the care of patients with suspected infections is complex, involves nuanced decision making, and contains multiple components (eg, whether treatment is indicated, selection of agent[s], dose, duration). Second, patient safety outcomes and resistant infection events are infrequent and may have multiple confounding factors that are either not modifiable or not attributable to the quality of inpatient antimicrobial stewardship. Third, the effort required to extract metrics for ASPs from the electronic health record, complete meaningful analyses, and then translate the analyses into actionable conclusions for program decisions may seem insurmountable. Many potential metrics for hospital ASPs have been proposed, but few have been adequately validated to warrant incorporation into routine program assessments [3, 4]. Furthermore, prior studies that have demonstrated reduced cost and improved processes of care through ASPs are not compelling from a patient care and safety perspective.

We aimed to gain expert consensus on a list of metrics both useful for assessing the impact of patient-level antimicrobial stewardship interventions and feasible to measure in acute-care hospitals with an electronic health record. The goals of this study were not to identify quality metrics to be used for external comparisons or value-based incentives, but rather to identify metrics most pertinent for internal ASP decisions.

METHODS

We performed a modified Delphi, expert consensus-building process to identify metrics useful for tracking the impact of patient-level antimicrobial stewardship interventions. The method differed from the Delphi process developed by the RAND Corporation because it did not include face-to-face meetings [5]. Rather, Web-based teleconferences and electronic surveys enabled the geographically diverse group of experts to participate without logistical barriers. The steps of the process included a comprehensive literature review to develop a candidate metrics list, 2 rounds of electronic surveys for metric rating, data collection, analyses, and feedback to the panel members, and structured, Web-based teleconference discussions between the electronic survey rounds.

Methods for Comprehensive Literature Review and Development of a Preliminary List of Metrics

A set of candidate metrics was compiled from a comprehensive review of published literature on antimicrobial stewardship outcomes and process measurement. First, a PubMed search was conducted using the following search terms for the time period prior to April 2015:

[((“antimicrobial management”) OR (“antibiotic management”) OR (“antimicrobial utilization”) OR (“antibiotic utilization”) (“antimicrobial utilisation”) OR (“antibiotic utilisation”) OR (“antimicrobial stewardship”) OR (“antibiotic stewardship”)) OR (“academic detailing” AND antibiotic OR antibiotics OR ((“Anti-Infective Agents”[Mesh]) OR “Anti-Bacterial Agents”[Mesh])] AND (patient safety OR patient outcome OR patient outcomes OR “Outcome and Process Assessment (Health Care)”[Mesh]).

Second, abstracts were screened by 2 physician and 1 pharmacist investigators (R. W. M., D. J. A., E. D. A.) to apply inclusion and exclusion criteria. Publications met inclusion criteria if they intended to measure the effect of a patient-level stewardship intervention, which was defined as involving (1) a patient-level clinical review (either medical record review or verbal review with a primary provider) and (2) recommendation(s) made to adjust antimicrobial therapy for a specific patient. Publications were limited to inpatient, acute-care ASPs. Exclusion criteria were as follows: (1) the publication was not related to antimicrobial stewardship, which targets adjustment, discontinuation, or optimization of antimicrobial therapy; or (2) the study intervention involved a “guideline” or “education” activity that did not include individual patient-level review and patient-specific intervention. The goal of the literature review was to capture a broad array of possible patient-level ASP metrics. Some publications included proposed metrics based on expert opinion; others directly measured and applied the metric in a study of intervention effect.

The third step of developing the preliminary metric list included review of each publication for extraction of proposed and utilized metrics. Each metric was placed into 1 of 5 metric categories: clinical outcomes, unintended consequences, utilization, process measure, or financial outcomes [6]. Primary references were added to the list for metric extraction as necessary. Duplicate entries were removed. Similar metrics were combined and summarized into a single description.

Assembly of the Expert Panel

The Structured Taskforce of Experts Working at Reliable Standards for Stewardship (STEWARDS) panel was assembled from geographically diverse areas of the United States (Table 1). All 19 invited experts agreed to participate and completed the modified Delphi process from September through December 2015. The panel included adult and pediatric infectious disease physicians and pharmacists with dedicated active practice in antimicrobial stewardship, healthcare epidemiologists, academic researchers, Veterans Affairs representatives, and Centers for Disease Control and Prevention (CDC) stewardship experts.

Table 1.

Structured Taskforce of Experts Working at Reliable Standards for Stewardship (STEWARDS) Panel

Name	Title(s)	Affiliation(s)a	Location
Deverick Anderson, MD, MPH	Adult Infectious Diseases Physician	Duke University Medical Center	Durham, North Carolina
Shawn Binkley, PharmD, BS	Antimicrobial Stewardship Pharmacist	Hospital of the University of Pennsylvania	Philadelphia, Pennsylvania
Michael Calderwood, MD	Adult Infectious Diseases Physician	Brigham and Women’s Hospital	Boston, Massachusetts
Sara E. Cosgrove, MD, MS	Adult Infectious Diseases Physician	Johns Hopkins Medical Institutions	Baltimore, Maryland
Elizabeth Dodds Ashley, PharmD	Antimicrobial Stewardship Liaison Pharmacist	Duke Antimicrobial Stewardship Outreach Network	Durham, North Carolina
Jeffrey Gerber, MD, PhD	Pediatric Infectious Diseases Physician	Children’s Hospital of Philadelphia	Philadelphia, Pennsylvania
Christopher J. Graber, MD	Adult Infectious Diseases Physician	VA Greater Los Angeles	Los Angeles, California
Keith Hamilton, MD	Adult Infectious Diseases Physician	Hospital of the University of Pennsylvania	Philadelphia, Pennsylvania
Adam L. Hersh, MD, PhD	Pediatric Infectious Diseases Physician	University of Utah	Salt Lake City, Utah
Lauri Hicks, DO	Director, Office of Antibiotic Stewardship Adult Infectious Diseases Physician	Centers for Disease Control and Prevention	Atlanta, Georgia
Kevin Hsueh, MD	Adult Infectious Diseases Physician	Washington University School of Medicine	St Louis, Missouri
David W. Kubiak, PharmD	Adult Antimicrobial Stewardship Pharmacist	Brigham and Women’s Hospital	Boston, Massachusetts
Kristi Kuper, PharmD, BCPS	Senior Clinical Manager Adult Infectious Diseases Pharmacist	Vizient, Inc	Houston, Texas
Rebekah Moehring, MD, MPH	Adult Infectious Diseases Physician	Duke University Medical Center Duke Antimicrobial Stewardship Outreach Network Durham VA Medical Center	Durham, North Carolina
Melinda M. Neuhauser, PharmD, MPH	National Pharmacy Benefits Management Clinical Pharmacy Program Manager, Infectious Diseases	Department of Veterans Affairs Pharmacy Benefits Management Services	Hines, Illinois
Christina Sarubbi, PharmD	Antimicrobial Stewardship Pharmacist	Duke University Medical Center	Durham, North Carolina
David Schwartz, MD	Adult Infectious Diseases Physician	John H. Stroger, Jr Hospital of Cook County	Chicago, Illinois
Arjun Srinivasan, MD	Associate Director for Healthcare-Associated Infection Prevention Programs	Centers for Disease Control and Prevention	Atlanta, Georgia
Robert A. Weinstein, MD	Adult Infectious Diseases Physician C. Anderson Hedberg, MD Professor of Medicine	Rush University Medical Center	Chicago, Illinois

^aAffiliation at the time of the STEWARDS panel participation.

Methods for Electronic Survey, Expert Panel Discussions, and Data Analysis

The preliminary list of metrics were compiled into a Web-based, electronic survey via Research Electronic Data Capture (REDCap) software hosted at Duke University [7]. Experts were asked to evaluate each metric using a 9-point Likert scale by rating their agreement in 4 separate criteria based on their expert opinion:

• 1.This metric is associated with improved antimicrobial prescribing.
• 2.This metric is associated with improved patient care.
• 3.This metric is useful in targeting antimicrobial stewardship efforts.
• 4.This metric is feasible to monitor in any hospital with an electronic health record.

Experts were encouraged to (1) submit free text comments on each metric or the group of metrics in each category and (2) add additional metrics that they believed should be considered for inclusion in subsequent rounds. The electronic survey also elicited experts’ suggestions for refinement of wording or description of each metric.

A priori rejection and retention criteria were used to analyze the results from the first electronic survey. Mean and 95% confidence intervals (CIs) were calculated for each metric and criterion. Ratings with a mean upper 95% CI bound <4 were deemed to have consensus to reject; ratings with a lower 95% CI bound >6 were deemed to have consensus to retain. Metrics that met criteria for consensus to reject in 3 or 4 criteria were removed. Metrics that met criteria for consensus to retain in 3 or 4 criteria were carried forward to the discussion and round 2 survey. All other metrics were considered “equivocal” and open for discussion, refinement, or reevaluation. All analyses and summaries of written comments were presented back to panel members by email prior to discussions.

Two Web conferences were held, each with half of the members of the expert panel in attendance. The discussion reviewed results for all metrics from the initial survey, confirmed agreement with retention of metrics by the a priori criteria, and allowed the panel to determine retention or removal of equivocal metrics. Discussions were moderated by a CDC qualitative research specialist (R. L. C.), who assured that every panel member was given opportunity to participate using a standardized script. Verbal consensus from the group was sought for final decisions to remove metrics, refine their description, suggest additional metrics, or retain metrics for rating in the next survey round.

A second electronic survey of the retained metrics was conducted using the same methods and criteria as round 1. The final list of accepted metrics deemed ready for immediate use and tracking was defined based on consensus acceptance in all 4 criteria. A second list of metrics identified for future study was defined based on acceptance in all criteria except the fourth feasibility criterion.

For all statistical analyses, SAS version 9.4 (SAS Institute, Cary, North Carolina) was used. The Duke University Institutional Review Board approved this activity as exempt.

RESULTS

Figure 1 details the process of literature and metric review based on prespecified exclusion criteria. The initial electronic survey included 90 metrics for rating by the panel. Round 1 survey format separated the numerator and denominator metrics in the utilization category (eg, days of therapy numerator was rated separately from the patient days denominator; Supplementary Table 1). All 19 panel members participated in the round 1 electronic survey. Round 1 survey ratings resulted in consensus to retain 14 metrics; the remaining 76 metrics were considered equivocal based on the a priori criteria and no metrics were removed. Eighteen panel members (95%) participated in the Web-based conferences. The discussions resulted in consensus to remove all 18 metrics in the financial outcomes category. This category was generally rated negatively during round 1, and the panel deemed these metrics as not relevant for patient safety (criterion 1). The panel removed an additional 30 metrics deemed to be difficult to interpret, unlikely to be meaningful for ASP decision making, better represented by other metrics under consideration, or too infeasible to capture and interpret. An additional 8 metrics were added for rating in round 2. Eight metrics were refined for the subsequent rating survey including defining utilization metrics as specific numerator/denominator pairings.

Figure 1.

Results of comprehensive literature review to identify candidate patient-level antimicrobial stewardship metrics. A comprehensive literature review included an initial PubMed search, followed by abstract review to apply exclusion criteria to best reflect metrics intended to demonstrate the impact of patient-level stewardship interventions in acute-care hospitals. Each included article underwent in-depth review for extraction of metrics. Primary references were added to metric review as necessary. The metrics list was de-duplicated; similar metrics were grouped together and summarized under a single description within each of the 5 broad categories.

The round 2 electronic survey included 41 metrics for the panel to reevaluate: 5 clinical outcomes, 6 unintended consequences, 10 utilization measures, and 20 process measures (Supplementary Table 1). All 19 panel members participated in the round 2 survey. Round 2 rating resulted in 6 metrics accepted in all 4 criteria and deemed ready for immediate tracking and use by hospital ASPs (Table 2). Fourteen additional metrics were accepted in all criteria except feasibility. These metrics were identified as needing further development in determining standard definitions, method of measurement, and implementation study before active use by ASPs could be recommended. The remaining 21 metrics did not receive expert consensus ratings high enough for acceptance as relevant and feasible metrics for antimicrobial stewardship.

Table 2.

Structured Taskforce of Experts Working at Reliable Standards for Stewardship (STEWARDS) Panel-Recommended Metrics for Assessing the Impact of Patient-Level Antimicrobial Stewardship Interventions

	Group 1: Ready for Immediate Use and Tracking	Group 2: Identified as Useful but Questionable Feasibility: Recommended for Future Study
Clinical outcomes	None	• Readmission: related to infectious diagnoses
Unintended consequences	• Clostridium difficile infection incidence: healthcare facility associated (includes NHSN LabID-defined community-onset, healthcare facility–associated and hospital-onset cases) • Clostridium difficile infection incidence: hospital onset (includes NHSN LabID-defined hospital-onset cases) • Drug-resistant infection: rate of resistant pathogen(s) isolated from clinical cultures (excludes nares and perirectal swabs used for active surveillance).	• Adverse drug events/toxicities
Utilization	• Days of therapy/admission • Days of therapy/patient-days	• Days of therapy/days present • Total duration/admission • Total duration/antimicrobial admission
Process measures	• Redundant therapy events	• Antimicrobial error (wrong drug, dose, route or frequency occurring during ordering or monitoring) • Appropriateness/inappropriateness per institutional guideline/expert opinion • Adherence to guidelines/formulary/protocol/bundle • Appropriate cultures performed per institutional guideline/expert opinion • Excess drug use (antimicrobial use that could have been avoided based on clinical guidelines, shorter recommended duration, stopping therapy due to earlier availability of culture results, etc) • De-escalation performed (number of occurrences) • Culture collected prior to antimicrobial being administered • Time to appropriate therapy • Proportion of patients who received initial antibiotic coverage for a targeted nosocomial pathogen who also had positive clinical cultures (blood, respiratory) for that target pathogen (eg, methicillin-resistant Staphylococcus aureus, Pseudomonas aeruginosa)

Group 1 metrics were accepted in 4 of 4 criteria. Group 2 metrics were accepted in 3 of 4 criteria; only the feasibility criterion was uncertain among expert panel members.

Abbreviation: NHSN, National Healthcare Safety Network.

DISCUSSION

The STEWARDS panel achieved consensus in identifying metrics for acute-care hospital ASPs to assess the impact of patient-level interventions for the purposes of internal program decision making. The panel identified 6 metrics ready for immediate use and tracking: 2 metrics capturing incidence of Clostridium difficile infection (hospital-onset and healthcare facility–associated infections), incidence of drug-resistant infection, 2 measures of antimicrobial utilization (days of therapy in rates per patient admission and per patient-days), and 1 process measure (redundant therapy events). An additional 14 metrics were identified that may prove useful for ASPs in the future, but currently have feasibility barriers that prevent their widespread use.

Prior expert consensus processes that focused on selecting metrics for antimicrobial stewardship have not specifically focused on the impact of patient-level interventions and the goal of informing internal program decision making. In contrast, other panels have attempted to select quality indicators to be used for external comparisons or focused on appropriateness of antibiotic use alone [8, 9]. Morris et al convened a panel of 10 US and Canadian experts to define quality improvement metrics for ASPs, including 2 measures to be used for public reporting [8]. The conclusions of this panel had some similarities to the STEWARDS panel: Both selected incidence of drug-resistant infection, including C. difficile infections, and antimicrobial utilization, specifically, days of therapy. In contrast to Morris et al, the STEWARDS panel did not select clinical outcomes such as 30-day unplanned readmissions or mortality due to drug-resistant organisms. The reluctance to use clinical outcomes as metrics for evaluating ASP impact in routine practice has also been demonstrated in a voluntary survey of physicians, administrators, and pharmacists [10].

The lack of acceptance of clinical outcomes as metrics ready for active use by inpatient ASPs is important. Many clinically important patient outcomes (eg, in-hospital mortality, length of stay, 30-day readmission) are already actively tracked by hospitals for quality improvement and thus do not have feasibility barriers like other proposed metrics. Members of the STEWARDS panel expressed a desire to demonstrate impact on clinical outcomes from ASP interventions. Their reluctance to include these metrics in assessments of patient-level stewardship interventions included concerns with the ability to detect changes in these events and then attribute this change directly to stewardship interventions. Namely, panel members expressed concern about the need for risk adjustment for confounding factors (eg, severity of illness, patient case mix, concurrent infection control activities). Also, clinical outcomes may be insensitive to change as a result of improvements in patient-level stewardship, especially for rare outcomes such as death. Clinical outcomes that may be more responsive to improvements in stewardship included infection-related mortality or readmission related to infectious diagnoses. These metrics, however, were not accepted by the STEWARDS panel in the feasibility criterion due to lack of standardized definitions and the need for more experience in measurement utilizing electronic health records. Furthermore, infection-related events are a subset of total deaths and readmissions, which would make it even more difficult to detect a change. Thus, the need for complicated analyses, large sample size, and therefore limitations in translating these data into actionable conclusions hampers the ability to adopt these metrics into routine surveillance practice for ASPs. Some STEWARDS panel members suggested that clinical outcomes may be more useful to prove “no harm” came from ASP interventions that aim to shorten duration, provide more narrow therapy, or avoid intravenous therapy. Clinical outcomes could be utilized as a complementary metric to reassure providers that interventions did not cause unintended negative clinical consequences. Although the ultimate goal for ASPs is to positively impact clinical and patient safety outcomes, members of STEWARDS acknowledged that perhaps a more practical place for individual ASPs to demonstrate impact is through measures of utilization and process.

Many metrics evaluated by the panel in the utilization and process measure category were rated in the neutral range due to experts’ limited experience with the metrics or the lack of a clear, previously validated, standard definition. Furthermore, several process measures did not reach acceptance in the feasibility criterion due to perceived barriers in capturing the required data elements from electronic health records. For example, de-escalation from broad to narrow antimicrobial therapy is an accepted, basic principle of antimicrobial stewardship that should be responsive to patient-level interventions. This metric was accepted in all criteria except the feasibility criterion due to the state of preliminary work in defining spectrum scores [11] and de-escalation events [12] from electronic data, the need for validation of these definitions in other study populations, and the need for more experience in implementing these metrics into routine practice. As another example, the panel achieved consensus that a days of therapy numerator over dominators of either patient-days or admissions were useful to capture in hospitals with electronic health records; however, several members voiced knowledge that many facilities lack the information technology resources to capture these data. The metric used in the National Healthcare Safety Network Antibiotic Use module includes days of therapy over days present, which several STEWARDS members deemed important given its adoption by the CDC for the US national surveillance system [13]. This metric was rated with uncertain feasibility due to experts’ experiences in the complexity of capturing patient movement data. The traditional denominator metric of patient-days, which is currently used for infection prevention surveillance, considers the count of patients housed on a unit measured at a certain time each day (eg, midnight census) as days at risk [14]. In contrast, days present counts the number of patients housed on a unit for any portion of a calendar day as days at risk [13]. Thus, the days present metric requires detailed information on patient movements throughout the calendar day. This feasibility barrier is slowly being addressed as more electronic health record vendors move toward adding antibiotic use reporting to their products. This and the other metrics that received an uncertain feasibility rating should be evaluated in future studies focused on measurement from electronic data (Table 2, group 2).

This study has limitations. First, the STEWARDS panel consisted of US physicians and pharmacists with infectious disease training, particularly those with antimicrobial stewardship expertise, public health interest, and healthcare epidemiology and antimicrobial stewardship research experience. Thus, the experts’ opinions and self-reported experiences may not reflect those of stewards working in other practice settings and systems. Second, the panel process did not include a face-to-face meeting, but instead involved 2 Web-based teleconferences, each with approximately half of the panel members in attendance due to scheduling limitations. This logistical barrier may have led to a reduction in direct sharing of ideas, but it did not result in failure to meet consensus on the final list of selected metrics. Finally, an important limitation in the output of this study is a continued generality or ambiguity in descriptions of some metrics selected in the final consensus list. For example, the STEWARDS panel did not come to a final recommendation for which measures of incidence of drug-resistant infections should be tracked or how they should be specifically defined and calculated. Based on knowledge of the many possible ways that drug-resistant events can be measured [15, 16], we believe that specific recommendations relevant to ASPs will need dedicated consensus building work in the future. Similar future work in standardized definition development will be required for multiple metrics with feasibility barriers identified during this process (Table 2, group 2).

CONCLUSIONS

The STEWARDS panel developed a list of 6 recommended metrics ready for active use and tracking for acute-care ASPs seeking to assess the impact of patient-level interventions. The selected measures align well with national priorities in improving and measuring antibiotic use and preventing drug resistance [17]. Measurement is a required task in both The Joint Commission antibiotic stewardship accreditation standard and the Centers for Medicare and Medicaid Services proposed antibiotic stewardship condition of participation [18, 19]. The metrics identified by this panel form a core set of measures that ASPs can start using immediately to both meet the measurement requirements and, more importantly, assess the impact of their efforts.

In addition, the panel identified 14 metrics for future study. Future work should focus on standard definition development and overcoming feasibility barriers for metrics that are based on electronic data elements. To this end, The Duke Antimicrobial Stewardship Outreach Network is partnering with CDC and the CDC Foundation to assess the most promising of these additional metrics. Lessons learned from these efforts will help guide the implementation of the next generation of antibiotic stewardship metrics.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the author to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the author, so questions or comments should be addressed to the author.

Notes

Acknowledgments. The authors thank Beverly Murphy, Duke Medical Center Library and Archives Librarian, for her assistance in guiding the literature search strategy;

Elizabeth Hermsen, Merck and Co., Inc., for her input on criteria development; and

Loria Pollack, CDC, for her assistance in study design and criteria development.

Disclaimer. The views expressed in this article are those of the authors and do not necessarily represent the position or policy of the CDC, US Department of Veterans Affairs, or the US government.

Financial support. This work was supported by the CDC Foundation; the Agency for Healthcare Research and Quality (grant number K08 HS023866 to R. W. M.); and the National Institutes of Health (grant number K23 AI095357 to D. J. A.). The source of this information is the Patient Tools for Antibiotic Stewardship Programs, a joint project made possible by a partnership between the CDC Foundation and Merck and Co., Inc.

Multiple STEWARDS panel members were supported by the CDC Epicenters Program grants to Duke University (Durham, North Carolina), Washington University (St Louis, Missouri), Rush University Medical Center (Chicago, Illinois), University of Pennsylvania (Philadelphia, Pennsylvania), and Harvard University (Boston, Massachusetts).

Potential conflicts of interest. Authors certifies no potential conflicts of interest. The authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.