Deriving Measures of Intensive Care Unit Antimicrobial Use from Computerized Pharmacy Data: Methods, Validation and Overcoming Barriers

Abstract

Objective

To outline methods for deriving and validating intensive care unit (ICU) antimicrobial utilization (AU) measures from computerized data and to describe programming problems that emerged.

Design

Retrospective evaluation of computerized pharmacy and administrative data.

Setting

ICUs from four academic medical centers over 36 months.

Interventions

Investigators separately developed and validated programming code to report AU measures in selected ICUs. Antibacterial and antifungal drugs for systemic administration were categorized and expressed as antimicrobial days (each day that each antimicrobial drug was given to each patient) and patient-days on antimicrobials (each day that any antimicrobial drug was given to each patient). Monthly rates were compiled and analyzed centrally with ICU patient-days as the denominator. Results were validated against data collected from manual medical record review. Frequent discussion among investigators aided identification and correction of programming problems.

Results

AU data were successfully programmed though a reiterative process of computer code revision. After identifying and resolving major programming errors, comparison of computerized patient-level data with data collected by manual medical record review revealed discrepancies in antimicrobial days and patient-days on antimicrobials ranging from <1% to 17.7%. The hospital for which numerator data were derived from electronic medication administration records had the least discrepant results.

Conclusions

Computerized AU measures can be derived feasibly, but threats to validity must be sought and corrected. The magnitude of discrepancies between computerized AU data and a gold standard based on manual chart review varies, with electronic medication administration records providing maximal accuracy.

Antimicrobial resistance rendering previously treatable infections unresponsive to most drugs is a significant and growing public health concern.^1,2 This threat has been recognized in the most recent national action plan for the prevention of healthcare-associated infections outlined by the Department of Health and Human Services,³ and calls for a coordinated national effort to monitor resistance and implement prevention and control efforts have been longstanding.^1,4 The Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America recently published guidelines promoting the implementation of antimicrobial stewardship interventions in hospitals.⁵ An integral component of evaluating the impact of any of these strategies is the accurate and continuous measurement of antimicrobial utilization over time.

The increasingly computerized processes of healthcare delivery have made the automated acquisition of antimicrobial utilization data possible. Indeed, most modern hospitals have universal computerization of laboratory, pharmacy, admission-discharge-transfer (ADT), and patient demographic and financial data.

However, just as these data sources have been incorporated into fully functional electronic health records for only a small minority of hospitals,⁶ the derivation of reliable and accurate reports based on computerized hospital data has generally been difficult to achieve.⁷ Because these data are stored at each hospital or health system in disparate information systems, procedures required for their collection, extraction, cleaning, validation, and computation are often complex and error-prone.⁸ With regard to antimicrobial use measures, specifically, published data that have used electronic data sources to derive these estimates have almost exclusively relied on propriety measurement systems and the methods underlying the acquisition, validation, and consolidation of such data have not been well described.

We describe our efforts to demonstrate the feasibility and validity of obtaining uniform measures of antimicrobial utilization in selected intensive care units (ICUs) in four academic medical centers by accessing pharmacy and administrative data contained in computerized data warehouses. Despite considerable expertise and relevant experience by healthcare informatics specialists and investigators in accessing such data, we encountered a number of problems that were largely unforeseen and, therefore, may be informative to the development of a standardized approach to deriving antimicrobial measures from electronic data.

Methods

Four tertiary-care, academic medical centers were recruited from institutions participating in the current Prevention Epicenter Program,⁹ funded by the Centers for Disease Control and Prevention (CDC), to participate in this study. The Institutional Review Boards of each hospital approved the study protocols, and waived requirements for patient and physician consent.

Hospital Intensive Care Unit Types and Information Technology Resources

The characteristics of the participating facilities, respective data warehouses, and medication ordering, dispensing, and administration systems are outlined in Table 1. Descriptions of these data warehouses have been provided elsewhere.^8,10–12 Antimicrobial use data for selected ICUs in each hospital over a 36-month time period (July 2004 – June 2007) were acquired, analyzed, and validated separately at each institution before being sent to one central Epicenter [Chicago Epicenter (DNS)] for collation and final analyses.

Table 1.

Intensive Care Units and Information Technology Resources of Participating Hospitals

Hospital Identifier	ICUs	Medication Ordering System (Physician Orders)	Medication Dispensing System (Pharmacy Order Entry)	Medication Administration System	Computerized Data Warehouse (CDW)	Programming Language	Numerator Data Source(s)	Denominator Data Source(s)
Hospital Aa	MICU, SICU, CCU	Paper, CPOE	SMS, Cerner	Paper, eMAR	CARP	SQL	Pharmacy dispensing, Physician orders	Pharmacy dispensing, ADT
Hospital B	MICU, CCU, STRI, RSCU, TICU	Paper	HELP Pharmacy	eMAR	HELP 1 and 2 Enterprise	Oracle SQL, C++	Medication administration	Medication administration
Hospital C	MICU, SICU, CCU	Paper	PDM Medication Control System	eMAR	Washington University and BJC Healthcare	Sybase SQL	Pharmacy dispensing	Pharmacy dispensing
					Medical Informatics Database
Hospital D	MICU, SICU, BMT	CPOE	Centricity, Siemens, Essentris OSUMC IW	Paper	OSUMC Information Warehouse (IW)	Oracle SQL	Physician orders	Physician orders

Note. ADT, admission-discharge-transfer; BMT, bone marrow transplant; CCU, coronary care unit; CPOE, computerized provider order entry; eMAR, electronic medication administration record; ICU, intensive care unit; MICU, medical ICU; RSCU, respiratory special care unit; SICU, surgical ICU; SQL, Structured Query Language; STRI, shock-trauma-respiratory intensive care unit; TICU, thoracic ICU.

^aA change in the pharmacy system vendor during the study time period necessitated use of two data sources.

Antimicrobial use numerators (described below) were computed from varying electronic data sources (Table 1). For Hospital A, whose pharmacy information system vendor changed during the study period, the numerator data source changed from pharmacy dispensing to computerized physician order entry (CPOE) data. Hospitals B, C and D obtained numerator data from the electronic medication administration record (eMAR), from pharmacy dispensing data and from CPOE data, respectively. Although eMARs were used to document medication administration in three of the four hospitals, eMAR data were available in a format amenable to analysis only at hospital B, which was able to distinguish antimicrobial doses that were administered from those that were ordered but not administered for any reason (e.g., because of doses held or refused). Denominator data (ICU patient-days) were derived from the same data sources as those from which numerator data were computed in all hospitals, with the exception of Hospital A where ADT data sources had to be queried separately after the change in pharmacy information system vendors.

Antimicrobial Use Measures

The primary numerator measure for antimicrobial use was antimicrobial days, defined as calendar days on which patients received each antibacterial or antifungal agent given by intravenous (IV) or oral administration. For example, one patient given two drugs for 5 days accrued 10 antimicrobial days. Drugs available in injectable, oral, or other systemically-administered form were counted just once per day irrespective of route of administration. Secondary numerator measures of antimicrobial use included patient-days on antimicrobials, antimicrobial starts, antimicrobial courses, and defined daily doses (Table 2). Our “antimicrobial days” have the same meaning as the “days of therapy” reported by Polk and colleagues¹³; however, we’ve avoided this latter term because of its resemblance to “patient-days on antimicrobials.” The programming logic used to compute numerator events from the different data sources is summarized in Table 3. Antimicrobial use measures were summed in each ICU for each calendar month over the study period and for each of the antimicrobial agents and predefined drug classes (Table 4).

Table 2.

Numerators Used in Antimicrobial Utilization Measures and Their Definitions

Measure	Definition	Example
Antimicrobial days	Sum of the calendar days on which each antimicrobial was administered	2 drugs given for 5 days followed by a different drug given for 5 days to one patient = 15 antimicrobial days
Patient-days on antimicrobials	Sum of the calendar days on which one or more antimicrobial drugs was administered	2 drugs given for 5 days followed by a different patient-days on antimicrobials
Antimicrobial starts	Sum of the calendar days on which each new antimicrobial drug was started, following 3 or more days without exposure to that drug	2 drugs given for 5 days followed by a different drug given for 5 days to one patient = 3 antimicrobial starts
Antimicrobial courses	Sum of the calendar days on which any antimicrobial drug was started, following 3 or more days without exposure to any antimicrobial drug	2 drugs given for 5 days followed by a different drug given for 5 days to one patient = 1 antimicrobial course
Defined daily doses (DDDs)	World Health Organization-standardized conversion of aggregate drug dosing data into number of doses26	200 grams of vancomycin dispensed divided by 2 grams per vancomycin DDD = 100 DDDs of vancomycin

Table 3.

Logic Used in Computing Numerators Used in Antimicrobial Utilization Measures from Different Computerized Data Sources

Data Source	Events Measureda	Logic Applied
Pharmacy dispensing	Antimicrobial doses dispensed from pharmacy	One or more doses of each antimicrobial dispensed during an ICU day constitutes an antimicrobial day; one or more doses of any antimicrobial dispensed during an ICU day constitutes a patient-day on antimicrobials.
Physician orders (CPOE)	Antimicrobial start and stop orders; ICU admission and discharge days	ICU days on which each antimicrobial is ordered for continuous scheduled administration and subsequent ICU days are counted as antimicrobial days until either that drug’s iscontinuation d is ordered or until ICU discharge. ICU days on which any antimicrobial is ordered and subsequent ICU days are counted as patient-days on antimicrobials until either the discontinuation of all antimicrobials has been ordered or until ICU discharge.
Medication administration (eMAR	Antimicrobial doses administered by nurse	One or more doses of each antimicrobial administered during an ICU day constitutes an antimicrobial day; one or more doses of any antimicrobial administered during an ICU day constitutes a patient-day on antimicrobials.

ICU day constitutes a patient-day on antimicrobials. Note. CPOE, computerized provider order entry; eMAR, electronic medication administration record.

^aNumerator events are counted only through the calendar day before ICU discharge.

Table 4.

Antimicrobial Classification System

Drug Class	Associated Antimicrobial Agents
Anti-Pseudomonals	piperacillin-tazobactam, imipenem, meropenem, ceftazidime, cefipime, aztreonam, levofloxacin, ciprofloxacin, gentamicin, tobramycin, amikacin
Anti-MRSA drug	vancomycin (parenteral only), linezolid, daptomycin, quinopristin-dalfopristin
Anti-MSSA drugs	oxacillin, nafcillin, dicloxacillin, clindamycin
Other beta-lactam drugs	cefazolin, cephalexin, cefoxitin, ceftriaxone, penicillin, ampicillin, ampicillin-sulbactam, amoxicillin, amoxicillin-clavulanate
Anti-Clostridium difficile drugs	metronidazole (oral only), vancomycin (oral only)
Macrolides	azithromycin, clarithromycin, erythromycin
Tetracyclines	doxycycline, minocycline, tetracycline
Other antibacterials	metronidazole (parenteral only), moxifloxacin, trimethoprim, trimethoprim-sulfamethoxazole
Antifungals	amphotericin B deoxycholate, liposomal amphotericin B, fluconazole, itraconazole, voriconazole, caspofungin, anidulafungin

Antimicrobial use rates were calculated using ICU patient-days as the denominator. An ICU patient-day was attributed to each patient occupying an ICU bed at midnight of each day, as previously recommended¹⁴, so that events occurring on ICU admission days were counted while those occurring on the day of ICU discharge were not.

Data validation

The investigators participated in regular teleconference calls to discuss problems with programming and data collection, providing a forum for shared learning as problems encountered at one institution were evaluated in the context of experience at the others. Inspection of data derived from draft program code for face validity sometimes identified the presence of programming errors before more detailed validation efforts were begun, prompting detailed examination of programming code to pinpoint and correct programming flaws, and subsequent validation studies to confirm data validity.

Each hospital used two methods to systematically validate their data. First, to measure how accurately medication administration records reflected actual medication administration to patients, convenience samples of at least 100 intravenous antimicrobial doses scheduled for administration in study ICUs were prospectively audited and observed at the bedside for timeliness of administration using the method of Itokazu, et al.¹⁵ The results (timely dose administration or not) were compared with the disposition of the dose as registered in the medication administration records. Second, retrospective validation studies were conducted by assembling cohorts of randomly selected antimicrobial recipients from each of the ICUs during the study period and comparing counts of numerator events compiled by applying draft computer queries to these cohorts with manual review of the same patients’ medication administration records, our gold standard. After identifying and correcting programming errors, numerator counts derived via revised program code and manual chart review were applied to new cohorts of antimicrobial recipients and results compared until no new programming errors could be identified.

Results

Investigators at the four participating institutions were able to generate antimicrobial utilization data for each of the selected ICUs over the first 24 months of the 36-month study period and reported preliminary intra- and inter-ICU antimicrobial use rate comparisons.¹⁶ However, preliminary retrospective and prospective validation studies revealed major discrepancies between numerator counts that substantially biased the preliminary results and prompted careful review of the code used in the computer queries for systematic error by programmers and investigators. A summary of the programming errors is provided in Table 5. These errors were detected after inspection of the resulting reports suggested a lack of face validity or after detection of inconsistencies between computerized reports of patient-level data and charted medication administration record entries during retrospective validation.

Table 5.

Errors Encountered During Validation of Computed Antimicrobial Utilization Rates from Computerized Data Sources

Data Source	Error	Cause	Solution
Numerator data
1. Physician orders (CPOE)	Spuriously high cefazolin utilization rates in an ICU in Hospital A after change to new pharmacy computer system.	Unknown to study personnel, an automated testing procedure added 233 cefazolin days over 7 months to non-existent patients.	Test entries were removed, with revised cefazolin utilization rates more comparable to rates ascertained from data from older pharmacy system.
2. Physician orders (CPOE)	Spurious increase in antimicrobial utilization rates in Hospital A after change to new pharmacy computer system.	Errant programming of ADT data in new system led to inappropriate attribution to the ICU stay of antimicrobial use that occurred following transfer from the ICU.	Programming was revised to limit numerator events to ICU patient- days as defined by the ADT tables and new results were validated.
3. Pharmacy dispensing	Patient-days on antimicrobials were calculated incorrectly, with rates exceeding the maximum 1,000 patient-days on antimicrobials per 1,000 ICU patient-days in Hospital C.	The programmer misunderstood the definition of patient-days on antimicrobials.	Rates were corrected and validated after the definition of patient-days on antimicrobials was clarified.
Denominator data
1. Medication administration (eMAR)	Spuriously high antimicrobial utilization rates in an ICU in Hospital B.	Communication error led to programming of ICU antimicrobial recipients only, rather than all ICU patients, in computing denominator data.	Rates were reprogrammed to include all ICU patients and new results were validated.
2. Pharmacy dispensing	Spuriously low antimicrobial utilization rates in one ICU in Hospital C.	Summed denominator days from two parallel nursing units comprising an ICU were used to calculate antimicrobial use rates for each nursing unit instead of calculating denominator ICU days from each nursing unit separately.	Rates were re-computed and validated after the programmer was made familiar with physical layout of the ICU.

Note. CPOE, computerized provider order entry; eMAR, electronic medication administration record.

After identifying and correcting all identifiable programming errors, numerator counts derived via revised program code and manual chart review were applied to new cohorts of antimicrobial recipients and results compared until no new programming errors could be identified. The discrepancies between computer-derived and manual counts of antimicrobial days and patient-days on antimicrobials presented in Table 6 reflect these final comparisons. The retrospective validation studies revealed variable levels of discrepancy by institution between numerator counts derived from application of final computer queries and manual review of medication administration records. Overestimation of counts of antimicrobial days and patient-days on antimicrobials generated by computer code were < 1% at hospital B, where antimicrobial utilization was computed from eMAR data. By contrast, programming of computerized pharmacy dispensing data at hospital C, where delayed delivery of paper medication orders from ICUs to pharmacy may have led to antimicrobial dispensing after drug discontinuation or patient discharge orders were written,¹⁵ counted 17.7% more and 14.5% more antimicrobial days and patient-days on antimicrobials, respectively, than manual record review did. Use of CPOE data to derive numerator antimicrobial measures at hospitals A and D generated intermediate levels of discrepancy (Table 6).

Table 6.

Retrospective Comparisons of Computerized Antimicrobial Numerator Data with Chart Review for Randomly Selected ICU Antimicrobial Recipients

Hospital Identifier	Numerator Data Source	Number of patients	Antimicrobial Days		Percent computer over- estimation	Patient-days on Antimicrobials		percent computer over- estimation
			Computer	Chart Review		Computer	Chart Review
Hospital Aa	Pharmacy dispensing	100	661	613	7.8	385	408	−5.6
	Physician orders (CPOE)	90	609	570	6.8	353	345	2.3
Hospital B	Medication administration (eMAR)	100	4551	4509	< 1%	2015	2018	< 1%
Hospital C	Pharmacy dispensing	100	1792	1523	17.7	953	832	14.5
Hospital D	Physician orders (CPOE)	100	1198	1082	10.7	595	563	5.7

Note. CPOE, computerized provider order entry; eMAR, electronic medication administration record.

^aA change in the pharmacy system vendor during the study time period necessitated use of two data sources.

Prospective bedside observations of the intravenous administration of routinely scheduled antimicrobial doses revealed > 95% concordance between the observed outcomes of dose administration events and the corresponding dose administration status recorded in the MARs in all study ICUs (data not shown).

Discussion

Our findings show that derivation of standardized patient-level measures of antimicrobial utilization from a sample of hospitals with disparate computerized pharmacy systems is feasible. However, our experience highlights a few important issues related to the use of computerized data sources to derive and report hospital antimicrobial utilization rates.

First, inter-institutional differences in pharmacy computer systems and available data sources (Table 1) necessitated the use of institution-specific computing strategies (Table 3), contributing to varying levels of fidelity between antimicrobial utilization results obtained by application of computer code versus manual review of patient records (Table 6). Until greater uniformity of hospital data systems is achieved or until valid antimicrobial measurement programs are included within commercial and governmental hospital pharmacy computer systems, institution-specific strategies for data programming, validation and interpretation will have to be developed to ensure that accurate and comparable measures of antimicrobial utilization data are derived and reported across hospitals.

Second, we found that programming of antimicrobial utilization measures based on computerized pharmacy and administrative data was complex and error-prone. Despite considerable experience in querying and analyzing computerized data from our respective institutions,^8,10–12 we made important errors in our initial attempts at deriving these antimicrobial utilization measures (Table 5). Our need to adopt separate, institution-specific computing strategies (Tables 1 and 3) and the complexities of computerized medical records, in general,¹⁷ and of pharmacy data, in particular,⁸ likely contributed to these problems. However, many of these mistakes stemmed from conceptual misunderstandings and inadequacies in communication between clinician- investigators and informaticists whose mitigation requires careful application of basic tenets of multidisciplinary collaboration and data review and validation (Table 7). Complementary methods – assessment of face validity, review of procedures for developing computer code and retrospective validation procedures – were necessary to fully identify and correct these errors, highlighting the importance of adopting a careful, systematic approach to collecting and validating data from electronic health records.

Table 7.

Potential Sources of Error in Computing Antimicrobial Use Rates from Computerized Pharmacy Data and Suggestions for Avoiding Them

Potential Sources of Error	Suggested Solutions
Data request/acquisition
Programmer’s understanding of clinical concepts and goals underlying investigator’s request for data may be incomplete.	Written request for data must clearly outline clinical concepts and goals. Programmer should have frequent access to investigator to clarify conceptual issues
Programmer’s understanding of the physical context and clinical processes and procedures relevant to the request for data may be incomplete.	The programmer should tour site(s) of care being studied. The programmer should be oriented to the processes of care relevant to the request for data (e.g., hospital processes of medication ordering, distribution and administration) emphasizing the point(s) at which the computerized data being studied are generated.
Investigator’s understanding of data structure and programming procedures may be limited.	Programmer should describe proposed approach to request for data, including anticipated shortcomings in data structure or availability, to investigator before writing program code. Programmer should review program code with investigator before executing programming procedures.
Data analysis
Computerized data may require careful review for outlier entries and may require re-formatting (e.g., converting free text to categorical tabular entries) before programming is possible.	Programmer should involve investigator in review of data tables and obtain investigator’s guidance in interpreting outlier entries. Magnitude of data cleaning and conversion efforts must be estimated and the necessary resources - primarily personnel time - allocated. Investigator must assist in designing data restructuring plan and provide necessary nomenclature and definitions.
Data validation
Investigator’s predisposition to trust integrity of programming processes and data derived thereby (“if it’s on a computer screen, it must be right”) may be misguided.	Reports based on queries of computerized data must be carefully reviewed for face validity: ➢ Are all expected data elements (e.g., antimicrobial names) represented? ➢ Are results comparable to previously validated data, if available? ➢ Do the results reflect anticipated variation? Reports based on queries of computerized data require validation, ideally via manual comparison of samples of computerized data with an acceptable gold standard. Systematic sources of error must be vigorously sought to explain recurring or substantial discrepancies in the results of these comparisons.

Third, after maximally validating program code in our respective institutions, we measured variable levels of over-estimation of computed numerator counts compared to a gold standard based on retrospective medical record review (Table 6). These discrepancies likely reflect inter-institutional variation in efficiency and coordination of medication ordering, distribution and administration procedures.^15,18 In particular, in hospital C which had the highest level of discrepancies, delays in transport of paper medication orders to a centralized pharmacy may have contributed to delayed pharmacy response to ordered changes in patient antimicrobial regimens, leading to pharmacy dispensing of antimicrobial doses to the ICU that were not administered and therefore omitted from the MAR (Tables 1, 3 and 5). By contrast, use of bedside charting of eMAR data for both computer-derived and manually collected numerator data in hospital B doubtlessly accounted for the high affinity between data obtained through these different sources. Our results suggest that eMAR is the most accurate source for pharmacy numerator data in hospitals where it is in use; however, this finding requires confirmation from other institutions. Measures based on pharmacy dispensing or physician orders, being further removed from the antimicrobial administration event, are more likely to overestimate actual utilization.

The optimal metric for reporting hospital antimicrobial utilization is unclear.^19–23 We chose antimicrobial days and patient-days on antimicrobials (Table 2) as primary numerator measures because they provide complementary information on the intensity and breadth of ICU antimicrobial use,²³ they are minimally affected by variation in antimicrobial dosing regimens and they should be readily extractable, given the current widespread availability of detailed, patient-specific computerized pharmacy data within U.S. hospitals. Previous studies have used “days of therapy,”¹³ analogous to our antimicrobial days, to rank and assess secular trends in antimicrobial utilization in an alliance of 22 U.S. academic health centers and in 130 U.S. hospitals.^24,25 However, those analyses were based on charge and billing data rather than the pharmacy dispensing, physician order or eMAR data that we employed. Also, the specific methods – proprietary in one instance – were not detailed and validation efforts were limited.^24,25

Older U.S. and European antimicrobial utilization surveys, performed where detailed, patient-specific data may not have been available, have used conversion factors such as the defined daily dose (DDD) or recommended daily dose ^26,27 to estimate patient-level antimicrobial use from aggregate antimicrobial use and census data to make inter-institutional or international comparisons,^27–29 analyze secular trends,^29,30 and correlate antimicrobial use with antimicrobial resistance.^31–33 However, pharmacy reports of aggregate antimicrobial utilization may substantially overestimate actual use in hospitals with poorly coordinated mechanisms of medication ordering, distribution and administration.¹⁵ Moreover, DDDs underestimate patient-level exposure to drugs requiring renal dose adjustment,³⁴ overestimate the use of antimicrobials for which the conversion factor is lower than doses that are commonly prescribed,^13,35,36 and do not apply to most pediatric patients.¹³

Adaptation of programming approaches to the disparate information systems and data sources available in each institution will be a formidable challenge that will require understanding and avoiding potential errors that can impede the valid and efficient measurement of antimicrobial use. The CDC is currently revising the data submission requirements for the Antimicrobial Utilization component of the National Healthcare Safety Network (NHSN) to receive standardized summary measures from eMAR systems directly. This may represent the most effective approach to achieving accurate centralized collection, analysis and reporting of antimicrobial utilization measures from multiple institutions.³⁷ Application of methods similar to ours by these and other surveillance efforts will be instrumental in achieving the widespread availability of valid and efficient measurements of antimicrobial utilization.