An Application for Monitoring Order Set Usage in a Commercial Electronic Health Record

Abstract

Organizations that use electronic health records (EHRs) often maintain a considerable amount of clinical content in the form of order sets, documentation templates, and decision support rules. EHR vendors seldom provide analytic tools for customers to maintain such content and monitor its usage. We developed an application for tracking order sets, documentation templates and clinical alerts in a commercial electronic health record. Using the application, we compared trends in order set creation and usage at two academic medical centers over a three-year period. In January 2012, one medical center had 873 order sets available to clinicians; the other had 787. Approximately 50–75 new order sets were added each year at each medical center. We found that 46% of order sets at the first medical center and 39% at the second medical center were unused over the three-year period.

Introduction

Health Information Technology is being rapidly deployed in hospitals, clinics and physician offices throughout the United States as a result of the federal Meaningful Use incentive program^1–4. As a consequence of these guidelines, a furious and sometimes haphazard scramble by EHR vendors and healthcare providers has ensued in order to complete EHR deployments before the Meaningful Use incentives shift to financial penalties in 2015^3,4. Amid the rush to EHR adoption, many healthcare delivery organizations may be neglecting long-term maintenance issues.

EHRs have the potential to reduce medical errors, eliminate workflow bottlenecks and bring about overall improvements to the experience of both clinicians and patients^5,6. To achieve these benefits, EHRs must be actively maintained to assure quality and effectiveness^7,8. When EHRs are not properly configured, they can introduce unintended negative consequences⁹. For example, McCormick et al. found that EHRs were associated with a notable increase in the number of imaging tests and blood tests ordered in one clinical setting¹⁰.

Defining a strategy for active maintenance and quality control of EHRs is challenging. In particular, localized content, such as order sets, documentation templates, and decision support rules can vary drastically across EHR implementations⁷. EHR vendors typically provide base systems that offer the ability to add locally-defined content. The EHR customer is typically responsible for all aspects of that content, whether it is developed and maintained internally or obtained from a third-party vendor. Without processes for maintaining EHR content and monitoring its use, efficiency, usability and cost savings can degrade¹¹. Information management tools are needed to identify functional gaps and assess EHR performance within and across organizations^7,8.

Automated tools for analyzing system usage data provide diagnostic information, compliance checking, usage trend detection, and content evolution quantification^7,8. Such tools may also provide hospital administrators and IT staff with information that can facilitate the identification of system defects or patterns of ineffectiveness, and they can monitor system performance as EHRs evolve^7,8,12,13. Despite the apparent need, there are few tools available that offer these features.

Vanderbilt’s EvidenceWeb is one example of an EHR content management and monitoring tool⁸. It is a Web-based application that was developed to track order set use and progress with the goal of improving clinical compliance with evidence-based care recommendations⁸. EvidenceWeb collects usage data from the Vanderbilt computerized physician order entry (CPOE) system and generates reports about order set activity at Vanderbilt University Medical Center^8,14.

To meet the needs of our institution, we developed a tool for tracking electronic clinical documentation usage trends within our institution’s EHR. Hundreds of documentation templates are used by a variety of clinicians in our academic medical centers; the application we developed made it easier to monitor the use of a continually expanding collection of templates¹⁰.

While both of these applications provide value to individual institutions, they are tethered to specific CPOE systems, limiting their broader use. Sittig, et al. identified the need for interoperable content measurement tools among the grand challenges for building clinical decision support tools that improve care and outcomes within healthcare organizations⁷. Order sets are one type of decision support content that would benefit from plug and play measurement and maintenance tools because they typically vary by site. An order set is a collection of orderable items that are grouped based on clinical purpose. Order sets may be used for diagnosis or treatment of a specific condition, or for guiding events in the care process¹³. Some examples include admission, laboratory testing, and post-operative patient care order sets. Order sets can provide many benefits to clinicians: they can improve order entry efficiency by automating the process of prescribing medications according to protocols, and they can potentially reduce errors by providing templated orderable items^13,15. Additionally, order sets can facilitate compliance with evidence-based care guidelines.

In this paper, we describe the design and use of a second content management application for tracking order sets, documentation templates and clinical alerts at our institution. The application, EHRAnalyzer, was built with interoperability in mind and was tested with our institution’s commercial electronic health record (Sunrise Clinical Manager, Allscripts Corp., Chicago, IL).

Methods

We developed EHRAnalyzer to support the analysis of usage data for order sets, clinical documentation templates, and alerts (Figure 1). The tool was developed in C# with the Microsoft .Net Framework 3.5, and uses a Microsoft SQL Server 2008 database.

Figure 1.

EHRAnalyzer, an application for tracking the use of order sets, clinical documentation templates, and alerts.

Application features include:

1. Asynchronous data collection and caching. Data-intensive features are efficiently loaded with multi-threaded processes and subsequently viewable off-line.

2. Content search. Templates, orders and alerts are searchable based on names, descriptions or relevant dates.

3. Multiple data views. Hierarchal displays and longitudinal usage graphs are available, along with lists that can be sorted according to a number of parameters. Additionally, usage statistics are viewable according to authoring clinician, content type and medical center campus.

4. Built-in data export and printing. Graphs and data are exportable as image or .csv files, enabling users to archive images or analyze data in other applications.

EHRAnalyzer is vendor and institution agnostic. The application includes a tool to extract data from a vendor EHR and store it in a middle-tier database (see Figure 2). The data import tool can be customized to accommodate any EHR that provides access to its database.

Figure 2.

EHRAnalyzer application architecture.

We used EHRAnalyzer to analyze order set usage data from two academic medical centers: NewYork-Presbyterian/Columbia University Medical Center (CUMC) and NewYork-Presbyterian/Weill Cornell Medical Center (WCMC). Data were collected for a 36-month period from January 2009 through December 2011. Both medical centers had been using the EHR for several years prior to the study.

The usage data was loaded into EHRAnalyzer for review. Results were also exported into spreadsheets for additional examination. The order set usage analysis, in particular, examined the rate at which order sets were both “activated” and “retired” from the system, the number of order sets that went unused, the order set usage rate per month, usage trend anomalies, and the most frequently used order sets during the study period. The term “activated” was used to describe the date on which an order set became available in the EHR. The term “retired” was used to describe the date an order set was decommissioned and no longer available for use by clinicians.

Results

Table 1 shows the ten most frequently used order sets during the 36-month period of review for both CUMC and WCMC. Medications and services that were ordered without the use of order sets were not included in this count. The two medical centers have separate, yet similar collections of order sets. The list has three order sets in common: ED Discharge Order Set, Medication Reconciliation Admission and Admit NYP Order set. Figure 3 shows the overall monthly rate of order set use for both campuses.

Table 1.

Most frequently used order sets at two academic medical centers over a 36-month period.

Columbia University Medical Center (CUMC)		Weill Cornell Medical (WCMC)
Order Set Name	Usage	Order Set Name	Usage
1. LAB -- Inpatient Routine Blood Tests	208,410	1. AM Lab Order Set	206,076
2. ED Discharge Order Set	119,326	2. ED Discharge Order Set	68,786
3. Routine Labs Order Set- AM Draw	114,456	3. Admit NYP Order set	42,982
4. AIM Common Orders	82,989	4. Medication Reconciliation – Admission	42,650
5. Amb Pediatric Common Orders	56,398	5. Anesthesia - Adult PACU Order Set	24,230
6. Amb Internal Medicine Common Orders	55,702	6. Urine Lab Order Set	22,988
7. Medication Reconciliation - Admission	52,168	7. Acute Care Profile-Extended Order Set	21,232
8. Routine Labs Order Set	40,856	8. CTS AM Labs Order Set	15,877
9. Admit NYP Order set.	35,174	9. Conditional Discharge Order Set	15,138
10. Amb OB/GYN Prenatal Common Orders	33,093	10. ED Order Set - General	14,838

Figure 3.

Order set usage at Columbia University Medical Center (CUMC) and Weill Cornell Medical (WCMC) over a 36-month period.

In January 2012, one medical center had 873 order sets available to clinicians; the other had 787. Approximately 50–75 new order sets were added each year at each medical center. We found that 46% of order sets at the first medical center and 39% at the second medical center were unused over the three-year period. There were 40 order sets retired at WCMC and 222 retired at CUMC. Figure 4 shows the number of order sets available in the WCMC and CUMC systems. Order sets were implemented at CUMC in 2005 and at WCMC in 2007.

Figure 4.

Order sets available in CPOE systems at two academic medical centers

Figure 5 and Figure 6 are a selection of graphs exported from EHRAnalyzer that demonstrate how usage trends can be identified and analyzed with the aid of the graphical tool. The CPOE system went live in the Emergency Department at CUMC in October 2010. Figure 5 shows usage details for the “Pain/ACS/HTN” order set, which would typically be ordered in the Emergency Department. This type of view is helpful for monitoring uptake when new order sets are introduced into the CPOE system.

Figure 5.

Usage trends for “ED Order Set – Chest Pain/ACS/HTN” that was activated in 2010.

Figure 6.1.

Aggregated usage data for four overlapping lab order sets that were replaced by another order set in August 2011.

Figure 6.2.

Usage data for one redesigned lab order set pre and post redesign in August 2011.

Laboratory order sets, as observed in Table 1, were among the most commonly used at both medical centers. Figure 6.1 shows aggregated usage data from four order sets that were relevant in overlapping care scenarios. A fifth order set was redesigned in order to replace the four duplicative sets in Figure 6.1. Figure 6.2 shows the usage of that replacement order set pre and post redesign. A side-by-side comparison of figures 6.1 and 6.2 shows increased use of the redesigned order set and decreased use of the outdated sets before and after August 2011, when the redesigned set went live.

Discussion

Order sets were commonly used at both academic medical centers. The number of available order sets increased and the frequency of order sets used grew over the three-year period observed. At the same time, however, a substantial number of order sets were never used, which may in part have been because complex order sets were developed to treat rare diseases and few outdated order sets were retired.

At a minimum, a basic order set takes approximately five hours to build. The development of a new order set includes reviewing the request, garnering hospital approval, developing the technology, deploying the content and testing it in the CPOE system. If we estimate each set took a minimum of five hours for CUMC IT to implement, then from 2009 through 2011, at least 2,425 man-hours were spent developing order sets across both NYP Hospital campuses that to date have not been used.

One example that illustrates the importance of optimizing order set content was a recent analysis we performed to redesign the process for ordering routine laboratory tests. We discovered that the layout of items in the existing order set encouraged excessive ordering of Magnesium Level tests. Once the order set was revised, we found that the number of Magnesium Level tests ordered at the medical center decreased by 17%, or 175 per day. While the laboratory costs for performing a magnesium test are small (approximately 56 cents based on the price of the reagent), it may be possible to apply the same optimization principles to order sets that contain items that are costly in terms of labor or materials.

Without tools such as EHRAnalyzer, it is difficult to identify when it is appropriate to create new order sets, retire old ones, and update the orderable items in existing sets. The Joint Commission requires that hospitals review and update order sets according to current evidence-based practice standards on a routine basis¹⁶. Evidence-based order sets can improve clinical quality and efficiency. However, the lack of standard practices for order set implementation and review can result in content that encourages the practice of outdated medicine and over-ordering of diagnostic tests¹¹.

Software tools that aggregate usage data from EHR systems can help clinicians and informatics professionals better manage clinical content. EHRAnalyzer, for example, was successfully used to identify usage trends and areas in need of content improvement at NewYork-Presbyterian Hospital.

This study was performed at two academic medical centers that used the same commercial EHR system. We believe that the problems we identified with management of EHR content are shared by many institutions. In future work, we will expand the EHRAnalyzer application to include more granular usage data about individual orderable items within order sets.

Conclusion

Our findings demonstrate the need for regular review and optimization of order sets and other clinical content in EHRs. Human and financial resources are not effectively utilized when EHR content is developed or purchased and not used in clinical practice. Content that is outdated or poorly designed may pose risks to patients and may adversely affect clinician efficiency and perceived satisfaction. EHR vendors and those who implement their systems should develop processes for managing content. Applications such as EHRAnalyzer may be beneficial in this area.