The Vanderbilt EvidenceWeb – Developing Tools to Monitor and Improve Compliance with Evidence-based Order Sets.

Abstract

CPOE systems hold the promise of both reducing medical errors and improving compliance with evidence-based guidelines. Evidence-based order sets have been used to make evidence-based guidelines actionable at the point of care. Investigators have conducted research into factors effecting order set use, but little has been documented regarding order set compliance. Under the direction of the Chief Medical Officer, Vanderbilt University Medical Center undertook a major effort to ensure order sets in the Vanderbilt CPOE system are evidence-based. The Vanderbilt EvidenceWeb was developed in order to monitor use and progress, and to provide feedback to clinical teams based on their requests for data. Monitoring tools and feedback to clinical groups regarding order set use and compliance have proved to be key components in improving delivery of evidence-based care.

Introduction

The landmark IOM report, To Err is Human: Building a Safer Health System¹ raised awareness of the problem of medical errors. In addition, a 2003 RAND study², showed that only 54.9% of patients received recommended care across a broad array of clinical conditions. Computerized provider order-entry (CPOE) systems are seen as a key component in reducing medical errors and improving care for patients. Use of CPOE has been associated with decreased errors in ordering^3–5, improved use of scarce resources⁶, improved ordering rates for preventive therapies⁷, improved odds a patient will receive recommended care⁸, and sustained practice change⁹. CPOE is not a panacea with respect to improving care, however, and has been linked to increases in certain errors¹⁰, and to increased mortality for pediatric inpatients¹¹, highlighting the need for careful implementation, integration with workflow, and a continuous improvement process post implementation.

CPOE systems typically offer several ways of entering orders for patients. Clinicians can enter orders singly or in groups through various means, such as typing in an order, entering orders through an advisor, or selecting orders from an order set. Order sets are collections of orders designed to streamline and standardize the order entry process. Order sets can range from a simple list of orderables, such as a list of commonly ordered laboratory tests, to more complex listings of orders related to a particular condition or procedure (e.g. community acquired pneumonia) and a particular phase of care (e.g., admission orders). Order sets have been used on paper, and also in CPOE systems to make evidence-based guidelines actionable by clinicians at the point of care^12–16, and have been associated with improved outcomes¹³. In essence, these evidence-based order sets are templates for care at a given point in time for a particular condition or procedure. Studies of evidence-based order set usage have focused on characteristics that effect adoption and use of order sets by a clinical community^14–16.

In addition to understanding factors affecting order set use, understanding clinician performance with respect to the particulars of an order set is also a necessary component for continuous feedback and improvement. For example, an evidence-based order set for initial treatment of Acute Coronary Syndrome (ACS) may contain an orderable for aspirin. Individual orders for patients who present with ACS can be compared to the template order set regardless of whether they originated directly from the order set. When looking at order set usage, one may find that the order set is used frequently, but that aspirin is not being ordered as frequently as expected. This information is useful by identifying an area for further investigation to uncover the cause of the lower than expected use of aspirin, and future interventions to improve use if needed. The ability to monitor order set usage both in terms of use, and in terms of orderable item compliance is an important component in managing evidence-based order set use and for providing information to clinical teams to improve care.

Vanderbilt University Medical Center (VUMC) has actively used a CPOE system (WizOrder¹⁷) for more than a decade. Under the direction of the Chief Medical Officer for VUMC a major effort was started to ensure that all of the order sets within the CPOE system contained the most up-to-date evidence that existed. The process to implement this effort involved physicians, case managers, nursing staff, pharmacists, and a support team that consisted of evidence-based support specialists, librarians, and informaticians. As this effort progressed it was apparent that providing usage and compliance information to the clinical teams was essential. The EvidenceWeb was created based on the need for data to help manage order set usage and adoption, as well as the expressed needs of the clinical teams for information.

EvidenceWeb

The Vanderbilt EvidenceWeb integrates order sets, web pages of the evidence that support them, and an order set monitoring tool. It utilizes data from the WizOrder orders database to create reports of order set usage and order set compliance. The WizOrder orders database is relational in design and is “orders centric.” This design proved inefficient for extracting data related to order sets, so information from the orders database was extracted and reconfigured to support efficient queries of order set usage and compliance. The EvidenceWeb database is relational and was built using MySQL. The client front-end is web-based and was programmed using HTML forms and PHP.

The EvidenceWeb was designed in conjunction with clinical teams to include data and functionality that benefits clinical care. The EvidenceWeb has two parts; an “evidence links” section that contains HTML pages that display key supporting evidence for order sets, and an order set tracking tool. The evidence links section allows direct linking from an order set and includes rationale statements and links to key published articles pertinent to the order set. The order set tracking tool incorporates a two step process for determining order set usage and compliance. First, the user identifies one or more order sets to track (see figure 1). Second, the user selects orderables from a list of items that will be tracked with respect to the identified order set(s). A report is then generated in either HTML or Microsoft Excel format (see figure 2). Individual patient encounters are listed in separate rows, and individual orderable items are listed across the top in separate columns. When a patient has an order for a specified orderable, a “YES” displays in the cell corresponding to that patient’s order regardless of whether or not the order originated from the order set. On the HTML version of the report, the “YES” is a clickable link that triggers display of a window showing details of the order (who ordered the item, when it was ordered, where the order originated, etc.).

Figure 1.

The Vanderbilt EvidenceWeb Order Set Search Screen.

Figure 2.

A Sample EvidenceWeb HTML Report.

Preliminary Results

The EvidenceWeb data reports have been particularly well received by clinical teams. Examples of how the EvidenceWeb reports have been used include the following:

• An order set was identified that was in conflict with an evidence-based order set. The competing order set was reviewed and disabled with resulting improvement in compliance with the evidence-based order set.

• Non Emergency Department (ED) Residents (e.g. Medical and Surgical Residents) who rotate through the ED had lower levels of compliance with evidence-based recommendations on ED order sets than expected. Changes to the way Attending Physicians work with rotating residents, along with changes to ED orientation were implemented and order set usage and compliance are being tracked.

• A report of order sets that are not frequently used, or not recently updated was generated and distributed to clinical teams. A systematic review of these order sets was performed by the clinical teams, and 50% of the order sets identified were retired as a part of this process.

• A report of evidence-based order set usage was distributed to clinical leaders to educate staff and residents toward more use of evidence-based order sets.

Other Uses

Development of a set of tools to provide feedback on order set usage and performance can help both developers and clinical teams manage appropriate care for patients. Potentially useful questions that could be asked of such a system are listed in table 1.

Table 1.

Potential Questions and Interventions.

Question	Potential Interventions
Are order sets being utilized as frequently as they should be?	Education that order set exists Add triggers for order set use Improve usability of the order set Integrate order set into clinical workflow Improve “buy-in” and “ownership” of the order set by the clinical team
What is the clinician performance with respect to an order set?	Education on reason for specific evidence-based orderables Add targeted reminders or alerts for particular items Tie order set performance to clinical outcome measures to show value
Are there any common deviations from orderables on an order set?	Investigate whether the deviations are appropriate or inappropriate and adjust accordingly
Are interventions improving use or performance?	Readjust strategy for obtaining appropriate use

Future directions

Future work includes linking order set usage information with clinical outcome measures and other measures of clinical care by integrating EvidenceWeb and other clinical and outcomes related data. Investigation of methods for identifying appropriate cohorts of patients in order to compare treatment and outcomes on patients who had order sets used with patients who did not have order sets used. Groups have used DRG data for this purpose¹⁵, but based on our initial investigation, DRG data may not be sufficiently granular to identify the correct populations for our purposes. The authors view integration of order set usage and compliance data with clinical outcome data as a key facet in a clinical “closed loop,” allowing clinical teams to make adjustments in evidence-based care guidelines based on actual outcomes.

In addition to the feedback items listed in table 1, order set usage and performance data are useful for evaluating the effectiveness of a broad array of order sets, allowing a hospital to specifically target problem areas for improvement. For example, in the ACS scenario above, if aspirin use is low, several interventions could be implemented to improve aspirin administration. Conversely, if aspirin use is high, then designing and deploying a targeted intervention to improve aspirin use can be an ineffective and inefficient use of limited resources. Using an order set monitoring tool can help an organization focus the use of specialized interventions where they are most needed.

The current tool provides a retrospective view of the data, and additional future possibilities include developing functionality to integrate real-time feedback on order set performance into clinical workflow to influence individual patient care as it occurs. Finally, refining the user-interface, and the reports that are available from the EvidenceWeb by creating screens and reports that efficiently and effectively communicate complex information is important to realize the full value of the tool.

Summary

Use of evidence-based order sets presents an opportunity to both reduce medical errors and improve delivery of evidence-based patient care. Monitoring of order set usage as well as order set compliance are key elements in providing proper feedback to manage both improvements to patient care, and development resources by properly targeting interventions where they are needed. Data reports from the Vanderbilt EvidenceWeb have proved useful in identifying target areas for interventions to improve compliance with evidence-based care recommendations. Future work to integrate order set compliance information with clinical outcomes data is a key next step to developing a closed loop for quality improvement.