Details of a Successful Clinical Decision Support System

Abstract

Computerized physician order entry (CPOE) with clinical decision support (CDS) is regarded as one of the most effective ways to improve the quality of health care and increase patient safety. As electronic medical records become more available, such systems will increasingly become the method of choice to achieve these goals. Creating a CPOE/CDS system is a complex task, and some fail despite time consuming and expensive development. The CPOE system at the Regenstrief Institute incorporates sophisticated CDS and is one of the oldest and most successful in the U.S. Many years in development, it is currently used by hundreds of providers. Our well established, successful system can serve as a template or model for the future development of similar systems. We recently completed a full analysis of our CPOE/CDS system and present details of its structure, functionality and contents.

Introduction

Multiple studies have shown that health care delivered in the United States is suboptimal. In 1999, the Institute of Medicine estimated that up to 98,000 patients die each year due to preventable medical errors.¹ Other studies have shown there is a major discrepancy between clinical care actually delivered and optimal patient care. In one study, roughly 30% of patients did not receive recommended acute care and approximately 40% did not receive recommended care for chronic conditions.²

The Institute of Medicine identified computerized physician order entry (CPOE) as an important component in improving clinical practice and reducing errors.¹ While clinical decision support (CDS) systems not associated with CPOE have been successful and shown to improve patient care,³ CDS coupled with CPOE has been shown to be even more effective.⁴ CPOE/CDS systems have been shown to significantly reduce medication errors and reduce rates of adverse drug events,⁵ and reduce nosocomial infection rates.⁶ In addition, CPOE/CDS implementation has been shown to result in improvement in prescribing practices and better adherence to recommended care standards.⁷

However, studies show that nearly 32% of CDS system implementations either fail to achieve provider acceptance and/or fail to improve patient care.⁸ In a recent analysis of 70 randomized controlled trials of CDS systems, researchers identified four components common to successful CDS systems: 1) Decision support provided automatically as part of provider workflow 2) CDS delivered at the time and location of decision making 3) actionable recommendations provided 4) is computer based.⁸ The CDS system developed at the Regenstrief Institute in Indianapolis, Indiana incorporates all of the above features. It is well accepted by providers and has been documented to improve patient safety and the quality of health care delivered.^{9, 10} We describe the general structure of this successful CDS system, and provide details regarding its rules, logics, and use of patient data elements. Future designers of such systems may wish to make use of the elements of our successful system as they begin their own development.

Background

The Regenstrief Medical Record System (RMRS) is one of the oldest electronic medical record systems in the U.S. and has been described elsewhere.¹¹ Briefly, the RMRS serves Wishard Hospital (a 264-bed primary care Indianapolis hospital) and its 40 local and 30 outreach clinics. It also serves as the clinical repository for Clarian Health Partners, a 1300 bed hospital system. The Medical Gopher, a CPOE system, was incorporated into the system in 1982. The Medical Gopher CPOE system is now used by providers for all inpatient and emergency department orders at Wishard Hospital. A large long term care facility and most of the outpatient clinics also enter all of their orders directly into the computer using the Medical Gopher. We used the CARE language¹² to implement CDS in 1974. The reminders displayed by the system at this initial stage were generally printed in advance of patient visits. In 1992, we re-designed the CARE language to provide decision support to the provider in real time, i.e. while they are entering data, such as diagnostic testing, problems, or treatments. We called this revised version of the CARE language G-CARE.¹² We still use this language to create all of our CDS rules. We follow a standard convention when naming rules which allows for increased human readability and understanding of each rules’ function.

Structure of our CDS system

Our CDS system consists of a series of interrelated but distinct rules written in the G-CARE language. There are seven types of rules as shown in Table 1. The majority of rules are of the selection type.

Table 1.

G-Care rule types and descriptions

Rule Type	Description
Algebraic	Rule evaluates a single expression and returns a numeric result
Logical	Rule evaluates an expression and returns a TRUE of FALSE result
Prompt	Displays a simple form to request data from the user
Special	Retrieves data from the Gopher CPOE database
Selection	Retrieves data from the patients electronic medical record (EMR)
Reminder	Returns one of a set of possible messages and/or suggested orders
Reminder.set	Returns one to many sets of possible messages and/or suggested orders
Reminder.set.text	Returns a message describing the logic or rationale of a reminder rule

All rules, regardless of type, contain the same structure. Information common to all rules includes rule name, type, author, date of rule creation/modification, rule duration, and the names of other rules which use this rule (rule ancestors) as well as the names of other rules this rule uses (rule daughters). Most rules also contain one or more logic sections, which are expressions that are evaluated to output either a specific value, a date, or a Boolean (True/False) value. The results of this expression are used by other rules that contain this rule in their logic. Most rules also contain rule exclusion criteria which if true, prevent the rules’ logic from evaluating. A rules’ duration or ‘life span’ determines when the rule should be re-evaluated. For example, the duration for the rule which determines a patient’s race is forever, while the rule which calculates the last potassium level is one day. Each reminder rule display message also can be assigned a guideline text pointer which is an index to full text clinical guidelines. The rules in our CDS system are arranged in a complex hierarchical structure. Highest level reminder rules will only evaluate to true if intermediate rules lower in the hierarchy are also true, which in turn will only evaluate to true if the lowest level ‘building block’ rules are also true

In our system, reminder rules can be triggered in four different ways: 1) the passage of time (such as flu shot reminders), 2) the entry of a patient problem or diagnosis (a diagnosis of deep vein thrombosis may trigger a reminder for heparin) 3) the entry of a diagnostic test (an order for a sigmoidoscopy may trigger a reminder that colonoscopy is the preferred test) 4) the entry of a treatment (an order for gentamicin triggers a reminder for gentamicin levels). Some reminder rules use the results of other reminder rules in their logic.

Most reminder rules contain multiple logic sections and are capable of displaying a unique message for each logic section that fires. Execution of a reminder rule proceeds through each logic section until the first one fires and its corresponding message displayed. Therefore, logic section precedence is easily adjusted by changing the order within the reminder rule itself. Our system currently does not contain methods (such as Bayesian networks) to manage uncertainty.

Reminder rules display one of three types of messages or alerts: 1) an informational message with no specific pre-written orders 2) a message with a recommendation and a pre-written order where the default action of a click or single keystroke is ‘omit’ 3) a message with a recommendation and a pre-written order where the default action of a click or single keystroke is ‘order’. For certain recommendations, we assigned the default user choice to ‘order’ because studies show that users are more likely to accept recommendations when they do not have to perform an additional action.¹⁰ These are rules where there is a high degree of certainty that the recommendation should be executed, and the recommendation is relatively non-invasive. Examples of such rules include ordering aspirin in patients with evidence of acute myocardial infarction and ordering an EKG for patients with none on record who have a history of cardiac disease.

We discovered early in the CDS rule development process that execution time for very complex rules could be prohibitively slow due to the large number of data elements involved (see Figure 1). Rules relating to treatment guidelines for conditions such as congestive heart failure and pneumonia can take up to 20 seconds to process- an unacceptable disruption to the workflow of the busy clinician. Therefore, using registration data, we execute certain complex rules the night before a patient visit and the results are stored in memory for fast retrieval.

Figure 1.

Hierarchical structure of a reminder rule to recommend an increase in diuretic therapy

Our rules were created by a total of 14 developers- four physicians serve as knowledge engineers who research and create the rules, and ten programmers who actually implement them. As part of our system maintenance program, we encourage user feedback of our CPOE/CDS system with biweekly pizza lunch meetings. We also perform an overall review of the entire system on a semiannual basis.

Details of our CDS system

We will now discuss in greater detail the contents of our CDS system. Our current production system includes 1,486 rules. Of these, 180 rules are used either within the actual CPOE (Gopher) code itself or embedded within a Regenstrief dictionary term, to simply display patient data to the provider when ordering specific items. For example, we have a rule which displays the last hemoglobin level when a provider is ordering a complete blood count. Since these types of rules function independently of other rules and do not display an actual reminder, we excluded these from our analysis (although they still are part of decision support). Using these criteria, we have a total of 1,306 rules in our CDS system. The distribution of rules by rule type is shown in Figure 2. Only 31% of all CDS rules are actual reminder rules.

Figure 2.

Distribution of rules by rule type

There are 360 basic ‘building block’ rules. We define building block rules as rules which do not use any other rules in their logic. All other rules in our system must use the results of one or more of these rules. The 360 building block rules consist of 277 selection rules, 78 special rules, and 5 algebraic rules. Approximately 70% of these rules retrieve laboratory or radiology result information (i.e. value of last potassium, date of last mammogram), 25% retrieve specific patient findings or diagnoses (i.e. last weight, active orders, problem lists). The other 5% retrieve various kinds of information such as admission date, patient age and race, etc.

There are 538 higher level ‘intermediate’ rules in our CDS system. These rules use the results of building block rules, and/or the results of other intermediate rules in their logic.

Our system contains 408 reminder rules. Approximately 72% are triggered by a treatment order, 15% by the order of a diagnostic test, 10% by entry of a diagnosis or problem, and 3% by the passage of time. There is an average of four logic sections per rule and the number of supporting rules used by one reminder rule ranges from 1 to 108. The most complex reminder rule is a rule that suggests an increase in antihypertensive treatment. It uses 108 supporting rules, and can display one of 69 different messages. On average, reminder rules use 10 supporting rules.

There are 38 reminder rules where the default user choice is ‘order’, 58 rules where the default user choice is ‘omit’ and 312 rules that display messages with no specific prewritten orders.

We examined which patient data elements were most useful when creating other rules. We ranked the 360 building block rules according to the number of higher level rules that use them. Table 2 shows the 10 top ranked building block rules. Not unexpectedly, data elements such as patient age, sex, weight, and last creatinine are frequently used by other rules. The highest ranked building block rule, used by 197 other rules, is ‘problems’ which retrieves the problem list of the patient. The top 5 (1.4%) highest ranked rules account for almost 10% of the total uses by other rules.

Table 2.

Top 10 rules ranked by their frequency of use by other rules

Rule Name	Rule Type	Rule Utilization
PROBLEMS	SPECIAL	197
ADMIT_DATE	SPECIAL	195
OUTPATIENT_PROFILE	SPECIAL	179
AGE	SPECIAL	172
SEX	SPECIAL	157
GUIDELINE_PAT	SPECIAL	90
PROBLEMS_INPAT_PREFORM	SPECIAL	82
INPUT_FORM_TYPE	SPECIAL	81
ORDERS_ACTIVE_INPAT	SPECIAL	75
CREATININE_LAST	SELECT.	67

Reminder rules generally are very complex and constitute a relatively small percentage of our total CDS system. The majority of our system consists of lower level supporting rules and creating a single reminder rule usually requires the use of multiple supporting rules. For example, a seemingly simple clinical question such as “does the patient have renal failure?” requires 15 rules to accurately answer. Figure 1 illustrates the complex structure of a typical reminder rule. Reminder rules with the highest complexity typically involve treatments such as recommending an increase in an antihypertensive medication, or the best medication for pneumonia.

An interesting aspect of our system is that rules which retrieve dynamic, often transient data from the CPOE, such as problems entered during the current order session, active drug orders (separate from medication lists), input form type (such as admission orders, discharge orders), the location of the patient (ICU, outpatient, medicine ward), and time concepts (current date and time), are the most frequently used by other rules. Relatively ‘static’ data from the patients’ EMR, such as the last creatinine, are used less frequently. Although most of the 360 building block rules are selection rules (276) (77%), special rules (which retrieve dynamic data from the CPOE Gopher system itself) are used the most frequently. We ranked the top 20 rules by frequency of use by other rules and found only 4 (20%) were selection rules while 16 (80%) were special rules (which retrieve dynamic data from the CPOE Gopher system itself). This is despite the fact that most of the 360 building block rules are selection rules (276) (77%).

We wanted to determine our ‘core set’ of building block rules- rules that involve patient data elements which are essential for the creation of most reminder rules. We analyzed the top 10 most frequently used building block rules (Table 2) and determined how many reminder rules used at least one of them in their logic. We found that 198 (49%) of reminder rules use at least one of these 10 building block rules.

Discussion

The Regenstrief COPE/CDS system is an established, well accepted, successful system. Many factors contribute to its success. First and foremost is the fact that our major consideration during design of our system was its potential impact on provider workflow. Decision support often introduces new steps and each step must be evaluated in terms of provider workflow before implementation. For example, we attempt to minimize the actions required by the user when an alert is displayed. Our system is designed to keep rule response times at the sub-second level. Users will not tolerate decision support that involves significant processing. We attempt to keep the messages in reminder rules short and focused. It is also easy to overwhelm the users with too many reminders. Our development team carefully considers a reminder rules’ significance and clinical importance before adding it to the system.

Several other key elements contribute to our system’s success. The design of our system allows CDS to be provided in real time during direct patient care, a method shown to be successful.⁸ Our system is designed with the philosophy that “the user is always right”. Users can override nearly every decision. We recognize that computer systems often lack fine details and reminder rules cannot anticipate every situation. We actively seek user feedback with regularly scheduled meetings. This feedback is critical to both creating a user-friendly system and addressing problems early. We take great pains to minimize the downtime of our system. Providers must be able to depend on the system and users will not tolerate erratic behavior (unpredictable reminders) or a system that is unreliable.

Beyond the obvious goal of improving healthcare quality and increasing patient safety, there are several other potential uses of a CPOE/CDS system. One additional use of such a system is in the accurate and timely identification of patients who are potential participants in ongoing clinical research. The purpose of several of our rules is to monitor for inclusion and exclusion criteria of specific studies.

As part of our CDS system evaluation, we installed a monitoring system that logs the frequency with which a particular rule fires. For each rule triggered, we record the date, the message displayed to the provider, the identifier of the provider who observed the message and the identifier of the patient referenced by the message. Such a monitoring system creates the opportunity to provide users with practical and specific feedback on how they may improve patient care.

Another interesting by-product of monitoring the frequency with which a rule is triggered is in estimating the cost savings provided by the CDS system. For example, we have a rule called block_hang_blood. This rule is designed to prevent an order for a blood transfusion when it potentially is not clinically indicated. It is estimated that the average cost of a blood transfusion (per unit) in a hospital setting is $155.¹³ We can conclude that each blood transfusion order which is canceled as a result of an alert from this rule being displayed, equates to a cost savings of $155 for the health care system. Similar calculations could be made for additional rules, such as the rule to block platelet transfusions not clinically indicated, and block an order for a lipid profile when only an LDL is likely needed. Monitoring the firing rates and outcomes of triggered blocking rules over a period of several months could be part of an overall system to approximate the cost savings in health care dollars provided by a CDS system.

The design and implementation of a successful CPOE/CDS system is a complex undertaking. By understanding the elements of an existing and successful system, the task of developing future systems will become more manageable.