A Description and Functional Taxonomy of Rule-based Decision Support Content at a Large Integrated Delivery Network

Abstract

Objective

This study sought to develop a functional taxonomy of rule-based clinical decision support.

Design

The rule-based clinical decision support content of a large integrated delivery network with a long history of computer-based point-of-care decision support was reviewed and analyzed along four functional dimensions: trigger, input data elements, interventions, and offered choices.

Results

A total of 181 rule types were reviewed, comprising 7,120 different instances of rule usage. A total of 42 taxa were identified across the four categories. Many rules fell into multiple taxa in a given category. Entered order and stored laboratory result were the most common triggers; laboratory result, drug list, and hospital unit were the most frequent data elements used. Notify and log were the most common interventions, and write order, defer warning, and override rule were the most common offered choices.

Conclusion

A relatively small number of taxa successfully described a large body of clinical knowledge. These taxa can be directly mapped to functions of clinical systems and decision support systems, providing feature guidance for developers, implementers, and certifiers of clinical information systems.

Introduction

Clinical decision support systems have great potential to improve the quality and lower the cost of health care. ^1–8 Such systems have been used to prevent errors, improve quality, reduce costs, and save time. The best evidence suggests that such systems, when used, can be extremely effective and provide significant value. ⁹ For example, a recent systemic review by Garg found that, in 100 studies of clinical decision support, “[Clinical decision support systems] improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome, including 4 (40%) of 10 diagnostic systems, 16 (76%) of 21 reminder systems, 23 (62%) of 37 disease management systems, and 19 (66%) of 29 drug-dosing or prescribing systems.” ⁴ A recent estimate suggests that adults today receive only 55% of all recommended, evidence-based care that they should be receiving. ¹⁰ It is estimated that if clinical decision support systems were in place within electronic medical record systems and used in all ambulatory care encounters, the country as a whole could reap potential savings of $44 billion. ¹¹ This sort of underprovision of appropriate care can be improved immensely by decision support systems, particularly computerized reminder systems. ¹²

Decision support systems are the subject of much interest and research given their great potential to improve care delivery and patient safety. However, communicating about decision support and classifying decision support systems can be difficult because of a lack of consensus on definitions, nomenclature, and models. Researchers have developed a number of taxonomies of clinical decision support from a variety of perspectives to help bridge this gap. In this article we present a new taxonomy of rule-based clinical decision support focused on the functional perspective. Our goal is to lay out a theoretical framework for understanding the functional capabilities that must be made available to effect clinical decision support, such as triggers, data elements, interventions, and offered choices.

Background

Prior Taxonomies in Clinical Decision Support

A number of clinical decision support taxonomies have been proposed. Most are based on expert opinion or designed for use in a specific task. One widely used prescriptive taxonomy is that proposed by Osheroff et al. ¹³ in their book, Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. They lay out a taxonomy of clinical decision support methods: documentation forms/templates, relevant data display, order creation facilitators, time-based checking and protocol/pathway support, reference information and guidance, and finally, reactive alerts and reminders. This taxonomy is very useful for considering which methods of intervention might be useful to solve a particular clinical or quality problem; however, it is not as useful for those who are seeking to develop or share a decision support system.

Other researchers have taken an empirical approach to developing taxonomies. A recently published paper by Berlin et al. ¹⁴ develops a taxonomy based on a review of 58 randomized controlled trials of clinical decision support systems consisting of 74 clinical decision support scenarios. They identify five categories: context, knowledge and data source, decision support, information delivery, and workflow. The context category describes the “setting, objectives, and other contextual factors of a system’s use” and includes taxa such as clinical setting and clinical task. The knowledge and data source category looks at the sources of clinical knowledge (such as guidelines) and patient data source (electronic medical record, direct entry into the system, etc.). The decision support category looks at the reasoning aspect of the system, such as the type of inference being made and the complexity of the recommended action. The information delivery category comprises taxa such as delivery format and mode—this category is particularly important because not all of the decision support systems reviewed in the article were fully computerized. Some, for example, result in printouts inserted into paper charts. The final category in the Berlin taxonomy is workflow, which includes taxa such as the user of the system (some systems target clinicians, whereas others are used directly by the patient), and system-workflow integration. This taxonomy provides insight into the design and intent of clinical decision support systems.

Another taxonomy has been proposed by Wang et al. ¹ based on their experience implementing a computerized physician order entry (CPOE) system at Cedars-Sinai Hospital. Their hierarchy has three levels. The top level describes the benefit of a clinical rule: process improvement, policy implementation, error prevention, or decision support. Under these benefits are a set of domains, such as laboratory (under the process improvement benefit), pharmacy (under the error prevention and decision support benefits), and Joint Commission on Accreditation of Healthcare Organizations requirements (under the policy implementation benefit). The lowest level of the taxonomic tree is termed class. Classes are used to logically organize clinical rules by content type and include such elements as drug-drug interaction checking, automated orders, and guided dosing.

The purpose of this taxonomy is to guide the organizational aspects of clinical decision support. For example, the investigators used the benefit classifications in presentations about their CPOE system to help end users understand the advantages of CPOE. The domain classification is used to determine the “owner” of a piece of decision support content—for example, the pharmacy and therapeutics committee at the hospital is responsible for decision support assigned to the pharmacy domain. The lowest level, class, is useful for knowledge management and implementation tasks. Unlike the Berlin taxonomy above, which is best suited for decision support research, this taxonomy would be most useful for applied use, in managing or administering a decision support program.

Miller et al. ¹⁵ also developed a taxonomy of clinical decision support content based on the experience at Vanderbilt. Vanderbilt has a well-known inpatient CPOE system called WizOrder, and Miller et al. used it to develop a taxonomy with three axes: the type of intervention (optimal ordering, patient-specific decision support, optimal care, just-in-time education), when in the workflow to introduce the intervention (such as when initiating a session, when selecting an order, etc.), and how disruptive the intervention should be (ranging from incidental display to pop-ups and complex protocols).

The taxonomy presented here differs from the taxonomies described above in two key ways. First, it is a functional taxonomy. It does not characterize the content or purpose of the decision support interventions in the system as the taxonomies by Osheroff et al. do. Instead, it describes the functional requirements of the decision support—those features that must be made available to decision support systems so that they can carry out their activities. Such a taxonomy is useful for designing clinical systems because it is generally the clinical system that exposes these features. It is also useful for developers of decision support standards and knowledge representation formalisms, as well as for those who might wish to design a decision support service, because the elements of the taxonomy correspond to the input and output requirements of such a formalism or service. In addition to its unique perspective, this taxonomy also differs from the taxonomies of Osheroff et al., Wang et al., and Berlin et al., (but not of Miller et al.) because of the breadth of its empirical basis—it is derived from a comprehensive analysis of the decision support content in use at Partners Healthcare System, a large integrated delivery network with a long history of computer-based point-of-care decision support. ¹⁶

Hypothesis

Some of the earliest clinical decision support systems, such as MYCIN and INTERNIST-1, were standalone, but most modern decision support systems are not. Instead, they tend to be tightly integrated with clinical systems, such as electronic health records, computerized provider order entry systems, and nursing systems. To integrate with decision support systems, clinical systems must expose a variety of capabilities to decision support systems. For example, if one wishes to build a decision support system for checking for drug-drug interactions and integrate it with a clinical system, that clinical system must, at a minimum, have the capability to notify the decision support system that a medication is being ordered and describe that medication. It must also have the capability to provide the list of other medications the patient is on, which the decision support system will check the new order against. However, defining the full set of capabilities that a clinical system must expose to a decision support system is difficult. Some approaches, such as the Health Level 7 (HL7) Decision Support Service ¹⁷ standard, specify that the entire set of capabilities that the HL7 Version 3 Reference Information Model specifies should be made available for integrating decision support. Other methods of integrating decision support, such as Arden Syntax, ¹⁸ specify a smaller set of capabilities, based mostly on the expert opinion of the standard developers. Our goal was to develop a taxonomy of these capabilities empirically. We began our study with the hypothesis that the number of capabilities needed would be finite and fairly small; and, as described in the Methods section, we carefully reviewed a large knowledge base of decision support content to develop a taxonomy that would validate or invalidate this hypothesis. Such a taxonomy would also be useful for developers of clinical systems, decision support systems, and standards developers.

Methods

Site and Content Description

Partners Healthcare System (Partners) is a federation of hospitals in the Boston metropolitan area. Its original founding members were the Brigham and Women’s Hospital and the Massachusetts General Hospital, both teaching affiliates of Harvard Medical School. Since its founding in 1994, Partners has grown and its current membership is described in ▶.

Table 1.

Table 1 Members of Partners HealthCare

Members
Brigham and Women’s Hospital ∗ †
Massachusetts General Hospital ∗ †
Faulkner Hospital
McLean Hospital†
Newton-Wellesley Hospital
North Shore Medical Center
MGH Institute for Health Professions
Partners Community Healthcare, Inc.
Partners Continuing Care

^∗ Founding members.

^† Teaching affiliates of Harvard Medical School.

Each member hospital maintains control over its own clinical systems and uses a combination of centrally or locally managed information system resources. For example, ancillary departmental systems are managed locally but may employ services from the centrally managed enterprise master patient index. In 2002, Partners launched an enterprise knowledge management effort with the goal of moving all decision support content in use at the sites into a centralized knowledge management portal. This portal now contains most Partners content, although a small amount has not yet been moved and remains hard-coded in applications. This portal provides a robust environment for collaborative development and review of clinical knowledge. The portal has been described in depth elsewhere. ¹⁹ The portal does not contain directly executable content. Instead it contains knowledge specifications in various forms that can be used by developers and implementers of clinical systems.

The clinical decision support knowledge base at Partners is extremely large, comprising a total of 181 rule types and 7,120 different instances of rule usage. A systematic review identified Partners as a benchmark institution and one of the major sources of research and development in clinical decision support, along with the Regenstrief Institute, the Department of Veterans Affairs, and LDS Hospital. ¹⁶ The knowledge base covers both inpatient and outpatient settings, a variety of decision support domains ranging from medication alerts to preventive care reminders, cost and efficiency notices, guideline and protocol support, and support for interpreting structured clinical data, crossing most medical and surgical specialties. The knowledge base has been refined over the past 2 decades by dedicated teams of informaticists, physicians, nurses, pharmacists, and medical scientists. As such, we believe it is an excellent pool to draw from in developing a functional taxonomy of clinical decision support.

Study Design

The investigators conducted an exhaustive review of the contents of the knowledge management portal to develop the taxonomy. Four functional categories were identified a priori:

• • Triggers: The events that cause a decision support rule to be invoked. Examples of triggers include prescribing a drug, ordering a laboratory test, or entering a new problem on the problem list.

• • Input data: The data elements used by a rule to make inferences. Examples include laboratory results, patient demographics, or the problem list.

• • Interventions: The possible actions a decision support module can take. These include such actions as sending a message to a clinician, showing a guideline, or simply logging that an event took place.

• • Offered choices: Many decision support events require users of a clinical system to make a choice. For example, a rule that fired because a physician entered an order for a drug the patient is allergic to might allow the clinician to cancel the new order, choose a safer alternative drug, or override the alert and keep the order as written but provide an explanation.

These categories were identified because together they fully describe and specify the components of an interface between a clinical decision support system and a clinical system.

The individual elements, or taxa, inside the categories were determined empirically over the course of the review, and each rule in the knowledge management portal was assigned to the appropriate taxon. After the assignment was completed, the taxa were reviewed and refined.

One challenge encountered in conducting this study was determining the unit of analysis. For the purpose of this analysis we introduce two concepts: the rule type, and the rule—analogous to the idea of a variable type and an instance in computer science. It is perhaps easiest to define these terms by example. At Partners, a single module encodes 1,561 drug substitutions, all of which use the same logical structure. This would be counted as one rule type, but 1,561 rules. Rule types range from containing only a single rule to containing several thousand, but most rule types contain only a few entries. Rule types often are aligned with clinical purposes, but rule types are sometimes partitioned to reflect organizational priorities, such as what medical division has oversight for a set of rules. The major unit of the analysis for this article is the rule type; however, all tables in the results section also present rule counts. The rule type was chosen because it is the unit of focus for the knowledge management process: each rule type has a source, an author, and one or more responsible persons, and each rule type follows a knowledge management lifecycle. This challenge of choosing a unit of analysis has been encountered in other studies ²⁰ ; however, in the end it poses only a moderate challenge because the universe of taxa is the same regardless of each taxon’s frequency.

Results

Triggers

A total of nine triggers were identified, and they are described in ▶. The distribution of triggers is extremely skewed—the top three triggers account for 94% of all rule types (and 94% of all rules). The top trigger is new-order entered. Many common decision support interventions, such as drug–drug interaction checking and test appropriateness verification, all fire when a new order is entered. The second most common trigger is laboratory result stored, such as a rule that alerts a clinician whenever a potassium level of >5 mEq/l is stored. The major difference between the top two triggers relates to their synchronicity—rules triggered by order entry are almost always synchronous—that is, their result is displayed to the clinician as part of the ordering process. Laboratory result–triggered rules, however, are asynchronous by definition. Depending on the clinical severity of the result and the clinical system that the rule is executing within, these rules may page a clinician, send an e-mail, alert a nurse, generate a patient letter, or simply add a low-profile flag to a patient’s electronic medical record.

Table 2.

Table 2 Triggers for Decision Support

Trigger	Rule Types	Rules	Example Rule
Order entered	99	6732	When digoxin is ordered, check potassium
Laboratory result stored	93	998	When glucose is stored, check value
Outpatient encounter opened	42	48	When a patient presents for a routine physical, order cholesterol test if needed
User request	4	152	When user requests them, show antibiotic utilization guidelines
Time	4	25	24 hours after admission, check for a medication list
Admission	3	151	When a patient is admitted for congestive heart failure, offer standard therapy
Problem entered	1	145	When asthma is diagnosed, request date of onset
Enter allergies	1	3	When a penicillin allergy entered, check drug list
Enter weight	1	3	When a patient’s weight is entered, ensure that it is reasonable

The third trigger, outpatient encounter opened, is fairly specialized. Almost all rules with this trigger occur in the Longitudinal Medical Record system (LMR). These LMR rules generally relate to prevention—for example, whenever a new encounter is opened, a rule fires that checks to see whether that patient is up to date with current National Cholesterol Education Program cholesterol management guidelines. ²¹ If the patient is not, an alert is shown along with appropriate remedial actions, such as ordering a cholesterol test or starting the patient on a statin.

The remaining triggers are much less frequent, and generally self-explanatory, although a few merit special discussion. The user request trigger relates to a number of guidelines and order sets that do not have any automatic trigger—instead, a user must intentionally request them. The time trigger has a number of different uses. For example, one rule fires every morning at 9:00 am to check the currency of laboratory values. Another rule fires 1 week after an abnormal mammogram, prompting the clinician to contact the patient to confirm that appropriate follow-up measures are underway.

A single rule can have multiple triggers, and many do. For example, one rule watches for hypokalemia in patients on digoxin. The rule is triggered whenever digoxin is ordered (a new order entered trigger), and also whenever a new potassium result is stored (a laboratory result–stored trigger).

Input Data

With 14 taxa, input data is the largest category in the taxonomy. ▶ shows the category. Although still quite skewed, it has the largest spread of any category, with eight of the 14 taxa being represented by at least 10 rule types. The most frequent input data elements consumed by rules are laboratory results/observations and medications, exactly mirroring the two most common triggers. Many rules use both of the data elements, such as the rule described above, which monitors potassium in patients on digoxin. It is important to note that the laboratory results/observations taxon includes not only standard laboratory results, but also structured observations, such as nursing documentation or data entered into standard templates.

Table 3.

Table 3 Input Data Elements Consumed by Decision Support Rules

Input Data Element	Rule Types	Rules	Example Rule
Laboratory result/observation	126	2,087	Check if latest hemoglobin A1C is >6%
Drug list	108	4,752	Active prescription for fluoxetine
Hospital unit	85	906	Coronary care unit
Diagnosis/problem	43	1,587	Decrease dose of cefuroxime in patients with renal insufficiency
Age	39	3,131	Warn about nifedipine use in the elderly
Nondrug orders	15	694	Patient has an active total parenteral nutrition order
Gender	12	1,595	Only suggest a mammogram in female patients
Family history	10	10	Suggest lipid panel more frequently for patients with family history of myocardial infarction
Allergy list	9	649	Check for a penicillin allergy when amoxicillin is prescribed
Weight	8	1,310	Suggest lipid panel more frequently for overweight patients
Surgical history	8	8	Do not recommend mammogram with history of bilateral mastectomy
Reason for admission	2	148	Suggest default orders when a patient is admitted for myocardial infarction
Prior visit types	2	2	Check for ophthalmology visit in the past year for diabetic patients
Race	1	1	Recommend a calcium channel blocker for patients with black race

The third most common data element is hospital unit. Many inpatient rules apply only to patients in certain units—for example, patients in the coronary care unit have more narrow cardiac parameters than patients in other units. There are also certain drugs and procedures that can only be ordered in specific areas—for example, succinylcholine can only be ordered in the intensive care unit. This data element also is sometimes used as a proxy for the patient’s condition. For example, a patient who is admitted to the coronary care unit is likely being treated for a heart problem.

As would be expected, a variety of demographic data also are used in decision support, such as age, gender, and race (which is used in only a single rule, which recommends calcium channel blockers in black patients). Family history is also used—it is encoded at Partners by disease and severity. The mammogram rule would recommend more frequent mammograms, beginning earlier, for a woman with an extensive family history of breast cancer. The rest of the data elements used are explained in ▶. It is worth noting that there are very few natural language processing systems in use at Partners, so none of the rules used in developing this taxonomy used clinical notes or reports directly, although some systems did use specific coded findings, which are exposed as observations, as described above.

Interventions

The interventions category is the smallest category. The most common member of the category is the most complex—notification. All forms of notification involve communicating a piece of information to a responsible clinical user, but these notifications can take many forms depending on the urgency of the information and the application context. These forms are described in ▶. In addition to a variety of notification forms, notifications frequently offer the user choices. These choices represent the fourth category of the taxonomy, and are described in the next section.

Table 4.

Table 4 Dimensions of Notification

	Synchronous	Asynchronous
Urgent	Pop-up messages	Paging, visual alerts on the hospital unit
Nonurgent	Informational messages	E-mail, clinical inbox

After notification, logging is the most common intervention. Logged messages are stored by the application and are available for analysis and review, but unlike notifications, they are not shown to the user and do not have any associated choices. Logging is frequently used in surveillance rules and monitoring rules, particularly for adverse drug events and in research studies. Providing defaults and picklists as an intervention is frequent with drug dosing rules. The Nephros system for renal dosing ²² and the Gerios system for geriatric medication ²³ use make substantial use of this response type. The remaining interventions are less frequent, and all interventions are described in ▶.

Table 5.

Table 5 Interventions by Decision Support Systems

Intervention	Rule Types	Rules	Example Rule
Notify	126	4,708	Alert the user when a patient’s potassium is >5
Log	58	173	Log all uses of ketorolac for utilization review
Provide defaults/pick lists	21	3,142	Compute recommended doses for a patient with renal impairment
Show guidelines	15	740	Show guidelines for use of antibiotics
Collect free text	8	391	Request a reason for overriding an alert
Get approval	3	662	Send order to endocrinology when growth hormone is ordered
Show data entry template	2	147	Request details when asthma is added as a problem

Offered Choices

As mentioned above, the offered choices category may be considered a child of the notify intervention. The members of this category are listed in ▶. The most common offered choice is to write an order. This comes up in a variety of clinical workflows—for example, Partners has a therapeutic substitution rule that recommends famotidine when other equitherapeutic, but more expensive, H2-receptor antagonists are ordered—this recommendation of famotidine would qualify as an order suggestion. This choice also occurs in laboratory result–oriented workflows. For example, when a low potassium value is stored for a patient on digoxin, clinicians are given the option of ordering potassium supplementation. The next two most frequent choice types, defer warning and override rule/keep order, both dismiss the notification received without changing the user’s current course of action—the defer warning choice dismisses the warning for a period of time, and the override rule choice dismisses it more permanently—until the condition worsens, or until the action that caused the notification is repeated. The next four choices, cancel existing order, cancel current order, edit current order, and edit existing order, all primarily occur in drug–drug interaction rules. When a clinician enters a potentially interacting order, he or she is offered the choice of canceling or editing either the new drug order or the existing order.

Table 6.

Table 6 Offered Choices as Part of Notification Interventions

Offered Choice	Rule Types	Rules	Example Rule
Write order	63	2,059	Change a ranitidine order to famotidine
Defer warning	47	94	Allow the user to defer a warning for 24 hours
Override rule/keep order	47	3,014	Keep an order that triggered a low-severity drug interaction rule
Cancel existing order	30	240	Discontinue an existing order for fluoxetine when it is flagged as duplicating a new order for paclitaxel
Cancel current order	29	3,110	Cancel an order for furosemide in a patient with a sulfa drug allergy
Edit current order	26	1,538	Change the dose of an order for 16 g acetaminophen
Edit existing order	23	42	Reduce digoxin when patient is hyponatremic
Set allergies	14	20	Decline a suggestion to order atenolol because the patient is allergic
Write letter	7	86	Send a letter to a patient with a normal mammogram
Write note	4	23	Provide default text for a note on a patient with an elevated low-density lipoprotein level
Edit problem list	4	4	Remove hypertension from the problem list in response to a suggestion for antihypertensive therapy
Enter weight, height, or age	3	787	Allow user to enter weight when ordering a drug with weight-based dosing

The set allergies choice most frequently occurs in response to interventions that suggest a drug. When the system suggests, for example, a statin, the user is offered the choice to turn the suggestion down because the patient is allergic to that statin. In addition to dismissing the alert, this also adds the appropriate allergy to the patient’s allergy list. The next two choices, write letter and write note, provide starting text for results letters to patients and for progress notes. They are primarily used in the Results Manager ²⁴ and LMR SmartForm modules ²⁵ at Partners. Finally, the edit problem list choice is used to add or remove items from the problem list, whereas the enter weight, height, and age choice is used to query the user for this information, generally before proceeding to a calculation or inference that requires this information.

Conclusions and Implications

This analysis indicates that a very large amount of decision support can be accomplished with a fairly small number of functional constructs across a finite set of categories. This suggests that the problem of integrating decision support into clinical systems, although nontrivial, should be tractable because these functional dimensions can be loosely translated into functional requirements and specifications for clinical applications and knowledge representation formalisms.

The taxonomy we developed closely parallels a number of efforts to standardize decision support. The four categories we identified appear in a number of knowledge representation formalisms. For example, Arden Syntax ¹⁸ rules are broken down similarly, with the Arden Syntax “evoke” section corresponding to our trigger category, the “data” section corresponding to our input data category, and the “action” section corresponding to our interventions category. In the final model there were also great similarities between the taxa identified by this methodology and those identified in other similar efforts 18,20,26,27 and to concepts frequently considered in informatics, such as elements in the HL7 Version 3 Reference Information Model. ²⁸

There are a variety of possible applications for a taxonomy such as this. One major and immediate use is the development of knowledge representation standards. This is a rich field, and a variety of formalisms, such as Arden Syntax, ¹⁸ GuideLine Interchange Format (GLIF), ²⁹ and Guideline Expression Language, Object-Oriented (GELLO), ³⁰ are available. It would be a productive exercise to see whether all of the taxa identified here could be properly mapped by each of these formalisms, and the taxa could likewise serve as a roadmap for future development of knowledge representation standards. For example, as mentioned earlier, Arden Syntax can map many elements of our taxonomy, but there are gaps—whereas we identify seven interventions (and 12 associated choices), Arden Syntax only covers one: notify (which it terms “write”).

Beyond knowledge representation standards developers, this taxonomy may be of interest to the broader standards community. It seems sensible that each trigger and data element identified should be representable according to a suitable message and vocabulary standard, but this is not currently possible. As an example, although good vocabulary standards are currently available and in use for drugs and laboratory tests, use of standard vocabularies for data elements such as family history, allergies, and problems is much less widespread. Given the data presented here, this is, perhaps, reasonable because drug and laboratory data are more frequently used than the other data elements. But in a perfect world, all data elements could be represented according to a standard and the results described here may help in prioritizing the development and adoption of such standards.

This taxonomy also should be useful to developers and implementers of clinical information systems and to clinical knowledge providers. Closely related to this use case is certification. A clinical system that implements features satisfying all of the functional dimensions described in this article is likely to be capable of a fairly comprehensive range of decision support, so these dimensions may be a useful starting point in framing certification requirements for decision support, just as the HL7 Electronic Health Record definition ^31,32 was used as a starting point for the Certification Commission for Health Information Technology ambulatory system certification criteria.

We also hope that the taxonomy described here is useful for decision support researchers. It is one tool for evaluating the generalizability of new decision support systems and knowledge representation formalisms. Some of the most exciting developments in decision support are likely to occur at the edge of the region of functionality defined by this taxonomy, necessitating revision and expansion as new areas are explored and new foci are developed.

Limitations

The major limitation of this study is that it looks at content in only a single integrated delivery network. Although this federation has a variety of hospitals, ranging from major academic medical centers to small community hospitals, there is some degree of homogeneity across the institutions. We are aware of certain classes of decision support in use at other institutions that cannot be fully described using this taxonomy. For example, some systems apply natural language processing techniques to progress notes or specialist reports (such as radiology reports) as a part of their inference process. ^33–35 No such systems were encountered in our analysis, so such reports are not described in the input data elements category of the taxonomy described in this article, and this taxonomy could not fully map such systems. There is a related possibility that some of the skew seen in the distribution of the four categories may be attributable to the ease of accessing the various taxa—for example, order entered and laboratory result stored were the most common triggers. However, it is difficult to discern whether this is because they are actually the most useful triggers, or simply because they are the most readily available and familiar to decision support developers. This is an inherent limitation of any empirically developed taxonomy—such a taxonomy can only include those taxa found in the site or sites on which the taxonomy is developed.

Future Directions

In future work, we intend to extend this taxonomy to other institutions with two primary goals. The first goal of this extension is to measure the generalizability of the taxonomy, and to measure the extent to which it can successfully describe content in other settings. The second goal is to extend the taxonomy to include new taxa in use at other institutions, and also to generalize it beyond rule-based knowledge to include other forms of decision support, as well as other elements, such as the targeted actor or relevant clinical situations and workflows.

We also intend to begin mapping the categories and taxa identified here to currently available message and vocabulary standards as a way to assess the adequacy of the current standards base for use in decision support systems. A complete analysis of the standards landscape through the lens of decision support would be useful for standards developers.