Order Sets in Computerized Physician Order Entry Systems: an Analysis of Seven Sites

Abstract

Most computerized physician order entry (CPOE) systems have built-in support for order sets (collections of orders grouped by a clinical purpose). Evidence and experience suggest that order sets are important tools for ordering efficiency and decision support and may influence ordering. Developing and maintaining order sets is costly, so hospitals often must prioritize which order sets can be created. We analyzed order set utilization at seven diverse sites with CPOE. The number of order sets per site ranged from 81 to 535, and the number of order set uses per discharge ranged from 0.48 to 9.89. We also compared the top ten order sets at each site, and found many commonalities, such as generic and condition-specific admission order sets, surgical sets and clinical pathways. We also found that, at each site, utilization of order sets was skewed, with a small number of order sets comprising the bulk of utilization. These findings may be useful for order sets developers, particularly in settings where resources are constrained and the most important order sets must be developed first.

Introduction

Computerized physician order entry (CPOE) with clinical decision support (CDS) has been shown to reduce errors [1]. The core function of CPOE is the capture and transmission of clinical orders; however, most CPOE systems incorporate support for a variety of time-saving devices such as quick orders, shortcuts, care protocols and order sets. Order sets are collections of orders grouped by a clinical purpose. Common order sets may include admission order sets, order sets related to the diagnosis or treatment of a particular condition, or order sets which embody a particular care process (such as the process for ruling out a myocardial infarction).

The addition of well-designed order sets to a CPOE system has at least two important benefits. First, order sets can be used to improve the speed at which physicians enter orders, since they can simply select from a menu of relevant orders rather than manually entering them. Second, order sets can encourage evidence-based, efficient care by influencing provider behavior [2–4], and many order sets are developed with this intent in mind.

Both evidence and experience suggest that order sets can be effective tools; however, little is known about how order sets are being developed and utilized in hospitals around the world. The best-known work on order sets was published in 2003, and looked at the use of order sets (as well as related CPOE artifacts) at the Veterans Affairs Health System [5]. The authors concluded that development of order sets was time consuming and resource intensive, but that order sets were widely used and important. Interestingly, in their analysis, they found that a small number of order sets accounted for the bulk of order set usage, and that many order sets which had been developed were not used at all over the course of a six month period.

In this paper, we conduct an analysis of order sets at seven purposively selected sites, with a focus on utilization patterns of these order sets. Our goal is to extend what is already known about order sets by doing the first multi-site analysis of order set utilization with a cross-site comparison of user behaviors.

Methods

Sample

We began by selecting a purposive sample of sites to provide information to us. We started with a set of ten sites – two were unable to provide the data we requested because they did not log order set utilization and one declined to participate, leaving a sample of seven sites. The characteristics of the sites are presented in the results section.

Dataset

We asked each site to provide us with anonymized logs from their CPOE system showing the date, time and order set name for each instance of order set utilization over the course of the year. Most sites provided us with raw logs, though two provided us with summary data only. Six of the seven sites provided data for a year-long period; however, one site provided us with only six months of data – we doubled the counts for this site to account for the shorter measurement period.

Analysis

We loaded all of the order set records from the participating sites into a Microsoft Access database (Microsoft Corporation, Redmond, WA, USA) and computing counts and summary information. We also aggregated the source data by order set, allowing us to compute the top order sets at each site. Where necessary, non-order set information (such as order menus) was cleaned from the dataset.

Results

Sites

Table 1 shows the characteristics of the seven sites, including the HIT system in use, general information, and hospital demographics from the American Hospital Directory (http://www.ahd.com/). The hospitals are geographically distributed across the United States and represent a mix of small and large hospitals and academic and community settings.

Table 1:

Characteristics of the seven sites.

Hospital	System	Location	Type	Staffed Beds	Discharges	Patient Days
Brigham and Women’s Hospital (BWH)	Brigham Integrated Computing System (BICS)*	Boston, MA, USA	Academic Med Center	750	44,571	261,076
Faulkner Hospital (Faulkner)	MEDITECH MAGIC	Boston, MA, USA	Community	153	7,637	38,510
Kaiser Permanente Northwest (KP)	Epic Systems EpicCare	Oregon/Washington, USA	Health System	208	14,817	58,983
Massachusetts General Hospital (MGH)	Physician Order Entry (POE)*	Boston, MA, USA	Academic Med Center	907	50,011	288,565
Sugar Land Hospital (SLH)	Allscripts EMR	Sugar Land, TX, USA	Community	77	5,133	17,536
NSMC Union Hospital (NSMC)	Siemens INVISION	Lynn, MA, USA	Community	415	20,457	109,601
Providence Portland Med Center (PPMC)	McKesson Horizon Expert Orders	Portland, OR, USA	Community	383	21,703	109,665

^*Self-developed system

Kaiser Permanente Northwest (KPNW) deserves special mention. Unlike the other sites, which are hospitals, KPNW is a health system with one hospital, twenty-six outpatient facilities and a 485,000 member health plan. KPNW is one of 8 regions of the Kaiser Permanente program. Statistics presented in this paper are filtered to focus on inpatient utilization; however, the separation was imperfect and some outpatient utilization may be represented in the figures presented.

Order set utilization

Order set utilization characteristics are presented in Table 2.

Table 2:

Order set utilization.

Hospital	Total Order Sets	Order Set Usages per year	Discharges	Uses per Discharge
BWH	81	26,732	44,571	0.60
Faulkner	35	3,692	7,637	0.48
KPNW	535	115,889	14,817	7.82
MGH	384	254,983	50,011	5.10
MH	341	14,281	5,133	2.78
NSMC	518	38,298	20,457	1.87
PPMC	508	214,654	21,703	9.89

The number of order sets and utilization of them varied across the sites. BWH, which has a very well-known order entry system, had only 81 order sets, while many of the sites had in the neighborhood of 500. The utilization of order sets also varied from less than one per discharge to about five, or in one case, to 9.89.

Top order sets

Table 3 shows the top ten order sets for each site. There are definite differences across each site, but some common order sets appear throughout the table:

Table 3:

Top 10 order sets at each of the seven sites.

Rank	BWH	Faulkner	KPNW	MGH	SLH	NSMC	PPMC
1	Basic Admission Orders	Admission	Generic ED Order set	Anesthesia - post-op	General Admission	Medicine Admission	Hospitalist Admission Orders
2	Patient Controlled Analgesia	General Detox	Diabetes Management with Insulin	Anesthesia - pre-op	Anesthesia - Same Day Surgery	Cardiology Admission	DVT Prophylaxis Guidelines
3	Post Partum Orders	Surgery Admission	Post-anesthesia care	Admission	Labor, Birth and Recovery Admission Orders	Newborn Admission	Blood Transfusion
4	Post-Cardiac Catheterization Orders	Parenteral Nutrition	Standard Admission Orders	Obstetrics Triage Orders	Cardiac Surgery ICU Post Op Orders	Obstetrics Admission	Critical Care Admission
5	Post-Cardiac Catheterization Admission	Rule-out MI	Pre-operative adult orders	Neonatal Admission - Nursery Level 1	Neurology Admission	Psychiatry Admission	Low Molecular Weight Heparin Ordering Advisor
6	Labor Admissions Template	Opiate Detox	Patient Controlled Analgesia	ED GI related	Same-day Surgery Generic Post-op	ICU Admission	Zosyn Ordering Advisor
7	Ischemia Pathway Admission	GI Admission	Expedited admission	Neonatal Circumcision	Orthopedic Surgery Post-op	Obstetrics Vaginal Birth Postpartum	Discharge Care Co-ordination Orders
8	Stroke Admission Orders	Psychiatry Admission	Chest Pain Nursing Orders	ED Cardiac Related Complaint	Pediatric Generic Admission	Pediatric admission (<40kg)	PICC Placement
9	Portland Insulin Protocol	Dialysis	ED Imaging Orders	Chest Pain - Low Risk	Medical ICU Admission	Rehabilitation admission orders	Ciprofloxacin Dosing Advisor
10	PICC line consult	ICU Admission	Blood Transfusion (adult)	Peripheral Nerve Block	Generic post-op template	Cesarean Section Post Op	Telemetry Admission - Universal

General admission orders: All sites had a general admission order set, although the names vary slightly. For example, some sites referred to a medicine admission order set, a generic set, or a hospitalist set, but in each case, these order sets represented common orders for admission to a general medical floor.

Surgical / Anesthesia Orders: The granularity and specifics of these orders varied. Sometimes they were for a particular condition, and in other cases they were more general such as for a clinical specialty. Also, some sites broke out pre-op and post-op orders, while others combined them.

Critical Care Admission: Many of the sites had orders for admission to higher acuity settings, such as a medical, surgical or neonatal ICU or other critical care unit.

Clinical Pathways: Most of the sites had some forms of pathway orders, such a chest pain or rule-out myocardial infarction.

Obstetrics: Many of the sites, particularly those with a high volume of obstetrics had obstetrics order sets, though they differed on granularity (e.g. whether admission, labor and recovery orders were separate or combined).

Diabetes Orders: Several of the sites had diabetes order sets which contained orders for the management of diabetes, usually with insulin. Remarkably, one site (BWH) actually had a specific order set (the Portland Insulin Protocol) which implements a care protocol developed at another site (Providence Health System) [6].

In addition to comparing the top order sets at each site, we also looked at the cumulative distribution of utilization (a Pareto analysis). The results of this analysis are shown in Figure 1. In all cases, the utilization of order sets is heavily weighted toward a small number of very frequently used order sets. This phenomenon is seen in many types of data (word usage, income distribution, retail purchases, etc.) and is sometimes referred to as the power law or, colloquially, the 80/20 rule. In our case, BWH and Faulkner had the steepest distributions because they have relatively fewer order sets. In all cases, no more than 69 order sets was required to account for 80% of order set utilization (at KPNW) and at BWH, the top six order sets accounted for 80.5% of all utilization.

Figure 1:

Cumulative distribution of order set utilization.

Discussion

Findings

Our first finding is that order sets are widely used at each site. Although the extent of utilization (characterized by order set uses per discharge) varied, all sites saw significant utilization. We believe that some of the variability in utilization is due to the design of the system. In some systems, order sets are the primary method for entering orders, while in others they are simply one option, along with, for example, menus for single orders.

Although this does not prove that order sets were useful, the fact that they were consistently utilized when other options (such as entering orders one at a time) were available suggests that users attach some utility to them.

We also found that the most commonly used order sets (the top 10) vary some by site but that there are many repeating themes. Perhaps one of our most significant findings was that some tacit sharing of content was already taking place. One of BWH’s top order sets was the Portland Insulin Protocol – a protocol developed by another participating site (PPMC, NB: the authors were not aware of this when sampling).

The Pareto analysis also provides important insight into utilization patterns for order sets. At each site, a relatively small number of order sets accounted for the majority of utilization. This suggests that, when resources are constrained, sites should concentrate their efforts on order sets which are likely to be commonly used (such as those that repeatedly occurred in our top ten).

Implications

The first implication of our work is that sites implementing CPOE should strongly consider including order sets when they implement their system. This is consistent with prior work (discussed in the background section) suggesting that order sets impart both quality and efficiency benefits.

Our finding that there is a reasonable degree of commonality across sites is important. Developing and maintaining order sets at a particular site is time consuming, and if there really is significant commonality, then sharing of order sets across sites (enabling division of labor and economies of scale) may be feasible.

Limitations

Our work has two limitations. First, we looked at only seven hospitals. Though diverse, they may not be representative of order set behavior at hospitals throughout the world. Second, our review process was somewhat subjective – determining what qualified as an order set at each particular site and which order sets across sites were related required some judgment, and other adjudicators may have reached slightly different conclusions (though we believe that the overall findings would likely be the same).

Future Directions

Our analysis suggests that order sets are widely used by adopters of CPOE, that adopters have found utility in having order sets and that there is a reasonable degree of commonality in order sets and usage patterns across clinical sites. In future research we plan to look at the top order sets at each site and their contents in more detail in order to determine the extent to which such content might be shared. In a similar vein, we also plan to explore mechanisms by which order sets might be shared in an interoperable way [7, 8]. Finally, all of the sites studied for this paper used manual processes to develop order sets – some automated processes have also been described in the literature by us and other and we intend to further explore these processes both as tools to develop novel order sets as well as tools for localizing shared order sets [9, 10].

Conclusion

Although the order sets used by the seven sites in this study varied, they shared several significant themes. Commonalities included generic and condition-specific admission order sets, surgical sets, and clinical pathways. Additionally, order set utilization follows a fairly uniform and steep distribution – a few order sets account for the majority of use at each site. Future developers and implementers of CPOE may find it useful to focus their efforts on these commonly occurring order sets.