Data-driven Order Set Generation and Evaluation in the Pediatric Environment

Y Zhang; JE Levin; R Padman

. 2012 Nov 3;2012:1469–1478.

Data-driven Order Set Generation and Evaluation in the Pediatric Environment

Y Zhang ¹, JE Levin ², R Padman ¹

PMCID: PMC3540526 PMID: 23304427

Abstract

Order sets as part of the Computerized Provider Order Entry (CPOE) system can improve care delivery through allowing faster and easier physician order entry guided by known best practices. This study examines current utilization patterns of order sets and “a la carte” orders in a pediatric environment with a preliminary investigation of methods to automate the creation and modification of order sets using historical ordering data. We examine the current usage of order sets associated with Asthma Minor and Appendectomy Minor patients to understand how physicians are utilizing order sets, and how order set usage is associated with the time of ordering and characteristics of order sets. K-means clustering was applied to orders to generate evidence-based order sets that are learned from historical hospital data. We demonstrate that coverage rate of order sets and ordering efficiency can be increased through modifications of existing sets and creation of new sets.

Introduction

As electronic health record (EHR) systems see steady increase in deployment across the US, hospitals are faced with new challenges beyond the implementation, particularly those raised by Meaningful Use criteria¹. A crucial component of EHR systems, and a major target of Meaningful Use, is Computerized Provider Order Entry (CPOE), currently used in 28% of all US Hospitals². CPOE has the potential to improve care through improved order legibility, completeness, and advanced clinical decision support features to guide care and catch errors before they reach the patient³^,⁴. Typically, orders are entered into CPOE system one item at a time, however even the earliest implementations included items grouped by clinical purpose into “order sets”⁵^,⁶. For example, an “Asthma Admission Order Set” groups together all the suggested items for patients admitted with asthma. Each item can be defaulted to ON, requiring no additional action to submit the order, or can be defaulted to OFF, requiring an additional mouse click. Preventing the ordering of an item that is defaulted to ON also requires a single click to turn it to OFF.

Order sets have the potential to improve care delivery by making it faster and easier for the physician to enter orders and by guiding care according to known best practices⁵^,⁷. Thus, the creation of order sets has been widely accepted as an important prerequisite to a successful CPOE implementation⁸^,⁹. Without using order sets, a physician admitting a patient may choose to enter all the individual items as “a la carte” orders. Reasons for ordering “a la carte” items include order set items not approved by physician, physicians not familiar with order sets, and order sets being inconsistent with current best practice. Some orders are not included in any order sets and “a la carte” is the only way to prescribe them. When ordering, clinicians can search for orders by typing the order names and the search result includes all a la carte orders and order sets that match with the keyword. Depending on the number of orders required, physicians will select either a la carte for one order or order sets for multiple orders.

The labor-intensive creation and maintenance of order sets based on published practice guidelines or the consensus of local experts who may be removed from front-line patient care can contribute to order sets being out-of-date or poorly-accepted¹⁰. In aggregate, the inefficiency resulting from inconsistent use of order set may be significant. Therefore even a small increase in efficiency, safety or consistency could have significant clinical and organizational impact. While order sets derived from scientific evidence-based guidelines are the industry standard, the increasing amounts of data related to orders that are compiled via CPOE and networked across enterprises via Health Information Exchanges provide us with the capacity to automate order set creation from practice-based evidence derived from historical data, including the ability to derive new types of order sets. We hypothesize that the creation of order sets based on actual order history, and informed by scientific guidelines, would have the benefit of rapid development, incorporation of local standards, and greater acceptance by physicians. Ultimately, implementation of efficient and well-accepted order sets should increase the speed of entering orders and help decrease unintended variation in care, thus improving quality and safety.

This study examines current utilization patterns of order sets and “a la carte” orders with a preliminary investigation of methods to automate the creation and modification of current order sets using historical ordering data. This is an early work in an ongoing study to examine the optimization of data and evidence driven order sets in the pediatric setting. Several previous studies from real-world EMR implementations have reported the number of order sets built and used⁹^,¹¹^,¹², and there have been several attempts to create ambulatory order sets or lab corollary orders through data mining techniques¹⁰^,¹³. However, we believe this study to be the first to evaluate the fraction and characteristics of orders entered via order sets and “a la carte”, and the first to comprehensively look at pediatric inpatient orders using extensive historical data and apply advanced machine learning algorithms. We estimate how much efficiency can be gained through modifications of existing sets and creation of new sets by looking at the ratio of items that are used versus not-used from an order set each time physicians open one. Previous reports have created laboratory simulations of ordering efficiency¹⁴^,¹⁵, but this study compares modified and newly created order sets using extensive historical data to evaluate the potential improvements in ordering efficiency. As the main focus in this study is to evaluate machine learning algorithm as a method to generate order sets based on ordering efficiency, extensive clinical validation of order sets created will be conducted in the next phase of the study.

Data and Methods

The study setting is the Children’s Hospital of Pittsburgh (CHP) of UPMC. Since October 2002, all inpatient orders at CHP have been entered directly into the CHP eRecord (Cerner Millenium™). On average, a patient at CHP is hospitalized for 5.5 days, and during that time, 36 unique individuals create 871 order actions. With over 12,000 patients admitted each year, this totals well over 10 million order actions. There are around 2000 departmental order sets designed in the eRecord system designed by CHP. With few exceptions, the system does not force physicians to order items from order sets, so they can also be ordered a la carte. Data on patients with Asthma Minor and Appendectomy Minor diagnoses were extracted from the CHP Clinical Data Warehouse, which accesses an exact copy of the production eRecord. All patient health information was removed to create a de-identified data set for this study (http://privacyruleandresearch.nih.gov/research_repositories.asp). Important variables include order, order action time, careset flag, de-identified patient ID, diagnosis, severity, age at admission, gender, and length of stay in hours. For this study, order action time is limited to the first 24 hours as most order set usage concentrate on the first day of admission. Diagnosis and severity are captured by the drug description and severity of illness using All Patient Refined Diagnosis Related Groups (APR-DRGs). Patient age is categorized into age groups following American Academy of Pediatrics (AAP) Age Groups: 1 = Neonate (< 30 days), 2 = Infancy (>= 30 days and < 1 year), 3 = Early Childhood (>= 1 year and < 5 years), 4 = Late Childhood (>= 5 years and < 13 years), 5 = Adolescence (>= 13 years and <= 18 years), 6 = Adult (>= 18 years). Careset flag is used to identify order set items against a la carte items. Careset flag = 0 means order is a la carte; = 1 means order is an order set parent; = 2 means order is an order set item; and = 3 means order is from an order set nested in another order set. In this study, we only include orders that have careset flag that are 0, 2, and 3.

We use usage rate and coverage rate as two performance measures to study ordering patterns. Usage rate is defined as the total number of items in order set divided by the number of all items ordered for a patient in a time interval. Let the time interval be t, the total number of order set items be o_t, and the total number of a la carte items be a_t. Then,

usage rate at time interval t = \frac{o_{t}}{a_{t} + o_{t}}

Coverage rate is defined as the number of items used from order set divided by the total number of items in the order set for a patient. Let c be a unique order set, p be a unique patient, i_p be the number of unique items prescribed to patient p from order set, and i_c be the total number of unique items in order set c. Coverage rate for patient p using order set c is $\frac{i_{p}}{i_{c}}$ . To compare usage among order sets, we take the average across patients per order set. Therefore,

coverage rate for order set c = \frac{1}{n} \sum_{p = 1}^{n} i_{p} / i_{c}

where n is number of patients on the particular order set, i_p is the number of unique items each patient used from order set c, and i_c is the total number of unique items in order set c. Also, given s order sets,

aggregate coverage rate for s order sets = \frac{1}{s} \sum_{c = 1}^{s} (\frac{1}{n} \sum_{p = 1}^{n} i_{p} / i_{c})

Coverage rate will be used later to compare the performance of current order sets as a whole against that of the order sets generated using machine learning techniques.

Previous studies have explored data mining techniques to automate order set generation process⁹^,¹⁰^,¹⁶, In this study, we will use K-means clustering, a well-known machine learning technique to cluster orders into new order sets that have higher coverage rate than current order sets. K-means clustering is used to identify clusters, a group of nodes with dense connections within groups and sparse connections between groups¹⁷. In our context, a cluster represents an order set, a node is an order, and the distance between two nodes signifies the relevance, or the frequency of cooccurrence of two orders.

Each order may be prescribed alongside one or more orders to a patient with varying age, gender, and treatment locations at different times since admission. Assuming that ordering patterns involve large variability due to differing patient conditions and needs, we first stratify patient population into subgroups using K-means clustering based on age, gender and length of stay (LOS). Then within subgroups and overall, we use order data to find orders that co-occur often, with the notion that orders that tend to be prescribed together to a patient should be grouped into the same order set. We develop and test these options using a modified version of relative risk as the co-occurrence metric to measure the likelihood of two orders being placed together¹⁸. Clustering of orders based on relative risk is subsequently applied to the subgroups and overall population.

For any pair of orders, order x and order y, let U_xy be the number of unique patients who were had both orders, V_x be the number of times order x was placed for patients, V_y be the number of times order y was placed for patients, and N be the total number of patients. Then relative risk, RR_xy, for this order pair is defined as

{Relative Risk}_{x y} = \frac{U_{x y} N}{V_{x} V_{y}}

Given ω number of orders, each order will have w - 1 number of relative risk measures when paired with each of the rest of the orders. K-means clustering is applied on this ω by ω matrix of RR_xy using Euclidean distance. Each order is categorized into an order set without any overlap as a result of clustering. Currently around 30 order sets are used to treat Asthma Minor and Appendectomy Minor patients, respectively. We test different numbers of order sets, from k = 20 to k = 40, to find the optimal order sets that yield the highest aggregate coverage rates. Orders that are clustered into order sets with only one item are treated as standalone orders, similar to current practice. Based on the historical data on patients and orders, we calculate coverage rates as if each order were part of new order sets and compare the new aggregate coverage rates with the ones from current order sets.

Results

Data used in this study are from the first 24 hours after admission of consecutive Asthma Minor and Appendectomy Minor patients discharged from CHP over a 3 month time period. Table 1 describes the patient characteristics. There are a total of 198 Asthma Minor and 91 Appendectomy Minor patients. Average length of stay (LOS) hours are similar, ranging from 20 to 50 hours, although there is one adult patient whose length of stay is significantly larger than the rest.

Table 1.

Patient Characteristics.

Asthma Minor				Appendectomy Minor
Age Group	Sex	Number of Patients	Average LOS hours	Age Group	Sex	Number of Patients	Average LOS hours
Infancy	F	2	30.6	Infancy	F	--	--
Infancy	M	4	21.1	Infancy	M	--	--
EarlyChildhood	F	31	32.6	EarlyChildhood	F	--	--
EarlyChildhood	M	74	32.9	EarlyChildhood	M	6	38.7
LateChildhood	F	22	36.5	LateChildhood	F	10	47.7
LateChildhood	M	45	36.5	LateChildhood	M	38	40.1
Adolescence	F	6	52.0	Adolescence	F	15	43.3
Adolescence	M	7	39.4	Adolescence	M	16	38.3
Adult	F	6	34.4	Adult	F	1	53.0
Adult	M	1	122.3	Adult	M	5	32.5

Open in a new tab

Table 2 presents summary statistics on ordering patterns. Order set items and a la carte items each comprise approximately half of the total orders. Some order items are placed as a la carte orders even though they are part of order sets. This is the reason why sum of unique a la carte items and unique order set items are larger than the total number of order items per patient in both Asthma Minor and Appendectomy Minor patients. On average, each Asthma Minor patient is prescribed 14 a la carte orders that include 9 standalone orders (64%), and each Appendectomy Minor patient is prescribed 21 a la carte orders including 14 standalone orders (67%). Modifying order sets so that more a la carte orders are placed as part of order set items instead should increase ordering efficiency via CPOE. Standard deviation for average number of unique patients per order set is large, indicating that usage rate varies largely across order sets.

Table 2.

Summary Statistics on Orders in the first 24 hours after admission.

	Asthma Minor	Appendectomy Minor
Number of patients	198	91
Average number of total unique order items per patient (sd)	30 (6)	34 (11)
Average number of unique order set items per patient (sd)	19 (5)	16 (7)
Average number of unique a la carte items per patient (sd)	14 (6)	21 (8)
Average number of unique a la carte items that are not part of any relevant order set per patient (sd)	9 (3)	14 (6)
Average number unique order sets per patient (sd)	6 (1)	3 (1)
Average number of unique patients per order set (sd)	39 (57)	9 (14)

Open in a new tab

Figure 1 shows order set usage rate (dotted line) and number of order activities (solid line) at 2-hour intervals for Asthma Minor patients. Except for a peak at time interval 2, and two lower peaks at time intervals 8 and 9, respectively, order set usage rate is less than 0.5, suggesting that there is considerable room for improvement in efficiency through creation of new order sets. The peak in number of order activities is at time interval 8. Order set usage rate and number of order activities seem to follow similar trend after time interval 4, when number of order activities start to increase. Reassuringly, as order activity builds up, the use of order set increases. It may be that increased order set usage leads to more orders being prescribed, or it may be that physicians find order sets facilitating their task of placing orders during busy time intervals and therefore tend to choose to prescribe from order sets. This phenomenon needs to be investigated further in ongoing studies.

Figure 2 shows similar order set usage rate (dotted line) and number of order activities (solid line) at 2-hour intervals for Appendectomy Minor patients. Order set usage is lower for Appendectomy Minor patients compared to Asthma Minor patients, with even the highest usage rate at time interval 7 being lower than 0.5. Similar arguments as above on the relationship between usage rate and number of activities can be made, and lower order activity may explain lower usage rate compared to Asthma Minor.

Figure 3 illustrates the change in average coverage rate by number of items in order sets for Asthma Minor and Appendectomy Minor patients. Number of items in order set follows a wide range from 2 to 96, but coverage rate seems to decrease as number of items increases in an order set. For example, all order sets whose sizes exceed 50 have coverage rates lower than 0.2. This suggests that number of items in order sets may be limited to under 50 to reduce the time physicians spend on de-selecting items off the order sets while prescribing.

Table 3 shows the average coverage rate of some order sets used within first 24 hours for Asthma Minor patients. Both Asthma Minor and Appendectomy Minor order sets had roughly 0.49 coverage rate overall. Larger order sets such as Admission Orders General Pediatric Medical Order Set tend to have lower coverage rate, whereas a smaller order set like MDI with albuterol has coverage rate of 1. This suggests that current, large order sets may be targeting multiple groups of patients with minor differences in their conditions. Modifications should be applied such that order sets target specific subgroups of patients who are prescribed smaller order sets with better-targeted orders. Of course, smaller size does not guarantee high coverage rate. For example, Mechanical Ventilation with 6 items used by Asthma minor patients has a 0.17 coverage rate, which is low for its size. In addition, we find that large order sets are used at earlier stages of hospitalization, when order activity is high but usage drops as order activity slows down, whereas smaller order sets are used throughout hospitalization. We may speculate that using large order sets causes active order activities, possibly resulting in over-prescribing. In redesigning future order sets, order set size may also be reduced to avoid such over-prescribing. On the other hand, all the order sets that have perfect coverage rates consist of 2 items. Since the benefit of order sets come from the fact that multiple orders can be entered at once in CPOE system, it may be reasonable to group some smaller order sets into relatively larger ones in order to maximize the efficiencies gained through order set usage as opposed to a la carte items.

Table 3.

Sample Order Set Coverage Rate for Asthma Minor and Appendectomy Minor Patients.

Asthma Minor			Appendectomy Minor
Order Set	Coverage Rate	Number of Items	Order Set	Coverage Rate	Number of Items
Continuous Neb with Racemic Epinephrine	1	2	MDI with albuterol	1	2
DA with budesonide	1	2	Nasal Cannula	1	2
Heliox Gas	1	2	Urine R&M/Culture	1	2
Nasal Cannula	1	2	PT/PTT	0.83	2
MDI with albuterol	1	2	Admission Orders Idiopathic Intussusception	0.77	13
Pulse Oximetry Continuous	0.99	2	Oxygen Therapy	0.75	2
Oxygen Therapy	0.99	2	Admission Orders Appendicitis Uncomplicated	0.65	14
DA with albuterol	0.96	2	Post Op Orders - Appendicitis Uncomplicated	0.60	15
DA with ipratropium	0.92	2	ED RN Abdominal Pain Orderset	0.60	4
DA with Racemic Epinephrine	0.83	2	Common Lab Stat	0.55	33
Continuous Neb with albuterol	0.53	4	Abdomen US Complete Set	0.54	2
Admission Orders Status Asthmaticus PICU	0.50	27	Abdomen US Limited Set	0.52	2
Restraint(s) for Non-Violent/Non-Self-Destructive (Medical)	0.50	9	Abdomen CT w/ + w/o IV Contrast Set	0.50	5
PFT Order Set	0.50	4	Abdomen CT w/ IV Contrast Set	0.50	4
Restraint(s) for PICU Non-Violent/Non-Self-Destructive	0.50	3	Admission Orders Appendicitis, Complicated	0.48	14
Chest CT w/ IV Contrast Set	0.50	2	Post Op Orders - Appendicitis Ruptured w/Gen. Peritonitis	0.42	24
Admission Orders Asthma	0.38	25	Abdomen CT w/o IV Contrast Set	0.38	4
ED Asthma (Status Asthmaticus) Orderset	0.35	16	Abdomen US w/doppler Set	0.33	3
ED I-STAT 8 Order Set	0.33	3	Post Anesthesia Care Orders - Pediatric	0.29	39
Post Anesthesia Care Orders - Pediatric	0.31	39	Admission Pediatric Surgery/Bowel Prep Orders	0.28	27
PICU Admission - Home Ventilator Patient Order Set	0.27	30	ED Abdominal Pain Work-up Orderset	0.25	28
Admission Orders General Pediatric Medical Order Set	0.18	58	Admission Orders General Pediatric Medical Order Set	0.18	58
Mechanical Ventilation	0.17	6	ED Febrile Infant or Child Orderset	0.17	30
PICU Admission Labs Order Set	0.15	26	Admission Orders General Trauma	0.12	96
Pulmonary Function Test Studies	0.13	16	Admission Orders DKA - ED/Inpatient	0.09	53
Admission Orders Unstable Patient Critical Care	0.12	89	Blood Products	0.05	20

Open in a new tab

Patient subgroups are formed using K-means clustering based on age group, sex, and average LOS. Further clustering on orders is performed on each subgroup separately. In Asthma Minor patients, subgroup 1 has average LOS of 43.4 hours, and subgroup 2 has 35. 3 hours, and subgroup 3 has 64.9 hours. In Appendectomy Minor patients, subgroup 1 has average LOS of 26.35 hours, subgroup 2 has 41.8 hours, and subgroup 3 has 35.1 hours, indicating that even within the same diagnosis group, there are distinct subgroups of patients for whom different order sets may need to be utilized.

Table 6 and 7 compare order sets generated by our clustering approach against the status quo in terms of aggregate coverage rate, number of order items captured as a whole, number of order sets whose coverage rate equals 1 and larger than 0.67 for Asthma Minor and Appendectomy Minor patients respectively. Comparison was performed using all orders, and also orders in the 3 patient subgroups. “Current” indicate status quo order sets, and “New” indicates order sets generated using our clustering approach. All clustering was performed using K-means clustering with relative risk as co-occurrence measure.

Table 6.

Patient subgroup analysis based on Co-occurrence: Appendectomy Minor.

	Overall		Subgroup 1		Subgroup 2		Subgroup 3
	Current	New	Current	New	Current	New	Current	New
Number of order sets	26	41	15	16	22	31	18	19
Number of order items captured	95	226	51	141	83	202	59	151
Aggregate coverage rate	0.49	0.62	0.51	0.70	0.47	0.65	0.51	0.70
One-sided T-test comparing coverage rates current vs. new: p-value with 0.05 sig. level	0.016^*		0.007^*		0.002^*		0.005^*
Number of order sets with coverage rate = 1	3	12	2	6	2	9	3	7
Number of order sets with coverage rate >= 0.67	6	17	3	7	4	13	4	10

Open in a new tab

Statistically significant at 0.05 significance level

Table 7.

Patient subgroup analysis based on Co-occurrence: Asthma Minor.

	Overall		Subgroup 1		Subgroup 2		Subgroup 3
	Current	New	Current	New	Current	New	Current	New
Number of order sets	32	35	25	38	17	17	26	40
Number of order items captured	97	235	75	158	56	80	87	190
Aggregate coverage rate	0.64	0.64	0.65	0.75	0.71	0.78	0.70	0.70
One-sided T-test comparing coverage rates current vs. new: p-value with 0.05 sig. level	0.467		0.096		0.225		0.241
Number of order sets with coverage rate = 1	10	13	8	14	8	6	10	15
Number of order sets with coverage rate >= 0.67	14	15	12	22	10	12	14	22

Open in a new tab

Numbers that are close to current number of order sets were chosen as k in K-means clustering. Hence there is not a large difference between the total number of current and new order sets. Despite this, statistically significant improvements from current to new are seen in the number of order items captured and the aggregate coverage rate for appendectomy minor patient subgroups. In addition, aggregate coverage rate tend to be higher in subgroups than overall group. This suggests that targeted order sets within same diagnosis and severity may help increase order set utilization. In this study, patients are divided into subgroups primarily based on length of stay. When treating patients, if a patient’s LOS can be estimated from historical data, physicians can choose to prescribe from order sets that are meant for short, moderate or long length of stay, respectively. Also, newly generated order sets tend to be utilized more completely (coverage rate = 1) than the current ones. This means that prescribing is done in one mouse-click without any addition or deletion of order items from order sets in more cases using the new order sets. Hence, clustering methods may help target patients more accurately, and increase operational efficiency of CPOE. Figure 4 compares the coverage rates of status quo and new order sets described above for Appendectomy Minor patients, where significant improvement was seen. Each dot represents the coverage rate (y-axis) for order set with a given number of items (x-axis). If there is more than one order set with same number of items, the average coverage rate is used in the graph. Exponential trend line is fitted for status quo and new order sets, respectively. As the graphs indicate, new order sets generally contain less number of items and have higher coverage rates. Also, trend lines are going downwards, reflecting the fact that larger order sets tend to have lower coverage rates.

Figure 4. — Current vs. New Coverage Rates by Number of Items in Order Set: Overall, Subgroup1, 2, and 3

We are not able to verify in depth the order set contents in this study, as it will be the focus in the next phase. However we briefly describe one relatively small order set designed as a result of clustering for Appendectomy minor patients. Our method produced an order set that contains ‘Cardio/Respiratory Monitor (continuous)’, ‘Dex 5% with 0.9% NaCl + 20 mEq/L KCl’, ‘Height’, ‘Magnesium Level’, ‘Phosphorous Level’, and ‘Weight’. Currently ‘Height’ and ‘Weight’ are in the general pediatric admission order set, but we may suggest those two items be included in a specific Appendectomy admission order set. We may also suggest that ‘Dex 5% with 0.9% NaCl + 20 mEq/L KCl’ be included as one of the optional item in the order set because it is frequently used along with other admission orders for Appendectomy patients.

Figure 5 and Figure 6 show the change in aggregate coverage rate and number of order items captured as a whole as the number of unique order set changes using the new order sets. For both Asthma and Appendectomy patients, aggregate coverage rate seems to decrease as number of unique order sets increases. On the other hand, number of order items captured increases as number of unique order sets increases, which seems natural. The pattern of aggregate coverage rate and number of order items captured follow similar trend for Appendectomy Minor patients, whereas in Asthma Minor patients, high coverage rate seem to correspond to low number of order items captured, and vice versa.

Conclusions

Ordering patterns at CHP are explored in this preliminary study with the long-term objective of re-designing order sets that are aligned with current best practices. Order set usage increase with order activities, but there is large variation in terms of order set items that are prescribed to patients. Also, low coverage rate of order set items reflect the fact that physicians often have to go through repetitive mouse-clicks to un-select items out of order sets when prescribing, resulting in inefficiency and non-use. Therefore, a data-driven order set generation and testing approach using historical data was attempted in this study to redesign order sets such that they have high coverage rate and capture as many relevant orders as possible.

We have demonstrated that order set compositions can be significantly improved from current practice using K-means clustering, a simple and widely used machine learning technique. Based on frequency of co-occurrence, groups of orders were identified that tend to be prescribed to similar patients and organized them into order sets. Performance comparisons suggest that order sets generated using clustering can capture more order items than current ones, and with higher coverage rate. Ongoing studies will incorporate temporal and order location factors for more appropriate order set sizing and higher coverage rates as the patient continues to receive care in the inpatient setting. This study was designed such that every order falls into one and only one order set. However, the same orders are often included in multiple order sets that are prescribed at different points in time. Clustering in heterogeneous information networks and hierarchical clustering that allow for overlaps among clusters will be investigated in future studies¹⁹^,²⁰^,²¹. Another important factor is the “must-have” order items. For example, a lab order for vancomycin level is not ordered commonly, but is almost always ordered when the drug vancomycin is first ordered. Prior knowledge about these corollary orders should be built into future order set generation algorithms.

Clinical content of the newly generated order sets was not the scope of this study, but will be investigated in the next stage as part of the order set labeling process. While one physician may feel that a particular drug is “unnecessary” and another physician may consider it as “desirable” when treating the same patient, the objective is to develop a collection of order sets that facilitate fast and accurate ordering while minimizing the challenges associated with explicit selection, de-selection and pop-ups in current CPOE systems. With improved order set quality, physicians will be able to quickly choose orders that are relevant to patient’s needs, leading to improved treatment quality, safety and efficiency.

Table 4.

Asthma Minor Patient Subgroups.

Patient Subgroup 1				Patient Subgroup 2				Patient Subgroup 3
Age group	Sex	Count	Avg LOS (hrs)	Age group	Sex	Count	Avg LOS (hrs)	Age group	Sex	Count	Avg LOS (hrs)
Infancy	F	--	--	Infancy	F	2	30.6	Infancy	F	--	--
Infancy	M	4	21.1	Infancy	M	--	--	Infancy	M	--	--
Early Childhood	F	17	20.0	Early Childhood	F	10	40.4	Early Childhood	F	4	67.9
Early Childhood	M	74	32.9	Early Childhood	M	--	--	Early Childhood	M	--	--
Late Childhood	F	--	--	Late Childhood	F	17	29.4	Late Childhood	F	5	60.7
Late Childhood	M	23	24.7	Late Childhood	M	17	40.9	Late Childhood	M	5	75.6
Adolescence	F	7	39.4	Adolescence	F	3	40.3	Adolescence	F	3	63.6
Adolescence	M	--	--	Adolescence	M	--	--	Adolescence	M	--	--
Adult	F	--	--	Adult	F	5	29.9	Adult	F	1	56.8
Adult	M	1	122.3	Adult	M	--	--	Adult	M	--	--

Open in a new tab

Table 5.

Appendectomy Minor Patient Subgroups.

Patient Subgroup 1				Patient Subgroup 2				Patient Subgroup 3
Age group	Sex	Count	Avg LOS (hrs)	Age group	Sex	Count	Avg LOS (hrs)	Age group	Sex	Count	Avg LOS (hrs)
Early Childhood	F	--	--	Early Childhood	F	--	--	Early Childhood	F	--	--
Early Childhood	M	1	25.0	Early Childhood	M	4	37.6	Early Childhood	M	1	2.4
Late Childhood	F	--	--	Late Childhood	F	10	47.7	Late Childhood	F	--	--
Late Childhood	M	--	--	Late Childhood	M	38	40.1	Late Childhood	M	--	--
Adolescence	F	8	28.3	Adolescence	F	--	--	Adolescence	F	7	60.54
Adolescence	M	9	25.5	Adolescence	M	--	--	Adolescence	M	7	54.7
Adult	F	--	--	Adult	F	--	--	Adult	F	1	2.2
Adult	M	4	26.6	Adult	M	--	--	Adult	M	1	55.9

Open in a new tab

Acknowledgments

This study was designated as exempt by the University of Pittsburgh Institutional Review Board.

References

1.Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010 Aug 5;363(6):501–4. doi: 10.1056/NEJMp1006114. Epub 2010 Jul 13. [DOI] [PubMed] [Google Scholar]
2.HIMSS Analytics: Healthcare IT Data, Research, and Analysis. [homepage on the Internet]. 2012 [cited 2012 Mar 14]. Available from: http://www.himssanalytics.org/hc_providers/emr_adoption.asp.
3.Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998 Oct 21;280(15):1311–6. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
4.Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003 Jun 23;163(12):1409–16. doi: 10.1001/archinte.163.12.1409. [DOI] [PubMed] [Google Scholar]
5.Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):322–9. doi: 10.1197/jamia.M1090. [DOI] [PMC free article] [PubMed] [Google Scholar]
6.McDonald CJ, Overhage JM, Mamlin BW, Dexter PD, Tierney WM. Physicians,information technology, and health care systems: a journey, not a destination. J Am Med Inform Assoc. 2004 Mar-Apr;11(2):121–4. doi: 10.1197/jamia.M1488. [DOI] [PMC free article] [PubMed] [Google Scholar]
7.McAlearney AS, Chisolm D, Veneris S, Rich D, Kelleher K. Utilization of evidence-based computerized order sets in pediatrics. Int J Med Inform. 2006 Jul;75(7):501–12. doi: 10.1016/j.ijmedinf.2005.07.040. [DOI] [PubMed] [Google Scholar]
8.Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003 May-Jun;10(3):229–34. doi: 10.1197/jamia.M1204. [DOI] [PMC free article] [PubMed] [Google Scholar]
9.Wright A, Sittig DF, Carpenter JD, Krall MA, Pang JE, Middleton B. Order sets in computerized physician order entry systems: an analysis of seven sites. AMIA Annu Symp Proc; 2010 Nov 13; 2010. pp. 892–6. [PMC free article] [PubMed] [Google Scholar]
10.Klann J, Schadow G, McCoy JM. A recommendation algorithm for automating corollary order generation. AMIA Annu Symp Proc; 2009 Nov 14; 2009. pp. 333–7. [PMC free article] [PubMed] [Google Scholar]
11.Munasinghe RL, Arsene C, Abraham TK, Zidan M, Siddique M. Improving the utilization of admission order sets in a computerized physician order entry system by integrating modular disease specific order subsets into a general medicine admission order set. J Am Med Inform Assoc. 2011 May 1;18(3):322–6. doi: 10.1136/amiajnl-2010-000066. [DOI] [PMC free article] [PubMed] [Google Scholar]
12.Ali NA, Mekhjian HS, Kuehn PL, Bentley TD, Kumar R, Ferketich AK, Hoffmann SP. Specificity of computerized physician order entry has a significant effect on the efficiency of workflow for critically ill patients. Crit Care Med. 2005 Jan;33(1):110–4. doi: 10.1097/01.ccm.0000150266.58668.f9. [DOI] [PubMed] [Google Scholar]
13.Wright A, Sittig DF. Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system. AMIA Annu Symp Proc; 2006. pp. 819–23. [PMC free article] [PubMed] [Google Scholar]
14.Chan J, Shojania KG, Easty AC, Etchells EE. Does user-centred design affect the efficiency, usability and safety of CPOE order sets? J Am Med Inform Assoc. 2011 May 1;18(3):276–81. doi: 10.1136/amiajnl-2010-000026. [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computerized physician order entry for medication ordering. Stud Health Technol Inform. 2009;150:532–6. [PubMed] [Google Scholar]
16.Hasan S, Duncan GT, Neill DB, Padman R. Automatic detection of omissions in medication lists. J Am Med Inform Assoc. 2011 Jul-Aug;18(4):449–58. doi: 10.1136/amiajnl-2011-000106. [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Lloyd SP. Least Squares Quantization in PCM IEEE Trans Inform Theory. 1982 Mar;28(2):129–137. [Google Scholar]
18.Hidalgo CA, Blumm N, Barabási AL, Christakis NA. A dynamic network approach for the study of human phenotypes. PLoS Comput Biol. 2009 Apr;5(4):e1000353. doi: 10.1371/journal.pcbi.1000353. [DOI] [PMC free article] [PubMed] [Google Scholar]
19.Sun Y, Han J, Yan X, Yu PS, Wu T. PathSim: Meta Path-Based Top-K Similarity Search in Heterogeneous Information Netowrks. PVLDB. 2011 Aug 29;4(11):992–1003. [Google Scholar]
20.Sun Y, Barber R, Gupta M, Aggarwal CC, Han J. Co-Author Relationship Prediction in Heterogeneous Bibliographic Networks. ASONAM. 2011 Jul 25;:121–128. [Google Scholar]
21.Huang J, Sun H, Han J, Deng H, Sun Y, Liu Y. SHRINK: A Structural Clustering Algorithm for Detecting Hierarchical Communities in Networks. CIKM. 2010 Oct 26;:219–228. [Google Scholar]

[b1-amia_2012_symp_1469] 1.Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010 Aug 5;363(6):501–4. doi: 10.1056/NEJMp1006114. Epub 2010 Jul 13. [DOI] [PubMed] [Google Scholar]

[b2-amia_2012_symp_1469] 2.HIMSS Analytics: Healthcare IT Data, Research, and Analysis. [homepage on the Internet]. 2012 [cited 2012 Mar 14]. Available from: http://www.himssanalytics.org/hc_providers/emr_adoption.asp.

[b3-amia_2012_symp_1469] 3.Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998 Oct 21;280(15):1311–6. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]

[b4-amia_2012_symp_1469] 4.Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003 Jun 23;163(12):1409–16. doi: 10.1001/archinte.163.12.1409. [DOI] [PubMed] [Google Scholar]

[b5-amia_2012_symp_1469] 5.Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):322–9. doi: 10.1197/jamia.M1090. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b6-amia_2012_symp_1469] 6.McDonald CJ, Overhage JM, Mamlin BW, Dexter PD, Tierney WM. Physicians,information technology, and health care systems: a journey, not a destination. J Am Med Inform Assoc. 2004 Mar-Apr;11(2):121–4. doi: 10.1197/jamia.M1488. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b7-amia_2012_symp_1469] 7.McAlearney AS, Chisolm D, Veneris S, Rich D, Kelleher K. Utilization of evidence-based computerized order sets in pediatrics. Int J Med Inform. 2006 Jul;75(7):501–12. doi: 10.1016/j.ijmedinf.2005.07.040. [DOI] [PubMed] [Google Scholar]

[b8-amia_2012_symp_1469] 8.Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003 May-Jun;10(3):229–34. doi: 10.1197/jamia.M1204. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b9-amia_2012_symp_1469] 9.Wright A, Sittig DF, Carpenter JD, Krall MA, Pang JE, Middleton B. Order sets in computerized physician order entry systems: an analysis of seven sites. AMIA Annu Symp Proc; 2010 Nov 13; 2010. pp. 892–6. [PMC free article] [PubMed] [Google Scholar]

[b10-amia_2012_symp_1469] 10.Klann J, Schadow G, McCoy JM. A recommendation algorithm for automating corollary order generation. AMIA Annu Symp Proc; 2009 Nov 14; 2009. pp. 333–7. [PMC free article] [PubMed] [Google Scholar]

[b11-amia_2012_symp_1469] 11.Munasinghe RL, Arsene C, Abraham TK, Zidan M, Siddique M. Improving the utilization of admission order sets in a computerized physician order entry system by integrating modular disease specific order subsets into a general medicine admission order set. J Am Med Inform Assoc. 2011 May 1;18(3):322–6. doi: 10.1136/amiajnl-2010-000066. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b12-amia_2012_symp_1469] 12.Ali NA, Mekhjian HS, Kuehn PL, Bentley TD, Kumar R, Ferketich AK, Hoffmann SP. Specificity of computerized physician order entry has a significant effect on the efficiency of workflow for critically ill patients. Crit Care Med. 2005 Jan;33(1):110–4. doi: 10.1097/01.ccm.0000150266.58668.f9. [DOI] [PubMed] [Google Scholar]

[b13-amia_2012_symp_1469] 13.Wright A, Sittig DF. Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system. AMIA Annu Symp Proc; 2006. pp. 819–23. [PMC free article] [PubMed] [Google Scholar]

[b14-amia_2012_symp_1469] 14.Chan J, Shojania KG, Easty AC, Etchells EE. Does user-centred design affect the efficiency, usability and safety of CPOE order sets? J Am Med Inform Assoc. 2011 May 1;18(3):276–81. doi: 10.1136/amiajnl-2010-000026. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b15-amia_2012_symp_1469] 15.Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computerized physician order entry for medication ordering. Stud Health Technol Inform. 2009;150:532–6. [PubMed] [Google Scholar]

[b16-amia_2012_symp_1469] 16.Hasan S, Duncan GT, Neill DB, Padman R. Automatic detection of omissions in medication lists. J Am Med Inform Assoc. 2011 Jul-Aug;18(4):449–58. doi: 10.1136/amiajnl-2011-000106. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b17-amia_2012_symp_1469] 17.Lloyd SP. Least Squares Quantization in PCM IEEE Trans Inform Theory. 1982 Mar;28(2):129–137. [Google Scholar]

[b18-amia_2012_symp_1469] 18.Hidalgo CA, Blumm N, Barabási AL, Christakis NA. A dynamic network approach for the study of human phenotypes. PLoS Comput Biol. 2009 Apr;5(4):e1000353. doi: 10.1371/journal.pcbi.1000353. [DOI] [PMC free article] [PubMed] [Google Scholar]

[b19-amia_2012_symp_1469] 19.Sun Y, Han J, Yan X, Yu PS, Wu T. PathSim: Meta Path-Based Top-K Similarity Search in Heterogeneous Information Netowrks. PVLDB. 2011 Aug 29;4(11):992–1003. [Google Scholar]

[b20-amia_2012_symp_1469] 20.Sun Y, Barber R, Gupta M, Aggarwal CC, Han J. Co-Author Relationship Prediction in Heterogeneous Bibliographic Networks. ASONAM. 2011 Jul 25;:121–128. [Google Scholar]

[b21-amia_2012_symp_1469] 21.Huang J, Sun H, Han J, Deng H, Sun Y, Liu Y. SHRINK: A Structural Clustering Algorithm for Detecting Hierarchical Communities in Networks. CIKM. 2010 Oct 26;:219–228. [Google Scholar]

PERMALINK

Data-driven Order Set Generation and Evaluation in the Pediatric Environment

Y Zhang, MS

JE Levin, MD, PhD

R Padman, PhD

Abstract

Introduction

Data and Methods