Reduction in Clinical Variance Using Targeted Design Changes in Computerized Provider Order Entry (CPOE) Order Sets

Abstract

Objectives

Unwarranted variance in healthcare has been associated with prolonged length of stay, diminished health and increased cost. Practice variance in the management of asthma can be significant and few investigators have evaluated strategies to reduce this variance. We hypothesized that selective redesign of order sets using different ways to frame the order and physician decision-making in a computerized provider order entry system could increase adherence to evidence-based care and reduce population-specific variance.

Patients and Methods

The study focused on the use of an evidence-based asthma exacerbation order set in the electronic health record (EHR) before and after order set redesign. In the Baseline period, the EHR was queried for frequency of use of an asthma exacerbation order set and its individual orders. Important individual orders with suboptimal use were targeted for redesign. Data from a Post-Intervention period were then analyzed.

Results

In the Baseline period there were 245 patient visits in which the acute asthma exacerbation order set was selected. The utilization frequency of most orders in the order set during this period exceeded 90%. Three care items were targeted for intervention due to suboptimal utilization: admission weight, activity center use and peak flow measurements. In the Post-Intervention period there were 213 patient visits. Order set redesign using different default order content resulted in significant improvement in the utilization of orders for all 3 items: admission weight (79.2% to 94.8% utilization, p<0.001), activity center (84.1% to 95.3% utilization, p<0.001) and peak flow (18.8% to 55.9% utilization, p<0.001). Utilization of peak flow orders for children ≥8 years of age increased from 42.7% to 94.1% (p<0.001).

Conclusions

Details of order set design greatly influence clinician prescribing behavior. Queries of the EHR reveal variance associated with ordering frequencies. Targeting and changing order set design elements in a CPOE system results in improved selection of evidence-based care.

Introduction

Computer systems have been utilized to augment the quality of medical care in a variety of contexts. Computer-based interventions have been effective in reducing unnecessary laboratory testing and in improving clinician response to critical laboratory results [1, 2]. In addition, multiple investigators have demonstrated the value of computerized provider order entry (CPOE) and automated clinical event monitoring in the detection and prevention of medication errors [3–5]. Computer applications have also been noted to be effective in assisting clinicians with antimicrobial management, [6] in preventing errors of omission [7] and in improving clinician prescribing behavior [8].

Most CPOE analyses to-date have been from the perspective of safety improvement and error reduction [3, 4, 9, 10]. CPOE systems often include advanced clinical decision support capabilities to further enhance prescribing safety and consistency. These features enable the CPOE system to compare the entered order with the appropriate medication and dose based on the patient’s weight, age, allergies and a number of other factors [11].

Researchers such as Luft and Wennberg have shown that care provided for similar conditions or procedures can vary widely between healthcare providers [12, 13]. Some variance is considered warranted and necessary to meet the care needs of the specific patient or specialized practitioner skills. Unwarranted variance is not uncommon in healthcare and has been associated with prolonged length of stay, diminished overall health and increased cost of care. Wells, et al. noted that children had significantly more frequent asthma exacerbations after hospital discharge, including emergency room visits and readmissions when physicians employed non-standard laboratory and management procedures [14]. Other investigators have noted considerable variance in the treatment of children with bronchiolitis in the United States and in children with asthma in several European countries [15, 16]. Meade, et al. noted wide variance in the management of patients with acute respiratory distress syndrome [17]. This variance was related to limited clinician awareness of relevant research, adherence to local practice patterns and conflicting interpretations of research findings. These authors suggested that clinicians are slow in implementing beneficial care and in responding to new evidence of harm from historic therapies.

Guidelines and protocols are recognized tools for reducing variance in medical care. Bero, et al. suggested that guidelines for care must be implemented in a way to ensure maximum compliance, including the use of interactive education, academic detailing, reminders, audit and feedback and computerized decision support systems [18].

Asthma is a disease which affects approximately 17 million people in the United States and over 7.5% of all children. This disease accounts for over 1.5 million emergency department visits, more than 500,000 hospitalizations and greater than 5,000 deaths annually [19]. Practice variance in the management of asthma is significant and few investigators have evaluated guideline implementation strategies in an effort to reduce this variance [20]. Guidelines and computerized decision support for asthma care have been studied with mixed results in adults and children [21–24]. Sarrell, et al. noted that targeted adult asthma guideline education resulted in modification in care practices in 40% of the studied physician population [25]. In a controlled trial in children, Kwan-Gett, et al. studied the impact of implementing a clinical pathway on the outcome of patients hospitalized with asthma [26]. These authors noted no significant differences in the use of steroids or peak flow meters, average length of stay or total charges as a result of pathway implementation. This study has been criticized in that the interventions utilized were relatively global (including clinical pathway education and training, a flowchart placed on the hospital chart, and nursing documentation of care variance from the pathway). Computerized reminder systems have been shown to be effective in improving clinician compliance with care guidelines. Nilasena, et al. implemented a computer-generated reminder system for diabetes care guidelines in a randomized controlled outpatient study [27]. The authors demonstrated improved compliance with recommended care secondary to facilitated documentation of clinical findings and ordering of recommended procedures. There are no studies which have utilized population-specific care variance data to design targeted workflow-integrated computer-based interventions aimed at reducing care variance in patients with asthma.

Order sets are condition-specific collections of care orders which may exist in either paper or electronic format as part of an EHR. Order sets in CPOE permit consistent initiation of content, guidelines and evidence-based medicine for common illnesses. In addition, order sets likely represent a broad-based tool for reduction of variance in inpatient medical care since most patients admitted to a hospital have their care guided through physician orders [28, 29]. Starmer, et al. noted that 50–60% of orders are placed via order sets, with the remaining orders being generated individually [30]. Order sets also represent a tool for the efficient delivery of disease or procedure-based care orders for busy clinicians. Unlike paper-based order sets, electronic order sets can be linked to supportive electronic resources to enhance safety, efficiency and/or regulatory compliance. Electronic order sets also permit electronic monitoring for compliance and performance improvement purposes [31, 32]. CPOE software systems often provide tools to enrich order set behavior such as the ability to add instructional text, choices to set individual orders within the order set as default on or off, and an option to embed conditional logic (for example “pick one of the following three antibiotics...”) into the order workflow [10, 33].

Chisholm et al studied pediatric asthma care with and without use of order sets in a CPOE system [34]. They studied the use of systemic corticosteroids, metered dose inhalers and pulse oximeter use and found absolute use increases of 19.3%, 7.7% and 11.9% respectively with the use of order sets.

We hypothesized that CPOE order sets could be utilized as a tool to study clinician choice and to reduce unwarranted variance in asthma care thereby improving adherence to evidence-based care. We concentrated on the role of specific design changes within order sets as a tool for focused variance reduction rather than the frequently studied question of whether CPOE or order sets themselves change behavior.

Patients and Methods

Institutional Background

The study was conducted at Cincinnati Children’s Hospital Medical Center (CCHMC), a 423 bed independent not-for-profit facility with 1,045 physicians and 1,750 nurses. CCHMC provides a wide spectrum of care to a diverse socioeconomic patient population. It serves as a primary care facility as well as offering highly specialized services such as organ transplantation, pediatric and cardiac intensive care, and cancer care for children. The hospital is an academic center with pediatric residency and fellowship programs as well as numerous clinical and basic research programs.

Evidence-Based Medicine at CCHMC

At the time of this study there were 29 diseases and conditions with evidence-based clinical practice guidelines and implementation tools at CCHMC [35]. Implementation tools include, but are not limited to, clinical pathways, paper and electronic order sets, algorithms, and education records. The order sets associated with these evidence-based guidelines were available only in electronic format in the EHR at the time of the study.

The Integrating Clinical Information System (ICIS)

ICIS is a web based electronic health record (EHR) at CCHMC which is utilized for inpatient care delivery and documentation throughout the institution. The ICIS core consists of Invision™ components from Siemens Medical Solutions (Malvern, PA) including a web-based portal, CPOE, clinical documentation, electronic medication administration record, results reporting, rules engine and data storage modules. This core system is integrated with multiple proprietary and institutionally developed applications including hospital policies and medication formulary, medical literature search function, evidence based medicine guidelines and implementation tools, a discharge summary system, clinical care rules and other advanced clinical decision support. The system was implemented between April and December of 2002. Over 120,000 orders are entered into the system each month, over 90% directly by prescribing clinicians.

Order Sets

ICIS contains over 500 diagnosis- or procedure-based order sets for children. Order sets are derived from evidence-based guidelines, published literature, consensus-expert groups within the institution or CCHMC practice groups. A formal order set approval, building and testing process is used prior to placing an order set into ICIS. In constructing order sets within ICIS, each individual order is listed in a standard organized fashion (admit, diagnosis, condition, vital signs, activity, nursing, diet, intravenous fluids, medications, laboratory studies, tests).

The order set building tools allow multiple ways of presenting or “framing” of each order. Each individual order default behavior can be set to be “included” (automatically ordered with the set unless unchecked) or “excluded” (requires a specific user check to be ordered). The clinician has the option of not acting on an included care item, in which case it will be automatically included as designed. Similarly, the clinician may utilize their judgment in excluding an included item. Clinicians have been widely educated to view an excluded care item as one which needs to be considered for inclusion for optimal care. For example, in the acute asthma exacerbation order set, all corticosteroid orders (with different dosing forms and routes) are listed as excluded. It is expected that one of these corticosteroid orders is selected and changed to an included item by the prescribing clinician. The acute asthma exacerbation order set is one of the best characterized and studied order sets in use at CCHMC. Acute asthma exacerbation also represents the most common diagnosis admitted to CCHMC with over 1,200 yearly admissions. It is for these reasons that the acute asthma exacerbation order set was selected for study.

Study Protocol

The study was divided into two 3-month time periods; Baseline and Post-Intervention. At the end of the Baseline period, ICIS was queried to determine the number of times the asthma exacerbation order set was utilized. In addition, the rate of utilization of “included” orders, rate of “excluded” order use and addition of orders not appearing in the order set were characterized. Following the Baseline period, individual order use was analyzed and the results presented to the research team. The team examined the results looking for orders with less than optimal performance and discussed opportunities for changes, with the goal of designing and implementing one or more interventions within ICIS or the order set specifically to improve optimal ordering. In general, orders with less than 90% performance were considered suboptimal and targeted for potential intervention. After designing and implementing the interventions over approximately 12 months, a 3-month Post-Intervention period was queried with the same analysis. Other than altering the existing order sets, there were no additional educational efforts associated with the change. The change was not communicated to the end-users.

Queries in the Baseline and Post-Intervention periods were performed by generating a report that retrieved all orders placed up to 24-hours post admission, including all orders prior to admission for all visits that had orders placed from an acute asthma exacerbation order set. Patients were excluded from the study if they did not have either an admission or discharge diagnosis of asthma exacerbation. In addition, patients admitted to the intensive care unit directly or transferred to the intensive care unit within 24-hours were excluded from analysis. Data was exported to an Excel® spreadsheet (Microsoft Corporation, Redmond, WA) and then imported into SPSS 14.0 for windows (SPSS Inc., Chicago, IL) for sorting, cleaning, and analysis.

Statistical Analysis

Previous medical record review data from CCHMC suggested that 91% of clinicians selected the acute asthma exacerbation order set in the care of children with this disorder. However, only 70% of the clinicians utilized the order set intact, with 30% either adding or deleting care items to the admission care orders. We anticipated that this 30% variance from evidence-based care could be effectively reduced to 15% by integrating evidence into the clinician workflow in the ICIS system. The pre-selected order is an indication to the prescriber that potential evidence exists for the use of this orderable. In order to reduce the 30% variance to 15%, assuming a two-tailed alpha of 0.05 and a power of 90%, the use of 161 acute asthma exacerbation order sets required study in the Baseline period and in the Post-Intervention period for a total of 322 order sets. Given an estimated 20% exclusion rate we elected to study a minimum of 400 admissions utilizing the asthma exacerbation order set. Care item performance comparisons between Baseline and Post-Intervention periods were analyzed using Chi-square for categorical variables and the Student’s t test for continuous variables. The Yates correction was used to adjust the Chi-square and therefore the p value for 2 x 2 tables to more accurately reflect the true distribution of the Chi-Square. Values are expressed as mean ± SEM. Significant p values were considered ≤0.05.

Results

The study was conducted between September 2004 and February 2006. During the Baseline period (September 1, 2004 to November 30, 2004) the ICIS query retrieved 286 patient admissions (276 patients) for which the acute asthma exacerbation order set was selected. Forty one admissions were excluded, leaving 245 admissions for analysis. Of the 41 admissions excluded, in 36 the patient was admitted directly to the ICU, in 3 the patient was transferred to the ICU within 24-hours of admission, and in 2 the patients had the diagnosis of asthma, but asthma was not the presenting problem. In the Post-Intervention period, the ICIS query retrieved 235 patient admissions (227 patients) for which the acute asthma exacerbation order set was selected. Using the same exclusion criteria, 22 admissions were excluded leaving 213 admissions (207 patients) for analysis. The demographic characteristics of patients are noted in Table 1. There was an average of 25±3.7 orders placed within the first 24-hours per admission in the Baseline period and 25±4.6 orders in the Post-Intervention period.

Table 1.

Demographic comparison between baseline and post-intervention study periods

	Baseline		Post-Intervention		P
Total Admissions with Order Set	245		213		0.99
Total Patients	240		207		0.81
Patients with Multiple Admissions	4		5		0.56
Gender	n	%	n	%
Male Female	143 102	58 42	125 88	59 41	0.98
Mean Age (years) Age Range (years)	6.1±0.3 0.5 to 21		5.8±0.3 0.3 to 22.2		0.50
Age ≥8 years	68/245	28	51/213	24	0.41

The frequency in which each care item was ordered in the Baseline period and in the Post-Intervention period is noted in Table 2. In the Baseline period, the overall frequency of use of automatically included orders was 96% while the frequency of use of excluded orders was 77%. The frequency of use of most orders in the Baseline period exceeded 90%. Care orders which fell below this 90% utilization frequency threshold were presented to the research team for further consideration.

Table 2.

Selected care item orders from an evidence-based guidelines order set in the baseline and post intervention periods. “Yes” indicates that the care item was automatically included. “No” indicates that the care item was not automatically included. The shaded rows indicate those items which were altered in the intervention period.

Automatically Included	Order Name/Description	Baseline (N = 245)	Frequency	Post-Intervention (N = 213)	Frequency	P
Yes	Admit	218	89.0%	199	93.4%	0.13
Yes	Utilize Evidence Based Guidelines	242	98.8%	208	97.7%	0.58
Yes	Vitals (Temperature, Blood Pressure, Respiration)	234	95.5%	203	95.3%	0.89
Yes	Blood Pressure Daily	232	94.7%	201	94.4%	0.95
No→Yes	Weight	194	79.2%	202	94.8%	<0.001
Yes	Pulse Oximetry	241	98.4%	211	99.1%	0.81
Yes	Start 02 if Sats = 90%, Adjust 91–94%	230	93.9%	201	94.4%	0.98
Yes	Wean 02/MD Order in 1st 4 hrs	239	97.6%	211	99.1%	0.38
Yes	Wean 02/Respiratory Assess after 4 hrs	240	98.0%	208	97.7%	0.92
Yes	Wean Orders Correct per Protocol	239	97.6%	207	97.2%	0.96
Yes	Avoid Strong Odors	242	98.8%	209	98.1%	0.85
Yes	Avoid Known Allergies	242	98.8%	209	98.1%	0.85
Yes	Avoid Visiting Animals	242	98.8%	209	98.1%	0.85
No→Yes	May go to Activity Center PRN	206	84.1%	203	95.3%	<0.001
No	Regular Diet, Encourage Fluids	206	84.1%	175	82.2%	0.67
No→Yes	Peak Flow Pre-Post Treatments	46	18.8%	119	55.9%	<0.001
No	Albuterol	240	98.0%	209	98.1%	0.81
Yes	When Aerosols are Every 3 Hours Change to Metered Dose Inhaler	240	98.0%	211	99.1%	0.56
No	Corticosteroid	236	96.3%	203	95.3%	0.76
Yes	Begin Education about Asthma	240	98.0%	210	98.6%	0.88
Yes	Utilize Discharge Instruction Sheet	239	97.6%	208	97.7%	0.81
Yes	Discharge When Patient Meets Goals and Notify MD	229	93.5%	208	97.7%	0.06

The research team was most concerned about the 79.2% frequency for the “Weight” order, the 84.1% frequency for the “May go to Activity Center PRN” order and the 18.8% frequency in utilizing the “Peak Flow Pre-Post Treatments” order in the Baseline period. According to the evidence-based criteria, peak flow testing is applicable for children >6 to 8 years of age. Therefore, additional analyses were undertaken to examine peak flow order utilization for the 68 study patients >8 years of age (Table 3). Of these patients, 42.7% had appropriate peak flow electronic orders, and 82.4% had peak flow measurements actually performed. In the Baseline period over 40% of the peak flow measurements were performed without an order. Performance of peak flow measurements on any given patient was at the discretion of the respiratory therapist caring for that child. When a respiratory therapist determined that a child was developmentally incapable of performing peak flow measurements, the therapist would document this issue in the medical record and relay this information to the care team. Peak flow measurements were infrequently noted to be performed in the study despite the absence of an order. These measurements were detected through respiratory therapy charting. There were no orders for peak flow testing placed without utilizing the order set.

Table 3.

Peak flow orders baseline and post intervention.

	Baseline		Post Intervention		P
	n	%	n	%
Peak Flow Order Pre-Post Treatments (all age groups)	46/245	18.8	119/213	55.9	<0.001
Peak Flow Order Pre-Post Treatments for Children ≥8	29/68	42.7	48/51	94.1	<0.001
Peak Flow Measurements Performed in Children ≥8	56/68	82.4	49/51	96.1	0.04
Peak Flows Performed with an Order in Children ≥8	26/68	38.2	46/51	90.2	<0.001
Peak Flows Performed without an Order in Children ≥8	30/68	44.1	3/51	5.9	<0.001
Peak Flow Order Pre-Post Treatments for Children <8 Mean Age (years) Age Range (years)	17/177  6.4±0.2  4.7 to 7.6	9.6	71/162  4.4±0.3  0.6 to 7.9	43.8	<0.001 <0.001
Peak Flow Performed n Children <8 Mean Age (years) Age Range (years)	10/177  6.3±0.6  0.9 to 7.6	5.6	9/162  6.9±0.2  5.8 to 7.7	5.6	0.84 0.35

As a result of this Baseline period analysis, the research team decided to implement 3 ICIS interventions. The acute asthma exacerbation order set orders for “Weight”, “May go to Activity Center PRN” and “Peak Flow Pre-Post Treatments” were changed from excluded to included status on November 29, 2005. A repeat ICIS query was performed on March 16, 2006 for the Post-Intervention period from December 1, 2005 to February 28, 2006. There were no significant differences in the demographic characteristics between the Baseline and Post-Intervention groups (Table 1). There were significant improvements in the frequency of orders for these 3 care items in the Post-Intervention period (Table 2). As noted in Table 3, the order set interventions resulted in a significant increase of “Peak Flow Pre-Post Treatments” ordering frequency in all age groups. In addition, there was a significant increase in the frequency of peak flow testing and peak flow testing with an order. The frequency of peak flow testing in children <8 years did not change between the Baseline and Post-Intervention periods. Children <8 years of age who did undergo peak flow testing tended to be older in this age category (mean age of 6.3 and 6.9 years respectively).

Additional orders generated in ICIS within 24-hours of admission that were not included in the order set were also analyzed. During the Baseline period, 36/245 (14.7%) of patient visits had orders for heart and respiratory rate monitoring, all but one in addition to an order for vital signs. In addition, 17/245 (6.9%) of visits had orders for the medication montelukast, and 6/245 (2.4%) had orders for chest radiographs. None of these orders were generated in ICIS within 24-hours of admission in the Post-Intervention period.

Discussion

Variance in health care decision making is widespread. Some of this variance is attributable to genetic and phenotypic variance in the presentation of patients with similar disorders. However, Wennberg has convincingly shown that significant variance is also due to physician behavior [36]. In the inpatient setting, physicians set the stage for the care delivery process when orders are generated to guide patient treatment. Several authors have noted that by combining computerized generation of physician orders with workflow-integrated electronic decision support, physician behavior can be effectively guided [1, 8, 10, 33, 37]. A great deal of research on CPOE has centered on alerts, rules and reminders to care providers, focusing on error prevention and patient safety as defined by error avoidance rather than by broader parameters of system performance and effectiveness. Modern industrial engineering pioneered by Deming has shown that a powerful way to reduce errors is not to focus on outliers but on controlling the central variance [38]. In medicine, CPOE likely represents such a tool to assist in variance reduction.

In constructing CPOE order sets, the designer is faced with many decisions including content, format, sequencing and choices. The manner in which decision choices are framed can have a powerful impact on how decisions are made. There has been considerable work in the last three decades regarding the science of decision making and the shortcuts the brain uses to make decisions (also known as heuristics). Many of these concepts are likely to apply to decision making in CPOE order sets.

Tversky and Kahneman have developed a methodology which notes that decision making does not simply reflect measures of probability and utility but instead depends heavily on how choices are framed [39]. Our research confirms and extends the concept of framing to CPOE order sets. We have demonstrated that framing the same order items differently results in dissimilar clinician decision making behaviors. Tversky and Kahneman have specifically described these decision making shortcuts as a series of heuristics they categorize as representativeness, availability, anchoring and adjustment [40]. Order set design can be analyzed using the availability and anchoring heuristics described by Kahneman and Tversky. Specifically, orders pre-selected to be in an order set can be seen to be more readily “available” than orders that need to be entered individually.

Anchoring and adjustment refers to the notion that a person will start with an implicitly suggested reference point (or anchor) and make adjustments to it to reach their decision. A prescriber may presume that the choices within an order set are based upon evidence and over time he or she will adjust to expect these behaviors. Our work suggests that clinicians will shift their standard of care to the content of the CPOE order set. The pre- and post-test interventions center on the default behavior of orders in order sets. In essence, orders that by default are “included” have higher availability and anchoring than orders that are by default “excluded”. This suggests that default behavior setting in CPOE order sets is extremely important and justifies considerable institutional investment. The ability to use default behaviors in CPOE to migrate clinicians toward evidence-based medicine is likely underestimated as a healthcare opportunity.

In this prospective trial we have shown that the design of CPOE order sets affects their use and the variance in their use. Specifically targeted improvements were noted in the use of three items in an order set by changing the default behavior of those items from “off” to “on”. In the setting of pediatric inpatient asthma care, we have demonstrated that by altering order set choice, weight measurement orders improved by almost 15%, activity center orders increased by 11%, and peak flow measurement orders improved by 40%. Furthermore, in the Baseline period 44.1% of patients ≥8 years of age were receiving peak flow measurements without an appropriate order from a prescriber. In the Post-Intervention period, physician prescribing behavior was significantly improved such that <6% of patients had peak flow measurements performed without an order.

There are several possible limitations to the study conclusions. First, there may have been bias introduced related to the difference in the Baseline and Post-Intervention study time periods. Orders were entered in the Baseline period between September and November. In the Post-Intervention period, orders were entered between December and February. As most orders are entered by house-staff who begin their training in July, it is possible that the training differences in these two study periods may have influenced the results. Second, it is possible that altered prescribing behavior may have been influenced by factors beyond order set design such as prescriber educational sessions, casual conversations regarding care delivery, or repeat admissions and patient/family influence on prescribing. Furthermore, it is possible that respiratory therapists and nurses developed unanticipated biases in the acceptability of certain therapies in the care of children with acute asthma exacerbation. Finally, it is possible that the introduction and adoption of evidence based order sets may result in acceptance of these care delivery orders as complete without the need for the prescriber to contemplate, nor substitute additional orders. Most of these limitations may be avoided in future studies through the implementation of a randomized, controlled study design. The authors are not aware of any additional asthma-specific changes within the organization between the Baseline and Post-Intervention periods that would have biased the interpretation of the data. The study was not focused on disease outcome differences and the design did not include an assessment of clinical measurements such as length of stay or cost of care. Rather, the focus of this study was the impact of order set design on prescriber decision making. Future studies would benefit from the tracking of clinical outcome metrics related to order set design. Though the focus of this study was on children diagnosed with asthma, there are likely many additional patient populations likely to benefit from the approach used in our study.

Conclusion

An organization’s specific approach to framing choice in order set development can have a significant effect on CPOE prescribing behavior and care delivery. Framing of order set choice is an excellent way for a health care system to control care delivery options without loss of physician autonomy in treating biologic patient variability. Organizations implementing CPOE should consider the catalog of CPOE tools which will allow them to frame the environment of order decisions within order sets. Furthermore, health care organizations should develop processes to both study and reduce variance in care delivery through the appropriate design of order set choices.

Conflict of Interest Statement

Donald Rucker, MD is an employee of the Siemens Corporation but does not have any conflicts of interest to report in regards to this research. Brian Jacobs, MD and Kim Ward Hart declare that they have no conflicts of interest in this research.

Protection of Human Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects and was reviewed and approved by the CCHMC Institutional Review Board.

Financial Disclosure

Supported by a research grant from Siemens Medical Solutions

Abbreviations

CCHMC

– Cincinnati Children’s Hospital Medical Center

CPOE

– Computerized Provider Order Entry

EHR

– Electronic Health Record

ICIS

– Integrating Clinical Information System

ICU

– Intensive Care Unit