Comparison of Ordering Tools on Adherence to Treatment Protocols in the Emergency Department

Abstract

Objective: To assess adherence to clinical protocols in the emergency department after the implementation of embedded order panels in the electronic health record. Materials and Methods: Due to infrequent use, a subset of order sets were redesigned as embedded order panels in a menu-style quick list. Usage was measured before and after implementation at seven departments. Results: There were 2,247 applicable encounters during the pre-intervention period and 1,723 post-intervention. The use of order sets increased significantly after implementation (14% vs. 33% or encounters, p < 0.001). Traditional order sets required at least seven keystrokes or mouse clicks, while embedded order panels required only two. Discussion and Conclusion: Use of order sets increased after implementation of embedded order panels; however, they were still only used for about one-third of applicable encounters suggesting that more work is needed to increase treatment protocol adherence and electronic health record efficiency.

Introduction

There are over 145 million emergency department (ED) visits each year in the United States (US)¹. As the complexity of medicine grows, ED physicians are pressured more than ever to accurately identify and manage life-threatening diseases. Professional liability, reimbursement, surge capacity, disrupted sleep cycles, multiple interruptions, and the variety of patient conditions are some of the challenges facing ED physicians^2-7. Not surprisingly, these challenges can contribute to adverse events that cause considerable harm to patients³. Research has estimated that 53-82% of adverse events are preventable^4-6.

Given these numerous challenges, much work has been done to improve clinical workflows using health information technology. The use of standardized treatment protocols can reduce unnecessary variation in the care of patients with specific conditions, resulting in higher quality care at lower costs.

One method of supporting standardized treatment protocols is implementing order sets in the electronic health record (EHR) as illustrated in Figure 1. Order sets are “predefined lists of steps that should be taken to deal with certain recurring situations in the care of patients⁸.” Order sets contain orders such as laboratory studies, medications, imaging studies, and nursing communication orders. The use of order sets promotes consistency in diagnosing and treating patients with specific complaints or conditions, and has been efficacious in a variety of settings, including chronic obstructive pulmonary disease (COPD), asthma, and sepsis^8-13.

Figure 1.

An example of how an order set used for shortness of breath in the emergency department might look. It contains nursing, medication, and laboratory orders, with some pre-selected.

Background and Significance

Geisinger is an integrated healthcare delivery organization in central Pennsylvania that serves more than 3 million patients. Geisinger adopted the Epic EHR in 1996 and has developed 55 order sets for use in the ED. In 2017, an upgrade to the EHR brought new functionality that allowed ED providers to choose from a pre-set menu of common orders called a quick list. The quick list opened by default whenever a user navigated to the ordering activity. The menu style ordering improved efficiency and satisfaction for providers. As shown in Figure 2, providers could simply click on the orders that were needed instead of manually searching for individual orders.

Figure 2.

An illustration of a quick list in the EHR. The quick list is the default screen in the EHR when providers navigated to the ordering activity. This example contains various sections which often include the most common emergency department orders.

Research on the topic of quick lists is limited. In 2008, Sard et al. evaluated the impact of a quick list on medication prescribing errors in a pediatric ED, concluding that the quick list led to a significant reduction in medication prescribing errors¹⁴. Our organization noted an unintended consequence of implementing this new quick list: the use of order sets significantly declined. Instead of searching for specific order sets, providers would individually select the orders they wanted from the quick list. Consequently, the advantages of order sets in terms of standardization, reduction of unnecessary variation, and increasing use of treatment protocols was put into direct conflict with provider efficiency and satisfaction.

In 2019, Geisinger’s EHR gained functionality to incorporate order sets directly into quick lists. This new functionality, called embedded order panels, allowed providers to select a group of orders using the familiar quick list, menu-style of ordering (Figure 3). Embedded order panels have the potential to realize benefits from both quick lists and order sets. Providers gain efficiency and ease of ordering from the quick list while still benefiting from an evidence-based group of orders organized within the panel.

Figure 3.

An illustration of a quick list incorporating order sets, or embedded order panels. In this example, when a user selects COPD in the Order Sets section, the embedded order panel appears, with the recommended orders pre-checked.

Materials and Methods

This study evaluated the effect adding embedded order panels within quick lists had on providers’ adherence to evidenced-based treatment protocols in the ED. We hypothesized that treatment protocol adherence would increase after the implementation of embedded order panels and tested this hypothesis with a pre-post study design and a comparison group. The study was approved by the Geisinger Institutional Review Board.

The study population consisted of emergency medicine providers who worked in nine EDs. These departments were diverse in size, geographic location, staffing models, patient populations, annual volumes, and available resources. Each department had its own quick list, but all departments shared the same order sets. Embedded order panels in quick lists were implemented at seven of the nine EDs. Two departments, which did not share staff with other departments, served as control sites. The control sites were not educated on embedded order panels but were included in data collection of order set usage.

Five of 55 ED order sets were implemented as embedded order panels on the quick lists of the intervention sites. The order sets were for congestive heart failure (CHF), chronic obstructive pulmonary disease (COPD), psychiatric evaluation, sedation, and sexual assault. The existing order sets were used as a template when creating the embedded order panels so that the tools were consistent. The quick lists for the intervention sites were updated to include a prominent section for embedded order panels.

Providers at the intervention sites received training on the use of the embedded order panels. The new functionality was demonstrated at multiple staff meetings and additional information was sent to providers in multiple bi-weekly email updates.

The new embedded order panels went live at the intervention sites on January 30, 2020. Given the seasonality of some diseases, baseline order set usage was collected between February 1, 2019 and March 15, 2019, and post-intervention usage was collected between February 1, 2020 and March 15, 2020.

To measure the percentage of encounters where order tools were used and to clearly isolate the dataset, the denominator was defined as encounters with specific diagnoses during the study period. For COPD, all encounters with a primary diagnosis of ICD-10-CM code J44, or one of its modifiers, were included. For CHF, all encounters with a primary diagnosis of ICD-10-CM code I50, or one of its modifiers, were included. For sexual assault, all encounters with a chief complaint of sexual assault or a primary diagnosis of ICD-10-CM code T74.2, were included.

Given the volume and variety of primary diagnoses for psychiatric encounters, using the chief complaint was more reliable than ICD codes for defining the denominator for this encounter type. Data for the total number of sedations performed in Geisinger’s emergency departments were collected using specific nursing documentation events that marked the start/stop of a sedation.

Using controlled testing scenarios, the number of clicks and keystrokes were measured for providers using the traditional order sets versus the embedded order panels. Clicks and keystrokes were counted starting from when the provider opened the ordering activity in a patient’s chart to when the orders were signed.

Statistical analyses were performed using Microsoft Excel (Microsoft Corporation, Redmond, WA) and PSPP (https://gnu.org/software/pspp). Odds ratios and p-values were used to investigate differences in order tool use before and after intervention by topic and facility.

Results

Across all sites, there were 37,095 patient encounters during the pre-intervention study period. Of these, 2,247 (6.1%) met the chief complaint and diagnosis criteria for using one of the five order sets. During the post-intervention study period, there were 39,190 patient encounters, with 1,723 (4.4%) meeting the inclusion criteria. The five order sets were used 258 times during the pre-intervention study period at all sites.

At the intervention sites, embedded order panels were used 367 times during the post-intervention period. Overall, there was a significant increase in the use of the order sets after implementation of embedded order panels (Table 1). This was driven by a significantly greater proportion of use of the order set for psychiatric complaints. For CHF, sedation, and sexual assault, there was a non-significant increase in the proportion of visits where order sets were used.

Table 1.

Order tool usage and statistical analyses at all intervention sites. OR = odds ratio. CI = confidence interval.

Topic	Pre-Intervention Use of Order Sets	Post-Intervention Use of Order Sets	OR (95% CI)	p-value
CHF	17/239 (7%)	15/137 (11%)	1.61 (0.77, 3.33)	0.203
COPD	32/247 (13%)	21/181 (12%)	0.88 (0.49, 1.59)	0.675
Psychiatric Complaint	152/864 (18%)	324/667 (49%)	4.42 (3.51, 5.58)	<0.001
Sedation	0/66 (0%)	2/99 (2%)	3.41 (0.16, 72.18)	0.431
Sexual Assault	5/17 (29%)	5/14 (36%)	1.33 (0.29, 6.04)	0.709
Total	206/1433 (14%)	367/1098 (33%)	2.99 (2.46, 3.63)	<0.001

At the control sites, use of order sets was not significantly different between the two study periods (Table 2). Similar to intervention sites, use of the psychiatric complaint order set was most common. In the post-intervention period, the sexual assault order set was used in 100% of applicable encounters.

Table 2.

Order set usage at control sites. OR = odds ratio. CI = confidence interval.

Topic	Pre-Intervention Use of Order Sets	Post-Intervention Use of Order Sets	OR (95% CI)	p-value
CHF	1/84 (1%)	0/84 (0%)	0.33 (0.01, 8.20)	0.677
COPD	0/50 (0%)	0/64 (0%)	0.78 (0.02, 40.15)	0.122
Psychiatric Complaint	46/643 (7%)	32/447 (7%)	1.00 (0.63, 1.60)	0.998
Sedation	5/30 (17%)	5/25 (20%)	1.25 (0.32, 4.93)	0.750
Sexual Assault	0/7 (0%)	6/6 (100%)	195 (3.37, 11285)	0.011
Total	52/814 (6%)	43/625 (7%)	1.08 (0.71, 1.64)	0.710

The minimum number of clicks and keystrokes required by providers to use a traditional order set was seven but could be higher depending on the name of the order set and the required fields within the orders. The minimum number of clicks required by providers to use the embedded order panels was 2. At least 5 fewer clicks were needed to use embedded order panels as compared to the traditional order sets.

Discussion

The evaluation of order sets embedded into quick lists as order panels has not been reported in the literature and offers a valuable research opportunity. Overall use increased when order sets were redesigned as embedded order panels accessed via quick lists. In our study, we found that the increase was driven by higher usage of the psychiatric complaint order tools. This could be explained by the larger population of patients seen for psychiatric complaints as compared to other complaints. If emergency medicine providers are seeing more of a specific complaint, they may be more motivated to use tools that save time for repetitive ordering. Furthermore, there were orders within the psychiatry order set that facilitated rapid communication with other treatment team members, including care management staff. The potential for saving time not only when ordering, but during the entire encounter, may have provided positive reinforcement.

Despite education and socialization of the tools, it was somewhat discouraging to observe overall low utilization of the embedded order panels. Beyond what is discussed in this study, such as the impact of quick lists, other possible causes for lower than expected uptake include poor order set design, lack of training or awareness, and lack of accountability for treatment protocol adherence. Evaluating usage of the order tools, particularly those used for infrequent situations like sexual assault, may warrant additional study.

Other studies have found that unclear prioritization of requests, lack of coordination between teams, and lack of communication between producers and requesters to be root causes in poor order set design¹⁵. Idemoto and colleagues showed that implementing a systematic and cyclic order set review process with defined responsibilities for various stakeholders and formalized communication can significantly improve the quality of order sets, as well as the usage of those tools¹⁵.

There were limitations to this study. Only five of 55 (9%) order sets were implemented as order panels. This limited implementation created fragmented workflows for ordering providers. Additionally, if a provider manually placed orders such that they adhered to the recommended treatment protocols, the analysis employed in this study would not classify the case as being adherent. It was unknown how often this may have happened, but manually placing orders in the EHR required much more time than using ordering tools. While education and training were available, provider behavior can be difficult to change^16-17.

Another limitation of the study was the relatively short period of observation. In the post-intervention study period, this was immediately after the implementation of the tools. Using a longer study period may have afforded providers time to become more familiar with embedded order panels. Alternatively, the lack of utilization of embedded order panels could indicate poor long-term use. Understanding the barriers of accepting new tools may be an area for further study.

The proportion of encounters that met the chief complaint or diagnosis criteria, outlined in the methods section, was significantly decreased in the post-intervention period. The COVID-19 pandemic that occurred during the post-intervention study period impacted the volume of these encounters. Additionally, the change in volume may have reflected organizational efforts to increase the availability of outpatient resources for the specific conditions evaluated. Finally, a recency effect associated with training may have been present in this study. The intervention group received training and additional communications on the importance of using treatment protocols through embedded order panels. The training and awareness alone may have contributed to the increase in order tool usage.

Conclusion

Emergency medicine providers treat patients experiencing a wide variety of disease processes and acuities. Treatment protocols are effective ways to reduce unwanted variation and standardize patient care. Within the EHR, order sets have been shown to improve treatment protocol adherence, as well as patient outcomes. Newer tools, such as quick lists, were designed to improve provider experience and facilitate easier point-and-click ordering. However, unintended consequences of improved provider satisfaction may conflict with organizational priorities of treatment protocol and order set compliance. This study demonstrated that implementing order sets as embedded order panels within the quick list increased treatment protocol adherence for specific use cases while also enabling greater efficiency for providers using the EHR.