Abstract
Robust order set catalogs are considered to be a vital part of a computerized physician order entry (CPOE) implementation. Tools and processes for building, localizing, and maintaining these content sets in a centralized repository are important in facilitating the knowledge management lifecycle. Collectively, these order sets represent a significant investment of effort and expertise in capturing and distributing best clinical practice throughout an enterprise. In order to address an important gap of understanding how order sets are both created and used in practice in a current EHR installation, we have developed tools to analyze how order sets are used and customized in clinical practice. In this paper, we present the capabilities of these tools. We further characterize early development patterns in our enterprise order set catalog in early phases of a system-wide vendor EHR rollout. We present data that show how personalized order sets (favorites) are authored and then used in clinical practice. We anticipate that this type of utility will provide useful insight and feedback for those tasked with content governance and maintenance in CPOE systems.
Introduction
Computerized physician order entry (CPOE) systems pose an interesting challenge in clinical knowledge management. Order sets, or standardized groupings of orders targeted to specific clinical conditions or scenarios, have been shown as an important part of user acceptance of CPOE. When physicians use order set templates, they often find personal benefit from the convenience of large groups of clinically-relevant orders being readily available for selection and customization. Organizations can also find benefit, often using order sets as a vehicle to drive standardization processes and reduce unnecessary variation in clinical care.
While electronic health record vendors provide tools and infrastructure for supporting computerized ordering processes, the population of knowledge in the form of order sets is often left to the institution itself. Typical implementations often involve a significant effort or investment in building (or purchasing), refining, localizing, and maintaining order set catalogs 1,2 These content libraries are designed to support routine procedures and common conditions treated by care teams in clinical domains3. Previous research4–9 has detailed many of the challenges associated with knowledge maintenance in order set catalogs, including:
Lack of personnel and resources to build order set libraries at the enterprise level
Immature and poorly-adopted standards for order sets and orderable catalogs impede the effective sharing of order sets from one institution to the next
Building and maintaining order set catalogs requires a sizable investment in time and resources
Implementations often suffer lower uptake of order sets within CPOE due to gaps in content coverage in order sets libraries
Conversion of purchased libraries to local practice requires refinement
Clinical knowledge advances rapidly and the need to update and maintain content within order sets is constant
The various phases of the knowledge management lifecycle are depicted in Figure 1. In preparation for an enterprise installation of a new EHR and CPOE system, we identified that visibility in order set authoring and usage data would be a key gap in our knowledge management processes10 Content authors were quick to indicate their desire for systems that present utilization data as feedback to assist in refining their order sets. Previous research has shown that automated processes for summarizing utilization data can be turned into reporting feedback loops that facilitate content change11–13.
Figure 1.
Knowledge Management Lifecycle. In this cycle, content is created, distributed, and used within a clinical system. Detailed usage data is captured, transformed into specific feedback, and provided back to content owners to facilitate further refinement.
Background
Intermountain Healthcare14 is a not-for-profit integrated healthcare delivery system based in Salt Lake City, Utah. It provides healthcare for the residents of the Intermountain West, including the entire state of Utah and parts of southern Idaho. Intermountain maintains a health insurance plan, 22 inpatient hospitals (including a children’s hospital, an obstetrical facility and a dedicated orthopedic hospital), more than 185 outpatient clinics, and 18 community clinics serving uninsured and low-income patients. Collectively, it provides primary and specialty care for nearly half of the residents of the state of Utah. Informatics teams provide support to clinical users in the form of clinical decision support, clinical knowledge management, content maintenance and software development for unmet clinical needs.
In late 2013, Intermountain Healthcare and Cerner Corporation announced a strategic partnership, including an enterprise rollout of the Cerner EHR platform, and a joint development effort aimed at delivering novel informatics solutions for healthcare organizations. Implementation of the system is still underway, with several sites in the enterprise currently live on the new software. One of the strategic priorities of Intermountain’s Cerner installation focuses on implementing and measuring utilization patterns of best practice care process modules into the system, in the form of order sets, decision support rules and care pathways. In order to address the need of understanding and reporting how knowledge content input into the system is actually being used we set about to build an interactive dashboard that illustrates the usage of order sets, personalized order sets derived from them (hereafter referred to as ‘favorites’) and how collective differences in the composition of favorites, in concert with utilization data can drive suggestions for optimizing order sets.
Methods
Working with technical staff and data analysts, we extracted data from relevant tables in our production Cerner Millennium database environment and moved the data to Intermountain’s data warehouse. For the purposes of development, approximately 12 months of usage data were extracted (February 2015 to February 2016). This resulted in data derived from nearly 300,000 order set usage instances. Filtering this data to exclude data derived from technical staff and/or testing data reduced the data set by approximately 20%.
Following several sessions with clinical sponsors focused on design and requirements definition, we used Tableau software to prepare an interactive dashboard that presents visual depictions of order set usage. Figure 2 shows the main view from this dashboard, in which all order sets in the catalog are visible, ranked by overall usage, with two main color depictions illustrating overall ordering volumes from the standard template, as well as the collective favorites that were derived from it.
Figure 2.
Main view of order set utilization dashboard, showing content, usage data, filters, and order volume derived from the main template and its associated favorites. This specific view is focused on order sets in the Critical Care (CC) domain.
Figure 3 presents a graphical view of overall usage of an order set over time, also with baseline template and favorite ordering volumes depicted. Dynamic filters allow the user to focus strictly upon specific regions, facilities, providers, roles, or timeframes of interest to the user. Drill-down capability allows the user to select portions of the display and view only the data that pertains to the selection itself.
Figure 3.
Time-based view of ordering patterns for a specific order set (CC General ICU Admission) in the Intermountain Healthcare order set catalog. Version, order volume, order session type, and associated dynamic filter sets are shown.
Figure 4 shows the ‘phylogeny’ of an order set, including timestamps, authors, and difference counts across the main versions of the order set itself. It further shows detail about the ordering physicians that have taken time to create favorites against the template, including timestamps, names, counts and drilldown views into the changes made in the favorite, and data regarding the number of times in which the favorite was used. At a high level, it illustrates the branching and evolution that can occur in the main order set template and the favorites derived from it.
Figure 4.
Version and Favorites Tree for the CC General ICU Admission Order Set from the Intermountain Healthcare order set catalog. This view shows the version history, favorites branches, as well as the corresponding timestamps, authors (names and roles), number of ordering instances, and modification counts derived from each knowledge artifact.
Figure 5 depicts a line-item view into the details of an order set. When set to show all versions, the visual cue of color blocks next to each of the orders demonstrates the presence or absence of an order from a particular version of an order set. When viewed against the entire version history of the same content (or against favorites derived from the template) the color patterns show the evolution of the content itself, including orders that are added, removed, or updated over time. The color of the blocks themselves represent selection status in this view, indicating whether the order was preselected or not. Numbers across the top show the various versions of the content itself, as well as corresponding ‘denominators’, which indicate the number of times in which that particular version of an order set was ordered against. Numbers overlaying the orderable blocks themselves show the frequency with which particular orders were ordered against in the ordering sessions that used these order sets.
Figure 5.
Line Item view inside the CC Sepsis ICU Admission Order Set. This view shows each order within the order set and depicts its presence/absence in the version history, as well as actual ordering data that corresponds with each element. Corresponding ‘denominators’ are found in the column headings that match each version.
This dashboard utility and these specific visualizations have allowed us to compile useful information about the scope of our current order set catalog and the extent to which it has been revised and used. It has also given us new insight as to the frequency with which our users are using personalized order sets to find, tailor, and adjust the enterprise order sets themselves. The insight is useful both in comparisons of templates to their derived ‘favorites’ collections, as well as across versions of the enterprise order set template itself.
Results
Our efforts in extracting these data sets and consolidating them have allowed us to summarize and describe the artifacts in the order set catalog itself. T able 1 contains data pertaining to the main order set templates in the library itself, their usage to date (with the understanding that our enterprise installation is still in early phases), and the number of authors responsible for the content. It shows that less than half of the available enterprise order sets available in the order set catalog have been used to date.
summarizes similar data for favorites saved to the order set catalog as well as corresponding usage data. This table contains two fewer order set types, as users do not have the ability to set favorites on future (day of treatment) and phased order sets in the Cerner system. It also shows how many order set templates have been ‘favorited’ and the average number of favorites per ‘favorited order set’. This table also shows the number of ‘cloned’ favorite order sets, in which the user has personalized the order set in name only (the content of the personalized order set does not differ from that of the template from which it was derived).
Figure 6 depicts the additions to the collection over time as authors added to and updated existing content in the set. It shows additions to the order set catalog in the year prior to the initial go-live in Intermountain’s Cerner installation (February 2015). Favorites are shown in orange, while main templates and revisions are shown in blue.
Figure 6.
Content additions to the Intermountain order set catalog (consisting of both enterprise order set templates in blue and the entire personal favorites catalog in orange) in 2014 through early 2016. Enterprise order sets and revisions are shown in blue, personalized order sets or ‘favorites’ are shown in orange.
Figure 7 shows the overall volumes of orders derived from order sets coming into the database from the initial enterprise installation to present (February 2015 to February 2016). This graph, combined with the data shown in
Figure 7.
Order Set usage data since initial Cerner implementation at Intermountain Healthcare. Order sets entered via favorites are shown in orange, enterprise-level content is shown in blue.
Table 1 and 2 shows that despite the large numbers of favorites available in the Intermountain order set catalog, most of the order volume derived from CPOE is coming from enterprise template order sets, not favorites. Growth in the volumes shown in the order curve are derived largely from new sites and regions coming online with the new Cerner EHR system.
Table 1.
Summary data for enterprise order set template data in Intermountain’s order set catalog
| TEMPLATE DATA | ||||
|---|---|---|---|---|
| Order Set Type | Care Plan | Phased | Pathway | Future Encounter |
| Order Sets available | 2,516 | 4,366 | 758 | 4,838 |
| Used – to date – | 970 (39%) | 1,725 (40%) | 386 (51%) | 390 (8%) |
| Not Used – to date – | 1,546 (61%) | 2,641 (60%) | 372 (49%) | 4,448 (92%) |
| Total # of Usage Instances | 131,498 | 137,869 | 3,134 | 4,785 |
| Grand Total of Usage Instances | 270,118 | |||
| Unique Content Authors | 107 | |||
Table 2.
Summary data for personalized or ‘favorites’ order set data in Intermountain’s Cerner order set catalog
| FAVORITES DATA | ||
|---|---|---|
| Order Set Type | Care Plan | Pathway |
| Favorite Order Sets available | 8,787 | 6694 |
| Derived from unique order sets | 693 | 275 |
| Average # of favorites per ‘favorited template’ | 12.7 | 24.3 |
| Used – to date – | 3282 (37%) | 3307 (49%) |
| Not Used – to date – | 5505 (63%) | 3387 (51%) |
| Total # of Usage Instances | 22,670 | 15,773 |
| # that are exact copies of template (clones) | 954 (11%) | 509 (8%) |
| Unique Content Authors | 1424 | |
Discussion
We have constructed tools that allow order set content owners to quickly identify their content, view its utilization over time, and understand specific details of how, where, and who is using the content. Further, material views make it relatively straightforward for content owners of order sets to understand how their content is being used. As we have developed and presented these views back to the clinical teams responsible for the content, they have expressed both interest and enthusiasm in being able to quickly visualize how and where their content is being used. Several key themes have emerged as we have reviewed this data and presented it back to users.
Departmental differences over Favorites
Different departments and clinical programs inside Intermountain Healthcare have strikingly different views on the place of personalized order sets within a CPOE system. Some clinical groups (like cardiology) are patently opposed to users utilizing them and have configured the system to disallow specific user roles from creating them. Other departments are much more relaxed in this regard and go as far as encouraging them and training users how and when to go about it. As a results of these differences, departmental level views of order set utilization can be very different, with cardiology-specific content showing almost no utilization of favorites, to the other extreme in obstetrics/gynecology where over 70% of postpartum discharge orders are entered via favorites. Despite the difference in philosophy or approach on personalized order sets, both departments have expressed value in being able to visualize the usage patterns of their content.
Namespacing or ‘initialing ‘ of content
As shown in Table 2, between 10 and 15% of all favorites in the Intermountain catalog are exact copies of the enterprise order set from which they were derived, or ‘clones’. In many cases, the only material difference between these personalized order sets and the templates from which they were built is the name. Many users applied a namespacing approach in which they prepended or appended their initials to the title of the order set. Others renamed them to facilitate searching using terms they would think of when using the search mechanism of the Cerner CPOE system. Due to the updating features of the Cerner EHR system, these users’ content will remain consistent with the enterprise template as it evolves.
Content gaps
The phylogeny-based view of the order set content also revealed interesting perceived gaps in the order set catalog itself. Simple inspection of the obstetrics/gynecology order set data showed large amounts of favorite use in the postpartum discharge order sets. Upon inspection of the favorite sets built against this template, key themes emerged from the order set titles, in particular the groundswell of desire from the users to separate the content into separate groupings of ‘C-section discharge orders’ and normal or vaginal delivery C-section orders. When content authors originally built this order set, they had grouped these two use cases (post C-section discharge orders and normal vaginal delivery orders) into one order set intended to support both. Yet the users’ creation and subsequent use of these two separate ‘favorites-based’ sets indicates a strong preference for separate content to support both scenarios. This type of feedback allows content owners to revisit and potentially re-prioritize content development efforts.
Unused content
In both the enterprise order set and favorites libraries there were significant numbers of order set templates that have not yet been used. This is not entirely unsurprising, as Intermountain is still relatively early in its implementation of the Cerner EHR system. The significant number of unused ‘favorites’ is somewhat more difficult to characterize, though some of that may be due to content created by physicians during their training with the software or time with physician coaches who are orienting them to the features present in software that is unfamiliar to them.
Over time, this usage data may bring to bear content that is present in the library, but semi-obsolete from a utilization perspective. Seldom-used order sets may be considered for deletion from the library in such circumstances, so as to optimize the users experience with the software, as well as to minimize the already significant maintenance costs that come with keeping them current with best practice and standards given in the medical literature.
Some clinical groups have been a bit surprised by the utilization data, in that they have in some cases created content that is rarely used. Some clinical workflows and even training routines that are part of implementation favor other methods for ordering, including quick orders, workflow-driven wizard screens and other modules. In some of these cases, content owners have shifted some of their focus away from order set development for all scenarios, targeting instead those where they feel that other options don’t address the clinical users’ needs adequately.
Ownership/governance
Tables 1 and 2 illustrate the scope of authors required to create the order set library that we are characterizing. It details that there are just over one hundred content authors who maintain the enterprise order set library and well over 1400 individuals that have personalized order sets. At Intermountain, the order set creation and update effort is spread across clinical programs at the corporate level, with regional representation from the various sites/regions and input from clinical, administrative, and central groups (e.g. compliance, legal, etc). It also has input from clinical service lines, local units, physician champions and specialists. Clearly there are many other individuals beyond the 100+ specific content authors who are expending significant effort in curating/refining content in the enterprise content development teams. The overall effort necessary to build these libraries is significant.
Order volume
Figure 7 shows that most of the order volume derived from order sets does indeed come from enterprise-level order sets, and not from personalized ones. More than 80% of the volume collectively comes from the standard content.
There are clear exceptions to this, as noted above, and these differences may reflect different philosophical approaches in clinical divisions, content gaps and/or other reasons specific to the users. Our hope is that presenting these data to content stewards can stimulate meaningful conversation amongst authors and users of these knowledge artifacts to optimize users’ experience with the order sets.
Limitations
We have not looked at complete ordering patterns within the Cerner toolset, only those derived from order sets. Quick orders and other workflows support ordering and encompass the broader picture of medical orders in the clinical setting. Our analysis has not yet extended itself to differences in order sentence construction. We have also not yet undertaken the visual analysis of phased or component order sets which are common to multiple parent or pathway order sets.
Next Steps
We intend to extend the capabilities of the visualization toolset to drill down to the details of the orderable/order sentence level, further illustrating the specifics of how the orderables are being used. We also plan to work with clinical analysts from each of the clinical specialties to begin to attach cohort specifications as an applicable filter for viewing order set utilization data (e.g. to show usage of the febrile infant order set specifically for patients who met the febrile infant definitions from the clinical data).
Conclusion
We have constructed tools that allow order set content owners to quickly identify their content, view its utilization over time, and understand specific details of how, where, and who is using the content. Materializing this information visually has illustrated some key themes of content usage, needed areas for refinement, and greater visibility into the actual consumption of knowledge content produced by central content teams. We anticipate that this vision into the data will lead to optimization and enhancement of content over time.
Acknowledgements
We acknowledge the Cerner technical engineers who have played an important role in orienting us to the nuance of the clinical data used in the analysis. We likewise express appreciation for the support from Intermountain Healthcare’s enterprise data warehouse team. We thank Sameer Badlani, former CHIO at Intermountain, for his inspiration to begin the effort.
References
- 1.Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and Use of Preconstructed Orders, Order Sets, and Order Menus in a Computerized Provider Order Entry System. Journal of the American Medical Informatics Association: JAMIA. 2003;10(4):322–329. doi: 10.1197/jamia.M1090. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Wright A, Sittig DF, Carpenter JD, Krall MA, Pang JE, Middleton B. Order Sets in Computerized Physician Order Entry Systems: an Analysis of Seven Sites; AMIA Annual Symposium Proceedings; 2010. pp. 892–896. [PMC free article] [PubMed] [Google Scholar]
- 3.Durieux P, Trinquart L, Colombet I, Nies J, Walton R, Rajeswaran A, Rege Walther M, Harvey E, Burnand B. Computerized advice on drug dosage to improve prescribing practice; Cochrane Database Syst Rev; 2008. 3 pp. [DOI] [PubMed] [Google Scholar]
- 4.Wright A, Feblowitz JC, Pang JE, et al. Use of Order Sets in Inpatient Computerized Provider Order Entry Systems: A Comparative Analysis of Usage Patterns at Seven Sites. International journal of medical informatics. 2012;81(11):733–745. doi: 10.1016/j.ijmedinf.2012.04.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Wright A, Sittig DF. Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system; AMIA Annu Symp Proc; 2006. pp. 819–23. [PMC free article] [PubMed] [Google Scholar]
- 6.Jacobs BR, Hart KW, Rucker DW. Reduction in clinical variance using targeted design changes in computerized provider order entry (CPOE) order sets: impact on hospitalized children with acute asthma exacerbation. Appl Clin Inform. 2012;3:52–63. doi: 10.4338/ACI-2011-01-RA-0002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Zhang Y, Levin JE, Padman R. Data-driven order set generation and evaluation in the pediatric environment; AMIA Annual Symposium Proceedings; 2012. pp. 1469–78. [PMC free article] [PubMed] [Google Scholar]
- 8.Chan J, Shojania KG, Easty AC, et al. Usability evaluation of order sets in a computerised provider order entry system. BMJ Qual Saf. 2011;20:932–40. doi: 10.1136/bmjqs.2010.050021. [DOI] [PubMed] [Google Scholar]
- 9.Munasinghe RL, Arsene C, Abraham TK, et al. Improving the utilization of admission order sets in a computerized physician order entry system by integrating modular disease specific order subsets into a general medicine admission order set. J Am Med Inform Assoc. 2011;18:322–6. doi: 10.1136/amiajnl-2010-000066. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Fischer G, Ostwald J. Knowledge management: problems, promises, realities and challenges. IEEE Intell Syst. 2001;16(1):60–72. [Google Scholar]
- 11.Hulse NC, Del Fiol G, Bradshaw RL, et al. Towards an on-demand peer feedback system for a clinical knowledge base: a case study with order sets. J Biomed Inform. 2008;41:152–64. doi: 10.1016/j.jbi.2007.05.006. [DOI] [PubMed] [Google Scholar]
- 12.Matheson NW. Things to come: postmodern digital knowledge management and medical informatics. Journal of the American Medical Informatics Association. 1995;2(2):73–78. doi: 10.1136/jamia.1995.95261908. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Zhang Y, Padman R, Levin JE. Paving the COWpath: data-driven design of pediatric order sets. J Am Med Inform Assoc. 2014 Apr. doi: 10.1136/amiajnl-2013-002316. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Intermountain Healthcare. 8 Mar 2016. www.intermountainhealthcare.org Accessed on.