Improving Computerized Provider Order Entry (CPOE) Usability by Data Mining Users’ Queries from Access Logs

Abstract

This study describes and evaluates a technique for using clinicians’ search preferences to reduce the time to query and select orders. A relational database was created to store user keystroke logs with the timing of each user activity. The most frequently selected orders associated with the popular query phrases were distilled from three months of data. The resulting table of popular queries and selected orders was incorporated into the order entry system so the most favored orders were placed at the top of the pick-list. A retrospective evaluation indicated the method significantly reduced orderable selection time (16.3% reduction for optimized queries using the method versus 5.7% for un-optimized queries).

INTRODUCTION

Among the various features that have been identified to make successful implementation and deployment of CPOE systems are:

1. The time it takes for a physician to enter an order^1–4.

2. The ability of the system to accommodate house staff preferences^2,5.

3. The ability to present the user only with essential information and usage of consistent terminology³.

The cornerstone of order entry activity is selecting the clinical concept to order⁶. WizOrder⁷, a computerized clinician order entry system developed and maintained by Vanderbilt University Biomedical Informatics faculty, staff, and trainees, uses a “completer” function to allow the user to search for orderables while other applications use menus and tabs to hierarchically organize orderables^7,8. The completer takes in as input a query string entered by the user. Incomplete terms in the query are completed using a dictionary of commonly used words. Then, a database of orders is searched for the close matches of each individual word in the query. A list of matched orders from the database is generated, and they are scored based on how well they matched the query string. The list is sorted and presented to the user. Figure 1 shows an example of a user entering “disch” into the completer and displays the list that is returned where ‘1.discharge’ is the highest ranked, ‘2.discharge wizards’ is second highest, etc.

Figure 1.

“Completer” search and results.

The order rankings returned by Wizorder are not always consistent with the desires of the users. That is, in some cases, when users query for Q, the completer returns a list of orders, O₁,O₂,O₃…O_n, that matches Q. However, users may frequently select orders from the bottom of the list, O₉ or O₁₂, where n = 15. On a typical clinical workstation, the user must scroll to see more than thirteen terms in the upper right pane. The act of scrolling and reading the list may take a few seconds depending on the length of the list. Since approximately 6,000 queries are made daily, even small reductions in individual search and order selection time result in a significant savings. We hypothesized that modifying the search algorithm to learn users’ preferences over time, and taking them into account when sorting the matched orders would have a significant, measurable impact on order selection time.

METHODS

Our approach was a two-step process in which (1) user preferences were learned and (2) the knowledge gained was used by the completer to sort the list returned respectively⁹.

A relational database was created to store and transform data from WizOrder user activity log files⁷. Figure 2 illustrates an example of a user querying “ns” and then selecting the “NS: SODIUM CHLORIDE 0.9%” orderable. Notice that user input is contained within {} braces and the system interpretation is contained in [] brackets. The time from query to response in this example was four seconds and the selected orderable was 2^nd in the returned list as denoted by {2}.

Figure 2.

Query and selected order from a log file

For the research outlined in this paper, a relational model between the user, the order entry session, the query, and the selected order was created. The most popular queries and their top selected orders were extracted from the database for a three-month period. A query was considered “popular” if it occurred more than five times in the three-month period. The five most frequently selected orders, or preferred orders, were obtained for each query phrase. The resulting query/preferred order(s) relationships were then saved to a popular queries file that is read into the WizOrder server as a hash table at startup. The choices regarding popular query definition and the number of selected orders to include were driven by a desire to preserve completer performance and minimize the method’s memory footprint inside the WizOrder server.

Currently, the popular queries file contains 4386 queries and 6137 orders (2157 distinct orders) and requires less than 50 kilobytes in memory. Table 1 presents the top 25 queries in the popular queries file, including their frequencies and percentages. Table 2 illustrates the preferred orders for the top two queries. The completer algorithm was modified to place the preferred orders from the popular queries file at the top of the ranked list of orderables to be presented to the user.

Table 1.

Top 25 user queries and their frequency.

Query Phrase	Frequency	Percentage (%)
BMP	16078	3.11
CBC	11887	2.30
NURSING	10965	2.12
CXR	10024	1.94
DISCHARGE	7179	1.39
MORPHINE	5110	0.99
ADMIT	4730	0.92
TYLENOL	4420	0.86
LASIX	4409	0.85
PCV	4377	0.85
NPO	4173	0.81
KCL	3969	0.77
LORTAB	3744	0.72
NS	3711	0.72
EKG	3500	0.68
ABG	3186	0.62
PT	3061	0.59
UA	2944	0.57
LABS	2803	0.54
PEPCID	2717	0.53
PHENERGAN	2618	0.51
CMP	2611	0.51
TRANSFER	2593	0.50
ATIVAN	2437	0.47
ALBUTEROL	2417	0.47

Table 2.

Top two queries and their preferred orders.

Query	Preferred Orders	Frequency	%
BMP	LYTES,CREAT,BUN,GLU	6904	43.0
	LYTES,CR,BUN,GLUC,W/ CA	6642	41.3
	ADMISSION LYTES,CREAT,BUN,GLU	719	4.47
	ADMIT LYTES,CR,BUN,GLUC,W/ CA	683	4.25
	VBG RESP	33	0.20
CBC	CBC / PLT CT	5777	48.6
	CBC / PLT CT / DIFF	4041	34.0
	COMMON LABS	31	0.26
	PCCU COMMON LABS	17	0.14
	PCCU HEMATOLOGY & CLOTTING LAB	12	0.10

The database was used to evaluate the method’s impact on search time, defined as the time in seconds that it takes a user to search for a query and select an order. Queries where the search time exceeded three standard deviations were excluded from analysis on the assumption that the lengthened response was most likely due to interrupted workflow.

Development of the database, the popular queries file, completer modifications, and testing took place between May 2004 and February 2005. To avoid seasonal variations, such as new interns arriving each July, the method was evaluated by comparing the search times for three months post-implementation (February 19 through May 19, 2005) against the same three months in the preceding year (February 19 through May 19, 2004). The number of orderable concepts in the system grew from 6,703 to 7,006. There were 4,368 distinct users in the pre-intervention period and 4,947 post-intervention.

The queries from each period were stratified into two categories: for the pre-intervention period, the queries were classified into potentially optimized and un-optimized groups. The potential optimized queries were those that would have qualified for entry into the popular queries file. For the post-intervention period, the queries were grouped into truly optimized and un-optimized categories. The optimized queries in this case were those that qualified and were actually included into the popular queries file.

RESULTS

Table 3 contains the result statistics for period, category, and includes the excluded queries for comparison. The average search times for excluded queries are consistent with interrupted workflow because it would be unusual for a user to be at the workstation for over two minutes without activity.

Table 3.

Search times (seconds) pre and post intervention for optimized and un-optimized queries.

	All Queries			Excluded Queries			Studied Queries
PRE INTERVENTION	# queries	avg	stddev	# queries	avg	stddev	# queries	avg	stddev
Potentially Optimized	386772	6.05	21.54	4462	146.03	128.99	382310	4.42	6.64
Potentially Un-optimized	26434	9.63	29.32	407	184.89	131.89	26027	6.89	10.64
POST INTERVENTION
Optimized	459362	5.19	19.59	5196	136.23	116.58	454166	3.7	5.81
Un-optimized	27313	9.16	28.12	432	174.55	125.17	26881	6.5	10.23

Figures 3 and 4 graph the search time distributions for the various query categories pre and post intervention. The reduction in search times of the optimized queries post-intervention is supported visually by the strong shift towards the left and the reduced tail of the graph. Figure 4 illustrates that un-optimized queries have a greater spread in search times.

Figure 3.

Pre and Post-Intervention search times in seconds for potential and actually optimized queries respectively.

Figure 4.

Pre and Post-Intervention search times in seconds for potential and actually un-optimized queries respectively.

The Wilcoxon rank-sum test was used to determine statistical significance. The reduction in search times was highly significant (p < 0.0001) for optimized and un-optimized queries. However, the magnitude of the reduction is much higher for the optimized queries (16.3% for optimized versus 5.7% for un-optimized).

DISCUSSION

If the 0.72 second average reduction in search time is extrapolated over a year, query optimization may optimistically save clinicians 367 hours annually. However, there was an unexpected decrease in search time for un-optimized queries. This may have been due to a hardware upgrade of the WizOrder servers that occurred between the pre- and post-intervention periods, the lifecycle replacement clinical workstations, or possibly network improvements. If the 0.39 second reduction in un-optimized search time is assumed to be due to technical enhancements that also benefited the optimized queries, then a 0.33 second reduction could be attributed to the method described. Conservatively, the potential annual savings would be 168 hours.

As this was a first attempt to systematically reduce search time based on activity data, the method used a simple knowledge representation and studied its impact. Since the method appears effective, there are several avenues for improvement.

Tuning the existing method

The criteria for a query and its preferred orders’ inclusion in the popular queries file could be adjusted and studied. Our definition of “popular”, selected more than five times in the past three months, was not based on rigorous analysis or experimentation. The current method allows adjustment by:

1. Changing the amount of user activity log data that is mined

2. Increasing the number of preferred orders that are stored with each query

3. Altering the number of query phrases included in the popular queries file.

Utilizing patient and user context

The patient’s age, location, admission service, primary diagnosis, and a variety of other factors could be used to improve order selection. WizOrder currently uses patient age and location to remove some orderables from the completer results. For example, certain forms of ferrous sulfate are restricted from use in the neonatal intensive care units. In another example, Enfamil is only used as a diet in pediatric patients. However, the idea of integrating patient specific context with knowledge discovered from activity data is unexplored. Both the patient’s admission service as well as the clinical service of the user may enhance the order selection process. There may also be seasonal variability in preferred orders. Orders related to treatment of influenza and bronchiolitis could be preferentially ranked during winter months. Calculation of the popular queries file may require seasonal adjustment.

Influencing order selection

Enabling the completer to use prior knowledge of user preferences is just one approach to guiding the user during order entry. Our experience managing the order entry system suggest there are four categories for guiding and influencing the user during search.

Technological Improvements in Search Algorithms

There have been considerable advances in search algorithms and natural language processing that are applicable across domains since Wizorder’s completer was implemented.

Domain specific knowledge

Prior to our research, the existing WizOrder completer already utilized existing knowledge bases contained in the Unified Medical Language System (UMLS) Metathesaurus to enhance search performance^10,11. Additional medical knowledge could be incorporated into the existing completer.

Knowledge Discovery in Databases

This research used a very simple approach to data mining. More sophisticated machine learning and knowledge discovery techniques could be applied to improve the completer’s ability to anticipate the user likely selection.

Institutional Encouragement/Discouragement

Prior to the development of the popular queries file, there was no mechanism forcing an order to appear at the top of the ranked list of orders. While synonyms could be added to orderables in an attempt to boost its ranking, the results were often frustrating. In the Spring, a new advisor was developed to order medical and surgical consults services and titled “consult med/surg service”.

Unfortunately, the advisor’s ranking by the completer was often below the thirteenth term and off the screen for common query phrases. In response, the popular queries file was perverted to force this high profile initiative to the top. The popular queries mechanism could customize the completer to promote other hospital policies as well.

CONCLUSION

The ideal completer would draw from all four areas. The user activity database would remain one source of knowledge. More robust machine learning techniques would generate relevant synonyms for orderables and ordersets as well as data mine user preferences. Additional knowledge bases from UMLS would be evaluated for inclusion in the completer. Finally, a knowledge base could contain the institution’s efforts to explicitly influence ordering behavior. Potentially, the hospital may subtlety encourage the use of evidence-based ordersets or discourage the use of non-formulary medications by merely changing the rankings.