Use of Classification Models Based on Usage Data for the Selection of Infobutton Resources

Abstract

“Infobuttons” are information retrieval tools that predict the questions and the on-line information resources that a clinician may need in a particular context. The goal of this study was to employ infobutton usage data to produce classification models that predict the information resource that is most likely to be selected by a user in a given context.

Methods

: data mining techniques were applied to a dataset with 13 attributes and 7,968 infobutton sessions conducted in a six-month period. Five classification models were generated and compared in terms of prediction performance.

Results

: all classification models performed statistically better than the implementation currently in use at our institution. Two to five attributes were sufficient for the models to achieve their best performance.

Conclusion

: the application of data mining tools over infobutton usage data is a promising strategy to further improve the prediction capability of infobuttons.

Introduction

Clinicians encounter numerous information needs while caring for their patients and most of these information needs are not met^1,2. These needs include questions concerning therapy, diagnosis, prognosis, and the meaning of test results. On-line health information resources offer great potential to solve part of these information needs but, despite the wide availability of these resources, their use at the point of care remains low². “Infobuttons” are tools that access information resources, guided by contextual information from within an electronic medical record system (EMR). They have been promoted as a potential solution to increase the effective use of information resources at the point of care^3–6.

The underlying principle behind infobuttons is that the context of a particular problem dictates workers’ information needs⁴. Therefore, it is assumed that information needs can be predicted based on the context of the interaction between the clinician and an EMR. Infobuttons attempt to predict these information needs, offering clinicians access to context-specific questions and resources. In an ideal scenario, an infobutton should take the clinician as close as possible to an answer and require the user to have minimal interaction with the computer^3–5.

Current Infobutton implementations adopt a simple rule-based approach to determine the optimal set of resources that might be relevant in a given context^4–6. These rules are based on a small number of context attributes, such as the main clinical concept associated with the infobutton and the EMR module (e.g., order entry, lab results) where the infobutton is located. Existing rules are hand-crafted, limiting the number of attributes that can be used for prediction purposes in a manageable and useful way.

The goal of this study is to investigate the feasibility of employing data stored in an infobutton audit log to produce a classification model that predicts the resource that is most likely to be selected by a specific user in a given context. With such a model, infobuttons would be able to automatically direct the user to the best candidate resource, minimizing the cognitive effort and hence the time users spend seeking relevant content.

Background

Intermountain Healthcare (“Intermountain”) is a not-for-profit integrated delivery system of 21 hospitals, over 120 outpatient clinics, an employed physician group with over 500 physicians, and an insurance plan located in Utah and southeastern Idaho. Clinicians at Intermountain have access to a web-based EMR called HELP2 that offers access to a wide variety of data and functions, including laboratory results, problem lists, and medication order entry⁷. These modules have offered infobutton links since September 2001 (Figure 1). Infobuttons were used more than 80,000 times by 3,590 users in the past 5 years. A prior analysis revealed that frequent infobutton users have a predictable behavior regarding the selection of resources and questions that are offered by the infobuttons⁵. This analysis suggested that more advanced prediction techniques would be helpful in determining the optimal set of resources to offer to a user in a given context.

Figure 1.

An infobutton screen, showing the resulting page when an infobutton next to the medication “Azithromycin” is selected. After clicking on the infobutton, the user is automatically directed to the most frequently used resource within the context of the order entry module (i.e., Micromedex). The left panel allows users to navigate to different questions (e.g., contraindications, adverse effects) and resources.

All infobutton sessions are recorded in the infobutton log, which is maintained in the Intermountain Enterprise Data Warehouse (EDW). Each session contains the user id, user discipline (e.g., physician, registered nurse), patient id, task (EMR module that contained the infobutton), the main search concept associated with the infobutton (e.g., a medication, a lab test result), the date and time of the session, and the resource and question that were selected by the user.

The infobutton implementation currently available at Intermountain directs the clinician to the most frequently used resource within the context of a given EMR module. For example, problem list infobuttons direct users to MDConsult, medication order entry to Micromedex, and lab results to Clin-eguide. Users can also access other resources from the infobutton navigation panel as illustrated in Figure 1.

Methods

In this study, the prediction performance of the current infobutton implementation at Intermountain was compared with five different classification models that were generated using data mining tools.

Data source:

the dataset for data mining consisted of a subset of the infobutton log containing 7,968 sessions conducted by 190 frequent infobutton users between September 2006 and January 2007. Frequent users were defined as those who accounted for approximately 80% of the infobutton sessions during the same period. The rationale is based on the assumption that frequent users are more knowledgeable than non-frequent users regarding the choice of the most relevant resource within a given context. Sessions involving infrequently selected resources (i.e., 20 times or less) were excluded since classification algorithms typically need a minimum number of instances of a given class value to achieve a reasonable performance⁸. This assumption was confirmed by a set of preliminary experiments. The final study dataset contained sessions where users selected one of the following resources: Micromedex (76% of the sessions in the dataset), UpToDate (14.9%), MDConsult (5.2%), Clin-eguide (3.9%), and Medline Plus (0.6%).

In addition to the attributes available in the infobutton log, the dataset was enriched with potentially relevant attributes that were computed from the infobutton log or obtained from external data sources, such as the HELP2 terminology server and EDW⁷. The complete list of attributes used in the data mining experiment is displayed in Table 1.

Table 1.

Attributes of the dataset used in the data mining experiment. Attributes obtained from external sources or derived from other attributes in the infobutton log are marked with an asterisk.

Attribute name	Description
User
User id	User’s unique identifier in the EMR.
Discipline	User’s discipline (e.g., physician, registered nurse).
Specialty*	User’s specialty (e.g., cardiology, family practice). Obtained from EDW.
Infobutton sessions*	Number of infobutton sessions conducted by the user in 2006. Derived from infobutton log.
EMR sessions*	Number of EMR sessions conducted by the user in 2006. Obtained from the HELP2 audit log in EDW.
Infobutton search concept
Search concept	Clinical concept associated with the infobutton (drug, lab test, or a problem displayed in a HELP2 screen).
Parent concept*	Parent of the search concept. Obtained from concept hierarchies represented in the HELP2 terminology server (e.g., antidepressant, anti-hypertensive).
Patient
Age*	Patient age from EDW.
Gender*	Patient gender from EDW.
Session time
Weekend flag*	Indicates that the session occurred on a weekend. Computed from the infobutton session timestamp.
Night shift flag*	Indicates that the session occurred during the night.
Task	EMR module where the infobutton was located.
Resource id	Unique identifier of the resource selected by the user.

Data mining procedure: classifiers were trained and analyzed using Weka, an open source data mining tool⁸, based on five techniques: decision tree (C4.5 algorithm), rules (PART algorithm), Naïve Bayes, Bayesian network, and support vector machine (SVM).

Prior to the training step, the best subsets of attributes for each technique were identified using Weka’s automatic feature selection module. Occam’s Razor principle (i.e., simpler models are preferred over more complex ones) was applied when multiple subsets of attributes had an equivalent performance for a given technique. Complexity was evaluated in terms of the number of attributes and real-time accessibility. For example, attributes that are routinely stored in the infobutton log are easier and faster to retrieve in real-time than those that need additional computation or database look-ups. The names of the latter attributes are followed by an asterisk in Table 1.

Performance analysis:

two thirds of the sessions were used to train the classifiers and one third was used for validation. The performance of each trained classifier was evaluated on 10 different datasets that were obtained by applying the bootstrap sampling technique until the original size of the study dataset was reached. Consequently, classifiers were processed over the same 10 datasets.

Performance was measured in terms of agreement with the actual user selections (kappa) and percentage of correct predictions. The current infobutton implementation, which chooses the most frequently used resource for a given EMR module, was used as the referent.

Kappa scores of the five different classifiers and the referent current implementation were compared using ANOVA. When a statistical difference was found, the paired t-test was employed to compare differences between each pair of classifiers and between each classifier and the referent. Significance was set at 0.05, two-tail. All statistical analyses were performed using STATA.

Results

The automatic feature selection indicated that the most predictive attributes overall were user id, discipline, infobutton sessions, parent concept, and task. Table 2 lists the optimal attribute subset for each of the evaluated classifiers. Of the tested classifiers, the Bayesian network and SVM performed the best with the smallest attribute set (task, user id) and no significant improvement was gained when additional attributes were added. The decision tree classifier required all five attributes to achieve its best performance while the Naïve Bayes classifier’s performance deteriorated as attributes were added.

Table 2.

Attributes used by each classifier to achieve the best performance.

	Task	User id	Parent concept	Infobutton sessions	Discipline
Bayesian network	✗	✗
Decision tree	✗	✗	✗	✗	✗
Naïve Bayes	✗	✗
Rules	✗	✗	✗
SVM	✗	✗

Comparisons among the classifier performances in terms of kappa scores (Table 3) revealed that: 1) according to the kappa interpretation proposed by Landis and Koch⁹, all classifiers showed an almost perfect agreement with actual user choices (greater than 0.8), while the current implementation obtained only fair agreement (less than 0.4); 2) there were no statistical differences among the Bayesian network, SVM, and decision tree (p=0.98); 3) Bayesian network and SVM were better than Naïve Bayes and rules (p=0.01); 4) Naïve Bayes performed better than rules (p=0.02); and 5) all classifiers performed better than the current implementation (p<0.0001).

Table 3.

Classifiers’ performance in terms of average kappa score and percentage of correct predictions. Standard deviations are shown in parenthesis.

Classifier	Kappa	% of correct predictions
Bayesian network	.876 (±.006)	95.2 (±.2)
SVM	.876 (±.006)	95.1 (±.2)
Decision tree	.875 (±.007)	95.1 (±.3)
Naïve Bayes	.870 (±.008)	94.9 (±.2)
Rules	.860 (±.012)	94.6 (±.5)
Referent (current implementation)	.389 (±.005)	82.0 (±.1)

Discussion

This study describes the development and evaluation of five classification models to predict information resources that a user is most likely to select in a specific context. To our knowledge, this is the first study that investigates the feasibility of using classification models based on usage data to select infobutton resources. The classifiers present some potential advantages over present implementations at Intermountain and other institutions^4–6.

First, our results indicate that the evaluated classifiers were consistently superior to our current implementation in predicting the resource that infobutton users will selected. If any of the classifiers were put into operation, infobutton users would be directed to their preferred resource in a given context at least 94.6% of the time. Even when the classifier fails to correctly predict the user’s preference, an infobutton navigation panel as depicted in Figure 1 would allow the user to switch to any of the other available resources, so the cost of misclassification in this application is fairly low.

Second, some of the techniques used in this study provide a ranked output, either in terms of probabilities or a score. Current implementations, on the other hand, do not provide a ranking mechanism^4,6. A ranking could be used to determine the resources that are to be displayed in a given infobutton session as well as the sequence in which these resources would be displayed. Resources with a probability that is lower than a pre-defined cutoff would not be offered. Such a method would enable infobuttons to elect a small set of relevant resources from a larger pool of resources that are available.

Third, in the current approach the logic that drives the selection of resources is hand-crafted and based primarily on users’ feedback and observational studies of clinicians’ information needs^2,10. Although these qualitative assessments are still essential to guide the development of infobuttons and configuration of new questions and resources, maintenance and ongoing improvement of infobuttons will likely become unmanageable with attempts to integrate advancing knowledge about information needs at the point of care and the contexts in which they arise. In contrast, data mining classifiers can be retrained periodically to reflect new users, changes in users’ behavior or the information system environment, and the addition of resources. In practice, frequent infobutton users would be rewarded with a more accurate resource prediction. A question for future research is how often a classifier needs to be retrained to keep its performance levels. Alternatively, Bayesian learning techniques could enable a classifier to continuously learn as new observations, i.e. infobutton sessions, become available¹¹.

Fourth and last, once information needs and the context in which these needs arise are better understood, it is possible that richer context representations, with a larger number of attributes, will be necessary to improve the prediction of resources. In such a scenario, an automated approach to the maintenance of infobuttons seems more appealing, as the number of possibilities to be covered by a prediction algorithm increases.

When comparing the different techniques evaluated in this study, Bayesian network, SVM, and decision tree achieved the best performances. Yet, the decision tree model required a larger number of attributes to obtain its best performance. One of these attributes is the number of sessions conducted by a user in a given period of time, which unlike other attributes is not readily available to infobuttons without a precomputation that needs to be performed periodically. In contrast, the two attributes used by the Bayesian network and SVM classifiers (user id and task) are routinely included as part of infobutton requests at our institution.

Although three of the classifiers achieved maximum performance with only two attributes, it is expected that as a larger number of HELP2 modules implement infobuttons and more resources become available, classifiers may need additional attributes to keep their performance close to the levels reported in this study. For example, resources that cover a very specific domain of knowledge or are targeted to a particular patient population or provider type (e.g., genetics, pediatric patients, uncommon lab tests) may benefit from a classifier that includes the main search concept, parent concept, patient age, or specialty attributes. We plan to reassess these attribute sets once a sufficient number of domain-specific resource sessions becomes available in the infobutton log as well as determine the classification techniques that scale best when accommodating additional attributes. In this study, the performance of the Bayesian network and SVM were not altered with the addition of new attributes, indicating that they might provide the desired level of scalability. This was not the case for the Naïve Bayes classifier, the performance of which worsened as new attributes were added.

Limitations:

The user id attribute was a useful predictor in all five classifiers. This is an expected finding since user id tells who the user is, and therefore indicates past preferences of a given user to help predict future choices. The user id is included in all infobutton requests, simplifying its use by any classifier that is eventually implemented at Intermountain. However, this attribute imposes an important limitation in the long term: only users who have infobutton sessions in the training set will benefit from the classifier’s highest performance. A potential solution to overcome this problem is to use proxy attributes that help characterize a user, such as discipline and specialty, in addition to, or in place of, the user id itself. The assumption is that a classifier would “consider” these proxy attributes more strongly when the user id is absent or not in the training set. Research is being conducted at Intermountain to evaluate the performance of classifiers using such an approach, especially when users not present in the training dataset are included in the test dataset.

These results are part of an exploratory study and therefore should be used with caution. Additional research is necessary to assess potential implementation issues, such as the frequency by which a classifier needs to be retrained and the limitations regarding the employment of the user id attribute. Moreover, it will be important to validate that infobuttons that offer resources based on predictions by classification models actually improve clinicians’ ability to effectively find answers to their questions at the point of care. Finally, since the dataset used in this study represents the use of specific infobuttons at one single institution with a home-grown EMR, generalizations of the results to other institutions should be made with caution.

Conclusion

Previous studies have shown that infobuttons provide answers to clinical questions at the point of care in a timely manner, with a high level of user satisfaction^6,12. This study indicates that infobuttons can be further improved by employing prediction models based on previous usage data to determine the resources that a clinician might choose. A similar approach may be adopted to address the prediction of questions that clinicians are likely to look for in a particular context.