Presenting Multiple Drug Alerts in an Ambulatory Electronic Prescribing System

Summary

Objective

This study explores alternative approaches to the display of drug alerts, and examines whether and how human-factors based interface design can be used to improve the prescriber’s perception about drug alert presentation, signal detection from noisy alert data, and their comprehension of clinical decision support during electronic prescribing.

Methods

We reviewed issues with presenting multiple drug alerts in electronic prescribing systems. User-centered design, consisting of iterative usability and prototype testing was applied. After an iterative design phase, we proposed several novel drug alert presentation interfaces; expert evaluation and formal usability testing were applied to access physician prescribers’ perceptions of the tools. We mapped drug alert attributes to different interface constructs. We examined four different interfaces for presenting multiple drug alerts.

Results

A TreeDashboard View was better perceived than a text-based ScrollText View with respect to the ability to detect critical information, the ability to accomplish tasks, and the perceptional efficacy of finding information.

Conclusion

A robust model for studying multiple drug-alert presentations was developed. Several drug alert presentation interfaces were proposed. The TreeDashboard View was better perceived than the text-based ScrollText View in delivering multiple drug alerts during a simulation of electronic prescribing.

1. Introduction

Electronic prescribing (e-prescribing) is the use of computing devices (clinical workstations, personal computers, or handheld devices) and drug information knowledge databases to enter, modify, review, output and electronically transmit drug prescriptions [1]. E-prescribing is a form of computerized physician order entry (CPOE). The latter can also allow for ordering of labs, imaging tests, procedures, etc. [2]. Studies suggest that e-prescribing with integrated decision support can reduce medication errors and adverse drug events (ADEs), increase healthcare efficiency, and improve patient safety [1-3].

Clinical decision support (CDS) in e-prescribing systems can improve patient safety and quality of care [1, 3, 4]. Teich and colleagues defined CDS as “providing clinicians or patients with clinical knowledge and patient-related information, intelligently filtered and presented at appropriate times, to enhance patient care” [5]. Clinical drug alerts/reminders are a form of CDS and have been commonly employed in CPOE/e-prescribing systems [1, 5]. These CDS systems use computer-generated messages that notify decision-makers when actions may be potentially unsafe. E-prescribing systems typically provide decision support in many domains that can promote safety, education, workflow improvement, communication between different stakeholders, and improved quality of care [1, 5-8]. ►Table 1 shows examples of these decision support areas [5, 8].

Table 1.

Decision support provided by e-prescribing systems

Drug-allergy Drug-drug Drug-disease Drug-food Drug-critical laboratory value Drug reference information including formulary information, cost, and generic alternatives Duplicate ingredients Geriatric precautions Lactation precautions Insurance information National/institutional/departmental guidelines Pediatric precautions Pregnancy precautions Recommended dosing limits including patient-specific limits on total dose, dose rate, etc.

In general, drug alerts/reminders are triggered based on pre-defined rules from CDS modules embedded in or connected to e-prescribing systems. For patients with complex medical regimens, the volume of drug alerts and precautions often results in large numbers of distinct alerts. In addition to these alerts being perceived as clinically insignificant, they also may not be well integrated into workflow [1, 2, 9, 10]. As a consequence, clinical decision support displays are often difficult to use in practice - resulting in the turning off of one or more decision-support modules to maintain or increase system adoption.

Studies suggested that e-prescribing adoption has increased [11] as recommended by groups such as the Institute of Medicine [12], the eHealth Initiative [1], and the Federal government [13], though there are no data to suggest that attention to drug alerts/reminders has improved. Despite the increase in e-prescribing adoption, there are continuing concerns about meaningful use of CDS related to usability [4, 14-16]. Issues such as too many alerts lead many clinicians to ignore important alerts. This condition is called “alert fatigue” [17-20]. Many studies have demonstrated extremely high override rates as high as 85–95% for drug interaction alerts [21], and 69–80% for drug-allergy alerts [22]. The high override rate suggests intrusive alerts do not alter prescriber practices. In addition, user interfaces often disrupt the work process and require inordinate time to comprehend, increasing what has been called the “attention cost” of the prescribing process [2]. Various studies have demonstrated the need for applying human factors engineering approaches to the challenge of CDS and the need for novel approaches to improving electronic health record (EHR) and CDS usability [3, 4, 6, 14-16, 23, 24]. Based on these notions, we postulated that alternative user interface designs may decrease the attention cost of alerts, as has been shown in inpatient CPOE systems [3]. We aimed to design and formally evaluate other user interfaces to display CDS output in an existing ambulatory e-prescribing system, to discover and evaluate how alternative human-computer interaction paradigms might impact the satisfaction with commercially available CDS knowledge database.

2. Methods

The study was conducted at Vanderbilt University Medical Center (VUMC) in Nashville, Tennessee between November 20 and December 15, 2009. To discover alternative interfaces for CDS, we used recommendations by authors such as Jacko [25] and involved clinician prescribers throughout all stages of interface development - a process known as user-centered design. As shown in ►Figure 1, we combined this process with two common human factors engineering techniques. First, we used expert heuristic review to rank candidates according to their usability, and iteratively suggest refinements. We then conducted formal usability testing on the highest ranking interface to evaluate its effectiveness at presenting multiple drug alerts in an existing ambulatory e-prescribing system.

Fig. 1.

Human-factor engineering design approach in this study

2.1 Interface development

We used three methods to develop and evaluate candidate user interfaces. First, we explored the literature about human-computer interfaces to find specific user interface approaches developed for multi-dimensionality alerting. Second, we used information mapping [26, 27] where we assigned drug alert attributes (type/category of alert, severity, frequency, strength of evidence, etc.) to different interface constructs (color, text, icon and shape) to ensure that the drug alert content could be readily captured by the interface. Third, we conducted an Expert Review using a think-aloud process [28-31] with a convenience sample of six domain experts (clinicians). The Expert Review can be conducted with 5 “domain experts”, as described by Nielsen [32], Kantner and Rosenbaum [33]. The think-aloud process requires that study subjects use a particular interface to complete an externally proposed task, such as creating a prescription for a medication. The subject is required to verbalize the cognitive processes and actions during the process of completing the task. The think-aloud technique allows for a more in-depth analysis about how specific information attributes influence the user’s cognitive process [30]. The experts were faculty/fellow members of a Biomedical Informatics training program. All participants were active practitioners who had at least two years of experience with EHR and e-prescribing systems.

Our literature review disclosed 4 candidate user interface paradigms (described below) that could represent the complexity of clinical decision support output. Using these descriptions, one of us (MX) constructed a prototype of each interface using the Borland JBuilder 2005 software. Then, domain experts used the think-aloud process and completed a survey designed to assess how well they were able to interpret the attributes of the alert, including the type of drug alert, the frequency of the alert, the severity of the warning, the strength of evidence, the description of the interaction, its clinical effect, and whether there was a detailed explanation (Monographs) available. We summarize the discrete data (from disagree to strongly agree on a 5-point scale) for each alert interface. Results from this survey were used to rank each of four drug alert presentation interfaces in terms of clarity and ease of use.

2.2 Usability Testing

2.2.1 Prototype development for multiple drug alert presentation

Based on the information mapping assessment, we implemented working prototypes for a novel drug alert interface that received the highest marks and a typical drug alert interface used as a control. Specifically, we implemented the TreeDashboard View (see Result section for description in details) to compare with the ScrollText View, which was similar to a standard e-prescribing user interface. The patient related information delivered by the prototypes included the patient’s age, the sex, the weight, the insurance information, the chief complaint for visit, the problem list, and the current medication list. The drug-related information delivered by the prototypes included the type of drug alert (drug-drug, drug-allergy, drug-disease, etc), the frequency of alert, the severity of the warning, the strength of evidence, the description of the interaction, its clinical effect, and whether there was a detailed explanation (Monograph) available.

2.2.2 Study subjects

For recruitment into the study, we targeted physician prescribers who were regular ambulatory e-prescribing users from Internal Medicine/Med-Peds (combined Internal Medicine and Pediatrics). None of them had participated in the first phase of the project – the Expert Review. We sent a personal email to each potential participant. The first twelve responders were invited to participate in the study. We used the other respondents as alternates if we could not schedule a session with any members of the original respondent group. The medical center’s Institutional Review Board approved the survey and research protocol.

2.2.3 Study design

Study participants completed the testing in a specially-configured cubicle to simulate attributes of a real-world ambulatory primary adult clinic setting. The participants used a workstation similar to what are available in this clinic. We created an audiotape of background noise from this clinic, and had each participant listen to this noise via a headset during the testing. The noise included ringing telephones, beeping pagers, and people talking and walking, and so that we could simulate the types of distractions that predispose to errors in prescribing decisions.

During the study, each participant was automatically randomized to one of 4 possible series of screens, as shown in ►Figure 2. We used a two-by-two counter-balanced presentation order scheme for case and drug-alert interface. Every clinician participant encountered one simulated patient case presented by TreeDashboard View and another simulated patient case presented by ScrollText View. We constructed two cases that were seen using different interfaces and in different sequences to control for training or fatigue effects. The patient cases represented different but common adult primary care prescribing situations that were similar in complexity. For instance, case I is a patient who is on warfarin. He is diagnosed with community acquired pneumonia. Erythromycin is newly prescribed. Case II is a patient who is on enalapril. He is diagnosed with congestive heart failure. Spironolactone is newly prescribed. Multiple care providers reviewed each case before testing. To complete the evaluation in a workflow-consistent manner, we integrated each case, prototype display, and study prompted into a test version of the local e-prescribing system, called RxStar, that has been in active use by more than 70% of our providers for over 5 years when the study was conducted.

Fig. 2.

Flow of events of formal usability testing

Each study subject received 15-minutes of training. During the training, the participants were given a tutorial and were required to walk through a simulated patient case that was different from the other simulated patient cases used in the formal usability testing. They then interacted with the two patient cases that each included demographic information, a diagnosis, current medications, available laboratory information, and medications to be prescribed. The participant was required to make a decision about whether or not to prescribe each drug, using all available information. The available decision options for the “Prescribe?” question included “Yes”, “No” or “Unsure”. The participant was not able to prescribe medications that were not part of the scripted cases. All participants followed the study instructions without any intervention from researchers, and all of them received a Starbucks gift card or iPod™ earphones for appreciation upon study completion. As each case was completed, we recorded the final decision and the time-to-decision. Each subject also provided us with his or her years using the EHR, their specialty, and role (attending or resident). Finally, at the end of each case, we asked each subject to assess:

1. usefulness of the displayed drug alerts;

2. ability of the interface to facilitate detecting critical information;

3. ability to accomplish a task;

4. sufficiency of information for making a prescribing decision;

5. user friendliness of the interface;

6. ease with which they could arrive at a decision.

Each question was answered on a 10-point scale with descriptive text anchors on either end [34], such as, from “1: Hard to detect critical information” to “10: Easy to detect critical information”. Survey data were collected electronically. Mean and median response times were calculated for each drug-alert interface. In this study, the attention cost was measured as response time of TreeDashboard View and ScrollText View. The response time was defined as the time from the display of the alerts to the time the prescriber made a decision to accept, to override or to cancel the prescription. We completed Wilcoxon signed-rank tests for significant difference in attention cost and precribers’ perception. All statistical testing was performed using SPSS Inc. SPSS software, version 14.0.

3. Results

3.1 Interface development

Seven drug alert attributes were included in our information mapping, as summarized in ►Table 2.

Table 2.

Description of drug alert attributes

Attribute	Description	Source
Type	Category of drug alert, e.g. drug-drug, drug-food, drug-disease	FDB*
Severity	Severity of the interaction or contraindication	FDB
Frequency	Frequency/prevalence of the interaction or contraindication	FDB
Strength of evidence	Strength of evidence supporting the warning	Fake data
Description	Description of the interaction found	FDB
Monograph	Abbreviated as “Mono”, which includes detailed information on drug’s adverse reactions, contraindications, pharmacokinetics as well as related drug monograph topics	FDB, if available
Clinical effect	Pharmacological mechanism of interaction or contraindication	FDB, if available

* FDB: First DataBank® drug knowledge database

We identified a set of potential user interface paradigms after an extensive literature search that we labeled ScrollText, Tree, TreeDashboard, and Thermometer Views. ►Figure 3 shows screenshots from the interface prototypes that we developed.

Fig. 3.

Four prototype drug alert presentation methods A) ScrollText View B) Tree View C) TreeDashboard View D) Thermometer View

The ScrollText View (►Figure 3A, top left) is a user interface that presents drug alert information in a plain text, non-graphical format, and in an essentially linear way [35]. The prescriber scrolls the text up and down to locate various indicators. Scrolling is a familiar action that does not require training, and is often used in CDS alerting.

The Tree View (►Figure 3B, top right) presents drug alert information hierarchically [36]. The level of various drug alerts is shown by indentation on the left side of tree nodes. A tree is a collection of one or more nodes. Each node represents a screening module, such as drug-drug interaction screening or drug-disease ccontraindication screening. Each node is the parent of zero or more children, which are also nodes corresponding to multiple drug alerts retrieved from a certain screening module. A tree can be expanded (tree children nodes are shown) or collapsed (tree children nodes are hidden). The way in which a collapsed or expanded node is displayed depends on definitive filtering rules.

The TreeDashboard View (►Figure3C, bottom left) shows the hierarchy of items and a matrix of additional data or item attributes in one unified structure. This view assembles information from multiple components into a unified display, and presents multiple drug alert information in a way that is easy to read, and easy to interact during e-prescribing. It also allows the prescriber to see a summary of various indicators. TreeDashboard is based on the concept of TreeTable [36], but it is more interactive in the way that the end users’ decisions, preferences and needs can be executed.

The Thermometer View (►Figure 3D, bottom right) presents multiple drug alerts using an analog view that is almost exclusively graphical [37, 38]. In this implementation, a thermometer represents each drug alert. The alert’s attributes are presented by the height of the dye in the thermometer, the stem’s color, the thermometer’s width, etc.

►Table 3 shows sample mappings of the attributes to our four representative interfaces.

Table 3.

Drug alert attributes mapped to each potential interface approach

Attributes	ScrollText	Tree	TreeDashboard	Thermometer
Type	Text in block	Tree node	Tree node	Text around thermometer
Severity	Colored number	Face icon, colored number / background	Face icon, colored number / background	dye color in stem
Frequency	Colored percentage	Colored percentage	Colored percentage / background	dye height in stem
Strength of Evidence	Colored number	Colored number	Colored number / background	Size of thermometer, number / dye color in bulb
Alert description	Text in block	Text in tree leaf	Text in tree leaf	Text around thermometer
Monograph	Text in block	Text in subpane	Text in subpane	Text around thermometer
Clinical effect	Text in block	Text in subpane	Text in subpane	Text around thermometer
Expand/Collapse Navigation	Unable	Able	Able	Unable

►Table 4 shows the six domain expert’s interface assessment. Each subject answered his/her drug alert related questions correctly. The subjects’ perceptions about the various drug alert presentation interfaces are summarized.

Table 4.

Comparison of prototype interfaces

Interface to present multiple ADEs		ScrollText View	Tree View	TreeDashboard View	Thermometer View
Interface concept		most textual	less graphical more textual	less textual more graphical	most graphical
Cognitive style for drug alert presentation		text-reader			image-visualizer
Domain expert Assessment	Easy to catch critical information?	+	++	++++	+
	Easy to interpret?	++++	++++	++++	-
	Is interface compact?	-	++	++++	+
	Information sufficient to make order decision?	++	++	+++	-

- disagree;

+ tend to agree;

++ slightly agree;

+++ moderately agree;

++++ strongly agree

3.2 Usability Testing

►Figure 4 and ►Figure 5 show the prototypes of TreeDashboard View and ScrollText View respectively that we implemented and integrated into an outpatient e-prescribing application and EHR system at VUMC.

Fig. 4.

Drug alert information delivered by TreeDashboard View

Fig. 5.

Drug alert information delivered by ScrollText View

Twelve physicians completed the usability testing. All were providers in Internal Medicine. Fifty percent of the subjects (n = 6) were attending physicians, while the other 50% (n = 6) were in residency. More than 75% (n = 9) of the subjects had used e-prescribing for 2 or more years. All participants completed the usability testing within 40 minutes before the exit survey was conducted.

One subject prescribed medications that were absolutely contraindicated according to drug-alert information presented by the ScrollText View. Three subjects prescribed medications in spite of high grade warnings (high risk for interactions, required monitoring or considering alternatives) presented by the ScrollText View. Two subjects prescribed medications in spite of high grade warning presented by the TreeDashboard View.

►Table 5 summarizes the comparison of participant perception on both drug-alert interfaces (ScrollText View and TreeDashboard View). Four questionnaire items addressed prescribers’ perception of quality of care. We considered that participants’ perception was strongly positive if the rating score was ≥8 on the 10-point scale. When asked about “the usefulness of drug alerts presented” (question 1), the response was positive with mean of 8.58 ± 0.793 for ScrollText View, and 9.00 ± 0.739 for TreeDashboard View, respectively. When asked “if provided information is sufficient for the participant to make prescribing decision” (question 4), the response was positive with mean of 8.67 ± 0.651 for ScrollText View, and mean of 8.50 ± 1.087 for TreeDashboard View. When asked about “how much the interface could help prescriber to accomplish prescribing task” (question 3), the response was positive with mean of 6.67 ± 1.497 for ScrollText View, and mean of 8.25 ± 0.754 for TreeDashboard View. When asked about “the ability to detect critical information” (question 2), the response was encouraging with a mean of 6.33 ± 1.826 for ScrollText View, and a mean of 9.08 ± 0.793 for TreeDashboard View. Two questionnaire items addressed proscribers’ perception of efficiency. When asked about “the ease of use” (question 5), the response was a mean of 6.42 ± 1.929 for ScrollText View, and a mean of 7.58 ± 1.505 for TreeDashboard View, respectively. When asked “if provided information is easy to find for making prescribing decision” (question 6), the mean response was 6.50 ± 1.931 for ScrollText View, 8.17 ± 1.030 for TreeDashboard View.

Table 5.

Results of Wilcoxon Paired Signed-Rank Test on prescribers’ perception

Questionnaire item	ScrollText View		TreeDashboard View		Paired Differences		p-value
	Mean score	SD	Mean score	SD	Mean	SD
Quality of care item
Question 1. Usefulness of drug alerts	8.58	0.793	9.00	0.739	-0.417	0.669	0.059
Question 2. Ability to detect critical info.	6.33	1.826	9.08	0.793	-2.750	1.960	0.005
Question 3. Ability to accomplish tasks	6.67	1.497	8.25	0.754	-1.583	1.676	0.001
Question 4. Information sufficient to make a prescribing decision	8.67	0.651	8.50	1.087	0.167	1.030	0.705
Efficiency item
Question 5. Ease of use	6.42	1.929	7.58	1.505	-1.167	3.099	0.234
Question 6. Information easy to find	6.50	1.931	8.17	1.030	-1.667	2.103	0.024

We also asked subjects to comment about different aspects of the interfaces. When asked “How enthusiastic would you be if VUMC implemented this interface in your clinic”, 11 of 12 subjects felt TreeDashboard View was more enthusiastic, while one subject felt that it was moderate.

There was no significant difference between the response times for TreeDashboard View and ScrollText View (152 ± 61 seconds versus 122 ± 50 seconds, z = –1.256, p = 0.209, α = 0.05).

4. Discussion

In the context of this study, we mapped the clinical data (drug alert attributes) to different visual representation metaphors (color, text, icon and shape) to ensure that the drug alert content could be readily captured and comprehended by clinician prescribers. After an iterative design phase, we examined four different interfaces for presenting multiple drug alerts [39]. The TreeDashboard View display appeared to be most favored among the four prototype interfaces studied. Our domain experts suggested that most available decision support applications delivered the drug alerts via a text-based presentation. Therefore, to simplify the study, we only implemented the TextScrolling View as a control prototype interface. While it is therefore true that we could implement all four interfaces, we believe we implemented the most standard and the most well accepted novel interface design in this study.

Formal usability testing of the most promising interface (TreeDashboard View) and controlled text-based ScrollText View demonstrated that physician prescribers agreed or strongly agreed that multiple drug alerts delivered by either platform were useful for e-prescribing (both interfaces scored >8.5 on a 10-point scale). The physician prescribers agreed or strongly agreed that patient-related and drug-alert information presented by both drug-alert interfaces were adequate for them to make prescribing decisions (both interfaces were scored ≥8.5 on a 10-point scale).

Formal usability testing also demonstrated that the physician prescribers had favorable impressions for drug alerts presented by the newly-designed TreeDashboard View. These favorable impressions relate to quality of patient care and efficiency when compared to the controlled text-based ScrollText View. Compared with the ScrollText View, the participants demonstrated a statistically significant improvement in the ability to detect critical information, the ability to accomplish tasks, and the easy of finding information using the TreeDashboard View (p<0.005, 0.001, and 0.024, respectively).

The study also showed that the physician prescribers’ response time to the same set of drug alerts varied substantially, reflected by a high standard deviation. The physician prescribers participating in the formal usability testing seemed to spend more time with multiple drug alerts presented by the Tree-Dashboard View. This result was contrary to our expectations – we initially hypothesized that the novel TreeDashboard View would reduce participant response time when evaluating multiple drug alerts.

Comments collected from survey questionnaires gave some explanations for this result. Traditionally, most drug alerts are delivered as text in popup windows, meaning that prescribers may be more familiar with the text-based ScrollText View. As a result, the novel TreeDashboard View interface may involve a learning curve. This idea was indicated by participant comments on negative aspects of the interface. In this study, both simulated patient cases contained 6 drug-drug interaction and drug-food interaction alerts. The text-based ScrollText View may be still sufficient to handle this limited number of multiple drug alerts. In addition, some participants noted that the TreeDashboard View encouraged them to seek more information, thus slowing them down, but potentially providing better care during prescribing. An improvement in our scaling system and more tutorial/training time may reduce the response time to TreeDashboard View in the future.

Nevertheless, the formal usability testing showed no significant difference between the response times for TreeDashboard View and ScrollText View. This suggested that both interfaces had similar attention cost, given the similar response time for the subjects to decide prescribing the medication or not during e-prescribing exercise. There was no evidence of alert fatigue for TreeDashboard View, given no subject prescribed medications that were absolutely contraindicated according to drug-alert information presented by the TreeDashboard View.

Our experience suggests that alert fatigue is one of significant barriers for meaningful use of drug decision support and adoption of e-prescribing application. When drug alerts repetitively contain irrelevant information that does not apply to a specific patient, they become more of a hassle and an interruption than a helpful tool. In this study, we aimed to explore different novel drug alert presentation interfaces. We expect that the novel interface with the highest marks would be used to improve the clinicians’ experience in exploring multiple drug alerts and the signal detection from potential high volume of alerts. We expect that the further usability study on such novel interfaces would aim on improving “alert fatigue”, perhaps decreasing the attention cost, therefore improving the meaningful use of drug decision support and adoption of e-prescribing application.

This study has limitations that merit discussion. First, the relatively small sample size (12 physician prescribers in the formal usability testing) limits statistical analysis. All participants were internists and half were residents, thus limiting generalization of the findings to community practitioners or specialists. In the future, we may need to expand the design to include more simulated patient cases, more test subjects, and more specialties. Additionally, the ScrollText View and the TreeDashboard View were implemented in a simple manner, without the extensive user interface refinements of a commercial interface. The prescribers may need more time to adapt to the multiple drug alerts delivered by the newly-designed TreeDashboard View. Third, there are limitations to the assessment of attention cost and alert fatigue. In this study, attention cost was measured as response time that was defined as the time from the display of the alerts to the time the prescriber made a decision to accept, to override or to cancel the prescription. Alert fatigue in this study was determined by the medication that was prescribed but absolutely contraindicated. Refinement of such assessment based on usability design principle are required for more validated evaluation. Four, this study only investigated a single in-house e-prescribing system with one commercial drug information knowledge database at one academic medical center. The participants were made up of house staff in Internal Medicine and Med-Peds who were familiar with the in-house developed EHR/e-prescribing systems in general. It is possible that results with other systems at other institutes may differ from those reported here.

5. Conclusion

This study explored issues related to the presentation of multiple drug alerts in an outpatient e-prescribing application integrated into an EHR system at VUMC. A robust model for studying multiple drug-alert presentations was developed. Several novel drug-alert presentation interfaces were introduced. Both expert evaluation and usability testing demonstrated that the TreeDashboard View was viewed more favorably than the text-based ScrollText View. Additional studies should be done on a refined version of this interface to improve its impact on accurate decision making and response time. The study result will be used to guide future work on the usability of multiple drug-alert presentation interfaces in an existing e-prescribing system.