Novel user interface design for medication reconciliation: an evaluation of Twinlist

Abstract

Objective The primary objective was to evaluate time, number of interface actions, and accuracy on medication reconciliation tasks using a novel user interface (Twinlist, which lays out the medications in five columns based on similarity and uses animation to introduce the grouping - www.cs.umd.edu/hcil/sharp/twinlist) compared to a Control interface (where medications are presented side by side in two columns). A secondary objective was to assess participant agreement with statements regarding clarity and utility and to elicit comparisons.

Material and Methods A 1 × 2 within-subjects experimental design was used with interface (Twinlist or Control) as an independent variable; time, number of clicks, scrolls, and errors were used as dependent variables. Participants were practicing medical providers with experience performing medication reconciliation but no experience with Twinlist. They reconciled two cases in each interface (in a counterbalanced order), then provided feedback on the design of the interface.

Results Twenty medical providers participated in the study for a total of 80 trials. The trials using Twinlist were statistically significantly faster (18%), with fewer clicks (40%) and scrolls (60%). Serious errors were noted 12 and 31 times in Twinlist and Control trials, respectively.

Discussion Trials using Twinlist were faster and more accurate. Subjectively, participants rated Twinlist more favorably than Control. They valued the novel layout of the drugs, but indicated that the included animation would be valuable for novices, but not necessarily for advanced users. Additional feedback from participants provides guidance for further development and clinical implementations.

Conclusions Cognitive support of medication reconciliation through interface design can significantly improve performance and safety.

BACKGROUND AND SIGNIFICANCE

Medication reconciliation is part of meaningful use guidelines and is a complex process wherein errors can greatly impact the quality of patient care.^1–3 It is a multistep process that is frequently complicated by the involvement of multiple providers in the patient’s care over the course of a hospital stay. During the discharge process, clinicians must obtain a list of home medications (usually created during the admission process) and one of current medications used during the hospital stay. These lists are then reconciled into a new list by continuing some medications and discontinuing others. Lastly, the final reconciled list must be communicated to the patient as well as the appropriate caregivers.⁴

Many issues arise during the construction and retrieval of medication lists, both during the admission and discharge process: patients being unable to communicate or unsure of their current medications, records being incorrect or incomplete due to data entry errors, and medication histories that are inaccessible or unavailable. After collecting the various lists of medications, the clinician must reconcile the lists into a single set of medications the patient should continue (and decide which medications should be discontinued). Various errors can occur during reconciliation, such as duplications (including one medication with multiple entries or two drugs of a similar class that were not intended to be given simultaneously), omissions (patient is discharged without necessary orders), dosing errors, or drug interactions (drugs administered together resulting in potentially harmful interactions).^5–7 These are all known complications that are made more likely by the influx of new medications to the market, the use of multiple names for the same medication, and the increase in combination drugs that contain multiple medications branded under a new trade name. Our work focuses on facilitating the task of deciding what medications to keep or reject from the lists. We recently introduced a novel user interface design, Twinlist,⁸ which uses spatial layout to reveal similarities and differences, staged animation to help first-time users, and carefully designed interaction to facilitate the reconciliation task. This study aims to quantify Twinlist’s effectiveness in helping medical providers during the reconciliation process.

Prior research has highlighted the need to improve the medication reconciliation process.^2,6,9–16 Patients are particularly vulnerable during care transitions,^17,18 due to changes in medication regimens. Studies have found that 94% of patients had medication errors at discharge, nearly all of which were eliminated with a medication reconciliation tool.³ However, complete and accurate reconciliation is difficult and is often overlooked or not performed.¹⁹ Three kinds of medication error outcomes have been classified¹²: harmful errors, also known as preventable adverse drug effects (PADEs); potentially harmful errors (near-misses); and harmless errors. Every year, 1.5 million harmful errors are reported,¹³ with most PADEs (75%) occurring during discharge.²⁰

Much research has considered the medication reconciliation problem, but few studies describe user interfaces used in clinical settings. Some focus on patient entering of adherence information; e.g., the Automated Patient History Intake Device.²¹ Others, such as the Pre-Admission Medication List,¹ use a “superlist” of medications from all sources grouped alphabetically by name. This at least allows users to compare medications based on the same names, but provides little support to help clinicians locate similar and unique drugs on the list. Other work has proposed algorithms to automatically detect similarities between medications^22,23 or infer links between medications and medical indications,^24–26 but that work does not focus on the user interfaces that take advantage of those linkages.

This study aims to examine the effectiveness of the novel interface, Twinlist,⁸ compared to a Control interface. We focused on one scenario in which a physician is discharging a patient. We propose the following hypotheses:

H1: With the same short training for both the Twinlist and Control interfaces, participants will perform the reconciliation task statistically significantly faster with Twinlist than with Control.

H2: Participants will perform the reconciliation task with significantly fewer interaction actions with Twinlist than with Control.

H3: Participants will make significantly fewer serious errors with Twinlist than with Control.

METHODS

Participants

Twenty participants were recruited through an email advertisement sent to two urban academic medical facilities. Both Emergency and Internal Medicine residents and attending physicians participated. They were all familiar with the process of medication reconciliation during patient discharge, but had no experience with Twinlist. This study was approved by the Internal Review Board and participant consent was acquired.

Apparatus and Materials

Because existing medication reconciliation user interfaces vary so greatly, two interfaces were developed from scratch: Control and Twinlist. The same team created both to assure consistency related to font size, color scheme, drug selection method, drug detail presentation, and user action logging. Otherwise, seemingly small design differences come into play; e.g., using tiny checkboxes separated from drug names versus large buttons directly labeled with drug names. Likewise, general system issues can easily confound experimental results; examples include differences in response time, differences in patient headers, and differences in other screen elements irrelevant to medication reconciliation. Our aim was to test an interface design, not to demonstrate that any particular operational/commercial display could be incrementally improved. This is a traditional method of evaluation in the field of Human– Computer Interaction.

We carefully designed the Control interface by reviewing dozens of user interfaces during the scope of the project. The proposed Control interface was reviewed by approximately 10 colleagues from the SharpC project who were familiar with medication reconciliation interfaces (non-coauthors). They were specifically asked whether the Control interface was reasonably representative of existing interfaces they had used or seen. They agreed with our design, and a few responded that the Control interface itself was an improvement over the worst interfaces they had seen. Our Control is not a perfect match for any existing interface, but we believe our design is reasonably representative of many current systems. Our comparison allows a measure of the impact of specific design elements in a controlled environment; i.e., the use of a novel spatial layout to reveal similarities and differences.

For this experiment, a very simple, ad hoc internal database was created by hand; no formal database connection was established to any commercial drug database or EHR system for the experiment. The information about the drug lists was manually assembled into the prototypes for five specific cases. Both interfaces showed the same drug class and generic and trade names in a “details” area. In addition, the Twinlist interface also made use of similarity information about the drugs based on the output of the algorithm described in Bozzo Silva et al.²² The Control interface did not have access to that information.

Control Interface Description

The Control interface presented two separate columns of medications organized in alphabetical order (user options are “as entered,” “generic,” or “brand”): Intake on the left (medications the patient was taking at home before admission) and Hospital on the right (medications the patient was prescribed during hospitalization). When the pointer hovered over a medication the background changed to dark gray (see Toprol in Figure 1) and details were displayed at the bottom of the screen (including generic and trade name, ordered strength, route, frequency, drug class, and indication). To indicate that a medication should be kept (i.e., continued), users simply clicked on the drug, and the background color changed to green (see Lipitor in Figure 1). To indicate that a medication should be rejected (i.e., discontinued) users either clicked twice or right-clicked on the drug. The deselected item was then grayed-out with a strikethrough (see rosuvastatin in Figure 1). Clicking multiple times on a drug cycled through all the states: undecided, accepted or rejected (e.g., a click on a rejected item changes it to undecided). In addition, controls were available at the top of each column to keep or reject all the remaining undecided items in a column (see buttons labeled “keep rest” and “reject rest” under the “Intake” and “Hospital” column titles in Figure 1). A “clear” button also reset items to undecided. Once all items had been decided upon; i.e., either kept or rejected, users could click on “sign off” at the bottom right of the screen. The sign-off button remained inactive and grayed-out until all decisions had been made (e.g., in Figure 1 we see that there are still 28 medications left to act upon).

Figure 1::

Control interface with the lists side by side. Two drugs have already been selected (Lipitor and metformin – shown in green), one was deselected (rosuvastatin, shown at the bottom, grayed-out). The pointer is currently on Toprol. It is highlighted in dark gray and more detail about this medication – including its drug class – is shown at the bottom of the screen.

While the Control interface does not match any particular vendor interface, we believe that it is a fair representation of many existing reconciliation interfaces. During our review of existing systems we found a variety of designs. Some used a single very long column with all medications from all sources, usually sorted by alphabetical order. Others used a side-by-side design similar to our Control interface. A few systems automatically merged and selected identical drugs. In addition, most interfaces required users to select small checkboxes, separate from the names of the drugs. Grouping by drug class is available in at least one system.

Twinlist Interface Description

A thorough description of Twinlist and its design rationale can be found in a previous paper;⁸ we summarize its functionality here. In Twinlist, the drugs initially began in a layout similar to Control but with the lists moved farther apart. A multistage animation begins immediately, changing the display into a five-column layout (see Figure 2). The animation was comprised of the following steps: identical items merge in the middle (“Identical” column), items that are unique (i.e., contain no similarities with any other items) were moved to the sides (“Intake unique” and “Hospital unique” columns), similar items were aligned horizontally, with small differences highlighted in yellow (“Intake similar” and “Hospital similar” columns), and finally, the display was compacted to save space by moving unique and identical items to the top and similar items below. Videos showing the animation are included in the Supplementary materials, and on the project webpage www.cs.umd.edu/hcil/sharp/twinlist.

Figure 2::

Twinlist with its five columns. The pointer is hovering over Toprol, so it is highlighted, but the similar drug (metoprolol) is also highlighted with the same dark gray background, while all the attribute differences are shown in yellow.

Hovering over a unique or identical medication had the same behavior as in the Control interface. However, when the pointer hovered over a drug that is similar to one (or more) other drugs, they were all highlighted with dark-gray backgrounds (see Toprol and metoprolol in Figure 2). When one of those similar drugs is selected (i.e., kept) the other(s) were automatically rejected (see Lipitor and rosuvastatin in Figure 2). Note that in the Twinlist interface it is not possible to keep two identical medications since they are displayed only once.

Twinlist Group by Drug Class and Other Features

Other features are available in Twinlist: buttons to add a new medication or modify an existing one, screens to review the instructions to the patients, options to customize the speed of the animation or remove it entirely, etc. The reconciliation tasks did not require participants to access these controls and they were instructed not to use them. At the end of the study trials, participants were asked for their feedback regarding one particular option: an alternate layout wherein items are grouped by drug class (see Figure 3). Drugs remain in the same column as with the standard five-column layout but are also grouped by drug class rather than simply listed by alphabetical order. Participants were only asked to give feedback on this “Group by Class” layout; they did not perform any experimental cases using it.

Figure 3::

Twinlist interface with grouping by class. The four antihypertensive medication entries are now grouped together for review.

Figure 4::

Average time for each participant to complete the two reconciliation tasks/cases, per interface.

Questionnaire

A questionnaire using a Likert scale (0 = strongly disagree, 9 = strongly agree) was designed to assess participant agreement with statements regarding clarity and utility and to elicit comparisons between interfaces.

Datasets

Five clinical scenarios were designed by two clinicians working together, then reviewed by five other clinicians and quality assurance officers: two cases of congestive heart failure (CHF), two cases of pulmonary disease (PD), and an appendectomy case for training purposes. Each included a narrative describing the situation, along with a home and hospital medication list. Each case included at least 12 medications in each list, including some lesser known medications and some overlapping drugs to mirror situations in which errors are likely to occur. The two CHF and two PD cases were designed to be of equivalent complexity to minimize confounding between tested interfaces.

Study Design

The study used a 1 × 2 within-subjects design with interface version as the independent factor. Participants performed medication reconciliation on two cases (i.e., one CHF and one PD case) using one interface, then the remaining two cases with the other interface, therefore using both interfaces and all four cases over the course of the experiment.

Outcome variables

Time

For each case, after participants read the narrative, we recorded the time between the click to start and the click to sign off. Interface task time is the time to reconcile one CHF case and one PD case using that interface and includes animation time for the Twinlist interface.

Number of interface actions

Efficiency was evaluated as the number of mouse clicks and scroll actions recorded during the use of each interface. Counts of low-level actions (such as keystrokes or mouse clicks) are traditional metrics used in the evaluation of interfaces.^27,28

Accuracy

The number and type of errors were reviewed by two clinicians, who were blinded to subject and interface used. A single case could lead to multiple errors. Each error was assigned a type and a severity score, independently. Consensus was reached after three rounds, including an independent review, combined review, and discussion regarding 10 of the identified errors. The developed common coding scheme is as follows:

Error severity

• Critical errors – High likelihood of patient return to the hospital (similar to PADEs in the literature¹²)

• Serious errors – Possible patient readmission (similar to potentially harmful errors) Other less than optimal choices for which there is no clear guideline in the literature about what to do were not counted as errors.

Error type

• Duplication errors – in which keeping similar medications might lead to an overdose.

• Omission errors – in which the patient was discharged without a required medication.

• One dosing error was found but it was not severe so it was not included in the analysis.

Procedure

The observer demonstrated the first interface using the appendectomy training case (1–3 min; the order of interface was counterbalanced between participants so that half started with Control and the other half started with Twinlist); participants practiced on their own using the same training case until they felt comfortable (1–2 min). The same Apple MacBook Pro and Google Chrome configuration was used by all participants, and all participants were permitted to use either a regular mouse or a touchpad. They were reminded that the observer would not be able to clarify the case scenarios during the experiment and to do their best with the information provided in the scenario narratives. They were allowed to use any drug references (e.g., pharmacopoeias) they would normally use during clinical work, as long as they used it in a consistent manner with both interfaces (seven participants used them). They were also encouraged to ask for help regarding the use of the interface should they forget what to click on (no one asked for help). Finally they were instructed to work as quickly and accurately as they would during a busy shift with real patients.

Participants read the narrative for their first test case (case ordering was also counterbalanced with one CHF case and one PD case per interface) and asked to click a button to start the interface when ready. After completing the first case, the second narrative was given and the participants again clicked to begin reconciling. This process (training then medical reconciliation for two fictional patients) was repeated for the other interface.

After all four test cases had been reconciled, the participants were asked to fill the first part of the questionnaire (regarding the features of the user interfaces they had just used, in particular the animation and groupings) and how those features may have helped them make better decisions. Afterward, participants were asked to provide feedback on the alternative spatial layout with grouping by drug class. They filled the last section of the questionnaire and received $50 for their participation.

RESULTS

Before conducting the main analysis, we verified that there were no statistically significant differences between pairs of similar cases (e.g., between the two PD cases). Thus, we could treat similar datasets as being equivalent, and analyzed the data using a paired t-test. Similarly, we verified that the interface ordering did not have a significant effect on any of our dependent variables.

Time

Participants completed both reconciliation tasks significantly faster (P = 0.006) using Twinlist (M = 211.4 s, SD = 54.5 s) compared to using Control (M = 293.2 s, SD = 133.3 s). The mean improvement was 81.8 s, an 18% improvement (See Figure 4).

Number of interface actions

The difference in number of clicks between Control (M = 84.2, SD = 29.1) and Twinlist (M = 47.3, SD = 16.05) was significantly different (P < 0.001). The mean improvement was 36.9 clicks, for a 40% reduction in the number of clicks.

The difference in number of scrolls between Control (M = 549.20, SD = 367.13) and Twinlist (M = 145.90, SD =  119.14) was statistically significantly different (P < 0.001). The average reduction in the amount of scrolling was 60% (See Figure 5).

Figure 5::

Total number of clicks (left) and scrolls (right) per participant per interface. Note that a single continuous scroll interaction may result in multiple scroll events fired (depending on how far and smooth the scroll was), hence the large number of scroll actions reported.

Accuracy

In total, participants made 43 errors: 31 errors were rated critical (24 with Control, seven with Twinlist) and 12 were rated serious (eight with Control, four with Twinlist). In terms of error type, 38 were duplication errors (29 with Control, nine with Twinlist) and five were omission errors (three with Control, two with Twinlist) (See Figure 6). The difference in average number of errors (critical + serious) between Control (M = 1.60, SD = 2.80) and Twinlist (M = 0.55, SD = 0.69) was not significantly different (P = 0.074). Three participants accounted for about a third of all errors (see “Discussion” section). The list of errors is in the Supplementary materials (Table 3).

Figure 6::

Error counts by severity and type, per interface.

DISCUSSION

Time

On average, participants using Twinlist were able to complete the medication reconciliation task 18% faster (H1 supported). A participant commented that each medication in the intake list should be compared against all of the hospital list medications for similarities. We believe Twinlist’s gains can be attributed to the following design elements: 1) spatial grouping positions similar items near each other thus reduces the search time for similar medications and 2) a compact display facilitates getting an overview of all the medications and of the final choices before sign-off. Highlighting differences also provides support for identifying the key differences between similar medications, and merging the identical items reduces the number of items to search through.

Number of interface actions

On average, participants using Twinlist used 40% fewer clicks and 60% fewer scrolls when using Twinlist (H2 supported).

This improvement is likely due to 1) fewer items to click (identical items are merged) and 2) automatic rejection of similar medications instead of having to explicitly reject them. Note that large rectangular buttons containing the name of the drugs are easy to click. The cost of the additional clicks (in terms of time and frustration) would be much greater if users had to select tiny checkboxes next to drug names.

Scrolling is likely reduced because 1) similar items are usually close together and 2) the display is more compact. Using larger screens or working on simpler cases would reduce the number of scrolls, but the amount of scrolling we recorded mirrors the effort needed to scan lists from top to bottom. Scrolling was easy for participants as they used the touchpad or a mouse with a scroll wheel, but the cost of scrolling (time and frustration) would be greater if participants had to manipulate the small handle of a scrollbar.

Accuracy

Overall, twice as many critical errors and three times as many duplication errors were made while using Control compared to Twinlist, which illustrates that interface design can have a major impact on reconciliation.

However, most participants were able to work accurately with either interface. Just three participants accounted for 22 of the 31 errors made in Control. The number of errors per participant was not statistically significantly different (critical, serious, or critical + serious). One participant was responsible for 13 of the 42 errors (29% of all errors); 11 while using Control and only two with Twinlist.

Information integrity had an effect on accuracy. Nine duplication errors were made in Twinlist involving drugs marked as unique and only one with Control, but six of those nine Twinlist errors involved the drug hydrochlorothiazide, which was not appropriately grouped next to Zestoretic, a combination of lisinopril and hydrochlorothiazide (see Figure 2 for the version used in the experiment, and Figure 7 for a partial screenshot of the correct grouping). These errors suggest that users rely on the grouping to make decisions and illustrate the potential risk associated with the implementation of decision support. If it were not for these nine errors (i.e., if the similarity algorithm was improved to better handle compound drugs), then the average number of errors would be statistically significantly lower (P = 0.033) for Twinlist (M = 0.25, SD = 0.44) than Control (M = 1.55, SD = 2.72). Thus H3 is only partially supported.

Figure 7::

A partial screenshot of the correct grouping of hydrochlorothiazide close to Zestoretic.

Questionnaire

The text of the 16 questions and associated statistics are in the Supplementary materials (Table 4 and Figure 1). Generally, participants agreed that the Twinlist interface was clear and helpful. There was strong agreement that Twinlist would improve the overall quality of the reconciled list, over Control, and over whatever interface they were currently using.

There was also strong agreement that, when starting using Twinlist, the five-column grouping, horizontal alignment, and use of highlight to bring attention to similar drugs were helpful. There was only weak agreement regarding the utility of animation. We observed seven participants visibly ignoring the animation or trying to begin before the animation completed. One participant who liked the animation commented that it helped him focus during the reconciliation process. Participants indicated that in the long run they would not use the animation on a regular basis, but they believed that the other features would remain very helpful.

Feedback from users

Participants supported spatially separating lists to show where drugs were used (home or hospital), as opposed to merging the lists with icons or text showing origin (as seen in many existing systems). For example, one participant said that it made it easy to discharge patients on their home medications.

When asked if they thought it would be better to automatically continue identical drugs, 16 out of 20 participants were against automation. Many said that it was safer to decide manually. One participant said that it was “best to click [on all drugs one by one] as this is the last check.” One participant mentioned that the interface made it easy to select each drug (direct selection with large targets, instead of small checkboxes as seen in many existing interfaces).

In contrast, the majority of participants agreed that rejecting a medication automatically when the similar one had been selected was appropriate. For example, one participant said that it significantly saved time and that it was “one of the nicest features” in Twinlist.

After seeing Twinlist with drugs grouped by class, only 45% of the participants reported that they preferred it over the normal view. Those opposed most often complained that drug class was redundant. They believed that other clinicians would know the drug classes. On the other hand, several suggested it would be useful in training situations or difficult cases. Similarly, most participants disliked an extension of this feature that listed drugs more than once when they belonged to more than one class (with the copies in secondary classes appearing in a paler gray). Their objection was based on the observation that there is usually only a single intent for a prescription. In summary, we believe that the grouping by class would be a useful option but should not be the default layout and further research is necessary to determine how best to handle complex lists and uncommon medications. Alternative means of grouping (e.g., by therapeutic intent) are likely to be more useful, but may in turn increase errors in those medications with multiple indications or interactions with other medications.

The Supplementary materials provides additional counts (Table 5).

Limitations and suggestions for future evaluations

Including more than two cases per interface may have helped better tease out differences in error rates. More details could be included in the case narratives, which would better guide participants and could also facilitate the severity coding of errors. Retrospectively we feel that it would have been useful to counterbalance participant experience (e.g., resident/PA or attending physician), and work environment (e.g., emergency department or inpatient). In addition, the role of other users (e.g., nurses, pharmacists, and technicians) in a team-based approach to medication reconciliation should be considered in future experiments. The development and use of benchmark cases, testing procedures, and standard error codes would facilitate the comparison of Twinlist with future interfaces as they become available. Further studies could also be done on evaluating Twinlist against other baselines (e.g., manual reconciliation done by hand, or newer designs entering the market), evaluating alternative layouts (e.g., grouping by therapeutic intent or drug class), and evaluating the learning effects of staged animation²⁹ with physicians.

CONCLUSION

This study evaluated the effectiveness of a novel interface (Twinlist), which uses a spatial layout to reveal similarities and differences, staged animation, and carefully designed interaction to facilitate medication reconciliation in a medical provider population. Twinlist reduced reconciliation time by 18% and the number of clicks and scrolls by 40% and 60%, respectively. Overall, three times as many errors were made with the Control interface than with Twinlist, driven by dramatic improvements observed for three participants. These results suggest that cognitive support through interface design can have a significant impact on patient safety.