Redesign of the Columbia University Infobutton Manager

Abstract

Background:

The Infobutton Manager (IM) is an application that provides clinical users with context-specfic links to health information resources. Usage of the first version (IM-1) suggested that the user interface was suboptimal.

Methods:

We conducted a laboratory-based observational study of IM-1 use, applied standard user interface design techniques to address observed problems, developed a new version (IM-2), conducted a second observational study and analyzed log files of the actual use of IM-1 and IM-2.

Results:

Modifications to the IM resulted in a reduction of “perusal time” (time between evocation of the IM and selecting a topic) from 11.13 to 5.92 seconds. However, evaluation of 14 months of usage logs did not show an appreciable effect on the perusal time or the rate at which users selected a topic once the IM was evoked.

Conclusions:

Laboratory analysis of the IM guided redesign that led to improved performance in the laboratory, but did not address factors that are influencing use.

Introduction

Infobuttons are context-specific links between clinical information systems and other online information resources, intended to anticipate and address clinician information needs.¹ Several developers have created such links in their own systems,^2,3,4 and they are beginning to appear in commercial systems.⁵ On the resource side, knowledge content providers, such as Thomson Micromedex and Elsevier Health Sciences, are enabling their products to respond to specific requests, such as those needed to service infobutton links.^6,7 An international standard for infobutton-related messages is under development by HL7.^8,9

Studies of infobutton use have shown that context-specific links to information resources are easy to use and helpful.^3,4 Timing studies have shown that users can get information very rapidly, with a minimal amount of time spent on navigation.⁴ In some cases, however, clinical information system users have been shown to have a variety of information needs in any given context – more than can be resolved with any particular link to a single resource.⁵ System developers have therefore created “infobutton managers” (IMs) that match contextual information (such as patient age and gender, user type, user task, and concept of interest) to potential information needs and then provide users with a set of links, each of which addresses a different need.^10,11

While evaluations of infobuttons have shown them to be easy to use and useful, the insertion of an IM into the process adds a level of complexity that might have a negative impact on these qualities. Del Fiol and colleagues have shown their IM requires only a modest effort on the part of their users.¹¹ A heuristic evaluation of the first IM at Columbia University, used by the New York Presbyterian Hospital (NYPH) and other institutions (IM-0), showed several design flaws.¹² As a result of that evaluation, a streamlined user interface (IM-1) was created (see Figure 1).

Figure 1.

Infobutton Manager Version 1 (IM-1). Note the multiple topics available by scrolling down or clicking on links next to “See also”.

Results from on-line and e-mail surveys regarding IM-1 showed results similar to those found by others evaluating their direct links: usability and usefulness were generally reported in positive terms.¹³ However, inspection of usage logs showed that, on average, users selected questions (“selection rate”) only 49% of the time they evoked the IM, with rates as low as 23% for certain user types in particular application contexts.¹³

User comments indicated that they were not finding questions of interest, although analysis showed that the desired questions were, in fact, usually present on the IM-1 page. Timing studies based on log file reecords showed that users were spending an inordinate amount of “perusal time” viewing the IM page, often failing to find the desired question. ¹⁴ This was in marked contrast to their use of a context-independent page of health resource (HR) links¹⁴ and to the experience of Maviglia and colleagues in their study of simple infobuttons.⁴

The need to improve IM usability, as reflected by objective measures, stimulated a reexamination of the IM-1 user interface. This paper describes the process we used to study the specific navigation issues with IM-1 and consider the design alternatives that might improve navigation. We report on the results of that process, as well as the development of Version 2 of the IM (IM-2), a usability evaluation of the new user interface, and a log file study to determine the ability of the new version to help users search for information in a timely manner.

Methods

Navigation Study of IM-1:

Because we could not identify from the log files the reasons for low selection rate or extensive perusal time, we designed an observational study to test clinicians' ability to navigate IM-1 to answer previously selected questions. We chose seven clinician questions (e.g., “Does cranberry juice alter the action of Warfarin?”) taken from the National Library of Medicine’s Clinical Questions Collection ¹⁵ and recruited clinician volunteers as subjects.

Subjects were presented a sample screen from a clinical information system with links to IM-1. They were instructed to imagine that they were reviewing patient information and that each of the seven questions arose. They were then to click on the link to IM-1 and attempt to obtain answers to each question. Questions consisted of two lab queries, three inpatient medication queries, one diagnosis query, and one outpatient medication query. We used Morae usability analysis software (TechSmith, Okemos, MI) to capture an audiovisual recording of each subject's activity. Sessions were reviewed to note times between page changes, number of mouse clicks and time required to obtain answers.

Redesign of IM User Interface:

Our initial approach was to seek ways to improve the IM to address workflow, aesthetics, and efficiency issues.

With regard to workflow, we found that the IM required only two mouse clicks to obtain information, while the use of the HR page required two clicks to reach a resource, manual entry of a search strategy, a third click to submit the query, and perhaps additional interaction to refine the search in ways that are carried out automatically by the IM. Given the high usage rate of, and satisfaction with, the HR page and its links,¹⁴ we concluded that number of steps, or clicks, required was not the sole predictor of user acceptance.

A second consideration revolved around the aesthetic design of the infobuttons. However, a side-by-side comparison of IM-1 and the HR page showed them to be very similar in terms of font, minimal use of color and graphics. We therefore concluded that aesthetics was not the key distinction between IM and HR.

Our redesign efforts ultimately focused on improving the efficiency of the IM user interface. We examined the navigability of IM-1 with respect to the Interaction Design Principles set forth by Bruce Tognazzini¹⁶ of the Neilsen group, including scanability, concise language, keyword menus and buttons. and design consistency, as well as Neilsen’s memorability criterion,¹⁷ the ease of reestablishing mastery of a tool after a period of disuse.

In IM-1, the user is presented with several “information need” questions, which vary both in number and order of presentation. If the concept of interest is related to other concepts of potential interest, questions about these concepts will be presented as well. For example, if the concept of interest is “potassium test”, the IM will present questions related to the test (such as how its specimen is collected or its result is interpreted), but it will also present questions about potassium (such as dosing and adverse effects) and related conditions (such hyperkalemia). When multiple concepts are considered, the IM-1 requires the user to scroll down to find the right question. This variability conflicted with the principle of design consistency, while the list length and question wordiness (as shown in Figure 1) conflicted with the scanability principle.

Redesign of IM Program:

IM-1 was evoked as a CGI (Common Gateway Interface) program, written in C++ and running on a Web server, that analyzed context parameters, and then generated and printed appropriate questions. The context parameters and concept of interest were passed to the program. The concept was classified and related concepts were identified using the Medical Entities Dictionary (MED).¹⁸ Questions were generated for each concept, and were printed in lists in concept-specific sections.

All computation was performed centrally on the Web server, and the fully generated HTML page was returned for rendering in the user's Web browser.

For IM-2, we chose to separate the computation and presentation functions of the program. This allows the basic processes of the IM – terminology translation, concept matching, and generation of links and questions – to be carried out independently from decisions made about the user interface design. Instead of generating a hard-coded HTML list of questions, the IM was modified to generate an array of questions that is passed to a display program that takes the array and generates appropriate code for visual output. This approach allows us to experiment with different designs and to develop display programs for different platforms (e.g., different browsers, operating systems, and handheld devices).

Navigation Study of IM-2:

We performed the same observational study on IM-2 with a new set of subjects. Morae software was used to capture each subject's activity. Sessions were reviewed to determine the reviewed for times between page changes, number of mouse clicks and time required for users to obtain answers.

Evaluation of Log Files:

Log file extracts were obtained, using a previously described method.¹³ The log file records contain a variety of information, including user ID, time, action, and (for the IM) the concept of interest. Actions include clinical information systems functions in which the IM links are embedded, evocation of the IM, selection of a question from the IM page, access to the HR page, and selection of a resource from the HR page. We used the institution's Lightweight Directory Access Protocol (LDAP) server to identify users' job titles.

Results

Navigation Study of IM-1:

Subjects were a convenience sample of six clinicians affiliated with Columbia's Department of Biomedical Informatics: two nurses and four physicians. One subject was excluded from analysis because he simply clicked on one link in the IM for each question but did not attempt to answer the questions. Subjects spent an average of 84.8 seconds answering each question (subject average range 47.4–169.7, SD 50.84). They used IM-1 an average of 1.68 times per question (SD 1.21); perusal time was 11.13 seconds (SD 9.7).

Redesign of IM User Interface:

The design of the user interface for the IM-2 was based on the requirements of improved scanability, conciseness and consistency, and the use of keywords rather than long statements. We chose to present the information needs as brief (one-word or two-word) topics, rather than questions. Unlike IM-1, which showed only relevant questions, IM-2 was designed to show the same topic list, in the same order, every time; relevant topics are “clickable”, while irrelevant topics are “greyed-out” (see Figure 2). In the event of multiple concepts, rather than requiring users to scroll down through several lists of questions for each concept, they see the same consistent topic list. When the user selects a topic, IM-2 presents the multiple concepts as a pop-up list (see, for example, the pop-list in Figure 2 containing links for “Hyperkalemia”, “Hypokalemia”, and “Potassium”).

Figure 2.

Infobutton Manager Version 2 (IM-2). Note the multiple subtopics in the pop-up list when a topic (in this case, “PubMed”) is clicked.

Redesign of IM Program:

In the new, topic-based design of the IM, each resource link in the IM's knowledge base is associated with a specific health information topic (drug dosage, patient instructions in Spanish, etc.) and each topic is assigned a label for display. The IM was modified to produce an array containing resource URLs and their labels. When the IM is called (as a CGI program), the array is returned to the user's browser, along with a JavaScript function that instructs the browser to dynamically build a page of links for display.^* The script first prints a shell containing all relevant concepts and consistently positions the topics, initially as grayed-out labels. Each of the resource links is then processed and their respective URLs are assigned to appropriate labels that are then converted to hyperlinks and colored blue. When two or more resource links are mapped to the same label by the IM, additional JavaScript code is generated that builds a dynamically generated dropdown list of links (see Figure 2).

Navigation Study of IM-2

Subjects comprised a new convenience sample of six clinicians affiliated with Columbia's Department of Biomedical Informatics: one nurse and five physicians. Subjects spent an average of 108.51 seconds answering each question (subject average range 74.9–138.3, SD 23.2). They used IM-2 an average of 1.78 times per question (SD 1.21); perusal time was 5.92 seconds (SD 5.0). Figure 3 shows the distribution of these times for each user, for each question, for both versions of the IM.

Figure 3.

Observed time spent perusing IM

Evaluation of Log Files

IM-2 was placed into use on November 1, 2006. Log files were obtained for the preceding ten months, and the subsequent four months. A total of 4,397 users used the IM (IM-1 and IM-2) 26,527 times; 3,525 users used the HR page 83,966 times during the same period. Monthly usage, selection rate, and perusal time for IM-1 and IM-2 are shown in Table 1, with corresponding statistics for the HR during the same periods included for comparison. There were no significant differences in overall performance between IM-1 and IM-2, nor were any significant differences found when subgroups were analyzed based on user type, application context or user experience with the IM.

Table 1.

Average IM and HR Monthly Statistics

	Time Period	Number of Uses (SD)	Selection Rate (SD)	Perusal Time (SD)
IM-1	1/06–10/06	1937 (250)	46.3 (2.5)	14.8 (1.4)
IM-2	11/06–2/07	1789 (154)	46.9 (2.9)	14.5 (1.5)
HR	1/06–10/06	5887 (268)	88.4 (0.9)	6.0 (0.5)
HR	11/06–2/07	6272 (535)	88.6 (0.5)	5.7 (0.4)

Statistics are separated by the period before and after the implementation of the IM-2. Numbers in parentheses are standard deviations.

Discussion

The Infobutton Manager provides an alternative method for retrieving relevant health knowledge, when compared to the standard approach of seeking out a resource and carrying out manual retrievals. The advantage of the automated IM approach, however, can be neutralized if the user interface is cumbersome or ineffective. The current study marks our second effort to improve the IM user interface.¹²

The first observational study confirmed our impressions that the low success rates and long perusal times were related to the effort imposed by the complex, variable, wordy IM-1 user interface. The HR, by comparison, has short resources names, presented in a consistent manner. Users who select the HR page know what is there and know how to get what they want. For example, they can click to open the page and position their mouse for the next click on a resource. The work presented in this study was an effort to approach that simplicity of design.

The challenge of making the IM more HR-like is that the questions and topics that arise in clinical environments vary radically from one context to another. The IM therefore needs to be dynamic. Furthermore the topics the IM deals with are more complex than simply the names of information resources. We turned to well-tested user interface design principles to develop IM-2. The resulting design seemed, to us, to address the problems we found with IM-1. Our second observational study supported our impressions.

Usage of IM-2 in real clinical settings, as reflected in the log files, shows a different story. The objective measures are inescapable: the introduction of IM-2 had, surprisingly, no detectable impact. There may be one or more reasons for low selection rates and long perusal times. Users may simply be unfamiliar with the user interface, compared with the HR page, which has been in use for many years. The IM also uses a different information retrieval paradigm (anticipatory retrieval) that users may have trouble embracing. Alternatively, the IM may not fit into the users' workflow or may require substantial cognitive investment by the users before they can match their information need with the ones presented by the IM.

Understanding which of these reasons, or what other reasons, might be contributing the users' difficulty with the IM will require more information than can be gleaned from log files; more direct information is needed from the users. The IM already includes user feedback functions, but they are rarely used. Surveys and focus groups may be somewhat useful but we have found them to be poor predictors of user information needs. We believe that only direct observation of clinicians in actual practice can provide the details we need. Although tedious and difficult to conduct, such studies will allow us to learn the actual information needed at the point of care, rather than resorting to log-file-based speculation.

Nevertheless, we do not feel that our efforts have been wasted. We applied appropriate methods and developed a new interface whose superiority is borne out, at least in a laboratory setting. We may never be able to match the efficiency provided by a simple list of 30 links. However, we continue to believe that the IM can provide more efficient start-to-finish retrievals, when compared to the HR, where users must do all the data entry and navigation themselves. Our design-development-evaluation process, as described in this paper, has helped us move forward. For example, the average time for subjects to answer the warfarin-cranberry question was 184.5 seconds using IM-1 and 47.5 seconds using IM-2. We believe this improvement will bear fruit for IM users, once we can identify and address the actual rate-limiting factors that are affecting IM usage.

Conclusions

We used standard approaches to improving the design of the Infobutton Manager user interface. The favorable results found in the laboratory were not reflected in actual use, suggesting that other factors are influencing users' ability to find the information they need in actual clinical care.