Cognitive Analysis of a Highly Configurable Web 2.0 EHR Interface

Abstract

We summarize findings of one case from initial in-lab usability and cognitive tests of 13 clinicians using MedWISE, a widget-based electronic health record (EHR) interface, to familiarize themselves with real patient cases and verbalize their assessment and plan. Multiple methods were used to examine patterns of use, time taken, use of new functionality, user-created interfaces, and diagnostic and human-computer interaction processes. All clinicians learned MedWISE quickly, most used more than half the new functionalities and found MedWISE easy to use and useful. We discuss implications for the design of future EHRs.

Introduction

MedWISE is a web 2.0 widget-based EHR interface which allows clinician users to select and arrange information and interfaces, and to share these created elements (as templates, custom lab panels, plots, mashups etc.) with colleagues(1). The intent is to provide far greater flexibility and user control than offered by conventional EHRs. In this paper, we report on in-laboratory user testing of 13 clinicians.

Background

As a highly configurable, widget-based system, MedWISE provides core new functionalities. For example, MedWISE gives the user the ability to:

• gather and spatially arrange any information elements from the EHR together on the same page, by click and drag

• make custom lab panels from any user-selected labs; and share these; likewise share user-created tabs (page interfaces) containing collections of notes, lab panels, plots, or other information

• set a tab containing user-gathered elements as a template, meaning that labs in the page are automatically updated with new information as it becomes available

• create multi-axis plots of any different types of lab test values together on the same plot (a mashup) encompassing all available patient data; pan, and zoom from a years-long scale to minutes/seconds

• collapse/expand widgets and edit header colors and titles

These features substantially expand the role of the user in system control. To assure it is the new features that are being tested, MedWISE information and menus match those of WebCIS, an EHR used at New York Presbyterian Hospital (NYP) for data review, with which all subjects are familiar. The usual WebCIS interface is simultaneously available. ‘Creating an interface’ in MedWISE is not a separate activity from using the system for assessment and does not necessarily require extra time or clicks.

We anticipate that MedWISE can increase flexibility and efficiency. The possible advantages, based on the literature, include decrease of the keyhole effect¹(2) which requires users to switch between screens and retain information in short-term memory (or copy it down), Although our overall study addresses disadvantages as well as advantages of such a system, in this paper, we focus on cognitive aspects and the following research questions:

1. How completely can users learn the system from a short (30 minute) training session/exercise?

2. What new functionalities do users use, for what purposes, and in what combinations?

3. How do users rate ease of use and usefulness?

4. What are the cognition-related user perceptions?

Because MedWISE has a substantially different interaction approach, laboratory testing with typical clinician users and tasks is important to identify potential flaws, risks, patterns of use, and impact on cognitive processes, workflow, and clinical performance. Think-aloud protocols with real cases and screen recording are an established method(3) which permits very detailed comprehensive analysis of user-system interaction, and has been used to identify flaws, errors, cognitive patterns (including lasting effects on clinician cognition), and routine patterns of use. This contributes to our understanding of the utility of the new features for EHR design.

Methods

Thirteen clinicians (10 medicine and nephrology residents, one attending, one physician assistant (PA), and one psychiatry resident) were recruited via a focus group announcement and email from the hospitalist and nephrology departments of NYP. Users were scheduled for 2-hour sessions and compensated $100. Approval was obtained from the Institutional Review Board of Columbia University. Subjects were given a short survey about clinical experience, demographics, computer proficiency and use of social networking tools (Table 1).

Table 1.

User Demographics and EHR Use

	Average	Range
Service at NYP	2.5yr	4mo–6.5yr
Work in field	3.3yr	4mo–7yr
WebCIS use	2.4yr	4mo–6.5yr
Eclipsys use	1.9 yr	4mo–4.5yr
Other EHR/CIS use	2.7yr	0–6.5yr
Hours/week WebCIS	26.6hr	8hr–80hr
Hours/week Eclipsys	25.8hr	8hr–80hr

All but two users (the attending and psychiatry resident) used Facebook; three others also used LinkedIn and one (the psychiatric resident) used Academica.edu. One rated himself ‘expert’ in computer knowledge, seven ‘above average’, and five ‘average’. Nine used EHR/CIS from other locations, usually home. Two had additional degrees to MD (PhD, MBA); two had BSE degrees.

We tested users in a two-part protocol. In part 1, the researcher presented the user with a 3-minute video showing several basic features of the system (e.g., the ability to select and arrange widgets, set up standing Pubmed searches as RSS feed widgets, make custom lab panels or multi-axis plots). Further demonstration of the system included timeline, templating, sharing, and other features. Users were given a short printed manual which included instructions and screenshots, but this was not used. The demonstration period took <5 minutes.

Users were then given a printed list of tasks to perform, with illustrative screenshots, and asked to carry out a series of ten basic tasks. Their on-screen actions and any verbalizations were recorded using Morae™ video-analytic screen capture software. All completed this section in 20 minutes or less.

In part 2, users were given five real patient cases, asked (via oral and printed instructions) to assume that they would be taking over care of the patient, and to use the MedWISE to familiarize themselves with the patient’s condition and state their assessment, diagnoses, and plan (after 10 minutes of system use for case analyzed in this paper). As in part 1, data were recorded using Morae(3). At the end of the five cases, clinicians rated their perceptions of system ease of use and usefulness on a survey.

The case was as follows. Patient 4 is a man in his mid fifties with a history of severe vascular problems including severe peripheral vascular disease centered around the aorta. He had an aortic valve resuspension, and treatment of thoracic aortic aneurysm followed by an AAA repair. Recently he was noted to have a Type A dissection of the thoracic aorta. Endovascular repair attempts were unsuccessful, and he subsequently developed a retroperitoneal hematoma requiring pressors and large scale transfusions.

Think-aloud protocols from the 13 users were transcribed and coded according to three coding schemata: 1) the schema of Hassebrock and Prietula(4) which pertains to diagnostic mental processes, 2) a schema identifying usability problems, system errors and user comments, and 3) a schema pertaining to human-computer interaction, some derived from the literature on the intelligent use of space in workplaces(5). The last was used to classify the different ways in which users used the new ability to arrange items to facilitate their thinking, design interfaces that reflected their categorization case information, signal others regarding relative priorities, and so on. Survey data were summarized using descriptive statistics.

Findings

Learning the System

All users were able to use core functions after the short training described above. A few had to ask the researcher basic questions (such as the link location for an advanced function).

Use of Data elements and Overall User Behaviors

First, we describe information access behavior common to all users, then the 4 different overall interaction strategies discovered, and then the use of specific new features. Table 2 summarizes key themes emerging from this study.

Table 2.

Key Themes:

All users have a common information access pattern

All used some new features

Four main interaction strategies: gather and view, tracking and juxtaposition of notes, purposive and organized, and mixtures.

Ability to gather items on same page is an advantage for: cognition, workflow, time (decreased back and forth navigation), errors (decreased copying)

Information externalization is an advantage, facilitates cognition

3 main spatial arrangements, left->right pattern

Users purposive in design for communication with others

Users categorize and mark items according to their own mental schemata (e.g. nephrologists creating custom lab panels)

Users use new features to solve problems

User are engaged, aware of problems, enthusiastic

Users varied widely in the number of elements they viewed. Sixty-five different elements were viewed by the 13 clinicians, with a range of 8–36 and an average of 16 elements per clinician. All users viewed a core set of information, starting with the latest clinical note and consult note; the latter was a detailed description of the patient’s vascular problems. From there, they viewed other information pertaining to the problem list. This included notes from other services, such as the chest and abdominal CT angiography reports, and last discharge summary. Most users then gathered lab panels relating to cardiovascular disease (e.g., lipid panels, hemoglobin, complete blood panel) followed by information to answer questions about the patient’s general health or suspected chronic disease (e.g., HbA1C, hepatic function).

Use of New Features

All users made some use of the new MedWISE features; this included novel uses unforeseen by the system designer. The most common (aside from drag and drop interface and template creation) was plotting creatinine to determine the patient’s state of kidney function, and setting it as a widget for future use. Spatial arrangement was used for categorization, workflow, and epistemic action²(6) (e.g., placement of panels for creatinine and urine protein above each other to facilitate ratio calculation, or of lab tests from different times together to view trends). Figures 1 and 2 contain examples of final user interfaces, showing the use of space, clustering, and marking (colored headers) as well as separation of note functions into information that persists relatively unchanged (e.g., patient and social history) and information that must be updated frequently (such as the current issues list) (Figure 2). One user explained this use of the new features to address the problem of excessive copying in notes and improve task-technology fit as “… the follow-up notes get copied and pasted over - you’re carrying on a lot of info that isn’t necessarily being reviewed at each visit, …for example you’re not taking a social history or family history at every single visit but it’ll show up in every single note., which takes up a lot of room and makes the note way longer than it has to be. The problem is that if you don’t include it, if you want to see it, you have to go clicking through note, note, note and it takes forever. So...have a section of things that aren’t necessarily going to be updated every single time, and if someone updates it it’s fine, but …my most recent progress note will be pretty tight. …things I’m going to be updating regularly are here, things I’m going to be looking back and forth … here. Things I might need to follow up on here.”

Figure 1.

Screenshot of MedWISE. Clicking on the links in the left-hand menu inserts data items (as movable rectangles) in the right-hand pane. A note and laboratory result plot are shown. Users can thus gather and arrange any desired elements of the clinical record together on the same page.

Figure 2.

The user arranged the orienting notes (clinical note and vascular service note) in the left column, labs down the middle (the brown bars indicate these are self-updating) and diagnostic reports with color-coded headers (red for cardiovascular-related items, black for EKGs, blue for brain-related items, white for the stress test) on the right.

This user used the ‘sticky note’ feature to set up separate summary line, history, current issues, healthcare maintenance, and to-do panels, with the less important ones collapsed, saving space while letting others know the information was there if needed. Several other users likewise split note functions into separate categories.

Overall Strategies

Users could be grouped into 3–4 different patterns in their use of the core select and arrange widget functionality. Some merely used the interface as a dynamic stage, collecting relevant widgets together (either reading each one after getting it, or waiting until all relevant widgets were gathered and then scanning the entire set at once, rearranging to juxtapose elements as necessary). This was one of the fastest methods of obtaining an overview, and several users commented on the possible time savings (two estimated 10 minutes would be saved per patient encounter). They also felt it reduced mental load because of decreased need to shift back and forth and remember or copy down things between screens). They also noted possible error reduction due to not having to copy.

Another strategy was to track the review process by stacking notes, closed, in one column, open two at a time in the second column, juxtaposed for comparison while reading, then close and move them to the third column. This kept reviewed material separate but with titles visible.

Some users were purposive and organized, carefully selecting and placing panels and using other design features to facilitate their future care of the patient or communication with colleagues. They often assigned regions to have particular significance; these fell into some common patterns, but could differ with personal preferences.

These patterns were: a) the user keeps the recent note they deem the most ‘helpful’ or complete in the left column, top row, with labs arranged down the central column, and ancillary notes or to-do lists or parameters to monitor on the right; b) the labs are arranged across the page in as many rows as needed, for maximum visibility on the screen; the user scrolls up and down to consider the evidence and gain an overview; c) the user assigns a space a meaning, for example ‘less important labs are in Column 3’ or ‘healthcare maintenance is at the bottom of Column 2’; ‘urine in the middle, blood work on the side’.

Another user arranged labs which needed monitoring in Column 3 with a sticky note at the top saying ’Screening labs’ as a header message to the next colleague. Two users (subjects 1 and 6) made little use of the new features; subject 1 likely because of an irritating navigation bug (subsequently corrected) and subject 6 because of his stated lack of being ‘used to so many windows’. We observed an overall pattern of progression from left to right corresponding to progression in the assessment, diagnosis and plan process; some users placed orienting notes in Column 1, illustrative material (plots, labs, imaging reports) in Column 2, and planning-related items, such as follow-up instructions, or to-do lists, to Column 3.

Nine users used the plotting function for trends, either as the plot composer mashup or single lab test values (e.g., creatinine) to answer questions such as ‘does this patient have early renal impairment?’ One user used this as a workaround for missing information; when unable to find very recent lab panels from menus she plotted the test, knowing it would include all data.

Some users identified the usefulness of the automatic saving and persistence of the created interface/history between sessions, and opined that the interface would be helpful in note writing as well as data review.

Several users made use of custom lab panel creation (see Figure 2). In particular, the two nephrologists conceptualized the necessary lab tests as sets; one created these as the bone panel, CKD (chronic kidney disease) labs, diabetes labs, etc. and shared or borrowed their own or others’ sets made previously.

Ease of Use and Usefulness

On a 5-point Likert scale for ease of use where 1=Very hard to use and 5=very easy to use, average user rating was 3.79. For a 5-point scale for usefulness, (1=not useful, 5=very useful) average user rating was 4.0. When asked what additional information would be useful, user responses included flowsheet data, scheduling functionality, and preset lab displays that include the entire lab test history.

User Perceptions Related to Cognitive Processes

User comments and the surprisingly short time required to learn core functions indicated high ease of use. Space prohibits full discussion of all user perceptions, however all but one user anticipated MedWISE being useful for some functions, mentioning note-writing, clinic evaluations, and inpatient care. One user liked that the single page space forced him to think about what was important:

“it allowed me to really quickly summarize relevant stuff, what I liked about it is sitting here thinking ‘how do I summarize this person succinctly - which is the art of medicine - and having one page to do it with - thinking about what’s the most relevant things, what do I want to follow, made me question what’s really important, so I think that was a positive thing.“

Others described the advantage of being able to assemble desired information on the same screen and also the ability to categorize and arrange items, as reflected in the three statements below:

“I probably approach the patient a little bit differently, ‘cause I can view the note and the lab”

“Being able to see a lot of stuff at once is the most obvious one. You can multitask much more easily because you can manage a lot more information at the same time, be able to look at notes and data at the same time, is really helpful.”

“[it takes] less [time] because it’s cool to be able to put things in different categories in your workspace. In WebCIS you end up having to click a lot to go back to information”

Users stated that the ability to externalize information (such as by using a sticky note, or seeing different elements together, the most liked feature) assisted their mental process: “I opened up a notepad (sticky note widget) to just maybe keep track of what was going on in my head, and maybe if possible to extract from these notes what’s important or what I would want to watch”. Several users stated that they normally did this with either paper or a Notepad document in a separate window, and that being able to do it in the EHR interface was even better.

Discussion

Our results suggest that clinicians can easily learn to use MedWISE and like the core features. Users’ interface creations confirmed our hypothesis that given a suitable set of tools, users would adapt the interface to fit their own needs and preferences, in the process revealing useful aspects of their mental and human-computer interaction processes and behaviors in organizing clinical information. Two users used few core features. Aside from the aforementioned bug, the first felt it would be useful for note writing and templates, but not the review task. The other was self-described as ‘an old guy, I’m not used to using so many windows’ – and just collected and viewed panels sequentially. Some predictions we made from the literature (e.g. that element selection can decrease the keyhole effect, juxtaposition would help thinking, or that work done in selection and arrangement would be shared and benefit others), were supported. This suggests that in a system with many users, their combined creations and modifications would result in a system better fitting their needs. In the words of one user “I feel like what we’re doing now is like the initial setup phase, the most intense portion, whereas everything else after that will be a time saver”.

Users in our study were engaged and several had definite and sometimes strong opinions about how clinical information systems should be designed, sometimes comparing MedWISE with other systems, suggesting valuable new features or interface improvements (some of which we incorporated), and even playing with the system and improving their interfaces after tasks ended and/or the researcher was out of the room. While this may be a result of a self-selection process, it also suggests that the idea of greater clinician control of the EHR is timely given that some estimate their EHR use at 80 hours/week.

Study limitations include the single patient case, the small number of users and the fact that the study was a laboratory study which may not fully represent field situations. These can limit its generalizability; however the use of real patient cases in a real system for a realistic task, are strengths.

In conclusion, the core features of MedWISE show potential for improving task-technology fit, time savings, clinical reasoning and user satisfaction.

Figure 3.

The user split note functions into summary, healthcare maintenance, and medications, in the yellow ‘sticky notes’. A custom plot of renal-related lab values is in the top middle widget, and custom labs for CKD and diabetes are in column 3.