Critical Conversations about Optimal Design Column: Thorough Error Testing a Requirement for Strong EHR Usability

Recently, while participating in a major nursing electronic health record (EHR) study, we were reminded of the negative impact of inadequate testing on the usability of technology. In this editorial, we provide a short overview of the study followed by a discussion of what we learned while observing our earliest subjects interact with the technology. Building on the Holden and Karsh (2010) and Neilson (1994) definitions, our team defines usability as the extent to which a technology is easy to use and learn, causes minimal errors, is used as intended, and valued by users. Most of the issues we uncovered inadvertently compromised intended use. Please note that the data driven findings of this study are forthcoming and will be reported elsewhere.

The Study Conditions

The four year National Institute of Nursing Research (Keenan et al, 2011–2015) funded study has two aims:

• Aim 1: to identify best nursing practices among hospitalized end of life patients from standardized data collected with the electronic care planning systems called HANDS; and,

• Aim 2: to translate the best practice findings into immediately useful decision support at the point of care

In the research phase described here, our Aim 2 team piloted a randomized clinical trial comparing the usability of 4 different decision support types with a representative group of 60 nurse subjects. The decision support features tested had been developed and refined during the first two years of the study and contained best practice suggestions derived from evidence generated by our Aim 1 team. Prior to the pilot, the script and software for each condition were carefully pre-tested by members of our team.

As in our earlier cycles, we took steps to ensure that the four conditions were tested by groups comparably sized and balanced on the characteristics of race, ethnicity, gender, education, and experience. Subjects were first provided a short orientation to a modified version of the HANDS basic electronic care planning system and then given simulated handoffs for two end-of- life patients. Next, subjects were randomly assigned to one of four conditions and asked to interact and adjust the electronically displayed plans of care as they saw fit. The four conditions were: 1) basic modified HANDS; 2) HANDS + narrative clinical decision support (CDS); 3) HANDS + (narrative + table CDS); 4) HANDS + (narrative + graph CDS). We then left subjects alone to interact with the software and monitored their interactions from a remote station located in an adjacent space out of view.

Discussion

At the start of the study, a monitoring station was used to keep an eye on the subject’s progress in completing the plans of care. Our main interest at this point was to get the data collected to help us learn how the different conditions ranked against each other on the rate and quickness of adoption of the items on our best practice list. Although the subjects’ facial expressions and interactions with the computer screens were being recorded for future analysis, we were able to observe these activities first hand as subjects adapted the plans of care in front of them. To our surprise we collectively identified more than 40 issues encountered by our early subjects that were seemingly invisible to them. Software glitches were the most prevalent but there were also issues with the orientation scripts and screens and software content.

The issues identified specifically involved breaches to the intended functionality and use of the system, both key components of usability. For example, although we had pretested the software, there were numerous software glitches found that appeared to interfere with the ability of the subject to use the system as intended. These included such things as failure of an alert to shut off when it was no longer applicable; the ability to add the same term (e.g, NOC) more than once and rate it differently (inappropriate), and the inability to add a term that was previously deleted. Other issues involved the training materials. In the absence of clear directions in our orientation to look at information available through CDS buttons immediately, some subjects chose to ignore these buttons until later in the care planning process. An unintended consequence was that subjects used other features to change the plans of care making it difficult for us to evaluate the value of the CDS feature in these instances. Adding or clarifying simple directions and visuals in our orientation corrected this and other similar issues. We also found errors in the content displayed. These included such things as wrong definition supplied for a term on an information screen or tool tip and wrong outcome ratings being displayed. Because we set up the capability to monitor use under real time conditions, all of these issues were reconciled early on but would have gone unnoticed without the vigilance employed in this study since users did not report them. For this reason, the first step toward building a technology system with strong usability is to quickly identify and eliminate all software issues that alter the intended functions and gaps in training materials.

Conclusion

As seasoned researchers in the field of electronic health records, we were surprised to see the large number of errors that our pretesting had failed to uncover in our software and training. Of greater concern is that these issues seemed not to be noticed by the subjects since none were reported even when subjects were given ample and easy opportunities to do so. This is problematic since it is our experience that current EHRs rely heavily on user reports of errors to fix systems. Insufficient testing, thus may well result in the use of EHRs with undetected functionality problems and the potential for dire consequences (e.g., clinicians taking inappropriate actions based on information in an EHR that is confusing or erroneous). To avoid the potential dire consequences of poor usability testing, we advocate for the use of strong testing methodologies that are designed to comprehensively detect systems issues such as those reported here. In the absence of such testing, it is nearly impossible to establish that the software works and is being used as intended and that the data collected with it are valid, an essential criteria in the era of big data. When buying and EHR, thus it is important to determine up-front how functionality and training issues are identified and reconciled both before and after implementation into practice.