Functional Analysis of Interfaces in U.S. Military Electronic Health Record System using UFuRT Framework

Abstract

The overall aim of this study is to evaluate the usability of U.S. military electronic health record (EHR) system AHLTA using a systematic work-centered evaluation framework UFuRT --- User, Functional, Representational, and Task Analysis. This paper with the focus of Functional Analysis (FA) of AHLTA explores operationalizable methods to study functions supported by user interfaces. A system hierarchy was created to map and uniquely identify all items on the interfaces. These items were then classified independently by 2 evaluators as Operations or Objects. Operations were further classified as either Domain or Overhead function. With acceptable inter-rater agreement, of the 1996 items in the interfaces, 61% were operations, around one fourth of which were Overhead functions. Overhead functions are hypothesized to be targets to be redesigned for improvements in usability.

Introduction

An Electronic Health Record (EHR) system is defined as being able to collect electronic health information longitudinally, provide immediate electronic access to the information by authorized users, provide knowledge and decision support, and support efficient healthcare delivery, according to 2003 Institute of Medicine (IOM) Letter Report. [1] AHLTA, previously named as Composite Health Care System (CHCS), is the EHR system implemented in the U.S. Military Health System (MHS). AHLTA assists health services comprehensively, including preventive care, readiness, acute care and long-term care, to serve the integrated needs of military health care delivery.

As the most widely rolled-out EHR in the United States, AHLTA faces one of the biggest challenges with respect to its usability and support of clinical workflow.

Usability as ISO 9241-11 definition refers to “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”. The healthcare work environment exhibits prominent safety-critical and time-critical characteristics. Because of this, a usable interface design of information systems in healthcare industry is required to save time, prevent error, and improve user satisfaction. However, in the healthcare industry, usability of information systems is especially critical in that poorly designed systems can compromise patient safety. Lack of understanding of clinical users and their workflow have led to numerous devastating failures. There has not yet been a widely accepted framework to assess and evaluate the usability of health information systems. A standard usability evaluation method is vital to assess the safety of existing systems and would provide valuable guidance to designers and developers to create interfaces which better suit the workflow of clinicians [2] and therefore contribute to successful implementation.

UFuRT---User Analysis, Functional Analysis, Representational Analysis and Task Analysis---is a systematic methodology developed for usability evaluation of information systems in healthcare industry.[3,4] UFuRT provides a conceptual framework based on work-centered principles. User Analysis identifies users’ characteristics including age, educational background, expertise and skills. Functional Analysis differentiates functionalities specific to the work domain from those dependent on the artifacts. Task Analysis examines the actions necessary to achieve goals and the time required. Representational Analysis assesses the cognitive effects of information display format on user interfaces.

In this study, we report an operationalizable method to conduct Functional Analysis of AHLTA user interfaces. The purpose of this analysis is to determine how well existing functions in an application supports the functions actually needed in the work. Based on this differentiation, usability of the user interfaces could be potentially improved.

Methods

1. System Hierarchy

In order to uniquely identify interface components, a system hierarchy was created to represent each item on the AHLTA user interface. The system hierarchy was created by visually inspecting the interface from top to bottom and left to right. Each interface item (label, field, drop-down menu etc.) was listed in an MS Excel spreadsheet and provided a unique identifier. The spreadsheet was subsequently imported into Protégé 3.3 to determine the total number of nodes and to visualize the resulting hierarchy.

2. Classification of Objects and Operations

Each interface item in the system hierarchy was classified as an Object or Operation. An Object was defined as when no actions or activities could be conducted on the item. Operations were items in which an action or activity could be conducted. For example, in the interface presented in Figure 2, under the tab A/P, the columns with the titles of “ICD” and “Diagnosis” can only display information and therefore are classified as Objects in the system hierarchy file shown in Figure 3; whereas up and down arrow buttons of “Priority” can be used to change the priority of each diagnosis and therefore are Operations. Two raters independently classified each item. Inter-rater agreement was calculated and discrepancies were resolved by consensus.

Figure 2.

The interface of diagnosis in AHLTA A/P section

Figure 3.

Overview of Coding Process for Functional Analysis of AHLTA A/P section

The purpose of this clarification to separate items where actions are possible from those that are not merely labels. Thus, items identified as Operations are ready to be further classified depending on their importance to the domain in the next step.

3. Classification of Domain and Overhead Functions

Each interface item that was classified as an Operation in the previous step was further classified as either a Domain or Overhead function. Domain function was Operations specific to the healthcare domain rather than dependent on the artifact or interfaces, whereas Overhead function was Operation related to the user interface instead of the healthcare domain. Two raters independently classified each item. Inter-rater agreement was calculated and any discrepancies were resolved by consensus.

Results

1. System Hierarchy

Three highest levels in the system hierarchy are visualized and presented in Figure 1. In total, 1996 interface items or nodes were documented. An example AHLTA interface and the corresponding part of system hierarchy file are shown in Figure 2 and 3, respectively.

Figure 1.

Visualization of top three levels of AHLTA user interfaces

2. Classification of Objects and Operations

Of all the interface items identified in the hierarchy, 61% were classified as Operations and 39% as Objects (see Figure 4). The inter-rater agreement in the classification of Object and Operation was statistically acceptable (kappa > 0.6).

Figure 4.

Percentage of Objects vs Operations in AHLTA

3. Classification of Domain and Overhead Functions

Of the 1218 items classified as Operations, 76% were identified as Domain functions and 24% as Overhead functions. Detailed results of AHLTA Patient Record section are shown in Figure 5. In general, there were more Domain functions than Overhead functions with the exception of “Summary” section. “Readiness” section has the lowest percentage of Overhead functions. The inter-rater agreement in the classification of Domain and Overhead Functions was statistically acceptable (kappa > 0.6).

Figure 5.

Percentage of Domain vs Overhead functions in AHLTA

Discussion

In this research, we developed a method to conduct Functional Analysis of AHLTA user interfaces based on the systematic usability evaluation framework --- UFuRT. Our approach was to develop a bottom-up method to identify “functions” by first classifying each interface item as an Operation or Object. We then classified all Operations as Domain or Overhead functions depending on their relevance to the domain.

We found that the AHLTA application is complex and contained nearly 2000 user interface items. After identifying Operations, we found that nearly 25% were Overhead and were not necessary for health related tasks. It is likely that the usability of an interface may improve by minimizing the number of Overhead functions. Overhead functions are likely to result in unnecessary actions by the users. In future work, we will individually inspect each Overhead function and identify specific strategies for how they may be eliminated or revised. We will also conduct Task and User Analysis to provide further evidence on the redundancy of some Overhead functions.

UFuRT usability evaluation framework is applicable to information systems in healthcare industry [3] and adopts the principles of work-centered design [2]. Functional Analysis is a critical component within UFuRT framework in terms of the evaluation of work-centeredness. Our three-step bottom-up method effectively and efficiently formalized the criteria to identify functions by classifying each interface item as Object or Operation and then Domain or Overhead. As the process is likely to be somewhat subjective it was important to calculate inter-rater reliability. We found acceptable inter-rater agreement which suggests the operationalizability of the definitions of Objects versus Operations, Domain versus Overhead functions. These definitions are straightforward and easy to understand as well as to apply in the evaluation process of Functional Analysis. In addition, the data collection and analysis can be accomplished in Microsoft Excel which is widely available and easy to use. This method of Functional Analysis does not require evaluators to have a highly specialized training in evaluation. Real users in healthcare industry could thus participate closely with designers and developers in evaluating the functionalities supported by the system or prototype. The ease of use of this method could facilitate the participatory approach [5,6] in a great extent.

This study used the bottom-up method of analyzing functionalities supported by the EHR system however, was not able to provide the ontology of the work domain from the top level. Further, our approach was to analyze each Operation independently. In future work, we will assess how such an approach can contribute to improving usability of an interface. This bottom-up method is very helpful in the iterative development and participatory approach, whereas the ontology of the work domain from the top level would be useful to guide a work-centered design from the first attempt.

Conclusion

Functional analysis under the UFuRT framework is a useful tool to identify and distinguish system functions that are essential for the domain and those which are overhead. This is a fundamental step to design systems that are useful and usable. About one fourth of the functions for AHLTA are Overhead functions. There are the functions that can be potentially eliminated or optimized to increase the usability of AHLTA. In summary, a system with good usability should foremost have good design of functionality that matches the work domain and good user interface that support efficient task performance by the users.

Figure 6.

Percentage of Domain vs Overhead functions in AHLTA Patient Record section