Usability Testing of a Web-based Tool for Managing Open Shifts on Nursing Units

Abstract

Using observational and interview approaches, we evaluated the usability of a web-based communication tool that allows managers to announce open work shifts and staff to request work shifts. We used Morae™ software to record screen shots and study participants’ (4 Nurse Managers, 3 Registered Nurses and 3 Patient Care Technicians) utterances during system use. Although features and functions used and patterns of use varied by role and individual, all participants found the system easy to use. Use of a particular function depended upon how effective the user perceived the function to be for the task that they wished to accomplish. User satisfaction was achieved when an effective function also allowed them to meet their task goal efficiently.

Introduction

The construct of “usability” is primarily found in the field of Human Computer Interaction (HCI) where it is described as the relationship between humans and computers. According to Staggers, nursing computer systems should be designed with an understanding of human-centered attributes and cognitive behaviors so that human computer interactions will seem natural, achieving the goal of computers as “assistants” to humans [1]. Moreover, the usability of a computer system is determined not only by the user-computer interactions, but also by the degree to which it can be successfully integrated to perform tasks in the intended work environment [2]. International Organization for Standardization (ISO) 9241-11 has defined usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [3]. This definition highlights three essential concepts: effectiveness, efficiency and satisfaction.

Evaluating technology usability is currently a rising issue in healthcare as technology has been implemented for purposes such as decision support, knowledge management, improved communication, effective resource management, reduction of medical errors and time saving [4]. Poor system usability has been identified as a barrier to technology adoption [5, 6]. Usability testing can identify areas for product improvement and subsequently potentially enhance technology acceptance.

Informed by the ISO 9241-11 definition, we evaluated the usability of a web-based communication tool for managing open shifts on nursing units. This study is part of a larger program of research focused on testing conceptual models related to technology acceptance using data from a variety of computer applications.

1. Research Questions and Objectives

The study employed a descriptive, qualitative design to address two research questions: 1) How do nurse managers and staff use the web-based communication tool for managing open shifts? and 2) What are users’ perceptions of the tool’s usability?

2. Methods

2.1. Sample and Setting

Participants were recruited via personal contact from a single community hospital in the Philadelphia area. Inclusion criteria were: 1) nurse manager or staff member (Registered Nurse or Patient Care Technician), 2) experience using the BidShift Open Shift Management tool for at least 6 months on a monthly basis, and 3) hospital employee pre- and post-BidShift implementation and for more than 1 year.

2.2. Web-based Communication Tool for Managing Open Shifts

BidShift (http://www.concerro.com /solutions) is a web-based communication tool that allows managers to announce open shifts throughout their organization, staff to request shifts for which they are best qualified based upon their profile, and managers to award shifts to a requesting staff member. The tool has demonstrated improve staffing effectiveness and the work environment in some organizations [7].

2.3. Procedures

Following informed consent, participants were asked to think aloud as they completed three sub-tasks associated with the open shift management process depending on their role of manager (M) or staff (S): 1) post/search for a shift, 2) award/request a shift, and 3) view report/schedule. Their utterances and screenshots were recorded using Morae™ software. After task completion, participants were asked three questions: 1) What do you like the most about the system and why? 2) What do you like the least about the system and why? 3) Do you have any suggestions for improving the system? Participants were probed about additional information regarding functionalities, features, processes, user interface, user-system interactions and manager-staff communication. Data collection time ranged from 30 to 60 minutes. Data were managed and coded using Morae™. Participants’ utterances and answers to interview questions were thematically analyzed and summarized [8].

3. Results

Four nurse managers, three Registered Nurses and three Patient Care Technicians participated in the usability testing. The majority had more than year of experience using Bidshift. Two themes were identified regarding how managers and staff use BidShift: prioritization/work patterns and frequency of use considered a measure of effectiveness (Table 1). Two other themes were identified in regards to users’ perceptions of system usability: satisfaction with the system and system impact. These themes were further deconstructed and regrouped into multiple categories that generalize the system usability.

Table 1.

Patterns of Use and Perceptions of Usability

Research Question 1: How do nurse managers and staff use BidShift?
Theme: Prioritization/Work Patterns
M: “The first thing I do when I open Bidshift to look at is I notice what shifts are starting within 72 hours and I would want to pick those first”, “Usually when I come early in the morning, I check all my bid shifts. […] I also pull shifts to see what my needs are to make sure they are on. I also double check the staff profile waiting to be approved”.
S: “I usually go in with a few days in mind, and times. […]And usually, you could find the shift for what day you’re on. I look at my own unit first, and then if not I’ll look for other telemetry units”
Theme: Frequency of use
M: “I rarely use report. […] I don’t need it.”, “I don’t use it (unit calendar) too much, because we have another way to see all this. I have access to this in a different way…from the staffing office.”
S: “I used that (my calendar) all the time […] and we usually print this out a lot, this includes all, […] We can see all we are already working so we can see what we can pick up.”
Research Question 2: What are users’ perceptions of system usability?
Theme: Satisfaction with the system
Category: Favorite functions = effectiveness + efficiency ( e.g. time saving, increased productivity, information organization, and convenience)
M: “This is ‘Create a shift’, which is the same as ‘PowerPost’, just it can only do one shift at the time. You saw me how I do a bunch (through PowerPost), and we like a bunch” , “You can see now I have already awarded 111 shifts, and this schedule only has been out for two weeks […] I have awarded more than half of what I need. […] It exceeded my expectation.”
S: ”I like being able to see what’s available everywhere.”
Category: Satisfaction with other functions (e.g. accessibility and information needs fulfilled)
M: “I can look up the staff profile if I need a phone number to call people for different shift”,
S: “I think it is great that you can look at it from home, and you can be awarded shifts and learn all that from home.”, “I like on the main page where it has ‘In my unit’, it just narrows it down for you.”
Theme: System impact
Category: Improved equity
M: “We send out an email to our per diem staff that our schedule is done and the shifts are posted, so everyone has the equal opportunity to apply for the shift. So we don’t get into the favoritism situation.”,
S: “I think that some people that I work with would prefer to go back to the old way, […] Because they maybe can get the shift they wanted more easily. Sometimes the fairness works for some, but not others. […] I work with who may felt they are getting less shifts now. Because they now have to go into a pool of being in the fairness with every body”
Category: Improved shared resources / improved staffing
M: “It exceeded our expectation in staff working on a unit they never work on before. I usually have needs listing on the board, and people wrote their name on it, and only my staff saw it. […] If they went up to 5C, they can see their needs. […] 5C never sign on mine. But now, they can”, “It opens it up to three campuses. […] I like the ability to staff from other hospitals. All the resources were there. ”
Category: Improved communication
M: “It eliminated me having to discuss with them (per diem). This is the shift we have […] so they just log on and take the available shifts. And it has worked out tremendously. I was skeptical initially, but I became a believer that it really does work.”
S: “If you are awarded, they send it right to your home email address. […] Then the only thing you need to do is to check the email. You don’t need to call anyone or talk to anyone. It just simplifies it.”

Note: M: manager; S: staff

Users considered Bidshift easy to use and easy to learn. However, some users also commented that they only use the simple functions that are needed for open shift management so they did not use “FAQ” or “Help” links. In general, users were very satisfied with Bidshift’s usability. A few experienced users identified interface issues that they considered to be cosmetic in nature.

4. Discussion

ISO 9241-11’s usability definition was useful in characterizing manager and staff perceptions of usability.

4.1. Mental Model and User Performance

All users were experienced Bidshift users, but varied their use according to task priority or their individual method for system use. This result is consistent with literature that suggests that users create their own mental models for interacting with target systems to perform tasks and that mental models vary among individuals. The mental model is formed through instruction or experience or both [9]. In this study, the mental model was primarily generated from experience since users were only provided with minimal guidance to use the system for the study tasks. Once a user’s mental model is built, it contributes to user performance. Because all users in this study were experienced users, it was easy for them to articulate their interaction with the system.

4.2. Effectiveness is the Reason for Use

Effectiveness is one of the indicators of usability, and was defined by ISO 9241-11 as “accuracy and completeness with which users achieve specified goals”. However, managers and staff did not use the term “effectiveness”. Instead, they expressed how often they used the certain functions in the system because those functions were useful for accomplishing their tasks. This is consistent with evidence that users’ perceptions of technology usefulness determine intention to use, which may lead to actual frequency of use [10, 11].

4.3. Effectiveness and Efficiency Determined Favorite Functions

The ISO 9241-11 concepts of effectiveness, efficiency and satisfaction are similar to the three concepts in the Technology Acceptance Model [12], perceived usefulness, perceived ease of use and attitude respectively. In this study, users identified their favorite functions (satisfaction) by the frequency of use (effectiveness) and the convenience of the function (efficiency). This is consistent with the TAM, which has demonstrated that perceived usefulness and perceived ease of use contribute to attitudes toward technology and predict user acceptance of new technology [12].

4.4. Experienced Users Explored Cosmetic Problems

All users expressed satisfaction with the web-based communication tool. Some very experienced users who had explored the breadth of system functions identified cosmetic issues in the interface that may impede their interaction with the system. This finding is consistent with another study that examined usability issues in expert users and novice users [13]. Novice users experienced general usability problems while experts had overcome general usability problems and focused on more cosmetic issues.

5. Conclusion

A web-based communication tool for managing open shifts, Bidshift, was well accepted by users. We found that patterns of system use varied among individuals during usability testing,. Their usage of each function depended on how effective they perceived the function to be in achieving a particular task. Users were satisfied when a function was both effective and efficient for accomplishing a task. Usability testing can identify potential barriers to technology use and targets for human-computer interaction improvement. Testing technology with intended users is critical for the design and refinement of tools to be used in nursing practice.