Usability Testing of a Digital Pen and Paper System in Nursing Documentation

Abstract

Usability testing was used to evaluate whether a new technology, a digital pen and paper system, would be usable for hospital nurses. Twenty-one nurses in a Labor and Delivery unit were randomly assigned into two groups, and a crossover design was used to compare the digital pen and paper system to conventional pens. Data collection included observations, interviews, and a questionnaire. Results showed that nurses had a positive attitude toward the system and could foresee its potential benefits, but they found that in its current design the system had poor usability and interfered with nurses’ work practices. Usability testing provided important insight into the needs of nurses and the suitability of this technology. This study is an example of how a user-centered approach can improve our understanding of the real needs of nurses and contribute to the design of useful and usable technologies for healthcare.

Introduction

Nursing information systems help nurses to organize their work, manage care plans, track diagnoses and outcomes, and support decision making. All of these functions depend critically on the absolute accuracy of data entered in the electronic health record. Therefore, nursing documentation and data entry are essential first steps to quality nursing care (1). Computer support for data entry, however, may be problematic for many clinicians. The goal of this research is to better understand the interaction of nurses with clinical information systems, focusing on technologies for nursing documentation or data entry.

Existing technologies for documentation and data entry are problematic. Most interaction with computers requires use of a keyboard and mouse, but these devices are poorly suited to bedside nursing. As a result, paper and pen continue to be preferred tools for recording clinical data in the majority of clinical settings. It is commonplace to see clinicians carrying paper and pen while collecting data on the wards, only to transfer this information into another information system, resulting in duplication of effort and potential for errors.

A number of new technologies have recently been developed for clinical documentation. These include barcode systems; speech recognition systems; and pen-based systems, including scanning technology, Tablet PCs, and Personal Digital Assistants (PDAs). Although helpful for medication management (2), barcode systems do not support free text to allow rich clinical data. Speech recognition systems work well where vocabularies are limited and dictation tasks are performed in isolated, dedicated workspaces, such as radiology or pathology (3), but are much less suitable in noisy public spaces, where performance is poor and the confidentiality of patient health information is threatened. Tablet PCs are limited by their weight and fragility, which reduce their acceptability to nurses (4, 5). PDAs, for most users, are not sufficiently easy for typing or writing (6). Scanning technology has the significant limitation of delays from document creation to availability, reducing the potential benefits (7), and without optical character recognition, other benefits such as search, retrieval, and analysis of data can not be realized (8).

Digital pen and paper system

The digital pen and paper system is a new technology which may address these limitations. The digital pen contains a camera to record the writing pattern on the digital paper, which contains an array of 0.3mm dots printed with slightly off-white color. Dots enable the camera to track the location of the digital pen relative to the paper within a virtual space equivalent to 60,000,000 km² (for comparison, Earth total land area is 150,000,000 km²). As the user writes with ink on the digital paper, a digital representation is created in the memory of the digital pen. Using a cradle, this representation can then be transferred to a computer at a later time. Thus, a paper representation and an electronic representation are created at the same time. One such digital pen and paper system is that developed by Anoto, a Swedish firm (9), available in the US through Logitech among others.

Usability testing in Nursing

According to Staggers, nursing computer systems should be designed with an understanding of human-centered attributes and cognitive behaviors so that human computer interaction seems natural, achieving the goal of computers as “assistants” to humans (10). While usability is a user-centered design attribute, the human-computer interaction is not the sole evaluation focus. It would be more accurate to say, as Bennett has proposed, that usability is developed through user-centered design methods but measured through the interaction of four components: user, tool, task and environment (11). In the present study Staggers’ model (12) has been extended by combining it with Bennett’s model to reframe the definition of usability: usability is a set of relationships among user, task, tool and environment, each with its own attributes, interacting with and interdependent upon each other.

The present study examines usability of a documentation device, the digital pen and paper system, using this broader, more comprehensive approach which includes users, tools, tasks and the actual working environment.

Hypotheses

Conventional pens are the natural comparison for the digital pen and paper system. To be successful, usability of the system must be comparable to conventional pens. This can be stated in the form of the following hypothesis: Hypothesis 1: There is no preference for digital or conventional pens. The study uses a crossover study design which introduces the possibility of recall bias due to the timing of exposure of the intervention. To test for this bias, a second hypothesis can be stated: Hypothesis 2: There is no time effect. The perceptions of subjects with early exposure to the intervention will not differ from the perceptions of subjects with later exposure.

Methods

Three strategies were used to obtain complementary information: observations, a crossover study using questionnaires, and interviews. Observations provide objective information and greater insight into the work processes. Questionnaires provided data for quantitative comparison. Interviews provide the informed perceptions of insiders, but are reported separately (Yen, P unpublished data).

The user

The population of interest in the study is hospital nurses. A convenience sample of 21 volunteers was recruited on a Labor and Delivery unit (accessible to the investigators), with an attempt to include representation from each shift. Nurses were randomly assigned to Group 1 using the digital pens for the first four weeks, then conventional pens, or Group 2 with exposure in reverse order.

The tool

The intervention in this study was the use of a digital pen and paper system (io Digital Pen, Logitech, Inc), adapted to meet requirement of the Labor and Delivery unit documentation. Nurses were provided with their own pens for the duration of a shift to reduce the need for interruption to obtain or share the device.

The task

Completion of the “Admission Database” form was chosen as the task for evaluating the digital pen and paper system, because it is a common task, a familiar and brief form, and is always completed by a single nurse rather than multiple nurses. The digital version of the form was virtually identical to the original, preserving the nurses’ familiarity with its format and content.

The task environment

The study was conducted in a moderately busy Labor and Delivery unit in an academic medical center with a diverse patient population, serving many patients with comorbid conditions threatening delivery.

Data collection and analysis

Instrument 1: Questionnaire

The questionnaire combined items from two existing and validated instruments, the Technology Acceptance Model (TAM) (13), and the Perceptions of Adopting an Information Technology (PAIT) (14) based on the work of Moore (15). The final version included 19 seven-point Likert scale items, reworded to reflect two directions of preference.

Instrument 2: Observer/Interviewer

Observations and the interviews were conducted by a trained nurse with graduate training in health informatics. Observations focused both on the experimental task and on the interaction of nurses with the device as they performed other duties. Field notes were transcribed within 48 hours. More than 110 hours of observations were conducted during some 200 visits.

The questionnaire provided ordinal preference data on a scale from one to seven, with zero indicating no preference. The data were analyzed using the Wilcoxon test (SPSS v10.0). Content analysis was used to analyze the qualitative data. Each transcript was read by the investigator, making note of themes and patterns. As additional transcripts were examined, similar themes were combined with attention to subjects’ statements relating to usability. Selected transcripts were reexamined to confirm the final themes reported below in Results.

Results

Table 1 shows the distribution of subjects across the three shifts in Groups 1 and 2. Most subjects work at least 0.7 full time equivalents suggesting that they had ample opportunity to use the system.

Table 1.

Shifts

	Group 1	Group 2	Total
N	10	11	21
Day shift	7	8	15
Evening shift	2	2	4
Night shift	1	1	2

Questionnaire Data

Questionnaire items were clustered according to factors (usefulness, ease of use, compatibility, and attitude) by calculating the within-subject means of the items associated with each factor. Figure 1 Shows the distribution of scores for the four scales.

Figure 1.

Histograms of four scales

Hypothesis 1: No Preference for Either Pen

Wilcoxon test results for the 19 items indicated a significant preference for conventional pens. Most of these items belong to the scales for “ease of use” and “compatibility”. Result for a composite of the 19 items combined also showed a statistically significant preference for conventional pens.

Table 2 shows the Wilcoxon test results for the four scales: usefulness, ease of use, compatibility, and attitude. Consistent with the individual item data, a statistically significant preference was found for the “ease of use” and “compatibility” scales. Although inspection of the raw data suggested a preference for digital pens on the “attitude” scale, this was not statistically significant.

Table 2.

Wilcoxon test results for the four scales

Scale	z scores*	p value
Usefulness	−1.410a	.159
Ease of use	−3.323a	.001
Compatibility	−2.661a	.008
Attitude	−2.262b	.793

^*a: trend toward conventional pen;

b: trend toward digital pens

Hypothesis 2: No Time Effect Between Two Groups

Group 1 and Group 2 data were compared using the Wilcoxon test for all 19 test items, a composite of the 19 items, and the four scales, with alpha set at 0.05. In none of these cases was a statistically significant difference found between the two groups.

Observations – Field Notes

Initial Excitement

In the beginning, most nurses appeared excited about using the digital pens. It generated conversation and curiosity, among those involved in the study and those not involved. Most nurses initially had a positive attitude about using the new technology.

Adopting for Routine Use

Nurses began to handle the digital pen like a conventional pen. For example, during conversation they would remove and replace the cap unconsciously as with a conventional pen. Similarly, nurses used the digital pen as a pointer, pointing to the white board, the computer screen, etc. It appeared to have been assimilated as a routine tool.

After the Honeymoon Period

After an initial positive response in the first few days, nurses began to use conventional pens for most tasks, reverting to the digital pen only for completion of the Admission Database form. Some explained that they simply reached for the most accessible pen.

Making Adjustments

Preferences began to appear after about a week. Several nurses used conventional pens at all times except when completing the Admission Database form. One acknowledged this, stating she used her digital pen only when she had to, otherwise, she would not use it. The reason: “it’s bulky”. Other nurses adopted it into routine use, using it as they would a regular pen. For most, the frequency depended upon the situation: if they were busy, they just grabbed the most accessible pen. Some made adjustments to use the digital pen: for example, while nurses kept a conventional pen in their front shirt pocket for easy access, the bulk and the weight of the digital pen made this impractical. Their work requires frequent bending, stooping, and leaning, so the pen would easily fall out of a shirt pocket and be damaged. As a result, nurses adjusted their behavior, placing the digital pen in their back pocket, though this is less convenient location to access. Some nurses adjusted their behavior even further. When carrying the digital pen, these nurses took care not to have a conventional pen with them. They said that they “quit” conventional pens to force themselves to use the digital pens at all times.

Discussion

Products designed for different users

It is not surprising that digital pens were not fully acceptable to nurses, since the system was designed as a personal assistant, not a clinician assistant. The disadvantages found here may be not problems for other uses, such as desk-bound office tasks. However, for nurses, the device proved to be a distraction, potentially interfering with patient care which requires full attention.

User acceptance of new technology

Interestingly, nurses’ attitudes toward the digital pens were positive even in spite of poor usability in the current design. Some noted the advantage that the digital pen could be used as a conventional pen. In general, nurses in this study saw the potential advantages of using technology, including the Electronic health records (EHRs) for patient care. Familiarity and training may increase users’ acceptance (16–21), even when, as in this case, the tool ultimately did not prove suitable for their work.

Potential issues of digital pens

Aside from usability, other issues must be addressed for this technology to be deployed, including the integrity of data transfer, system robustness during downtimes, and the quality of character recognition, which were not evaluated in this study.

Bennett’s Model

The study findings are consistent with Bennett’s Model of usability, which takes into account user, tool, task, environment, and the interactions among them. A change in any element will influence others (Figure 2). For example, digital pens change users’ habits, such as the nurses’ writing behavior, and also the way they usually work with conventional pens. Digital pens change the tasks, such as the requirement to remember to download data after completing the form. Digital pens will require changes in the environment because of the necessity for widespread installation of pens, cradles, etc. Thus, usability cannot be assessed by examining only one or two components in the model. All of them contribute to usability, and a change in one will effect changes in others (Figure 3). Ammenwerth describes a similar model combining the Technology Adoption Model (TAM) of Davis, the Information Technology Adoption Model (ITAM) of Dixon and the Task-Technology-Fit Model (TTF) of Goodhue and confirmed by a case study (22).

Figure 2.

Bennett’s Model

Figure 3.

Fishbone diagram in Usability

Limitations

There are some limitations due to the methodology of the study. First of all, subjects in the study were volunteers who may not be representative of the general population. Similarly, the study was held in only one unit, which also may limit generalizability. Second, since there are no previous studies of this device, there is no opportunity for comparison. Third, the study only examined a single task. It is possible that we would have different results with other tasks. Fourth, observations were made by a single observer, a potential source of bias. Fifth, this was a partial and temporary implementation. There may be problems in full implementation that we can’t predict now.

Implications for nursing practice and nursing informatics

These results are similar to those of Allen’s in 1998. Even though nursing records are valuable for nursing process, the attitudes of nurses toward nursing documentation are different among nurses. Nurses may not realize directly the full benefit of documentation with paper records because of limitations in availability and retrieval. As a result, nursing documentation becomes not a means to improved care as much as a routine chore (23). Where evidence-based practice is concerned, nursing records can be a most powerful resource for discovering relationships between nursing interventions and patient outcomes. Without electronic nursing records, it is neither feasible nor affordable to make full use of clinical nursing data for quality improvement or research. Nurses in this study already foresee the potential advantages of computer documentation. With full implementation and direct experience with the added benefits to patient care, research, and quality improvement, it is expected that nurses’ acceptance of this technology will be still greater.

Conclusion

Electronic health records (EHRs) are increasingly a routine part of clinical practice. New devices for interacting with information systems in clinical settings can help clinicians and patients realize the full benefits of the EHR. But first, these devices must be compatible with the work environment, tasks, and goals of clinical practice. The nurses in this study showed a positive attitude toward the digital pen and paper system, but ultimately rejected it due to the problems with usability. Nurses must be able to choose the devices most suitable to their workstyles, tasks, experience and abilities. A suitable spectrum of input devices would allow nurses whose typing is faster to use a keyboard as desired, while others whose writing is faster could use a pen-based system. By supporting nurses’ preferences and abilities, nursing documentation would be less of a burden and the benefits of the EHR could be more fully realized.