Comparison of Two User Interfaces for Accessing Context-Specific Information Resources Related to Hazards and Near Misses

Abstract

The Hazard and Near Miss Reporting System (HNMRS) was designed to promote patient safety mindfulness as part of a patient safety curriculum for Advanced Practice Nursing (APN) students. We are extending the functionality of the system beyond reporting to Just-in-Time learning by providing context-specific links to internal and external information resources related to the type of hazard or near miss reported. As part of this process, 55 APN nursing students compared two different interfaces on ease of use and reported their perceptions of usefulness and intention to use the information resources links integrated into the HNMRS. The students demonstrated a significant preference for the Category-based Interface as compared to the Question-based Interface (p< .001). Mean scores for perceptions of usefulness and intention to use the context-specific links in the HNMRS for reference purposes reflected moderate to strong agreement.

Introduction

“An adverse event results in unintended harm to the patient by an act of commission or omission rather than the underlying disease or condition of the patient”. ¹ Hazards and near misses are well recognized precursors of adverse events that can prevent future adverse events by improving the environment, procedure, or system ² Consequently, identification of hazards and near misses is an important component of creating high reliability organizations. Despite this important role in adverse event prevention and increasing patient safety mindfulness, hazards and near misses are typically under reported. To address this issue, we designed a web-based hazard and near miss reporting system (HNMRS) as part of a patient safety curriculum for Advanced Practice Nurses (APN) students in a combined BS/MS program. ³ In order to extend the functionality of the system beyond reporting to Just-in-Time learning, which allows students to access educational materials at the time of reporting or at another time when an information need arises, we are developing context-specific links (infobuttons) to internal and external resources related to hazards and near misses.

The purpose of this paper is to describe a usability study that compares two user interfaces for context-specific links within the HNMRS. As background, we first provide a brief overview of the HNMRS. Second, we summarize preliminary work related to the development of content for context-specific links. Third, we specify the rationale and process for development of the two user interfaces for the context-specific links: category-based Interface and Question-based Interface. Fourth, we describe the theoretical framework for the study, the Technology Acceptance Model (TAM).

Background

Hazard and Near Miss Reporting System

Our research team developed the web-based HNMRS based on the Medical Event Reporting System for Hospitals (MERS-TH) used for reporting adverse events and near misses at Columbia University Medical Center (CUMC). ⁴ The purpose of the HNMRS is to improve student’s patient safety mindfulness through identification and reporting of hazards and near misses during their clinical rotations. The HNMRS consists of two main questions:

• Q1: On your shift today, were there any "dangerous situations" that could cause a future event?

• Q2: On your shift today, were there any near misses (i.e., events that almost happened)?

If the answer to one or both questions is yes, the student classifies the hazard or near miss into one of 13 categories: Accident (non-fall), Environment Hazard/Safety, Equipment/Device, Fall, Food/ Nutrition, Infection, Laboratory, Medication, Patient Disappearance, Procedure/Treatment, Restraint, Transfusion, and Other (Figure 1). Students in the first year of a combined Bachelor of Science/Master of Science (BS/MS) program complete a report for each shift during five clinical rotations: Medical-Surgical, Childbearing Family, Pediatrics, Community, and Mental Health even if no hazards or near misses occurred. During a ten-week period in Autumn 2006, 156 students reported a total of 3086 hazards or near misses. ³

Figure 1.

Hazard and Near Miss Reporting System

Context-specific Links

A number of authors have identified nurses information needs in clinical practice ^{5, 6} and described the difficulties associated with accessing relevant information to meet those needs despite recognition of the importance of this task to evidence-based practice and patient safety.⁷ Context-specific links (i.e., infobuttons) have demonstrated utility in meeting information needs at the point of care. ^8–10 In the current version of the HNMRS, context-specific links are limited to provision of definitions of the categories of hazards and near misses.

As an initial step toward the development of context-specific links for Just-in-Time learning, we conducted a focus group with four students in the baccalaureate year of the APN curriculum to understand the students’ information needs related to hazards and near misses. ¹¹ Students were asked to provide information about the hazards and near misses they had reported and their information needs related to particular types of hazards and near misses. Students were also asked to express their information needs as questions which were then analyzed and organized into the thirteen categories. ¹¹ Questions were also classified into three information types: (S) subject-specific (e.g., what is my patient’s laboratory test result?), (D) domain-specific (e.g., what are the potential side effects of this medication?), or (I) institution-specific (e.g., what is the institutional standard of care for pressure ulcer prevention?). ¹² Students experienced information needs related to hazards and near misses, such as “How do I use this type of device?”, “What is the institutional protocol to do this procedure?”, and “How do I educate my patient about the procedure?” Context-specific links to reliable and valid institutional or external information resources may help to prevent future adverse events through Just-in-Time learning at the time a hazard or near miss is reported. ^{1, 11}

Interface Development

As the next step in transforming the HNMRS to support Just-in-Time learning, we want to implement a user interface for context-specific access to information resources related to hazards and near misses that users would perceive to be easy to use, useful, and would intend to use. Consequently, we developed two different interfaces (Category-based Interface and Question-based Interface) for accessing context-specific information resources related to hazards and near misses.

Category-based Interface is a common interface structure. It categorizes objects first and then leads to questions associated with the object (Figure 2). This results in a flow of General Category → Specific Category → Questions → Resources.

Figure 2.

Category-based Interface

The Question-based Interface (Figure 3) narrows the number of objects by filtering by question first. This results in a flow of General Category → Questions → Specific Category → Questions → Resources.

Figure 3.

Question-based Interface

Theoretical Framework: Technology Acceptance Model (TAM)

Technology Acceptance Model (TAM) (Figure 4), introduced by Davis ¹³, models users’ acceptance and usage of a new technology. The model suggests that perceived usefulness and perceived ease of use influence users’ attitudes toward using the technology and contribute to their behavioral intention to use the technology. TAM has been applied in healthcare to understand clinicians’ technology adaptation in healthcare environment and evidence regarding its usefulness to predict clinicians’ acceptance of a new technology is mounting.^14–18

Figure 4.

Technology Acceptance Model (TAM) by Davis 1989

The purposes of this study were to compare APN students’ ease-of-use ratings for two user interfaces for accessing context-specific information resources related to hazard and near misses and to assess their perceptions of usefulness and intention to use the context-specific links to information resources within the HNMRS.

Methods

Research Questions

1. Do APN students perceive the Category-based Interface or the Question-based Interface for accessing context-specific information resources related to hazards and near misses as easier to use?

2. Do APN students perceive context-specific links to information resources within the HNMRS as useful?

3. Do APN students intend to use context-specific links to information resources within the HNMRS??

Study Design

A descriptive, correlational study was used to compare the interfaces. Students used both interfaces and rated their preferences. To eliminate the potential effect of testing order, students were randomized to first receive either the Category-based Interface or the Question-based Interface.

Sample

One hundred and fifty eight students in the first year of the BS/MS program were invited via email to participate in the user interface evaluation study. The institutional review board approval for the evaluation of the patient safety curriculum covered assessments related to the ease of use, perceived usefulness, and intention to use informatics tools. Respondents were entered into a lottery for $100 gift card in which the odds of winning of 1:31. Those who volunteered were randomly assigned into two groups that determined the order in which they evaluated the two user interfaces. The same scenario (Figure 5) was given to both groups prior to the review of the two interfaces. Group 1 first reviewed the Category-based Interface and then reviewed the Question-based Interface; Group 2 reviewed the interfaces in the opposite order.

Figure 5.

Scenario

Questionnaire

Perceived Ease-of-use, Perceived Usefulness and Intention to Use constructs in the TAM ^{19, 20} were selected as variables for evaluating the usability of the system. Perceived ease-of-use questions were used to assess which interface (Category-based Interface or Question-based Interface) APN students perceived as easier to use. Four additional questions assessed their perceptions of usefulness and intention to use the links to context-specific information resources integrated into the HNMRS.

Perceived Ease of Use was measured with three 7-point Likert scale items: clear and understandable, easy to use, and mental effort required. The scale was chosen to encourage the APN students to distinguish which interface best met the characteristic.

Perceived Usefulness and Intention to Use were assessed from two perspectives - at the time of reporting hazards and near misses and at other times for reference purposes. Each of the four items was assessed using a 7-point Likert scale (1 = strongly disagree; 2 moderately disagree; 3=somewhat disagree; 4 neutral (neither disagree nor agree); 5 = somewhat agree; 6 = moderately agree; 7 = strongly agree). One open-ended question was asked to gather general comments.

Data analysis

The Mann-Whitney U test was performed to examine if there was any order effect between two groups related to which user interface they evaluated first. Nonparametric tests were performed to analyze the three Ease-of-Use questions that compared the two user interfaces. Prior to the sign tests, the data were transformed to a 1 to 7 scale, where 1 represented the strongest preference for the Category-based Interface, 7 represented the strongest preference for Question-based Interface, and 4 indicated no preference. Because of the three related comparisons, alpha was set at p<.017 to control for galloping alpha. Data related to Perceived Usefulness and Intention to Use were summarized using descriptive statistics. All analyses were performed using Microsoft Office 2003 Excel StatPlus Macro and SPSS v14.0. Answers to the open-ended question were analyzed based on the three major TAM constructs: ease of use, usefulness, and intention to use.

Results

Fifty-nine of the 158 students agreed to participate and 55 students completed the evaluation (Figure 6). A total of 23 comments were written in response to the open-ended questions. Using alpha of p<.05, there were no significant effects related to which user interface was first assessed by the student: clear and understandable (p= .435); easy to use (p=.722), and mental effort (p=.835). The description of study findings in the following paragraphs integrates both quantitative ratings and qualitative comments.

Figure 6.

Randomization flowchart

Ease of Use

A series of sign tests indicated that APN students’ rated the Category-based Interface as significantly easier to use (p<0.17) than the Question-based Interface (Table 1) on all three items. Student comments supported their preference for the Category-based Interface ‘I preferred the category-based version, it’s more user-friendly’, ‘…the questions interface made it somewhat confusing’.

Table 1.

Comparison of Perceptions of Ease of Use of Two User Interfaces (N=55)

	Preference	No	Preference	Median	p-value	Lower	Upper
	for CBI	preference	for QBI			95%	95%
Clear and understandable	31	15	9	2	0.001	1.71	4.00
Easy to use	37	11	7	3	0.000	2.00	3.00
Mental effort	30	18	7	3	0.000	2.00	4.00

CBI = Category-based Interface; QBI = Question-based Interface.

Ho: Median = 4 (No preference); Ha: Median ≠4 (There is preference)

Usefulness

Mean scores for usefulness indicate that students somewhat agreed that the context-specific information resource links are useful at the time of reporting hazards and near misses and somewhat agreed to moderately agreed that the information links are useful at other times for reference purposes (Table 2). Comments positively supported the idea of context-specific information resources: ‘I like the information resources idea. I think that it would be very helpful in clinical practice’. ’When I KNOW a situation is hazardous, I don't always know how to remedy it. I think this is the information I need to resolve these questions.’

Table 2.

Perceived Usefulness and Intention to Use (N=55)

	Mean	Median	SD
Usefulness at the time of reporting hazards and near misses	4.93	5.00	1.54
Usefulness at other time for reference	5.56	6.00	1.42
Intention to use at the time of reporting hazards and near misses*	4.37	5.00	1.78
Intention to use at other time for reference	5.36	6.00	1.58

1 = strongly disagree; 2 moderately disagree; 3=somewhat disagree; 4 neutral (neither disagree nor agree); 5 = somewhat agree; 6 = moderately agree; 7 = strongly agree

^*N=54 due to missing data.

Intention to Use

Mean scores for intention to use suggest that students were about neutral (neither disagree nor agree) in regards to use at the time of reporting hazards and near misses, but somewhat agreed that they intended to use the context-specific information resource links at other times for reference purposes (Table 2). The quantitative results were consistent with the general comments that indicated that the some students would prefer to use the HNMRS with context-specific links to information resources as a teaching tool after clinical practice because they were concerned about not having sufficient time or computer access during clinical practice: ‘I'm not sure I would always have the time as a nurse on the floor to use the interface for additional information. Sometimes there aren't enough computers. Sometimes someone calls my attention to something more immediate’. Other comments supported the likelihood of using it on the clinical unit: ‘I would be more likely to use it when I needed the information right away’. ‘I'm more likely to look up this info while I am in clinical, rather than when I return home and enter the info into the hazard and near miss reporting system’.

Discussion

The APN students perceived the Category-based Interface as clearer to understand, easier to use and requiring less mental effort than the Question-based Interface. This is likely due to students’ familiarity with Category-based Interfaces because of their wide use in Web sites and book indexes. Moreover, the Question-based Interface required one more step in accessing context-specific information resources than the Category-based Interface.

Usefulness scores suggest that the APN students believed that access to context-specific information resources would be more useful at times other than the time of reporting hazards and near misses. Qualitative comments indicated that they might choose to access the information from home for reference purposes or in the clinical setting. Findings related to intention to use followed a similar pattern. This is consistent with other research that documented that due to time constraints, nurses sought information only for specific tasks or problems and were more likely to consult other clinicians because of time efficiency. ^{21, 22}

There is a high demand for reliable and up-to-date information resources to support nurses information needs ²³. A strategy such as the one we proposed and evaluated in this study, providing context-specific access to up-to-date and reliable information, has the potential to meet the demand for efficient access to high-quality resources.

There were several limitations to this study. First, only two interfaces were tested. It is possible that another interface would be more positively evaluated by the students. Second, the response rate for the survey was only 34.8% and there exists the possibility that the respondents are not representative of the population of HNMRS users. Third, the sample comprised one group of potential users (students in the first year of their BS/MS program). Other types of users or users at a different spot in their curriculum might differ in preference for the structure of the interface, perception of usefulness, and intention to use.

Conclusion

Technology that is perceived to be easy to use and useful is more likely to be used. In this study, participants preferred the Category-based Interface finding it easier to understand, easier to use, and requiring less effort. Moreover, they agreed that context-specific links to information resources would be at least somewhat useful. By extending the functionality of the system beyond reporting to Just-in-Time learning through providing context-specific links to internal and external information resources related to the type of hazard or near miss reported, there is a potential for students to improve knowledge and behaviors related to prevention of adverse events.