Usability of Emergency Department Information System Based on Users' Viewpoint; a Cross-Sectional Study

Abstract

Introduction:

The emergency department is of special importance due to its emergency and vital services, the high volume of referrals, and the patients' physical condition. Thus, it requires a well-designed information system with no usability problems. This study aimed to evaluate the usability of the emergency department information system from users' perspectives.

Methods:

This was a cross-sectional study. The research setting was the emergency department of 3 hospitals. The research instrument was a 37-item questionnaire adapted from the USE and ISO Metrics questionnaires, consisting of five dimensions measuring the usefulness of the system, ease of use, ease of learning, user satisfaction, and suitability for the task. The content validity of the questionnaire was examined using the content validity ratio and content validity index, and its reliability was assessed using Cronbach's alpha (α = 0.88).

Results:

Fifty questionnaires were administered in the three hospitals, and the response rate was 80%. According to the findings, 55% of the respondents were female. The highest mean scores belonged to usefulness in emergency department information system (EDIS) A, ease of use in EDIS B, ease of learning in EDIS A, user satisfaction in EDIS C, and suitability for the task in EDIS A. According to the usability evaluation criteria, ease of learning (3.66 ± 0.74), usefulness (3.53 ± 0.87), and suitability for the task (3.47 ± 0.96) received the highest scores, and the lowest scores belonged to user satisfaction (3.29 ± 1.01) and ease of use (3.12 ± 1.00).

Conclusion:

In terms of usability criteria, the emergency department information system is at a relatively good level. The usability of these systems can be further enhanced by considering the users' working needs, improving software flexibility, customizing the software, using data visualization tools, observing consistency of features and standards, and increasing the quality of information and system services.

1. Introduction:

The emergency department is characterized by several major features, including unpredictable admission, patients' physical condition, and time constraints to perform diagnostic-therapeutic processes (1-4). These challenges, in turn, result in irrecoverable consequences such as increasing patient mortality, heavy costs, increasing waiting time and patient dissatisfaction, increasing medical errors and the incidence of violence, and disrupting medical services(5-7). Therefore, compared with other departments, the emergency department requires considerable flexibility and instant planning of resources(8).

Persistent collection, and correct and timely processing of data may improve emergency department management (9). Emergency department information system (EDIS), a component of the hospital information system, plays a key role in information management and care, as well as management processes of the emergency department (10-12).

The EDIS enjoys numerous advantages such as improving emergency department performance (13), providing easier and faster access to patient information (14), recording much better and more accurate clinical and managerial information, assisting in decision-making (14), and eliminating the limitations of paper systems such as multiple users' simultaneous access to information and information illegibility (15). Despite its increasing use in recent years in different countries (16), the EDIS is still not widely adopted and applied (17-19). In some studies, poor display, a lack of workflow support, reduced efficiency of the emergency department, and congestion have been reported as the main disadvantages (20, 21).

Given the vital importance of the emergency department's activities, the emergency department's information systems must be free of usability problems to prevent errors. In doing so, systems and applications must be designed and used appropriately in accordance with scientific principles (22). One method to ensure the proper design of applications and health information systems is evaluating their usability. Usability deals with various features of the software, including ease of learning, efficiency, ease of use, memorization, error prevention, and user satisfaction. Evaluation also plays a pivotal role in software development (23, 24). According to various studies, it is necessary to observe usability principles in designing the user interface of the EDIS (25, 26). The usability index evaluates the performance of a product in terms of user satisfaction and increased productivity (27-30). A common method of evaluating the applicability of information systems is using the standard 9241/10 ISO Metric Questionnaire and the USE Questionnaire, with approved validity and reliability (30, 31). Several studies have been conducted on the evaluation of hospital information systems’ subsystems in military hospitals. Based on results, the same maturity was observed in military and civilian hospitals, i.e., the third stage of the EMRAM model. Meanwhile, the potential benefits of these systems were not yet fully exploited in hospitals (32, 33).

The present study aimed to evaluate EDISs from the perspective of users in five dimensions of usefulness of the system, ease of use, ease of learning, user satisfaction, and suitability for the task.

2. Methods:

2.1. Study design and setting

This was a cross-sectional study conducted in 2021 in three hospitals affiliated with AJA University of Medical Sciences, Tehran, Iran, which employed an EDIS. The questionnaires were distributed from 6 to 15 November 2021. The data were collected five days later, from 20 to 27 November. The research setting was the emergency departments of hospitals affiliated with AJA University of Medical Sciences.

Multi-stage sampling was carried out. The goal was to include the emergency department of hospitals with 100 or more active beds, which led to the selection of the three hospitals. The emergency department system in two hospitals had been designed by two software companies, and in the third hospital, it had been developed by the hospital. The population comprised of users of EDISs (emergency department nurses and secretaries).

A sample of 50 was selected from the mentioned centers through convenience sampling. To maintain confidentiality, the identities of vendors and hospitals remained anonymous and they were labeled as EDIS A, EDIS B, and EDIS C.

This study was approved by the Ethics Committee of AJA University of Medical Sciences, Iran (IR.AJAUMS.REC.1400.215).

2.2. Participants

The inclusion criteria were as follows: having > five years of work experience and 1 year of experience with the EDIS. Fifty emergency information system users met the aforementioned criteria in the three hospitals.

2.3. Data collection

To evaluate the usability of EDIS, the standard 9241 ISO Metric Questionnaire (part 10) and the USE Questionnaire were administered (appendix 1). The 9241 ISO Metric Questionnaire consists of 75 questions based on seven principles: suitability for tasks, self-description, controllability, error tolerance, suitability for personalization (customization), and suitability for learning (34). The validity and reliability of the 9241 ISO Metric questionnaires have been confirmed in various studies (30, 31). The USE Questionnaire includes 30 questions that assess the following dimensions: usefulness, satisfaction, ease of use, and ease of learning (31). The validity and reliability of the USE Questionnaire have also been confirmed in a number of studies (35, 36). The questionnaire administered in this study was a combination of the two mentioned questionnaires, which covered five dimensions of system usefulness, ease of use, ease of learning, user satisfaction, and suitability for the task. The questionnaire encompassed a total of 37 questions on a five-point Likert scale ranging from completely disagree (score: 1) to completely agree (score: 5). The number of questions for each dimension was as follows: usefulness of the system (eight questions), ease of use (10 questions), ease of learning (six questions), user satisfaction (seven questions), and suitability for the task (six questions).

Table 1.

Baseline characteristics of the respondents

Variable	Number (%)
Gender
Male	18 (45.0)
Female	22 (55.0)
Education Level
High-school diploma	3 (7.50)
Associate degree	2 (5.00)
Bachelor’s degree	28 (70.00)
Master's degree	5 (15.00)
Doctoral degree	1 (2.50)
Occupation
Nursing supervisor	3 (7.50)
Nurse	26 (65.00)
Secretary	11 (27.50)

Experts determined the content validity of the questionnaire using the content validity ratio (CVR) and content validity index (CVI), and its reliability was assessed using Cronbach's alpha (α = 0.88). Then, the questionnaire was administered to 50 system users.

2.4. Statistical analysis

We employed the SPSS 22 software (IBM SPSS, Armonk, NY, USA) for data analysis. Descriptive tests (mean, percentage, and frequency) were used to assess the data. For data analysis, the means and standard deviations (SD) were calculated, and then, the systems were compared against usability criteria.

The relationship between the dimensions of usability and participants’ variables was assessed using Pearson correlation coefficient or Spearman correlation coefficient.

3. Results

Baseline characteristics of participants

Fifty questionnaires were administered in the three hospitals (response rate = 80%). The baseline characteristics of the respondents are given in Table 1. 55% of the respondents were female and 70% had a bachelor's degree. The mean age of the respondents was 29.8 ± 8.7 years. Also, their mean work experience and mean duration of involvement with the EDIS was 8 ± 5.1 and 7.1 ± 4.3 years, respectively.

3.1. Usefulness

The highest mean scores for this criterion were reported for EDIS A (3.59 ± 0.78), EDIS B (3.54 ± 1.12), and EDIS C (3.48 ± 0.73), respectively.

In EDIS A, the highest score belonged to the item "EDIS saves time (3.92 ± 0.49)", and the lowest score belonged to “EDIS meets your needs (3.3 ± 0.94)". As for EDIS C, the highest score was associated with "EDIS is useful and valuable (4 ± 0.42)", and the lowest score was related to "EDIS meets your needs (2.83 ± 0.93)" (Table 2).

Table 2.

Mean score of emergency department information systems (EDIS) usability criteria in three hospitals (A, B, and C)

Usability criteria					Systems
Suitability for the task	User satisfaction	Ease of learning	Ease of use	Usefulness
3.53±0.94	3.30±1.08	3.77±0.73	3.10±1.05	3.59±0.78	EDIS A
3.41±1.01	3.26±0.98	3.54±0.81	3.13±1.07	3.54±1.12	EDIS B
3.49±0.95	3.33±0.97	3.67±0.70	3.13±0.90	3.48±0.73	EDIS C
3.47±0.96	3.29±1.01	3.66±0.74	3.12±1.01	3.53±0.87	Total

Data are presented as mean ± standard deviation.

1. Ease of use

The highest mean scores for this criterion belonged to EDIS B (3.13 ± 1.07) and EDIS C (3.13 ± 0.90), respectively, while the lowest scores were reported for EDIS A (3.10 ± 1.05).

In EDIS B, the highest score belonged to "Using the EDIS does not necessarily require much effort (3.53 ± 1.06)", whereas the lowest score pertained to “Using the EDIS, I can quickly and easily recover errors (2.8 ± 1.26)". As for EDIS C, the highest score was associated with "EDIS is user-friendly (3.66 ± 0.65)" and the lowest score was related to "Using the EDIS, I can quickly and easily recover the errors (2.83 ± 0.83)" and "EDIS is flexible (2.83 ± 0.83)". With regard to EDIS A, the highest score belonged to "Using the EDIS is easy and simple (uncomplicated) (3.23 ± 0.83)" and the lowest score pertained to "I noticed no inconsistencies when using the EDIS (2.76 ± 1.02)".

1. Ease of learning

For this criterion, the highest mean scores were obtained by EDIS A (3.77 ± 0.73), EDIS C (3.67 ± 0.70), and EDIS B (3.54 ± 0.81), respectively.

In EDIS A, the highest score was related to "I will learn to work with the software quickly (4.00 ± 0.49)" and "If I do not use the emergency information system for a long time, I will most likely re-learn and re-use it easily (4.00 ±0.57)". As for EDIS B, the highest score belonged to "I quickly learn the required skills to use the EDIS (4.00 ± 0.53)" and "I quickly learn to work with software (4.00 ± 0.53)", whereas the lowest score pertained to "In order to use the EDIS correctly, you have to remember a lot of details (2.60 ± 1.05)".

Regarding EDIS C, the highest score was associated with "I learn to work with software quickly (4.25 ± 0.45)" and the lowest score was related to "Learning the principles and instructions of working with software is easy (3.25 ± 0.75)".

1. User satisfaction

For this criterion, the highest mean scores were obtained for EDIS C (3.33 ± 0.97), EDIS A (3.30 ± 1.08), and EDIS B (3.26 ± 0.98), respectively.

In EDIS C, the highest score was related to "I feel that such a system is needed (3.66 ± 0.77)" and the lowest score was associated with "The EDIS works as expected (2.91 ± 0.90)". Regarding EDIS A, the highest score was related to "I feel that such a system is needed (3.69 ± 1.01)" and the lowest score pertained to "Using the emergency information system is fun (3 ± 1.15)". In the case of EDIS B, the highest score was allocated to "I feel such a system is needed (3.77 ± 0.77)" while the lowest score was reported for "The Emergency Information System is excellent (2.8 ± 1.08)".

3.5. Suitability for the task

For this criterion, the highest mean scores were obtained for EDIS A (3.53 ± 0.94), EDIS C (3.49 ± 0.95), and EDIS B (3.41 ± 1.03), respectively.

In EDIS A, the highest score belonged to "Software outputs (reports) are suitable for user tasks (3.92 ± 0.95)" and the lowest score was related to "Screen fields are fitted to user tasks (3.15 ± 0.89)". As for EDIS C, the highest score was related to "The information required by the user is available on the screens (3.66 ± 0.88)" while the lowest score was related "EDIS provides support for users' daily activities (3.16 ± 1.02)".

As for EDIS B, the highest score pertained to "EDIS provides support for users' daily activities (3.66 ± 0.81)" and the lowest score pertained to "It is possible to adjust the display of results (information) to the user's needs (3.26 ± 1.16)".

1. Relationship between usability criteria and participants’ characteristics

With increase in the level of education, age, work experience, or experience of working with the system, user satisfaction with each dimension of system usability declined. Evidence also shows that there is a stronger correlation between job and satisfaction with usability than between other variables (table 3).

Table 3.

Relationship between the usability criteria of the emergency department information system (EDIS) and baseline characteristics of participants

Variable	Usefulness	Ease of use	Ease of learning	User satisfaction	Suitability for task
Education Level
CC	0.01	-0.03	0.03	-0.04	-0.02
P-value	0.92	0.81	0.84	0.79	0.90
Occupation
CC	0.12	0.14	0.27	0.21	0.16
P-value	0.42	0.36	0.08	0.18	0.32
Age
CC	-0.20	-0.37	-0.20	-0.36	-0.23
P-value	0.19	0.01	0.20	0.02	0.14
Work experience
CC	-0.32	-0.28	-0.30	-0.34	-0.33
P-value	0.03	0.07	0.05	0.02	0.03
Experience with EDIS
CC	-0.24	-0.34	-0.12	-0.24	-0.21
P-value	0.12	0.03	0.43	0.12	0.17

CC: correlation coefficient.

4. Discussion:

Based on the findings of this study, the ease of use criterion received the lowest score among the criteria. Consistency in the displayed information and quick recovery of errors received the lowest scores as the sub-criteria of ease of use.

In the study by Khajouei et al. (37), the most important recommendations for creating consistency and standards were: arranging numbers, fields, menu items, and the cursor equally and according to the standard on all pages; having an active exit button on each window; and using the same icon for buttons having the same function. As for the feature of helping users identify and correct errors, the most important recommendations were: presenting all the messages in one language (Persian) with accurate and consistent grammar, without using exclamation marks, and in positive statements. Indicating the severity of the error, the cause of the problem, and the necessary activities to recover the error are also recommended. In case of an error related to several data entry fields, after viewing and confirming it, the user should be directed to the relevant fields.

Moreover, the software meeting the expectations and users enjoying working with the software had the lowest score in the user satisfaction criterion. In the study by Kalankesh et al., factors such as the quality of the information provided by the system, system quality, quality of services provided by the system, quality of vendor company support for the software and its users, and the compatibility of system performance with expectations were deemed useful for improving users' satisfaction with the information systems (38). Various studies show that paying attention to the design of the system's user interface (in terms of allowing the background color, fit, order, and sequence of icons, and menus to be adjusted with the work processes of the emergency department), the use of data visualization tools, software speed, personalization, and system font size may increase user satisfaction with the EDIS (39, 40). In an applied research on the usability of hospital information systems, Sadoughi et al. concluded that user satisfaction, perceived usefulness, meeting the expectations, and the intention to continue using the system were indicators of the success or failure of the system; thus, it is essential to resolve the deficiencies that reduce satisfaction with the system (41).

Findings related to the suitability for the task criterion showed that the studied systems have a relatively optimal score in this dimension(3). Among the strengths of the systems examined in the emergency departments were the suitability of software outputs (reports) for the user's tasks, and the existence of information required by the user on the screen. Still, these systems scored poorly in terms of the fit of the screen fields and the possibility of personalizing the screen for user tasks. In the study by Farley et al., the poor user interface design was reported as a limitation of the EDIS, which caused difficulties for users in accessing the required information(12). In the study by Callen et al., despite improving coordination and communication between providers, the use of this system had limitations in terms of quick access to patient information, increasing information registration time, and user-friendliness, which necessitated the use of innovative methods for collecting and displaying emergency department information (42). Agharezaei et al. also found that from the users' point of view, the systems are simple, working with these systems is not particularly complicated, and they are easy to learn, but they lack flexibility. It is thus suggested to pay more attention to this component when designing the system (43).

Findings related to the usefulness criterion indicated that the systems under review received good scores regarding increasing efficiency and productivity, facilitating activities, and better control and management of tasks. The results related to this criterion are in line with the findings of other studies examining ways to increase system usefulness (44, 45). Furthermore, the ease of learning criterion had the highest score compared to the other criteria. The results of this study showed that features such as few learning details when working with the system and the small number of learning instructions were positive features of the systems under study. The literature shows that educating users about using the system and paying attention to the design of these systems to allow users to easily navigate the systems and learn them with little effort promote the learning of the system (46-48). Therefore, Observing the principles of usability in health information systems can promote efficiency and user satisfaction, reduce errors, and ultimately improve patient safety and quality of care (49). This is even more critical in systems used in departments such as the emergency department, where due to the critically ill status of patients, as well as numerous and unpredictable referrals, the users' focus is diminished and they need easy-to-use systems (50, 51).

5. Limitations

To evaluate the usability of EDIS, we administered a researcher-made questionnaire based on the standard 9241 ISO Metric Questionnaire part 10 and the USE Questionnaire. The questionnaires had previously been used in other studies assessing the usability of hospital information systems (28, 52). However, in this study, only the quantitative method was adopted to evaluate the usability of the EDIS. A mixed-methods study (quantitative and qualitative) is recommended to gain better insight into the usability problems of the EDISs.

Other limitations of this study include the lack of cooperation of some system users and users' unfamiliarity with this system. Due to the existence of different processes and structures in the emergency departments, the results of this study may not be generalizable to other settings. Still, out of the three examined information systems, two had been developed by different companies, and one had been designed within the hospital, and this diversity may improve generalizability.

To promote the usability of these systems, the following recommendations are made: understanding the needs of users and the work environment, identifying the type of users and their diverse needs in a versatile environment, paying attention to work processes and seemingly unimportant issues, designing the interface according to common principles and models, involving end-users in information system design, conducting usability tests, and applying final edits before implementing the system.

6. Conclusion:

The results of this study show that the studied emergency systems are at a desirable level in terms of usability. Nevertheless, considering the following factors in designing these systems might increase the usability of EDIS: foreseeing users' work needs, greater software flexibility, consistency of features and standards, similarity across the system, customizing users' screens according to their needs, incorporating data visualization tools, quality of the information provided by the system, system quality, quality of after-sales services, software maintenance support, and displaying information such that users can easily understand them.

6. Declarations

6.1. Acknowledgments

We would like to express our sincere gratitude towards the personnel of the emergency departments of hospitals affiliated with AJA University of Medical Sciences.

6.2. Authors’ Contribution

 S. Almasi, N. Mehrabi, F. Asadi and M. Afzali: Concept and design. S. Almasi: Literature search. S. Almasi: collecting the data. S. Almasi and N. Mehrabi: Data analysis and interpretation. S. Almasi: Manuscript drafting. N. Mehrabi, and F. Asadi and M. Afzali: Editing and critical review. All authors wrote and reviewed the manuscript.

6.3. Conflict of Interest

No potential conflict of interest relevant to this article was reported.

6.4. Funding

This study was financially supported by AJA University of Medical Sciences.

6.5. Competing interests

The authors declare that they have no competing interests.

6.6. Availability of data and materials

The datasets used and/or analyzed in the current study are available from the corresponding author upon reasonable request.