Abstract
Objectives
As the largest group providing healthcare services, nurses require well-designed information systems in their practice. This study aims to evaluate the usability of nursing information systems (NIS).
Methods
This cross-sectional survey was conducted in 2015. The settings of the study consisted of four hospitals affiliated with three medical universities in Tehran (Iran). The subjects of the study included nurses who had access to and used a NIS developed by four major software companies. The data were collected using a modified version of a usability questionnaire known as IsoMetrics, based on the International Standard ISO 9241, Part 11. The questionnaire is composed of 35 questions divided into seven general criteria. The validity of the questionnaire was determined by experts in the field, and the reliability was checked using Cronbach's alpha (α = 0.91). The questionnaire was then distributed to 184 nurses.
Results
The response rate was 64.6%. Among the seven ISO usability criteria, suitability for the task (3.10 ± 1.24) and suitability for learning (3.10 ± 1.27) had the highest mean value. The lowest mean value (2.37 ± 1.29) was related to the suitability for individualization.
Conclusions
Addressing issues related to individualization and self-descriptiveness could improve the usability of nursing systems. Considering usability requirements in the design of a NIS will lead to the efficient and effective use of these systems.
Keywords: Information Systems, Hospital Information Systems, Evaluation Studies, Nursing Informatics, Nursing
I. Introduction
The recent advancements in information technology and computer science have led to the development of computerized information systems in different areas. In healthcare, these systems, known as healthcare information systems, are used as supportive tools to facilitate clinical and administrative tasks [1]. These systems can help to minimize errors, reduce costs and improve the delivery of services [2]. Nursing information systems (NIS) are among the key components of healthcare information systems [3]; they can help nurses to use data more effectively while managing their nursing activities, allowing them to improve patient care and develop their knowledge [4]. Nurses use a NIS to develop care plans, perform interventions, document patient care, organize information, and assess the effects and quality of nursing services [5,6,7,8].
The efficient and effective use of a NIS depends to a great extent on the appropriate designs of these systems and how well they take the expectations and needs of nurses into consideration [9,10]. A poorly designed NIS can complicate the daily routines of nurses, possibly leading to problems related to the acceptance and use of these systems [11,12]. The proper design of the user interface is one of the crucial elements affecting user interactions with the NIS. These systems should also be easy to learn and use [5]. The format of the data display, the format of the navigation instructions, and the data entry and data retrieval sequences should be meaningful for nurses, and they should find the NIS relevant to their work [13]. In other words, interactive health information systems including nursing information systems should be designed considering their usability features [14]. Usability is inherently linked to the quality of an information system [15] and by definition affects the extent to which such a system can be efficiently and effectively used to achieve specific goals and to ensure user satisfaction. Usability is directly related to productivity, error rates, alert fatigue, and user satisfaction [15,16]. With respect to a NIS, usability problems mainly arise due to the limited functionality of these systems and the poor fit between the systems and the nurses' professional needs. Rogers et al. [17], in a study conducted using a scenario-based usability test, indicated that the application of human factors such as recognition rather than recall regarding the functions of a NIS and a good match between the system and the real world can help to improve the designs of NISs with regard to nurses' needs. Several researchers have undertaken usability testing with regard to other health information systems. Georgsson and Staggers [18], in their usability evaluation of a system for diabetes, used measures from the International Organization for Standardization (ISO 9241-11), specifically the IsoMetric usability questionnaire. Finding correct options and remembering the steps to complete tasks were among the issues reported. Similarly, Hamborg et al. [19] evaluated the usability of hospital information systems using the IsoMetric questionnaire. The findings showed low consideration of human factors such as learnability of the system and its suitability for performing tasks. Although several researchers have evaluated the usability of healthcare information systems, relatively few have investigated the usability of nursing information systems. This study aims to evaluate the usability of a NIS from the perspective of nurses. In detail, the following study questions are defined: (1) To what extent, do nursing information systems meet the ISO 9241-11 usability criteria? (2) What are the recommendations for further improvement of these systems?
II. Methods
This was a cross-sectional study conducted in 2015. The settings of the study were four hospitals affiliated with three medical universities in Tehran (Iran) that used a NIS developed by four major companies. Sampling in these settings considered two inclusion criteria. The first of these was that there had to be at least one hospital for every company that has a system installed in more than one hospital. The second was based on the number of hospital beds in general hospitals (i.e., general hospitals with the most beds). Initially, five hospitals were selected (one hospital from university 1, one hospital from university 2, and three hospitals from university 3). However, at the time of data collection, one hospital from university 3 did not agree to take part in the study and was thus excluded from the study. The final sample included four hospitals with four different NISs. To maintain confidentiality with regard to the information collected, the identities of the vendors and hospitals are obscured here. Instead, the hospitals are denoted as hospitals A, B, C, and D. Similarly, the software packages used in the hospitals are referred to here as A, B, C, and D, respectively.
The target population consisted of nursing staff at each hospital. However, a limited number of nursing staff members had access to the systems and used them. As a result, no sampling method was used, and all of the nurses who used the systems were invited to take part in the study (n = 184) (Table 1).
Table 1. Number of nurses in the target population and eligible nurses.
The data were collected using a usability questionnaire known as IsoMetric, which is based on the International Standard ISO 9241, Part 11. This questionnaire utilizes a 5-point Likert scale with options ranging from predominantly disagree (score 1) to predominantly agree (score 5), along with a ‘no opinion’ option (score 0). The original questionnaire contains 75 statements categorized in seven main groups (criteria): ‘suitability for the task,’ ‘self-descriptiveness,’ ‘controllability,’ ‘conformity with user expectations,’ ‘error tolerance’ (sensitivity), ‘suitability for individualization,’ and ‘suitability for learning’ [20]. However, due to the nurses' workload, which was expected to influence the response rate, the number of questions was reduced to 35, and each section consisted of five questions. The minimum and maximum score for each section was between 0 and 25. The validity of the questionnaire was assessed through expert opinions. The reliability was confirmed using Cronbach's alpha (α = 0.91). The questionnaires were distributed to the nurses at the four hospitals through the systems.
Data analysis was performed using SPSS ver. 19.0 (IBM SPSS, Armonk, NY, YSA). To analyze the data, the mean values and standard deviations were calculated, after which the systems were compared according to the usability criteria.
III. Results
The response rate was 64.6% (n = 119). Ninety-seven percent of the participants were female. With regard to academic degrees, 97% of the nurses had a bachelor's degree and 3% had their master's degrees. In terms of the participants' age, the 24–30 age group accounted for 32% of all nurses, and the lowest frequency (8.4%) was that of age group of 45 years and above. The average age of the participants was 34 years (SD 7). Work experience for the majority of nurses (49%) amounted to 1–10 years, while 11.7% had work experience of more than 20 years. The average amount of work experience was 10.3 years (SD 6.9). In addition, for the majority of the nurses, the amount of experience with a nursing information system was between 1 and 5 years. The average was 5.4 years (SD 2.3).
1. Suitability for Tasks
The findings related to ‘suitability for the task’ indicated that the highest mean score was for system B (1.35 ± 3.51) while the lowest mean score was for system D (1.35 ± 2.91). Systems A (1.27 ± 3.06) and C (1.08 ± 3.05) were ranked second and third, respectively (Table 2). In relation to system B, the highest mean score was found in relation to support for the user during their daily work routine (1.22 ± 4.13); in system D, the lowest mean score was related to the requirement to undertake different steps to complete a task (1.28 ± 2.53).
Table 2. Frequency distribution of responses to ‘suitability for the task’ in the studied hospitals.
Values are presented as number (%).
2. Self-description
The findings related to the “self-description” criterion showed that the highest mean score was for system B (3.9 ± 1.11) while the lowest mean score was for system A (2.56 ± 1.12). Systems C (2.81 ± 0.94) and D (2.69 ± 1.31) were ranked second and third, respectively. In system B, the highest mean score was for ease of understanding messages displayed by the system (3.35 ± 1.07) and for clarity of the terms used in the system (3.35 ± 1.11). In system A, the lowest mean score was related to the ability to display general explanations and relevant examples (1.97 ± 0.98).
3. Controllability
The findings of the ‘controllability’ criterion indicated that the highest mean score was for system B (3.43 ± 1.33) with the lowest mean score assigned to system A (2.87 ± 1.37). Systems C (2.97 ± 1.03) and D (2.94 ± 1.19) were in second and third place, respectively. In system B, the highest mean score was for the ability to return to the main menu from any screen (3.70 ± 1.29), to stop running procedures (3.70 ± 1.29), and for ability to move (forward and backward) between screens (3.70 ± 1.14). In relation to system A, the lowest mean score were related to stopping running procedures (2.60 ± 1.35) and requiring a fixed sequence of steps to complete tasks (2.60 ± 1.22).
4. Conformity with User Expectation
The findings related to ‘conformity with user expectations’ showed that the highest mean score was for system B (3.29 ± 1.39) and that the lowest mean score was for system A (2.85 ± 1.32). Systems C (3.06 ± 0.98) and D (2.90 ± 1.29) were ranked second and third, respectively. The highest mean score in system B was related to using the same designations in all parts of the system (3.70 ± 1.11), while in system A, the lowest average score was for predicting the time required to perform tasks by the user (2.60 ± 1.06).
5. Error Tolerance
The findings for the ‘error tolerance’ (sensitivity) criterion showed that system B had the highest mean score (1.39 ± 3.31). The lowest mean score was given to system A (2.81 ± 1.43). Systems D (2.97 ± 1.39) and C (2.83 ± 1.29) were in second and third place, respectively. Regarding system B, the highest mean score was for requiring user confirmation before performing an action, such as deleting information (3.91 ± 1.24). The lowest mean score in system A was related to warning the user about potential problem situations (2.37 ± 1.44).
6. Suitability for Individualization
The findings related to the ‘suitability for individualization’ criterion indicated that the highest mean score was for system B (2.87 ± 1.19), while system A had the lowest mean score (1.99 ± 1.34). For system B, the highest mean score was for the ability to change the terminology of the commands and actions based on the user vocabulary (3.00 ± 1.38), while in system A, the lowest mean score was for the ability to change forms, screens, and menus according to user preferences (1.89 ± 1.21) and for setting the system's response time proportional to the user's working speed (1.89 ± 1.41).
7. Suitability for Learning
In relation to the ‘suitability for learning’ criterion, the highest mean score was for system D (3.20 ± 1.25) and the lowest mean score was for system B (3.07 ± 1.23). Systems C (3.12 ± 1.21) and A (3.08 ± 1.44) were in second and third place, respectively. In system D, the highest mean score was related to the lengthy amount of time required to learn how to use the system (3.63 ± 1.27); in the system B, the lowest mean score was for using the system without having to ask coworkers for help (2.13 ± 1.01). A summary of the findings is presented in Table 3.
Table 3. Frequency of responses for the seven usability criteria of isometric.
Values are presented as number (%).
As shown in Table 3, among the seven usability criteria, the highest mean value (3.12 ± 1.27) was related to ‘suitability for learn’ and the lowest mean value (2.37 ± 1.29) belonged to ‘suitability for individualization.’ A comparison between the usability features of the nursing information systems is presented in Figure 1.
Figure 1. Mean scores of seven ISO 9241 criteria for the systems under study.
IV. Discussion
Research suggests that the design of a health information system must fully match users' practices and must support them as they undertake their tasks; otherwise, the system will not be accepted by users or its uptake could face difficulties [21]. Among the criteria related to ‘suitability for the task,’ disagreement was mainly found for the factors of ‘easy to find required commands’ and ‘finding necessary information on a given screen.’ This is in line with the findings of Rogers et al. [17], which showed that nurses are occasionally unable to complete a task due to navigation issues and difficulties in finding the required functions. Another study indicated that displaying user-required information on different screens affected the efficient use of the systems and ultimately user satisfaction [22]. Similarly, research suggests that information about a given task should be displayed in a specific part of a system [23]. In general, features related to the ‘suitability for the task’ factor appeared to be considered when designing the systems evaluated in this study. The provision of information considering user needs, progressing through short steps to accomplish tasks, ease of access to required commands, and the display of necessary information on a single screen will improve the satisfaction of the user with the system.
In relation to ‘self-description,’ most of the nurses agreed that the NISs were self-described systems. However, the mean score obtained indicates that adequate attention was not paid to self-descriptiveness to the extent that it should when designing the systems. Regarding the sub-criteria of self-descriptiveness, the nurses agreed mainly on the clarity of the terms used in the systems and in the ease of understanding the messages displayed. This finding is in good agreement with the findings of Rogers et al. [17], in which nurses were reportedly positive about the consistency and clarity of the terms used in all aspects of the nursing system. However, the main area of disagreement was related to the capability of the systems to provide general explanations and relevant examples; these features should be considered in further developments of nursing systems.
Another feature of usability, in this case ‘controllability,’ as the findings showed, was taken into account in the designed systems. However, in the systems investigated by Rogers et al. [17], functions such as saving frequent actions or using shortcuts were not often readily available. Research points out controllability as a key feature of a well-designed information system [24]. The capability of moving between screens, stopping running procedures when needed, and returning to the main menu from any screen could help to improve the controllability of systems. Such features in turn affect user satisfaction.
Similarly, the majority of nurses agreed that the NISs conformed to their expectations. However, they expected to have more flexibility in terms of using identical keys for similar functions in all aspects of the systems. Studies have demonstrated the significance of considering user expectations and the related impact on user satisfaction with information systems [25]. Research suggests that information systems which do not meet user expectations are at risk of failure [26]. Using designations consistently in all parts of the systems, displaying messages at the same screen location, and predicting the time required to perform a given task are among the approaches that could help to improve system conformity levels with regard to user expectations.
In relation to the capability of the nursing systems to prevent errors, i.e., ‘error tolerance,’ as the findings indicated, more attention should be paid to this aspect in further developments of NISs. This finding is supported by the findings of Viitanen et al. [16], in which nurses reported making errors when performing documentation tasks, noting that the failure protection system was insufficient. These findings suggest that an appropriate interface design and reduced complexity of the systems could improve the error prevention factor. Studies have also reported the importance of information systems which are capable of preventing errors [27,28]. Providing users with functions such as recovery from an error situation, restoring a previous session and correcting errors with little effort could help to improve the ‘error tolerance’ capability of a nursing information system [29].
Among the criteria outlined in ISO 9241, the lowest mean score was seen for ‘suitability for individualization.’ Near 50% of the nurses did not find their NIS suitable for individualization, and they pointed out that they were unable to adjust the response time of the NIS according to their working speed. This aspect should therefore be addressed in further studies of nursing systems, as it could help users to make more efficient use of the systems and could facilitate their work routines. A key factor related to ‘suitability for individualization’ is consideration of users' levels of knowledge and skills [29]. Studies have reported that information systems should be designed considering the knowledge and experience of their users. In addition, the systems should display information in an appropriate format to meet users' needs [29].
With respect to the last feature of usability, i.e., ‘suitability for learning,’ the nursing information systems were found to be easy to learn by the nurses, and the highest mean score among the seven criteria outlined in ISO 9241 was related to this feature. Nurses did not require much time to learn how to use the systems. In addition, it was easy for them to relearn how to use the systems after a lengthy interruption. However, in a study by Viitanen et al. [16], nurses found nursing information systems difficult to learn and not intuitive to use. Knowing that suitability for learning can influence user satisfaction levels [30], the designs of these systems should allow users to navigate the systems conveniently and learn them with little effort.
In this study, a modified version of the IsoMetric questionnaire was used to make it simpler to complete by the respondents. Although the reliability of the modified questionnaire was confirmed (α = 0.91), the study could benefit from using the original questionnaire to provide a broader picture of NIS usability issues. Moreover, in the current study, only a quantitative method was used to evaluate the usability of the nursing information systems assessed here. Conducting a mixed-method evaluation study (quantitative and qualitative) is suggested for future researchers to gain better insight into the usability problems of these systems.
In conclusion, this study evaluated the usability of four nursing information systems using the metrics recommended in ISO 9241-11. The findings showed that there are areas in need of further improvement. Providing nurses with the capability of individualizing the systems according to their work routines could help them to use the systems more efficiently and effectively. In addition, the designs of these systems should enable nurses to find their own way with little effort when using them. Moreover, the systems should guide users on how to perform their tasks conveniently. Finally, designing nursing information systems according to usability principles and considering user requirements can help to improve the effective use of these systems.
Footnotes
Conflict of Interest: No potential conflict of interest relevant to this article was reported.
References
- 1.Su KW, Liu CL. A mobile Nursing Information System based on human-computer interaction design for improving quality of nursing. J Med Syst. 2012;36(3):1139–1153. doi: 10.1007/s10916-010-9576-y. [DOI] [PubMed] [Google Scholar]
- 2.Kern R, Haase R, Eisele JC, Thomas K, Ziemssen T. Designing an electronic patient management system for multiple sclerosis: building a next generation multiple sclerosis documentation system. Interact J Med Res. 2016;5(1):e2. doi: 10.2196/ijmr.4549. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Kahouei M, Molanoroozi Z, Habibiyan M, Sedghi S. Health information technology in the knowledge management of health care organizations. Middle East J Rehabil Health. 2015;2(3):e29229. [Google Scholar]
- 4.Rachmawaty R. The development of the electronic nursing record system (ENRS) in the hospital setting: an integrative literature review. Am J Health Sci. 2015;6(1):1–6. [Google Scholar]
- 5.Kimiafar K, Sadoughi F, Sheikhtaheri A, Sarbaz M. Prioritizing factors influencing nurses' satisfaction with hospital information systems: a fuzzy analytic hierarchy process approach. Comput Inform Nurs. 2014;32(4):174–181. doi: 10.1097/CIN.0000000000000031. [DOI] [PubMed] [Google Scholar]
- 6.Nakate G, Dahl D, Drake KB, Petrucka P. Knowledge and attitudes of select ugandan nurses towards documentation of patient care. Afr J Nurs Midwifery. 2015;2(1):56–65. [Google Scholar]
- 7.Xu WL, Yang LQ, Zhang HY. Nursing informatics in clinical practice in China. Comput Inform Nurs. 2013;31(5):214–218. doi: 10.1097/NXN.0b013e31828a0db0. [DOI] [PubMed] [Google Scholar]
- 8.Alkouri O, AlKhatib AJ, Kawafhah M. Importance and implementation of nursing documentation: review study. Eur Sci J. 2016;12(3):101–106. [Google Scholar]
- 9.Schiza EC, Neokleous KC, Petkov N, Schizas CN. A patient centered electronic health: eHealth system development. Technol Health Care. 2015;23(4):509–522. doi: 10.3233/THC-150919. [DOI] [PubMed] [Google Scholar]
- 10.Janols R, Lind T, Göransson B, Sandblad B. Evaluation of user adoption during three module deployments of region-wide electronic patient record systems. Int J Med Inform. 2014;83(6):438–449. doi: 10.1016/j.ijmedinf.2014.02.003. [DOI] [PubMed] [Google Scholar]
- 11.Kushniruk AW, Borycki EM, Anderson J, Anderson M, Nicoll J, Kannry J. Using clinical and computer simulations to reason about the impact of context on system safety and technology-induced error. Stud Health Technol Inform. 2013;194:154–159. [PubMed] [Google Scholar]
- 12.van Engen-Verheul MM, Peute LW, de Keizer NF, Peek N, Jaspers MW. Optimizing the user interface of a data entry module for an electronic patient record for cardiac rehabilitation: a mixed method usability approach. Int J Med Inform. 2016;87:15–26. doi: 10.1016/j.ijmedinf.2015.12.007. [DOI] [PubMed] [Google Scholar]
- 13.Cipriano PF, Hamer S. Special report: nursing, technology and information system. Am Nurse Today. 2013;8(11):1–19. [Google Scholar]
- 14.Issa T, Turk A. Applying usability and HCI principles in developing marketing websites. Int J Comput Inf Syst Ind Manag Appl. 2012;4:76–82. [Google Scholar]
- 15.Freire LL, Arezes PM, Campos JC. A literature review about usability evaluation methods for e-learning platforms. Work. 2012;41(Suppl 1):1038–1044. doi: 10.3233/WOR-2012-0281-1038. [DOI] [PubMed] [Google Scholar]
- 16.Viitanen J, Kuusisto A, Nykanen P. Usability of electronic nursing record systems: definition and results from an evaluation study in Finland. Stud Health Technol Inform. 2011;164:333–338. [PubMed] [Google Scholar]
- 17.Rogers ML, Sockolow PS, Bowles KH, Hand KE, George J. Use of a human factors approach to uncover informatics needs of nurses in documentation of care. Int J Med Inform. 2013;82(11):1068–1074. doi: 10.1016/j.ijmedinf.2013.08.007. [DOI] [PubMed] [Google Scholar]
- 18.Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc. 2016;23(1):5–11. doi: 10.1093/jamia/ocv099. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hamborg KC, Vehse B, Bludau HB. Questionnaire based usability evaluation of hospital information systems. Electron J Inf Syst Eval. 2004;7(1):21–30. [Google Scholar]
- 20.International Organization for Standardization. Ergonomic requirements for office work with visual display terminals (VDTs) -- Part 11: Guidance on usability. Geneva, Switzerland: International Organization for Standardization; 1998. (ISO 9241-11:1998) [Google Scholar]
- 21.Shojaei S, Farzianpour F, Arab M, Rahimi Foroushani A. An Investigation on the Status of Implementation of Communications and Information Management System (MCI) in Khorasan Razavi Hospitals. Glob J Health Sci. 2015;8(5):110–119. doi: 10.5539/gjhs.v8n5p110. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Kahouei M, Zadeh JM, Roghani PS. The evaluation of the compatibility of electronic patient record (EPR) system with nurses' management needs in a developing country. Int J Med Inform. 2015;84(4):263–270. doi: 10.1016/j.ijmedinf.2014.12.006. [DOI] [PubMed] [Google Scholar]
- 23.Nabovati E, Vakili-Arki H, Eslami S, Khajouei R. Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system. J Med Syst. 2014;38(4):35. doi: 10.1007/s10916-014-0035-z. [DOI] [PubMed] [Google Scholar]
- 24.Jayawardena AS. The electronic hospital information system implemented at the District General Hospital Trincomalee: an experience of business process reengineering. J Community Med Health Educ. 2014;S2:001. [Google Scholar]
- 25.Karimi F, Poo DC, Tan YM. Clinical information systems end user satisfaction: the expectations and needs congruencies effects. J Biomed Inform. 2015;53:342–354. doi: 10.1016/j.jbi.2014.12.008. [DOI] [PubMed] [Google Scholar]
- 26.Farzandipour M, Sadoughi F, Meidani Z. Hospital information systems user needs analysis: a vendor survey. J Health Inform Dev Ctries. 2011;5(1):147–154. [Google Scholar]
- 27.Birjandi F, Gharehbaghian A, Delavari A, Rezaie N, Maghsudlu M. Blood donor deferral pattern in Iran. Arch Iran Med. 2013;16(11):657–660. [PubMed] [Google Scholar]
- 28.Jahanbakhsh M, Ehteshami A, Shafiei A. Assessment of the hospital information system in compliance with certification commission for healthcare information technology standard at Isfahan University of Medical Sciences' academic hospitals. Int J Educ Psychol Res. 2016;2(2):105–110. [Google Scholar]
- 29.Ahmadi M, Shahmoradi L, Barabadi M, Hoseini F. A survey of usability of hospital information systems from the perspective of nurses, department secretaries, and paraclinic users in selected hospitals: 2009. J Health Adm. 2011;14(44):11–20. [Google Scholar]
- 30.Rafique I, Weng J, Wang Y, Abbasi MQ, Lew P. Software learnability evaluation: an overview of definitions and evaluation methodologies for GIS applications; Proceedings of the 7th International Multi-Conference on Computing in the Global Information Technology (ICCGI); 2012 Jun 24-29; Venice, Italy. [Google Scholar]