Evaluating hospital information system according to ISO 9241 part 12

Abstract

Objectives

Compliance with standards in designing information systems leads to better utilization and ease of use for users. In this study, the compliance of a widely used hospital information system (HIS) with ISO 9241 part 12 was assessed.

Methods

This applied research is a descriptive, cross-sectional study in which the HIS of 8 hospitals affiliated with Kerman University of Medical Sciences was evaluated based on ISO 9241 part 12. Data were collected by using ISO 9241/12 checklist. The data was analyzed in SPSS 16 using descriptive statistics.

Results

The analysis of data showed that the total compliance of the software with the ISO 9241/12 was 72%. The compliance of the software based on different groups of recommendations was 79% with Organization of information, 91% with Graphic objects, and 58% with Coding techniques. Compliance with different subgroups of ISO recommendations ranged from 28% related to “color coding” in coding techniques to 97% related to “General recommendation for graphical objects” in Graphic objects.

Conclusion

According to this study, the design of a widely used HIS has fairly good compliance with the standard but still suffers from some problems. Considering the role of accurate, valid and timely information in management of the hospitals, and the difficulty of system optimization after implementation, it is necessary that software developers follow existing standards when designing health information systems.

Introduction

Despite many benefits of hospital information systems (HIS) and their capabilities, it is sometimes not easy to use these systems.^1–3 This leads to frustration of users, withdrawal of the systems and increasing the number of errors.^4–7 In some cases, instead of increasing patient safety, they may introduce new risks to the health of patients .Many of these problems are related to poor design of the systems making their acceptance hard by doctors and other users.^8,9 Hence, improving the usability of these systems is an important step in their implementation and acceptance by their users.

Important factors such as efficiency and flexibility of HIS and data recording method have an impact on employee engagement with the system.¹⁰ To ascertain effectiveness and efficiency of information system continuous evaluation of their usability for a wide range of potential users is required.^11,12

According to the International Organization for Standardization (ISO), usability is the extent to which a particular product can be used by specific users to achieve specific goals in a specific area with efficiency, effectiveness and the satisfaction.¹³ In this definition, effectiveness depends on the accuracy and completeness of the measures taken, the efficiency is related to the amount of resources spent (for example time) to achieve a specific goal and user satisfaction can imply system ease of use and ease of access to all system functionalities. Therefore, products such as HIS should be designed based on the needs and expectations of users to guarantee their efficiency, effectiveness and user satisfaction.

Attaining these three groups depends on how the user interface of a system and the display of its components are designed. If the information on the user interface of a system is displayed in accordance with the users’ mental models, it will be more intuitive for the users. Thus, users can independently work with the system while committing a lower number of errors. Based on the results of previous evaluation studies of health information systems, the complexity of the user interface and the lack of understanding of organization of the information by users were considered as two of the main problems of information systems.^5,14,15 Other studies have pointed to the willingness of physicians and other providers to adopt systems that are easy to use, efficient to work and can reduce errors.^6,16–18 To facilitate the optimal use of healthcare information systems the presentation and the organization of information on user interface should be standardized. These user interfaces should comply with design principles and standards which are put forward by standard developing organizations. ISO in part 12 of standard ISO 9241 provides recommendation for presentation of information on the user interface of information systems.¹¹ According to this standard, the display of information on screen should help users to do their cognitive activities (such as the search for information on a computer screen) with efficiency, effectiveness and satisfaction. This standard focuses on the clarity, discriminability, conciseness, consistency, detectability, legibility, and comprehensibility features to be considered when designing visual information. According to our investigation and knowledge, this standard has already not been used for the evaluation of health information systems. So far, most studies on domestic and international papers in this field^19–21 used part 10 of ISO 9241 standard for evaluation of health information systems. This standard only provides principles for the design of dialogues and does not exist anymore. The only study used part 12 of this standard²² evaluated pathology and radiology information system in Kerman University of Medical Sciences (KUMS), and reported that the design of these systems had fairly good compliance with standards but still suffers from some problems. The purpose of our study is to evaluate the presentation of information on the user interface of a HIS, based on part 12 of the ISO 9241standard.

Methods

In this cross-sectional study, a hospital information system (HIS) used in seven hospitals (Afzalipour, Shafa, ShahidBeheshti, Emam Ali, Sina, Khatamolanbia, Qaem) and a large clinic (Besat) affiliated with Kerman University of Medical Science was assessed using ISO 9241/12 checklist. The HIS is currently used in more than 150 hospitals and health centers in Iran and consist of 13 subsystems such as Admission, cash, Laboratory, Radiology, Rehabitation, and Emergency. The system is already implemented and used for at least 11 years (from 2008) in all studied medical centers.

The data collection tool was a checklist provided in the appendix of the standard ISO 9241/12. The Checklist is based on the recommendations provided in this part of the standard concerning presenting information on user interface of information systems. It is used for evaluating both the applicability of, and adherence to, the conditional recommendations in this part of ISO 9241. The Checklist consists of the summary of 123 recommendations, categorized in 16 subgroups and three main groups: Organization of information, Graphical objects and Coding techniques. HIS evaluation was done by three researchers with Health Information Technology education. They were familiar with health information systems. The evaluators independently inspected the user interface of all subsystems of the HIS and used the checklist to evaluate their applicability and adherence to the recommendations of ISO 9241/12. The percentage of recommendation that their applicable and adherence options of them were ‘Yes’ has been calculated. The collected data were reviewed at a joint meeting and combined in a final evaluation list. Any disagreement was resolved through discussion. The data was analyzed by SPSS 16 using descriptive statistics (frequency and percentage). ISO 9241/12 compliance scores were classified as follows: 0–50 percent (Undesirable), 50–65 percent (not satisfactory), 66–75 percent (satisfactory), 76–85 percent (fair), 86–100 percent (good).

Results

The results of this study are shown in Tables 1 to 3. The total compliance of the software with ISO 9241/12 was 72 percent and the most compliance among the three main groups of recommendations was related to the group Graphical objects with 91percent (Table 1).

Table 1.

Mean score of compliance of HIS with ISO 9241/12 in terms of three main categories of recommendations.

Group	Compliance (%)
Organization of information	79
Graphical objects	91
Coding techniques	58

Table 2.

Mean score of compliance of HIS with ISO 9241/12 in terms of subgroups of recommendations.

Group	Subgroup	Compliance (%)
Organization of information	Recommendation for windows	80
	Areas	92
	Input/output area	92
	Groups	90
	Lists	54
	Tables	91
	Labels	89
	Fields	46
Graphical objects	General recommendation for graphical objects	97
	Cursors and pointers	85
Coding techniques	General recommendations for codes	53
	Alphanumeric coding	85
	Abbreviation for Alphanumeric code	72
	Graphical coding	28
	Color coding	38
	Markers	78
	Other coding techniques	50

Table 3.

Compliance rate of HIS sub-systems with ISO 9241/12.

Sub-system	Compliance (%)
Medical record	78
Laboratory	77
Operating room	75
Food unit	75
Rehabilitation	75
Pharmacy	73
Admission	72
Access Level	71
Radiology	69
Emergency	68
Endoscopy	68
Ward	67
Cash	64

Among the subgroups, the most compliance of the system was with subgroup “general recommendations for objects” with 97 percent. The compliance of system with 6 of 8 subgroups related to organization of information was equal or above 80%. The compliance with other two subgroups (lists and fields) was about 50%.

The compliance rate of different groups does not follow a specific pattern. In the group Organization of information, compliance rate was variable from 46% belonging to “fields” subgroup to 92% belonging to “areas” and “input/output area” subgroup. In Graphical objects group, compliance rate was variable from 85% belonging to “cursors and pointers” subgroup to 97% belonging to “general recommendation for graphical objects” subgroup. In Coding techniques group, compliance rate was variable from 28% belonging to “graphical coding” subgroup to 85% belonging to “alphanumeric coding” subgroup (Table 2).

Information concerning compliance rate of each HIS subsystem is provided in Table 3. The compliance rate of most subsystems with standard was almost equal or more than 70 percent. The most compliance was related to Medical record. Cash subsystem with 64% had the lowest compliance among all HIS subsystems.

We have different compliance rate of subgroups in each subsystem. For example, Lab subsystem have 100% compliance rate with standard in “Area”, “Input/output area”, “Groups”, “Table” and “labels” subgroup and the lowest compliance rate belongs to Access level subsystem with 28% in “Fields” subgroup.

Description of incompliance with ISO 9241/12 based on the recommendations of each evaluated group

Organization of information

Problems due to incompliance with this group were fragmented throughout the system. Some of the prominent problems were combination of letters and numbers in a given field of information (such as National Code field, Identification number field, and tell field), obscurity of the length of fixed data fields (such as National Code field) and using zero instead of natural (counting) numbers in data entry fields (such as address field).

Graphical objects

The studied system had the most compliance with the subgroups of this group of standard than with others. The major problems were related to “cursors and pointers” subgroup.

Coding techniques

The number of problems due to incompliance with this group, compared with other groups, was more. The most important problems were lack of using codes, meaningful numbers and appropriate format (such as doctor code and ward code); and lack of using colors for displaying different information.

Discussion and conclusion

Based on the results of this study total compliance of the software with ISO 9241/12 was percent which is rather satisfactory. To our knowledge no original study has used this part of ISO 9241 to evaluate health information systems. Only one study¹⁸ evaluated the pathology and radiology subsystems of a HIS. Aligned with our results, this study showed that design of these subsystems had fairly good compliance with standards. In a systematic review that focused on the design aspects of computerized physician order entry systems,⁶ the studied aspects have been mapped to the recommendations of ISO for the design of computer screens, including part 12 of ISO 9241. Other researchers^19–21 have used part 10 of this standard. Similar to the results of our study, the compliance of evaluated systems with ISO 9241 part 10 in these studies^20,21 was satisfactory.

In this study compliance of the systems with Organization of information group of ISO 9241/12 was 79%. According to this result it is necessary that the programmers and developers pay more attention to the recommendations of this group especially concerning “list and field” subgroup.

The compliance with Graphical objects group of ISO 9241/12 was 91%, that is good. In this field maximum mismatch belongs to the cash subsystem other subsystems are almost in compliance with this group of recommendations.

The compliance with Coding technique group was 58%, that is not satisfactory. In order to improve compliance with this part, it is necessary that system programmers and developers pay special attention to the following recommendations in software upgrade cycle.

• Using codes and numbers with meaningful and appropriate format

• Using colors and graphical objects in presentation of information

• Considering the rules of designing markers and icons

Analysis of all the studied subsystems showed that all of these subsystems, except cash subsystem, are completely in compliance with ISO 9241/12 Standard in “areas” subgroup of Organization of information group; “general recommendation for graphical objects” subgroup of Graphical objects group.

The maximum and minimum compliance with ISO standard is related to Medical record (78%) and Cash (64%) subsystems.

The studied subsystems have the most compliance with “general recommendation for graphical objects” subgroup of Graphical objects group (97%) that is good, and with “graphical coding” subgroup of Coding techniques group has the least compliance (28%) that is undesirable.

Although the studied information systems were designed and developed by a particular company, but the software is already being used in about 150 health care centers in Iran, and the other existing HIS in Iran are designed with similar capabilities. Therefore, the problems reported in this study may exist in other information systems used in Iran.

It is recommended that based on the results of this study proper modifications and adjustments be made on current information systems. Since most of the health information systems used in Iran are commercial (vendor-based) systems, health care organizations have no permission to modify the system after implementation. Therefore, in order to improve systems it is crucial that encourage the designers and developers to use appropriate standards for the design and development of information systems. Moreover, continuous assessment of the systems for their accordance with existing standards and users’ needs can improve the design of the system and its adoption in health care settings. This assessment can be done in different stages of system life cycle including design, purchase, and implementation.

The focus of this study was on the presentation of information on the user interface of HISs based on recommendation provided in ISO 9241/12 standards. Other type of evaluation can be done for evaluating other aspects of system such as user satisfaction, system efficiency and effectiveness.

The research findings show that the presentation of information in the user interface of studied hospital information systems is relatively satisfactory. However, it is suggested that the software developers seriously consider standard recommendations especially concerning aspects of codes, meaningful numbers, appropriate format and the use of graphics and colors tools in display based on ISO9241/12. These sorts of problems can have negative effects on users’ performance and may hinder a successful interaction with the system.

Since, all the involving aspects such as human, technical and organizational factors should be considered for the assessment of information systems,²³ it is recommended to combine the results of this study with those focusing on user interaction with system in a specific context.

Limitations

This study has three limitations. First, this standard only focuses on the rate of compliance of the system with established and cannot specifically specify the type of problems. To our knowledge, this study is one of the few studies that shed lights on the extent of non-adherence of information presentation in a widely used health information system with established standard. Future studies can augment this method with other methods such as usability studies to deeper identify the problems lead to non-adherence of a system.

Second, this standard is intended to improve Clarity, Discriminability, Conciseness, Consistency, Detectability, Legibility, and Comprehensibility features of the information system follow this standard. Since the checklist was not prepared based on the discrimination of these items, it was not possible to analyze the data accordingly.

Third, Since the time of study, a new standard (ISO 9241-125) has replaced the standard we used in our study, although a number of the features we evaluated are the same in the new standard, we recommend future studies employ ISO 9241-125 to evaluated health information systems.

Recommendations

This study showed the extent of non-adherence of hospital information systems with the standard. According to the results of this study, other researchers can use other methods to find the reasons for non-adherence in information systems. We performed this evaluation on HIS. other evaluation can be done on other information systems.

Considering that the compliance rate with the standard in this study was 77% and we had 23% non-adherence, it is better to use design principle standards when designing these systems to improve adherence.