Evaluating the usability of Iran’s national comprehensive health information system: a think-aloud study to uncover usability problems in the recording of childcare data

Abstract

Introduction

Health information systems play a crucial role in the delivery of efficient and effective healthcare. Poor usability is one of the reasons for their lack of acceptance and low usage by users. The aim of this study was to identify the usability problems of a national comprehensive health information system using the concurrent think-aloud method in the recording of childcare data.

Methods

A descriptive cross-sectional study was conducted in the health centers of Kashan University of Medical Sciences, Iran, in 2020. Ten healthcare providers as system's users were purposively selected to evaluate the system. To identify problems, a concurrent think-aloud evaluation was conducted. Two administrators of the system designed scenarios for ten childcare data recording tasks. By analysing the recorded files, usability problems were identified. The severity of the problems was then determined with the help of the users and problems were assigned to usability attributes based on their impact on the user.

Results

A total of 68 unique problems were identified in the system, of which 47.1% were rated as catastrophic problems. The participants assigned 47 problems (69%) to the user satisfaction attribute and 45 problems (66%) to the efficiency attribute; they also did not assign any problems to the effectiveness attribute.

Conclusion

The problems identified in the national comprehensive health information system using the think-aloud method were rated as major and catastrophic, which indicates poor usability of this system. Therefore, resolving the system problems will help increase user satisfaction and system efficiency, allowing more time to be spent on patient care and parent's education as well as improving overall quality of care.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-024-02746-2.

Introduction

Health information systems play a critical role in providing effective, timely, and efficient care [1, 2]. These systems should have high usability to facilitate the users' tasks. However, studies have proven that despite the high cost of development and implementation of such systems, some of them have many usability problems and are not accepted by the users for several reasons, such as making new errors, taking so much time to perform the tasks, and not paying attention to the needs of end-users [3–6]. Based on Iran's healthcare system reform plan, the Integrated Health System (SIB in Persian), a national comprehensive health information system, was established by the health deputy of the Ministry of Health and Medical Education. It has been in use since 2015. As of September 2023, more than 90% of people's health data had been recorded in this system [7]. The SIB system was designed and implemented to collect primary care information of people in the community as the basis for electronic health records. The key capabilities of SIB include recording critical patient data such as demographics, screening age groups, and vital events such as births, deaths, marriages, divorces, diagnoses, immunizations, and e-prescriptions. The aggregated population health data in the system enable public health initiatives, such as the family physician and referral system, and the creation of monitoring systems [7].

The national SIB system is the sole electronic platform used by numerous healthcare providers across primary care facilities to document crucial patient health information daily, having replaced traditional paper-based medical records. As the only government-run comprehensive health information system in primary care without competition, therefore, high usability and user-friendliness of SIB is critical.

In previous studies [8, 9] that evaluated the usability of the system, it was found that the system has catastrophic and major problems from the experts' point of view. However, in expert-based evaluation, experts identify general system problems and fewer user-related problems [5]. Perhaps considering the widespread use of this system that was mentioned, correction of these general problems was not prioritized.

However, health centers must use this system to record the data for all care activities. For such a high-volume platform, usability and technical problems can severely affect user experience, satisfaction, efficiency, effectiveness, and data quality. The results of the study by Khajouei et al. showed that 45% of the problems identified by the think-aloud method had an effect on user satisfaction, 21% on efficiency, 7% on effectiveness, 10% on learnability, and 16% on errors [5]. Therefore, identifying problems that affect the users who frequently use this system to record data may be important for system administrators and designers. To achieve this, it is better to evaluate the system using a user-centered method. The think-aloud (TA) method is a user-based evaluation approach that involves users in the process. This allows them to express their thoughts and difficulties as they use the system [10, 11].

Given the widespread use of the system, the importance of childcare data entry in primary care as a frequent care process comprising the largest user group responsible for inputting this critical data, and the importance of usability evaluation, this study evaluated the usability of the SIB system for recording childcare data from the end-user viewpoint using the think-aloud method. To determine the types of identified problems and assess system usability based on them, users rated the problems. In addition, the problems were categorized into usability attributes based on their impact on users.

Materials and methods

In this study, the usability of the SIB user interface to record childcare data was assessed using TA, and the problems were identified.

Study setting

This study was conducted in 14 comprehensive health service centers affiliated with the Kashan University of Medical Sciences, located in Kashan city, Iran, in 2020.

Participants

Although some research has indicated that 4–5 participants can identify 80% of usability problems [12], another study [13] found that only five participants could identify 77–85% of problems, but, other study has suggested utilizing more participants. Faulkner's research [14] discovered that increasing the number of test participants from 5 to 10 led to more identified usability problems. Specifically, the average percentage of identified problems increased from 85 to 95% when going from 5 to 10 participants. While small sample sizes can detect a considerable proportion of usability problems, increasing the number of participants typically yields a higher rate of problem identification. Considering the willingness of health service centers and healthcare providers to participate, as well as the objective of identifying a comprehensive set of system problems and a review of the relevant literature, the researchers conducted the study with 10 childcare providers.

The study inclusion criteria were as follows: having at least an associate degree, being responsible for childcare, having experience working with SIB for recording childcare data, and participating in the instruction session to learn how to use the think-aloud evaluation methods.

Assessed tasks

Considering the wide variety of SIB tasks, a form was designed to recognize the most frequent tasks. Twenty experienced users were given this form, and based on their answers, ten childcare tasks were selected for evaluation as the most important and frequent system tasks (Table 1).

Table 1.

Main and common Tasks related to child care data registration through the SIB

Number	Task
1	Data entry regarding children’s danger signs
2	Data entry regarding children’s jaundice
3	Data entry regarding children’s weight
4	Data entry regarding children’s height
5	Data entry regarding children’s head circumference
6	Data entry regarding children’s nutritional status
7	Data entry regarding children’s vision status
8	Data entry regarding children’s developmental status
9	Data entry regarding children’s vaccination
10	Data entry regarding children’s supplement use

Accordingly, two administrators of the SIB designed ten scenarios for children’s healthcare tasks that the participants had to perform (Supplementary File 1). The selection criteria for these two people were more experience working with the SIB system because they faced many cases, were skilled in scenario design with different complexities, and were aware of the gold standard for each task.

Evaluation method

The TA method was used to identify the usability problems of the system, and their severity was determined based on participants’ opinions. The problems were classified according to their effects on usability attributes.

Think-aloud method

TA is a user-based usability evaluation in which users are asked to loudly express their feelings, thoughts, and observations while performing the tasks, and the researcher observes the user's problems and uses audio or video recording to discover problems [11, 15].

Data collection

First, TechSmith Camtasia software version 2020.0.12 was installed on the laptops to record detailed audio and video of the verbal and interactive data exchanged between the healthcare provider and the system. Before starting the evaluation, all participants were verbally informed of the study objectives and instructed to use the TA method. Verbal informed consent to participate in the evaluation was obtained. Demographic information, including age, work experience, and educational level, was recorded for the participants, who were then asked to log in to the system using laptops. The participants were given ten designed scenarios consisting of childcare-related tasks to complete. The evaluator was present as an observer during the evaluation process and instructed the healthcare providers to perform the tasks using the system while concurrently verbalizing their thoughts and feelings aloud. The evaluator documented any usability problems observed during the sessions that were not verbally expressed by participants. The participants were repeatedly prompted by the evaluator to concurrently verbalize their thoughts and feelings aloud during the tasks. Each usability evaluation session lasted 30 min, during which each participant completed 10 task scenarios.

Data analysis

The usability evaluation sessions generated two sources of data: video files of the participants completing the task scenarios, as well as evaluator observations of problems not captured in the video. Immediately after each TA session, the evaluator watched the video file, listened to the voice of the user, identified the problems, and then matched the problems noted with those identified by the evaluator. All usability problems from both sources were compiled into a comprehensive list for the analysis. Healthcare providers then participated in a meeting to review and confirm the identified problems. The problem was considered a system usability problem if confirmed by a minimum of 5 participants. A list of identified problems was given to the participants, and they independently rated the severity of them by considering three factors: the exposure frequency of risk, the impact of the problem on the user experience, and the persistence of the problem [16, 17]. Five degrees were used to score the problem severity, ranging from 0 to 4 according to Table 2; therefore, quantitative data were obtained from qualitative data. Finally, the average severity rate was calculated and rounded to the nearest integer.

Table 2.

Rating scale used to rate the severity of usability problems

Problem	Severity	Description
No problem	0	I don’t agree that this is a usability problem at all
Cosmetic	1	Need not be fixed unless extra time is available on project
Minor	2	Fixing this should be given low priority
Major	3	Important to fix, so should be given high priority
Catastrophe	4	Imperative to fix this before product can be released

Next, two usability experts guided the participants in classifying problems and resolving ambiguities. Both experts had previous experience in designing user interfaces and were familiar with usability evaluation methods and health information systems. Then, usability attributes were explained using examples for the participating users, and they were asked to assign problems considering their impact on each attribute of usability. If three or more users assigned a problem in a usability attribute, the problem is categorized in that attribute, finally, the data was analyzed in Excel 2013 using descriptive statistics.

Results

Demographics

The demographic information of the participants is presented in Table 3.

Table 3.

Demographic characteristics of the participants in think- aloud evaluation

Variable		Number (%)	Variable		Number (%)
Gender	Man	0	Education Major	Public health	7(70)
	Female	10(100)		Midwife	3(30)
Age	20–25	4(40)	Job	Public health	7(70)
	26–30	4(40)		Midwife	3(30)
	31–35	2(20)
Education	Bachelor's degree	9(90)	Work experience (year)	1–3	6(60)
	Master's degree	1(10)		4–6	4(40)

Identified problems

A total of 100 problems were identified in the SIB; after removing duplicate problems, 68 unique problems remained. Of the 68 unique problems, two (2.9%) were cosmetic, 11 (16.2%) were minor, 23 (33.8%) were major, and 32 (47.1%) were catastrophic (Table 4). Based on the results in Table 4, the three most common problems identified during the tasks were asking general questions of parents, requiring data entry on multiple pages, and providing improper diagnosis and treatment recommendations.

Table 4.

Frequency of usability problems identified in think- aloud evaluation and their intensity by task

Task	Total Number of Problems	Mean severity	Severity				Examples of Catastrophe and Major Problems for Each Task
			Cosmetic	Minor	Major	Catastrophic
Task 1	6	1/8	-	4	2	-	• Repeating the selection of the No option for healthy children and Yes for unhealthy children • Incorrect treatment recommendations, such as quick referral when choosing Yes to questions • Placing information on multiple pages
Task 2	5	2/6	-	1	2	2	• Improper treatment recommendations When choosing an answer Yes to the question of whether the patient has severe jaundice • Providing the same treatment recommendations for all ages instead of age-appropriate recommendations • Placing information on multiple pages
Task 3	6	3/3	-	-	1	5	• Placing information on multiple pages • Asking general questions for this type of care
Task 4	8	3/1	-	-	2	6	• Failure to provide a warning when answering a question incorrectly • Asking general questions about this type of care • Placing information on multiple pages
Task 5	9	2/9	-	-	4	5	• Determining the time for recording information and not recording information after that period • Asking general questions of parents regarding this care placing information on multiple pages
Task 6	9	2/6	1	1	3	4	• Improper diagnosis and treatment recommendations such as "child with nutritional problems and the need for follow-up" if you choose to consume breast milk with formula • Displaying the same treatment recommendations for all ages instead of age-appropriate recommendations • Determining an inappropriate follow-up date
Task 7	8	2/3	1	3	1	3	• Determining the follow-up date if the patient does not have vision problems
Task 8	6	2/5	-	1	4	1	• providing the same treatment recommendations for all ages • Asking general questions of parents regarding this care
Task 9	4	3/2	-	-	1	3	• Recording information on vaccination at birth • Recording vaccination information in two parts: action and service provision • Determination of inappropriate follow-up date
Task 10	7	2/8	-	1	3	3	• Asking inappropriate questions to determine the child's status when receiving a medication supplement • Recording drug supplements on a separate page in the drug registration section • Determining the time for recording information and not recording information after that time
Total (%)	68(100)	2.71	2(2/9)	11(16/2)	23(33/8)	32(47/1)

Problems classification

The results of the classification of problems based on their impact on usability attributes showed that 27 problems (40%) affected effectiveness, 67 problems affected efficiency (99%), 68 problems (100%) affected satisfaction, 4 problems (6%) affected learnability, 7 problems (10%) affected memorability, and 23 problems (34%) were classified as error attributes. However, if a problem was assigned by three or more users to one of the usability attributes, it is classified as that attribute. After the review assignment of each problem, 47 problems (69%) were classified as user satisfaction and 45 problems (66%) were classified as efficiency attributes, and one problem (1.5%) was classified as error attributes. The task of data entry regarding children’s height had the greatest number of problems affecting efficiency, and the task of data entry regarding children’s nutritional status had the greatest number of problems affecting efficiency.

Discussion

A total of 68 usability problems were identified in the user interface of SIB for recording childcare data using the TA method. The majority of the identified problems were classified as major and catastrophic, and common problems included: Asking general questions of parents, Data entry on multiple pages, Improper diagnosis and treatment recommendations. Also, more problems were classified under the satisfaction and efficiency attributes.

The majority of the identified problems in this study were classified as major and catastrophic. The results of studies [18] (16 participants), [19] (10 participants), [20] (5 participants), and [21] (8 participants) indicated that the usability problems identified using TA were predominantly major issues, which is rather consistent with the present study's findings. The presence of catastrophic problems in SIB may be related to the scope of the system and its nationwide implementation. Evaluations utilizing heuristic and cognitive walkthrough methods identified major and catastrophic issues in SIB [8, 9], indicating severe problems exist regardless of the evaluation approach. These findings necessitate taking serious corrective steps.

Common problems across most tasks are discussed in the following paragraphs

Asking general questions from parents

This problem was evident across tasks 3, 4, 5, 8, and 10. Asking specific questions about indicators directs users' attention and emphasizes completeness [22]. Specific questions elicit more accurate and reliable patient information [23]. Accurate and reliable information facilitates monitoring children's health status and the level of care provided. Therefore, it is suggested that before system design, healthcare providers trained in childcare determine specific questions based on health indicators, paper records and their experience, and then incorporate these questions into the system.

Data entry on multiple pages

This problem was evident across tasks 1, 2, 3, 4, and 5. The placement of data elements on multiple pages has caused a waste of manpower time and reduced system efficiency. The results of this study are in line with the results of Farrahi et al.'s study, in which users mentioned the placement of information on multiple pages as the reason for the prolongation of tasks and poor system efficiency [24]. It is recommended to remove unnecessary data elements or to use the space of the user interface more efficiently to insert additional data elements, thereby minimizing the number of pages required for each task.

Improper diagnosis and treatment recommendations

This problem was evident across tasks 1, 2, 6, and 10. Health information systems are designed to improve the efficiency and safety of healthcare processes [25]. The content of these systems must be evidence-based, accurate, complete, and error-free. This information must be understandable to authorized users, prioritized based on user needs, and applicable [26]. If the recommendations provided by the system do not fit the patient's condition, the user will be confused and spend a lot of time reviewing. On the other hand, if the system causes doubts for the patient, it will lead to unnecessary follow-up of treatment. To prevent unwanted consequences and safety problems for the patient, it is recommended that the content of the system be determined by clinical specialists, then the system be tested in a real environment with real information after designing it.

More problems were classified under the satisfaction and efficiency attributes. This shows that the users are not satisfied with SIB and, from their point of view, the system does not demonstrate efficiency. Hu et al. (2008), in evaluating the user interface of a commercial website using the think-aloud method, stated that 39 problems (32.2%) out of 121 identified problems effected user satisfaction. Similarly, in our study, the efficiency attribute, with 36 problems (29.8%), was ranked second after satisfaction in terms of the number of problems [27].

Additionally, the results of our study align with Khajouei's findings, which showed that problems identified by the TA method had a 45% effect on user satisfaction and a 21% effect on system efficiency [5]. In the study by Sadeghi et al. (2021), most of the problems identified using the heuristic method in the nursing information system were classified under the satisfaction and effectiveness attributes. Similar to Sadeghi et al.'s study, the majority of problems identified in this study were classified under the satisfaction attribute. However, in our study, most problems after satisfaction were classified under the efficiency attribute. In contrast, effectiveness was the second most common classification in Sadeghi et al.'s research. The reason for this inconsistency may be attributed to the evaluation method [28].

In the expert-based method, experts cannot complete the tasks with the system because it is not part of their daily tasks. In contrast, in the user-based evaluation method, users are obliged to perform their daily tasks with the system, which allows them to complete the tasks, although system problems may prolong the time needed. The classification of most problems under the satisfaction attribute can be linked to factors such as the lack of user participation in the development of the system and the utilization of their experiences, insufficient organizational support in addressing system issues, and users' attitudes toward using the SIB system. Therefore, it is recommended that the usability of systems be evaluated using a user-based method in the initial stages of software design. This approach allows for the identification and correction of problems that negatively affect user satisfaction through iterative revisions.

Strengths and weaknesses

This study has several strengths. First, it was conducted with the participation of end-users in a real-world environment using realistic scenarios. This enabled the identification of problems with the SIB system, providing system designers and developers with valuable insights to guide improvements. Users contributed voluntarily. Additionally, this study classified the identified problems under the supervision of experts who were familiar with usability attributes. The classification was based on the participants' opinions as system users, considering the impact of the specified problems. In this qualitative investigation, to quantify and enhance comprehension of user-expressed issues, participants were prompted to assess problem severity using the heuristic evaluation scale. Subsequently, the identified problems were categorized, so that the ones that were major and catastrophic would be prioritized for correction.

There were some limitations in our study. The- think-aloud method was used only for a limited set of tasks in the SIB system, making it difficult to generalize the results of this study to the entire SIB system. However, we tried to select the most frequently used tasks to maximize the relevance of the findings to typical usage scenarios. Users might not receive comprehensive training on the TA method, which has an impact on the identification of the problem. This problem was somewhat mitigated by providing a training session to acquaint users with the TA approach and by having the researcher serve as a facilitator during the evaluation session. The transcription errors by the single evaluator were decreased because the transcription was performed with the evaluator's presence during the think-aloud sessions, and also the number of video files from think-aloud sessions was low. Participants were less willing to express their thoughts due to their personality traits or fatigue. However, the aim was to select active users with high communication skills since the TA method is suitable for participants who freely express their ideas. Health care providers were in the health centers affiliated with Kashan University of Medical Sciences, and all ten participants were female. This, because the caregivers of the children in comprehensive health centers are women.

Future implications

Although this system has completely replaced paper documentation in primary care centers and has overcome its disadvantages, users have reported that the existing problems and shortcomings of the system prolong the care process and data collection, reduce face-to-face communication with children and parents, shorten the time for training parents to care for children, and lead to user dissatisfaction. Therefore, it is recommended that the problems identified from the users' point of view be resolved by design and support companies with the support of the organization, as this is one of the factors influencing user satisfaction. Once the problems are corrected, user satisfaction and quality of childcare can be improved through careful examination and better training. It is also recommended that future studies evaluate the processes related to prenatal care, as this is one of the most important care processes in health centers.

Conclusion

Usability problems of the national health information system were identified using the TA usability evaluation and classified into major and catastrophic categories, which primarily indicate the poor usability of the user interface in registering data related to childcare. Most of these problems impact efficiency and user satisfaction. Therefore, revising system problems will help improve user satisfaction, devote more time to caring for and educating parents, and improve quality of care.

Abbreviation

TA

Think Aloud

Authors’ contributions

E.N., F.R.J., and R.B. designed the study. F.R.J. supervised the project. R.B. and R.F. performed the experiments. E.N., R.B. and R.F. analyzed the data. All authors discussed the results and reviewed and approved the final manuscript. R.F. wrote the final manuscript.

Funding

This study was part of an M.S. thesis in the field of health information management which was funded the entire research process by the deputy of research at Kashan University of Medical Sciences with the grant number Reg. Code: 96049. It should also be noted that the funding bodies played no role in the design of the study, the collection, analysis, and interpretation of data, or in writing the manuscript.

Data availability

All data presented in the result but the raw data in a text file that support the findings of this study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study was part of a M.S. thesis in the field of health information management which was approved by the Ethics Committee of Kashan University of Medical Sciences (IR.KAUMS.MEDNT.REC. 1396.6). In accordance with the opinion of the Ethics Committee mentioned above and given the fact that no information about participants is provided in this paper, participants who participated in this study gave informed verbal consent to participate in this research.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.