Development, Implementation, and User Evaluation of COVID-19 Dashboard in a Third-Level Hospital in Iran

Abstract

Introduction  The implementation of a dashboard enables managers to make informed and evidence-based decisions through data visualization and graphical presentation of information. This study aimed to design and implement a COVID-19 management dashboard in a third-level hospital in Mashhad, Iran.

Materials and Methods  This descriptive developmental applied study was conducted in the second half of 2020 in three stages, using user-centered design methodology in four phases: (1) specification of the application context, (2) specification of requirements, (3) creation of design solutions, and (4) evaluation of designs. Data collection in each phase was performed through holding group discussions with the main users, nominal group techniques, interviews, and questioners. The dashboard prototype for the data display was designed using the Power BI Desktop software. Subsequently, users' comments were obtained using the focus group method and included in the dashboard.

Results  In total, 25 indicators related to input, process, and output areas were identified based on the findings of the first stage. Moreover, eight items were introduced by participants as dashboard requirements. The dashboard was developed based on users' feedback and suggestions, such as the use of colors, reception of periodic and specific reports based on key performance indicators, and rearrangement of the components visible on the page. The result of the user satisfaction survey indicated their satisfaction with the developed dashboard.

Conclusion  The selection of proper criteria for the implementation of an effective dashboard is critical for the health care organization since they are designed with a high-tech and content-based environment. The dashboard in the present study was a successful combination of clinical and managerial indicators. Future studies should focus on the design and development of dashboards, as well as benchmarking by using data from several hospitals.

Background and Significance

The coronavirus disease 2019 (COVID-19) was first detected in Wuhan, China, on December 30, 2019. The World Health Organization recognized the disease as a global epidemic on March 11, 2020. 1 The outbreak quickly turned into a critical condition for the health systems around the world. 2 3 Given the priority of treating COVID-19 patients, health care systems devoted the majority of their resources to combat the epidemic, which in turn led to the reorganization of health care services. 2 4 Decision making during the COVID-19 pandemic was a challenge for health system managers, especially hospital managers at all organizational levels, due to the unknown characteristics of the emerging disease; lack of vaccines, drugs, and acceptable therapies; the complexity of the disease in terms of clinical manifestations and modes of transmission; and the unpredictable consequences of the disease on one hand and its devastating and deadly waves on the other. 5 Moreover, the decisions of health system managers at different levels have an important impact on the effectiveness of health services 6 and the success of health organizations in controlling the COVID-19 pandemic. 4

In the case of the COVID-19 pandemic, the reaction of health systems has been faster and much more complex than normal. 7 However, health system managers and decision makers faced more challenges in the process of decision making and the provision of appropriate and timely responses due to the lack of proper understanding of the environment and the high volume of information they are confronted with in critical situations. 8 9

Based on the previous experiences, the main challenges of decision making in the current situation include misunderstanding the scope of the current situation, as well as information overload experienced by the decision makers. 8 In such critical situations, it is very important to design a suitable system that collects, analyzes, and reports information from available sources. 10 In addition, having access to high-precision reports can facilitate the implementation of support operations, identification of system weaknesses, and provision of effective solutions for reliable planning. 11 12

In this regard, technical advancements and integration of all provided information in a suitable field accelerate decision making. 13 14 Data visualization has provided a huge capacity for the examination of different dimensions of information and the visual exploration of the relationships among different components. 15 Dashboards are one of the most important data visualization tools. 16 They make data evaluation easier and assist users in analyzing information through the processing of information and identification of factors that can be evaluated. 17 Consequently, dashboards provide the possibility of informed and evidence-based decision making for the managers. 15

The first step in the development of a dashboard is the selection of main performance indicators and the determination of a basic relationship between them. 18 Research showed that a dashboard should provide users with content that is tailored to their needs and be designed in a way to be easily understood by different users. 19 20 It should be noted that valid and reliable data for the user should be displayed completely, accurately, and timely. 20 Moreover, the dashboard should be easy to use and facilitate the process of decision making. 21

The challenges of patient care administration and monitoring suspect cases made it vital to know about the number of daily cases, diagnosis results, and transfers of COVID-19 patients to hospitals for treatment. During the COVID-19 epidemic, various studies have designed information dashboards at different levels to better manage COVID-19 patients. 22

In most previous related studies, information dashboards have been designed nationally to address the demographic characteristics of patients with COVID-19. Berry et al developed open access epidemiologic data and an interactive dashboard to monitor the COVID-19 outbreak in Canada, which included such information as demographic characteristics, location, report date, travel history, and exposure source. All data are openly accessible and updated daily. 23

Dong et all developed an online interactive dashboard, hosted by the Center for Systems Science and Engineering at Johns Hopkins University, Baltimore, Maryland, United States, to visualize and track reported cases of COVID-19 in real time. 24

Moreover, some studies have designed hospital-level dashboards to track the status of COVID-19 patients in hospital wards. For example, Vizcaychipi et al developed a COVID-19 near real-time traffic light system in an acute hospital setting. The input variables were age, gender, first recorded blood pressure, respiratory rate, temperature, heart rate, indices of oxygenation, and C-reactive protein. 25

Therefore, this study aimed to design (develop) and implement a COVID-19 management dashboard at the hospital level in Iran.

Methods

This descriptive developmental applied study was conducted to develop and assess a COVID-19 dashboard in a big hospital in Mashhad, Iran, using a focus group technique and a panel of experts. This dashboard was designed in Imam Reza Hospital, which is a tertiary care teaching hospital in Mashhad, Iran. This hospital has approximately 1,000 beds and provides care to around 6,000 inpatients and 19,000 emergency patients monthly.

User-centered design (UCD) was adopted for the development of the dashboard. The UCD is an iterative design process in which designers and other stakeholders focus on the users and their needs in each phase of the design process. 26 The UCD process ensures a stable integration of later users throughout all phases of the project. With more user focus, the UCD method is one of the most suitable methods for the recognition of the user's problems. 26 27

The general phases of the UCD process include:

1. Specification of the context of use: Identification of the people who are supposed to use the product, the purpose of use, and the conditions of use.

2. Specification of requirements: Identification of any business requirements or user goals that must be met for the success of the product.

3. Creation of design solutions: This part of the process may be done in stages, building from a rough concept to a complete design.

4. Evaluation of designs: Evaluation—ideally through usability testing with actual users—is an integral part of software development. 26

In other words, UCD steps for the development of a dashboard include identification of goals, users, data, design and layout, user feedback, dashboard design, and overall best practices.

This study was performed in four steps according to UCD stages.

The main goals of designing this dashboard for real-time analysis and management of COVID-19 disease and the general features of this dashboard were examined through brainstorming during the first focus group discussion. This focus group discussion was held with the presence of 10 potential users of the system, including hospital managers, nursing managers, heads of relevant departments (infectious disease specialists), and heads of COVID-19 committees (emergency medicine specialists). The main criteria for the selection of participants included their experience and knowledge of hospital management and their willingness to participate in the study.

In the second phase, the focus group discussion was held in the hospital, with those who were present in the previous focus group, and the meetings were chaired by the hospital manager. Initially, users were asked such questions as “What information do they need to better manage COVID-19 in their field?,” “Why do they need this information?,” “When do they need this information?,” “How should they access this information?,” and “Where should they get the needed information?.”

Moreover, the focus group discussed any user goals that must be met for the success of the product. Therefore, the necessary approach for getting access to each piece of information, as well as the information sources for inclusion in the COVID-19 information management dashboard was identified subsequently.

At the end of this phase, the comments were written and summarized, and voting was done using the nominal group technique in the form of a list of information that needs to be placed in the COVID-19 information management dashboard. The nominal group method was used to analyze the focus group data and get a conclusion at the end of the sessions. Access to each required indicator and information resource was provided and approved by the majority (75%) of the participants.

Subsequently, the study team including five specialists (i.e., two physicians, one nurse, one health information management, and one manager) classified indicators in three classifications of inputs, process, and output after the second focus group meeting, according to related previous studies.

In the creation of the design solutions phase, the specialized software design team developed the dashboard prototype using the Power BI Desktop software based on the identified data elements and requirements from the previous step.

The use of colors in a dashboard can be a plus; however, it is necessary to choose them according to known rules, as indicated in the study conducted by Pestana et al. 28 In this regard, different colors were applied to indicate the importance of information. In this regard, dark-colored fonts were used to highlight information that required more user's attention, 28 and the barely discernable pale background color was used to provide a more soothing and less starkly contrasting surface on which the data can reside. 29

Afterward, the designed prototype was integrated with the hospital information system (HIS) and its capabilities and visual features were displayed to the users. In the final phase (i.e., evaluation of the designs), 1 week after the prototype was available to users, the third focus group was held with the same participants who had attended the second focus group. These 10 main users were asked to comment on suggestions provided for upgrading the designed dashboard. Furthermore, their satisfaction was measured based on the end-user satisfaction model with such items as content review, accuracy, ease of use, and timeliness using a questionnaire that was scored based on a five-item Likert scale, similar to that used in the Rouhani and Zamenian study, 21 in which very good = 4, good = 3, moderate = 2, bad = 1, and very bad = 0. The validity and reliability of the questionnaire were confirmed. The dashboard software was then provided to participants to evaluate after working with the dashboard through the completion of a designed paper questionnaire. The mean ± standard deviation (SD) of the questions were calculated as well.

Eventually, the final version of the dashboard system was developed after the comments and suggestions received from users and the required changes were applied to the designed prototype.

Results

Opinions from potential users of the COVID-19 information management dashboard were gathered in the initial phase of the study. According to the comments, objectives of dashboard development included the possibility of informed and evidence-based decision making for the managers, easier analysis of information through information processing, and identification of the main performance indicators and factors that can be evaluated. Regarding general features dashboards, participants agreed that dashboards are likely to solve the problems of presentation format and information load when certain visualization principles and features are present (e.g., high data-ink ratio and drill-down features). Dashboards should have some level of flexibility so that users can switch between alternative presentation formats. Pop-ups and warnings can help users select an appropriate presentation format.

At the end of the session, the main users of the dashboard were given their votes. The criteria for selecting these individuals included membership in the hospital board of directors or one of the COVID-19 committees, such as the Nosocomial Infection Committee, and at least 2 years of working experience in the hospital.

The mean ± SD age and mean ± SD work experience of the selected main users were 44 ± 4.11 and 12 ± 6.04 years, respectively. The participants in the study included four females and six males and their demographic information is presented in Table 1 .

Table 1. Demographic characteristics of participants.

Variable	Frequency (%)
Gender
Female	4 (40)
Male	6 (60)
Age a
36–41	3 (30)
42–47	5 (50)
48–53	2 (20)
Education status
Bachelor of Science	4 (40)
Master of Science	3 (30)
Specialist	2 (20)
PhD	1 (10)
Field of study
Health economics	1 (10)
Health information management	2 (20)
Management	2 (20)
Nursing	3 (30)
Medicine	2 (20)
Work experience b
5–10	4 (40)
11–16	2 (20)
17–22	3 (30)
23–27	1 (10)

^aMean ± standard deviation (SD) age is 44 ± 4.11.

^bMean ± standard deviation (SD) work experience is 12 ± 6.04.

In the second phase, the main information requirements of caring for COVID-19 patients were identified with the participation of the main users. Based on the available data, 25 indicators have been selected to be displayed in the COVID-19 dashboard. For a better understanding of information, participants decided to classify the selected indicators into three categories (performance, clinical, and demographic categories) ( Table 2 ). Data on these indicators were extracted from the HIS, automatically. Moreover, at the request of participants in the focus group, users could visualize the data summary of each patient and compare the data obtained from different departments. It was suggested that the number of admissions, discharges, and deaths over a period of time could be observed as well. Moreover, it was decided to use a table to show the average request for the diagnostic tests.

Table 2. Indicators for display in a dashboard.

Criteria	Indicator	Percentage of agreement
Input	The number of patients admitted to triage	76%
	The number of patients admitted to the emergency department	88%
	The number of patients referred from other centers	78%
	The Number of empty beds	94%
	The gender of patients	96%
	The age of patients classified into 19 groups (based on the standard table of life expectancy pertaining to World Health Organization [57])	81%
	Location of living (Urban or rural)	77%
	Job title (Based on 6 job titles, including unemployed, employed, retired, housekeeper, student, health professional identified in HIS)	96%
	The number of severe COVID-19 patients	92%
Process	The Number of occupied beds	75%
	The mean length of stay in departments for COVID-19 disease	95%
	The mean length of stay in ICUs for COVID-19 disease	95%
	The mean time between graph request and response	99%
	The mean time between laboratory tests request and response	84%
	The mean time between CT-scan request and response	93%
	The number of laboratory tests requests	84%
	The number of requested radiographies	82%
	The number of CT-scan requests	93%
	The number of COVID-19 specific drugs	93%
	The number of patients ready to be discharged from the inpatient department	91%
	The number of patients ready to be discharged from an emergency department	87%
Output	The number of deaths	93%
	The number of patients discharged	84%
	The number of patients referred to other centers	98%
	The number of patients leaving the triage	88%

Abbreviations: CT, computed tomography; HIS, hospital information system; ICU, intensive care unit.

Dashboard users could view demographically classified data, such as gender-based data. In total, eight items were suggested by the second focus group as dashboard requirements: (1) the ability to access the system through Web browsers without the need to install a specific platform or operating dashboard, (2) the ability to connect to the health information system for data gathering and analyzing, (3) the capability to define access levels for the users, (4) the possession of user-friendly dashboard, (5) the ability to change, develop, configure, and service the dashboard, (6) the use of color capabilities in the dashboard, (7) the ability to report online and drill down, and (8) the possibility to visualize data over a period of time.

Subsequent to the development of the initial dashboard prototype using the Power Bi server, each user was given a URL to use the dashboard, and access levels were also defined.

According to the access level set in the initial focus group of the prototype dashboard, it was possible to provide daily, weekly, and monthly reports according to the selected indicators. It was also possible to create exception reports based on the users' requests. This prototype dashboard utilized data from March 2020 to August 2020 based on HIS. The HIS is an administrative and clinical database containing data from all departments in the hospital, which provided care to COVID-19 patients. The prototype dashboard presents summary data of COVID-19 patients at the hospital level for the main users.

In the third step of the focus group, suggestions were made by the members of the focus group to upgrade the dashboard according to the feedback provided by the users. After consultation with the technical team for the modification of the relevant dashboard, it was agreed that the number of deaths, admissions, and discharges should be displayed in three separate graphs, and the distribution of patients by days of hospitalization and the average length of stay be displayed in a graph as well. Afterward, a questionnaire was provided to participants and their satisfaction with the designed dashboard was measured. The internal and external validity, reliability, and statistical validity of this questionnaire have been approved by Aggelidis and Chatzoglou. 30 The average user satisfaction of the model is presented in Table 3 .

Table 3. User satisfaction of the dashboard.

Variables based on EUCS model 29	Research variables	Average
The independent variable format	Satisfaction with the dashboard format	4.782 ± 0.321
The independent variable being up to date	Satisfaction with up-to-date status of the dashboard	4.402 ± 0.987
The independent variable ease of use	Satisfaction with the dashboard's ease of use	3.985 ± 0.654
The dependent variable final user satisfaction	Overall satisfaction of the dashboard	4.36 ± 0.451
The independent variable content	Satisfaction with dashboard content	4.46 ± 0.475
The independent variable accuracy	Satisfaction with the accuracy of the dashboard	4.191 ± 0.430

Abbreviation: EUCS, end-user computing satisfaction.

The images of the report page of the designed information dashboard are presented in Figs. 1 2 3 .

Fig. 1.

The final dashboard visualizes data for patients screened for COVID-19 disease. The prototype dashboard presents a deidentified patient record populated with clinical data (imaging diagnostic tests).

Fig. 2.

The number of daily admissions, daily discharges, and daily deaths in a sample time period.

Fig. 3.

Distribution of length of stay and mean ± standard deviation (SD) of patient age in deidentified department.

Discussion

As previously mentioned, the main objective of the present study was to develop and implement a dashboard system to manage the crisis caused by COVID-19 in a third-level hospital. In this regard, the main indicators and requirements of such a system were identified. Therefore, identification of the main indicators and requirements of the developed dashboard through the participation of potential users in all stages of development and implementation was the difference between this study and similar studies. 25 31

Some of the previous studies on dashboard design for pandemic diseases, such as COVID-19, 1 4 severe acute respiratory syndrome, and H1N1 influenza, 32 33 34 had their focus on the clinical aspect and analysis of data outcomes. The ignorance of users' views and their participation in designing such information systems has led to their lack of interest in such systems. 35 36 Therefore, the present study focused on the identification of the information needs and requirements of such a system from the users' point of view. Based on the study results, several methods were used to identify and extract indicators and requirements of the dashboard system which included investigation of similar information systems, user interviews, and questionnaires. 37 38 39

In this study, a UCD was adopted for the development, implementation, and evaluation of the COVID-19 dashboard. The results of a study performed by Setyawan et al indicated that this method had a positive relationship with supporting user's participation in the process of application development. 27 In studies in which a similar method has been used, researchers designed a safety dashboard for patients as well. Their findings showed that this method increased patients' participation. 40 41 Another study based on this method showed that according to the users' feedback, visualizations improved situational awareness and have probably provided valuable information to facilitate informed operational decision making. 42

In this regard, indicators and requirements of the dashboard system were identified through the examination of similar dashboard systems and interview with users through focus group meetings.

Users tend to utilize a variety of tools, such as mobile devices, especially smartphones, as well as other electronic devices, including personal computers (PCs), laptops, and tablets to access information in Web-based applications, such as dashboards. 37 43 The designed dashboard in the present study is Web-based and can be accessed by all users through common browsers on electronic devices, such as PCs, laptops, and tablets, without the need for installation on a specific operating system.

Reasonable decision making is a key element in health care organizations at managerial levels. A dashboard is a tool that can integrate data from several different sources, summarize the key performance indicators (KPIs), and provide them to users in a colorful graphic in real-time to facilitate informed decision making. 38 Studies on the design of COVID-19 management dashboards showed that these dashboards connect to and integrate with other health information systems, such as electronic health records and health information systems, to receive data from these systems. 14 39 44 45 Similarly, the dashboard designed in the present study was integrated with the HIS. Therefore, the data could be used in real-time to display and manage COVID-19 disease.

Determination of accurate KPIs is one of the primary requirements in the development of a dashboard. These indicators should be developed in accordance with the needs and goals of the given organization to provide valid and reliable measurements. 46 47 48 Demand for intensive care unit (ICU) services will increase under conditions similar to the COVID-19 pandemic. Therefore, patient triage is critical in situations like this since there is always a risk that patients may not be properly admitted to the ICU which may cause more harm and injury to the patients. 35 Regarding the fact that the triage, emergency, and ICU departments are involved in the process of providing health care services to COVID-19 patients, it is important to monitor the workflow of these units. Accordingly, the indicators required for COVID-19 management in the present study were identified through the assessment of the information needs of dashboard users. These indicators were classified into three main groups based on clinical and managerial needs of users, including performance, clinical, and demographic criteria. Subsequently, the output data and reports of the dashboard were displayed to managers and clinical specialists in form of color charts and tables. Based on the results of previous studies, the use of colors in dashboard design led to a better understanding of the situation and improved data visualization which in turn reduced the response time and error rate. 36

Similarly, in the study conducted by Grange et al, the related key indicators were identified and applied to the designed COVID-19 management dashboard. 14 Furthermore, interactive Web-based dashboards have been designed during the COVID-19 pandemic surge in different countries to help managers and clinicians better manage the limitations in human resources, medicine, and medical equipment and take more effective actions. 1 31 49 50

In addition, user satisfaction is one of the important factors affecting the acceptance and application of different software. 51 However, the review of studies related to the design of COVID-19 dashboards showed that user's satisfaction with the designed dashboard has been examined in a small number of them. 52 In the same line, user's satisfaction with the designed dashboard was examined in the present study in terms of the simplicity of use and the extent to which the information and operational needs of dashboard users were met.

As a result, the examination of users' satisfaction and application of their comments and feedback in dashboard design led to improved dashboard performance and increased users' acceptance of the designed dashboard.

It seems that the structure and minimum data included in this dashboard can be used for similar hospitals due to the similar nature of the disease.

Regarding the limitations of the present study, one can refer to the fact that due to the critical situation of the COVID-19 pandemic, focus group interviews with potential users were withheld. Moreover, since the present study was conducted in only one center, the number of participants involved in the development and implementation of the dashboard was small.

Conclusion

Information dashboards are important for the provision of access to necessary information in the shortest possible time. Information dashboards keep managers informed about the process of patient care, admission, discharge, and incidents, such as the number of deaths and the use of hospital services. The application of this tool leads to more evidence-based decision making in the crisis and reduces the likelihood of error and failure in crisis management.

Furthermore, the simplicity of design and the possibility of connection to the HIS database allows administrators and users to access the reports they need with higher confidence and speed and avoid the repeated collection of the required information.

Eventually, the dashboard developers need to receive periodic feedback from the users, update their information needs, and evaluate the quality of data and dashboard information to preserve the users' desire to use this system continuously and make the most of such information management dashboards.

Clinical Relevance Statement

This study can be used as a guide for the implementation and development of a dashboard for managing various crises, such as COVID-19 disease in hospitals.

Multiple Choice Questions

1. According to the USD method, which is not one of the general phases?

   1. Specification of the context of the use

   2. Specification of requirements

   3. Creation of design solutions

   4. Development a prototype

   Correct Answer: Option a is the correct answer. The final phase of the USD method is evaluating designs: Evaluation—ideally through usability testing with actual users—is as integral as quality testing is to good software development.

2. Dashboard indicators in this study are divided into how many main categories?

   1. One

   2. Two

   3. Three

   4. Four

   Correct Answer: Option c is the correct answer; three categories including inputs, process, outputs.