Systematic mapping of digital health apps – A methodological proposal based on the World Health Organization classification of interventions

Abstract

Coronavirus disease 2019 was identified as a pandemic and Brazil is one of the major epicentres. One of the Brazilian states affected is Ceará, where this research group works. This group was challenged by a Hospital stakeholder to develop a communication channel with the health professionals and the coronavirus disease 2019 patient's family. This article presents a part of this whole project. The main methodological approach was the user-centred design based on user experience elements. Benchmarking was applied to understand the state-of-art of Brazilian apps that were related to coronavirus disease 2019. The research process was based on a systematic approach that was carried out by a multidisciplinary team that worked through four work cycles (identification, classification, screening, analysis). This work was based on two main points: (a) World Health Organization digital health guidelines, specifically digital health interventions (b) System Usability Scale. As a result, apps features were gathered according to the digital health interventions and their experiences were analysed on System Usability Scale. This work has provided an overview of apps that were available and how they support the coronavirus disease 2019 context. Another valuable contribution is the understanding of how the industry was satisfying the user's needs. These two results can provide a holistic view for future product development that can be used in different contexts of health issues. One of the highlighted conclusions was that digital health interventions should be adapted to the local context because these World Health Organization guidelines were open. Moreover, the System Usability Scale is an effective method to compare different digital health solutions.

Introduction

Coronavirus disease 2019 (COVID-19) was identified as a pandemic by the World Health Organization (WHO) in 2020¹. Within the context above, Brazil is one of the major disease epicentres. This country identified this pandemic as a Public Health Emergency of National Importance.¹ It is a fact that all Brazilian states were impacted by COVID-19.² One of them is Ceará, where this research group works.

The Ceará health services were overloaded during the pandemic stage and one of the stakeholders that suffered more was the Unified Health System (System Usability Scale (SUS)). This public health system is one of the largest universal systems in the world.³ As a consequence of this emergency in Ceará, the bedload rates in outpatients and intensive care units kept above the Brazilian level, reaching 100% in some periods. It led SUS to face a huge challenge, increasing the risk of intra-hospital transmission. This scenario resulted in a ban on hospital visits and consequent communication difficulties between patients, their family and healthcare professionals. Family members complained about the gap or complete absence of clinical information. The patient suffered an interruption of contact with their relatives after admission.⁴ Health professionals felt overwhelmed by the emotional issues, with the increasing number of admissions and deaths.

In light of this challenging scenario, the research team was challenged by the stakeholder Hospital Geral de Fortaleza to overcome the communication problem refered before. This is the largest public hospital of Ceará and it is a reference in highly complex procedures, such as transplants, neurosurgeries, and stroke care assistance. However, this hospital was dedicated to support the COVID-19 treatment.

The project had the main goal of developing a channel to support the communication between the hospital and the patient’s family. This channel should provide information about the COVID-19 patient journey in the hospital context. As a key factor for the project’s development in the field of digital health, the research team established to base their work on WHO guidelines,⁵ namely the Classification of digital health interventions (DHIs) v1.0.⁶ This classification intends to support the interoperability of health systems based on Digital Health Technologies sustainability. It targets priority points such as ethical, safe, reliable, equitable, and sustainable digital systems focussed on patients’ needs.

The methodological approach of the whole project was the user-centred design (UCD) proposed by Garrett.⁷ Strategy is the initial phase and it was previously carried out through field studies in Hospital Geral de Fortaleza. During this phase, the research team gathered and analysed the user needs (health professionals, COVID-19 patients, and their families). Besides that, the team decided to add a new method to understand the market – Google Play store and their offer about mobile apps. The project involves a complex context because there are problems coming from different knowledge areas. This led the head researchers who are experts in Benchmarking method to invite a multidisciplinary team to answer these challenges. Thus, the research team consists of eleven participants: three user experience/user interface (UX/UI) experts and two junior researchers; one information and communication technology (ICT) senior and two junior researchers; three Public Health researchers.

The head researchers in Benchmarking discussed and established a set of questions for this project phase. These questions guided the whole research process. The main one was:

• What is the state-of-art of Google Play store apps related to COVID-19 communication context running in Brazil?

Besides that, there were other questions which emerged based on the main question:

• How are these apps supporting the citizens? It is intended to measure the user experience (UX).

• Are those apps considering WHO digital health guidelines? The purpose is to understand if the apps are aligned with WHO strategy.

• How is the industry interpreting the user’s needs? The goal is to provide a holistic view to the future product development.

The researchers involvement was articulated through four work cycles. One interesting result was the amount of apps that met the criteria, from the 30 apps identified in the I Cycle, only seven were selected to be analysed at the end – IV Cycle. Figure 1 summarizes what was explained previously.

Figure 1.

Research infographic.

The main goal of this article is to present a methodological framework and the results of this process. The reason why this team decided to create this framework is that the authors didn’t find a similar research process in previous literature. The article is divided into five sections. The first section presents the context where the research is happening. The second section describes the state-of-the-art, articulating the subject of Digital Health with Benchmarking. The third section presents the methodological framework composed of four work cycles (identification, classification, screening; analysis) within three phases (Planning, Implementation, Results). The fourth and fifth sections are addressed to discussion, conclusion, and future works.

The Digital Health landscape

The COVID-19 pandemic is testing the capacity of answering the challenges that are emerging in health systems. Several stakeholders such as governments, the private sector, and the academy are trying to provide fast, accurate, and efficient solutions to the population. Digital Health initiatives are one of the paths these stakeholders developed more. This course of action demonstrates to be essential because it keeps strengthening the activities in all health sectors.^4,8 Digital health tools have been crucial to overcome uncertainty in information, communication, and technological fields related to the health environment. Apart from this, these tools are considered solutions for patient’s empowerment. This enables them to achieve distinguished health and health care quality statements.⁵

For stakeholders, additional support and recommendations about Digital Health landscape are promoted at different levels: global (e.g., WHO); regional (e.g., Pan American Health Organization (PAHO)), and national (e.g., Brazil). Therefore, the WHO is considered the major player that publishes clinical, public health policies, and data recommendations worldwide. The regional office (e.g., PAHO) is aware of local issues adopting global recommendations. On the national level, each government should implement these recommendations through policies, procedures, and digital tools.⁹ The WHO is aware that this task is crucial and has been providing recommendations for several distinguish goals in digital health field. One example of that is the Global Strategy for Digital Health 2020–2025.¹⁰ This initiative aims to strengthen health systems interoperability through a sustainable development of Digital Health technologies. It established priorities regarding basic knowledge to benefit users on an ethical, safe, reliable, equitable, and sustainable digital system focussed on patients’ needs. In addition, these recommendations are centred on the principles of accessibility, scalability, replicability, interoperability, privacy, security, and confidentiality. On the national level, those entities which provide digital health services should consider them when they are developing the referred systems. To avoid misunderstood in this context, WHO created a classification⁶ presenting a taxonomy linked to different DHIs. This classification considers the way how the technologies are being used to support health care. It also aimed to understand how to implement digital health services promoting accessibility and establish the appropriate system requirements where the functionalities are defined regarding the users’ specific needs.

The DHIs are organized into four groups according to a target user: (a) Clients: they are potential or current users of health services, patient and caregivers; (b) Health professionals: are those considered as the major workforce of health services; (c) Health system or resource managers: both are involved in the administration and supervision of public health systems. The interventions related to the resource managers audience are reflected in the administration functions, health care financing, and human resource; (d) Data services: consists of developing specific functionalities that support a set of activities related to data. This task includes procedures such as collection, management, use, and exchange.

The WHO global vision for digital health is spread worldwide through regional offices. PAHO is one example of the Inter-American System as a specialized health agency. PAHO highlights the benefits of joining efforts to achieve a common goal in transforming the digital health sector.¹¹ It recommends eight principles – Figure 2 – that are connected with the Sustainable Development Goals published in 2030¹²:

1. Universal connectivity – this means to achieve digital interdependence in the health sector.

2. Digital goods – it involves the co-create process of digital public health products.

3. Inclusive digital health – it focusses to accelerate progress toward Digital Health implementation.

4. Interoperability – it establishes open, sustainable, interoperable channel enabling digital health solutions to communicate between them.

5. Human rights – it ensures the protection of human rights through working the legal instruments in all areas of digital health.

6. Artificial intelligence – this technology should participate in the global cooperation of any emerging digital health system.

7. Information security – it develops mechanisms for the confidentiality and security of information in digital public health environments.

8. Public health architecture – this recommendation defends that the design field works as a renewed architecture for the era of digital interdependence in the health sector.

Figure 2.

‘8 principles for digital transformation of public health’⁹.

Considering the national environment, the Brazil government has been developing several initiatives in Digital Health: National Policy of Information and Informatics in Health (2003); e-Health Strategy for Brazil³; Action Plan, Monitoring and Evaluation of Digital Health for Brazil 2019–2023²,¹³ and Digital Health Strategy for Brazil 2020–2028 (ESD28)³. This last initiative joins all the previous ones, updating and expanding them. The Brazilian Digital Health Strategy ESD28 presents an action plan where the main goal is to implement the Brazilian Digital Health Strategy answering the governance needs of SUS. Besides that, this plan is divided into three strategic paths: (a) strengthen the National Health Data Network (RNDS)⁴ to provide essential services in digital health; (b) building legal and policy actions to allow collaboration between distinguished stakeholders; (c) implementing a digital health ecosystem that promotes effective synergies between stakeholders.

Based on those entities’ policies and recommendations, it is needed to go beyond the previous literature and discuss with the local stakeholders about what are their new health and health care challenges. This research team started to develop a digital health system during the COVID-19 Pandemic, involving a public hospital. One of their major missions is to articulate the stakeholder challenges with WHO, PAHO and Brazilian guidelines considering the patient as the key participant of the health problem.

The development of a digital health solution

User-centred design proposal

The research project aims to develop a digital health solution promoting communication between patients, their families, and the hospital. To answer the project needs, UCD was chosen.^14–17 This process supports the development of a solution based on users’ needs, gaps, and desires. This strategic approach helped the research team in guiding project management, technology development, and product design. The team followed Garrett’s Framework⁷ defining a UX design strategy that helped with all work plan and research phases definition. This is an iterative process and the results from each phase are used as insights for the next phases. The goal behind this approach is to show the stakeholder involved in the project, that a digital health solution should promote positive experiences and it is often a result of a collective effort. To achieve that purpose, implementing a plan that goes beyond what is functional and aesthetic, reflects the user’s perspective.

The methodological structure⁷ that guided the research project goes from an abstract idea to a concrete product. This type of development succeeded through different phases: (a) strategy; (b) scope; (c) structure; (d) skeleton; (e) surface. The initial phase – Strategy – is the starting point. It will guide all process and it influences the achievement of goals. This step shows us a double task: first, gathering the user context and second, specifying the solution goals. Although Garret has mentioned these tasks in his book, he does not quote which are the specific UX methods that fit better with each phase. The research teams consider this issue unclear. Since the team have experience with previous projects from the same field, they decided to add some other methods to the design process: one is participatory design.^18–20 Helps with the understanding of user context. The other one is, Benchmarking²¹ to comprehend how the digital health solution goals were established. The last one should be used even to identify the available digital health solutions. This task will guide the research team with the solution objectives specification. Because it is needed to know what solutions are in the market, identifying the gaps to overcome and offer the right solution for the user target. The research team involved in the project believes that this strengthening of the Garret methodoly is more accurate and suitable for digital health solutions development.

According to the WHO,⁵ it is necessary to gather the acceptability of digital health solutions by patients and health professionals. This statement should guide the research deployment in everyday needs. This idea centres the clinical research issues on patients’ needs. For that, WHO suggests that digital solutions development should be focussed on UX.⁹ In view of this WHO report, the UCD approach has been used as a tool for the healthcare areas since 1992.²² Some studies also adopted a similar methodological approach²³ with the same characteristics as UCD, or using terms such as patient-centred design²⁴ and human-centred design (Rinkus et al., 2005).

Currently, there is a real impact on end users when involving UCD approach in project development in health context. It is clear that Design has now spread and strengthened its application as a process for several health challenges: Long-term healthcare; aging population; non-communicable diseases; well-being & mental health; social interaction & support; active living; environment and lifestyle.²⁵

UX benchmarking as a measuring tool

The benchmarking method serves to raise the strengths and weaknesses of the competition, allowing researchers to be aware of the performance levels currently provided by the market. Another interesting point is that this procedure reveals how the market solved the problem. It supports selecting the more suitable answer and adapting the product development.

As referred before, the research team adopted several methods to understand the user context in the Strategy phase from Garret framework.⁷ The results from this previous phase were used to base the project goal and consequently the gathering and analysis of current market offers. With this, the research team was able to discuss the gaps and identify future directions to develop the digital health solution. The type of Benchmarking that was chosen to conduct was UX Benchmarking. The reason why is because the team would like to measure the UX that is provided by screened apps (IV Cycle) This decision is based on the recommendation available in a Nielsen Norman Group article “7 Steps to Benchmark Your Product’s UX!⁵ that states it should be applied a usability test to compare the UX from different solutions. Although this article does not suggest a specific method to measure, it recognizes the importance of collecting quantitative data through UX metrics.

The research team decided to use the SUS instrument to measure the UX.²⁶ This instrument was chosen because it is one of the essential UX Benchmarking tools commonly used to measure the usability of digital solutions.²⁷ This is a quick and dirty usability test that works with a Standardized Usability Questionnaire.²⁸ This survey includes ten questions that should be answered using the Likert scale²⁹ (IV Cycle). SUS instrument, as a part of a whole research process, is used to compare the current UX in mobile apps available on the online market – Brazilian Google Play Store⁶. To implement the UX Benchmarking explained before, the research team defined a protocol including four phases.

This article explains the protocol, the methods and discusses the results of UX Benchmarking application. The whole procedure (Figure 3) was planned to be a systematic approach by two expert researchers in Benchmarking. It allowed them to conduct the analysis and evaluation without bias concerns. Results are described in the third section, IV Cycle – Analysis.

Figure 3.

User experience (Ux) benchmarking.

Methodological instrument

The proposed method carried out a comparative analysis process called Benchmarking. Its purpose was to validate if the current Brazilian apps development were based on the DIHs.⁶ As previously referred this process followed a systematic approach to avoid any bias concerns (Figure 3) Before the starting point, the head researchers assume the responsibility of these two tasks: (a) set up a goal for app’s research – the system should be focussed on a communication channel between professionals who work inside hospitals and families of COVID-19 patients; (b) state the researcher profiles to conduct the process – the Classification and Screening phases should involve a multidisciplinary team with researchers from UX/UI Design, ICT and Public Health fields. This strategy helped to minimize any source of bias because multiple team members were considered even if the whole process was guided by two expert researchers in the Benchmarking method with strong knowledge in Marketing and UX/UI design. After that, the systematic approach followed four work cycles summarized in Figure 3:

• I Cycle – Identification. It was the established search criteria and identified apps from the Brazilian Google Play store according to defined keywords.

• II Cycle – Classification. The identified applications from the I Cycle were classified according to the DHIs.

• III Cycle – Screening. The classified applications from the II Cycle were screened according to the matching DHIs related to the project goals.

• IV Cycle – Analysis. The screened applications were analyzed focussing on their functionalities and usability.

Furthermore, each work cycle was divided into three phases: (a) planning: In this initial phase is intended to list and organize all the future research actions; (b) implementation: It is planned to carry out the researches actions previously listed; (c) results: This last stage is where these actions are consolidated and reported into results. Thus, this article in the next sub-sections will explain each cycle with its three phases. Figure 4 advances a summary of the global results of the three cycles.

Figure 4.

PRISMA flowchart.

I Cycle - identification

Planning

This is the first stage where the search strategies and the selection procedures were planned through a systematic mapping approach.^16,29 This work was led by the UX/UI design team. The key advantages in terms of what led the decision into a clear understanding of how far the market offers related to COVID-19 generic apps goes, focussing the apps search on those which are related to the communication between hospitals and families. This activity was narrowed to the following cycles. The protocol was created through a joint discussion that followed a plan of action:

• Search source – Google Play Store. The decision behind choosing this specific store is because it is the most used store by the Brazilian population⁷.

• Analyzed categories – apps, without defined subcategory. The reason why is that the project aimed also to raise application trends.

• Keywords – COVID, 2019-nCov, Coronavirus, COVID-19, Sars, Sars-cov-2, Sarscov-2, Sars-CoV-2. The UX/UI team discussed with the head researchers a broad range of words related to the COVID pandemic context.

• Filling out a table with the following app characteristics – name, description, goals, downloads, number of ratings, rating score, playstore category, last update, if it is free or paid, Android version, institution, developer address, Google Play Store link and related keywords.

It was known that the mobile location services influenced the app search. Because of this, the junior researchers that were based in Brazil only searched for available local apps. After that, the head researchers established the following inclusion/ exclusion criteria:

• Inclusion criteria: should be an app; must be related with COVID-19.

• Exclusion criteria: not matching the above criteria.

Implementation

The head researchers improved the UX/UI team knowledge with an extra section explaining the importance of a well-established protocol. They aimed the team to comprehend the search strategies and recognize the inclusion and exclusion criteria. Moreover, they got together and carried out some examples to confirm if the protocol was suitable or not. The keywords range was divided into two groups: (a) COVID, 2019-nCov, Coronavirus, COVID-19; (b) Sars, Sarscov-2, Sarscov-2, Sars-CoV-2.

Thus, each researcher received a group of keywords and searched for one word at a time in the Google Play Store. The individual search was consolidated in individual tables. In cases of duplicate apps found, only one was filled out. After this stage, one head researcher and these two UX/UI junior researchers reviewed the results and the individual tasks. After this initial meeting, the juniors conducted a comparative app analysis joining the information from each table. The duplicates were discarded. During the search task, this part of the team found an issue using a different keyword, ‘SARs’.

The implementation process also returned many irrelevant results such as games apps for example. These outcomes were analyzed the application names and icons and, as the case may be, they were eliminated.

Results

The search phase explained before resulted in a list of 30 applications (Table 1). This outcome was used in the next phase. A detailed app characteristics see online for Appendix 1.

Table 1.

Google Play Store search results (I Cycle).

1	Coronavírus – SUS	11	Rede COVID-19 MOC	21	EscoLAR
2	Monitora COVID-19	12	A Salvus	22	Aula Paraná
3	SOS COVID-19	13	Tô de Olho	23	SASI
4	OpenWHO: Conhecimento para Emergências em Saúde	14	Personal Health	24	Ouvidoria 156 – Prefeitura do Município de Maringá
5	Atende em Casa	15	Agenda Fácil – Prefeitura SP	25	iPES – Plataforma Eletrônica de Saúde
6	WHO Info	16	ALEP SDR	26	Infoleg
7	Dados do Bem	17	Corona Rodeio	27	SeuVale Empresas
8	SOS Corona – DDCOVID19	18	Centro de Mídias SP	28	COVID19 – DXB Smart Ap
9	Colab	19	Educa + Mogi	29	Sars HRMS
10	Cachoeirinha Contra o Coronavírus	20	CAIXA|Auxílio Emergencial	30	SARS disease

WHO: World Health Organization; COVID-19: coronavirus disease 2019.

II Cycle: classification

Planning

After completing the I Cycle analysis, the head researchers concluded that there are many apps related to COVID-19 but not focussed on health context. Therefore, it was needed to select only those which were directly related to health. The head researchers also decided to organize a multidisciplinary expert team to discuss and help to identify what were the apps that matched with DHIs.

Implementation

The head researchers presented the whole process – objectives, practical examples, and expected outcomes – for the multidisciplinary expert team. This team was composed of four researchers (two head researchers coming from UX/UI fields, one expert on Public Health, and another expert on ICT). Each one of these received a file with the I Cycle table. Individually, they analyzed and matched the app’s description with one or more DHIs. The apps that were not related to the health field were discarded. After the conclusion of this task, the team returned to a new meeting to compare and discuss the individual results. In case of disagreement, a discussion session between all members of the team was carried out and the screens of the applications were revisited in Google Play Store.

Results

The multidisciplinary team checked that there were 18 apps related to health and twelve not related. The whole set is presented in Table 1. The apps not related were focussed on: (a) Educational issues that provided student office and virtual classroom –‘Centro de Medias SP’; ‘Educa + Mogi’; ‘EscoLAR’; ‘Aula Paraná’; (b) Public management that intended to be a channel between citizens and government –‘SASI’, “Ouvidoria 156 – Prefeitura do Município de Maringá’, ‘Infoleg’; (c) Business management that had goals such as purchase support, employee tracking – ‘SeuVale Empresas’, ‘Sars HRMS’; (d) social policy with the purpose of government support for citizens – ‘CAIXA | Auxílio Emergencial’, ‘Rede COVID-19 MOC’); (e) eGovernment that supported actions to keep remote governance activities (ALEP SDR). All these mentioned 12 apps were discarded.

The research team identified 19 DHIs that in their opinion were matched to apps (See online for Appendix 2). This analysis enabled them an understanding of what were the most common DHIs used on apps. 1.6.1 client, looked up in terms of health information, was the intervention with the highest achievement (9/19). The other eighteen gathered DHIs did not reveal a prevalence rating considering the 1.6.1 intervention.

The 19 DHIs are presented by ascending order⁸: 2.4.1 remote consulting between clients and healthcare providers (4/19); 1.4.3 active data capture/documentation by client (3/19); 1.5.2 reporting of public health events by clients (3/19); 4.3.3 map location of clients and households (3/19); 1.1.2 transmit targeted health information to the client(s) based on health status or demographic (2/19); 1.1.4 transmit diagnostics result, or availability of results to clients (2/19); 1.2.1 transmit untargeted health information to undefined population (2/19); 1.4.1 access by the client to own medical records (2/19); 2.3.3 screen clients by risk or another health status (2/19); 2.4.2 remote monitoring of client health or diagnostic data by the provider (2/19); 4.3.1 map location of health facilities/ structures (2/19); 1.3.1 peer group for clients (1/19); 1.5.1 reporting of health system feedback by clients (1/19); 2.8.1 provide training content and reference material to healthcare providers (1/19); 4.1.1 non-routine data collection and management (1/19); 4.1.3 data synthesis and visualizations (1/19); 4.1.4 automated analysis of data to generate new information or predictions on future events (1/19); 4.3.2 map location of a health event (1/19). In supplemental material, the results presented above are even more detailed. Thus, the range was very diverse, but it had one type with a relevant quantity (1.6.1).

The team concluded that in a range of eighteen health apps there were 1ten that had more than one related DHIs (10/19). It was found out these app matched to DHIs (See document Table 2): ‘Monitora COVID-19’ (6/19); A Salvus (5/19); ‘Tô de olho’ (5/19); ‘COVID19 – DXB Smart Ap’ (4/19); ‘Colab’ (3/19); ‘Cachoeirinha Contra o Coronavírus’ (3/19); ‘Personal Health’ (3/19); ‘Coronavírus – SUS’ (2/19); ‘Atende em casa’ (2/19); ‘Dados do bem’ (2/19); ‘SOS COVID-19’ (1/19); ‘OpenWHO: Conhecimento para Emergências em Saúde’ (1/19); ‘WHO Info’ (1/19); ‘SOS Corona – DDCOVID19’ (1/19); ‘Agenda Fácil – Prefeitura SP’ (1/19); ‘Corona Rodeio’ (1/19); ‘IPES – Plataforma Eletrônica de Saúde’ (1/19); ‘SARS Disease’ (1/19). The ‘Monitora COVID-19’ was the app with the highest value of DHIs classification (6 DHIs: 1.1.2, 1.2.1, 2.3.3, 2.4.1, 2.4.2, 4.3.3). This app was developed by the Brazilian government. The second place was divided between two apps: ‘A Salvus’ (5 DHIs: 1.31, 1.6.1, 2.4.2, 4.1.4, 4.3.2) and ‘Tô de olho’ (5 DHIs: 1.1.2, 1.5.1, 1.5.2, 1.6.1, 4.3.3). ‘A Salvus’ is a solution created to support business and ‘Tô de olho’ is an academic initiative. These three apps also included location services and two of them use geolocation (‘Monitora COVID-19’ and ‘Tô de olho’). The third place was ‘COVID19 – DXB Smart Ap’, an app created by the United Arab Emirates country. This last one was discarded because it is from a foreign country.

Table 2.

Apps category with WHO Classification of Digital Health Interventions (II Cycle).

APPS		WHO classification of digital health interventions
1	Coronavírus - SUS	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	0	0	2	0
2	Monitora COVID-19	1	0	1	0	0	0	0	0	0	1	1	1	0	0	0	0	0	0	1	6	1
3	SOS COVID-19	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0
4	OpenWHO: Conhecimento para Emergências em Saúde	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0
5	Atende em Casa	0	0	0	0	0	1	0	0	0	0	1	0	0	0	0	0	0	0	0	2	0
6	WHO Info	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	0
7	Dados do Bem	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	1	2	1
8	SOS Corona - DDCOVID19	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	1
9	Colab	0	0	1	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	3	0
10	Cachoeirinha Contra o Coronavírus	0	0	0	0	0	1	0	1	1	0	0	0	0	0	0	0	0	0	0	3	0
11	A Salvus	0	0	0	1	0	0	0	0	1	0	0	1	0	0	0	1	0	1	0	5	1
12	Tô de Olho	1	0	0	0	0	0	1	1	1	0	0	0	0	0	0	0	0	0	1	5	1
13	Personal Health	0	0	0	0	1	0	0	0	0	0	1	0	0	0	0	0	1	0	0	3	1
14	Agenda Fácil - Prefeitura SP	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1
15	Corona Rodeio	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	0
16	IPES - Plataforma Eletrônica de Saúde	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1
17	COVID19 - DXB Smart Ap	0	1	0	0	0	0	0	0	1	0	1	0	0	0	1	0	0	0	0	4	2
18	SARS Disease	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	0
	TOTAL	2	2	2	1	2	3	1	3	9	2	4	2	1	1	1	1	2	1	3	43	10
Ref.	NAME	112	114	121	131	141	143	151	152	161	233	241	242	281	411	413	414	431	432	433	Categories Averiage	Relevant Categories Averiage
		Client									Health Provider				DATA SERVICES

WHO: World Health Organization; COVID-19: coronavirus disease 2019; SUS: System Usability Scale.

III Cycle: screening

Planning

Despite the 18 classified apps being related to the health field, not all of them were connected to the project goal – to provide a communication channel between professionals who are working inside hospitals and families of COVID-19 patients. Thus, it was necessary to screen them again based on what were the DHIs related to this specific goal. The head researchers decided to add another two researchers to the multidisciplinary team. These invited researchers are experts in Public Health and WHO guidelines and are already involved with the project as a principal investigator and responsible for Public Health search activities. The head researchers worked on the II Cycle table, eliminating the app’s names and highlighting the DHIs, creating another table (See online for Appendix 3).

Implementation

The group of six researchers presented above got together and discussed, finding an agreement with the project/work phase purpose and the reasons for carrying out this method in particular. After that, this team analyzed the II Cycle table. They discussed and selected which were the DHIs more related to the project goal. This discussion supported the screening process of those systems that will be analyzed in the next cycle.

Results

The team discussion revealed two different results: first, a list identifying the DHIs focussed on the project goal. The interventions which the team considered more relevant were:

1.0 clients:

1.1.4 Transmit the diagnostic result, or availability of the result, to the client(s).

1.4.1 Client access to own medical records.

4.0 Data Service:

4.1.1 Non-routine data collection and management.

4.1.3 Data synthesis and visualizations.

4.1.4 Automated data analysis to generate new information or predictions about future events.

4.3 Location mapping

4.3.3 Customer and household location map.

As a second result emerged and a list with the apps that presented those DHIs which are more associated with the project goal – to provide a communication channel between professionals who are working inside of hospitals and families of COVID-19 patients. From these 9 screened systems only 8 will be analyzed on IV Cycle. The ‘COVID19 – DBX Smart App’ was excluded because it was launched in Dubai, United Arab Emirates. The results are presented in below – Table 3.

Table 3.

Results of III Cycle.

Mobile Apps
Monitora coronavirus disease 2019 (COVID-19)
Dados do Bem
SOS Corona – DDCOVID19
A Salvus
Tô de Olho
Personal Health
Agenda Fácil – Prefeitura SP
IPES – Plataforma Eletrônica de Saúde
COVID19 – DXB Smart App

IV Cycle – analysis

Planning

The two head researchers decided to challenge the project’ ICT team. They thought it would be a proper strategy to involve the developers in this process. The reason why this choice emerged was that the developer team would have the possibility to find an initial perspective of the features. This strategy was though in its preparation for future work. The head researchers invited the senior ICT researcher to plan the whole analysis procedure with them. The senior ICT researcher also supervised two junior researchers to carry out the gathered analysis.

Conduction

The head researchers explained the Benchmarking process to the senior ICT researcher who lately trained the two junior ICT researchers. The results of III Cycle were delivered in a sheet to the two junior ICT researchers that individually worked. Each of the juniors downloaded and interacted with the apps. These researchers listed all found features in a new table – Supplemental material d. The identified features were discussed by the all ICT team. After that, the results of the discussion were consolidated through a new column in the same file. If an additional feature came out, the juniors shared the information, opened a new column without identifying the applications for the team. However, the juniors and the senior ICT researcher did not agree in some cases. When that happened, a new column was opened with a different feature interpretation. Each app received the value (1) if it had the feature and (0) if it did not have it. Each identified feature was related to an app screen, and they were registered in a digital file by each junior researcher. This digital file became an overview of each app because also includes screenshots of each system.

On a second moment, each junior played the role of a common user and answered a SUS survey^26,28,30 for each application. Brooke²⁶ described SUS as a ‘quick-and-dirty’ usability scale and, this method has become one of the most popular tools for user tests. It is suitable to measure efficiency, efficacy and satisfaction.³¹ The SUS analysis resulted in a score that measures the application’s usability degree. The measurement process uses the Likert Scale with five points. The team of researchers opted for this scale because it is used in the collection of opinions. The whole procedure was followed by the senior ICT researcher.

Finally, the senior ICT researcher presented the results to the head researchers who confirmed the accuracy of the process. The senior ICT researcher consolidated the junior individual results into three products: a single features list A single SUS evaluation and a single document with the app screen screenshots.

Results

At the first moment, the junior researchers were able to analyze only seven apps because ‘Agenda fácil – Prefeitura SP’ requested a specific address in São Paulo City. After that, each junior researcher freely interpreted what was seen on apps. This analysis resulted in 27 and 31 identified different features. The senior researchers thought the different amount was because each developer had their own interpretation. Some features are highlighted and they are related to the project goal (e.g., chatting to a healthcare professional, user location, etc.). For more detailed data see online for Appendix 4.

At the second moment, a usability test was carried out for every seven apps through SUS method.³¹ Each junior researcher individually answered the survey. The most important contribution from the SUS was to support the comparative analysis of how each app works. The results of the SUS score is presented in Table 4. For more detailed information, see all individual scores results on Table 5 available in Appendix.

Table 4.

SUS Score results (IV Cycle)

Mobile apps	SUS score
A Salvus	93.75
Personal health	93.75
Dados do bem	88.75
Tô de olho	86.25
Monitora COVID-19	78.75
SOS Corona – DDCOVID19	73.75
IPES – Plataforma Eletrônica de Saúde	36.25

COVID-19: coronavirus disease 2019; SUS: System Usability Scale.

According to Sauro & Lewis book³¹ where they observed data from 446 studies and over 5000 individual SUS responses, they concluded the overall mean score of the SUS is 68 points. Thus, Sauro²⁸ the SUS score above 68 would be considered above average and anything below 68 is below average. From the seven analysed apps (Table 4), only one did not achieve the average suggested by Sauro²⁸ which is ‘iPES – Plataforma Eletrônica de Sáude’. According to Bangor, Kortum e Miller classification³⁰ the most suitable adjective for ‘IPES’ experience is poor, because it is approximately 35.7. Following the same classification tool, the results are as follows: ‘Monitora COVID-19’ and ‘SOS Corona’ are good, because they reach approximately 71.4; ‘Dados do bem’ and ‘Tô de olho’ are excellent, with a result close to 85.5; ‘A Salvus’ and ‘Personal Health’ are the best imaginable because they achieve above 90.9.

Discussion and conclusion

COVID-19 pandemic challenged the world in answering global several health issues. In this context, Digital Health initiatives emerged worldwide trying to support government management, private and public health sector. One of the most urgent problem that challenged the Brazilian hospital management case was the communication gap between families of COVID-19 patients and this hospital. This problem was presented to our research team in order to find a digital health solution to get together the hospital and population in the same communication channel.

Before the development phase, our team decided to conduct several research stages. One of them included the identification in mobile apps marketplace of the available solutions in Brazil and the understanding of how they support the citizens. This was the reason why it was chosen Benchmarking as a research methodology. The typology followed was UX Benchmarking (NNGroup) because the team focussed on the user experience through usability tests in the final phase – IV Cycle. UX Benchmarking is a suitable method to discover UX gaps in ICT.

One of the primary purposes of our research team was to establish a strict research approach that should be applied in all our work phases. This purpose was set up to avoid bias in the research procedure because the project is complex and involves a multidisciplinary team. Besides that, the team chose to undertake a systematic mapping of the apps through four cycles: I Cycle – Identification; II Cycle – Classification; III Cycle – Screening. The last Cycle, the IV, presents features analysis and results of usability carried out through the SUS method.

As a multidisciplinary team, our Public Health experts defined that WHO strategies were an essential tool. One of the concerns of WHO⁹ is that digital systems have been unsystematic, slow, prone to error, which has resulted in poor transparency and traceability. WHO decided that it was needed to overcome this issue, and one of the initiatives for that was to establish DHIs. These guidelines inspired all our research plans.

DHIs are recent guidelines launched in January of 2020, that aims to promote a holistic understanding of how to implement and manage digital solutions, health policies, and recommendations. The DHIs are a pioneering initiative that is in the first version, but the work presented in this article guides us to a conclusion: these DHIs should be adapted to local policies, procedures, and digital tools. After conducting II Cycle (Classification), the research team thought that these WHO guidelines were widely open. The terminology causes some doubts about what a specific DIHs meant and what it dealt with. The II Cycle was one of the most discussed by the researchers. Thus, the DIHs should be more narrow to support an analysis process closer to the current health context. After we undertook this method, we found a recent WHO article⁹ suggesting to optimize this tool. In this article, WHO request the research community to improve future versions of DHIs focus on local health policies and challenges.

Our method can be optimized in the future. Our team concluded that one of the cycle’s tasks could be carried out earlier, on the starting point. In this case, the task of III cycle that involved setting up the DHIs was more related to the project goal. In future projects that involve working with UX Benchmarking method, the experts in Public Health and WHO Guidelines will be invited to contribute at the beginning of the method. We believe that it will be more agile.

The final cycle was the most relevant for the research team. This part was a result of work and discussion that was started by ICT team and supported by firstly by head researchers and after by the entire multidisciplinary team. This initial step was important to develop a better final system because it helps to understand what already exists and to develop a work basis. Besides that, it provided valuable insights into the state-of-the-art of Brazilian digital health solutions for the COVID-19 pandemic. This discussion and analysis gave us an overview of what are the available solutions and how they operate in the health context, what helps us to find a market gap through others’experience.

There was an issue that was different in the amount of identified features by junior researchers. The senior researchers thought this normal because the junior researchers did not have access to development codes. That is, this analysis method needs to be flexible. Another valuable insight that was raised up was the understanding of how the industry was interpreting the user’s needs. This allied with methods focussed on user context can provide ideas on where we should work, as well as a holistic view to base the future product development. The SUS method showed us to be an effective method. Firstly, it is clear enough to user test with non-designers. Secondly, it measures the overall UX, which allows us to compare different solutions.

Future works

The research team invites other researchers to carry out this methodologic approach for digital health product development. It is important to understand the state-of-art of digital solutions before developing products. This allows us to identify the gaps in user needs and line up the functional requirements for them. However, it needs to highlight that a systematic approach is welcome to base the methodological choices and steps. One of the ways to support these choices in the health sector is through WHO guidelines. The DHIs can be used with some adaptation to the context.