Skip to main content
NCBI home page
PLOS One logoLink to PLOS One
. 2020 May 29;15(5):e0233269. doi: 10.1371/journal.pone.0233269

Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study

Sarita Rodríguez 1,*,#, Ana María Sanz 1,#, Gonzalo Llano 2,‡, Andrés Navarro 2,‡, Luis Gabriel Parra-Lara 1,3,#, Amy R Krystosik 4,‡, Fernando Rosso 1,3,5,#
Editor: Abdallah M Samy6
PMCID: PMC7259752  PMID: 32469894

Abstract

Background

Vector-borne diseases are a public health problem in Colombia, where dengue virus infection is hyperendemic. The introduction of other arboviruses, such as chikungunya and Zika in the last three years, has aggravated the situation. Mobile health (mHealth) offers new strategies for strengthening health care and surveillance systems promoting the collection, delivery, and access of health information to professionals, researchers, and patients. Assessing mobile application performance has been a challenge in low- and middle-income countries due to the difficulty of implementing these technologies in different clinical settings. In this study, we evaluate the usability and acceptability of a mobile application, FeverDX, as a support tool in the management of patients with febrile syndrome and suspected arboviruses infection by general practitioners from Colombia.

Methods

A pilot implementation study was conducted to evaluate the usability and acceptability of FeverDX using the modified version of the Mobile Application Rating Scale (uMARS). The evaluation form included 25 questions regarding quantity and quality of information, engagement, functionality, aesthetics, impact, and acceptability by healthcare workers. Each item uses a 5-point scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent). A global score was obtained for the evaluation form test by determining the median scores of each subsection. A descriptive statistical analysis of the data obtained was performed.

Results

Between December 2016 and January 2017, a total of 20 general practitioners from the Emergency room and hospitalization areas evaluated FeverDX. Less than half (9/20) of the evaluators had a comprehensive knowledge of the Colombian Ministry of Health’s guidelines for the diagnosis and management of arboviruses, and evaluators partially (4/9) or completely (5/9) agreed that the content of the application follows the management guidelines. On uMARS scale, FeverDX excelled regarding impact (median 5; IQR = 5–5), functionality (median 5; IQR = 4.8–5), and information and scientific basis (median 4; IQR = 4–4). FeverDX scored well regarding user feedback (median 4; IQR = 4–4.5), design and aesthetics (median 4; IQR = 4–4.3), and subjective assessment of quality (median 4.5; IQR = 4.3–4.8).

Conclusions

FeverDX, a mobile application, is a novel mHealth strategy to strengthen care processes and facilitate the detection and reporting of notifiable surveillance diseases. It could improve adherence to clinical practice guidelines for the management and prevention of prevalent diseases as arboviruses in healthcare settings.

Although this pilot study used a small sample size, FeverDx performed adequately in a simulated emergency consultation. Further implementation studies are needed to increase the reliability of mHealth technologies in different scenarios.

Introduction

In Latin America and the Caribbean, arboviruses are a major public health problem [1]. Before the arrival and subsequent spread of chikungunya and Zika viruses, Aedes spp. transmitted dengue virus with a record of 2,38 million cases in the Americas in 2016 [2]. During 2014, Colombia experienced the importation and rapid transmission of chikungunya, with approximately 96,687 cases reported. In September 2015, Colombia reported its first case of Zika, reaching a total of 11,712 the same year [3].

The limited availability of diagnostic tests in endemic areas in developing countries, represents a challenge for differential diagnosis, especially in the case of co-infection [4–6] because arboviruses infections share non-specific clinical manifestations [7]. This situation encourages the development and implementation of innovative initiatives in different scenarios, as health services and public health surveillance entities. An example is mobile health (mHealth), referring to the practice of medical and public health supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants, and other wireless devices [8]. For more than a decade, the World Health Organization (WHO) has promoted the development and application of mHealth as an essential resource in the provision of health services [9]. According to a survey conducted by WHO, only 12% (14/115) of the WHO Member States conducted formal evaluations of mHealth-related initiatives [8].

Supply and demand for mobile applications development in the healthcare area have been recently increasing [10]. Although mHealth performance evaluations are growing [11–13], regarding the majority of commercially mobile health applications in our research field, companies and researchers have not yet incorporated an extensive performance evaluation, including usability and app quality. Furthermore, additional tailored and rigorous evaluation studies on the mHealth clinical impact and reliability are still required [14,15].

FeverDx is a mHealth tool designed by a group of physicians for healthcare workers (HCW) in a local context where digital transformation is limited, making this tool tailored to the specific needs of the end-user in terms of time and ease of use, access to national health guidelines content, and is a critical step of the reporting process of mandatory notifiable diseases. Moreover, the mobile app provides rapid access without an internet connection to locally stored clinical data regarding the most frequent fever syndromes clinical guidelines and supports general practitioners (GPs) in the approach and management of patients with a fever syndrome and suspected arboviral infection.

This study evaluated the usability and acceptability of FeverDx mobile application, a mobile application to guide general practitioners (GPs) in the approach and management of patients with a fever syndrome and suspected arbovirus infections. GPs from different clinical areas at Fundación Valle del Lili (FVL) performed FeverDx.

Materials and methods

We conducted a cross-sectional, pilot implementation study of FeverDx in Cali, Colombia.

Setting

The study was conducted at Fundación Valle del Lili (FVL) University Hospital between December 2016 and January 2017. FVL is a non-profit university hospital that works as a referral health center for the southwestern region of Colombia, and it is associated with Universidad Icesi. In 2016, a total of 17,321 cases of dengue, 1,204 of chikungunya, and 14,638 of Zika were reported in the city by the Local Health Authority [16].

The study protocol was approved by the Comité de Ética en Investigación Biomédica from FVL (IRB/EC No. 584–201, followed the ethical principles for medical research outlined by the Declaration of Helsinki [17] and the declaration 8430 of 1993 from Colombia’s Ministry of Health [18].

Mobile application

FeverDx is a mobile application (app) developed by the FVL in association with Universidad Icesi. The app was created following the Methodology for the Development for Mobile Applications (MDAM) [19]. This guideline recommends five phases intending to guide the process of app design and implementation: analysis, design, development, performance testing, and delivery. The minimal requirements for the use of the app are a tablet or smartphone device supported by either Android and iOS operating systems with at least intermittent access to Wi-Fi or mobile network connectivity.

This app runs based on two modules: the clinical guideline and the evaluation modules. The guideline module allows GPs to obtain a rapid-offline reference to the CPGs for acute febrile syndromes such as Zika, dengue, chikungunya, and other common infections in tropical areas (leptospirosis). Each guideline includes general topics concerning clinical features, diagnostic, treatment, and prevention (Fig 1).

Fig 1. Screenshots of the FeverDx clinical guideline module.

Fig 1

Open in a new tab

A. Main Menu; B. Clinical Guidelines; C. Guideline Display.

The patient evaluation module allows the GP to create a local record (mobile phone database) of the patient data, which can be consulted, modified, or deleted at any time. Fig 2 shows the algorithm integrated into the application to support the approach and management of patients with a febrile syndrome. This algorithm highlights the essential points of patient care, such as the evaluation of general symptoms, vital signs, and the alarm signs, severity signs, which include all dengue, Zika, and chikungunya warning signs.

Fig 2. Flowchart of the patient evaluation module process.

Fig 2

Open in a new tab

Patient information obtained from medical history, physical examination, and laboratory tests are recorded using a graphical interface (checklists and alphanumeric fields) (Fig 3).

Fig 3. Patient clinical information collection views in FeverDx application.

Fig 3

Open in a new tab

A. Risk Conditions; B. General Evaluation; C. Clinical Evaluation; D. Physical Exam.

The mobile app shows a variety of messages from an internal feedback “library” which is based on the Colombian Ministry of Health´s clinical practice guidelines (CPG) for the diagnosis and management of arboviruses (dengue, chikungunya, and Zika) and additional common infections in tropical countries [20–23]. An infectious disease specialist reviewed the included recommendations.

Obtained data from registration, evaluation, and reporting is stored on a local server, where it is available to be consulted by national epidemiological surveillance entities.

The FeverDx app is compliant with the Policy of treatment and protection of personal data (apps.co) from the Colombian Ministry of Information and Communications Technologies [24].

Study procedures

The pilot study was conducted in a controlled setting where GPs from the emergency room and hospitalization areas, that had no prior experience with the app, were recruited through an individual invitation. The usability evaluation was carried out in a single session according to the following stages:

  1. Delivery of materials (an Android or iOS device, medical records, evaluation forms) and instructions about the application use and the evaluation process.

  2. Simulated emergency consultation (Interaction with the application).

  3. Individually and confidentially usability evaluation by GPs.

The convenience sampling method was implemented to select a sample of 100 different clinical records of patients who consulted FVL for acute febrile illness from the Institutional Epidemiological Surveillance Committee and Statistics Department Databases. Demographic, clinical, and laboratory data were obtained from the FVL medical charts. Five clinical records were randomly assigned to each of the twenty GPs who used the app to perform a simulated emergency consultation interacting with the patient evaluation module.

During the clinician-application interaction, feedback messages related to patient risk- group, signs of alarm, severity, and recommended management, appear intended to guide the GP in the approach and management of the patient.

The simulated emergency consultation that included the entry of all patient-related data, lasted 6 minutes on average. Afterward, there was no time limit to experience FeverDx freely. The evaluation of usability and acceptability was performed anonymously, individually, and secretly by each GP. The assessment included the perception of usefulness and user experience satisfaction.

Evaluation tool and analysis

Implementation outcomes were extracted into a template using a Microsoft Excel® spreadsheet. We developed the evaluation form based on four different evaluation systems for mHealth applications: 1) mobile health (mHealth) evidence reporting and assessment (mERA) checklist [25]; 2) iSYScore index [26]; 3) Mobile App Rating Scale (MARS) [27]; 4) User Version of the Mobile Application Rating Scale (uMARS) [28]; and 5) additional evaluation methods [29] and implementation outcomes (acceptability [30] and usability [31]).

The evaluation form comprises 25 questions measuring quantity and quality of information (Section A), engagement (Section B), functionality (Section C), aesthetics (Section D), the impact of patient management from the practitioner perspective (Section E) and acceptability by HCW (Section F) (S1 File).

Knowledge evaluation regarding the CPG for vector-borne disease management was included in the first two questions of the evaluation form. Each item uses a 5-point scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent) (S1 File). A global score was obtained for the evaluation form test by determining the median scores of each subsection.

A descriptive analysis was performed using STATA® statistical software (Stata Corp, 2011, Stata 12 Base Reference Manual, College Station, TX, USA). Continuous data were summarized as median and interquartile ranges, and categorical variables presented as frequencies and proportions.

Results

Between December 2016 and January 2017, a total of twenty GPs tested and evaluated FeverDx mobile application.

CPG knowledge

All data regarding CPG knowledge evaluation was gathered from GPs with no prior use of the FeverDx app. The proportion of GPs with comprehensive knowledge of the Colombian’s CPG for diagnosis and management of arboviruses (dengue, chikungunya, and Zika) was less than half of GPs evaluated (9/20), and less than one-third of them applied them in their practice (6/20). Of those GPs who reported having comprehensive knowledge of CPG, more than half (5/9) wholly agreed and, 4/9 partially agreed that the application information matched with CPG information.

Application evaluation

Table 1 presents an overview of the information quality, engagement, functionality, aesthetics, quality, and impact categories evaluation scores. The median of the global evaluation score was 4.0 (out of 5.0). Impact and functionality obtained the highest score with a median of 5 (IQR = 5–5) and 5 (IQR = 4.8–5), respectively. Among GPs, 19 out of 20 partially/wholly agreed that the application could increase their knowledge and understanding about dengue, chikungunya, and Zika infection management.

Table 1. Category and item scores evaluated in the FeverDx application (n = 20).

Category Score Median (IQR)
Information/scientific and clinical foundations 4 (4–4)
    Implementation has specific, measurable and achievable objectives 4 (3–4)
    Quality of information 4 (4–5)
    Quantity of information 4 (4–5)
    Visual information 4 (4–4.3)
    Credibility 4 (3–4)
    References 4 (3.5–5)
    The application can increase your knowledge/ comprehension about vector-borne diseases approach and management 5 (4.7–5)
    I had to learn many things before I could use the application 2 (1–2)
User feedback 4 (4–4.5)
    Interest 5 (4–5)
    Interactivity 4 (4–5)
    Application content is appropriate for the end user 4 (4–5)
Functionality 5 (4.8–5)
    Throwput 4 (4–5)
    Ease of use 5 (4–5)
    Navigation 5 (4–5)
    Gestural Design (Interactions: Taps/Swipes/Pinches/Scroll) 5 (4–5)
Aesthetic—graphic design 4 (4–4.3)
    Design 4 (4–5)
    Graphs 4 (4–5)
    Visual appeal 4 (4–5)
Impact 5 (5–5)
    This application can increase motivation/intention to improve adherence to handling guidelines of dengue, Zika, and Chikungunya 5 (4–5)
    Attitudes towards the approach and management of patients with dengue, Zika, and chikungunya are likely to change. 5 (4–5)
    This application has the potential to be efficiently introduced into multiple emergency departments of different levels of care in Colombia 5 (4.8–5)
Acceptability 4.5 (4.3–4.8)
    I would recommend this application 5 (4–5)
    Would frequently use this application 4 (4–5)
Open in a new tab

IQR: Interquartile Range

The highest score is five and the lowest score is one, except in section 1, item 8, where the highest score equaled one, and the lowest score equaled 5.

Regarding the user feedback, GPs scored a median of 4 (IQR = 4–4.5), and “interest” obtained the high scored of this category 5 (IQR = 4–5). Almost one-third of participants (6/20) reported the mobile application content as suitable for physicians (without content problems) and more than half (13/20), that the application information (content) was correctly oriented with minor issues. Only one of the participants considered the content as acceptable and it was not geared toward physicians.

Concerning impact, the intention to change domain was assessed with a median score of 5 (IQR = 4–5), and the majority (18/20) of GPs partially/wholly agreed that the app will probably change attitudes towards clinical management of dengue, chikungunya, and Zika. Moreover, all the GPs would recommend the app, and most of them (17/20) rated it with 4 out of 5 according to the overall star rating (Fig 4).

Fig 4. Overall star rating of mobile application FeverDx.

Fig 4

Open in a new tab

Discussion

To our knowledge, this is the first study evaluating the usability and acceptability of a mobile application for the clinical approach to acute febrile syndrome management in Colombia. In general, there is a lack of evidence regarding the assessment and performance of mobile apps in healthcare facilities. For this reason, more initiatives to evaluate mHealth developments are needed in different contexts to know the real impact of these technologies, and to generate evidence about its performance in the field [15].

In this study, we described an alternative approach to evaluate mHealth through validated scales such as uMARS, which had demonstrated a reliable degree of app quality evaluation and the ability to obtain valuable information from users about mobile apps [28]. Our findings showed that FeverDx evaluation obtained a high overall score as well as in most of its subscales. Hence, the use of FeverDx in regions where arboviruses are endemic or hyperendemic potentially could help GPs and stakeholders in the management of acute febrile syndromes such as arboviruses infections. Although this is a pilot study, our findings are a starting point for the development of new implementation studies that allow the evaluation of application performance in different contexts and scenarios.

One of the challenges in low and middle-income countries is the surveillance of vector-borne diseases and their epidemiology. The epidemiological surveillance process includes the notification, collection, and analysis of data, which allows for the generation of timely, valid and reliable information for stakeholders and policymakers, and thus to guide public health policy and create prevention and control strategies [32]. However, this process is often time-consuming. We believe that FeverDx could help in the solution of this problem and facilitate this process through innovative development that is easy to reach and manage from an app that can be used in devices such as smartphones.

Mobile applications designed for epidemiological surveillance of arboviruses are limited in their scope. A study conducted in 2018, which reviewed the available Google Play and App Store applications, showed that 26 applications relevant to surveillance are available. This review included 7 surveillance applications related explicitly to dengue and Zika, aimed at case reporting and outbreaks monitoring through the use of interactive maps. Some of these applications were limited by location, operating system (Android only), and although most of them were developed for global use, most of them were in the English language, which could narrow their applicability in different contexts [33,34]. Additional mHealth developments with a different approach are available such as surveillance (DengueWatch, Dengue map, ZikaTracke, Kidenga App, Break Zika, Predict and Beat dengue) and general information (Dengue–Manejo clínico, Arbo App, Dengue App, Dengue síntomas tratamiento y prevención, Dengue Treatment, WHO Zika App), however evidence regarding implementation in clinical scenarios is scarce [34,35].

There are relevant conditions to be considered for the mobile health applications development and adoption: the local demographic, geographic, connectivity, culture, and socio-sanitary aspects; and the particular needs of the potential end-users, general public, healthcare workers or surveillance authorities [34,36]. Among the commercially available mHealth developments, a small proportion combines different functionalities such as vector-borne disease management (evaluation, classification, and treatment) or rapid offline clinical information review.

In this sense, FeverDx is innovative in the field of arboviruses, it is in the local language, and has been evaluated for usability and acceptability. It offers a solution to local and national surveillance systems, based on these identified needs, to fulfill an early and timely case report. Furthermore, it could potentially support the decision-making process in real-world clinical environments.

Another relevant finding was that a not insignificant proportion (25%) of the GPs had not read the Colombian CPGs for the management of vector-borne diseases. Previous studies in this matter have shown limitations in the adoption of CPGs. For example, a study conducted in Zimbabwe in 2011 that evaluated the attitudes of GPs towards the use of CPGs showed that 57.9% of GPs felt that guidelines would not improve their diagnostic ability, even though that 65.9% were prepared to use guidelines in their practice [37]. Furthermore, qualitative studies conducted to assess GPs' experiences when applying multiple guidelines found that GPs’ responses clustered around two major topics: 1) Complications for the GPs of applying multiple guidelines; and, 2) Complications for their patients when GPs apply multiple guidelines [38]. Thus, further strategies for CPGs implementation and real-world use are needed. For this reason, we propose friendly mobile technologies as a feasible and reliable strategy for promoting the adoption of CPGs.

Finally, mobile technology offers unique opportunities to reach and engage HCW to enhance health literacy and public health authorities for real-time disease surveillance, and that the real-time case reporting feature could improve the current manual registration and report process [33].

FeverDx is a user-centered app intended to drive clinical information to GPs and facilitate epidemiological surveillance. Our findings could promote the adoption and engagement of this app in a clinical setting; thus, improving the care of patients with febrile syndromes in health centers in tropical areas resulting, eventually, in better clinical outcomes.

Limitations

Our study is not without limitations and results should be interpreted in the context of the study design. First, the study included twenty GPs from the emergency department and hospitalization areas of one center highly specialized in the treatment of severely ill patients, which limits the generalizability of our study and the applicability of our results in other environments. Furthermore, the GPs included in the study and the mobile application developers come from the same institution, which may have introduced a courtesy bias. To face this limitation, we assured an anonymous and independent evaluation process for the participants. Second, the fact that we tested the app in a tertiary care center makes our study prone to selection bias. Third, the implementation of FeverDx is currently restricted to Colombia because its design was mainly based on the Colombian Ministry of Health´s mandatory notification sheets and CPG. Application content was strengthened with the international recommendations given by the WHO and with the review by an infectious diseases specialist with experience in arboviral research to mitigate this issue.

Despite our limitations, we consider that the application, in general, has acceptable performance, and our study advances the knowledge of mHealth technology for the diagnosis of febrile syndromes (arboviruses and other common infections) in settings where innovative initiatives are needed to improve patient outcomes. Finally, further research with a larger sample size and with the participation of other healthcare centers need to be conducted to support our initial findings.

Supporting information

S1 File. Evaluation form (English version).

(PDF)

Click here for additional data file. (93KB, pdf)
S2 File. Evaluation form (Spanish version).

(PDF)

Click here for additional data file. (117.1KB, pdf)
S1 Dataset. Raw evaluation results.

(PDF)

Click here for additional data file. (260.3KB, pdf)

Acknowledgments

We thank the team of developers at Universidad Icesi and the editorial group at Fundación Valle del Lili for their support in this project.

Data Availability

All relevant data are within the manuscript and its Supporting Information files.

Funding Statement

This study was funded by Fundación Valle del Lili and Universidad Icesi. The funders don't play role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

  • 1.Marcondes CB, Contigiani M, Gleiser RM, Reisen W. Emergent and reemergent arboviruses in South America and the Caribbean: Why so many and why now? J. Med. Entomol. 2017;54:509–32. 10.1093/jme/tjw209 [DOI] [PubMed] [Google Scholar]
  • 2.World Health Organization. Dengue and severe dengue. Newsroom 2019. [Google Scholar]
  • 3.Alvis-Guzmán N, Zakzuk-Sierra J, Vargas-Moranth R, Alcocer-Olaciregui A, Parra-Padilla D. Dengue, Chikunguña y Zika en Colombia 2015–2016. Rev. MVZ Córdoba 2017;22:5994 10.21897/rmvz.1069 [DOI] [Google Scholar]
  • 4.Cardoso CW, Paploski IAD, Kikuti M, Rodrigues MS, Silva MMO, Campos GS, et al. Outbreak of Exanthematous Illness associated with Zika, Chikungunya, and Dengue viruses, Salvador, Brazil. Emerg. Infect. Dis. 2015;21:2274–6. 10.3201/eid2112.151167 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Parreira R, Centeno-Lima S, Lopes a. , Portugal-Calisto D, Constantino a., Nina J. Dengue virus serotype 4 and Chikungunya virus coinfection in a traveller returning from Luanda, Angola, January 2014. Eurosurveillance 2014;19:6–9. 10.2807/1560-7917.ES2014.19.10.20730 [DOI] [PubMed] [Google Scholar]
  • 6.Furuya-Kanamori L, Liang S, Milinovich G, Soares Magalhaes RJ, Clements AC a, Hu W, et al. Co-distribution and co-infection of chikungunya and dengue viruses. BMC Infect. Dis. 2016;16:84 10.1186/s12879-016-1417-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Mattar S, Tique V, Miranda J, Montes E, Garzon D. Undifferentiated tropical febrile illness in Cordoba, Colombia: Not everything is dengue. J. Infect. Public Health 2017:6–11. 10.1016/j.jiph.2016.09.014 [DOI] [PubMed] [Google Scholar]
  • 8.World Health Organization. mHealth: New horizons for health through mobile technologies. Observatory 2011;3:66–71. 10.4258/hir.2012.18.3.231 [DOI] [Google Scholar]
  • 9.World Health Organization. 58a Asamblea Mundial de la Salud. Resoluciones y Decisiones. Cibersalud. 58a Asam. Mund. La Salud 2005:128–30. [Google Scholar]
  • 10.Jahns R-G, Houck P. Mobile Health Market Report 2013–2017. Research2Guidance 2013:116. [Google Scholar]
  • 11.Boyce SP, Nyangara F, Kamunyori J. A mixed-methods quasi-experimental evaluation of a mobile health application and quality of care in the integrated community case management program in Malawi. J. Glob. Health 2019;9:010811 10.7189/jogh.09.010811 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Haque F, Ball RL, Khatun S, Ahmed M, Kache S, Chisti MJ, et al. Evaluation of a Smartphone Decision-Support Tool for Diarrheal Disease Management in a Resource-Limited Setting. PLoS Negl. Trop. Dis. 2017;11:1–19. 10.1371/journal.pntd.0005290 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Hamrioui S, Torre-díez I De, López-coronado M, Herreros J. Implementation and Performance Evaluation of a Mobile Health Application for Auto-controlling of Cardiac Diseases. Int. J. Clin. Med. Informatics ISSN 2018;1:13–7. [Google Scholar]
  • 14.Fougerouse P-A, Yasini M, Marchand G, Aalami OO. A Cross-Sectional Study of Prominent US Mobile Health Applications: Evaluating the Current Landscape. AMIA… Annu. Symp. Proceedings. AMIA Symp. 2018;2017:715–23. [PMC free article] [PubMed] [Google Scholar]
  • 15.Sim I. Mobile devices and health. N. Engl. J. Med. 2019;381:956–68. 10.1056/NEJMra1806949 [DOI] [PubMed] [Google Scholar]
  • 16.Rojas Palacios JH. Boletín epidemiológico No. 10. Cali: 2016. [Google Scholar]
  • 17.World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2013;310:2191–4. 10.1001/jama.2013.281053 [DOI] [PubMed] [Google Scholar]
  • 18.Ministerio de Salud. Resolución 8430 (octubre 4): Por la cual se establecen las normas científicas, técnicas y administrativas para la investigación en salud 1993:1–19. [Google Scholar]
  • 19.Gasca Mantilla MC, Camargo Ariza LL, Medina Delgado B. Methodology for mobile application development. Tecnura 2014;18:20–35. [Google Scholar]
  • 20.Ministerio de Salud y Protección Social. Guía para la atención clinica integral del paciente con dengue Bogotá D.C.: Ministerio de Salud y Protección Social; 2010. [Google Scholar]
  • 21.Ministerio de Salud y Protección Social. Lineamientos para el manejo clínico de los pacientes con el virus chikungunya. Bogotá D.C.: 2013.
  • 22.Ministerio de Salud y Protección Social. Lineamientos para la atención clínica integral de pacientes con zika en Colombia. Bogotá D.C.: 2016.
  • 23.Instituto Nacional de Salud (INS). Protocolo de vigilancia en salud pública: leptospirosis. Bogotá D.C.: Ministerio de Salud y Protección Social; 2016. [Google Scholar]
  • 24.Ministerio de Tecnologías de la Información y las Comunicaciones. Política de tratamiento y protección de datos personales. Apps.Co available at.
  • 25.Agarwal S, LeFevre AE, Lee J, L’Engle K, Mehl G, Sinha C, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174 10.1136/bmj.i1174 [DOI] [PubMed] [Google Scholar]
  • 26.Grau I, Kostov B, Gallego J a., Grajales F, Fern??ndez-Luque L, Sis??-Almirall A. Método de valoración de aplicaciones móviles de salud en español: el índice iSYScore. Semergen 2016;42:575–83. 10.1016/j.semerg.2015.12.001 [DOI] [PubMed] [Google Scholar]
  • 27.Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR MHealth UHealth 2015;3:e27 10.2196/mhealth.3422 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR MHealth UHealth 2016;4:e72–e72. 10.2196/mhealth.5849 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Grundy QH, Wang Z, Bero L a. Challenges in Assessing Mobile Health App Quality: A Systematic Review of Prevalent and Innovative Methods. Am. J. Prev. Med. 2016;51:1051–9. 10.1016/j.amepre.2016.07.009 [DOI] [PubMed] [Google Scholar]
  • 30.Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda 2011;38:65–76. 10.1007/s10488-010-0319-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Madan A, Dubey SK. Usability Evaluation Methods: a Literature Review. Int. J. Eng. Sci. Technol. 2012;4:590–9. [Google Scholar]
  • 32.García Perez C, Aguilar PA. Vigilancia epidemiológica en salud. Arch. Médico Camagüey 2013;17:784–805. [Google Scholar]
  • 33.Mohanty B, Chughtai A, Rabhi F. Use of Mobile Apps for epidemic surveillance and response–availability and gaps. Glob. Biosecurity 2019;1:37–49. 10.31646/gbio.39 [DOI] [Google Scholar]
  • 34.Early J, Gonzalez C, Gordon-Dseagu V, Robles-Calderon L. Use of Mobile Health (mHealth) Technologies and Interventions Among Community Health Workers Globally: A Scoping Review. Health Promot. Pract. 2019;20:805–17. 10.1177/1524839919855391 [DOI] [PubMed] [Google Scholar]
  • 35.Agarwal S, Perry HB, Long LA, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: Systematic review. Trop. Med. Int. Heal. 2015;20:1003–14. 10.1111/tmi.12525 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Barrera Valencia John Jairo. Análisis de la salud móvil en Colombia 2016–2017. Universidad CES, 2018. [Google Scholar]
  • 37.Bhagat K, Nyazema N. General practitioners and clinical guidelines. East Afr. Med. J. 2001;78:30–4. 10.4314/eamj.v78i1.9109 [DOI] [PubMed] [Google Scholar]
  • 38.Austad B, Hetlevik I, Mjølstad BP, Helvik AS. Applying clinical guidelines in general practice: A qualitative study of potential complications. BMC Fam. Pract. 2016;17:1–8. 10.1186/s12875-016-0490-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

Decision Letter 0

Abdallah M Samy

3 Jan 2020

PONE-D-19-31941

Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study

PLOS ONE

Dear Dr Rodriguez,

Thank you very much for submitting your manuscript "Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study" (#PONE-D-19-31941) for review by PLOS ONE. As with all papers submitted to the journal, your manuscript was fully evaluated by academic editor (myself) and by independent peer reviewers. The reviewers appreciated the attention to an important health topic, but they raised substantial concerns about the paper that must be addressed before this manuscript can be accurately assessed for meeting the PLOS ONE criteria. Therefore, if you feel these issues can be adequately addressed, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. We can’t, of course, promise publication at that time.

We would appreciate receiving your revised manuscript by Feb 17 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Abdallah M. Samy, PhD

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements:

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors present the results of a pilot acceptability and usability mobile application to help GPs with diagnosing arbovirus and leptospirosis (although it is ignored in most of the text which is centered on arboviruses), and allow them to report cases directly for surveillance purposes. The paper is well written, and I found the mhealth application interesting, useful and well-though. However, the study presented here is very limited, biased and, although it is a good first step, the study needs to be expanded in order to warrant a manuscript rather than a short communication. I would suggest expanding the study to other settings and the sample size.

In the introduction, the authors need to introduce why FeverDX is particularly suitable to improve the adherence to CPGs and surveillance, who is it designed for and how are mhealth tools particularly suitable (something novel is not necessarily better). Some of this information is already in the discussion, but I think it would help construct the argument better if placed in the introduction (for example, lines 208-215 and lines 216-219). Moreover, the whole argument is centered around that performance evaluations have not been performed for mhealth interventions (lines 69-70), which is not entirely true. There is a whole body of literature that does exactly that, and although not all mhealth tools are validated, there are performance evaluations. The reference provided for that claim, does not support it exactly.

In Methods, I would suggest making the first paragraph describing the mobile app clearer. As I continued reading, I understood the app but it is very confusing at the beginning. I would make Fig 2, Fig 1 instead. However, I am most worried about the usability evaluation. Did it happen during the training session, after the simulated emergency consultation? Or some time after? Did the authors measure actual engagement after the training session? I worry that the high scores are influenced by courtesy bias, apart from the selection bias already mentioned by the authors. Besides, the sample size is too small to even speculate about the external validity of the results.

In the results:

How was CPG knowledge measure?

In lines 164-165 the authors mention that the majority of the GPs agreed that the app matched the CGP information, but in the previous sentence that almost half actually read them… so that could be also indicating courtesy bias?

In the discussion, the first sentence speaks about the novelty of the study but uses a very restrictive situation so, yes it is novel for that…

Also, I do agree that more usability and feasibility studies are needed for mhealth but is this study enough to state this? Is a one-time (not longitudinal) and with a very small sample. This limits the interpretation of the results and I worry about speculations as the one presented in lines 205-207.

In lines 219, the authors may have omitted other apps. Of the top of my head I can think of Kidenga, for example. What are the other 26 apps relevant to surveillance referring to?

Lines 236-237 seem to talk about the general public but this app is not intended for that.

Minor:

Lines 145-147: it is not very clear how the evaluation form was given in this sentence and Table 1 is presented in Results. I would suggest including the evaluation form as supp. material and improve this sentence.

Lines 150-151: I assume that the authors referred to median and IQR as is stated in the following paragraph.

Line 183: “as (un)acceptable”?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 May 29;15(5):e0233269. doi: 10.1371/journal.pone.0233269.r002

Author response to Decision Letter 0

18 Feb 2020

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

We appreciate this observation; thus, to compliant the PLOS Data Policy, we will make raw data concerning the results of the FeverDx usability and acceptability evaluation that supports our work fully available as part of the supporting information.

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors present the results of a pilot acceptability and usability mobile application to help GPs with diagnosing arbovirus and leptospirosis (although it is ignored in most of the text which is centered on arboviruses), and allow them to report cases directly for surveillance purposes. The paper is well written, and I found the mhealth application interesting, useful and well-though. However, the study presented here is very limited, biased and, although it is a good first step, the study needs to be expanded in order to warrant a manuscript rather than a short communication. I would suggest expanding the study to other settings and the sample size.

We thank you for this comment. We agree that in some sentences in the abstract and the introduction, it seemed that we focus on arboviruses and leptospirosis. However, we included leptospirosis and other infections in our environment due to the frequency of presentation and that these infections are part of the differential diagnosis of dengue, Zika, and chikungunya. Consequently, we replaced "Leptospirosis" from Lines 52, 185, 457 for "other common infections" in the file “Manuscript with track changes”.

We share your concerns about the extend, bias, and other limitations of our pilot study. We agreed that our study could be perceived as a first approach, especially in a sanitary context, and that lacks evidence-based mHealth interventions. We believe that these findings warrant further research as the second phase of this study that includes a larger sample size and different participating centers.

In the introduction, the authors need to introduce why FeverDX is particularly suitable to improve the adherence to CPGs and surveillance, who is it designed for and how are mhealth tools particularly suitable (something novel is not necessarily better). Some of this information is already in the discussion, but I think it would help construct the argument better if placed in the introduction (for example, lines 208-215 and lines 216-219).

We appreciate this suggestion. We have edited the introduction section and included the following text in lines 123-131 of the file “Manuscript with track changes”:

“FeverDx is a mHealth tool designed by a group of physicians for healthcare professionals in a local context where digital transformation is limited, making this tool tailored to the specific needs of the end-user in terms of time and ease of use, access to content and particularly in a critical step of the reporting process of mandatory notifiable diseases. This particular feature could potentially improve the currently manual registration and report process, mostly using physical resources (paper forms, physical files)[10]. Moreover, the mobile app provides rapid access without an internet connection to locally stored clinical information regarding the main topics of frequent fever syndromes clinical guidelines”.

Moreover, the whole argument is centered around that performance evaluations have not been performed for mhealth interventions (lines 69-70), which is not entirely true. There is a whole body of literature that does exactly that, and although not all mhealth tools are validated, there are performance evaluations. The reference provided for that claim, does not support it exactly.

We thank the reviewer for pointing out that we needed reference support about performance evaluations for mHealth interventions. We have added additional sources (Lines 133 and 153) and edited Lines 111-131 in the “Manuscript with track changes” file with the following text.

Although mHealth performance evaluations are growing[12–14], regarding the majority of commercially mobile health applications in our research field, companies and researchers have not yet incorporated an extensive performance evaluation, including usability and app quality. Furthermore, additional tailored and rigorous evaluation studies on the mHealth clinical impact and reliability are still required[15,16] .

In Methods, I would suggest making the first paragraph describing the mobile app clearer. As I continued reading, I understood the app but it is very confusing at the beginning. I would make Fig 2, Fig 1 instead. However, I am most worried about the usability evaluation. Did it happen during the training session, after the simulated emergency consultation? Or some time after? Did the authors measure actual engagement after the training session? I worry that the high scores are influenced by courtesy bias, apart from the selection bias already mentioned by the authors. Besides, the sample size is too small to even speculate about the external validity of the results.

We agree that the mobile app description might appear confusing. We reviewed the drafting and structuring of the section, where we describe the general and functional characteristics of the mobile application. Therefore, we reorganized almost the entirely Mobile application section (Lines 171-227). We included reviewers' recommendations regarding the designation of figures 1 and 2, which we changed (Fig1 instead of Fig 2) in the "Manuscript with track changes." Moreover, the features and functionality of each application module were described separately.

Reviewer's comments regarding the usability evaluation allowed us to include the following text (Lines 237-242) to make more precise the process described in the study procedures section.

"The usability evaluation was carried out in a single session according to the following stages:

1. Delivery of materials (devices, medical records, evaluation forms) and instructions about the application use and evaluation process

2. Simulated emergency consultation (Interaction with the application)

3. Individually and confidentially usability evaluation by GPs"

Thank you for pointing out the engagement measure, which is an essential part of the evaluation and was included in the usability evaluation and refers to sections B (user engagement) and F (subjective quality assessment) in the evaluation form (included as Supporting information).

Although the usability evaluation was performed by each GP anonymously, individually and secretly, we shared the reviewer's concern about the possible introduction of a courtesy bias as a disadvantage of this approach [8]; therefore, we included it as a limitation of our study (Lines 490-493). As a strategy to reduce the risk of courtesy bias, further research could include different assignment strategies for the evaluation, such as the use of digital channels or the use of evaluation tools embedded in the mobile application as the log reporting system.

As a recognized limitation, the sample size and extent of the study should be tackled towards a further stage of clinical impact evaluation. Despite these limitations, study results can be considered as a potential strategy to support GPs in primary healthcare centers, rural areas with a limited internet connection.

In the results:

How was CPG knowledge measure?

In lines 164-165 the authors mention that the majority of the GPs agreed that the app matched the CGP information, but in the previous sentence that almost half actually read them… so that could be also indicating courtesy bias?

The first two questions of the evaluation form qualitatively measured CPG's knowledge (S1 Evaluation form). Nevertheless, through these questions, we gathered a general observation regarding the GP's knowledge level of Ministry of Health’s guidelines for vector-borne disease management. Since our findings revealed a surprising proportion of GPs with a lack of knowledge, we decided to include them as part of the evaluation.

Regarding the GP's perception of the app's content and GCPs agreement, we have clarified in lines 359-361 these results the proportions discriminated by subcategories with the following text:

"The proportion of GPs who reported having more comprehensive knowledge about the guides was 9/20, more than half (5/9) ultimately agreed, and 4/9 partially agreed that the app matched the CPGs."

Based on the evaluation form (S1 Evaluation form) the answer options for the first question included the multiple levels of knowledge about CPGs. The second question's option "Not applicable" was included for those who answered in the first question that they did not know the GCPs, which may have generated confusion among some evaluators. Therefore, we hope that including evaluation form facilitates results interpretation.

In the discussion, the first sentence speaks about the novelty of the study but uses a very restrictive situation so, yes it is novel for that…

We appreciate this comment. We agreed that we present this study under a narrow perspective. However, we wanted to frame the discussion around essential conditions to considerer for the development and evaluation of mobile health interventions by adding the following text in lines 440-446 of the "Manuscript with track changes" file.

"There are relevant conditions to be considered for the mobile health applications development and adoption: the local demographic, geographic, connectivity, culture, and socio-sanitary aspects; and the particular needs of the potential end-users, general public, healthcare workers or surveillance authorities[34]. Among the commercially available mHealth developments, a small proportion combines different functionalities such as vector-borne disease management (evaluation, classification, and treatment) or rapid offline clinical information review."

Also, I do agree that more usability and feasibility studies are needed for mHealth but is this study enough to state this? Is a one-time (not longitudinal) and with a very small sample. This limits the interpretation of the results and I worry about speculations as the one presented in lines 205-207.

We agree that a pilot study approach with the mentioned limitations could also limit the interpretation of results. With the previous text, we intended to narrow the scope of our results and suggested the potential (Lines 416-420) of FeverDx as a support tool susceptible to improvement regarding functionality throught iterative evaluation. We added the following text:

“Hence, the use of FeverDx in regions where arboviruses are endemic or hyperendemic potentially could help GPs and stakeholders in the management of acute febrile syndromes such as arboviruses infections. Although this is a pilot study, our findings are a starting point for the development of new implementation studies that allow the evaluation of application performance in different contexts and scenarios”.

In lines 219, the authors may have omitted other apps. Of the top of my head I can think of Kidenga, for example. What are the other 26 apps relevant to surveillance referring to?

We have added relevant information regarding mobile health developments vector-borne diseases related that we omitted in the revised manuscript. We mentioned the apps related to dengue, chikungunya, and Zika surveillance and some of their features (Lines 432-434). Besides, in lines 403-407, we included commercially available mHealth from the most popular online stores (Google Play Store and Apple Store) to complement the description with the following text (Lines 435-439):

"Among the commercially available mHealth developments, a small proportion combines different functionalities such as vector-borne disease management (evaluation, classification, and treatment) or rapid offline clinical information review."

Lines 236-237 seem to talk about the general public but this app is not intended for that.

We appreciate this observation, and we agree that this statement might appear out of context. We changed the original statement in lines 469-471, and included the following text:

"Finally, mobile technology offers unique opportunities to reach and engage healthcare professionals to enhance health literacy and public health authorities for real-time disease surveillance."

Minor:

Lines 145-147: it is not very clear how the evaluation form was given in this sentence and Table 1 is presented in Results. I would suggest including the evaluation form as supp. material and improve this sentence.

Regarding the use of the evaluation form, in the mentioned lines, we described the components of the evaluation form. As the reviewer suggested, we rewrite the description in lines 333-345 to offer a more accurate description. The evaluation form will be included as supporting information to clarify the description of the evaluation form components. In lines 237-240 (Study procedures section), we described the process in which the evaluation form is given to the evaluators.

Lines 150-151: I assume that the authors referred to median and IQR as is stated in the following paragraph.

In line 347 we referred to median and IQR

Line 183: “as (un)acceptable”?

Lines 391-393 (Manuscript with track changes file) in the results section refers to the evaluation form, B section (user engagement). Question 13 inquiry about the application contents and its suitability for the group of users it is intended. The options for the answer are totally unappropriated, mostly unappropriated, acceptable (selected by one GP), adequately oriented, correctly targeted. This is clarified with the inclusion of the evaluation form.

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment

Submitted filename: Response to Reviewers.docx

Click here for additional data file. (41.3KB, docx)

Decision Letter 1

Abdallah M Samy

18 Mar 2020

PONE-D-19-31941R1

Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study

PLOS ONE

Dear Dr Rodriguez,

Thank you very much for submitting your manuscript "Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study" (#PONE-D-19-31941R1) for review by PLOS ONE. As with all papers submitted to the journal, your manuscript was fully evaluated by academic editor (myself) and by independent peer reviewers. The reviewers appreciated the attention to an important health topic, but they raised substantial concerns about the paper that must be addressed before this manuscript can be accurately assessed for meeting the PLOS ONE criteria. Therefore, if you feel these issues can be adequately addressed, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. We can’t, of course, promise publication at that time.

We would appreciate receiving your revised manuscript by May 02 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Abdallah M. Samy, PhD

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have responded to my concerns in a satisfactory way and although my concerns about bias and external validity remains, they have been able to make those limitations more explicit. They have greatly improved the manuscript, although minor changes need to be addressed before I can recommend it for publication.

Abstract

1- In the background it says: “A large number of mHealth tools are available; however, very few have been evaluated regarding usability and acceptability.”

This is not appropriate as discussed in the previous review and addressed in the reviewer’s comments. Look at the second paragraph of your introduction and use that argument. Why mhealth can help address the problem.

2- In the results is says “Seventy-five percent of the evaluators reported being aware of the Colombian Ministry of Health’s guidelines.”

This is not in the results though, either include in the results or only indicate that less than half had comprehensive knowledge (see comments on the CPG knowledge section of results).

3- In the conclusion it says “Despite a large number of mHealth tools available, the literature lacks evaluated and evidence-based mobile technology.” This is not completely true as described previously. The new discussion section address this pretty well, so you need to revisit that argument here and match the discussion.

4- Conclusions section is too general, you need to answer why FeverDx could help advance or improve practice based on the results of the pilot study. Also briefly mention the limitations (e.g., although it was a pilot study with a small sample size, FeverDx…)

Introduction

5- Where it says “…time and ease of use, access to content and particularly in a critical step of the reporting process of mandatory notifiable diseases.” Change for “…time and ease of use, and access to content in a critical step of the reporting process of mandatory notifiable diseases.”

6- Where it says “This particular feature, we hypothesize that could potentially improve the currently manual registration and report process, mostly using physical resources (paper forms, physical files)[10].” Which feature are you referring to? Maybe rephrase it as: “We hypothesize that these features (ease of use and rapid access to content) can potentially…”

7- Where it says “There are a recently increasing supply and demand of mobile applications development in the healthcare area”. Change for “Supply and demand of mobile applications development in the healthcare area have been recently increasing.”

8- Where it says “This study evaluates the usability and acceptability of FeverDx…” Briefly mention who valuated it and where. Use the first sentence from Methods.

Methods

9- Where it says “Moreover, an infectious disease specialist reviewed the included recommendations.” Moreover can and should be removed.

10- Where it says “Obtained data from registration, evaluation and reporting is storage” Change storage for stored.

11- The following phrase is confusing: “The evaluation form comprises a 25 item measure that includes four objective quality - information quality, engagement, functionality, aesthetics, and two subjective subscales-quality and impact.”

All items are subjective in the strict sense, based on the S1 File provided, I would recommend that you rephrase it as:

“The evaluation form comprises 25 questions measuring quantity and quality of information (Section A), engagement (Section B), functionality (Section C), aesthetics (Section D), impact of patient management from the practitioner perspective (Section E) and acceptability by health practitioners (Section F) (S1 file).”

12- Where it says “The evaluation test is scored by determining the mean scores of the app quality subscales and the total mean score.” You should rephrase it as: “A global score was obtained for the evaluation form test by determining the median scores of each subsection.”

13- S1 file: I suggest some changes:

Seccion E: the brief description is the same as Seccion D. This section measures impact on patient management from the practitioner perspective.

Seccion F: This section measures acceptability of users. Include this in the description.

Results

14- In the CPG knowledge section, to improve readability, I suggest rephrasing the whole section as:

“The proportion of GPs with comprehensive knowledge of the Colombian’s CPG for diagnosis and management of arboviruses (dengue, chikungunya, and Zika) was less than half of GPs evaluated (9/20) and less than one third of them applied them in their practice (6/20). Of those GPs who reported having comprehensive knowledge of CPG, more than half (5/9) wholly agreed and, 4/9 partially agreed that the application information matched with CPG information.”

15- Table 1: change subjective assessment of quality by acceptability

16- Where it says “Regarding the "interest generated by the application" item, GPs scored a median of 5 (IQR=4-4.5).”, which section is this in the table? Be consistent with names. Also the median doesn’t match the IQR, check that.

17- Where it says “Moreover, all the GPs recommended the app”, change it for “Moreover, all the GPs would recommend the app”

Discussion

18- Where it says “Mobile applications designed for epidemiological surveillance of arboviruses are scope limited”, rephrase it as: “Mobile applications designed for epidemiological surveillance of arboviruses are limited in their scope”

19- Where it says ”On the one hand, a study conducted in 2018”, “On the one hand” is not appropriate. You should remove it.

20- Where it says “...that aimed case reporting and outbreaks..”, change for aimed at case…

21- Where it says “…some limited by country, the language with narrow applicability…” Do you mean that the language limited their applicability because they were mostly in English? Were they designed to be used by English-speaking users or in developed countries? Making that more explicit will help your argument.

22- Where it says “However its implementation in clinical scenarios is scarce”, do you mean they are not designed to be used by GPs in clinical scenarios? Rephrase accordingly.

23- Where it says “A striking find was”, change for “A striking finding was”

24- This phrase “…although the majority of the GPs agreed that the application content was according to management guidelines.” is not necessary here and interrupts the flow. Moreover, using although implies that is in contrast with the previous statement which is not. I would remove this phrase altogether.

Limitations

25- Consider including a statement about their applicability mostly restricted to Colombia, given the country specific CPG and surveillance systems.

Reviewer #2: Even though the article is clear and the mobile app is interesting the findings does not correspond to an orignial article, it is more suitable for a short communication.

Abstract: in methods I would add the grade scale of uMARS, if not results are not understandable without reading the full article.

Introduction: it is not clear which is the potential impact of the using of the app among GP. Line 67, this paragraph should be at the end of the introduction or even in methods.

Methods:

Fig2, when referring to alarm signs that includes all dengue warning signs? if so it should be stated.

Results: CPR knowledge prior of after using the app?

It is not clear if the app was used in clinical context, with how many patients, if it was useful for the final diagnosis.

Discussion: there is no future plans for further research or for the app.

General thoughts: First of all, it could be missleading using an app for fever which do not discriminate malaria or bacterial infections at the begining. An algorithm of fever in the tropics should rule out the most dangerous causes first, and if it is not a malaria zone it should be stated; nevertheless bacterial infections are distributed worlwide.

Although the info about this app should be shared with the community little information could be extracted from this article. I would suggest to compare two GP groups, one using the app and the other not using it and seeing the differences; but the most important thing is that the app is not validated with confirmed cases, apparently the purpose of the app is only to apply the guidelines, if it is so it should be explained. Could be much more interesting if this app can be used by health care workers who not are doctors. Maybe the app should focuse more in epidemiological vigilance.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2020 May 29;15(5):e0233269. doi: 10.1371/journal.pone.0233269.r004

Author response to Decision Letter 1

28 Apr 2020

Reviewers

PLOS ONE

April 28th, 2020

Dear reviewers,

We thank you for a thorough revision and valuable suggestions regarding our manuscript and for the opportunity to revise and resubmit. We are pleased to submit the improved manuscript. On the following pages, we include our response to each recommendation and comments. On behalf of my co-authors, I thank you for your consideration of this resubmission. We appreciate your time and look forward to your response.

Sincerely,

Sarita Rodríguez R, M.D.

Corresponding author

sarita.rodriguez@fvl.org.co

+573173820407

Attachment

Submitted filename: Response to Reviewers_07_04_2020.docx

Click here for additional data file. (50KB, docx)

Decision Letter 2

Abdallah M Samy

4 May 2020

Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study

PONE-D-19-31941R2

Dear Dr. Rodriguez,

We are pleased to inform you that your manuscript, "Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study" (PONE-D-19-31941R2), has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Abdallah M. Samy, PhD

Academic Editor

PLOS ONE

Acceptance letter

Abdallah M Samy

13 May 2020

PONE-D-19-31941R2

Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study

Dear Dr. Rodríguez:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Abdallah M. Samy

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 File. Evaluation form (English version).

    (PDF)

    Click here for additional data file. (93KB, pdf)
    S2 File. Evaluation form (Spanish version).

    (PDF)

    Click here for additional data file. (117.1KB, pdf)
    S1 Dataset. Raw evaluation results.

    (PDF)

    Click here for additional data file. (260.3KB, pdf)
    Attachment

    Submitted filename: Response to Reviewers.docx

    Click here for additional data file. (41.3KB, docx)
    Attachment

    Submitted filename: Response to Reviewers_07_04_2020.docx

    Click here for additional data file. (50KB, docx)

    Data Availability Statement

    All relevant data are within the manuscript and its Supporting Information files.

    Articles from PLoS ONE are provided here courtesy of PLOS

    RESOURCES