Implementing nationwide facility-based electronic disease surveillance in Sierra Leone: Lessons Learned

Abstract

The Global Health Security Agenda aims to improve countries’ ability to prevent, detect, and respond to infectious disease threats by building or strengthening core capacities required by International Health Regulations. One of those capacities is the development of surveillance systems to rapidly detect and respond to occurrences of diseases with epidemic potential. Since 2015, the US Centers for Disease Control and Prevention (CDC) has worked with partners in Sierra Leone to assist the Ministry of Health and Sanitation (MoHS) in developing an Integrated Disease Surveillance and Response (IDSR) system. Beginning in 2016, CDC, in collaboration with the World Health Organization and eHealth Africa, supported MoHS in the development of Android device mobile data entry at the health facility (HF) for electronic IDSR (eIDSR), also known as HF-based eIDSR. HF-based eIDSR was introduced via a pilot in one district, and national rollout began in 2018. With over 1,100 HF now reporting, the Sierra Leone eIDSR system is substantially larger than most mobile-device health (mHealth) projects found in the literature.

Several technical innovations contributed to the success of HF-based eIDSR in Sierra Leone. Among them were data compression and dual-mode (internet and text) message transmission to mitigate connectivity issues, user interface design tailored to local needs, and a continuous-feedback process to iteratively detect user or system issues and remediate challenges identified. The resultant system achieved high user acceptance and demonstrated the feasibility of a mHealth-based surveillance system implemented at national scale.

Background

During the 2014–2016 West African Ebola epidemic, the US Centers for Disease Control and Prevention (CDC) initiated multiple activities in Sierra Leone under the Global Health Security Agenda (GHSA). The purpose of GHSA is to improve countries’ ability to prevent, detect, and respond to infectious disease threats through core capacities required by International Health Regulations (IHR).¹ GHSA efforts are categorized into eleven “action packages” that correspond to specific technical areas.^1,2 Within the “real-time surveillance” and “reporting” action packages, CDC and World Health Organization (WHO) supported the Sierra Leone Ministry of Health and Sanitation (MoHS) and partners to revitalize an Integrated Disease Surveillance and Response (IDSR) system in 2015.^3,4 Originally, IDSR was paper-based. MoHS and partners implemented electronic reporting, including mobile-device-based reporting beginning in 2016.⁵

The use of mobile devices for health applications – whether for surveillance, reference, or clinical guidance – is commonly described as “mHealth.” Although previous literature has documented effective implementation of mHealth for public health surveillance,^6,7,8,9,10 including IDSR,¹¹ most reports have described pilots or small-scale implementations (<50 sites). A 2013 review observed that there is “a lack of [literature documenting] mHealth applications and services operating at scale in low- and middle-income countries.”¹² With successful implementation of a nationwide system, this report contributes an answer to that deficit.

Methods I: System Development/Implementation

Initially, IDSR was a paper-based system implemented at health facilities (HF) including hospitals and peripheral health units across Sierra Leone. An IDSR focal person at each HF completed a paper Weekly Reporting Form compiling total suspected cases of 26 priority diseases or conditions (Table 1) from the HF’s clinical (paper) register. Focal persons submitted IDSR data via telephone or hard-copy form to the District Surveillance Officer (DSO) at their District Health Management Team (DHMT) office. DSOs would compile reports from all HF into a weekly summary, which was submitted to the national surveillance program by email.

Table 1:

Integrated Disease Surveillance and Response (IDSR) priority diseases and conditions reported weekly in Sierra Leone

Acute flaccid paralysis Acute viral hemorrhagic fever Acute jaundice syndrome Adverse event following immunization (AEFI) Animal bite (dog, cat) Anthrax Buruli ulcer Chikungunya Cholera Dengue fever Diarrhea with severe dehydration in children under 5 years of age Dracunculiasis (Guinea Worm) Dysentery (Bloody diarrhea) Influenza due to new subtype

Malaria Suspect case Tested (RDT or microscopy) Tested positive Malnutrition (severe) in children under 5 years of age Maternal death Measles Meningococcal meningitis Monkeypox Neonatal Tetanus Plague Severe Pneumonia Smallpox Typhoid Fever Yellow Fever

Source: Sierra Leone Technical Guidelines for Integrated Disease Surveillance and Response²

To increase timeliness and completeness of reporting and to facilitate analysis, MoHS initiated electronic IDSR (eIDSR) in 2015 with support from CDC, WHO, eHealth Africa (eHA) and other partners.³ eIDSR was rolled out in two phases:

Phase 1:

Computers, internet and data entry staff were placed in DHMTs. A national web-based surveillance database was developed on the open-source District Health Information System 2 (DHIS2),¹³ and DHMT personnel were trained to enter weekly data into a secure eIDSR website. Only the DHMTs used eIDSR during Phase 1; HF focal persons continued to use paper forms until Phase 2.

Phase 2:

eIDSR was implemented at the HF level using Android tablets for data entry with district-level review of submissions. HF-based eIDSR is the first at-scale use of mobile systems for healthcare (mHealth) in Sierra Leone.

eIDSR Mobile App Development

Development of HF-based eIDSR was overseen by a Technical Working Group (TWG) consisting of representatives from MoHS, CDC, WHO, and eHA. The TWG developed initial requirements for HF-based eIDSR (Table 2 and Figure 1). Requirements specified that the mobile application would enable data entry directly from HF to the national database, incorporate data validation, and have capacity to transmit data both via internet and text messages using short message service (SMS) or Short Message Peer-to-Peer (SMPP) communication modes.

Table 2:

Initial Requirements for the Health-Facilty (HF)-based Electronic Integrated Disease Surveillance and Response (eIDSR) mobile application

HF-based eIDSR must transmit data into existing DHIS2-based eIDSR national system Use of android devices to enable validation and message coding Data entry screen similar to IDSR paper forms (Figure 1) Require completion of mandatory data fields Ability to store completed reports for later transmission

Confirmation of entry for reports of priority diseases Data validation (e.g. number of positive malaria tests not greater than total tests) Encoding of transmitted reports done by the application from plain-English data entry, rather than relying on the user to encode a report Capacity to transmit data by both Internet (HTTP POST) and text (SMS) protocols

Figure 1:

IDSR Paper Form and HF-based elDSR App Entry Screen, Sierra Leone, 2017

While DHIS2 remained the foundation for HF-based eIDSR, incorporation of requirements outlined in Table 2 required substantial custom programming, including a new mechanism to receive SMS/SMPP messages. eHA performed these modifications for Sierra Leone, and all code has been submitted to DHIS2 repositories.

HF-based eIDSR was first implemented at six HF in Port Loko district during November-December 2016. Subsequent cohorts were initiated in April and May 2017. National rollout to the remaining 13 districts began in April 2018 and was completed in June 2019.

Personnel from all IDSR-reporting HF were trained on eIDSR applications and procedures. They were issued a tablet with software and mobile data installed. Trained HF personnel reported in parallel using both paper and mobile application, and were monitored until the district achieved quality metrics set by the TWG. Once approved, HF-based eIDSR became the sole reporting method.

Methods II: System Assessment/Data Analysis

A mixed-methods approach was employed to assess system performance. Continuous-feedback monitoring during development and throughout deployment provided rapid visibility into operational issues and permitted iterative improvements. Usability, user acceptance, and data quality were assessed among pilot users and results returned to the continuous-feedback process. Finally, overall system performance was assessed through centralized data analysis.

Continuous-Feedback

System performance and user experience were continuously assessed through technical support calls and direct observation of users in training and at their work locations. As issues were identified, they were addressed through iterative updates to both the eIDSR application and the back-end system (Table 3).

Table 3:

Continuous-Feedback Implementation for Sierra Leone Electronic Integrated Disease Surveillance & Response (eIDSR) mobile system: Issues and Solutions

Issues Identified	Solution implemented
Users had difficulty understanding error messages, particularly when a field was left blank or when an entry failed error checks.	Messages modified to use simplified language, to display in-screen popups which require user response. Colored outlines highlight the specific field in which a correction was needed.
Some SMS/SMPP (text) transmissions were failing mid-transmission when network signals failed. This was traced to the fact that reports were transmitted using six to seven SMS messages for a single report.	Reporting message was compressed to a single SMS/SMPP message. This also reduced data expense.
Users were unable to transmit SMS messages due to lack of account credit	Implemented Short-Messaging Peer-to-Peer (SMPP) protocol which did not have a per-message charge.
Users did not know when a submission was successful or not. Sometimes they submitted multiple reports for the same week; other times they did not re-attempt a failed transmission, resulting in a non-reported week.	The eIDSR app was configured to notify the end-user of successful or failed transmission.
Users reported weak or intermittent network signal on their tablets, and that phones using a different mobile service provider showed a stronger signal.	Contracts were negotiated with the two largest mobile providers in Sierra Leone, and tablets were configured to use the provider best suited to signal strength at each HF.
DSOs identified errors in data reported from facilities in their district	Data review process was created to enable DSOs or their designees to approve all HF submissions from their district before the data are visible in the national system.
Data Quality Assessment identified transcription errors between paper records and tablet/DHIS2 data. Most of these errors came from entering data on the wrong line of the data entry screen.	Additional user training and clearer on-screen guidance mitigated reporting errors.
Data from previous reports on tablet disagreed with the DHIS2 database due to correction by the DSOs. Users also requested ability to see their reporting history so they could analyze trends in their own facilities.	All reported data, including corrections by DSOs, were made available to the end-users through synchronization of the mobile device with the DHIS2 database.
Unauthorized software on tablets was corrupting the system, and excessive personal use was exhausting data credit on accounts. Additionally a few tablets went missing.	Mobile Device Management (MDM) was implemented to restrict mobile devices to the specific applications and functions needed for HF-based eIDSR, and to permit remote deployment of software updates. MDM also enabled device tracking and locking it down if lost or stolen.

Usability/User-Acceptance Assessment

In July 2017, a Usability/User-Acceptance assessment was conducted in Port Loko. Of 112 HF that were then operational for facility-based eIDSR, 47 HF were assessed. Facilities were selected to represent varying levels of mobile connectivity, physical accessibility, and frequency of recent reports to the IDSR database in DHIS2.

Methodology consisted of a structured interview, direct observation of each user demonstrating basic operational tasks, and completing a mock eIDSR report submission. Interview questions were asked of the user, who answered using a five-point Likert scale from “strongly agree” to “strongly disagree.” For operational tasks and data entry, observers rated each user as showing No Experience, Novice (limited experience), Intermediate (demonstrated practical ability to use the application), or Expert (demonstrated full and confident ability) capacity.

Assessments were conducted at the HF where the focal person normally worked, except for six HF that were inaccessible due to flooding during this period. On-site assessments permitted observation of functionality issues, such as app and network performance, not readily apparent in training locations. Issues observed were incorporated into the continuous-feedback process described above.

Data Quality

Data quality was assessed simultaneously with user acceptance. Data were collected from Weekly Reporting Forms, HF-based eIDSR application logs, and the national DHIS2 database. Numbers reported for suspected malaria cases and deaths June 5-July 2, 2017 were compared between all three sources. Malaria was chosen because it is the only condition for which cases reported are consistently greater than zero.

System Performance

To assess overall system performance, data on eIDSR submissions from April 9-June 16, 2019 were exported from DHIS2 and analyzed using R 3.6.0.¹⁴ Communication mode for reports was categorized as manual (paper-based), mobile-internet or mobile-SMS/SMPP.

Results

Continuous-Feedback

This process identified multiple issues with the HF-based eIDSR application as initially implemented. Issues detected and solutions implemented appear in Table 3.

Usability/User-Acceptance Results

Of 47 users assessed, 43 (91%) reported preference for tablet-based eIDSR over paper-based reporting. Forty-six (98%) agreed or strongly agreed with the statement “I find the eIDSR mobile application easy to navigate,” and all 47 (100%) agreed or strongly agreed with the statement “I feel very confident using the eIDSR mobile application.” Of the 47 respondents, 35 (74%) were evaluated by the observers as Expert, 6 (13%) as Intermediate, and six (13%) as Novice.

Data Quality

Of 188 reports assessed for data quality, 138 (74%) had complete records in paper logs, HF-based eIDSR application, and DHIS2. Of these 138 reports, 126 (91%) were identical in all three sources, and 12 (9%) had at least one disagreement between sources. Discordant records showed disagreement between paper and HF-based application. Five of 138 (3.6%) also showed disagreement between the application and DHIS2.

System Performance

As HF-based eIDSR rolled out nationally, the proportion of IDSR submissions from HF through paper forms (manual) decreased and mobile submissions increased (Figure 2), while the number of facilities reporting remained roughly constant. The proportion of submissions using the Mobile-Internet vs Mobile-SMS/SMPP modes fluctuated with varying network conditions.

Figure 2: Electronic Integrated Disease Surveillance & Response (elDSR) Submissions from Health Facilities, by Communication Mode.

Sierra Leone, April 9,2018 through June 16,2019

As of June 10, 2019, 1,316 HF from all 14 Sierra Leone districts had been trained in HF-based eIDSR. Of these HF, 1,156 (88%) submitted reports using the mobile application, 116 (9%) by manual reporting, and 44 (3%) failed to report. Of the 1,156 who reported through the eIDSR application, 538 (47%) submitted by Internet and 618 (53%) by SMS/SMPP.

Discussion

Sierra Leone successfully implemented HF-based electronic reporting of IDSR data at national scale, with nearly 90% of HF reporting through eIDSR. In November 2016, the official Joint External Evaluation (JEE) of Sierra Leone’s GHSA targets scored indicator D.2.2, an “interoperable, interconnected electronic real-time reporting system,” at level two out of five.¹⁵ eIDSR may help Sierra Leone reach IHR compliance and potentially raise the JEE D.2.2 score.

A major strength of our assessment is the countrywide implementation of Sierra Leone’s HF-based reporting system. Multiple studies have discussed mHealth utility to improve healthcare delivery or reporting in lower-income countries, but they tend to describe small-scale interventions.^6–11 With over 1,100 HF in all 14 districts reporting via mobile HF-based eIDSR, Sierra Leone demonstrates successful scaling of a mHealth application in public health surveillance.

Several publications have reported health workers’ ability to use mHealth applications even with little or no prior computer experience.^16,17 Our User-Acceptance assessment reported high (>90%) acceptance of the mobile eIDSR application, and over 80% of users demonstrated at least intermediate skill in its use. Braun et al. describe the value of in-software guidance to users.¹⁸ Through the continuous-feedback process, we observed that user-guidance and confirmation-of-submission modifications markedly improved usability of the application for HF-based end users.

Continuous-feedback design was vital to the success of HF-based eIDSR and helped to identify issues around usability and function. Although not commonly described in public health literature, the principle of continuous user feedback is well-known in software development.^19,20,21 As Kazi and colleagues observed, “…iterative development with early and continuous engagement of end users ensures that the final product is efficient and usable, both key to widespread adoption.”¹⁰ We concur with Kazi et al. and add that monitoring users’ experience in their actual work environments also provides vital insights into real-world operational issues and identifies needs for additional training.

The importance of improving internet connectivity for global health has been described elsewhere.^{3, 22,23} Phalkey and colleagues²³ observed that ineffective communication and transport systems adversely affect surveillance systems. Coverage throughout Sierra Leone ranges from locations with stable mobile data, to locations where only voice or SMS are accessible, to locations where there is no signal at all. In some remote HF, focal persons reported having to walk or ride some distance to a spot where they could obtain a signal. Furthermore, mobile connectivity in Sierra Leone is supplied by several providers, and the geographic coverage for each provider is variable. It is crucial to monitor shifts in connectivity that may happen during the life of a program, which may affect reporting rates. Mitigation strategies described in this paper may become less important as network infrastructure improves.

Diwan and colleagues²⁴ observed that functionality to store data for later transmission helps to address network limitations, and the Sierra Leone experience supports this observation. Our innovation in capacity to transmit by both Internet and text may be a useful adaptation to consider in locations with connectivity challenges. Our results demonstrate that SMS/SMPP capacity remains an important conduit for eIDSR in Sierra Leone, even among facilities that sometimes have mobile internet access.

Our text compression modification was of vital importance to system functionality. Initially, the mobile app coded a single report into a series of six to seven text messages transmitted in sequence; however, in areas with weak signal, reports sometimes failed when not all messages were successfully transmitted. Compression to a single text message mitigated risk of transmission failure. In-app confirmation to users of submission success or failure, and reminders for unsubmitted reports are also important.

This work is subject to a few limitations. First, this report is restricted to technical aspects of system implementation and operation and does not address eIDSR effectiveness for public health surveillance. Data quality as addressed in this report refers to completeness and concordance between data at HF and that reported to the system. A more robust evaluation according to CDC surveillance evaluation guidelines²⁵ would include measures of actual disease reports and outbreaks detected. Such an evaluation would require additional on-site assessments beyond the scope of, and resources available to this project and is recommended as a future strategy.

Second, usability and data quality were assessed at only 47 early-implementation HF which may not be representative of all HF. While we believe the large scale of present implementation implies high user acceptance, additional evaluations could confirm our results and reassess data quality.

The final limitation is the question of sustainability. Continued success of HF-based eIDSR in Sierra Leone depends upon ongoing support for technical staff to maintain, update and troubleshoot the system. Sierra Leone’s technical innovations have been shared with the DHIS2 community, and incorporation into the core software may mitigate future support costs; however, continued financial support for mobile data contracts and software subscriptions is necessary.

Although MoHS strongly supports the eIDSR initiative, both staffing and financial requirements currently depend upon external partners.²⁶ As with any innovation in a low-resource environment, long-term ownership and sustainability depend upon reconciliation of many priorities competing for limited funds.

Conclusion and Lessons Learned

The Sierra Leone HF-based eIDSR system successfully receives and consolidates eIDSR reports from over 1,100 health facilities in 14 districts. Staff at these facilities have demonstrated the ability to learn and operate tablet-based reporting software. Other countries implementing electronic surveillance or mHealth activities in resource-constrained settings may benefit from several lessons learned:

1. mHealth depends upon reliable network connectivity. Adverse impacts of limited or variable connectivity can be mitigated through:

   • Functionality to store data for transmission when on-network;
   • Capacity to transmit data by multiple modes, such as internet and SMS/SMPP;
   • Data compression to limit size and quantity of transmitted messages; and
   • Utilization of multiple data providers to maximize coverage.

2. An application that is easy to use is more likely to be accepted and used successfully. mHealth application design should incorporate features such as context-specific data checks and feedback messages adapted to end users. Error messages should highlight specific actions required, and be validated through field-testing with end users.

3. Continuous user feedback and rapid integration of modifications can enhance overall system quality and user acceptance.

The success of eIDSR in Sierra Leone demonstrates the feasibility of national-scale, facility-based, mHealth surveillance systems in limited-resource settings. Other countries may consider similar efforts to enhance their surveillance and reporting for global health security purposes.