Assessment of VaxTrac electronic immunization registry in an urban district in Sierra Leone: Implications for data quality, defaulter tracking, and policy

Abstract

Background:

In 2016, the Sierra Leone Ministry of Health and Sanitation (MoHS) piloted VaxTrac, an electronic immunization registry (EIR), in an urban district to improve management of vaccination records and tracking of children who missed scheduled doses. We aimed to document lessons learned to inform decision-making on VaxTrac and similar EIRs’ future use.

Methods:

Ten out of 50 urban health facilities that implemented VaxTrac were purposively selected for inclusion in a rapid mixed-method assessment from November to December 2017. For a one-month period, records of six scheduled vaccine doses among children < 2 years old in VaxTrac were abstracted and compared to three paper-based records (register of under-two children, daily tally sheet, and monthly summary form). We used the under-two register as the reference gold standard for comparison purposes. We interviewed and observed 10 heath workers, one from each selected facility, who were using VaxTrac.

Results:

Overall, VaxTrac captured < 65% of the vaccine doses reported in the paper-based sources, but in the largest health facility VaxTrac captured the highest number of doses. Two additional notable patterns emerged: 1) the aggregated data sources reported higher doses administered compared to the under-two register and VaxTrac; 2) data sources that need real-time data capture during the vaccination session reported fewer doses administered compared to the monthly HF2 summary form. Health workers expressed that the EIR helped them to shorten the time to manage, summarize, and report vaccination records. Workflows for data entry in VaxTrac were inconsistent among facilities and rarely integrated into existing processes. Data sharing restrictions contributed to duplicate records.

Conclusion:

Although VaxTrac helped to shorten the time to manage, summarize, and report vaccination records, data sharing restrictions coupled with inconsistent and inefficient workflows were major implementation challenges. Readiness-to-introduce and sustainability should be carefully considered before implementing an EIR.

1. Introduction

Sierra Leone’s health system was recovering from a devastating civil war when the country was struck by a large outbreak of Ebola Virus Disease in May 2014 [1]. The Ebola outbreak lasted for almost two years and disrupted primary health care services [2] including routine childhood immunization [3]. For example, a survey conducted in three communities in Western Area district revealed that vaccination coverage of the first dose of measles-containing vaccine declined from approximately 71% in August 2014 before the peak of the outbreak to 46% in April 2015 after the peak [4]. Since the Ebola outbreak in 2014, the US Centers for Disease Control and Prevention (CDC) and partners have been assisting the Sierra Leone Ministry of Health and Sanitation (MoHS) to strengthen integrated health systems to prevent, detect, and respond to health threats through the Global Health Security Agenda [5,6].

Childhood immunization is one of the most effective public health interventions [7]. CDC’s efforts to strengthen health systems in Sierra Leone focused in part on improving immunization services [6]. Based on a request from the MoHS in June 2016, CDC agreed to support the pilot of an electronic immunization registry (EIR) in health facilities in Western Urban district, Sierra Leone [8]. VaxTrac was the EIR selected to improve the management of children’s vaccination records using a biometric identifier. The EIR had other features to quickly identify children who are due for, or have missed, one or more scheduled vaccination doses. While VaxTrac has been introduced elsewhere prior to the pilot in Sierra Leone, evaluations of those efforts have not been conducted.

In 2010, the Bill & Melinda Gates Foundation provided $100,000 seed funding to a non-profit software development group to support the initial development and field testing of “a mobile phone-based vaccination registry that uses fingerprint scans to track those who have received immunizations in hopes of reducing redundant doses and boosting coverage levels in developing countries” [9]; this system became known as VaxTrac. In 2012, VaxTrac was implemented as a pilot project in 38 health facilities in Benin and eventually expanded to 100 health facilities [10]. VaxTrac was also introduced in health facilities in Nepal through the Vial-to-Child project [11].

Based on a request from the Sierra Leone MoHS to address the challenges of using paper-based registers, the CDC supported the piloting of VaxTrac to understand its feasibility in an urban setting including acceptability among health workers and caregivers. The biometric feature of VaxTrac was particularly attractive given the lack of a unique national registration number for children, low proportion of children with a birth certificate, and the high frequency of similar names in the paper-based immunization records. Identifying children who are due for or have missed scheduled vaccine doses is supposed to be done regularly by health facilities but health care workers (HCWs) face many challenges performing these tasks manually with paper-based registers. For instance, to identify children who are due for the second dose of measles-containing vaccine, the HCW would need to manually search the child health register going back at least 15 months from the due date to identify the age-eligible children. The reason for this is because the order in which children are entered in the paper register is based on their date of birth. The poor physical condition of the paper-based registers also makes it difficult for HCWs to abstract data from the paper-based registers. Similarly, summarizing doses administered using paper-based registers is time-consuming and often leads to inaccurate counts.”

The main objective of piloting VaxTrac in Sierra Leone was to better understand the feasibility of an EIR to help health workers improve management of vaccination records, improve data quality, and make the reporting of data timely. A secondary objective was to enhance the tracking of children past due for one or more vaccination doses (defaulted children) in urban areas. Western Urban district, hosting the country’s capital, was selected for testing the feasibility of VaxTrac because of the unique complexities in delivering immunization services in urban settings – including frequent population movements between and within localities, sprawling of informal settlements, and heterogeneity of the population [12].

eHealth Africa installed and configured VaxTrac on Android-based tablets. Use of quick response (QR) barcode technology [13], on registers maintained at health facilities and linked to children’s vaccination record, served as a back-up unique identifier to retrieve records if the fingerprint scanning feature malfunctioned. Users were able to enter data into VaxTrac using an offline mode to later synchronize them to a central data hosting server. Existing workflow guidance specified that initial data entry was to be made to the paper-based register of under-two children. Data sharing was enabled among geographic groupings of seven to twelve health facilities (Fig. 1).

Fig. 1.

Grouping of health facilities for VaxTrac data sharing, Western Urban, Sierra Leone, 2017.

By June 2017, VaxTrac had been implemented in 50 out of 70 health facilities in the district. eHealth Africa trained 115 health staff on how to use the system – two staff from each health facility plus 15 national- and district-level immunization staff [14]. No evaluation has been conducted on the extent to which the EIR possibly helped health workers improve management of vaccination records, improve data quality, and enhance the tracking of children due for one or more vaccination doses. We therefore aimed to document the successes and challenges from piloting VaxTrac in Sierra Leone to inform decision-making on future use of the system in the country or elsewhere.

2. Methods

2.1. Design and sampling

We conducted a mixed-methods assessment in ten health facilities that implemented VaxTrac in Sierra Leone. The assessment was carried out between November and December 2017. Health facilities were purposively selected for inclusion into the assessment to ensure variability in facility type, time since staff were trained, and geographic distribution. Four hospitals and six community health centers that had implemented VaxTrac for about one year were included in the assessment. At least one hospital and one community health center were selected from each of the three geographic wards (East, West and Central) in the district. Proportionally more health facilities were selected from the East ward because it comprised about half of all facilities that were using VaxTrac at the time. Ola During Children’s (ODC) Hospital was included in the assessment because it was the first health facility where VaxTrac was introduced and is the largest pediatric referral hospital in Sierra Leone delivering immunization services to children from various wards in the district. Geographic distribution of the ten selected health facilities is shown in Fig. 2.

Fig. 2.

Geographic distribution of the ten health facilities included in the VaxTrac assessment, Western Urban, Sierra Leone, 2017.

2.2. Data sources

We examined vaccination records from four data sources, which comprised two child-based registers (under-two register and VaxTrac EIR) and two aggregate data sources (daily tally sheet and monthly reporting form). As per guidance from Sierra Leone’s Expanded Program on Immunization (EPI), whenever a child under the age of two years was vaccinated, records of the vaccines administered were entered in the under-two register. The under-two register contained basic demographic information about the child (e.g. date of birth, name, sex, mother’s name, address) and records of the date each vaccine dose was administered. Records from the under-two register were then supposed to be manually entered in VaxTrac and linked to a unique identification number using a QR barcode affixed onto the under-two register in real-time during the vaccination session. The vaccination data entered in VaxTrac were stored locally on the Android tablet and synchronized regularly with a web-based hosting server where the data were permanently stored. Data sharing was restricted to five groups of seven to twelve health facilities (Fig. 1).

The daily summary of vaccine doses administered were supposed to be tracked using tally sheets. At the end of the month, health facilities were then required to aggregate the cumulative vaccine doses administered for each antigen in the immunization schedule using a monthly summary form, called the Health Facility Form 2 [HF2]. We did not identify any MoHS guidance on which data source to use to aggregate the monthly doses into the HF2. Monthly aggregated data reported in the HF2 were submitted to the District Health Management Team for entry into the District Health Information Software Version 2 (DHIS2). DHIS2 is an electronic health information system used in Sierra Leone and other countries [15]. Based on our conversations with immunization stakeholders in Sierra Leone, the under-two register is the primary data source on childhood immunization in the health facilities. We therefore considered the under-two register as the gold standard. The under-two register is also most appropriate for 1:1 comparison to VaxTrac because both are child-based registers whereas the tally sheets and HF2 summary forms only contain aggregate data.

2.3. Abstraction of vaccine doses from immunization records

From the four aforementioned data sources, we abstracted facility-level aggregated monthly doses administered for Bacillus Calmette–Guérin (BCG) vaccine scheduled at birth, first dose of pentavalent vaccine¹ (Penta-1) scheduled at six weeks, second dose of pentavalent vaccine scheduled at ten weeks (Penta-2), third dose of pentavalent vaccine scheduled at 14 weeks (Penta-3), first dose of measles-containing vaccine scheduled at nine months (MCV-1), and second dose of measles-containing vaccine scheduled at 15 months (MCV-2). We abstracted data for the month of September 2017 for the first batch of health facilities visited in November 2017, and abstracted data for October 2017 for the second batch of health facilities visited in December 2017. Of the ten health facilities assessed, eight had data available for all four sources.

2.4. Semi-structured interviews

We conducted face-to-face interviews with ten full-time health staff – one from each of the health facilities in our sample. To be eligible, health staff had to have been trained by eHealth Africa on the use of VaxTrac and had to have a direct role in vaccinating children at the health facility. Health staff who were trained by peers were excluded. Interviews were conducted using a semi-structured questionnaire administered by authors MJF, DO, and MSJ. Questionnaire domains included health facility characteristics, user characteristics, VaxTrac training and user experience, biometrics and QR codes, utilization of VaxTrac data, and caregiver acceptability of VaxTrac (Supplementary Material). Finally, we conducted interviews with two technical staff responsible for managing VaxTrac to follow-up on issues that emerged from the interviews with the health staff.

2.5. Observations

Interviewers directly observed the ability of the health workers to use key functions of the EIR including using the biometric device and QR barcodes, creating new registrations, updating existing registrations, identifying children who are due for vaccination, identifying children who have missed a scheduled vaccine dose, and summarizing doses administered. Workflows for entering data in the paper-based systems and VaxTrac were observed as well as the availability of power supply and Internet connectivity at the time of the visit. Observations were recorded in the same questionnaire used for interviews. We did not observe how HCWs used paper-based records to identify children who were due for or had missed scheduled vaccine doses.

2.6. Data analysis

We compared the cumulative number of vaccine doses administered that were captured in VaxTrac to each of the three paper-based sources (hereafter referred to as data congruity) using Pivot-Tables in Microsoft Excel (2016 version). Bar graphs were generated to show variations by vaccine dose and data source. Individual child-based records (under-two register and VaxTrac) were not directly compared in the data congruity analysis. For standardization in comparisons, two health facilities (out of 10) that had one or more unavailable data source(s) were excluded from the data congruity assessment² but included in all other analyses. Finally, we descriptively analyzed closed-ended questionnaire items and qualitatively synthesized open-ended items.

2.7. Ethical approval

The assessment was approved as a public health program activity and received non-research determination from the CDC’s Center for Global Health. All respondents provided verbal consent before participating in the assessment. Permission to access the health facility data was obtained from the EPI Manager from MoHS.

3. Results

The ten health facilities assessed collectively served an approximate catchment population of 231,845, which included 41,037 children under the age of five years and 9,272 children under the age of two years (Table 1). A total of 46 HCWs were working full-time across the facilities at the time of the assessment, of whom 41 (89%) were reportedly using VaxTrac and 21 (46%) were directly trained on VaxTrac by eHealth Africa. Of the ten VaxTrac users interviewed, nine were female, nine were State Enrolled Community Health Nurses, six had worked for the past five years in childhood immunization, and nine had prior experience in using a smart phone or tablet (data not shown in table).

Table 1.

Characteristics of health facilities sampled, VaxTrac assessment, Western Urban, Sierra Leone, 2017.

Facility Name	Facility Type	Ward Location	Catchment population&	Under-5 population&	Under-1 population&	EPI staff*	Staff using VaxTrac#	Staff trained on VaxTrac
Facility visited in November 2017 (N = 5)
1. Ola During	Hospital	East	35,500	6,284	1,420	8	5	2
2. Kuntorloh	CHC	East	38,210	6,763	1,528	12~	12	2
3. Kissy CHC	CHC	East	42,904	7,594	1,716	4	2	2
4. Military 34	Hospital	West	11,734	2,077	469	4	4	3
5. Murray Town	CHC	West	20,110	3,560	804	3	2	2
Facility visited in December 2017 (N = 5)
6. Mabella	CHC	Central	11,663	2,064	467	2	1	2
7. King Harman	Hospital	Central	11,384	2,015	455	4	4	2
8. Calaba Town	CHC	East	12,805	2,266	512	3	3	2
9. Allen Town	CHC	East	15,436	2,732	617	3	5^	2
10. Rokupa	Hospital	East	32,099	5,682	1,284	3	3	2
Total			231,845	41,037	9,272	46	41	21

^&Based on official estimates posted in the health facility wall-charts provided by the Sierra Leone Ministry of Health and Sanitation.

^*Based on information provided by interviewed health staff or facility-in-charge regarding health workers who were assigned to EPI.

^#Full-time health staff using VaxTrac at the time of the assessment (including health workers that may not have been dedicated to EPI alone).

^{^}2 health workers in the health facility who were not part of the EPI staff were also reportedly using VaxTrac when they help with EPI.

^~This facility did not have dedicated EPI staff and all 12 health workers in the facility rotated with EPI duties.

EPI = Expanded Program on Immunization; CHC = community health center.

Under-5 = Number of children under the age of five years living in the catchment area.

Under-2 = Number of children under the age of 1 year living in the catchment area.

Fig. 3 shows the distribution of cumulative vaccine doses administered at the eight health facilities that had all four data sources available for the six scheduled vaccine doses assessed for a one-month period. Three notable patterns emerged from the data congruity analysis. First, the aggregated data sources (HF2 and tally sheets) reported higher doses administered compared to the two child-based registers (under-two register and VaxTrac). Second, VaxTrac reported fewer doses administered compared to all paper-based sources. Third, all the data sources that need to be completed in real-time during the vaccination session (tally sheet, under-two register, and VaxTrac) reported fewer doses administered compared to the HF2 summary form that is aggregated manually at the end of the month. Tally sheets captured 2,936 doses, under-two registers captured 2,368 doses, HF2 summary forms captured 3,209 doses and VaxTrac captured 2,198 doses. It should be ODC Hospital made up substantial proportions of the cumulative doses captured (38% of the tally sheet data, 39% of the under-two register data, and 59% of VaxTrac data) (supplementary material). For this reason, data congruity is shown separately for ODC Hospital (Fig. 4) and for the other smaller seven health facilities (Fig. 5).

Fig. 3.

Distribution of cumulative vaccine doses administered in eight health facilities with data for all four sources, VaxTrac assessment, Western Urban, Sierra Leone, 2017.

Fig. 4.

Congruity between data captured in VaxTrac compared to three paper-based data sources for cumulative vaccine doses administered at Ola During Hospital, VaxTrac assessment, Western Urban District, Sierra Leone, 2017.

Fig. 5.

Congruity between data captured in VaxTrac compared to three paper-based data sources for cumulative vaccine doses administered at seven health facilities excluding Ola During Children’s Hospital, VaxTrac assessment, Western Urban, Sierra Leone, 2017.

At ODC Hospital, VaxTrac captured the highest number of cumulative vaccine doses administered (n = 1,306) compared to the tally sheet (n = 1,125; 86% of VaxTrac doses captured), HF2 (n = 1,121; 86% of VaxTrac doses captured), and under-two register (n = 925; 71% of VaxTrac doses captured), (Fig. 4). However, in the seven smaller health facilities, VaxTrac captured substantially fewer vaccine doses: 43% compared to the monthly summary form, 49% compared to the tally sheet, and 62% compared to the under-two register (Fig. 5).

Qualitative themes and direct observations from the semi-structured interviews with health staff are outlined in Table 2 to identify successes and challenges regarding VaxTrac’s user training, user experiences and functionality of the system, data sharing, workflows, and integration of community health workers. A summary of the implementation successes and challenges from all the ten health facilities are highlighted below.

Table 2.

Themes demonstrating successes and challenges related to VaxTrac implementation based on interviews and direct observations, VaxTrac assessment, Western Urban, Sierra Leone, 2017.

Thematic Area	Successes	Challenges
User training	Ample opportunities to practice using the key functions of VaxTrac.	Nine of the ten health staff did not know how to identify children due for an upcoming vaccine even though the system had this feature.
User experience & functionality	Health workers were able to retrieve and update vaccination records, generate lists of defaulters, and create monthly summaries vaccines administered*. Efficient use of QR barcodes as unique identifiers to manage vaccination records. VaxTrac was helpful in quickly summarizing cumulative vaccine doses in the monthly reporting form. All tablets had Internet connectivity and were sufficiently charged*.	Malfunctioning of finger printing devices caused delays during vaccination sessions. Health workers desired to but did not know how to undertake troubleshooting for simple recurring issues with the tablets. Only one of the ten health facilities had a functional biometric fingerprinting device*. Only two facilities reported experiencing challenges with electricity supply; only one facility did not have electricity*.
Data sharing	Enabled within groups of 7 to 12 health facilities based on geographic proximity. Creation of limited data sharing group helped to speed up data synchronization between offline tablets and online data hosting servers.	When a child moved to a different locality outside of the original data sharing group, his/her record could not be accessed. Data sharing restrictions led to creation of duplicate records in VaxTrac using a new QR barcode.
Workflows	ODCH had an efficient workflow that integrated VaxTrac into existing process and with clear roles and responsibilities*.	Inconsistent and inefficient workflows in all other nine health facilities observed*.
Integration of community health workers	All health facilities reported having ten CHWs assigned to their catchment areas. CHWs may have identified children who defaulted and refer them to health facilities for catchup vaccination doses without using VaxTrac data.	None of the health facilities visited were routinely tracking defaulters even prior to the introduction of VaxTrac despite the MoHS recommendation to do so four times monthly. Although trained staff knew how to generate defaulter lists in VaxTrac, the lists were never shared with CHWs to track defaulted children.

^*Direct observations at the time of the health facility visits

QR = Quick response.

ODCH = Ola During Children’s Hospital.

MoHS = Sierra Leone Ministry of Health and Sanitation.

CHWs = Community Health Workers.

3.1. Implementation successes

Regarding the training by eHealth Africa, all health staff said that they were given ample opportunities to practice creating new registrations, updating existing vaccination records, generating monthly summaries of doses administered, and identifying children who have defaulted within their geographic catchment areas. They expressed that VaxTrac helped them to quickly summarize their monthly vaccine doses. We directly observed that all health workers interviewed knew how use the system to identify children who have defaulted for scheduled vaccine doses (Table 2).

3.2. Implementation challenges

The malfunction of fingerprinting devices required health workers to reboot the tablet frequently and some data had to be re-entered. There were instances where caregivers reportedly left vaccination sessions before their children were vaccinated due to the prolonged waiting time caused by the malfunctioned fingerprinting devices. All facilities included in the assessment reported no longer using the fingerprinting devices and having transitioned to only using QR barcodes affixed onto the Child Health Card given to caregivers. Moreover, when a child moved to a different locality outside of the original cluster, the child’s record could not be accessed using the QR code from the previously registered health facility due to the limited geographic scope of data sharing system in place. Health workers at the new location had to create a duplicate VaxTrac record for the child. According to the eHealth Africa team managing the system, removing the data sharing restriction would have required synchronization of a larger volume of data across all facilities which may have faced challenges due to the system’s capacity. Another crosscutting challenge was the lack of consistent workflow set up to facilitate complete data capture across all the data sources. All health workers we observed did not know how to use VaxTrac to identify children who will be due for an upcoming dose (not yet defaulted), even though the system had this capability. Lastly, although trained staff knew how to use VaxTrac to generate the lists children who have missed one or more scheduled vaccine doses (defaulted), no well-planned efforts were made to ensure the use of the lists by Community Health Workers (CHWs) in tracking children who defaulted (Table 2).

4. Discussion

Simplicity, flexibility, data quality, and user acceptability are important attributes of an effective EIR [16]. Applying our results to these system-wide attributes paints a mixed review of VaxTrac’s implementation in Sierra Leone. Health workers found VaxTrac to be simple-to-use. Compared to paper-based methods, health workers reported that VaxTrac allowed them to more quickly retrieve vaccination records, determine the child’s next due date for vaccination, and summarize monthly doses administered. We observed that health workers were able to use most of the key functions of the application except for identifying children who are due for an upcoming dose. Outside of the large pediatric hospital, our results showed that VaxTrac only captured about two-thirds of the cumulative vaccine doses in paper-based records. While this finding strongly pointed to incongruences between VaxTrac and paper-based sources, our findings also demonstrated gaps in data completeness across paper-based sources as well. However, it was difficult for us to ascertain which data source should be considered gold standard based on data completeness, partly because we could not account for potential duplicates and we did not have access to home-based records kept by the caregivers. It was unclear how VaxTrac’s introduction may have impacted data quality in the paper-based records given the burden of multiple entries and lack of a coherent workflow process. Cumulative vaccine doses captured in VaxTrac were less congruent with those in the under-two register compared to tally sheets or HF2 monthly summaries, suggesting at least in part that the step of transcribing data from the under-two register (the recommended initial step of data entry) to VaxTrac was not always occurring.

Based on our assessment, child-based registries such as VaxTrac and the under-two register may have faced challenges with completeness because they required more time to update individual child-based records compared to the tally sheet or HF2 summary form. Tally sheets may have provided inflated data due to fear of reporting high vaccine wastage among health workers. Genuine human errors may have also occurred wherein health workers forgot to update the child-based registers to reflect inputs made on the tally sheets. HF2 data may have been more aligned with tally sheet records because health facilities in Sierra Leone normally use the aggregated vaccine doses from the tally sheet to do their monthly summary reporting in the HF2. Compared to single dose vaccines (e.g. BCG), the counts of doses administered for multiple-dose vaccines (e.g. pentavalent) may have been inaccurate for the respective doses but added to a more accurate cumulative count across all vaccines in the series. Needing to do entries in multiple systems and the lack of consistent workflows to guide the process may also partly explain some of the data quality gaps we found. Additionally, the malfunctioning of fingerprinting device in the system’s initial introduction could have resulted in less use of the VaxTrac system – especially if health workers primarily used paper-based records for their required monthly reporting.

Incompleteness of data and the lack of procedures to address duplicate records posed serious limitations to VaxTrac’s potential in generating real-time estimates of vaccine uptake. Data sharing restrictions may have led to incomplete or duplicate records in the system. According to the technicians managing VaxTrac, the lack of an active community of software developers supporting the VaxTrac platform posed an inherent threat to its sustainability. The ability for countries to lead the development of and subsequent updates to the EIR software over time (e.g. when new vaccines are introduced into the schedule) has been documented as a critical component of sustaining EIR in other settings. Modifications to the EIR software in Honduras were initially done by the software developer but the role was later assumed by the country’s MoH Information Management Unit [19]. In Panama, software adjustments were coordinated between the national Department of Statistics and the national immunization program [19]. In Sierra Leone, VaxTrac was implemented as a pilot project to understand its feasibility in an urban setting. While the Sierra Leone MoHS was involved in the process of designing and implementing the project, technical aspects of the software configuration and data hosting/synchronization were handled by eHealth Africa.

Several EIRs have been implemented with varying levels of success in high-, middle-, and low-income countries [17,18,20–24]. In China’s Jingsu Province, data sharing across all health facilities was an important element in improving data quality and generating real-time vaccination coverage [20]. Experiences from Latin America prominently highlight the need to standardize processes and procedures for data flows, mechanisms for resolving duplicate records, timely synchronization between offline data capture devices and centralized data hosting servers, flexibility in accommodating changes such as new vaccine introductions, ongoing training of health staff, and supportive supervision [19]. In Zambia and Tanzania, successful EIR introductions were proceeded by the establishment of national advisory groups that carried out iterative assessments to identify functional and system requirements using the Collaborative Requirements Development Methodology [24]. In both countries, it took over three years to identify, test, deploy, and refine the EIRs to meet the agreed upon requirements. For example, in Zambia, when the initial EIR software no longer met the functional requirements, the country switched over to a new EIR software to address the shortfalls. A key lesson from Zambia points to the need for flexibility and adaptation in the evolution and optimization of an EIR. The combined experiences from countries in Latin America and sub-Saharan Africa emphasize that EIR introductions should not be conceived as one-off projects; proper planning that account for long-term sustainability and integration into broader health information ecosystems are important for success [19,24].

In Sierra Leone, VaxTrac was initially piloted in just one large pediatric referral facility in Western Area Urban district. Given the success of using the system at that referral facility, the MoHS requested for an expansion of the system to 49 additional health facilities in the district. Our assessment was conducted after one year of continuous piloting the system in the 50 health facilities. Preliminary findings from our assessment were shared with in-country partners at the 2018 Annual Joint Appraisal Meeting. Given the totality of the findings from our assessment coupled with sustainability concerns and other competing priorities in the resource-constrained immunization program in Sierra Leone, the MoHS decided to suspend the use of VaxTrac in 2019. It is unclear if the MoHS will ever resume its implementation or that of another EIR. Nevertheless, our findings have shed light on important issues that will need to be addressed in any future implementation of an EIR in Sierra Leone. Cross-cutting findings from Sierra Leone may also hold important lessons for EIR implementations in other low-income countries.

4.1. Limitations

In each sampled health facility, we only abstracted one month of data. We attempted to address this limitation by abstracting two different months for the first and second rounds of data collection. However, the months captured in our analysis (September and October 2017) only provided a snapshot that may not be representative of all months. Given the current data captured in our analysis, we could not account for potential duplicates in VaxTrac or inflation of aggregated doses in paper-based records, which makes it difficult to interpret the data congruency results. Our assessment initially aimed to glean rapid insights to inform a more comprehensive evaluation of VaxTrac in the country. However, in 2019 after our preliminary results were shared with the country, the Sierra Leone MoHS decided to suspend the system due to sustainability concerns. Therefore, a follow-up comprehensive evaluation was never undertaken. To the best of our knowledge, the current assessment is the first to evaluate the implementation of VaxTrac in any country. In addition to identifying data incongruencies, the qualitative results we have presented provide an enhanced understanding of the range of potential barriers when considering the introduction of VaxTrac or other similar EIRs in low resource contexts.

4.2. Conclusion

Data quality improvement should be an ongoing effort when implementing an EIR in any setting [17,18]. In this rapid assessment, we identified immunization data quality issues and barriers in the implementation of VaxTrac. Gaps in data completeness were linked to underlying systemic challenges among health workers and inconsistent workflows for managing vaccination records, especially in smaller health facilities. In addition, the failure to leverage existing CHWs to use VaxTrac data for defaulter-tracking was a missed opportunity. Despite these challenges, health workers perceived VaxTrac to be user-friendly and helpful in managing vaccination records. Countries looking to introduce an EIR should first consider conducting a readiness assessment to determine feasibility, minimum system requirements, and sustainability of the electronic system in the context of existing processes, workloads, and workflows in health facilities. EIR implementers should proactively identify emerging and recurring barriers while continuously exploring mitigation strategies to improve data quality and systemwide performance throughout the EIR lifecycle. Rapid assessments coupled with rigorous monitoring and evaluation should be undertaken as part of any EIR implementation to inform longer term decision-making and policy.