Limitations of Using Administratively Reported Immunization Data for Monitoring Routine Immunization System Performance in Nigeria

Abstract

Background.

Efforts are underway to strengthen Nigeria’s routine immunization system, yet measuring impact poses a challenge. We document limitations in using administrative data from 12 states in Nigeria and explore alternative approaches.

Methods.

We compared state-reported coverage with the third dose of diphtheria-tetanus-pertussis vaccine (DTP3) to district-reported coverage and data from coverage surveys conducted during 2006–2013. We used district-reported data during 2010–2013 to calculate the annual change in immunization coverage, the percentage of the target population that was unimmunized, and the number of vaccine doses administered. Data quality indicators were also assessed.

Results.

State-reported DTP3 coverage was 66%–102% in 2010, 49%–98% in 2011, 38%–84% in 2012, and 75%–123% in 2013 and was a median 46%–114% greater than survey coverage during 2006–2013. The mean local government area (LGA)–reported coverage varied substantially (standard deviation range, 10%–33% across years). For 2010–2013, the mean annual percentage change in LGA-reported DTP3 coverage was −15% from 2010 to 2011, −9% from 2011 to 2012, and 74% from 2012 to 2013; the mean annual percentage change in the percentage of the target population unimmunized was −62%, 426%, and −62%, respectively; and the mean annual percentage change in the number of doses administered was −13%, −7%, and 90%, respectively. Annually, a mean 14% of LGAs reported DTP3 coverage of >100%.

Discussion.

Assessing immunization system performance by using administrative data has notable limitations. In addition to long-term improvements in administrative data management, alternatives for measuring routine immunization performance should be considered.

The Global Polio Eradication Initiative (GPEI) began in 1988 with the key eradication strategy of reaching all children with multiple doses of polio vaccine through routine and supplementary immunization activities. By 2012, polio remained endemic in only 3 countries, including Nigeria. Since the declaration of polio as a public health emergency by the World Health Assembly in May 2012, international efforts have increased to assist Nigeria in strengthening its immunization system [1, 2].

Historically, the immunization system in Nigeria has struggled to achieve the high (>80%) routine immunization coverage that is needed to reduce or eliminate the incidence of vaccine-preventable diseases, including polio. Recent immunization coverage surveys indicate particularly low routine immunization coverage, as measured by data on receipt of the third dose of diphtheria-tetanus-pertussis vaccine (DTP3), in Nigeria’s northern states [3–7]. In 2012, Nigeria had an estimated 3.8 million children unimmunized with DTP3, the second highest population of unimmunized children of any country in the world [7].

Strengthening routine immunization services is emphasized as a way to help eradicate polio, to ensure GPEI investments in the immunization system are sustained, and to allow for the successful introduction of inactivated poliovirus vaccine [8]. The Nigerian government’s National Primary Health Care Development Agency (NPHCDA) and development partners have demonstrated a renewed interest in strengthening routine immunization in Nigeria, with substantial time and resources being targeted in an effort to improve coverage and program accountability [9, 10].

Indicators are needed to track improvements in routine immunization coverage and to monitor accountability at all levels. Page 51 of the GPEI Plan 2013–2018 includes a global indicator of “at least 10% annual increase in DTP3 coverage achieved in 80% of high-risk districts in all focus countries from 2014 to 2018” [8]. However, historically, countries have rarely achieved a 10% annual increase in DTP3 coverage at a national level [11]. Tracking annual changes in immunization coverage at a district level will entail the use of health-system-reported immunization data (commonly known as administrative data) or implementation of yearly district-level immunization coverage surveys; both approaches have distinct advantages and disadvantages [12–15]. Coverage surveys, despite their limitations, generally provide reliable coverage estimates [16, 17]; however, they are resource intensive, require special training to administer, and are not intended to provide real-time coverage. Administratively sourced coverage provides nearly real-time information and allows for monitoring of all administrative layers of the health system, but it depends on a reliable data-reporting system, which still remains a goal in Nigeria [12]. Administrative coverage calculations also require an estimate of the size of the target population (infants <1 year of age), which may be inaccurate because of population movement and outdated census information. Documenting the challenges in using administrative data to measure indicators can be a useful exercise in identifying whether and how to further invest in these 2 approaches for monitoring coverage improvement.

To evaluate the strengths and weaknesses of using various administratively reported indicators at the local government area (LGA)–level (the Nigerian administrative equivalent of the district level), we used administrative data to assess (1) the relative annual change in DTP3 and measles immunization coverage, (2) the relative annual change in the percentage of the target population unimmunized against DTP3, and (3) the relative annual change in DTP3 doses administered. Additionally, we documented key data quality issues encountered in the analysis during 2010–2013 and compared state-level DTP3 coverage that was calculated using administrative data to results obtained from representative immunization coverage surveys during 2006–2013.

METHODS

Administrative Data Reporting System

Administrative routine immunization data in Nigeria are collected via monthly reports sent by health facility staff to the LGA immunization officer. The health facility reports are compiled into LGA summaries and sent to state government and World Health Organization staff, who enter the data into the district vaccine data-management tool (DVDMT) database. The state data are conveyed via zonal offices to the national level, where the DVDMT database is made accessible to NPHCDA and partner agency staff.

Data Sources and Definitions

For 2006–2009, state-level administrative data were available; for 2010–2013, both state and LGA-level administrative data were available. We extracted monthly data on doses of DTP3 and measles vaccine administered and annual data on target populations in 12 states at high risk for polio (Bauchi, Borno, Gombe, Jigawa, Kaduna, Kano, Katsina, Kebbi, Niger, Sokoto, Yobe, and Zamfara) [18] for 2006–2009 and from each of the LGAs within these states for 2010–2013. For 2006–2008, state estimates for the total population of infants <1 year of age were available. For 2009–2013, the numbers of live births and surviving infants were available. Live births were estimated as 4% of the total population of a state or LGA, while the figure used by NPHCDA to calculate surviving infants at state and LGA levels varied each year, as follows: 3.6% for 2009, 3.5% for 2010–2012, and 3.7% for 2013. For 2006–2008, we used 3.6% of the total population (or 90% of the total population of infants) to calculate the number of surviving infants. We used the number of surviving infants as the target population for DTP3 and measles immunizations in our analyses.

Immunization coverage survey results were obtained from Demographic and Health Surveys (DHS) and from Multiple Indicator Cluster Surveys (MICS) conducted in Nigeria during 2006–2013 [3–6].

Data Analysis

We obtained the total annual doses of DTP3 and measles immunizations administered in a state or LGA by summing the monthly doses administered during 12 calendar months. We calculated DTP3 and measles immunization coverage by dividing the annual number of doses administered by the annual size of the target population, defined as the number of surviving infants in a state or LGA. We assessed the percentage of the target population unimmunized with DTP3 by subtracting the calculated DTP3 coverage from 100%.

We compared state-level administrative coverage to results from immunization coverage surveys conducted during 2006–2013. Since results from coverage surveys were collected from children aged 12–23 months, we compared survey results with administrative reports from a year prior to the year of survey data collection.

We calculated the relative annual percentage change in DTP3 and measles coverage by subtracting the DTP3 or measles coverage in the prior year from a given year’s DTP3 or measles coverage and then dividing the result by the annual coverage percentage from the prior year. We used the same calculation for the relative annual percentage change in the number of individuals in the target population who were unimmunized with DTP3 and the relative annual percentage change in the number of doses of DTP3 administered.

We assessed data quality by calculating the percentage of months during 2010–2013 in which DTP3 doses administered were not reported by an LGA (incompleteness), months in which 0 doses of DTP3 administered were reported by an LGA (0-dose months), and by calculating the percentage of months in which LGA administrative coverage exceeded 100%. Last, we assessed the relative annual change in the LGA-level target population to identify any notable inconsistencies (defined as a change of >5%) during 2010–2013.

RESULTS

Administrative Coverage

Administratively reported DTP3 coverage for the 12 states at high risk for polio varied between 66% and 102% in 2010, between 49% and 98% in 2011, between 38% and 84% in 2012, and between 75% and 123% in 2013 (Figure 1). Mean annual LGA-reported DTP3 coverage was similar to state-reported DTP3 coverage (Table 1); however, the standard deviation for the mean annual LGA-reported coverage indicated a substantial variation in coverage across LGAs within each state, ranging from 10% to 33% across the 12 states over the 4-year period. On average, in most states, the DTP3 coverage declined from 2010 to 2012 and improved in 2013.

Figure 1.

State and local government area (LGA)–level coverage with the third dose of diphtheria-tetanus-pertussis (DTP3) vaccine per administrative data sources and immunization coverage surveys in 12 Nigerian states at high risk for polio, 2006–2013. One LGA in Borno state reported 244% DTP3 coverage in 2013 (data not shown). Abbreviations: Admin, administrative (reported) data; DHS, Demographic and Health Surveys; MICS, Multiple Indicator Cluster Survey.

Table 1.

State and Mean Local Government Area (LGA)–Level Immunization Coverage Per Administrative Data Sources in 12 Nigerian States at High Risk for Polio, 2010–2013

		2010		2011		2012		2013
State	LGAs, No.	State Coverage, %	LGA Coverage, %, Mean ± SD	State Coverage, %	LGA Coverage, %, Mean ± SD	State Coverage, %	LGA Coverage, %, Mean ± SD	State Coverage, %	LGA Coverage, %, Mean ± SD
Bauchi	20	74	75 ± 19	51	52 ± 12	49	49 ± 18	97	77 ± 22
Borno	27	69	67 ± 22	63	63 ± 24	71	71 ± 31	75	76 ± 59
Gombe	11	72	70 ± 11	73	70 ± 13	84	83 ± 16	108	94 ± 13
Jigawa	27	87	90 ± 19	81	83 ± 22	51	54 ± 24	123	109 ± 32
Kaduna	23	80	82 ± 25	56	55 ± 10	39	38 ± 13	101	81 ± 19
Kano	44	66	65 ± 20	49	49 ± 18	44	44 ± 20	82	69 ± 21
Katsina	34	70	72 ± 25	68	69 ± 18	67	67 ± 16	123	105 ± 15
Kebbi	21	75	75 ± 17	51	51 ± 13	38	39 ± 13	110	87 ± 28
Niger	25	90	88 ± 17	67	67 ± 14	48	49 ± 18	99	84 ± 17
Sokoto	23	102	105 ± 18	71	72 ± 14	70	71 ± 16	113	102 ± 15
Yobe	17	78	76 ± 17	64	64 ± 17	58	58 ± 15	95	89 ± 16
Zamfara	14	84	85 ± 18	98	100 ± 33	78	80 ± 20	100	91 ± 25

Reported state-level DTP3 coverage for the 12 states at high risk for polio was always higher than corresponding state-level DTP3 coverage estimates from the MICS or DHS during 2006–2013. Differences varied by year, with the greatest median absolute difference occurring in 2006 (114%; range, 64%–133%) and the lowest median absolute difference occurring in 2012 (46%; range, −4%–72%; Figure 1).

Figure 2 illustrates the annual change in DTP3 coverage in the 286 LGAs in the 12 states at high risk for polio during 2010–2013. Similar to the trend observed in state-level DTP3 coverage, there was an overall decline in LGA-level DTP3 coverage during 2010–2012, followed by an increase during 2013: only 61 LGAs (21%) in 2011 and 100 LGAs (35%) in 2012 reported an increase in annual coverage, while 256 LGAs (90%) reported an increase in annual coverage in 2013.

Figure 2.

Administratively reported coverage with the third dose of diphtheria-tetanus-pertussis (DTP3) vaccine in local government areas (LGAs) in 12 Nigerian states at high risk for polio, 2010–2013 (n = 286). The solid line represents 0 change in annual coverage, while the dashed line represents the indicator for a 10% relative annual increase in coverage. One LGA in Borno state reported a DTP3 coverage of 80% in 2012 and 244% in 2013 (data not shown).

Indicator Achievement

Of the 286 LGAs in the 12 states at high risk for polio, 40 (14%) in 2011, 75 (26%) in 2012, and 240 (83%) in 2013 met the indicator for a 10% relative annual increase in DTP3 coverage (Figure 2 and Figure 3A). The mean annual percentage change (±SD) in DTP3 coverage was −15% ± 27% in 2011, −9% ± 33% in 2012, and 74% ± 71% in 2013.

Figure 3.

Distribution of local government areas (LGAs) by relative annual change in coverage with the third dose of diphtheria-tetanus-pertussis (DTP3) vaccine (A), coverage with measles vaccine (B), percentage of the target population unimmunized with DTP3 (C), and DTP3 doses administered (D), calculated using administrative data collected in 12 Nigerian states at high risk for polio, 2010–2013. N = 286 LGAs represented.

In 2011, 121 LGAs (42%) met the indicator for a 10% relative annual increase in measles coverage, while in 2012 and 2013, the indicator was met by 73 (26%) and 205 (72%), respectively (Figure 3B). The mean annual percentage change (±SD) in measles coverage at the LGA level was 38% ± 114% in 2011, −15% ± 44% in 2012, and 64% ± 88% in 2013. Of the 286 LGAs, 89 (31%) in 2011, 92 (31%) in 2012, and 245 (86%) in 2013 exceeded the indicator of a 10% relative decline in the percentage of individuals unimmunized against DTP3 (Figure 3C). The mean annual percentage change in the percentage (±SD) of unvaccinated individuals was −62% ± 1067% in 2011, 426% ± 7286% in 2012, and −62% ± 254%) in 2013. In 2011, 42 LGAs (15%) had at least a 10% increase in the number of doses of DTP3 administered, while 82 (29%) met the indicator in 2012, and 254 (89%) met the indicator in 2013 (Figure 3D). The mean annual percentage change (±SD) in the number of doses of DTP3 administered was −13% ± 28% in 2011, −7% ± 33% in 2012, and 90% ± 77% in 2013.

Data Quality Assessment

The 286 LGAs had the opportunity to report monthly doses of DTP3 administered 3432 times annually during 2010–2013. Incomplete reporting of DTP3 doses administered was infrequent: 13 (0.4%) in 2010, 1 (0.0%) each in 2011 and 2012, and none in 2013. In contrast, 0-dose months were more frequent: 185 (5.4%) in 2010, 484 (14.1%) in 2011, 647 (18.9%) in 2012, and 215 (6.3%) in 2013. The number (and percentage) of LGAs reporting >100% DTP3 coverage was 49 (17.1%) in 2010, decreased to 13 (4.5%) in 2011 and 10 (3.5%) in 2012, and increased to 92 (32.2%) in 2013.

The relative annual change in total population estimates, used as a basis for calculating the target population during 2010–2013, varied by LGA. The median relative annual change in total population during 2010–2013 was 3.2% for each year during 2010–2013, but the range of annual change varied each year, with the widest range occurring during 2011 (−37% to 109%), and the smallest range occurring in 2013 (2.9%–3.5%). The percentage used by NPHCDA to calculate surviving infants changed from 3.5% of the total LGA population during 2010–2012 to 3.7% of the total population in 2013. Consequently, the target population increased by the same median and range as the total population during 2011–2012 but increased by a median of 9.1% (range, 8.8%–9.4%) in 2013. Subsequently, the number of LGAs that had a >5% relative annual change in the target population was 7 (2.5%) in 2011, 5 (1.8%) in 2012, and 286 (100%) in 2013.

DISCUSSION

Health system reports of vaccine doses administered are generally the only systematic and routinely available sources of information on routine immunization outcomes at the lowest administrative levels and consequently hold the potential to provide crucial data on performance gaps to drive decision making. After variable levels of coverage during 2006–2009, reported DTP3 coverage for the 12 states at high risk for polio in northern Nigeria declined during 2010–2012, followed by an increase in coverage in 2013 to approximately 2010 levels. Annual rates of LGA-reported coverage varied substantially from year to year, resulting in marked volatility in the direction and magnitude of the relative annual change in DTP3 coverage at this level. Comparisons between administratively reported state-level immunization coverage and coverage estimates from surveys indicate that the data-reporting system substantially overestimates immunization coverage. In comparing the value of several indicators for assessing immunization system performance (the relative annual change in coverage, the percentage of the target population that was unimmunized, and number of doses administered), our analysis revealed limitations when used within the constraints of an unreliable reporting system.

During 2010–2013, multiple events may have influenced routine immunization coverage. The release of routine immunization program funding was delayed in 2011, which affected vaccine purchasing and resulted in stockouts of measles and DTP vaccine throughout the year. In 2012, a national stockout of DTP-containing vaccine occurred before the introduction of pentavalent (diphtheria, tetanus, pertussis, hepatitis B, and Haemophilus influenzae type b) vaccine. In 2013, concurrent with the reported increase in DTP3 coverage back to 2010 levels in the 12 northern states, pentavalent vaccine introduction activities were ongoing (including wide-scale healthcare worker training and improvements in vaccine cold chain and distribution throughout the country). Additionally, multiple government agencies and development partners initiated a renewed investment in strengthening routine immunization as part of Nigeria’s emergency plan to eradicate polio.

Although the reported state-level coverage trends may have adequately captured these macro level events, the accuracy of the reported coverage appears poor when compared to estimates from coverage surveys and may have been affected by multiple challenges identified in the course of our analysis. We have demonstrated that, while the mean LGA-level DTP3 coverage is similar to the reported state-level coverage, substantial variability in coverage exists across the LGAs within a state. This volatility might be explained by uncertainties related to target population estimates. While a fixed annual growth factor was used to calculate the total population in most LGAs, many increased or decreased their target population by >5% in a single calendar year. Since the last national census in Nigeria occurred in 2006 [19], it is unclear whether these local changes are based on population movement or shifting geopolitical boundaries, perhaps as a result of insecurity in the region. While these adjustments may be a positive improvement in the accuracy of the target population for planning at the local level, the large proportion of LGAs reporting >100% coverage indicates a general problem with target population accuracy. We also observed a substantial number of monthly 0-dose reports of DTP3 administration in an LGA; uncertainty exists as to whether these truly reflect that no vaccines were administered or whether they were in fact incomplete reports potentially impacting the accuracy of the annual coverage estimate for the LGA. All of these data-related challenges limit our ability to use reported data to calculate indicators and make informed programmatic decisions.

Despite these limitations, administrative data are currently the most readily available source of data for monitoring routine immunization system performance in close to real time at the district level. We assessed 3 indicators which could be used for assessing changes in immunization system performance by using reported immunization data and identified both advantages and disadvantages of each. First, we assessed the proportion of LGAs in 12 states at high risk for polio in northern Nigeria that had at least a 10% relative annual increase in coverage during 2010–2013; few LGAs met the indicator in 2011 and 2012, but >80% of LGAs achieved the goal in 2013. External factors could have influenced this achievement: as previously mentioned, the decline in coverage during 2010–2012, followed by the increased focus and resources invested in routine immunization in 2012–2013, set the region up for a marked improvement in coverage in 2013. A similar trend was observed using measles coverage data during the same period. Use of the relative annual change in reported coverage as an indicator of performance is beneficial because it is familiar and easily understood by healthcare staff and because it readily shows the percentage of immunized and unimmunized individuals within a defined target population. However, relative changes in coverage may be easier to achieve at a low baseline level of coverage than at levels approaching the ceiling of 100% coverage [11]. For instance, a 10% relative annual increase in DTP3 coverage for a district at 20% baseline coverage would only require a 2% improvement, while a similar relative increase for a district at 80% baseline coverage requires an 8% improvement. Consequently, it is possible that resources may end up being focused on districts with high coverage rather than on districts with low coverage, where there is greater need, in an effort to meet the indicator.

The 2 other performance indicators that were assessed, a 10% relative annual decrease in the proportion of the target population unimmunized and a 10% relative annual increase in the number of doses administered, also have advantages and disadvantages. Use of the relative change in the proportion of individuals who are unimmunized as an indicator could increase in usefulness as coverage reaches the 100% ceiling, since the ability to achieve this goal would be more realistic than, for instance, a country at 90% coverage achieving a 9-percentage-point change if the indicator goal were a 10% relative annual change in coverage. However, similarly to coverage, calculation of the proportion unimmunized is dependent on a reliable estimate of the target population. With a large proportion of LGAs having coverage of >100% during 2010–2013, it is not logical to expect a 10% relative increase to a coverage estimate that is already >100%, or a 10% relative decline in a negative percentage of unimmunized individuals. The use of an indicator measuring relative annual change in the number of doses administered has the benefit of removing target population from the calculation; however, drawbacks include the lack of an upper threshold with which to measure against to ensure all children are adequately immunized and that expected population growth is not taken into account. Use of the drop-out rate between the first and third doses of DTP administration has also been suggested as an indicator; again, this would avoid the effect of an inaccurate target population, but this indicator has the same limitation that there is no indication of the system performance relative to the overall goal of reaching every child.

Considering the constraints identified with using health-system-reported data in Nigeria, we conclude that consideration be given to multiple options for obtaining dependable estimates of program performance to monitor current investments to eradicate polio and improve routine immunization. Representative coverage surveys at the LGA level may provide more-reliable figures, but they are resource intensive, have a significant time lag, and may suffer in quality if conducted at such a large scale; state-level surveys are an alternative but do not provide the granularity needed to assess district-level performance. A select number of high-risk LGAs could act as sentinel sites, with annual coverage surveys conducted to provide more-reliable figures and also to obtain data for comparison with reported coverage rates. Incorporating the monitoring of process indicators may also be beneficial and identify local reasons for poor performance [20]. Example process indicators include frequency of vaccine stockouts, frequency of immunization sessions, release of funding for immunization activities, frequency and quality of supervisory visits, and frequency and quality of social mobilization activities.

In 2013, immunization system partners in Nigeria began conducting an assessment of data-management practices with the final objective of rolling out an enhanced routine immunization data module as part of the National Health Management Information System. The information system improvement process includes extensive health worker training on data reporting, increased supervisory visits focused on early problem detection and intervention, and data entry at a lower administrative level (the LGA health office with health-facility-level granularity) into a cloud-based data-collection system. The process also incorporates a new system of annual health facility routine immunization microplan updates to improve target population estimates and encourages more frequent data use by managers at all administrative levels via a user-friendly, Internet-based dashboard of coverage and monitoring indicators. This dashboard raises the profile of these indicators, which will ideally result in timely identification and solutions to data-quality problems. Postimplementation reassessment of the indicators we have presented may help identify whether the reliability of reported immunization data has improved.

Considering trends in reported LGA-level coverage during 2010–2013, a consistent >10% relative annual increase in DTP3 coverage in LGAs in the 12 states at high risk for polio during 2014–2018 may be difficult to reliably measure using reported coverage. Similarly, use of alternative indicators for measuring whether these LGAs have achieved the objectives laid out in the GPEI Plan 2013–2018 have substantial limitations. Alternative short-term methods for measuring improvements in the routine immunization program, including sentinel LGA- or state-level coverage surveys and tracking of system process indicators, may be needed. In the long-term, investments in a well-functioning health information management system with reliable target population estimates is required to track routine immunization system performance at the LGA level.