Auditing the quality of immunization data in Tunisia

Abstract

Objective

To carry out a data quality audit for the consistency and quality of immunization monitoring in Kasserine governorate in Tunisia, where immunization coverage rates more than 100%, negative dropout and conflicting coverage rates had been reported.

Methods

In a randomly selected sample of 12 primary health care facilities in 2 districts all files used to record immunization data were checked to verify the accuracy of the reported 3rd dose of diphtheria-tetanus-pertussis vaccine during a calendar year.

Results

The results found frequent over-reporting or under-reporting of doses being administrated in all health facilities. The regional verification factor was estimated as 85% and quality index was 55%.

Conclusions

Data quality audit provides a effective tool for self-assessment implementation and developing a critical analysis of the quality of routine immunization data at all levels of health service.

1. Introduction

Monitoring of immunization coverage is one of the most important components of the expanded programme on immunization (EPI). Estimates of coverage are generally based on 2 sources of empirical data: reports of immunizations performed by service providers (administrative data); and household surveys of children's immunization history (coverage surveys). For estimates based on administrative data, the immunization coverage is derived by dividing the total number of vaccinations given by the number of children in the target population. Estimates based on administrative data have the advantage of giving an annual and regular estimate which is more convenient than estimates based on surveys every 5 years. Unfortunately, administrative data collected through routine immunization reporting systems have been criticized for their inaccuracy.

In Tunisia, the immunization coverage estimate from round 3 of the United Nations Children's Fund (UNICEF) Multiple Indicator Cluster Surveys (MICS3)[1], conducted in 2007 was different from the estimate based on administrative data for the same year[2]. For example, the coverage rate for the 3rd dose of diphtheria-tetanus-pertussis vaccine (DTP3) was 91.3% based on MICS2 data and 115.7% based on administrative data. According to the MICS3 results, Kasserine governorate had the poorest indicators, with the routine immunization reporting system suffering weaknesses such as a coverage rates higher than 100%, negative drop-out or inconsistent coverage between antigens given during the same visit[1].

This paper reports a data quality audit (DQA) performed in Kasserine governorate in order to improve the quality of immunization data, to familiarize regional and local health teams with a new tool for data quality assessment, to develop a critical analysis of the quality of immunization data reported by the administrative route and to be able to plan a similar exercise for other health programmes.

2. Materials and methods

2.1. Description of the routine immunization reporting system

In Tunisia, the immunization data concerning children and mothers are collected within an integrated child health reporting system that was launched in 1998[3],[4]. The flow of information starts in the public primary health care (PHC) facilities where the vaccinations are first recorded. The system does not include vaccinations carried out in private health facilities which are only a very small proportion of immunizations. As recommended by EPI guidelines[5], when a health worker administers a dose of vaccine, it is immediately recorded on an immunization register, a daily tally sheet and the child's individual vaccination card.

The PHC facility should send a monthly report to the district health office not later than the 5th of the next month, keep a copy of all reports sent and display the cumulative number of doses administered in a graph on the wall to monitor progress towards coverage targets. The individual vaccination card stays with the child's parent, while the register, tally sheets and monthly reports are archived in the health facility.

At the district level, immunization manager receives the monthly PHC facility reports, checks for completeness and timeliness and follows up on late reports. Before the 10th of the next month, a monthly report is sent to the regional level. The manager also archives the received reports, various tabulations and copies of the reports that have been sent to the regional level.

At the regional level, a senior nurse is usually in charge of the EPI. This person receives the district monthly reports, checks for completeness and timeliness, follows up on late reports and sends a monthly report to the national level not later than the 15th of the next month. In addition, the responsible person carries out supportive supervisory visits to district levels.

At the ministry level, national tabulations are made, and every year before 15th April, the country sends an official report to the World Health Organization (WHO) and UNICEF, using the joint reporting form, containing the available vaccination information for the previous year.

2.2. Sample

The districts of South Kasserine-Zouhour and Foussana were recommended by the Kasserine regional health authority for inclusion in the study, according to their immunization performance. The first district was considered as the best performing and the second as the worst performing district. The PHC facilities to visit were randomly selected. Randomization was done from the whole list of facilities in each district. A total of 6 PHC facilities in each district were selected.

2.2.1. Data collection

The immunization DQA procedure initiated by WHO in 2003[6] and the data quality self-assessment tool suggested by the WHO vaccine assessment and monitoring consultants briefing in 2004 were used to assess the accuracy of immunization data and to estimate the quality of the whole monitoring system at the district and PHC facility level.

2.2.2. Accuracy assessment

The consistency of the reporting system was estimated by determining the proportion of DTP3 vaccines reported as being administrated that could be verified by written documentation by health facilities and districts. In order to evaluate data accuracy, the research team visited PHC facilities and districts, recounted the number of vaccinations administered at each PHC facility (as recorded in the facility's immunization register), and checked the number of vaccinations reported by the monthly health facility report to the district level and the displayed graph of cumulative number of doses administered by health facilities. At the district level, the written documentation was the computerized tabulation presented as the reference data. It was compared with the monthly district report to the regional level. The study period for South Kasserine-Zouhour district was the first 10 months of 2010. The study period for Foussana district was the calendar year 2009.

2.2.3. Completeness and timeliness

Completeness and timeliness of reporting were checked through the availability of the PHC facilities' daily and monthly reports and the district monthly reports; quality of their use by local teams; editing of arrival and departure time of reports between different levels; and the formal identification of the officer who had signed the arrival or departure time of the related report. Reasons for missing or late submission reports and their impact on the quality of the reporting system were also discussed with local teams.

2.2.4. Global quality of the monitoring system

The quality of the monitoring system was measured through a questionnaire covering a number of quality indices for different components of the monitoring system, depending on the health system level: recording and reporting (paper); archiving practices; monitoring and evaluation; recording and reporting (computer); and denominators. For each component items were assigned a value from 1 to 3. These were then summed to give a total for each component and a total quality index for each level (health facility, district and region).

2.2.5. Denominators and verification factor

For each PHC facility and district, a verification factor was calculated. The factor was calculated as the total DTP3 vaccinations recounted from the register in the selected PHC facilities (or total of them for the district) divided by the total number of DTP3 doses found in the tabulation to the district level as reported by PHC facilities. To avoid errors and discrepancies, the verification factor can help to adjust the data coming from the PHC facilities or district by the routine way (monthly report). Applying the verification factor to total DTP3 vaccines reported via monthly reports should give the most accurate number. Denominators, the size of the targeted population of the PHC facility or district, are estimated every year according to national census data.

3. Results

3.1. Description of visited primary health care facilities

Table 1 summarizes vaccination statistics of the visited PHC facilities. The size of the target population and the number of DTP3 vaccinations during 2009 shows that large urban PHC centres as well as small rural ones were included in the study. All except 2 of the facilities were managed by only 1 member staff.

Table 1. Profile of the vaccination programme at the study health facilities DTP3 = third dose of diphtheria-tetanus-pertussis vaccine.

District/health facility	Population served No.	Target population No.	DTP3 vaccinated (average) No.	Workflow (vaccination sessions) No.	Vaccinators available No.	Qualification of vaccinators
South Kasserine- Zouhour district
A	20 784	347	294	2/week	2	Nurse + nurse helper
B	7 817	121	126	1/week	2	2 nurses
C	1 692	32	20	1/month	1	Nurse
D	199	4	7	1/month	1	Nurse
E	2 758	51	30	2/month	1	Nurse helper
F	785	14	17	1/month	1	Nurse
Foussana district
G	13 253	240	308	1/week	1	Nurse helper
H	3 257	60	66	1/month	1	Nurse helper
I	1 500	28	30	1/month	1	Nurse
J	1 373	25	17	1/month	1	Nurse
K	1 500	30	20	1/month	1	Nurse
L	2 000	42	145	2/month	1	Nurse helper

Accuracy Table 2 shows the accuracy ratios for 4 stages of reporting in the 12 PHC facilities. There was evidence of low accuracy of reporting data by all PHC facilities in both districts. Over-reporting was more frequent than under-reporting. PHC facilities with a high number of vaccinations tended to over-report and those with low rates under-reported. Table 3 presents the district accuracy ratios. The recounts carried out from the registers of the PHC facilities were considered as the most accurate figures. They were compared with the recounts carried out from the monthly reports and of the monitoring tabulations displayed at the district level. As seen in Table 3, both districts were over-reporting, with accuracy ratios of 109% and 129%. Foussana district remained as over-reporting even if health facility L, which presented very inconsistent data, was excluded. If we assume that the governorate picture could be similar to that of the investigated districts, the regional accuracy ratio would be 118%, which is an overall situation of over-reporting. From these recounts, we also calculated a verification factor (Table 3) which could be used for to adjust the data for final validation by the health authorities. Verification factors ranged from 18% to 142% across the individual PHC facilities and were 77% and 92% for the 2 districts. The regional verification factor was estimated as 85%.

Table 2. Accuracy ratios for different stages of the reporting system for number of people vaccinated with third dose of diphtheria-tetanus-pertussis vaccine (DTP3) by health facility and district.

District/health facility	Accuracy ratiosa				Result
	Health facility monthly report %	Health facility tabulation %	District monthly report %	District tabulation %
South Kasserine- Zouhour district
A	114	113	113	113	Over-reporting
B	94	95	94	94	Under-reporting
C	95	85	110	110	Over-reporting
D	85	85	85	85	Under-reporting
E	117	117	117	117	Over-reporting
F	108	-	100	100	Over-reporting
Subtotal	District monthly report/district tabulation 111%				Over-reporting
Foussana district
G	103	108	102	102	Over-reporting
H	109	108	125	125	Over-reporting
I	110	-	103	103	Over-reporting
J	83	95	75	71	Under-reporting
K	79	90	79	79	Under-reporting
L	563	-	559	537	Over-reporting
Subtotal	District monthly report/district tabulation 102%				Over-reporting

Accuracy ratio at each stage of reporting=(No. of people recorded as vaccinated with DTP3/No. recorded vaccinated in health facility register)×100.

Table 3. Verification factors for number of people vaccinated with third dose of diphtheria-tetanus-pertussis vaccine (DTP3) by health facility and district.

District/health facility	Total DTP3 register No.	Total DTP3 district report No.	Total DTP3 district tabulation No.	District accuracy ratioa %	Verification factorb %
South Kasserine- Zouhour district
A	334	381	381	-	88
B	102	96	96	-	106
C	20	22	22	-	91
D	13	11	11	-	118
E	12	14	14	-	86
F	13	13	13	-	100
Subtotal	494	537	537	109	92
Foussana district
G	303	308	308	-	98
H	53	66	66	-	80
I	29	30	30	-	97
J	24	18	17	-	141
K	19	15	15	-	127
L	27	151	145	-	19
Subtotal	455	588	581	129	78
Both districts	949	1125	1118	119	85

aDistrict accuracy ratio = (no. of people recorded as vaccinated with DTP3 in district tabulation /no. recorded vaccinated in health facility register) × 100. bVerification factor = (no. of people recorded as vaccinated with DTP3 in health facility register/no. recorded vaccinated in district tabulation) × 100.

3.2. Completeness and timeliness

Previous annual reports for the study period were physically available at the time of the assessment for all districts and PHC facilities, except the tally sheet which was not used in 8 PHC facilities. In some cases, copies of health facility monthly reports to the district level were found. Regarding timeliness, we were not able to determine how many reports were received by the deadline as the reception date was no longer recorded.

3.3. Quality of monitoring system

3.3.1. Quality by level

The results related to the aggregated quality of the monitoring system (Table 4) showed a similar quality index in both districts (57% and 53%). Quality index scores of PHC facilities ranged from 51% to 69%. In each district, half of PHC facilities had somewhat better scores (quality index between 60% and 70%) than the other half (quality index between 50% and 60%). Health facility L showed a very big gap between the recounted register data and reported data (quality index of 34%).

Table 4. Quality indices of different components of the reporting system and aggregated quality index by health facility and district.

District/health facility	Recording & reporting (paper)%	Archiving practices %	Monitoring & evaluation %	Recording & reporting (computer)%	Denominators %	Quality index %
South Kasserine- Zouhour district
A	79	100	40	-	-	69
B	59	60	40	-	-	51
C	79	100	14	-	-	57
D	59	100	40	-	-	63
E	79	100	27	-	-	63
F	79	100	0	-	-	51
Subtotal	58	-	58	46	59	57
Foussana district
G	79	100	13	-	-	57
H	79	100	27	-	-	63
I	81	80	27	-	-	57
J	79	100	27	-	-	63
K	79	100	27	-	-	63
L	61	61	0	-	-	34
Subtotal	39	-	58	46	72	53

3.3.2. Quality by component

As Table 4 shows, the factor which most affected the final score at district level was the electronic recording of data. At PHC facilities, the factor which most affected the final scores was follow-up and evaluation.

3.4. Denominators

The denominators used to calculate immunization coverage of targeted populations for the districts or PHC facility came from the National Institute of Statistics (INS). At the beginning of each calendar year, the regional level submits to the districts the estimated population by district and PHC facility. However, we found a difference between denominators submitted from the regional level. The denominators used to estimate 2010 coverage were much lower than those of 2009. Indeed, assuming that the INS estimations were not valid due to the large decrease in the birth rate, it was decided to use revised denominators. For 2010, the number of 1st doses of DTP (DTP1) administrated in 2009 was considered as the target population. In spite of these corrections, coverage rates higher than 100% were noted.

4. Discussion

The DQA method has 2 important limitations[8]. The first relates to the lack of precision of the verification factor in validating the reported number of children receiving 3 doses of DTP, due to the small sample size and a large variation in verification factors among districts in the same country. The second weakness is related to the fact that in the same audit, doses which are recorded but are not administrated and vice versa cannot be verified. Community-based verification is necessary if such inaccuracy is suspected.

In our study, we did not plan to calculate a verification factor in order to adjust the figures given by the PHC facility or district immunization administrative data. The exercise aimed to help health teams to learn how to implement self-assessment, to have a critical analysis of the quality of routine immunization data and to plan the same evaluation for use in other reporting systems. Calculating a total verification factor for the governorate using the data collected from the 2 districts might not be useful. We should include all districts, calculate a weighted verification factor and propose it as a tool to adjust data coming from the PHC facilities or district by the routine way (monthly reports). Calculating a crude verification factor as we did was just an example to show how it can be done. We could also benefit from this audit by examining a number of other issues. The recount of DTP3 of the 949 children recorded on the registers of the visited health facilities enabled us to check the background of childbirth (the proportion of childbirth), the age at DTP1 vaccine administration, the time between the first contact and DTP3 administration and the drop-out rate. This also enabled us to check the wastage rate of vaccines, the proportion of children protected against tetanus and the proportion of vaccinated women.

Differences were found at both health facility and district levels when comparing data from health facility registers with the tally sheets, monthly reports and tabulations. Regarding the large discrepancy found between the target population and the total number of DPT3 doses administered in PHC facility L (145 versus 42), certain problems were noted: the health centre was rarely supervised because it is located in a remote area with difficult access, the health worker responsible was not well trained (only 6 months of registration) and there were frequent changes to the immunization register.

A process evaluation study carried out in Mozambique in Niassa province between January and March 2003 showed differences for all the types of vaccines by comparing the reports of the health facilities with the reports of the district[9]. Onta et al. conducted a study in the hill district in the western Nepal to compare the number of children who received various vaccines during 1 year as reported by 3 sources: immunization registers of 3 primary health care centres, monthly reports of the health centres and monthly reports of the Official Services of Statistics[10]. They found differences between the 3 examined sources for all types of vaccines. Ronveaux et al. evaluated the consistency and quality of immunization monitoring systems in 27 countries during 2002-2003 using standardized DQAs that had been launched within the framework of the Global Alliance for Vaccines and Immunization[8]. They showed that many countries had inconsistencies in their immunization data, hampering their ability to manage their immunization programmes.

The accuracy problem we met was more often due to problems of over-reporting than under-reporting. A total of 8 health facilities out of 12 studied, especially those with high patient flows, were over-reporting vaccinations. Centres with low patient flow were generally under-reporting. Both the districts studied were over-reporting and, if we assume that the 2 districts are representative of the whole governorate, we have an overall situation of over-reporting.

Ronveaux et al. showed that among 557 health units assessed in 27 countries, 40% were over-reporting and 7% were under-reporting DTP3[8]. Onta et al. in Nepal showed that for all vaccines the number of children vaccinated as recorded in the monthly reports of health centres was higher than the number recorded by registers[10]. The number of vaccinations in monthly reports of the Official Services of Statistics were higher than the number recorded by the health facility monthly reports for BCG, DTP and measles vaccines and were lower for poliomyelitis vaccine.

The verification factor which could be used to adjust the reported vaccination rate of Kasserine governorate was estimated as 85. Ronveaux et al. reported that the national verification factors ranged from 40 to 100% in 27 countries. The proportion of verified DTP3 doses was lower than 85% in 16 countries and 33% of countries had verification factor suggesting considerable over-reporting (verification factor <0.70%). The DQAs conducted in Cameroon[11] and in Ivory Coast[12] during 2001 reported verification factors of 47.7% and 59.4% respectively.

The discrepancies between various sources of information were found mainly in the PHC facilities, where differences between the immunization register data, tally sheets, monthly reports and tabulations were usually noted. Nurses at health facilities needed to record every dose 3 times-on the vaccination card, register and tally sheet-and later to recalculate the number of doses administered in order to complete the monthly report. It seems to be an extra workload which is does not favour accurate registration. Introduction of real-time computerized recording and automatically generation of reports may improve the accuracy of data at the health facilities. Adams et al. carried out a study to evaluate data quality, nursing satisfaction and reduction in documentation burden after the introduction of a point-of-service immunization entry system in an inter-city paediatric PHC centre[13]. They showed that entry by nurses at the time of immunization preparation improved the quality of tracking data, reduced misclassification of immunization needs, saved time and was well accepted. Ronveaux et al. found that difficulties in verifying the doses administered often arose at the peripheral level of the health service, usually as the result of discrepancies in information between health units and their corresponding districts or because completed recording forms were not available from health units[8]. The Cameroon DQA explained the lack of accuracy as due to bad storage of reports and other immunization documents leading to loss of data[11].

Regarding completeness, all tally sheets and registers related to the study period were available in the health facilities and districts. Regarding timeliness, it was not possible to know how many reports were received by the deadline. The notation of the date of reception on the reports was not recommended in the vaccination programme guidelines. In Cameroon, the DQA showed that 94.1% of the reports of the various districts were found and 0% had arrived on time[11]. In Ivory Coast, these proportions were 91.1% and 1.1% respectively[12]. Ronveaux et al. noted that all countries studied had poor monitoring of completeness and timeliness of reporting[8].

We found the quality index scores were found to be good in Cameroon (72.1%)[11] and in Ivory Coast (75.0%)[12]. For Ronveaux et al., quality index scores varied widely at all levels, and the overall analysis showed substantial deficiencies in all countries. The mean national quality index was 63% and the mean health unit quality index was 58%[8].

At the health facility level, the quality index score was most affected by the score on the monitoring and evaluation component. The factor which seemed to most affect the quality index score for the district level was electronic recording of the data.

For most vaccine programme the target population is the national annual number of births or number of surviving infants. This denominator usually comes from the official statistical services, which organize a national census every 10 years. This means that the target population estimates become increasingly more imprecise in the years between censuses. In our study, the denominators used to estimate the 2004 immunization coverage were much lower from those of 2003; yet despite this, immunization coverage rates higher than 100% were reported. The Cameroon DQA found similar immunization coverage rates higher than 100%. The denominator problem is a constant problem in all countries throughout the world. The best approach is to consistently aim to improve and strength the routine monitoring system of the EPIs in order to increase consistency and to consider the DTP1 population as the best estimate of the target population of the programme.

Although administrative data might be the most widely available information for assessment of immunization coverage, their validity for measuring changes in coverage over time is highly questionable. Because of the intense international efforts to achieve the Millennium Development Goals, there is a need for accurate immunization data collected on an ongoing basis. The DQA is a tool that provides a way to independently assess the quality of immunization monitoring systems at all levels of a health service and serves as a point of entry to make improvements. It provides a useful example for other global health initiatives.

Comments

Background

The introduction section provides an essence of the EPI in Tunisia and the hierarchical organizational facilities which report the registered annual vaccinations. The paragraphs are gently organized in a way that one may comprehend the necessity for evaluating the quality of immunization data in Tunisia.

Research frontiers

According to the previous researches undertaken in most developing countries, the reported data regarding the quality of immunization may be affected by several organizational factors within the PHC facilities and thus, its quality may be confounded especially under resource limited settings. Thereby, it would be of great interest to conduct studies via which the quality of the national records regarding EPI data are audited and further evaluated.

Related reports

The data acquired to audit the quality of immunization data have been extracted from national information system such as integrated child health reporting system and the national tabulation reports which are used for providing reports for WHO.

Innovations & breakthroughs

The present study provides valuable data on the auditing of the quality of data on immunization in Tunisia. Therefore, from a subjective point of view in addition to the paucity of data, auditing of the immunization data would be a novel subject in order to assess the quality of the represented national reports in the developing world.

Applications

Considering the importance of the subject with regard to the EPI coverage in Tunisia, the report of the present study may be used for further enhancing the accuracy of the annual PHC facility reports. Correspondingly, the necessary steps may be undertaken in order to further increase the accuracy of immunization data, as a vital component of national health reports.

Peer review

In general, the study is well-structured. The material and methods section thoroughly describes the nominators and denominators as well as sources of the data extraction. However, it would be better to reconsider the paragraph organization in the discussion part, and fuse the comparisons which describe the same results in accordance to the results of the current study. Also, I wonder if it would be applicable to provide other data on further vaccinations in the EPI such as MMR, TB, etc; and compare them with the data regarding DTP3.