Collecting and using reliable vaccination coverage survey estimates: Summary and recommendations from the “Meeting to share lessons learnt from the roll-out of the updated WHO Vaccination Coverage Cluster Survey Reference Manual and to set an operational research agenda around vaccination coverage surveys”, Geneva, 18–21 April 2017

Abstract

Household surveys are frequently used as means of vaccination coverage measurement, but obtaining accurate survey estimates present several challenges. In 2015, the World Health Organization (WHO) released a working draft of its updated Vaccination Coverage Survey Reference Manual that moved well beyond the traditional Expanded Program on Immunization (EPI) survey design. In April 2017, WHO convened a four-day meeting, to review lessons learned using the updated manual and to define an agenda for operational research about vaccination coverage surveys. About 70 stakeholders, including EPI managers and participants from 10 countries that have used the updated Survey Manual, survey experts, statisticians, partners, representatives from WHO regional offices and headquarters, and providers of technical assistance discussed methodological issues from sampling to accurately ascertaining a person’s vaccination status, optimizing data collection and data management and conducting appropriate analyses. Participants also discussed data sharing and how to best survey data for immunization decision-making. The lessons learned from the use of the updated WHO Survey Manual related mainly to operational issues to implement better quality vaccination coverage surveys. It resulted in a list of 23 recommendations for WHO, donors and partners, immunization programs, and household surveys that collect immunization data. Similarly, 14 research topics, categorized in six themes (overall survey conduction, sampling, vaccination ascertainment, data collection, data analysis and use, and inclusion of questions on knowledge, attitudes and practices) were prioritized. Top areas of further work included improving our understanding of the accuracy of caregiver recall when documented evidence of vaccination is not available, improving engagement and coordination between immunization programs and entities conducting multi-purpose household surveys such as Demographic and Health Survey and Multiple Cluster Indicator Survey, improving mechanisms for sharing vaccination survey datasets and documentation, and making better use of survey results to translate data into knowledge for decision-making. This manuscript summarizes the meeting proceedings and provides an update of actions taken by WHO since this meeting.

1. Background

Vaccination coverage is an important indicator to track and guide immunization programs at the global, national and sub-national levels [1]. While coverage is ideally continuously monitored through routine administrative systems and registries, data can be incomplete or inaccurate, particularly in low and middle-income countries (LMICs) [2], [3]. Therefore, household surveys are frequently used as a supplement to administrative data or, in some cases as the primary means of vaccination coverage measurement [4]. Nevertheless, obtaining quality vaccination survey coverage estimates also presents challenges. Methodological issues include accurately ascertaining a person’s vaccination status, the potential for selection bias given difficulties in conducting probability sampling, optimizing data collection and data management techniques and conducting appropriate analyses [5]. There are also strategic and organizational challenges including engaging appropriate stakeholders and ensuring decision-makers understand the results, including their limitations, and use the data to their full potential.

Since the 1980s, the World Health Organization (WHO) has provided guidance on designing, conducting, and utilizing vaccination coverage surveys [6], [7]. In 2015, WHO released a working-draft of its Vaccination Coverage Survey Reference Manual that moved beyond the well-known “30x7” Expanded Program on Immunization (EPI) survey design [8]. The update was motivated by growing complexities of EPI in the 21st century [9]; a need for more accuracy and precision with increasing coverage levels [10]; global emphasis on accountability and transparency [11]; and increasingly sophisticated statistical and computational capacities in LMICs. Table 1 presents the main differences between the updated manual and previous WHO guidance on vaccination coverage surveys.

Table 1.

Main changes in the updated WHO Vaccination Coverage Cluster Survey: Reference Manual compared to previous guidance on vaccination surveys.^a

Topic	Previous WHO guidance on vaccination surveys	Updated WHO Vaccination Coverage Cluster Survey: Reference Manual
Sampling	Non-probabilistic sampling, analysis gave equal weight to every respondent (non-interpretable CIs)	Probabilistic sampling, weighted analysis and meaningful confidence intervals (CIs)
	Data collectors selected households to visit and randomly selected first dwelling, usually using spin the pen/bottle technique	Households (HHs) to be interviewed are pre-selected (requires good maps and usually field visits prior to interviewers’ field work)
	Quota sampling. Usually 30 clusters of 7 children each	Sample size to be defined according to survey objectives (estimation, hypothesis testing or classification). Pre-defined number of HHs to find an approximate number of children in each cluster
	Assumed design effect (DEFF) of 2 (intra-cluster correlation of 1/6)	Recommends DEFF depending on number of eligible people per cluster
	No attempts at revisits recommended	Recommends at least two revisits to obtain interviews in pre-selected HH; document outcomes of each visit
Eligibility	Proposed the inclusion only of persons who had resided in the area for at least six months	Removes the length of residence as an inclusion criteria, and instead, it proposes adding a question to the questionnaire on how long the individual has been living at the present residence.
Vaccination ascertainment	Relied on home-based records (cards) and/or maternal/caregiver recall	Relies on home-based records (cards) and/or maternal/caregiver recall, but encourages visits to health care facilities to document vaccination from facility records
		Recommends photographing cards, when possible
Data collection	Only paper-assisted personal interviewing (PAPI)	Includes section on computer-assisted personal interviewing (CAPI) (using mobile devices for data collection)
Report writing	Not clear guidance on report writing	Encourages using the results for action Encourages detailed report writing to clearly understand limitations
Overall quality		Renewed emphasis on taking steps to reduce bias and improve overall survey quality

^aAdapted from “2018 WHO Vaccination Coverage Cluster Survey: Reference Manual”, section 1.4 [12].

In April 2017, WHO convened a four-day meeting to review lessons learned using the updated manual and to define an operational research agenda about vaccination coverage surveys; in practice, the meeting ended-up covering a broader set of survey issues. About 70 stakeholders, including EPI managers and participants from 10 LMIC countries that recently used the updated Survey Manual, survey experts, statisticians, partners, representatives from WHO regional offices and headquarters, and providers of technical assistance, shared experience through presentations, panels, and break-out sessions, each followed by plenary discussion. Following the meeting, a questionnaire was sent to all attendees to help prioritize 14 potential research topics and potential WHO actions proposed during the meeting. Questionnaire results are shown in Table 2 and Fig. 1. The draft manual was updated after the meeting, mostly with editorial changes, and a final version released in 2018 [12].

Table 2.

Priority actions related to vaccination coverage surveys, by stakeholder. WHO actions are ranked by the proportion of meeting participants assigning each item a priority rating of 4 or 5 (on a 0–5 scale) in the post-meeting online poll (N = 19).^a The poll did not solicit feedback on recommended actions for non-WHO stakeholders.

Actions, by stakeholder
WHO (as a facilitator or lead)	% 4 or 5 priority rating
1. Lead conversations and reflection on how to translate data into knowledge for decision making, including discussing early on how the coverage survey will be analyzed and used.	80%
2. Improve standards and technology for sharing datasets and documentation.	68%
3. Develop a standard template for EPI survey reports to standardize critical outputs – Tables could mirror DHS and MICS standards to allow for easy comparability.	63%
4. Create a set of quality criteria that can be used to grade survey results to better inform the users on potential limitations or survey quality issues.	63%
5. Explore using online tools to support survey planning and analysis, including publicizing existing tools such as annual equity analysis.	63%
6. Develop or identify tools for collecting useful vaccination coverage information at the district and local level, that are more practical and affordable than doing surveys in all districts	58%
7. Document/compile budget and sample information from surveys to demonstrate budget/sample size trade-offs and drivers of costs in different settings.	50%
8. Develop guides/toolkits to help interpret results and highlight actions to be taken based on the survey findings.	50%
9. Work to ensure countries have a good rationale for doing a survey, and that those without sufficient rationale are discouraged.	45%
10. Continue strengthening collaborations between EPI, DHS and MICS.	40%
11. Document/compile case studies of what went right and wrong when implementing vaccination coverage surveys, mainly when using the WHO Survey Manual.	37%
12. Examine how to ensure health facility visits are worth the effort (e.g.: when should they be done? can you collect other info while there?).	37%
13. Develop standard questions on household and demographic characteristics, but that still need to be adapted and tested in each country.	32%
14. Consider oversampling selected areas or populations as needed for decision making, rather than all or no district level strata	16%
Country Immunization Programs
15. Designate an individual or working group to engage closely with DHS/MICS on the vaccination components of their surveys, from planning to report writing and result dissemination. This individual or group can advise on the formulation of vaccination questions, training of supervisors and enumerators, pilot testing and fieldwork protocols to maximize the quality of vaccination data collection, in order to increase the credibility of results for the EPI manager and reduce the need for parallel EPI surveys.
16. When an EPI survey is needed, consider coupling the EPI survey to MICS or DHS, when feasible and appropriate
17. Take the lead in defining the EPI needs that can be addressed via a vaccination coverage survey. Actively participate in a Vaccination Coverage Survey design (including expected tables and graphs), piloting, training, facilitation of field visits and access to registers in health facilities, and report writing and dissemination with all stakeholders. The latter also applies to engaging with the team leading a DHS, MICS and any other survey collecting vaccination data
18. Take provisions to make Immunization coverage survey reports and datasets available to the global community.
Donors and Partners
19. Promote collaboration between EPI and DHS, MICS and other household surveys that include immunization indicators.
20. Consider measures to prevent EPI coverage surveys in countries with a recent or upcoming MICS or DHS survey, unless specific questions or reasons warrant the implementation of an EPI survey. When an EPI survey is needed, consider coupling the EPI survey to MICS or DHS as appropriate.
21. Ensure that non-technical staff dealing with countries better understand the role of surveys, vis-à-vis other available tools to answer specific questions.
22. Encourage immunization programs to identify their needs for secondary survey analyses.
Household surveys that collect immunization data (DHS, MICS, SMART, and others)
23. Communicate potential survey plans as early as possible to WHO and country immunization programs. This will facilitate coordination and collaboration, and allow EPI to account for DHS, MICS and other household surveys in their annual and multi-year planning.

^aRelative to all meeting attendees, respondents were disproportionately from research or partner institutions.

Fig. 1.

Vaccination coverage survey research agenda, by the proportion of meeting participants assigning each item a priority rating of 4 or 5 (on a 1–5 scale) in the post meeting online survey (N = 19^*).

This paper describes the main discussion points, recommendations, and conclusions from the meeting and subsequent poll.

2. Collaboration among survey implementers and national immunization programs

Vaccination coverage is estimated in surveys commissioned by national immunization programs, Demographic and Health Surveys (DHS), UNICEF-supported Multiple Indicator Cluster Surveys (MICS), national health surveys [4], [5], among others. Between 2000 and 2015, there were 61 instances where a country conducted a vaccination coverage survey within one year before or after a DHS or MICS (unpublished results presented in the meeting). While the two surveys sometimes had similar results, the findings often diverged substantially, leaving decision-makers unsure what to believe or do and providing an opportunity to discount results that reflect poorly on their program.

Participants agreed that countries, WHO, partners, and donors should standardize and harmonize methods as much as possible and avoid expending unnecessary resources on parallel surveys. First, it was recommended improving communication and coordination between DHS and MICS with WHO and within countries between National Statistical Offices (NSOs) and Ministries of Health, so that immunization programs account for these surveys in their annual and multi-year plans. Second, every national EPI could designate a focal point to closely advise DHS/MICS or similar multipurpose surveys, on current vaccination schedules, recent vaccine introductions, different home-based records (HBR) or vaccination cards in use, formulating vaccination survey questions, training supervisors and interviewers, and designing questionnaires and fieldwork protocols. This would improve the credibility of vaccination results, increase EPI’s confidence in DHS/MICS and other survey results, and reduce the perceived need for separate EPI surveys.

Meeting discussions included balancing the needs of EPI with the operational and technical structure of DHS/MICS. For example, EPI may wish to collect additional information, such as reasons for non-vaccination, that lie beyond a standard DHS/MICS. On a case-by-case basis, coupling or integrating the EPI needs with DHS/MICS should considered as was successfully completed in Mongolia in 2013 when the Ministry of Health, WHO, and UNICEF conducted a Social Indicator Sample Survey that integrated aspects of the DHS, MICS, and UNFPA-funded Reproductive Health Survey [13] modules. The 2016–2017 Nigeria MICS included reasons for non-vaccination and additional clusters were sampled in selected states to achieve state-level vaccination coverage estimates [14]. However, it was noted that DHS and MICS must strike a balance between customizing surveys to meet a country’s needs and maintaining international comparability, and that if all health programs added elements, a standard survey could become unmanageable. Other options discussed were conducting more qualitative studies, for example on barriers-to-vaccination rather than a survey.

NSOs commonly provides assistance with sample design and the sampling frame and maps to visit the selected locations. Country representatives reported that failing to engage NSOs early is a common occurrence. It was recommended that when conducting a survey, the EPI sign a formal memorandum of understanding with the NSO and other actors to define roles early on.

3. Survey objectives and design

To decide whether, or when, to conduct a vaccination coverage survey, a country must clearly define survey objectives and estimate costs and timeline. This is newly emphasized in the second chapter of the WHO Survey Manual [12], and should be based on careful consideration of how the survey results will be used and the desired balance between quality, granularity, precision, cost and time.

3.1. Estimation vs. classification

EPI cluster surveys are used to estimate vaccination coverage (i.e., generate point estimates and sometimes confidence intervals) or to classify coverage (i.e., generate labels like acceptable/not acceptable or pass/fail). For the latter, Lot Quality Assurance Sampling (LQAS) has also been used [15], though some authors have highlighted LQAS shortcomings [16]. The updated WHO Survey Manual provides a consistent framework for sample size calculations and data analysis for estimation and classification.

Classification is often more relevant to surveys conducting following supplemental immunization activities (SIA) or campaigns. However, it is often unclear what to do with results where the threshold falls within the confidence intervals. The new WHO survey guidance includes an indeterminate outcome category when the sample size is insufficient to confidently classify whether coverage is above or below a programmatic threshold. Nevertheless, meeting participants reported that in a 2016 post-SIA survey in Kenya, country-level staff found the new ‘indeterminate’ category to be unsatisfying and difficult to act upon. While more work is needed to define how to better deal with indeterminate results, it was recommended that countries define a priori how they will describe and act on results that are neither clearly above nor below a predefined threshold.

3.2. Granularity of results

Survey designers must also decide the level they will report coverage results (i.e., national only, provincial, and/or district levels). Country participants expressed a strong preference for results aligned with the country’s administrative levels, typically districts. Some countries recently completed or are planning surveys designed for precise district-level estimates including Burkina Faso 2016, Uganda 2016–2017 (reports not yet shared with WHO, as of July 2018) and Kenya 2016 following a vaccination campaign [17]; Pakistan is conducting a district-level survey in 2018. However, district-level surveys require a very large sample size, thus they are costly, complicated and can suffer from inconsistent fieldwork and supervision quality. For these reasons, WHO, UNICEF and Gavi participants suggested that district-level surveys are usually not cost-effective, though country participants reiterated their perceived value for local decision-making.

As an alternative to district-level surveys, sub-national surveys that seek to answer specific questions about particular areas or populations can be done. Discussions also revolved around (a) implementing simple-but-insightful assessment tools at the district level (see next section) and (b) developing experience with statistical methods known as small-area estimation where less data from less granular surveys might be combined with administrative covariates to estimate coverage at the district level with less expense than a full every-district survey. Small-are estimation was ranked as one of the top research priorities in the post-meeting poll.

3.3. Rapid monitoring tools

Several participants expressed the need for tools that could complement administrative data to quickly and inexpensively provide information for local immunization program management. A rapid monitoring tool from the Pan American Health Organization (PAHO),² module 3 from PAHO’s “Toolkit to monitor vaccination coverage and preventive chemotherapy coverage,” was identified as a potential model [18], although statisticians and survey experts emphasized that such tools are not equivalent to rigorous probability sample surveys [19]. It was re-emphasized that improving administrative data is of highest priority. Also, WHO was asked to provide better practical guidance around when a targeted or rapid monitoring is justified, how it should be implemented, and how to interpret and use those results.

3.4. Combining post supplementary immunization activities and routine immunization surveys

Special considerations arise when considering whether to piggyback routine immunization (RI) questions on a post-SIA/campaign coverage survey [4], [12]. Adding RI questions will increase the number and duration of interviews conducted and the total time and budget and will magnify the complexity of data entry, management and analysis. Depending on whether RI results are designed to be precise only at the national level or in every survey stratum, the added requirements may substantially increase the number of households to be visited. And if the RI requirement is an afterthought, the planning might delay the start of survey fieldwork, making it difficult for SIA respondents to recall whether their child was vaccinated.

Adding an RI component to a post-SIA survey is possible, but it requires timely planning. The lowest-impact scenario is one where survey planning commences concomitantly with SIA planning and RI questions are added, but the number of households visited per cluster is driven by SIA survey goals. Each cluster may yield very few RI respondents; thus, RI results may only be precise at the national level.

4. Probability sampling

Probability sampling is an important element of the updated WHO survey manual, bringing EPI cluster surveys in line with established standards from the broader household survey community. Probability sampling means that every eligible person has a non-zero and quantifiable probability of being selected for the survey; making the survey results representative of all eligible respondents – not only the respondents who are interviewed – and making it possible to size the survey to achieve a desired precision.

4.1. Sampling frames and spatial sampling

Primary sampling units are usually selected from lists or frames of census-based enumeration areas (EAs) and their probability of selection is tied to estimates of population size. In many countries, the frame is out-of-date, which is even more problematic with the increasing population movements, so selection probabilities and survey weights likely do not represent the current spatial distribution of eligible respondents; consequently, survey results are likely biased. Even when a country has a very recent census, often the EAs hold many more households than the survey requires per cluster, so a process must be devised to rigorously randomly select a subset of households and the selection must be well documented to calculate survey weights. The updated WHO Survey Manual includes guidance on conducting probability sampling and conducting weighted analyses.

An important sampling challenge is ensuring that no populations are missed, especially if those groups are also likely to be missed for vaccination. Meeting attendees conceptualized two types of special populations: those excluded from the sampling frame entirely, and those who are in the frame but inadequately sampled. Context-specific examples include: seasonally inaccessible rural areas; persons with no fixed abode; refugees integrated into host populations; internally displaced people; migrants; nomadic populations; unregistered individuals; indigenous populations; people working in the black market; areas with security concerns; gang members; and communities that refuse to participate in surveys because they have been over-surveyed in the past, or wealthy individuals in gated communities. Potential solutions include negotiating with leaders to access hard to reach areas; spatial sampling to include individuals not on official registries [20] and alternative survey designs specifically targeting these populations [21].

Spatial sampling offers an alternative to census-frame selection and several spatial methods are under development to measure coverage of health interventions [22], [23]. Participants learned about gridded datasets of likely population counts where grids are seeded with census data and updated using modern sources like satellite imagery and cellular telephone usage. An R package entitled ‘GridSample’³ has been developed that makes gridded sampling a viable option for future surveys [24]. Spatial sampling has been successfully used to measure routine immunization coverage in some areas in Pakistan [25] and to evaluate a cholera vaccination campaign in Lusaka [26]. Spatial sampling was identified as an active area of work to learn about and consider using.

5. Ascertaining vaccination status

Meeting participants discussed many challenges to accurately ascertain a person’s vaccination history.

5.1. Home-based records

In addition to providing frontline health workers with a standardized patient history that is convenient, comprehensive and vital to making informed decisions about the need for care and immunization services, home-based records (HBRs) are an important source of documented evidence of vaccination history [27]. Efforts are on-going to revitalize HBRs as a critical tool within immunization service delivery including focused activity on immunization beyond the first year of life and on reducing missed opportunities for vaccination. Participants discussed use of survey coverage estimates derived solely from documented evidence [28] and agreed that HBR-only coverage would likely underestimate coverage in most settings.

Meeting participants agreed research is needed on how to increase HBR availability during surveys, including advertising the survey ahead of time. Participants shared experiences from Bangladesh, Bolivia, Burkina Faso, Lebanon [29], and a DHS in South Africa that have tested photographing HBRs to facilitate data cleaning as some queries could be resolvable without returning to the field, and in the case of Lebanon, for data extraction. Although HBR imaging during field work is feasible, additional thought and guidance is needed to ensure their proper management (including filing, data security and confidentiality) and their use for survey inference. Countries were urged to publish their experiences using photographed evidence of vaccination history.

5.2. Health facility visits

The updated Survey Manual encourages survey planners to consider conducting facility trace-back exercises and visit health facilities to search for documented evidence of vaccination history. Such exercises are only beneficial if immunization services are recorded in name-based facility registers. Many challenges were highlighted in experiences with facility trace-back exercises conducted in eastern Europe and more recently in Ethiopia [30] in DHS surveys and in Senegal and Bangladesh in EPI surveys [31]. Challenges included children who received vaccinations from several facilities or campaigns and therefore were not identified in a single facility register; lack of record standardization across facilities; poor organization of register information; and inconsistency in children’s identifying information. The potential use of electronic immunization registries was not discussed. Careful thought should be given before including health facility trace-back exercises in a coverage survey. Further exchange of documented experiences is needed to inform the role of pre-survey piloting for feasibility, as well as use of different approaches such as assigning a special team to conduct the facility visits, collection of additional facility-based data and conduct of health worker interviews. Seeking data at health facilities was highlighted as an area that can help detect issues with data recording practices, that not only affect surveys but also likely affects administrative coverage estimates.

5.3. Caregiver recall

Vaccination schedules have become much more complex than they were in the early days of EPI [32]. There is substantial debate about whether and how a caregiver’s memory recall should be considered when estimating vaccination coverage. A 2013 systematic review by Miles et al. observed that recall has low sensitivity and specificity when compared to facility records [33]; this was based largely on studies in high-income countries. An updated review on recall focusing on LMICs was commissioned following this meeting.

Some participants advocated for dropping recall as an ascertainment method in surveys but most participants acknowledged a need to retain it. It was agreed that more research is needed on the formulation of recall questions and on considering analysis adjustments for those with only recall data, based on comparing documented vaccination evidence with recall. For instance, the human papilloma virus (HPV) vaccine might be better recognized as the “cancer vaccine” in some settings; visual cues might improve recall; and vaccines targeting older children (e.g.: HPV and certain measles-rubella (MR) campaigns) may be better assessed through combined caregiver and child recall, especially if the vaccine is administered at school. Ascertainment via recall was the second highest ranked research priority in the post-meeting questionnaire.

6. Knowledge, attitude, and practice questions

Meeting participants discussed whether knowledge, attitude, and practice (KAP) questions should be routinely included in vaccination coverage surveys; the WHO Survey Manual does not include this component. Although KAP questions can be informative for action, a standard set of validated immunization focused KAP questions remains in development. Given that such questions are sensitive and may require additional interviewer training and more nuanced interpretation to be useful, participants decided to await forthcoming results of a UNICEF-led working group that will recommend KAP questions. The KAP questions added to a coverage survey are likely different from quantitative and qualitative methods specifically designed to uncover ideas that may represent obstacles to receiving immunization services.

7. Electronic data collection (CAPI)

Computer Assisted Personal Interviewing (CAPI) is increasingly used for vaccination coverage surveys [34] The technology, which requires dedicated support plus substantial investment in devices and interviewer training, has some acknowledged advantages and challenges (Box 1). Country experiences with CAPI for vaccination coverage surveys highlighted challenges with different software versions across interviewers, incomplete data transmission due to poor connectivity and handling duplicate records [35]. The potential for date-related data entry errors on touch screen devices was discussed following presentation of a study suggesting error rates on dates >10% with a commonly-used default smartphone interface (unpublished results presented in the meeting). It was agreed that further work is needed to develop and test interfaces and instructions to reduce error rates below 1%, a rate commonly achieved with paper forms and keyboard double-data entry [36], [37]. Participants further agreed that in the absence of in-country CAPI experience and support, paper forms are certainly acceptable, using double-entry with computer identification of discordant entries and rigorous data cleaning. [38].

Box 1. Advantages and challenges of computer assisted personal interviewing in vaccination coverage surveys.

Advantages

• •Assist with household selection when GPS is built-in;
• •Automation of skip patterns and respondent eligibility;
• •Provide interviewer indication of warnings and messages;
• •Allow for forms for more than one HBR format;
• •Facilitate linkages of photographs of HBRs with respondent questionnaire;
• •Display images on the questionnaire; and
• •Facilitate daily electronic data transfer to a central office for “near real-time” quality monitoring and timely analysis

Challenges

• •Power/charging requirements for electronic devices;
• •Device breakage or heat damage, theft and viruses;
• •Increased technical assistance requirements;
• •Unfriendly user interface;
• •More data entry errors and
• •Interviewer use adaptations (i.e., learning shortcuts) they wouldn’t use on paper that ultimately compromise data quality.

8. Survey costing

The real costs of vaccination coverage surveys, with any methodology, are largely undocumented and therefore ill-understood. Although national surveys are relatively practical and inexpensive, surveys powered for precise routine immunization coverage estimates at sub-national level are among the more complex and costly endeavors. Some real expenditure data were presented along with a hypothetical costing study. It was noted that costs will vary enormously between settings and all participants agreed there is a need to study a compilation of survey budgets and sample information from surveys to understand the trade-offs and cost drivers in different settings. Better documentation of the time until results are available and concrete examples of how survey results drive immunization program management decisions would also help inform the perceived cost-effectiveness of investing in surveys.

9. Reporting and using survey data

9.1. Standardized reporting

In contrast to the standardized questionnaires, protocols, and reports utilized by DHS and MICS [39], [40], vaccination coverage survey reports rarely follow a standard form, making comparisons across countries more challenging. WHO has developed a suite of programs to calculate vaccination coverage survey indicators in a documented, standardized and replicable manner. The programs are known as the Vaccination Coverage Quality Indicators (VCQI) [41]. Indicator definitions and software specifications are included in the VCQI documentation. VCQI’s first version was released in 2016 and it continues being updated as more surveys use it. Meeting participants encouraged WHO to also develop a standard EPI survey report template perhaps drawing from examples used by DHS, MICS and Standardized Monitoring and Assessment of Relief and Transitions (SMART) surveys [42].

9.2. Timely reporting

To be programmatically useful, delays in communicating results need to be shortened. Final DHS and MICS survey results are often released at least one-year following the completion of field work. Vaccination coverage survey reports are also often finalized (if at all) several months after survey field work. Reporting may be delayed to the extent that results are no longer useful for EPI. To better ensure timely availability of results, MICS has begun producing a preliminary report focused on the final tabulated results more so than narrative content. Participants encouraged greater attention towards improving timely release of survey results and documentation of obstacles to this. EPI surveys that name and code response variables in a manner consistent with recommendations in the VCQI (see above) documents should be able to generate preliminary results rapidly using VCQI software [41].

9.3. Analytical tools and additional analyses

Secondary analyses from surveys already conducted are often not done due to absent foresight, absent availability of databases or poor data documentation, limited analytical capacity, and lacking standardized survey documentation. Countries were encouraged to conduct further analyses of existing survey data beyond estimating vaccination coverage. The WHO Survey Manual includes guidance on analyses such as comparisons between sub-groups, timing and source of vaccination, drop-out rates, and missed opportunity for vaccination. Several of these analyses are already included in VCQI. Other uses of survey data include comparisons of survey results with other data sources, insights into administrative data weaknesses, trend analysis, and may include small area estimation. Meeting participants encouraged WHO to publicize existing tools such as its inequality analyses [43], [44] and explore online tools to support capacity-building around survey analyses. To the latter end, in 2017, WHO successfully trained professionals from a variety of countries using a distance learning approach [45], [46].

9.4. Data sharing

In contrast to DHS and MICS for which public use files are the norm, few vaccination coverage survey datasets are made publicly available. Those that are available often lack adequate documentation i.e., Information about sampling/weighting/design, data dictionary, analysis code. In line with current models used by DHS, MICS and the Bill and Melinda Gates Foundation, it was recommended that countries be supported to include data sharing agreements in memorandums of understanding and protocols during survey planning and contracts and that datasets be made available to facilitate secondary analysis. Participants encouraged WHO to prioritize improvement of standards and technology for sharing anonymized datasets and documentation.

9.5. Using data for decision-making

When used effectively, vaccination coverage survey data can be a powerful tool to inform decision-makers and educate stakeholders as well as track progress in immunization service delivery. Accomplishing these goals requires that results be communicated in a timely manner, understood, accepted and used. The temptation for ministries of health or EPI teams to reject some survey results (e.g. those suggesting suboptimal EPI performance) needs to be overcome by demonstration of survey quality and reliability. Concerned with challenges to immunization program ownership in the face of trends to outsource survey implementation, participants encouraged the development of guidance for EPI program managers focused on linking survey findings to other data and potential actions (Table 2, Table 3). To this end, WHO is working to develop guidance and learning initiatives targeted to immunization decision-makers in countries [47], [48].

Table 3.

Recommendations to WHO (as a facilitator or lead).

Status update as of 7 July 2018. • Finalize and publish the revised 2015 Vaccination Coverage Cluster Survey Reference Manual o The content of the Manual remained mostly unchanged. Editorial changes were made for clarity in some sections, such as those on weights (section 6.2 and annex J) and adding a “map to the Manual” in the form of a table with survey steps and where to find these topics the Manual. The only substantial change was the removal of the former annex M on calculating coverage by 12 months of age, given the doubtful validity of vaccination recall. Thus, the calculation of vaccination by 12 months assuming that children without cards would be vaccinated just like those with documented evidence of vaccination was considered undependable. Annex M was replaced with more details on suggested graphical display of coverage results. o Final version available at http://www.who.int/immunization/documents/who_ivb_18.09/en/ • Provide guidance to ensure countries have a good rationale for doing a survey, and that those without sufficient rationale are discouraged. o Chapter 1 of the Vaccination Coverage Cluster Survey Reference Manual includes guidance. o Survey Scholar distance learning initiative, Module A1, focused on this issue along with designing a survey concept note; 130 participants from various countries successfully completed this module in 2017 [43]. • Continue strengthening collaborations between the Expanded Programme on Immunization (EPI), the Demographic and Health Surveys (DHS), the UNICEF Multiple Indicator Cluster Survey (MICS), and the Standardized Monitoring and Assessment of Relief and Transitions (SMART) teams. o WHO-led “Expert consultation on estimating vaccination-related indicators in multipurpose household surveys” conducted in April 2018. Meeting materials and report available upon request at vpdata@who.int. • Lead conversations and reflection on how to translate data into knowledge for decision making, including discussing early on how the coverage survey will be analyzed and used. o Ongoing WHO technical assistance to countries. o Seminar on vaccination coverage surveys held at Gavi, the vaccine alliance in 2017. o Survey Scholar distance learning initiative, Modules A1, A2 and A3 covered the uses of vaccination coverage survey estimates for immunization program; 130, 90 and 75 participants successfully completed each module, respectively, in 2017 [43]. • Develop or identify methods/tools for collecting rapid assessment (or for estimation/validation) useful vaccination coverage at the district and local levels, that would be more practical and affordable than doing surveys in all districts. o Experiences from PAHO using rapid monitoring for routine immunization, and a variety of countries worldwide using rapid monitoring following vaccination campaigns ongoing. • Consider providing further guidance on oversampling selected areas or populations as needed for decision making, rather than all or no district-level strata. o Partially covered in Module A1 of the Survey Scholar distance learning initiative [45]. o WHO document to be produced. • Create a set of quality criteria that can be used to grade survey results to better inform the users on potential limitations or survey quality issues. o Work ongoing on a survey checklist (WHO in collaboration with UNICEF). This list will be first used for new surveys to be considered for the 2018 or 2019 session of the WHO/UNICEF Estimates of National Immunization Coverage (WUENIC) [48]. • Document/compile case studies of what went right and wrong when implementing vaccination coverage surveys, mainly when using the 2015 WHO Survey Manual. o Work ongoing. Draft document for Burkina Faso 2016 experience available. o Other summaries included in this meeting presentations/report. • Document/compile budget and sample information from surveys to demonstrate budget/sample size trade-offs and drivers of costs in different settings. o Work ongoing • Develop standard questions on household and demographic characteristics, as well as barriers and reasons for non-vaccination or knowledge of the immunization services (see below, under operational research) noting that these contents will still need to be adapted and tested in each country. o Work ongoing, led by UNICEF under the umbrella of the KAP for immunization working group. • Examine how to ensure health facility visits are worth the effort (e.g. when should they be done? can you collect other information besides vaccination status during the same visit?). o Experiences being compiled: Bolivia (2013), Senegal (2017), Ethiopia (DHS 2016, JSI in 3 districts 2017) and to be included in a vaccination coverage survey in Madagascar 2018 • Develop a minimum standard template for EPI survey reports to standardize critical outputs -Tables could mirror DHS and MICS standards to allow for easy comparability. o This was discussed in the WHO-led “Expert consultation on estimating vaccination-related indicators in multipurpose household surveys” in April 2018 o A draft White Paper, provisionally entitled “Harmonizing vaccination coverage measures in household surveys: A primer”, is being circulated among partners and participants from the Survey Scholar training for comments. • Define a set of additional survey analysis (beyond coverage) and how to best standardize them. o Several standard secondary analyses included in the revised WHO Vaccination Coverage Cluster Survey Reference Manual [12] and software developed for conducting these analyses (VCQI). o PAHO’s “Toolkit to monitor vaccination coverage and preventive chemotherapy coverage”, module 6, includes several additional analyses [18]. • Work with immunization programs so they can identify their needs for additional/secondary survey analyses. o WHO-led “Hands-on training workshop for the comprehensive analysis of vaccination coverage” held in Nepal in 2017 [46]. o Partially covered in the Survey Scholar distance learning initiative, modules A1 and A3 [45]. o Work ongoing with countries requesting technical assistance on vaccination coverage surveys from WHO or UNICEF. • Describe and explain what are the differences between the DHS, MICS and EPI methodologies, including details in indicator calculation, and in the way results are presented. o Discussed in the WHO-led “Expert consultation on estimating vaccination-related indicators in multipurpose household surveys”, April 2018 • Compile an exhaustive list of possible sources of bias in vaccination coverage surveys, which countries can use as a checklist of issues to discuss in their report limitations or strengths sections. o Partially covered by a survey checklist being developed by WHO in collaboration with UNICEF. o Covered on Module A2 the Survey Scholar distance learning initiative; 90 participants successfully completed this module in 2017 [43]. o Partially covered in PAHO’s “Toolkit to monitor vaccination coverage and preventive chemotherapy coverage”, module 5 [17] and in other documents [5]. • Develop guides/toolkits to help interpret results and highlight actions to be taken based on the survey findings. o Work ongoing. o Covered on Module A3 the Survey Scholar distance learning initiative; 75 participants successfully completed this module in 2017 [43]. Module A3 being re-run (June-July 2018) o WHO is developing a handbook on data quality and use for immunization that includes survey and it should be published by the end of 2018. o Partially covered in PAHO’s “Toolkit to monitor vaccination coverage and preventive chemotherapy coverage”, modules 1, 5 and 6 [17]. • Explore using online tools to support survey planning and analysis, including existing tools that explore analysis of disparities such as WHO’s annual analysis on inequalities [41] and UNICEF Equist [42]. o Collaboration with UNICEF-led immunization equity working group. • Improve standards and technology for sharing datasets and documentation. o WHO is exploring this area.

10. Conclusions and recommendations

In conclusion, the main lessons learned from the initial use of the updated WHO Vaccination Coverage Survey Manual were less about the manual itself and more about operational issues and the pressing need for WHO and its partners to help implement the updated guidance. Also, more efforts are needed to better collaborate with institutions with statistical expertise and develop a cadre of practitioners with sufficient understanding of probability sampling and how to improve survey quality. Meeting participants also highlighted the need to bridge the desire for district-level coverage estimates with the operational and cost implications of undertaking surveys with district-level representativeness. This meeting led to several recommendations for implementing better quality vaccination coverage surveys (Table 2). As per priority operational research topics, these were grouped around sampling, [vaccination] recall, survey design and instruments, data collection, and analysis/use of survey results (Box 2 and Fig. 1).

Box 2.

• •
  Sampling

  • o
    Use of gridded data and other computer-assisted approaches for sampling.
  • o
    Studies to quantify who (e.g., nomadic populations, persons without fixed houses in urban areas) is missing from sampling frames and develop methodologies to improve the sampling frames’ coverage.
• •
  Recall

  • o
    Study the extent and impact of recall response bias, especially in low and middle-income countries.
  • o
    Studies seeking to understand factors influencing poor recall in different contexts. This could be done by first identifying statistically significant characteristics (from persons being interviewed and from interviewers) associated with poor recall that could then guide a qualitative or mixed methods study.
• •
  Survey design and instruments

  • o
    Extent and impact of survey tools and interviewers in recall response bias, especially in low and middle-income countries.
  • o
    How to formulate questions about vaccine coverage, particularly in the case of recall, including use of visual cues.
  • o
    Defining a standard set of KAP questions that could be added to surveys, based on proper social science methodologies.
• •
  Data collection

  • o
    Test the accuracy of data entry using different electronic collection formats and platforms; develop evidence-based recommendations for interfaces and for data entry protocol (e.g., enter the dates and take one or more clear photos; after data entry have a partner read the dates from the HBR out loud while you review them on the screen).
  • o
    Study the role of publicizing the survey in the selected clusters ahead of time to increase the number of HBRs available, provided that this activity will not discourage those without HBRs from participating
  • o
    Improve recommendations on taking photos during data collection and how to better use them.
• •
  Analysis/use of survey results

  • o
    Further develop small-area estimation methods as a possible alternative to estimating district and other local levels vaccination coverage.
  • o
    Study statistical adjustment approaches for survey-based vaccination coverage estimates, particularly to address possible bias due to recall (based on respondent characteristics, length since vaccination to recall, etc.).
  • o
    Test interventions to promote use of data for evidence-driven decision making.

Given the importance of having accurate vaccination coverage estimates, WHO will continue working with its Member States and partners to improve the quality, accuracy and use of vaccination coverage survey estimates in support of immunization programs.

All meeting materials, including presentations and additional files are available upon request at vpdata@who.int.