The value of demonstration projects for new interventions: The case of human papillomavirus vaccine introduction in low- and middle-income countries

ABSTRACT

Demonstration projects or pilots of new public health interventions aim to build learning and capacity to inform country-wide implementation. Authors examined the value of HPV vaccination demonstration projects and initial national programmes in low-income and lower-middle-income countries, including potential drawbacks and how value for national scale-up might be increased. Data from a systematic review and key informant interviews, analyzed thematically, included 55 demonstration projects and 8 national programmes implemented between 2007-2015 (89 years' experience). Initial demonstration projects quickly provided consistent lessons. Value would increase if projects were designed to inform sustainable national scale-up. Well-designed projects can test multiple delivery strategies, implementation for challenging areas and populations, and integration with national systems. Introduction of vaccines or other health interventions, particularly those involving new target groups or delivery strategies, needs flexible funding approaches to address specific questions of scalability and sustainability, including learning lessons through phased national expansion.

Introduction

Demonstration projects or pilots of new public health interventions are widely used.¹ Geographically and time-limited, they aim to build learning and capacity to inform country-wide implementation.^1,2 Between 2007 and 2011, demonstration projects were the main approach to delivering new vaccines against human papillomavirus (HPV) to prevent cervical cancer in low-income and lower-middle-income countries (LLMICs).^3,4 Initial pilots, with vaccines donated by Merck & Co., Inc. or GlaxoSmithKline Biologicals to governments and external partners, allowed countries to gain experience vaccinating adolescent girls, who were not routinely targeted for immunisation.^5,6 With the exception of Rwanda and Bhutan, vaccine donations for national delivery were not generally available, so demonstration projects enabled resource-poor countries to gain HPV vaccination experience.^7,8

From late 2012, Gavi, the Vaccine Alliance, began supporting HPV vaccination demonstration projects or national programmes if countries had prior experience vaccinating girls aged 9–13.⁹ The principle objective of Gavi support was to ‘learn by doing’. Gavi provided 2-year funding for vaccines and operational costs to allow time to assess delivery strategies and potential integration with other adolescent services, develop tools, and prepare applications to Gavi for national program funding. By mid-2015, Gavi had approved demonstration project funding for 25 countries, of which 6 (24%) had already conducted at least one pilot, while another 3 were approved for national support.⁹ Despite 55 pilots or demonstration projects being completed before 2015, only 7 countries – and 3 more in 2015-2016 - transitioned from demonstration to national provision.^2,10

This article aims to examine the value of HPV vaccination demonstration projects to date, including potential drawbacks and how value for informing national scale-up might be increased, drawing from a review of published and unpublished documents from 37 countries and key informant interviews from 23 countries conducted in 2015.² Countries with data in the public domain are specifically named, while others were anonymised with an identification number. Lessons are relevant for countries intending to introduce HPV vaccination and more broadly for funders supporting introduction of new health interventions in LLMICs.

Value and drawbacks of demonstration projects

Demonstration projects allowed both national and external partners to gain valuable experience in planning and budgeting for a new intervention, enumerating target populations, developing community acceptability and consent procedures for adolescent services, designing and piloting new reporting forms and systems, coordinating with the Ministry of Education for school-based vaccination, and using standardised evaluation tools.^11-13

While countries had nuanced experiences and gained from the ‘learning by doing’ process, lessons from later demonstration projects generally repeated those reported earlier,² indicating that while individual countries might require first-hand experience, sufficient collective learning has been generated.

Several drawbacks of demonstration projects were noted. First, the limited scale of projects did not allow assessment of potential health system integration, particularly as demonstration projects were often implemented outside routine services. Second, communication and social mobilisation activities had to be carefully restricted to avoid perceived inequity among those not receiving vaccination. Third, ‘demonstration project’ was an unclear term for many stakeholders with some communities concluding that the vaccine rather than delivery method was being piloted. Fourth, many were conducted in areas primarily selected for convenience (e.g. with higher routine vaccination coverage, more extensive infrastructure, and better education levels than national averages), potentially providing few lessons applicable to national scale-up. Fifth, the resource-intensive delivery strategies used in demonstration projects were potentially unsustainable once vaccine donations and external support for operational costs ended. Finally, demonstration planning was sometimes as intensive as that for national programming.

Thus, opportunities to explore novel or sustainable delivery strategies were often missed. A focus on project evaluation and demonstrating at least 50% vaccination coverage to secure Gavi support for national introduction, led some countries to choose safe options (e.g., school-based delivery) known for achieving good coverage but potentially too costly to be sustainably expanded nationally. Few projects purposefully included hard-to-reach girls.¹⁴ Projects in only 3 countries simultaneously or sequentially tested different vaccination venues, only 2 simultaneously tested different eligibility criteria,^15,16 and only one tested different vaccination timings.¹⁶ As approximately 13 projects tested HPV vaccination integration (e.g. with tetanus toxoid vaccination, deworming, vitamin A supplementation, health education), learning around combination interventions was largely missed.

Increasing the value of demonstration projects for national scale-up

Many of the drawbacks above seemed to originate from design choices. Value would increase substantially if projects were designed to learn specific lessons and inform realistic national scale-up. For example, the prevailing ‘learning by doing’ objective appeared disconnected from learning for national scale-up, while ‘attaining good coverage,’ as a performance indicator for funding allocation, could create distortions similarly unsupportive of learning for scale-up. Instead, countries could test and cost different delivery strategies in different districts simultaneously, e.g., school-based and health facility-based, to inform sustainable national expansion.¹⁶ Alternatively, donors could fund further economic modeling of different strategies to inform both demonstration projects and national scale-up. Evaluation is critical for any testing approach and cost and coverage evaluations were included in a few demonstration projects. However, evaluations often lacked sufficient evidence to inform decision-making, with data missing or derived from pre-set grants that generally magnified operational costs.

Twenty completed demonstration projects before 2015 without announcing plans to scale-up, 5 of which ceased implementing HPV vaccination without attempting to co-finance vaccine and operational costs, 2 conducted further demonstration projects, and the remainder have not indicated a decision. Implementation often stalled due to vaccine donations ending, funding constraints, or lack of ownership by national immunisation programmes. Demonstration projects may thus delay decision-making or even discourage national scale-up due to the high-cost strategies tested during the demonstration phase.

Lessons from demonstration projects were not always relevant to scale-up. Of seven LLMICs that scaled-up from demonstration to national implementation, only one reported that project lessons were useful for expansion, while another reported national expansion as more valuable for testing possible implementation strategies. Several indicated that demonstration projects were too small to inform national expansion or selected a different delivery strategy than had been tested.

Several countries implemented successive demonstration projects. Of these, 3 used lessons from Gardasil Access Program (GAP) projects to test different delivery strategies as part of Gavi-funded demonstration projects while 6 conducted multiple, sequential non-Gavi projects for reasons unclear from the data. While some scale-up delays are probably due to insufficient national immunisation program engagement with early demonstration projects, an overall discontinuity exists between implementing demonstration projects and national expansion. Longer-term comprehensive planning of projects and expansion, or changing to phased national introduction approaches, could accelerate scale-up.

The pathway from demonstration to scale-up is not straightforward and major challenges to sustainability persist. Financial sustainability was identified as the main barrier to scale-up. Interviewees indicated that the expensive campaign-style delivery and reduced external support during scale-up challenged national expansion efforts. However, the question remains whether demonstration projects may have become a way to delay or discourage commitment to scale-up. With many lessons already from 55 demonstration projects, learning may be saturated and further learning likely to be most effective during national implementation, or in the context of phased national expansion. This would maintain political commitment to scale-up and avoid loss of integration with national health systems.

Conclusions

Well-designed demonstration projects can test multiple delivery strategies, implementation for challenging areas and populations, and integration with national systems.^16-18 Countries implementing new interventions benefit through ‘learning by doing’ and may need initial experiences to hone social mobilisation, delivery strategies, and reporting. However, demonstrations can distract momentum from national introduction and designs that test alternative scalable options are thus crucial. Demonstration projects were valuable when HPV vaccine was first offered to LLMICs before national funding was available. However, with many lessons already documented, few new lessons observed, and additional funding available, their value has decreased. Projects were designed to ‘demonstrate’ whether LLMICs could implement HPV vaccination rather than whether they could implement it sustainably and at scale. While the latter is crucial, countries were reportedly reluctant to risk experimenting and potentially lose funding.

In the example of HPV vaccination, initial demonstration projects quickly provided consistent lessons. However, scale-up is critical to maximise health impact, and further demonstrations could distract momentum and decision-making. Any new demonstration projects should have guidelines that maximise value for national implementation, and flexibility for phased transition to national scale-up, without repeated funding applications. Thus, introduction of vaccines or other interventions, particularly those involving new target groups or delivery strategies, needs flexible funding approaches so that pilots can address specific questions of scalability and sustainability. This could include phased national introduction. Lessons from HPV vaccine introduction are relevant for other potential interventions that may be introduced in the coming years (e.g., MenACWY vaccine, RTS, S vaccine).

Disclosure of potential conflicts of interest

No potential conflicts of interest were disclosed.

Acknowledgements

Authors thank interviewees for their participation and source authors for their contributions.

Author contributions

NH and SMJ wrote the manuscript. NH, SMJ, KG and SK contributed to data collection, analysis, and interpretation. DWJ, DSL, UG, HB and SMJ designed the study and contributed, along with MF, to data interpretation and critical review. All authors approved the version submitted.

Funding

This article is based on research funded by the Bill & Melinda Gates Foundation. The findings and conclusions contained within are those of the authors and do not necessarily reflect positions or policies of the Bill & Melinda Gates Foundation or PATH.