LiST as a catalyst in program planning: experiences from Burkina Faso, Ghana and Malawi

Abstract

Background African countries are working to achieve rapid reductions in maternal and child mortality and meet their targets for the Millennium Development Goals (MDGs). Partners in the Catalytic Initiative to Save One Million Lives (CI) are assisting them by providing funding and technical assistance to increase and accelerate coverage for proven interventions. Here we describe how the Lives Saved Tool (LiST) was used as part of an early assessment of the expected impact of CI plans in Malawi, Burkina Faso and Ghana.

Methods LiST builds on country-specific demographic and cause-of-death profiles, and models the effect of changes in coverage for proven interventions on future levels of mortality among children less than 5 years of age. We worked with representatives of Ministries of Health and their development partners to apply LiST to assess the potential impact of CI plans and coverage targets, generating a short list of the highest-priority interventions for additional scale-up to achieve rapid reductions in under-5 mortality.

Results The results show that in each country, achieving national coverage targets for just four or five high-impact interventions could reduce under-5 mortality by at least 20% by 2011, relative to 2006 levels. Even greater gains could be obtained in Burkina Faso and Ghana by scaling up these high-impact interventions to 80%.

Discussion LiST can contribute to the development of stronger programmes by identifying the highest-impact interventions in a given epidemiological setting. The quality of LiST estimates is dependent on the available data on coverage levels and causes of death, and assumes that the target levels of coverage are feasible in a given context while maintaining service quality. Further experience is needed in the feasibility and usefulness of LiST as part of the program planning process at district and subdistrict levels.

Introduction

Earlier papers in this issue have described the Lives Saved Tool (LiST) and how it can make state-of-the-art evidence about intervention effectiveness available to policy makers and programme planners as a basis for sound decision making about how to allocate resources to achieve the Millennium Development Goals (MDGs). Here we describe early applications of the tool in the context of the Catalytic Initiative to Save One Million Lives (CI).

The CI aims to accelerate coverage of interventions proven to be efficacious in reducing mortality from major causes of mortality in children under the age of 5 years, resulting in rapid reductions in child mortality at scale.¹ As part of the CI, UNICEF is implementing the ‘Integrated Health System Strengthening’ project with support from the Canadian International Development Agency (CIDA) to accelerate reductions in under-5 mortality in six countries in sub-Saharan Africa (Ethiopia, Ghana, Malawi, Mali, Mozambique and Niger). In a complementary effort being implemented under the CI ‘umbrella’, WHO, UNICEF and UNFPA are working with support from the Bill & Melinda Gates Foundation (BMGF) to accelerate coverage of proven interventions and reduce child mortality in Burkina Faso, Malawi and Mozambique. These efforts aim for reductions in under-5 mortality by 2011 that are at least 20% greater in the project acceleration areas than in other areas of each country where the CI is not being implemented. By late 2008, all countries receiving CI funds had developed specific plans for scaling-up interventions within the context of their respective national strategies for maternal, newborn and child health.

The Institute for International Programs (IIP) at the Johns Hopkins University is responsible for conducting independent effectiveness evaluations of CI activities at country level in collaboration with in-country research partners. These evaluations are prospective, require innovative evaluation designs and are guided by principles that differ from most traditional evaluations: (i) the evaluations are based on a stepwise approach that includes early assessments of the programme design and the likelihood of achieving impact; (ii) use of LiST to provide detailed input to programme planners allowing them to make changes to initial plans and increase the probability of impact; (iii) independence from programme implementers with responsibility for providing regular feedback to improve programme effectiveness; and (iv) designs that are flexible and realistic, recognizing that good programmes change over the course of their implementation in response to contextual factors and ongoing feedback from internal and external monitoring and evaluation.²

In 2007 and 2008, country teams from Burkina Faso, Malawi and Mozambique submitted proposals for support from the Catalytic Initiative that described plans to accelerate coverage for proven interventions and produce reductions of at least 20% in rates of under-5 mortality in the project areas by 2011, over and above those achieved in areas of each country not implementing the accelerated CI approach. Country teams proposed to supplement existing government plans by accelerating the scale-up of between 13 (Burkina Faso) and 20 (Malawi and Ghana) interventions simultaneously in large areas of each country, raising concerns among CI leaders (CIDA, BMGF, the Doris Duke Foundation, UNICEF and WHO) about the feasibility of achieving targeted increases in coverage during the time frame of the project. It was therefore proposed that Ministries of Health and their CI counterparts be invited to participate in applications of LiST with the aim of narrowing the focus of their project plans to a smaller set of interventions that together could achieve the mortality reduction targets of the CI. We report here on the first three of these country applications in Malawi, Burkina Faso and Ghana.

Methods

IIP prepared for the visits by building preliminary LiST scenarios using the most recent available estimates of country-specific under-5 deaths by cause³ and coverage targets as presented in the country-specific CI plans. In Malawi, model inputs were national coverage estimates from a Multiple Indicator Cluster Survey (MICS) conducted between July and November 2006⁴. In Burkina Faso, coverage estimates for most interventions were available from a 2006 MICS.⁵ The national plan in Burkina is based on two or more antenatal care visits rather than the global consensus indicator of more than four visits, and this indicator had not been calculated for the 2006 MICS. We therefore adjusted the 2006 estimate for one or more visits using the reported relationship between at least one and two or more visits reported from the 2003 DHS. In Ghana the scale-up will focus in the Central, Northern, Upper East and Upper West Regions, for which no recent coverage data were available at the time of the LiST workshop. National data sets had limited applicability due to the huge variability by geographic region in child mortality burden and intervention coverage levels. We therefore used coverage estimates from a 2007 survey conducted in the Central, Northern, Upper East and Upper West Regions.⁶ The model was applied only to the Upper East Region because there was only time in the workshop to run one set of models.

Table 1 summarizes the participants and the context for the LiST application in each country. In each setting, the IIP team asked in-country partners to identify a few individuals familiar with the national plans and comfortable with computer applications to work with the IIP team to apply LiST using the interventions and targets specified in the country plan. This smaller group worked together to apply LiST and prepare a summary of the results for presentation to the Ministry of Health leaders and a larger group of stakeholders. Uncertainty estimates were not presented as this aspect of LiST is still under development.

Table 1.

Context for LiST applications in Malawi, Burkina Faso and Ghana

	Malawi	Burkina Faso	Ghana
Focus area for CI activities (total population)	10 districts (5.1 million)a	9 districts (2.4 million)b	37 districts (3 million)c
Area used in LiST application (total population)	National (14 million)d	National (14 million)d	8 districts (1 million)c
Date of LiST application	December 2008	February 2009	March 2009
Setting	Residential workshop to design independent evaluation of CI	MOH offices	Meeting with MOH and other partners convened by UNICEF
Advance team working with IIP/JHU	UNICEF/WHO	MOH (Family Health Division, Health Information System, National Statistics Institute) UNICEF, UNFPA, WHO	Ghana Health Service, MOH, UNICEF, WHO
Who presented LiST results to stakeholders?	UNICEF	IIP/JHU	Ghana Health Service
Participants in stakeholder meeting	MOH (all levels, Director to community health worker), WHO, UNICEF, independent evaluation team, BMGF, CIDA	MOH leaders and partner stakeholder	Ghana Health Service, MOH, WHO, UNICEF, BMGF, CIDA

^a2008 Malawi population census.

^bMinistry of Health, Burkina Faso (Extrapolation based on 2006 census).

^c2008 estimates from the Ghana Statistical Service (http://www.statsghana.gov.gh/docfiles/gh_figures_2008.pdf).

^d2006 estimates from the United Nations Population Reference Bureau.

Some interventions in the country plans could not be included in the LiST modelling exercise. Table 2 shows these interventions and the reasons for their exclusion. In addition, coverage levels for DTP, measles, tetanus and Hib (where implemented) vaccinations were estimated at over 80% in 2006 in all three countries;⁷ the LiST applications did not assume any further scale-up for these interventions as targets had already been achieved. In Malawi, vitamin A coverage was >80% in 2006⁴ and we assumed coverage would be sustained at this level through 2011.

Table 2.

Reasons why interventions present in country plans were not modelled in LiST application

	Malawi	Burkina Faso	Ghana
Evidence of effectiveness in reducing child mortality rates not sufficient to warrant inclusion in LiST	De-worming, family planninga	Maternal vitamin A supplementation, IPTp	Family planninga
Evidence of effectiveness exists, but intervention not yet incorporated in LiST	Therapeutic feeding for severe malnutritionb	Caesarian section, breastfeeding within 1 hour of birth	Caesarian section, Intrapartum care
Country data on coverage not adequate or compatible for use in LiST, or target coverage not set in programme	Complementary feeding, kangaroo mother care		Complementary feeding, kangaroo mother care
Not part of CI acceleration programme	Interventions to prevent mother-to-child transmission of HIV

^aFamily planning reduces the number of child deaths; evidence for an effect of birth spacing on child mortality is under review and is not currently reflected in LiST.

^bNow incorporated into LiST.

Coverage targets play an important role in determining the relative contribution of each intervention to mortality reductions as modelled by LiST, and considering alternative scenarios with higher and lower targets is an essential part of a full LiST application during the pre- or early-implementation period of a programme. The early applications reported on here began with the intervention-specific coverage targets defined in the national plan for the CI acceleration, most of which were generated through a workshop-based application of the UNICEF-supported Marginal Budgets for Bottlenecks (MBB) tool.⁸

In-country presentations of the results of applying LiST to initial implementation plans focused on: (i) the total percent reduction in under-5 mortality between baseline (around 2006) and 2011 if national coverage targets were achieved; and (ii) the contribution of individual interventions to the modelled reductions in child mortality, permitting a comparison of the relative contribution of different interventions to the achievement of the plan’s outcome objectives. Here, we also present the total and intervention-specific reductions in under-5 mortality that would result from achieving 80% coverage with the highest-impact interventions in each country to illustrate the maximum impact that could be achieved by accelerating coverage for the most effective interventions focused on the major causes of child death in each country.

Results

Table 3 presents the number of interventions included in the country scale-up plan, the number of interventions modelled and the overall level of child mortality reduction between 2006 and 2011 if national coverage targets are achieved. The table also presents the interventions that will have the greatest impact on under-5 mortality in each country if national coverage targets are achieved, representing a cumulative reduction of at least 20% relative to baseline, as well as their baseline and target levels of coverage. All projections assume that 2006 coverage levels for childhood immunizations are sustained through 2011.

Table 3.

Summary of country plans before and after LiST applications

	Malawi	Burkina Faso	Ghana
Programme focus and impact
Number of interventions included in acceleration plan	18	13	20
Number of interventions modelled	13	9	17
Percentage reduction in under-5 mortality if all targets in plan achieved for modelled interventions	36	24	26
Number of interventions required to achieve mortality reduction of ≥20% reduction in under-5 mortality, as modelled by LiST	4	5	5
Minimum set of interventions to achieve mortality reduction of 20%, with current and target coverage levels
Pneumonia treatment with antibiotics	✓(29; 67)	✓(30; 50)	✓(33; 60)
Diarrhoea treatment with ORS and zinc	✓(55a; 85)	✓(4a; 60a)	✓(42a; 60)
Malaria prevention with insecticide-treated nets	✓(23; 69)	✓(10; 70b)	✓(40; 55)
Malaria treatment with ACTs	✓(27; 69)	✓(48; 57)	✓(65; 70)
Vitamin A supplementation		✓(67; 90)
Improved sanitation			✓(18; 70)

^aORS only; zinc not included in national policy.

^bOr indoor residual spraying.

Figure 1 shows trends in rates of under-5 mortality for the three countries if coverage levels for all interventions in the national plan are sustained at 2006 levels, and for two scenarios generated by LiST: (i) trends in mortality if coverage targets in the national plan are achieved; and (ii) trends in mortality if the minimum number of interventions in the national plan that could achieve a 20% reduction in under-5 mortality by 2011 are scaled up to 80% coverage. Under-5 mortality rates increase slightly in the null scenario (if baseline coverage levels are maintained) due to historical trends in nutritional status and HIV prevalence among women of reproductive age.

Figure 1.

Trends in rates of under-5 mortality from 2006 to 2011 in Malawi, Burkina Faso and Ghana if 2006 coverage levels remain unchanged (baseline) and for two LiST scenarios: (i) coverage targets in the national plan are achieved; and (ii) the minimum number of interventions in the national plan that could achieve a 20% reduction in under-5 mortality by 2011 are scaled up to 80% coverage

Full implementation of the national plan in Malawi is projected to result in a 36% reduction in under-5 mortality—from 123 in 2006 to 78 in 2011. Four interventions account for 55% of this reduction: correct treatment of childhood pneumonia, diarrhoea and malaria, and the use of insecticide-treated nets for the prevention of malaria. The second scenario (achieving 80% coverage for these four interventions) would result in a lower level of mortality reduction (31%) than the current national plan in this 5-year period, because the plan sets a target of 85% for ORS and zinc in the treatment of diarrhoea, which is the single largest cause of under-5 deaths, and the scenario target of 80% represents a reduction from this level of coverage.

Full implementation of the national plan in Burkina Faso is projected to result in a 24% reduction in under-5 mortality—from 170 in 2006 to 129 in 2011. Five interventions account for over 80% of this reduction: treatment of childhood pneumonia, diarrhoea and malaria, the prevention of malaria at household level through either insecticide-treated nets or indoor residual spraying, and vitamin A supplementation. Achieving 80% coverage for these five interventions would reduce under-5 mortality from 2006 levels by 40%, saving a total of about 20 000 additional child lives relative to the current national plan in this 5-year period.

Full implementation of the national plan in Ghana is projected to result in a 26% reduction in under-5 mortality—from 107 in 2006 to 79 in 2011 in the Upper East Region. Five interventions account for over 75% of this reduction: treatment of childhood pneumonia, diarrhoea and malaria, the use of insecticide-treated nets for the prevention of malaria, and improving sanitation at the household level from a baseline of 18–70%. Achieving 80% coverage for these five interventions would reduce under-5 mortality by 37% in this 5-year period, saving a total of ∼500 additional child lives relative to the current national plan in the Upper East Region. A larger number of deaths would be averted in the 54 districts that comprise the CI intervention area, but available data are insufficient to support an estimate.

Responses from Government and Partners

In all three countries, responses to the LiST results highlighted the need for country-specific estimates of intervention effectiveness as inputs to programme priority setting. LiST can stimulate reassessments of existing coverage targets for the interventions with the highest potential for rapid mortality reduction. The ability of LiST to produce estimates of the point at which increases in coverage for specific interventions would result in measureable decreases in mortality was also appreciated.

Some stakeholders, and especially donor representatives, were struck by the fact that many more interventions were included in the scale-up plans than was necessary to achieve the mortality reduction target, leading to discussions about how to fine-tune the plans to give greater focus to the highest-impact interventions and achieve the greatest impact, while at the same time increasing feasibility of implementation. In Malawi, where the results were presented at the start of a week-long workshop convened to plan for the independent evaluation of the CI, high levels of concern about achieving coverage targets while maintaining quality resulted in a reorganization of the workshop programme to allow a separate working group of MOH personnel and their UN counterparts to focus on how to refocus implementation plans and reduce barriers to rapid acceleration of coverage for these interventions. In Burkina Faso, the advance team included high-level representatives of the Ministry of Health, and several targets were changed to result in higher impact during the LiST application (original results not shown here) and prior to the presentation of the results to a larger group of stakeholders. A stakeholders’ meeting in Ghana provided a forum for presentation and discussion of the LiST results. In all three countries, the LiST application and resulting discussions highlighted the fact that national mortality reduction targets could be achieved if coverage for the existing immunization schedule was maintained and programmes were scaled up to achieve national coverage targets for up to five additional interventions including correct treatment of childhood diarrhoea, pneumonia and malaria. The LiST applications also sparked discussions about how barriers to rapid increases in coverage could be addressed.

Participants in the advance teams in each country were able to use LiST independently by the close of the preparatory period. In all three settings, local individuals took the initiative to explore alternative scenarios that went beyond the estimation of impact for the CI plans and targets. Members of the advance teams in two of the three countries have applied LiST independently since being exposed to it in the workshop. The ability to try out alternative scenarios—setting higher or lower targets for particular interventions and examining the effects on under-5 mortality—was particularly appreciated.

Discussion and implications for programming

Applications of LiST as a part of programme planning exercises in Malawi, Burkina Faso and Ghana showed that in each country a set of four or five proven interventions could reduce under-5 mortality by at least 20% in a period of 5 years if existing national coverage targets are able to be achieved without jeopardizing the quality of the interventions. Even greater gains could be obtained in Burkina Faso and Ghana if the five most effective rapid-mortality-reduction interventions in each country were scaled up to coverage levels of 80%. These findings assume that existing high coverage levels for childhood immunizations in these countries will be maintained.

These conclusions are limited in several important ways. First, the estimates of lives saved are presented here as point estimates despite obvious uncertainties resulting from variability in baseline mortality levels (estimated from survey-based birth histories), in cause of death distributions (modelled for most countries), in baseline coverage levels (also estimated through surveys or in some cases from information systems), and in parameter specifications (e.g. assumptions about effect of specific interventions on cause-specific deaths), among other factors. The magnitude of the differences among interventions in the numbers of lives saved are sufficiently large in most cases to make the implications for programming clear, but further work is needed to address this issue. Second, the estimates produced by LiST assume that interventions will be delivered at levels of quality sufficient to produce effects on mortality equivalent to those assumed in the model. Third, coverage targets for some interventions in these countries were unrealistically high. In Ghana, for example, the national plan calls for an increase in improved sanitation from 18 to 70%.

Despite these limitations, these first applications of LiST highlight important considerations for the design of programmes to accelerate child survival. In all three countries, many more interventions are included in national plans than are needed to achieve the planned impact. Given severe resource constraints, it may not be possible to implement all currently planned interventions at scale simultaneously. A tighter focus on the highest-impact interventions could reduce the complexity of intervention packages and increase the probability that coverage and impact targets are achieved. Governments and their partners should assess the trade-offs between the rapid achievement of mortality targets by accelerating coverage for the highest-impact interventions and slower gains across a broader range of interventions that are less effective in their contexts.

These exercises also make clear, however, that LiST is only a tool and cannot in and of itself strengthen maternal and child policies and programmes. At the time of this writing (June 2009), none of the three countries has changed their formal targets or plans for accelerating coverage to reflect the outcomes of the LiST application. MOH coauthors highlight the challenges of the policy process and the barriers to rapid action based on LiST results (Box 1). In Ghana, for example, revising targets within the current 5-year plan of work (2007–11) would require revisiting not only the complicated process of target setting, but also the ‘triggers’ for funding that donors have set based on those targets. Overcoming this and other barriers to evidence-based programme planning will require that LiST is embedded into the broader cycle of programme planning and monitoring—not an easy thing to do. Despite the difficulties in changing the formal country plans to reflect the LiST results, MOH representatives in all three countries did report that they have changed the implementation plans for the scale-up to give greater emphasis to the four or five interventions within their existing programmes with the greatest impact.

Box 1 Challenges in incorporating LiST results into program policies, plans and targets.

Coauthors from Ministries of Health in Malawi, Burkina Faso and Ghana (HN, GYA, NN, SKF) identified three factors that are likely to affect the speed and extent to which LiST results are taken up by governments and their counterparts and reflected in changes in official plans and resource allocation:

1. Political commitment. National public health policies and plans are shaped by strong forces other than scientific evidence, including technical clarity about what changes need to be made and their costs and benefits to the population, and competition for attention and resources among disease- and intervention-specific advocacy groups, some of whom are backed by enormous resources. LiST can contribute to efforts to put maternal and child survival at the top of the priority list at national and district levels, especially once the costing module is available.

2. Consensus. National plans, and especially sector-wide approaches (SWAps), are the result of an intensive collaborative process between governments and donors, with release of funds increasingly tied to achievement of milestones and targets. Changes reverberate through this system and are therefore difficult to make once the plan is approved.

3. Timing. National plans run in three- to five-year cycles, and once set are difficult to change as described above. District plans are usually developed on an annual basis and can therefore change more rapidly if supportive national policies are in place. LiST applications should be embedded in the policy and program planning processes at national and district levels.

These early applications of LiST also highlight important gaps in essential data for programme planning and evaluation at country level. First, countries and districts need accurate information on the causes of child deaths in order to generate accurate LiST results that can be used to prioritize interventions and allocate resources. When feasible, these data should be collected routinely through verbal autopsies conducted as a part of nationally representative household surveys; where they are not available, the most recent cause-of-death profiles developed by WHO and the Child Health Epidemiology Reference Group can be used in their stead.⁹ Second, countries and districts need frequent measures of coverage for all the interventions included in their national plans. In Burkina Faso, the most recent available coverage estimates for some interventions was 2003. Efforts by national and international partners to conduct household surveys at more frequent intervals and to maintain consistent indicator definitions across surveys to allow trend analyses are important steps; making those surveys feasible and affordable for more frequent use at national and even district levels is a continuing priority.

LiST must not be viewed as a ‘magic bullet’ that will solve all problems in planning and implementing strong public health programmes in child survival. First, the LiST applications reported here occurred in the context of specific programmes focused on accelerating intervention coverage and reducing child mortality. There are other worthy public health goals for which LiST may hold less immediate relevance. Second, those applying LiST as a part of programme planning exercises, even within child survival programmes, should keep in mind that the resulting estimates of lives saved assume that interventions that are currently reaching high proportions of the population will need to be sustained. Third, the potential contributions of LiST as part of an ongoing programme management process must be documented further at national and regional levels, and further experience is needed at district and subdistrict levels where most operational planning actually occurs.

Funding

This work was supported by the Bill and Melinda Gates Foundation through grants to the World Health Organization to support the rapid scale-up of proven interventions in Burkina Faso, Malawi and Mozambique and to the US Fund for UNICEF to promote evidence-based decision making in designing maternal, neonatal and child health interventions in low- and middle-income countries, and by the Canadian International Development Agency through a grant to the Institute for International Programs at the Johns Hopkins Bloomberg School of Public Health for real-time monitoring of child mortality under the Catalytic Initiative to Save One Million Lives.