Evaluating global health programmes targeting under-5 mortality: problems and recommendations

Abstract

Reducing the under-5 mortality rate (U5MR) is a key public health priority and remains a persistent challenge in low- and middle-income countries. Many interventions have been developed in attempts to lower U5MR through partnerships between governments, funding partners, researchers, product developers and non-governmental organisations. The well-intentioned desire to use data to set priorities has led to many large-scale evaluations that specify U5MR as a primary outcome. The results of these evaluations are then used to inform investments, policies and advocacy efforts. While important insights can be gained from tracking U5MR at the population level, these evaluations often present serious scientific, pragmatic and ethical challenges. In this article, we outline six problems associated with evaluations targeting a U5MR outcome. We describe why U5MR is primarily informative as an outcome measure for trials of new clinical procedures or treatments designed to cure a life-threatening condition. For interventions that aim to lower U5MR by preventing disease or other potentially fatal events at the population level, or by improving implementation of existing evidence-based interventions, we recommend evaluations that specify intermediate outcomes that offer timelier information and more reliable causal inference. We present limitations to using results of a single study as the basis for policy or funding decisions about an intervention or implementing partner. Finally, we reinforce the need for rigorous methodology, national periodic U5MR measurement, verbal and social autopsies, strong mixed-methods implementation research and reliance on local implementer knowledge.

SUMMARY BOX.

• When evaluating interventions that aim to reduce under-5 mortality rate (U5MR), targeting intermediate outcomes (rather than U5MR) will generally result in greater causal validity, provide increased insight into mechanisms of impact, and have higher statistical power, while lowering both study length and costs.

• It is important to make policy and funding decisions based on multiple studies and sources of evidence to facilitate robust decision-making, promote pragmatic studies, maintain equipoise, encourage transparent representation of results and minimise publication bias.

• Uncontrolled before-and-after U5MR measurements are insufficient to draw conclusions about the effectiveness of an intervention designed to reduce mortality.

• Overemphasis on measuring U5MR limits time, resources and capacity for intermediate outcome measurement, process evaluation, qualitative analysis, quality improvement or implementation research, all of which are critical components of a thorough evaluation.

Introduction

Reducing the under-5 mortality rate (U5MR) is a key public health priority and remains a persistent challenge in low- and middle-income countries (LMICs) despite significant public health advances over the past decades.¹ The U5MR is the probability a child will die before reaching their fifth birthday, expressed per 1000 live births. In places without standardised death reporting or electronic medical records, reliable U5MR data typically come from large population surveys, such as those conducted through the Demographic and Health Survey (DHS) programme² formerly funded by USAID,³ in which women of reproductive age are asked a number of questions including a full history of all of their pregnancies. These surveys are critical to our overall understanding of the state of child health in a country and lead to a rich data source that is widely used for tracking progress against national child health goals,⁴ projection over time,5,7 subnational spatial prediction^{8 9} and risk factor analysis.^{10 11}

With persistently high U5MR in many LMICs,⁶ many interventions have been developed to lower U5MR through partnerships between governments, funding partners, researchers, product developers and non-governmental organisations.12,14 At a high level, these consortia must answer two critical questions: (1) which interventions work in a given context and (2) which implementing partners and strategies should receive funding to scale these interventions? The well-intentioned desire to make data-driven decisions around these two questions has led funders to encourage or require large-scale randomised or quasi-experimental studies directly evaluating the impact of programmes on U5MR.¹⁵ The results of these evaluations are then used to inform funding decisions, together with factors related to organisational capacity, performance, resource availability, scalability, risk, political will, cost effectiveness, opportunity costs and prior evidence of effectiveness. On the surface, this may seem beneficial due to the importance of practical evidence for what works, and U5MR is an intuitively appealing metric for programmes that target child survival. However, there are often serious scientific, pragmatic and ethical challenges with using U5MR as the primary endpoint of an impact evaluation.

Several previous papers have attempted to guide the evaluation of programmes that seek to lower U5MR.16,20 Much like traditional monitoring and evaluation logical frameworks, these works focus on choosing indicators corresponding to inputs, outputs, outcomes and impact, with U5MR nearly universally present as a core impact indicator. Consequently, there is often pressure for evaluations to focus on U5MR as the primary measure of impact.^{15 21} However, these frameworks go into little detail on how this impact should be measured (with the exception of,¹⁷ discussed below) and tend to overlook key issues related to how, when and why to design an evaluation targeting a U5MR outcome.

In this article, we aim to build on the methodological literature by elucidating problems with current practice and making recommendations for designing future evaluations that are rigorous, practical, cost-effective and mutually beneficial to funders, policymakers, programme designers and implementers.

Problems with evaluations targeting under-5 mortality

Six key problems with current practice in evaluations targeting a U5MR outcome are summarised in box 1; in this section, we consider each in detail.

Box 1. Six problems associated with evaluations targeting an under-5 mortality rate (U5MR) outcome.

1. Timeliness: U5MR evaluations often last 3–5 years, meaning that information on effectiveness is substantially delayed; by the time the evaluation is over, the implementation context may be considerably different.

2. Causal validity: There is a misconception that a study targeting U5MR is always the most rigorous option. A study targeting an intermediate outcome will often have greater causal validity, allow for a more rigorous design, and have greater statistical power, while lowering both costs and study length.

3. Decision-making: Impact evaluations are increasingly used by funders as a tool for deciding which programmes or implementing partners to support. However, in addition to the problems associated with drawing conclusions from a single study, using evaluations to decide who to fund can lead to non-pragmatic studies, lack of scalability, misrepresentation of results, and publication bias.

4. Ethical considerations: Randomised U5MR studies are unethical unless there is genuine uncertainty around whether the intervention is more effective than an alternative (ie, equipoise).

5. Interpretation of results: The time-averaged nature of a U5MR estimate is often ignored, leading to misinterpretation of what changes over time represent.

6. Generalisability: To effectively lower U5MR, a typical intervention involves many components and must be highly tailored to a specific context; while desirable from a public health standpoint, this leads to evidence that is not easily generalised to other settings and makes it difficult to know which components are responsible for effects.

Timeliness

Prospective evaluations that measure U5MR typically involve very large surveys and last a long time (commonly 3–5 years), meaning that information on effectiveness is substantially delayed. To understand why these evaluations take so long, consider figure 1, which depicts three stages typical of any intervention including pre-intervention, implementation and post-intervention. The time required for implementation (during which effects on mortality are ramping up) typically depends on factors related to the nature of the intervention, the target population and other contextual details, and does not depend on evaluation design choices. On the other hand, when measuring U5MR, evaluators typically must wait years for enough events to accrue in the post-intervention period before conducting a follow-up survey, since achieving sufficient power depends on the number of deaths that occur rather than just the number of individuals in the study.²² As global U5MR decreases and child deaths become rarer, these studies must be larger or take even longer. Alternative outcomes, in particular those that can be measured for any child at a single point in time, allow for the post-intervention measurement period to be shortened from years to months. Although clever study design choices sometimes help, the only reliable way to shorten the time required for an evaluation measuring U5MR is to increase the sample size, which may necessitate increasing the geographic scope of the intervention. When relevant stakeholders must wait years for the post-intervention period to pass before knowing whether the intervention works, priorities may have shifted, the implementation context will likely have changed, funding for scale-up efforts may not be available, and the ability to programmatically correct course is hindered.

Figure 1. The hazard of mortality over the three stages of an intervention.

Causal validity

Attributing observed changes in an outcome to an intervention is much more difficult for U5MR than for an intermediate outcome, particularly for quasi-experimental studies, which must statistically account for confounders of the relationship between the intervention and U5MR for causal inference to be valid. A variety of economic, social and contextual factors influence U5MR, and the likelihood that all relevant confounders are measured and have sufficient overlap between intervention and control areas is low, particularly for cluster-level confounding effects which will rarely be balanced and are more difficult to statistically control for. In contrast, the set of possible confounders that influence an intermediate outcome (such as vaccination rates) will always be smaller, and thus it is far easier to achieve the ‘no unmeasured confounding’ assumption of causal inference.

Furthermore, many studies do not involve a control group and simply measure changes over time; in such uncontrolled before-and-after studies, it is not possible to know what would have happened in the absence of the intervention. Because of long-term secular trends toward fewer under-5 deaths in most settings, U5MR reductions will commonly be seen in before-and-after studies over long enough time periods, even in the presence of interventions that do nothing (or are harmful). Conversely, one may observe increases in U5MR after implementation of a beneficial intervention if other factors lead to U5MR increases. For intermediate outcomes, the validity of an uncontrolled before-and-after study depends on the likelihood of alternative explanations for observed changes and whether an unrelated secular trend is plausible.

Decision-making

Individual impact evaluations are increasingly used by funders as a tool for deciding which programmes and/or implementing partners to support.¹⁵ However, this can lead to error-prone decision-making, since multiple studies are typically necessary to have true confidence that an intervention works in a given location. A study with 80% power will (by definition) have a 1 in 5 chance of not detecting a true effect even if one exists, and recent research suggests that most studies that estimate 80% or 90% power actually have much lower power in reality.²³ Studies targeting a U5MR outcome are often underpowered since mortality rates tend to decrease over time, a point often overlooked in power calculations.

A related issue is that the same group who is designing and implementing a programme is often asked by funders to run an evaluation of that programme. This is not inherently problematic, but when funding decisions depend on the outcome of an evaluation, there are potential conflicts of interest that arise from incentives to disproportionately allocate resources towards implementation in that study area, leading to non-pragmatic studies that are unrepresentative of how interventions may realistically scale over time. Similarly, there may be incentives to selectively choose favourable indicators to report, emphasise non-prespecified secondary analyses, and delay or suppress dissemination of unfavourable results, all of which distort the evidence base and deprive stakeholders of a nuanced understanding of the intervention.

Finally, while reducing under-5 mortality is a critical fundamental objective, it is not the only one that matters. There is a tendency to judge an intervention as unsuccessful if the study does not show an impact on under-5 death even though it may show meaningful improvements in morbidity, welfare or socioeconomic outcomes and even if subsequent implementation research could lead to a better intervention. Such a conclusion can misguidedly lead to the cessation of valuable interventions.

Ethical considerations

Randomised clinical research studies are only ethical if there is genuine uncertainty as to whether the intervention is more effective than the alternative in the target population, a concept known as equipoise. Organisations may be incentivised, due to funding decisions, to not conduct a randomised trial until they are highly confident that the intervention is effective,²⁴ which conflicts with the equipoise principle. It is similarly unethical to conduct a randomised study to get a better understanding of effect sizes, cost-effectiveness or return on investment for an intervention versus a control condition known to be inferior in terms of reducing U5MR. Stepped wedge cluster randomised trial designs were formerly proposed as a ‘solution’ to the equipoise issue, but this idea has been contested and the current consensus is that if equipoise does not hold, it is generally not considered ethical to randomise the order in which clusters receive the intervention or to delay intervention rollout for the sake of an evaluation.²⁵

Interpretation of results

A point estimate of U5MR represents an average over a long period of time rather than a reflection of what is happening at a single time point. However, it is common to see comparisons of U5MR at ‘baseline’ versus ‘follow-up’, with the time-averaged nature of the actual measurement dismissed as a technical detail. A ‘baseline’ estimate will always represent an average over many years (typically, based on a 5-year birth history) prior to the start of an intervention, and since U5MR tends to trend downward over time, the true risk of under-5 mortality at the start of an intervention will typically be overestimated. Furthermore, the time window corresponding to a ‘follow-up’ estimate may include the implementation period (see figure 1), and if the effect of the intervention varies with time, effect sizes may be underestimated, particularly in stepped wedge cluster randomised trials.²⁶ HRs related to under-5 mortality may be a more appropriate guide if interest lies in what is happening at a single point in time, but strong modelling assumptions are required to do so, and causal interpretation can be difficult.^{27 28} Finally, it is important to distinguish between whether an evaluation is estimating an effect on the entire population or just the participants enrolled in the study.

Generalisability

It is usually only possible to power U5MR studies for large effect sizes (eg, a 20% decrease in U5MR). To achieve these effects, an intervention must combine a number of strategies, which are often highly tailored to the implementation context. This is typically the right approach from a public health standpoint to improve mortality outcomes, but it means that even a well-designed randomised trial cannot disentangle the relative effects of specific components of the interventions or interactions between the components. Studies focused on more proximate outcomes are better suited to identifying the relative effects of specific components. Furthermore, an intervention package designed for the specific epidemiological challenges and sociopolitical context of one geographic area may not work in another. A recent systematic review of 16 single-centre trials targeting mortality that had a follow-up multicentre trial of the same intervention found that just a single one of the multicentre trials was able to detect a statistically significant mortality reduction.²⁹

Recommendations for programme evaluation

Given the challenges associated with evaluations targeting a U5MR outcome, this section provides high-level recommendations on alternative modes of evaluation to guide policymakers, funders and implementers.

Choice of outcome

For interventions designed to reduce under-5 deaths, the best possible evaluation will often not use U5MR as the primary outcome. Interventions designed to lower U5MR can broadly be classified into three categories, which we refer to as treatment interventions, preventative interventions and implementation interventions. The first category (treatment interventions) consists of new clinical procedures or treatments designed to cure a life-threatening condition, for which there is genuine equipoise. For these interventions, large-scale randomised (often cluster-randomised) trials using a mortality outcome are typically warranted, either structured to measure efficacy (impact in ideal conditions) or effectiveness (impact under real-world conditions) depending on the study objectives. The second category (preventative interventions) consists of devices, prophylactic medication, behavioural strategies or other interventions designed to prevent the onset of a disease or other potentially fatal event, such as insecticide-treated nets (ITNs) for malaria or water source chlorination for diarrhoeal disease. These interventions target diseases, injuries, and other conditions that have a clear causal link to mortality. Evaluations of preventative interventions should use an outcome directly relevant to what they seek to prevent; for example, a study of an ITN-based intervention could target malaria incidence. These will typically be structured as effectiveness studies, perhaps augmented with secondary outcomes related to implementation, as is done in hybrid trials.³⁰ The third category (implementation interventions) consists of interventions designed to increase the coverage, access, quality, fidelity, or use of an evidence-based preventative or treatment intervention that has already been shown to be effective. Examples include task-shifting delivery of a medication from one cadre of healthcare workers to another, strategies for improving medication adherence, and multilevel interventions that combine community-level, facility-level, regional and national strategies. Interventions falling in this category should rarely use U5MR as a primary study outcome. Instead, we recommend focusing on outcomes related to whether the underlying preventative and treatment interventions are reaching the target population, implemented as designed, delivered with high quality and properly integrated into existing government health systems. For example, a community health worker programme implementing integrated community case management for malaria, pneumonia, and diarrhoea could focus on the proportion of children experiencing disease who receive the correct medications at correct dosages in a timely manner. This simple framework will not cover all scenarios, but it can serve as a useful starting point for choosing an appropriate outcome. In general, use of an intermediate outcome can solve for problems related to timeliness and interpretation of results, since they often correspond to a single point in time or a very short time window (eg, receipt of treatment for child pneumonia cases in the past 2 weeks). They can also mitigate problems related to causal validity, since the set of possible confounders will be smaller.

Methodological rigour

Evaluations should use rigorous study designs, high-quality data collection, and modern analytic methods. Uncontrolled before-and-after studies examining changes over time can provide helpful contextual information but cannot be reliably used to estimate the causal effect of a single intervention on U5MR. Some authors have written that uncontrolled before-and-after evaluations are ‘better than no assessment, if results are interpreted with the necessary caveats’.¹⁷ However, the caveats are substantial, frequently ignored, and often misunderstood. As noted above, the multitude of confounders and simultaneous causes of U5MR creates a more serious problem for studies targeting U5MR than it does for intermediate outcomes.

For studies of treatment interventions targeting a U5MR outcome, a randomised design should be used across a large geographic area, involving groups with extensive experience designing and running trials, and should follow best practices.^{22 31 32} Researchers should take measures to prevent or mitigate sources of errors that are particularly pronounced for U5MR data, including recall bias,³³ social desirability bias,³⁴ survivorship bias,³⁵ and misreporting of dates or ages.³⁶ Studies of preventative interventions and implementation interventions targeting an intermediate outcome should use the best available context-appropriate study design and involve estimation of counterfactual outcomes, ideally through direct measurement in a control (or delayed treatment) population. Approaches based on mathematical modelling, such as those implemented in the Lives Saved Tool,³⁷ can sometimes be used to extrapolate U5MR impact from changes in intermediate outcomes; this can help compare the relative impact or cost-effectiveness of two different interventions, but results from modelling methods should be interpreted with caution. With any study, efforts should be made to measure quality and fidelity of implementation, intervention coverage, population acceptability and use, as well as any contextual factors that may influence effectiveness. This should ideally be guided by established implementation science frameworks, such as the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework³⁸ or the Consolidated Framework for Implementation Research.³⁹

Funding decisions

Funders and policymakers should not overinterpret individual studies or make funding decisions based on a single impact evaluation, since multiple studies are typically necessary to prove or disprove that an intervention works in a given location. Research should be used for creating generalisable knowledge around an intervention, but not as a means of comparing the performance of implementing partners. There are many frameworks for making evidence-based and principled funding decisions that account for organisational capacity,⁴⁰ performance⁴¹ and cost-effectiveness⁴²; a full discussion of this topic is outside the scope of the present work.

National periodic U5MR measurement

At a national level, it is essential to continue regular measurement of U5MR through large representative national surveys, such as the DHS. These surveys are a pillar of public health research and can facilitate assessment of whether global targets for U5MR are being met, as well as spatial prediction, temporal projection, and risk factor analysis. However, they are only useful for programme evaluation against a U5MR endpoint in rare circumstances where the timing and geographic rollout of a programme allows for a quasi-experimental evaluation using national survey data; even then, the issues discussed around timeliness, causal validity, and interpretation are present. In some situations, measuring subnational U5MR trends over time can be informative in a descriptive sense, but trends should be interpreted with caution.

Understanding holistic causes of U5MR

At a local level, it is critical to have a deep understanding of both medical causes of under-5 death and systemic, structural, behavioural and societal factors contributing to death to effectively design and evaluate interventions that prevent them. Time, resources, and capacity related to an evaluation should ideally allow for process evaluation, qualitative analysis, quality improvement, and implementation research. Mixed methods studies focused on high-risk populations can be particularly useful to understand barriers to accessing high-quality preventative and curative healthcare services. Implementation of death registries and regular verbal and social autopsies,⁴³ along with consistent ongoing community engagement, can inform ongoing programmatic learning in the context of adaptive evaluations.

Conclusions

While reducing under-5 mortality remains a critical global health priority, there are major challenges with evaluations targeting U5MR as a primary outcome due to concerns with methodology, interpretation, decision-making, and ethics. This paper highlights the limitations of such evaluations and argues in favour of prioritising intermediate outcomes that offer greater causal validity, timelier insights, and enhanced programmatic relevance. Future evaluations should emphasise context-appropriate study designs that are rigorous, rapid, and involve meaningful community engagement. Individual studies should rarely be used to make funding decisions. Importantly, child survival is only one aspect of child well-being; children must also grow up well-nourished, free from preventable illness and disability, and in communities that support personal development.

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study.