Evaluating performance-based financing in low-income and middle-income countries: the need to look beyond average effect

Summary box.

• Performance-based financing (PBF) is widely implemented despite limited available evidence on its effectiveness.

• There is a need to assess how PBF works in different contexts within and across settings in order to inform country-level decisions as to whether PBF may be a suitable health financing option and how it should be shaped to suit different contexts within a country.

• This commentary intends to inspire research looking beyond average programme effects into understanding heterogeneity in PBF programmes and their effects.

• As a starting point, we propose to document and analyse all potential sources of heterogeneity in PBF.

• We further provide initial guidance to conduct systematic analyses of heterogeneity of PBF by measuring and understanding differential effects and what drives or explains them.

Introduction

Over the last decade, performance-based financing (PBF) has gained momentum as a health financing innovation, which combines linking healthcare payments to performance with increased provider autonomy and supervision.^{1 2} The combination of these elements is expected to boost supply-side efforts towards increasing quantity and quality of service provision, triggering a demand-side response towards improved service utilisation.^{1 3 4}

A recent paper by Paul et al has critically questioned the widespread introduction of PBF in light of the limited available evidence on its effectiveness.⁵ The response to this paper has been varied, with authors advancing arguments for and against PBF. Some African PBF implementers have claimed that PBF is an evolving strategy with potential benefits on health systems despite its existing challenges.⁶ Others have drawn attention to the unintended consequences of PBF⁷ or to the need to assess the economic value of such an approach.⁸ Beyond their diverse arguments, however, most authors have concurred with Paul et al⁵ on the limited scope of currently available evidence and have postulated the need to better assess how PBF works under different contextual constraints within and across settings.^{9 10}

Our commentary positions itself against this background, acknowledges the limited scope of current evidence on PBF, and explicitly argues in favour of devoting more effort to unravel heterogeneity across and within settings. Our argument is based on the recognition that by virtue of how impact evaluations are designed, the focus has been on the average effect, which masks important heterogeneity across settings, providers and users.^11–13 To date, only a handful of studies have assessed heterogeneity of PBF effects across population subgroups^{4 14 15} or across health providers.^16–18 Similarly, little attention has been devoted to understanding which factors can explain heterogeneity in the response to PBF or why PBF stimulates changes in some instances, but not in others.^{3 4 10}

In light of the above, we call for more systematic analyses of heterogeneity, defined in relation to both the need to report differential effects and the need to understand what drives or explains such differential effects within and across settings. We first define and outline potential sources of heterogeneity and then offer initial guidance on how to measure and understand heterogeneity.

Potential sources of heterogeneity in PBF

PBF interventions are not uniformly designed or implemented across settings. Variations in design, implementation models, and settings, inevitably lead to substantial heterogeneity in PBF programmes and eventually effects.^{3 4 19} However, publications rarely detail these variations, inevitably omitting underlying programme heterogeneity and hence nurturing a false view of PBF as a monolithic intervention. Furthermore, within a single setting, PBF addresses organisational units and actors whose intrinsic diversity represents an additional source of heterogeneity for PBF implementation and effects. Hereafter, we address three key sources of heterogeneity.

Design features of PBF

First, we highlight the need to account for PBF design features as part of a researcher’s core engagement into unravelling heterogeneity. PBF programmes vary in many aspects, all worth reporting given their potential to shape responses to the programme: what services are incentivised and how (restricted vs extensive service package; case based vs target payments); who is eligible for incentives (individual providers vs health facilities vs both); the level of autonomy granted to providers; frequency and content of supervision and verification visits.¹⁹ It is also relevant to describe how performance is measured and along which dimensions and to consider whether PBF is implemented on its own or in conjunction with demand-side interventions.

Contextual and structural factors

Relevant contextual characteristics can be identified at international, national, local and institutional level, as well as at the level of the individual actors.³ At national level, countries differ in terms of health system setup (eg, financing mechanisms, resource availability) and pre-PBF performance. In some instances, heterogeneity arises as PBF is implemented alongside another health financing reform (eg, health insurance, user fee removal) or in a fragile and conflict-affected region.⁹ At subnational level, PBF effects may for instance vary substantially across districts and facilities, levels of care or public and private facilities. This heterogeneity may be due to variation in: resource availability; complexity and scope of service provision; health staff’s availability, competence and motivation; facility remoteness and spread of catchment populations; and pre-PBF performance in regard to service coverage and quality.^{15 17 18}

Implementation processes and progress

Variations in implementation fidelity and/or implementation failures can also explain heterogeneity both within and across PBF programmes and settings.³ Process analyses are useful to detect differences in implementation fidelity across settings as well as within single programmes. Researchers are advised to document losses in implementation fidelity at different levels (from programme orientation to implementing activities to timeliness of PBF payments) as an important potential source of heterogeneity.

How to measure heterogeneity in PBF

Having acknowledged diverse sources of heterogeneity, we provide some initial practical guidance on how to look beyond average programme effects using both quantitative and qualitative methods, the former being oriented at measurement and the latter at in-depth understanding. The propositions below can be applied to assess heterogeneity across a wide range of PBF-related outcomes, including providers’ motivation and performance, health service utilisation and quality of service provision.

Capture health systems factors, community and population characteristics

One’s capacity to assess heterogeneity begins with having access to comprehensive information on the characteristics of both healthcare providers implementing PBF and target communities. Similarly, information on contextual elements, implementation models and implementation fidelity at different levels of the health system is needed. Depending on the specific focus of one’s analysis, this information can be captured quantitatively through use of routine data sources (eg, Health Management Information Systems), facility-based and population-based surveys, or qualitatively through means of document reviews, observations and interview methods.

Identify relevant subgroups to analyse heterogeneity of PBF effects

The identification of potential sources of heterogeneity can be done through data disaggregation. This entails categorising data by key actors (eg, different sets of providers/users), by setting (eg, rural–urban, different districts), by different implementation models and different levels of implementation fidelity, or by any other conceptually pertinent factor, including, for example, facilities baseline performance levels or managerial capacity. Assessing heterogeneity quantitatively or qualitatively effectively entails having sufficient information to allow for subgroup analysis (comparison between groups) in relation to any of the above-mentioned categories. This of course relies on having engaged in comprehensive data collection efforts from the onset of the study and on defining relevant subgroups in relation to relevant sources of heterogeneity, the specific research questions being addressed, the study conceptual model or the intervention broader contextual elements.

Select an appropriate analytical approach

When assessing heterogeneity quantitatively, the choice for an appropriate quantitative analytical approach largely relies on the typology of the data available—time series data, panel data or repeated cross-sectional data—as well as on the specific research question being addressed. Within the PBF literature in low-income and middle-income countries, only a few studies to date have explicitly used interactions terms and/or stratified analysis to assess differential impacts across subgroups of providers and users.^14–17 Some authors have proposed additional analytical options to estimate heterogeneity of programme effects. For instance, some authors have used quantile-specific treatment effects,^{11 13} while others have used hierarchical linear models on propensity score strata to estimate the distributional impact of policies.²⁰

Assessing heterogeneity qualitatively means assessing how different constituencies within and across settings define and experience PBF in light of broader contextual and implementation-specific elements. Thematic descriptive qualitative analysis has largely dominated this body of literature.¹⁸ We note that with a few exceptions, quantitative and qualitative analytical approaches have to date rarely been merged into a single research effort to explore specifically heterogeneity in PBF effects.

Conclusion

With this brief commentary, we wish to inspire the research and policy community to look beyond average programme effects, devoting more attention towards measuring and understanding heterogeneity in PBF programmes and their effects. Understanding heterogeneity represents an essential step towards establishing the needed evidence base to inform country-level decisions as to whether PBF may be a suitable health financing option for a given setting. In addition, understanding heterogeneity and its sources also represents the initial step towards adjusting design and implementation to take into account the different actors and realities that may co-exist within a single setting.