Skip to main content
NCBI home page
Gates Open Research logoLink to Gates Open Research
letter
. 2020 Nov 30;4:176. [Version 1] doi: 10.12688/gatesopenres.13201.1

What next after GDP-based cost-effectiveness thresholds?

Y-Ling Chi 1,a, Mark Blecher 2, Kalipso Chalkidou 1,3, Anthony Culyer 4, Karl Claxton 4, Ijeoma Edoka 5, Amanda Glassman 6, Noemi Kreif 4, Iain Jones 7, Andrew J Mirelman 4, Mardiati Nadjib 8, Alec Morton 9, Ole Frithjof Norheim 10, Jessica Ochalek 4, Shankar Prinja 11, Francis Ruiz 1,3, Yot Teerawattananon 12, Anna Vassall 13, Alexander Winch 3
PMCID: PMC7851575  PMID: 33575544

Abstract

Public payers around the world are increasingly using cost-effectiveness thresholds (CETs) to assess the value-for-money of an intervention and make coverage decisions. However, there is still much confusion about the meaning and uses of the CET, how it should be calculated, and what constitutes an adequate evidence base for its formulation. One widely referenced and used threshold in the last decade has been the 1-3 GDP per capita, which is often attributed to the Commission on Macroeconomics and  WHO guidelines on Choosing Interventions that are Cost Effective (WHO-CHOICE). For many reasons, however, this threshold has been widely criticised; which has led experts across the world, including the WHO, to discourage its use. This has left a vacuum for policy-makers and technical staff at a time when countries are wanting to move towards Universal Health Coverage .

This article seeks to address this gap by offering five practical options for decision-makers in low- and middle-income countries that can be used instead of the 1-3 GDP rule, to combine existing evidence with fair decision-rules or develop locally relevant CETs. It builds on existing literature as well as an engagement with a group of experts and decision-makers working in low, middle and high income countries.

Keywords: Cost-effectiveness thresholds, cost-effectiveness analysis, health opportunity cost, priority setting

Introduction

Public payers around the world are increasingly using Health Technology Assessment (HTA) to inform resource allocation decisions ( Leech et al., 2018; MacQuilkan et al., 2018; Tantivess et al., 2017; World Health Organization, 2015). These decisions are often based on evidence of the expected additional intervention costs and health benefits summarized as the incremental cost-effectiveness ratio (ICER). This is a measure of the value of resources that are actually needed in a specific location and at a specific time to produce one unit of health (most commonly a Quality Adjusted Life Year – QALY- or a Disability Adjusted Life Year – DALY averted). ICERs can be used to compare competing interventions or can be evaluated against a pre-defined decision rule referred to as a Cost-Effectiveness Threshold (CET). The CET sets, on average, the maximum financial investment a public payer will commit to generate a unit of health ( Cameron et al., 2018) and is typically used alongside other information to inform decisions around resource allocation in health, particularly around the introduction of new treatments and benefits.

The use of CETs (and ICERs) is typically associated with a goal of health maximisation. They can also, however, be used in conjunction with other criteria. For instance, in Norway a CET was used to maximise both health together with a fair distribution of health ( Ottersen et al., 2016). While health system objectives might vary between jurisdictions, maximising health is one aspiration that we assume is widely shared within and across jurisdictions ( Culyer, 2016).

There is still much confusion about the meaning and uses of the CET, how it should be calculated, and what constitutes adequate evidence base for its formulation ( Ochalek et al., 2015). There are three broad bases on which CETs that are used by public payers are set: willingness to pay (WTP), precedence and opportunity cost (for further discussion, see Santos et al., 2018 and Vallejo-Torres et al., 2016). However, a commonly used approach following none of the three bases is to set the CET at 1-3 times GDP per capita, including in Low and Middle-Income Countries (LMICs). It is often attributed to the Commission on Macroeconomics, and was later adopted in the WHO guidelines on Choosing Interventions that are Cost Effective (WHO-CHOICE) ( World Health Organization Commission on Macroeconomics and Health, 2001). Using this approach, an intervention that averts one DALY for a cost less than GDP per capita is considered “very” cost effective, but an intervention could still count as cost-effective if the ICER did not exceed three times GDP per capita. One rationale behind the rule is that GDP per capita is a proxy for earnings ( Robinson et al., 2017). In other words, if an intervention averts a DALY at less than one GDP per capita, then its return on investments in the wider economy (through increased labour productivity) would offset its implementation costs. Because of its simplicity of use and interpretation, the 1-3 GDP threshold has gained much popularity in recent years. A recent review of cost-effectiveness analyses (CEA) found that 66% of published research studies between 2000 and 2015 in LMICs used a GDP-based CET ( Leech et al., 2018) i.

The 1-3 GDP per capita criterion in the Commission on Macroeconomics and Health report was, however, never intended to be used to determine CETs. It was to be used to value health in benefit-cost analyses (e.g. to make the case for resource allocation to health as opposed to other sectors) ( Robinson et al., 2017). Moreover, GDP-based CETs have no direct relation to a country’s healthcare budget, technical capacity, population preferences or social values ( Marseille et al., 2015). Several experts have also shown that GDP-based CETs can lead to the adoption of interventions that are not in practice locally affordable ( Marseille et al., 2015). One may argue that there is no harm in casting the net wide by setting a high CET. This is false, however, because spending health resources inevitably creates opportunity costs. Health opportunity costs arise because resources committed to one intervention are no longer available to fund alternative, perhaps more cost-effective, interventions. As a result, allocating resources to an intervention that would not be included by reference to a more realistic CET can paradoxically result in a loss of health and increase in avoidable deaths, by displacing more health than it creates ( Revill et al., 2018). Health opportunity costs are higher in LMICs because spending on the wrong interventions might deplete a country’s resources to pay for affordable and effective interventions. In stylized terms, in a country where a life could be saved by spending $1000 dollars, the misallocation of that same $1000 will cause a death.

These considerations have contributed to a growing unease with the use of GDP-based CETs, especially in LMICs, given how widely they are used in research ( Leech et al., 2018). More recently, several experts, including some at the WHO where the practice has been referenced for decades, have advised against the use of GDP-based CETs as a sole decision rule ( Bertram et al., 2016).

The retraction of the 1-3 GDP rule leaves a vacuum for policy-makers and technical staff at a time when LMICs are aspiring to Universal Health Coverage (UHC) and making decisions that will set the path of health policy and spending for decades to come. As others have observed ( Global Burden of Disease Health Financing Collaborator Network, 2019), mobilising additional resources for health is a lengthy and challenging process; making decision rules such as CETs to ensure adequate spending all the more important. An appropriately set CET could support the realisation of the most generous possible form of UHC by ensuring that existing resources are primarily directed toward cost-effective interventions.

In addition, countries have been encouraged to develop in-country capacity and institutions to support the use of HTA in decision-making. This is fully supported by the WHO and is reflected in the World Health Assembly WHA67/23 resolution ( World Health Organisation, 2014). In several countries (e.g. Indonesia, Kenya, Ghana, South Africa, India and the Philippines), plans to institutionalise HTA or set up HTA committees were announced in official communications ( Addo, 2019; Authority of the Republic of Kenya, 2018; Congress of the Philippines, 2019; Ministry of Health and Family Welfare, 2019; Sharma et al., 2020). These committees and agencies are likely to include CETs as part of their decision-making processes.

The theory of CETs is discussed elsewhere and reviews document how countries with existing CETs have defined them ( Cameron et al., 2018; Santos et al., 2018; Thokala et al., 2018; Vallejo-Torres et al., 2016). Shortcomings of existing approaches to CETs, not exclusively focusing on GDP-based CETs, have also been discussed ( Bertram et al., 2016; Marseille et al., 2015; Newall et al., 2014; Vallejo-Torres et al., 2016). What is missing is the creation of practical alternatives to the use of GDP-based thresholds. This article seeks to address this gap by offering five practical options for developing locally relevant CETs (i.e. ones that are informed by local data) or utilising evidence (existing, complementary to CEA) to support decision-makers facing urgent resource allocation decisions.

The paper builds on a consultation between academics, country technical staff and global donors from high, middle, and low income countries (although not representative), organised by the International Decision Support Initiative (Secretariat based in the UK), Centre for Health Economics (University of York) and the Health Intervention and Technology Assessment Program (Thailand), at the Rockefeller Foundation Bellagio Centre between December 3-7, 2018. The group spent three and a half days discussing the role of CETs in achieving UHC in LMICs ii. Our conversation focussed on supply-side approaches. There is one major reason for this. The willingness-to-pay approach raises question of whose willingness matters, a political matter on which we do not feel able to advise, and can also lead to the use of aspirational CETs that are not connected to budget constraints or health opportunity costs. The supply-side approach by contrast is focussed on the resources available, how they are currently used, and what is most likely to be sacrificed when they are used in one way rather than another. These are concrete matters that decision-makers face on a daily basis.

During this meeting, our group collectively defined five options that can replace the GDP based threshold (in the absence of a formal CET) when decision-makers are faced with a new intervention that they need to consider. We have ordered the options from the least to the most resource and data intensive, with option 5 being a within-country empirical estimation of supply-side CET that reflects health opportunity costs. The options are not mutually exclusive: it is possible to combine several approaches according to context and need.

Option 1. Use existing estimates of national health opportunity cost thresholds derived from cross-country data

Two recent contributions ( Ochalek et al., 2018; Woods et al., 2016) provide estimates of health opportunity costs (i.e. what is given up as a consequence of introducing an intervention), country by country, for a number of LMICs. Woods et al. (2016) extrapolate health opportunity costs in LMICs from UK estimates produced by Claxton et al. (2015) by applying data on the income elasticity of the value of health to generate ranges of cost per QALY gained estimates in LMICs. Ochalek et al. (2018) expand country estimates from Bokhari et al. (2007) on the effect of changes in health expenditure on health outcomes and, following Claxton et al. (2015), apply country-specific data on health expenditure, epidemiology and demography to calculate a range of cost per DALY averted thresholds estimates in LMICs. Country by country estimates are available in their supplemental materials. To date, those two studies are the best attempts to estimate health opportunity costs using cross-country data sources, despite methodological caveats iii.

Although adopting different approaches, both Woods et al. (2016) and Ochalek et al. (2018) estimate CETs averaging iv roughly half of GDP per capita, albeit with a substantial range. On this basis, if a GDP based rule is applied, it would probably be a secure rule to deem as cost-ineffective all interventions averting 1 DALY at more than 1 GDP per capita. Half of GDP per capita is more in accord with countries’ realities than the 1-3 GDP per capita, and could be used as an interim rule of thumb rather than the 1-3 GDP rule. A handful of recent papers have already started using half of GDP per capita v. Francke et al. (2016) estimated the cost-effectiveness of diagnostic of HIV infection in early infancy in South Africa, and based on ‘ emerging literature’, used a CET of half of GDP per capita. Other studies also used such a CET, although none provided a justification for doing so ( Bilcke et al., 2019; Campos et al., 2018; Mezei et al., 2018). Finally, half of GDP per capita was also referenced in the Disease Control Priorities 3 (DCP3) as an example CET in highly resource constrained countries ( Watkins et al., 2017). Half of GDP per capita will lead to underestimating or overestimating health opportunity costs in roughly half of LMICs ( Ochalek et al., 2020b).

Option 2. Use existing evidence from other settings

Short cuts can sometimes offer an informed way forward when decision-makers are uncertain about appropriate methodologies or lack the necessary skill and evidence base for more sophisticated procedures. One shortcut could be to look at evidence elsewhere by asking the following questions:

  • Have regulatory authorities such as the Food and Drug Administration licensed this product and for which indication? This question can first identify ‘wasted buys’ ( i.e. interventions that have a harmful or non-beneficial effect for a patient) if the intervention under consideration was not licensed (for the right indication), without identifying a CET or conducting a CEA. For instance, a review of the Romanian procurement decisions found that bevacizumab was being used for the treatment of metastatic breast cancer, despite having been withdrawn from FDA approval for this indication ( Lopert et al., 2013; US Food & Drug Administration, 2011). Other agencies in charge of ensuring the safety, efficacy and security of drugs and products can be considered.

  • Was this rejected for funding elsewhere? The National Institute for Health and Care Excellence (NICE) in the UK publishes its technology appraisal guidance, which contains information on the technology under consideration and the accompanying recommendations. NICE’s negative recommendations can be a useful starting point since interventions not recommended in a high-income country are very unlikely to be appropriate choices in LMICs unless it was thought that the United Kingdom’s health system and financial capacity is very different from the country under consideration. On the other hand, a NICE positive recommendation is not to be followed slavishly in LMICs, because of differential health opportunity costs.

CEA estimates are not always transferrable across settings, and there is little guidance on how to make decisions about suitability of estimates in a local context ( Drummond et al., 2009). Nonetheless, if an intervention was found not to be cost-effective in a high-income setting, then it is unlikely to be cost-effective in an LMIC; and it would need important differences in disease epidemiology, intervention costs or health state preferences to warrant adoption in an LMICs.

Option 3. ICERs and budget impact to inform cost-effectiveness and affordability

Budget impact analyses (BIA) can also support decision-makers in determining whether an intervention is affordable to the country ( Bilinski et al., 2017). BIA is a method of assessing predicted short-term changes in expenditure were a new intervention were introduced. It reflects not only the total cost of its introduction, but also the coverage and uptake rates, as well as potential new health costs (or savings) ( Sullivan et al., 2014). There is often a disconnect between the cost-effectiveness of an intervention, and its affordability to a country ( Bilinski et al., 2017; Howdon et al., 2019; Lomas, 2019; Wiseman et al., 2016). Presenting BIA alongside CEA can ensure that decision-makers can anticipate the resource implications of a new intervention for the allocation of their budget ( Mohara et al., 2012). For instance, treatment of Hepatitis C was found to be cost-effective in many settings, but providing universal access to all eligible patients would have significant resource implications, even in middle-income countries ( Urrutia et al., 2016). In the United Kingdom, treatment of Hepatitis C was found to be cost-effective but this decision was found to be controversial due its budget impact ( Lomas et al., 2018).

Even in countries where a CET is used to inform policy, there is growing consideration of BIA. In Thailand, the ICER is presented alongside budget impact. For instance, the inclusion of Imiglucerase for Gaucher disease type 1 was approved due to the low budget impact, equity concerns and disease severity (terminal condition) even though the treatment was well above the Thai CET ( Leelahavarong, 2019). Since 2017, budget impact for the first three years of use is assessed in the United-Kingdom. If it exceeds a certain threshold for the entire National Health Service (currently £20 million), a phased implementation or price negotiation with manufacturers is initiated ( National Institute for Health and Care Excellence, 2018). However, Bilinski et al. (2017) report that fewer than 3% of the 384 published CEA included in their review contained a full report of BIA.

Option 4. A league table for Health Benefits Package design

A league table is a list of health interventions in order of their ICER. It can be a useful approach to allow decision-makers to appraise a wide range of interventions in one summary table.

When considering a new inclusion to an existing Health Benefits Package (HBP), a league table can be used to identify the least cost-effective intervention that has been funded under the HBP, which can serve as a benchmark to infer what the maximum investment the country was willing to commit to producing an additional unit of health when developing the HBP – or to initiate more detailed evaluation of its likely cost-effectiveness. In other words, the league table helps identify a proxy of the shadow CET. This approach can be appropriate if an existing package of services is available in the country and ICERs can be derived for a reasonable number of the interventions included in this package. Using this method, decision-makers could gain confidence that a new entrant would not be included unless it produced more health benefit than the least cost-effective intervention already covered.

On the other hand, when developing an HBP de novo, a league table can also be used to set a CET when combined with data on coverage and utilisation. For this, the budget envelope will need to be defined from the onset. This may not be easy, especially in countries where contributions from external partners is significant. For instance, in a study in Malawi from Ochalek et al. (2018), the authors highlight that donor funds (often off budget, disbursed through conditionalities) make up 70% of total health expenditure. This creates uncertainties on how the budget line is set (the authors calculate the budget line considering all funding, regardless of the source).

In this option, the budget line determines the CET: a league table is constructed in descending order of cost-effectiveness, and estimates of utilisation are used to calculate the budget impact for each intervention. Culyer (2016) uses a metaphor of a bookshelf of healthcare interventions, in which each book is ranked according to its height ( i.e., its effectiveness-cost ratio) and the thickness of the book represents the cost of providing the intervention ( i.e., the budget impact). The threshold corresponds to the least cost-effective intervention affordable to the country before the (fixed) budget is exhausted. This approach was reported to be implemented in Oregon’s Medicaid scheme in the 1990s to define health benefits (although a review ( Tengs et al., 1996) later showed that there was no correlation between the final benefits list and the economic literature or Oregon’s own cost-effectiveness data).

While the data requirements for this option appear be high to some, they can be much reduced by using expert opinion and international evidence to identify a narrower range of likely candidates for more detailed evaluation. This enables decision makers to focus their attention on a manageable number of possible interventions together with their relevant uncertainties. One challenge in LMICs is the low availability of evidence on cost-effectiveness, as well as differences in the methodological specifications employed in studies ( Drummond et al., 1993; Mauskopf et al., 2003). Comparing them usefully requires local epidemiological and economic skills. However, there are several global sources which can be compiled to inform country level estimates: DCP3, the TUFTS GHCEA Registry and WHO-CHOICE. These sources were used by the Ochalek et al. (2018) study, although important data gaps remained and were highlighted as a limitation to the study. In Ethiopia, a similar approach was used to develop the Essential Health Services Package ( Ministry of Health Ethiopia, 2019). It is worth noting that WHO-CHOICE present average cost-effectiveness ratios (ACERs) instead of incremental ones ( Arnold et al., 2019). This is sometimes raised as a concern because average cost-effectiveness ratios compare interventions to a doing nothing scenario, which is only very rarely an appropriate comparator ( O’Day & Campbell, 2016).

Option 5. Estimating a health opportunity cost CET using within-country data

Notwithstanding the different approaches to defining a CET, this group found that defining CETs based on health opportunity costs using within-country data was particularly suitable in LMICs given the high opportunity costs created by severe budget constraints. However, each country may want to develop its own national CET, relevant to their own situation.

Relying on health opportunity cost estimates makes it possible to estimate whether the health gains produced by an intervention are greater than the health lost from displacing other interventions in other parts of the health system ( Claxton et al., 2015). This approach derives from the goal of health maximisation. Where health must be sacrificed to improve distributional outcomes (or to meet goals other than health maximisation), a health opportunity cost CET can help quantifying the trade-off. Unlike the WTP method, which bears no link to public budgets, the health opportunity cost method calibrates the CET against the reality of local budget constraints ( Brouwer et al., 2019; Leech et al., 2018).

The seminal works of Claxton et al. (2015) in the UK pioneered the estimation of health opportunity costs CETs. They used a very detailed programme budgeting data from the English National Health Service (NHS) to estimate a health opportunity cost CET, which was much lower (£12,936) than the one then applied (£20,000-30,000 and up to £100,000 in certain cases).

Attempts to apply a similar estimation framework have been made in China, Indonesia, India and the Republic of South Africa ( Edoka & Stacey, 2020; Ochalek et al., 2020), although with different methods and data (given the paucity of the latter in LMICs). These estimates have two parts: estimating the elasticities of health outcomes with regard to health expenditure using an econometric analysis, and translating the elasticity estimates to health opportunity cost thresholds. In order to estimate health spending elasticities, data on health expenditure and health outcomes ( e.g. mortality rates, DALYs or QALYs – and ideally age and gender specific) at a low level of aggregation (e.g. local health authorities or districts and provinces) is required. Ideally, these data will need to be collected across several time periods/years. Furthermore, the estimation of health spending elasticities could be strengthened if controlling for potential confounders (e.g. poverty, literacy rate) and the application of robust estimation strategies to account for unobserved heterogeneity and reverse causality ( Edoka, 2019). One such approach consists of using an instrumental variable (IV) (i.e. a variable that has no direct impact on the health outcome but indirectly, through its impact on health expenditure). IVs should be selected by researchers to fit the local context. One example in the UK has been the use of the ‘funding rule’: local jurisdictions in the UK receive a share of the total budget based on local characteristics, however, the funding rule is revised periodically and this change creates exogenous changes in local funding and generates data for econometric estimation of elasticities ( Claxton et al., 2018).

Elasticities will need to be translated into population estimates of cost per DALY averted or QALY gained. If researchers estimated elasticities using mortality data, then those must be converted into QALYs or DALYs using additional assumptions. This requires data on the age and gender structure of the population as well as the morbidity burden of disease. It is worth noting that if the elasticities are estimated from the subset of the population (e.g. children), they will need to be extrapolated for the whole population.

There are several sources of uncertainty attaching to this approach, as there are to other approaches. Given a shortage of vital population statistics, health outcomes are often drawn from survey data, which come with their own shortcomings. Moreover, data on health expenditure is often incomplete (e.g. missing budget items), poorly collected (e.g. inconsistency in recording practice) and unavailable at a low level of aggregation or across several years. There might be uncertainty stemming from the methods employed (e.g. use of an appropriate instrumental variable).

Discussion

Setting priorities is more than ever before seen as a pre-requisite for achieving global development goals and UHC ( Wiseman et al., 2016). This prerequisite was recognised at the United Nations High-Level Meeting on UHC in 2019 ( United Nations General Assembly, 2019). There is now a push for using HTA across the world to drive more efficient resource allocation in health. Addressing the vacuum left by the abandonment of the 1–3 times GDP per capita CET has therefore become central to determining what services or interventions will be included or excluded within the UHC agenda. In addition, a CET or clear decision rule can signal maturity in resource allocation practices for the health budget to Treasuries or Ministry of Finance. This may support further investments in the health sector, especially in LMICs where health receives low priority within the broader government budgeting process.

This article lays out five alternatives to GDP-based CETs for decision-makers faced with urgent resource allocation decisions, building from our meeting bringing together a selected group of practitioners and researchers. Estimating a health opportunity cost CET using local data is the long-run solution for LMICs since it will explicitly link CETs to budget constraints and help articulate the trade-offs that inevitably arise in coverage decisions. Stakeholders should understand how a CET can be estimated, what the possible alternatives are, and be able to assess the adequacy of the arrangements in, or proposed for, their country. This will require engagement and communication from the onset.

While an empirically estimated supply-side CET using local data, should be the long term aim for all countries, the other four suggestions provide LMICs with tools to structure what is often a difficult ad hoc conversation on value for money and affordability. This intermediate step would represent a significant improvement on current practice and the first four options require no additional or only modest additional resources. In combination, these suggestions can help form deliberative processes that are as evidence-based as possible and that face explicitly up to the complexity of the choices faced by decision-makers ( Baltussen et al., 2016; Chalkidou et al., 2016).

The dangers of applying thresholds that are set too high have been discussed widely elsewhere ( Bertram et al., 2016; Leech et al., 2018; Marseille et al., 2015). It is likely that our proposed approach would lead to more conservative CETs compared to the 1–3 times GDP per capita rule. For this reason, it is worth considering what the implications and risks of under-estimating the CET would be. The first obvious consequence would be that cost-effective interventions would be mistakenly ruled out, causing a loss of health at the population level relative to what would have been possible if resources were fully allocated to cost-effective interventions. Some have also argued that a low CET reduces innovation by discouraging manufacturers from seeking to develop new products. Finally, it may be thought that a more conservative CET is incompatible with other social objectives of the healthcare system that do not align or may even conflict with the goal of health maximisation (e.g. priority to the poor or more broadly equity).

These concerns need to be addressed seriously. Further research should help to make more reliable any estimates of a local CET based on health opportunity costs. Further work on the LMIC estimates of Woods et al. (2016) and Ochalek et al. (2018) would enable better methods and reduce some of the uncertainty and data challenges. Moreover, it is worth noting that CETs are not only used as a simple inclusion/exclusion rule, but also as a basis for price negotiation with manufacturers. In Thailand, economic evaluation has successfully been used to bring down drug prices: for instance, the price of Tenofovir was cut down more than two third from the original to the negotiated price using the CET ( Teerawattananon et al., 2014).

On innovation, there is growing evidence that the vast majority of new products approved for use are only marginal improvements on existing ones, and new market introductions are often priced well above the existing thresholds, especially in LMICs. For example, a discussion of cancer drugs highlights that new introductions are often ‘prohibitively expensive’ and therefore unaffordable for publicly funded systems in LMICs ( Gyawali & Sullivan, 2017). Recent studies have also pointed to the fact that high CETs created perverse incentives on prices, as manufacturers can use the CET to calculate a maximum ceiling price for their products to be accepted ( Gronde et al., 2017). There is no evidence that high aspirational CETs encouraged innovation or access to novel treatments ( Claxton et al., 2009), so the disincentive, if that is what it is, of lower CETs may be similarly unimportant. More important is for LMIC countries individually or collectively to identify the kinds of innovation they would most like to see and then to engage in a discussion with manufacturers and other stakeholders as to suitable incentives (or removal of disincentives). The inclusion of social objectives other than health maximization, like equity, positive discrimination, managerial capacities at various healthcare delivery levels, is independent of the level at which a threshold is set, and should be discussed alongside cost-effectiveness in any HTA framework ( Cookson, 2016). More realistic CETs might well be intrinsically more equitable than high ones because the inclusion of wasted buys leads to crowding out of resources that would otherwise be spent on cost-effective services, usually benefitting the poor.

This piece has provided a menu of options that can be used as an alternative to GDP based thresholds. The primary audience for this piece has been national decision-makers, but our recommendations have ramifications for the global health community. Development partners (DPs) should consider supporting countries in the challenging estimation of locally relevant CETs, working with research institutions with this expertise. The use of CETs to inform resource decisions by DPs has not been widely researched ( Drake, 2014; ( Morton et al., 2018). Should DPs use a single global threshold (as most countries do) or rely on country-specific thresholds, whether estimated by them or the countries in question? This is a contentious issue. Locally estimated CETs reflect the local health opportunity costs, which will be important for countries in transition with increasing co-financing from domestic resources ( Silverman, 2018). This will raise consistency issues if different payers adopt different CETs, or ones that conflict with those preferred by recipient countries. For example, the Global Fund affords priority to the fight against AIDS, Tuberculosis and Malaria, but recipient countries may apply different threshold values to the same programs. On the other hand, using country thresholds would mean that DPs would apply different decision rules to different recipient countries. There is a universal inverse relation between GDP per capita and opportunity cost-based CETs, so using national thresholds may signal a mean that an intervention covered in a middle income country may not be covered in low income one, which may be politically challenging. Conversely, it may also signal to DPs that spending in low income countries is more impactful, as a DALY averted or QALY gain can be realised at lower cost. The matter plainly needs further thought and investigation.

Data availability

Underlying data

No data are associated with this article

Notes

i There is no published review of the use of CETs by decision-makers.

ii For more information about the meeting and its participants, go to: https://www.idsihealth.org/blog/developing-cost-effectiveness-thresholds-to-support-universal-health-coverage/

iii Data limitations are acknowledged in the two papers and use of strong assumptions.

iv Unweighted average

v In preparing this commentary, we searched the cost-effectiveness analyses conducted in LMICs in the last five years using the TUFTS database. Conclusions from this search will be subject to a different piece focusing on the use of CETs in the past five years (forthcoming).

Funding Statement

This work was supported by the Bill and Melinda Gates Foundation [INV-006987]. This work was also supported by the Rockefeller Foundation. IE is supported by the South African Medical Research Council (23108).

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 1; peer review: 2 approved]

References

  1. Addo C: MOH inaugurates Steering and Technical committees for Health Technology Assessment. Ghana News Agency. 2019. Reference Source [Google Scholar]
  2. Arnold M, Griffin S, Ochalek J, et al. : A one stop shop for cost-effectiveness evidence? Recommendations for improving Disease Control Priorities. Cost Eff Resour Alloc. 2019;17:7. 10.1186/s12962-019-0175-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Authority of the Republic of Kenya: The Kenya Gazette. . Vol. CXX —No. 692018. Reference Source [Google Scholar]
  4. Baltussen R, Jansen MP, Mikkelsen E, et al. : Priority Setting for Universal Health Coverage: We Need Evidence-Informed Deliberative Processes, Not Just More Evidence on Cost-Effectiveness. Int J Health Policy Manag. 2016;5(11):615–618. 10.15171/ijhpm.2016.83 [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bertram MY, Lauer JA, De Joncheere K, et al. : Cost-effectiveness thresholds: pros and cons. Bull World Health Organ. 2016;94(12):925–930. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bilcke J, Antillón M, Pieters Z, et al. : Cost-effectiveness of routine and campaign use of typhoid Vi-conjugate vaccine in Gavi-eligible countries: a modelling study. Lancet Infect Dis. 2019;19(7):728–739. 10.1016/S1473-3099(18)30804-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Bilinski A, Neumann P, Cohen J, et al. : When cost-effective interventions are unaffordable: Integrating cost-effectiveness and budget impact in priority setting for global health programs. PLoS Med. 2017;14(10):e1002397. 10.1371/journal.pmed.1002397 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Bokhari FAS, Gai Y, Gottret P: Government health expenditures and health outcomes. Health Econ. 2007;16(3):257–73. 10.1002/hec.1157 [DOI] [PubMed] [Google Scholar]
  9. Brouwer W, van Baal P, van Exel J, et al. : When is it too expensive? Cost-effectiveness thresholds and health care decision-making. Eur J Health Econ. 2019;20(2):175–180. 10.1007/s10198-018-1000-4 [DOI] [PubMed] [Google Scholar]
  10. Cameron D, Ubels J, Norström F: On what basis are medical cost-effectiveness thresholds set? Clashing opinions and an absence of data: a systematic review. Glob Health Action. 2018;11(1):1447828. 10.1080/16549716.2018.1447828 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Campos NG, Lince-Deroche N, Chibwesha CJ, et al. : Cost-Effectiveness of Cervical Cancer Screening in Women Living With HIV in South Africa: A Mathematical Modeling Study. J Acquir Immune Defic Syndr. 2018;79(2):195–205. 10.1097/QAI.0000000000001778 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Chalkidou K, Li R, Culyer AJ, et al. : Health Technology Assessment: Global Advocacy and Local Realities Comment on "Priority Setting for Universal Health Coverage: We Need Evidence-Informed Deliberative Processes, Not Just More Evidence on Cost-Effectiveness". Int J Health Policy Manag. 2016;6(4):233–236. 10.15171/ijhpm.2016.118 [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Claxton K, Longo R, Longworth L, et al. : The Value of Innovation. 2009. [PubMed] [Google Scholar]
  14. Claxton K, Lomas J, Martin S: The impact of NHS expenditure on health outcomes in England: Alternative approaches to identification in all-cause and disease specific models of mortality. Health Econ. 2018;27(6):1017– 1023. 10.1002/hec.3650 [DOI] [PubMed] [Google Scholar]
  15. Claxton K, Martin S, Soares M, et al. : Methods for the estimation of the National Institute for Health and Care Excellence cost-effectiveness threshold. Health Technol Assess. 2015;19(14):1–503, v-vi. 10.3310/hta19140 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Cookson R: Equity-informative health technology assessment - A commentary on Ngalesoni, Ruhago, Mori, Robberst & Norheim. Soc Sci Med. 2016;S0277-9536(16)30590-1. 10.1016/j.socscimed.2016.10.022 [DOI] [PubMed] [Google Scholar]
  17. Congress of the Philippines: Republic Act No 11223. 2019. Reference Source [Google Scholar]
  18. Culyer AJ: Cost-effectiveness thresholds in health care: a bookshelf guide to their meaning and use. Health Econ Policy Law. 2016;11(4):415–432. 10.1017/S1744133116000049 [DOI] [PubMed] [Google Scholar]
  19. Drake T: Priority setting in global health: towards a minimum DALY value. Health Econ. 2014;23(2):248–252. 10.1002/hec.2925 [DOI] [PubMed] [Google Scholar]
  20. Drummond M, Barbieri M, Cook J, et al. : Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report. Value Health. 2009;12(4):409–418. 10.1111/j.1524-4733.2008.00489.x [DOI] [PubMed] [Google Scholar]
  21. Drummond M, Torrance G, Mason J: Cost-effectiveness league tables: more harm than good? Soc Sci Med. 1993;37(1):33–40. 10.1016/0277-9536(93)90315-u [DOI] [PubMed] [Google Scholar]
  22. Edoka IP: Returns to health spending in low- and middle-income countries.In: Oxford Research Encyclopedia of Economics and Finance.Oxford: Oxford University Press.2019. 10.1093/acrefore/9780190625979.013.437 [DOI] [Google Scholar]
  23. Edoka IP, Stacey NK: Estimating a cost-effectiveness threshold for health care decision-making in South Africa. Health Policy Plan. 2020;35(5):546–555. 10.1093/heapol/czz152 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Francke JA, Penazzato M, Hou T, et al. : Clinical Impact and Cost-effectiveness of Diagnosing HIV Infection During Early Infancy in South Africa: Test Timing and Frequency. J Infect Dis. 2016;214(9):1319–1328. 10.1093/infdis/jiw379 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Global Burden of Disease Health Financing Collaborator Network: Past, present, and future of global health financing: a review of development assistance, government, out-of-pocket, and other private spending on health for 195 countries, 1995-2050. Lancet. 2019;393(10187):2233–2260. 10.1016/S0140-6736(19)30841-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Gronde T van der, Uyl-de Groot CA, Pieters T: Addressing the challenge of high-priced prescription drugs in the era of precision medicine: A systematic review of drug life cycles, therapeutic drug markets and regulatory frameworks. PLoS One. 2017;12(8):e0182613. 10.1371/journal.pone.0182613 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Gyawali B, Sullivan R: Economics of cancer medicines: for whose benefit? New Bioeth. 2017;23(1):95–104. 10.1080/20502877.2017.1314885 [DOI] [PubMed] [Google Scholar]
  28. Howdon DDH, Lomas JRS, Paulden M: Implications of nonmarginal budgetary impacts in health technology assessment: A conceptual model. Value Health. 2019;22(8):891–897. 10.1016/j.jval.2019.04.001 [DOI] [PubMed] [Google Scholar]
  29. Leech AA, Kim DD, Cohen JT, et al. : Use and Misuse of Cost-Effectiveness Analysis Thresholds in Low- and Middle-Income Countries: Trends in Cost-per-DALY Studies. Value Health. 2018;21(7):759–761. 10.1016/j.jval.2017.12.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Leelahavarong P: Health Technology Assessment for RareDisease: Lessons Learned from Thailand. paper presented at the World Health Summit.2019. Reference Source [Google Scholar]
  31. Lomas JRS: Incorporating Affordability Concerns Within Cost-Effectiveness Analysis for Health Technology Assessment. Value Health. 2019;22(8):898–905. 10.1016/j.jval.2019.05.003 [DOI] [PubMed] [Google Scholar]
  32. Lomas J, Claxton K, Martin S, et al. : Resolving the "Cost-Effective but Unaffordable" Paradox: Estimating the Health Opportunity Costs of Nonmarginal Budget Impacts. Value Health. 2018;21(3):266–275. 10.1016/j.jval.2017.10.006 [DOI] [PubMed] [Google Scholar]
  33. Lopert R, Ruiz F, Chalkidou K: Applying rapid ‘de-facto’ HTA in resource-limited settings: experience from Romania. Health Policy. 2013;112(3):202–208. 10.1016/j.healthpol.2013.07.019 [DOI] [PubMed] [Google Scholar]
  34. MacQuilkan K, Baker P, Downey L, et al. : Strengthening health technology assessment systems in the global south: a comparative analysis of the HTA journeys of China, India and South Africa. Glob Health Action. 2018;11(1):1527556. 10.1080/16549716.2018.1527556 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Marseille E, Larson B, Kazi DS, et al. : Thresholds for the cost-effectiveness of interventions: alternative approaches. Bull World Health Organ. 2015;93(2):118–124. 10.2471/BLT.14.138206 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Mauskopf J, Rutten F, Schonfeld W: Cost-effectiveness league tables: valuable guidance for decision makers? PharmacoEconomics. 2003;21(14):991–1000. 10.2165/00019053-200321140-00001 [DOI] [PubMed] [Google Scholar]
  37. Mezei AK, Pedersen HN, Sy S, et al. : Community-based HPV self-collection versus visual inspection with acetic acid in Uganda: a cost-effectiveness analysis of the ASPIRE trial. BMJ Open. 2018;8(6):e020484. 10.1136/bmjopen-2017-020484 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Ministry of Health Ethiopia: Essential Health Services Package of Ethiopia.2019. Reference Source [Google Scholar]
  39. Ministry of Health and Family Welfare: The Health Technology Assessment Board Act.2019. Reference Source [Google Scholar]
  40. Morton A, Arulselvan A, Thomas R: Allocation rules for global donors. J Health Econ. 2018;58:67–75. 10.1016/j.jhealeco.2018.02.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Mohara A, Youngkong S, Velasco RP, et al. : Using health technology assessment for informing coverage decisions in Thailand. J Comp Eff Res. 2012;1(2):137–146. 10.2217/cer.12.10 [DOI] [PubMed] [Google Scholar]
  42. National Institute for Health and Care Excellence: Guide to the processes of technology appraisal. National Institute for Clinical Excellence.2018. Reference Source [PubMed] [Google Scholar]
  43. Newall AT, Jit M, Hutubessy R: Are current cost-effectiveness thresholds for low- and middle-income countries useful? Examples from the world of vaccines. PharmacoEconomics. 2014;32(6):525–531. 10.1007/s40273-014-0162-x [DOI] [PubMed] [Google Scholar]
  44. Ochalek J, Claxton K, Lomas J, et al. : Valuing health outcomes: developing better defaults based on health opportunity costs. Expert Rev Pharmacoecon Outcomes Res. 2020b;1–8. 10.1080/14737167.2020.1812387 [DOI] [PubMed] [Google Scholar]
  45. Ochalek JM, Lomas J, Claxton KP: Cost per DALY averted thresholds for low-and middle-income countries: evidence from cross country data. CHE Research Paper. 2015;122 Reference Source [Google Scholar]
  46. Ochalek J, Revill P, Manthalu G, et al. : Supporting the development of a health benefits package in Malawi. BMJ Glob Health. 2018;3(2):e000607. 10.1136/bmjgh-2017-000607 [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Ochalek J, Wang H, Gu Y, et al. : Informing a Cost-Effectiveness Threshold for Health Technology Assessment in China: A Marginal Productivity Approach. Pharmacoeconomics.Epub ahead of print.2020;38(12):1319–1331. 10.1007/s40273-020-00954-y [DOI] [PubMed] [Google Scholar]
  48. O’Day K, Campbell DJ: What’s The Matter With Average Cost-Effectiveness Ratios? A Simple Graphical Explanation. Value Health. 2016;19(3):A107–A108. 10.1016/j.jval.2016.03.1696 [DOI] [Google Scholar]
  49. Ottersen T, Forde R, Kakad M, et al. : A new proposal for priority setting in Norway: Open and fair. Health Policy. 2016;120(3):246–51. 10.1016/j.healthpol.2016.01.012 [DOI] [PubMed] [Google Scholar]
  50. Revill P, Walker S, Cambiano V, et al. : Reflecting the real value of health care resources in modelling and cost-effectiveness studies-The example of viral load informed differentiated care. PLoS One. 2018;13(1):e0190283. 10.1371/journal.pone.0190283 [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Robinson LA, Hammitt JK, Chang AY, et al. : Understanding and improving the one and three times GDP per capita cost-effectiveness thresholds. Health Policy Plan. 2017;32(1):141–145. 10.1093/heapol/czw096 [DOI] [PubMed] [Google Scholar]
  52. Santos AS, Guerra-Junior AA, Godman B, et al. : Cost-effectiveness thresholds: methods for setting and examples from around the world. Expert Rev Pharmacoecon Outcomes Res. 2018;18(3):277–288. 10.1080/14737167.2018.1443810 [DOI] [PubMed] [Google Scholar]
  53. Sharma M, Teerawattananon Y, Luz A, et al. : Institutionalizing Evidence-Informed Priority Setting for Universal Health Coverage: Lessons From Indonesia. Inquiry. 2020;57;46958020924920. 10.1177/0046958020924920 [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Silverman R: Projected Health Financing Transitions: Timeline and Magnitude - Working Paper 488.Washington, DC: Center for Global Development.2018. Reference Source [Google Scholar]
  55. Sullivan SD, Mauskopf JA, Augustovski F, et al. : Budget impact analysis-principles of good practice: report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value Health. 2014;17(1):5–14. 10.1016/j.jval.2013.08.2291 [DOI] [PubMed] [Google Scholar]
  56. Tantivess S, Chalkidou K, Tritasavit N, et al. : Health Technology Assessment capacity development in low- and middle-income countries: Experiences from the international units of HITAP and NICE. [version 1; peer review: 2 approved]. F1000Res. 2017;6:2119. 10.12688/f1000research.13180.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Teerawattananon Y, Tritasavit N, Suchonwanich N, et al. : The use of economic evaluation for guiding the pharmaceutical reimbursement list in Thailand. Z Evid Fortbild Qual Gesundhwes. 2014;108(7):397–404. 10.1016/j.zefq.2014.06.017 [DOI] [PubMed] [Google Scholar]
  58. Tengs TO, Meyer G, Siegel JE, et al. : Oregon’s Medicaid ranking and cost-effectiveness: is there any relationship? Med Decis Making. 1996;16(2):99–107. 10.1177/0272989X9601600201 [DOI] [PubMed] [Google Scholar]
  59. Thokala P, Ochalek J, Leech AA, et al. : Cost-Effectiveness Thresholds: the Past, the Present and the Future. Pharmacoeconomics. 2018;36(5):509–522. 10.1007/s40273-017-0606-1 [DOI] [PubMed] [Google Scholar]
  60. United Nations General Assembly: Political declaration of the high-level meeting on universal health coverage.2019. Reference Source [Google Scholar]
  61. Urrutia J, Porteny T, Daniels N: What does it mean to put new hepatitis C drugs on a list of essential medicines? BMJ. 2016;353:i2035. 10.1136/bmj.i2035 [DOI] [PubMed] [Google Scholar]
  62. US Food & Drug Administration: Avastin (bevacizumab) Information FDA.[September 9, 2019].2011. Reference Source [Google Scholar]
  63. Vallejo-Torres L, García-Lorenzo B, Castilla I, et al. : On the Estimation of the Cost-Effectiveness Threshold: Why What, How? Value Health. 2016;19(5):558–566. 10.1016/j.jval.2016.02.020 [DOI] [PubMed] [Google Scholar]
  64. Watkins DA, Jamison DT, Mills A, et al. : Universal health coverage and essential packages of care.In D. T. Jamison, H. Gelband, S. Horton, P. Jha R. Laxminarayan, C. N. Mock and R. Nugent (eds), Disease control priorities: improving health and reducing poverty. Washington (DC): The International Bank for Reconstruction and Development / The World Bank.2017. 10.1596/978-1-4648-0527-1_ch3 [DOI] [PubMed] [Google Scholar]
  65. Wiseman V, Mitton C, Doyle-Waters MM, et al. : Using Economic Evidence to Set Healthcare Priorities in Low-Income and Lower-Middle-Income Countries: A Systematic Review of Methodological Frameworks. Health Econ. 2016;25 Suppl 1(Suppl Suppl 1):140–161. 10.1002/hec.3299 [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Woods B, Revill P, Sculpher M, et al. : Country-Level Cost-Effectiveness Thresholds: Initial Estimates and the Need for Further Research. Value Health. 2016;19(8):929–935. 10.1016/j.jval.2016.02.017 [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. World Health Organisation: WHA67.23 - Health Intervention and Technology Assessment in Support of Universal Health Coverage. WHA Resolution; Sixty-seventh World Health Assembly. World Health Organisation, Geneva.2014. Reference Source [Google Scholar]
  68. World Health Organization: 2015 Global Survey on HealthTechnology Assessment by National Authorities. Geneva: World Health Organization.2015. Reference Source [Google Scholar]
  69. World Health Organization Commission on Macroeconomics and Health: Macroeconomics and health: investing in health for economic development. Geneva: World Health Organization.2001. Reference Source [Google Scholar]
Gates Open Res. 2021 Feb 1. doi: 10.21956/gatesopenres.14409.r30155

Reviewer response for version 1

Tharani Loganathan 1

Setting a cost-effectiveness threshold is crucial for making the right decisions on allocating limited national resources. Many nations, especially LMICs depend on the 1-3 GDP thresholds for its ease of use and standardisation. This paper reviews the current arguments and details five options to replace GDP based thresholds. At the end of the day, setting priorities are national decisions. And decision rules must be made at the national level, using national data and capacity, while being transparent enough for policy-makers and non-economists to apply with trust. Thus the tools should be simple, relevant and prioritise health systems goals of health maximation and equity while staying within budget.

Thus as stated, this article is written for an audience of national decision-makers but considers the global health community.

Does the article adequately reference differing views and opinions?

Yes

Are all factual statements correct, and are statements and arguments made adequately supported by citations?

Yes

Is the Open Letter written in accessible language?

Yes

Where applicable, are recommendations and next steps explained clearly for others to follow?

Yes

Is the rationale for the Open Letter provided in sufficient detail?

Yes

Reviewer Expertise:

Vaccine economics

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

Gates Open Res. 2021 Jan 13. doi: 10.21956/gatesopenres.14409.r30128

Reviewer response for version 1

Christopher James Sampson 1

This article provides a helpful overview of the implications for policymakers of using cost-effectiveness thresholds without relying on GDP-based estimates.

I have no major criticisms or essential revisions to suggest. My specific points – presented mostly for the purpose of demonstrating that I have fully reviewed the paper – are as follows.

  1. In places, the article conflates decisions about new investments, decisions about coverage, and decisions about the optimal allocation of a budget within or beyond health care. The extent to which cost-effectiveness thresholds (as described) can or should inform such decisions is a matter for debate. Therefore, it would be helpful if the article more clearly articulated the types of decisions to which their discussion relates.

  2. I believe that the framing of the Options as ‘alternatives’ is unhelpful. Arguably, Options 1 through 4 are – collectively – an alternative to 5. But the most appropriate recommendation is surely that decision-makers consider pursuing all five options rather than just one ‘alternative’. The five options would be better characterised as a to-do list than a menu – as complements, not substitutes – ‘Steps’ or ‘Strategies’ rather than ‘Options’.

  3. The authors do identify its limitations, but the article still presents Option 5 as if it is a panacea. Indeed, ‘Options’ 2 and 3 in particular remain relevant even with the realisation of Option 5.

  4. The first footnote claims that “There is no published review of the use of CETs by decision-makers.” I’m not sure this is true. I believe that Cameron et al (2018), which is already cited, and Zhang and Garau (2020) do this, at least to some extent. Perhaps the claim is not precisely articulated.

Does the article adequately reference differing views and opinions?

Yes

Are all factual statements correct, and are statements and arguments made adequately supported by citations?

Yes

Is the Open Letter written in accessible language?

Yes

Where applicable, are recommendations and next steps explained clearly for others to follow?

Yes

Is the rationale for the Open Letter provided in sufficient detail?

Yes

Reviewer Expertise:

health economics, cost-effectiveness analysis, health technology assessment, cost-effectiveness thresholds

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

References

  • 1. : On what basis are medical cost-effectiveness thresholds set? Clashing opinions and an absence of data: a systematic review. Glob Health Action.2018;11(1) : 10.1080/16549716.2018.1447828 1447828 10.1080/16549716.2018.1447828 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2. : International Cost-Effectiveness Thresholds and Modifiers for HTA Decision Making. OHE.2020; Reference source

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Availability Statement

    Underlying data

    No data are associated with this article

    Articles from Gates Open Research are provided here courtesy of Bill & Melinda Gates Foundation

    RESOURCES