Are Estimates of the Health Opportunity Cost Being Used to Draw Conclusions in Published Cost-Effectiveness Analyses? A Scoping Review in Four Countries

Abstract

Background

When healthcare budgets are exogenous, cost-effectiveness thresholds (CETs) used to inform funding decisions should represent the health opportunity cost (HOC) of such funding decisions, but HOC-based CET estimates have not been available until recently. In recent years, empirical HOC-based CETs for multiple countries have been published, but the use of these CETs in the cost-effectiveness analysis (CEA) literature has not been investigated. Analysis of the use of HOC-based CETs by researchers undertaking CEAs in countries with different decision-making contexts will provide valuable insights to further understand barriers and facilitators to the acceptance and use of HOC-based CETs.

Objectives

We aimed to identify the CET values used to interpret the results of CEAs published in the scientific literature before and after the publication of jurisdiction-specific empirical HOC-based CETs in four countries.

Methods

We undertook a scoping review of CEAs published in Spain, Australia, the Netherlands and South Africa between 2016 (2014 in Spain) and 2020. CETs used before and after publication of HOC estimates were recorded. We conducted logit regressions exploring factors explaining the use of HOC values in identified studies and linear models exploring the association of the reported CET value with study characteristics and results.

Results

1171 studies were included in this review (870 CEAs and 301 study protocols). HOC values were cited in 28% of CEAs in Spain and in 11% of studies conducted in Australia, but they were not referred to in CEAs undertaken in the Netherlands and South Africa. Regression analyses on Spanish and Australian studies indicate that more recent studies, studies without a conflict of interest and studies estimating an incremental cost-effectiveness ratio (ICER) below the HOC value were more likely to use the HOC as a threshold reference. In addition, we found a small but significant impact indicating that for every dollar increase in the estimated ICER, the reported CET increased by US$0.015. Based on the findings of our review, we discuss the potential factors that might explain the lack of adoption of HOC-based CETs in the empirical CEA literature.

Conclusions

The adoption of HOC-based CETs by identified published CEAs has been uneven across the four analysed countries, most likely due to underlying differences in their decision-making processes. Our results also reinforce a previous finding indicating that CETs might be endogenously selected to fit authors’ conclusions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40258-021-00707-8.

Key Points for Decision-Makers

In health systems aiming to maximise population health from a constrained budget, published cost-effectiveness analyses (CEAs) should compare incremental cost-effectiveness ratios (ICERs) with cost-effectiveness thresholds (CETs) reflecting the health opportunity cost (HOC) of funding decisions.

HOC values were cited in 28% of CEAs in Spain and in 11% of studies conducted in Australia, but they were not referred to in CEAs undertaken in the Netherlands or South Africa.

Through regression analyses, we found that more recent studies, studies without a conflict of interest and studies estimating an ICER below the HOC value were more likely to use the HOC as a threshold reference.

Introduction

In most countries health technology assessment (HTA) processes are conducted to inform coverage decisions, which often incorporate formal cost-effectiveness analyses (CEAs). CEA results are often summarised by the incremental cost-effectiveness ratio (ICER), which measures the technology’s incremental cost per unit of effectiveness gained, where effectiveness is commonly quantified in quality-adjusted life years (QALYs). However, information on the incremental cost per QALY gained of technologies that are more costly and provide better outcomes than their comparators is not enough to judge whether an intervention is cost-effective. To draw any conclusion regarding cost-effectiveness, the ICER of the technology needs to be compared with a ceiling value that determines whether the introduction of the new technology leads to an efficient use of limited resources; this ceiling value is known as the cost-effectiveness threshold (CET).

Among the many countries that use CEAs to inform funding decisions (such as the United Kingdom, Australia, New Zealand, Canada, Sweden, the Netherlands, Finland, Norway, Denmark and others), very few are explicit about the CET figure used to draw recommendations and to inform decisions. Two exceptions are the National Institute for Health and Care Excellence (NICE) in England and Wales and the Dutch National Health Care Institute (Zorginstituut Nederland; ZIN). Since 2004, NICE guidelines explicitly refer to a range of £20,000–£30,000 per QALY to judge the acceptability of a technology as “an effective use of NHS (National Health Service) resources” [1]. In the case of the Netherlands, ZIN formally adopted in 2015 a range of €20,000–€80,000 per QALY [2]. In most other countries, health authorities do not explicitly report using a CET value to inform decisions, but specific values are commonly cited in the literature. This is the case, for instance, in the USA and Australia were the figure of $50,000 per QALY is widely cited [3, 4], as well as ranges from $20,000 to $100,000 [5, 6] in the USA and Canada, and, more recently in the USA, from $100,000 to $150,000 [7]. The basis for these values is sometimes unclear, but often relies on the observation of past decisions or on the World Health Organization (WHO) rule of thumb of using one to three times a country Gross Domestic Product (GDP) per capita [8]. The lack of theoretical and empirical basis regarding available CETs has led to an increasing body of empirical research.

Empirical research estimating a CET has been grounded in two different conceptual views that differ on what the CET ought to reflect, either a societal monetary valuation of health gains or the opportunity cost resulting from the disinvestment required to adopt a new technology [9]. The former view has been considered relevant under a context of flexible budgets and when a broad societal perspective is taken into account [10]. In the most commonly operating contexts where fixed budgets are allocated to healthcare and coverage decisions are taken from a healthcare system perspective, information on the opportunity cost of funding decisions becomes the relevant information to inform a CET [10, 11].

The opportunity costs of a funding decision are the gains that are necessarily forsaken when the resources required to fund an intervention are no longer available for its best alternative use. In practice, it is extremely rare to know the specific alternative that would get displaced or not funded in a given real-world decision. A proposed alternative to measure the opportunity costs of healthcare funding decisions consists of empirically estimating the marginal cost per unit of health produced by the healthcare system on average. Following seminal work in the UK [12–15], empirical estimates of the health opportunity cost (HOC), approximated by the marginal cost-effectiveness of current spending in healthcare, have been recently published in a number of countries, including Spain [16] , Australia [17], the Netherlands [18, 19], Sweden [20], South Africa [21], China [22] and the USA [23]. However, the extent to which this information is used in the recent health economics literature has not been investigated.

The primary aim of this study is to identify the CET values used to interpret the results of CEAs published in the scientific literature before and after the publication of jurisdiction-specific empirical HOC-based CETs in four countries. Analysing the use of HOC-based CETs by researchers undertaking CEAs provides a quantitative assessment of the impact that HOC studies have had on the CEA empirical research, and by taking a multi-country approach, the analysis provides valuable insights to further understand barriers and facilitators to the acceptance and use of HOC-based CETs.

To conduct this study, leading authors of each of the HOC-based CET empirical studies were contacted and invited to participate. Countries were included in our study if at least one country-based author agreed to participate as a member of the review team; these included Spain, Australia, the Netherlands and South Africa. We undertook a scoping review of CEAs published in these four countries to identify the values and sources of the CET figures being used (if any) before and after the publication of the estimates of the HOC in each country. We then explored the study characteristics that explained the use of HOC-based CETs. In addition, following a previous finding indicating that selection of CETs might be endogenous to the ICER of the technology being evaluated [24], a secondary aim of this study was to explore whether the quantitative value of the selected CET was associated with study characteristics, and particularly to the reported ICER result.

Health System Characteristics and the Role of Cost-Effectiveness Thresholds (CETs) in Decision-Making

The nature and values of CETs used in empirical research are inevitably connected to the decision-making context they are applied to. In this section, we briefly describe the health system characteristics and the role that CEAs have in decision-making in each analysed country.

In Spain, the public health service is financed through general taxation and offers universal coverage to all residents. While the Law (Royal Decree 16/2021) clearly states that health-funding decisions ought to be based on scientific evidence regarding the cost-effectiveness of health technologies, the role of CEA in decision-making is not clear, particularly for pharmaceutical products. Formal HTA processes are followed though by the Spanish Network for Health Technology Assessment and Services of the NHS (RedETS), which assesses mostly non-pharmaceutical technologies and often includes CEAs. An explicit CET figure has not been formally adopted by health authorities, although the value of 30,000€ per life year (LY) or per QALY is widely cited, after a literature review of economic evaluations conducted in Spain concluded that the authors of identified papers were likely to recommend the adoption of the technology under study if the ICER value was below this number [25]. In 2015, the Ministry of Health commissioned RedETS to provide further information on how to estimate a CET. Part of that project was an empirical analysis that estimated the marginal cost per QALY of the Spanish NHS to be between €22,000 and €25,000 [16, 26].

In Australia, cost-utility analysis and the incremental cost per QALY gained are used to inform Commonwealth government decisions to subsidise the funding of pharmaceuticals and medical services, for which evidence of value for money to the taxpayer is a criterion for subsidisation (National Health Act). The Pharmaceutical Benefits and Medical Services Advisory Committees (PBAC and MSAC) assess evidence on new pharmaceuticals and medical services, respectively, to make recommendations for public subsidisation to the Minister for Health. In public summary documents, the PBAC and MSAC refer to ICERs as being acceptable or not, but the use of a CET is not acknowledged. The most recent review of PBAC funding decisions, between 2005 and 2009, indicated that one-third of proposed pharmaceuticals with an ICER between AUD45,000 and AUD75,000 per QALY gained were funded [27], and in the CEA literature the figure of AUD50,000 is commonly cited. Research to estimate empirical HOC values was funded through competitive investigator-driven research funding, which generated an estimate for Australia of AUD28,033 per QALY gained (95% confidence interval (CI) 20,758–37,667) [28].

The Netherlands’ system of managed competition between health insurers includes a broad statutory benefits package, defined by the Ministry of Health, Welfare and Sports. All health insurers must cover the statutory benefits under mandatory basic health insurance. The statutory benefits package is mostly open ended, where new technologies are automatically accepted when compliant with ‘established medical science and practice’ [29]. An exception is new drugs with a high budget impact have to be formally approved by the Minister of Health. The Advisory Committee Benefit Package (Adviescommissie Pakket; ACP), part of ZIN, drafts an advice for the Minister based a number of criteria, including cost-effectiveness [2]. Since 2015, ZIN uses a CET range of €20,000–€80,000 per QALY, depending on burden of disease [2]. These values were based on a 2006 advice of the Dutch Council for Public Health and Healthcare (Raad voor Volksgezondheid en Zorg; RVZ) [30]. Concerns of displacement by new technologies incentivised ZIN to commission a research programme in 2017 to estimate a CET empirically. The Radboud University and Medical Centre Nijmegen estimated a marginal cost per QALY of hospital spending of €73,400 per QALY (95% CI 53,000–93,000) [18]. Simultaneously, the Erasmus University Rotterdam estimated a CET based on cardiovascular spending of €41,000 per QALY [19].

In South Africa, public healthcare is financed through general taxation, which provides services to approximately 85% of the total population, largely free of charge at the point of care. In choosing health technologies to fund within the public sector, several decision-making bodies at national and provincial levels employ some form of HTA process that often includes the review of evidence of the cost-effectiveness of interventions. However, there is no formal CET recommended by these decision-making authorities. As with other low- and middle-income (LMIC) countries, the most widely cited threshold in South Africa has been the WHO one- to three-times GDP per capita threshold [31]. In 2020, a study was conducted to assess the appropriateness of WHO recommendations for use in South Africa. This study estimated an empirical threshold of US$3,015 per disability-adjusted life year (DALY) averted, approximately 50% lower than the widely cited WHO threshold [21].

Methods

Search Methods

A systematic scoping review of published CEAs was performed following accepted methods and presented according to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines [32, 33]. A scoping review was conducted due to the relatively broad scope of the review, which aimed to identify how conclusions are drawn on current cost-effectiveness research [34]. Scoping reviews maintain the same methodological rigor as traditional systematic reviews, with a key difference being that a critical appraisal of identified studies is not required. The search was assisted by a librarian of the Flinders University Library Services and conducted in international electronic databases (MEDLINE, SCOPUS, ISI Web of Science, International HTA Database by INAHTA, Econlit and Embase) on 2 November 2020. Full details of the search strategy can be found in Electronic Supplementary Material (ESM) S1. Records were imported into Thomson Reuters Reference Manager v.10 to remove duplicates and then exported into Rayyan (https://rayyan.qcri.org/) for title and abstract screening against inclusion criteria.

The inclusion criteria consisted of original studies or study protocols: (1) published between January 2016 (January 2014 for Spain) and 2 November 2020, (2) undertaking a full CEA in Spain, Australia, the Netherlands or South Africa, (3) using LY/QALY/DALYs as the effectiveness unit, and (4) reporting an ICER value (unless the evaluated technology was found to be dominant (dominated), i.e., intervention was less (more) costly and provided better (worse) outcomes). A full CEA was defined as a study comparing the costs and health outcomes of at least two competing interventions. Study protocols were included to identify the planned use of CETs in ongoing CEAs. Starting search dates were selected to include a period of 2 years prior to the first publication of the empirical HOC estimate in each country, which in the case of Spain was first published as a report to the Ministry of Health in January 2016 [26].

Identified abstracts were independently reviewed and assessed by two reviewers. Full texts of selected records were then reviewed and classified as included or excluded according to the specified inclusion criteria. For all included papers, information on the general study details, the CET value and CET source cited (if any) was extracted by a researcher and checked by a second researcher. This information was used for the descriptive analysis of the four analysed countries, which compared the CET values and sources used before and after the publication of the empirical HOC estimate in each setting. A subset of these studies was also evaluated in regression analyses (regression methods described below) to explore the characteristics explaining the adoption of the HOC estimates and the CET value selected in countries where studies citing the HOC estimate as the benchmark CET were found. Further study characteristics for this subset of studies were extracted. Critical appraisal was not conducted as it was irrelevant to the purpose of the study. Table 1 describes the extracted information used in descriptive and regression analyses.

Table 1.

Extracted information of included studies

Variable	Values/Categories
General information for descriptive analysis
Country	Spain, Australia, the Netherlands, South Africa
Received datea	DD/MM/YYYY
Received date after publication of HOC estimate = 1; otherwise = 0	Spain: if received date was after 04/01/2016 (online publication of report to the Ministry of Health [25]) Australia: if received date was after 22/12/2017 (online publication of Edney et al. [21]) The Netherlands: if received date was after 01/10/2018 (online publication of van Baal et al. 2019 [19]) South Africa: if received date was after 03/03/2020 (online publication of Edoka et al. 2020 [21])
CET valueb	Numerical (e.g., 20,000€/QALY)
CET source	Text (e.g., Edney et al. 2018) [21]
CET descriptionc	Text (original text where authors justify the use, if any, of the selected CET)
Information extracted/constructed for regression analyses
Dependent variable
Used HOC value (binary)	HOC estimate used as CET = 1; otherwise = 0
CET value (continuous)c	Numerical expressed in US$
Explanatory variables
Time since HOC publication	Numerical (days from HOC estimate publication date to study submission to journal)
Conflict of interest	Yes; Potentiald; None; Not reported
Disease/condition	ICD-10 chapters
Rare disease	No; Yese
Intervention type	Pharmaceutical/vaccine; device; screening/tests; surgery/procedure; educational/behavioural; other
Comparator	Do nothing/placebo; usual care; another alternative
Methodology	Model-based; observational; controlled trial; other
Perspective	Societal; public healthcare system; healthcare provider; patient; other
Population	Newborns and infants; children and adolescents; adults; elderly non age-specific
CEA study result categories: Spain/Australiaf	Dominant; ICER < €20k/AUS$20k; ICER between €20k/AUS$20k to €30k/AUS$50k; ICER > €30k/AUS$50k or dominated
ICER value (continuous)	Numerical expressed in US$
Country	Spain; Australia

CEAs cost-effectiveness analyses, CET cost-effectiveness threshold, HOC health opportunity cost, QALY quality-adjusted life year, ICER incremental cost-effectiveness ratio, ID International classification of diseases

^aWe used the received date of the paper by the journal (submission date) when indicated to categorise the paper as before/after the publication of the HOC estimate instead of the publication date to allow for the lag between submissions and publications of accepted papers

^bIf authors used a range, the upper limit was recorded

^cThis information was collected to aid understanding the rationale for authors to select specific CET values. The most common reasons reported included that the value was “the CET most commonly cited” (for arbitrary values), “the value applied by regulatory bodies” (for policy thresholds), and “a recent empirical estimation” (for HOC-based values)

^dPotential conflict of interest was recorded when study co-authors reported having received funding from industry in the past

^eIf study authors reported the disease was rare

^fRanges were defined differently in Australia and Spain to relate categories to whether the ICER was below/above the commonly cited CET figure used in each country

Regression Analysis Methods

We conducted a regression analysis exploring the factors associated with the probability of using HOC estimates as CET among the countries where studies citing the HOC estimates were identified. The unit of observation in these analyses was the identified studies. If studies reported more than one ICER as a base case, we recorded the lowest ICER reported. The following logit regression models were specified:

$${P(HOC}_i=1)={\beta }_0+\sum _{k=1}^K\sum _{i=1}^{\begin{array}{c}\\ N\end{array}}{\beta }_K{X}_i+{\mu }_i$$

where ${\mathit{HOC}}_i=1$ if the HOC estimate was used as CET by study i and ${\mathit{HOC}}_i=0$ if otherwise. The $X_i$ covariates were defined as the characteristics of study i, showed in Table 1, ${\beta }_0$ and ${\beta }_K$ are the constant and the parameters associated to covariates $X_i$ and ${\mu }_i$ the error term.

In addition, to explore the secondary aim of this study, we conducted a second econometric specification to identify the drivers of the selection of the CET value among the studies where a CET was used (i.e., studies not reporting any CET were excluded), and which also reported an ICER result (i.e., studies where the evaluated technology was dominant/dominated were excluded). The following linear regression using an ordinary least square (OLS) model was estimated:

$${\mathit{CET}}_i={\alpha }_0+\sum _{k=1}^K\sum _{i=1}^{\begin{array}{c}\\ N\end{array}}{\alpha }_K{X}_i+v_i$$

where ${\mathit{CET}}_i$ was defined as the CET figure used to compare the ICER reported by study i. If authors used a CET range, the upper limit was recorded. Similar to the previous model, the $X_i$ covariates were defined as the characteristics of study i, ${\alpha }_0$ and ${\alpha }_K$ are the constant and the parameters associated with the covariates $X_i$, and $v_i$ is the error term.

We ran full models including all potential covariates and reduced models using stepwise regression to remove covariates that did not show associations. The selection of the variables included as potential covariates was based on (i) characteristics that have been argued to warrant a higher CET (e.g., the focus on specific disease conditions such as cancer disease, and whether the disease is considered rare), (ii) characteristics that might make the HOC-based estimate less relevant (e.g., studies taking a broader perspective, intervention types that might be associated with broader costs and benefits, and whether the CEA study was conducted soon after the publication of the HOC value), and (iii) characteristics that might introduce a bias towards selecting CETs that suits a hypothesis (e.g., the presence of a conflict of interest, and whether the ICER result was above HOC values). When HOC-based values are below alternative CET sources commonly cited in a given setting, we expect the above variables to decrease the probability of citing the HOC value as the benchmark CET as well as to increase the quantitative value used as CET. For some other variables, i.e., the target population, the methodology and the comparator used, we did not have an a priori hypothesis regarding its effect but its introduction in the full models was explorative. Models were conducted in the combined sample of identified studies and include country fixed effects. In the linear specification, CET and ICER values were converted into US dollars to be expressed into a common currency (using average 2020 exchange rates €1 = US$1.18 and AUD1 = US$0.74). We ran sensitivity analyses excluding studies with a received date within 6 months of the publication date of the HOC estimation and analyses setting the continuous CET value (in the second specification) as the midpoint of the reported range instead of the maximum value. Analyses were undertaken in Stata 16 [35].

Results

Search Results

17,172 records were identified (see ESM S1). After removing duplicates and screening by title/abstract, 2018 records were assessed for eligibility, and finally 1171 studies were included after full-text screening. In the regression analyses that evaluated the use of HOC estimates, only completed CEAs for countries where the HOC estimate was found to be cited and that were conducted after the publication of the HOC estimate were included, i.e., 242 studies (Fig. 1).

Fig. 1.

Flow diagram of study selection

Descriptive Analysis

Table 2 shows the descriptive analyses of the CETs used in the identified studies per country. Assessed studies include study protocols and empirical CEAs. In the vast majority of study protocols (94% across the four analysed countries), the CET intended to be used to judge the cost-effectiveness of the evaluated intervention was not indicated, and therefore we did not record separately the sources used among the few studies that did so. Percentages of study protocols that do not cite a CET ranged from 75% in South Africa to 100% in Spain prior to 2016 (see Table 2). In the following paragraphs we describe the threshold values and references used (if any) in empirical CEAs in each analysed country.

Table 2.

Descriptive analysis of CETs used in identified studies per country

Spain	Before (2014–2015)	After (2016–2020)	Australia	Before (2016–2017)	After (2018–2020)	Netherlands	Full period (2016–2020)	South Africa	Full period (2016–20)
Total studies	129	144	Total studies	162	174	Total studies	511	Total studies	51
Protocols	14 (11%)	21 (15%)	Protocols	46 (28%)	55 (32%)	Protocols	161 (32%)	Protocols	4 (8%)
CEAs	115 (89%)	123 (85%)	CEAs	116 (72%)	119 (68%)	CEAs	350 (68%)	CEAs	47 (92%)
CETs used in protocols			CETs used in protocols			CETs used in protocols		CETs used in protocols
None	14 (100%)	17 (81%)	None	44 (96%)	48 (87%)	None	156 (97%)	None	3 (75%)
CETs used in CEAs			CETs used in CEAs			CETs used in CEAs		CETs used in CEAs
1. €30,000	78 (68%)	39 (32%)	1. AUD50,000	75 (65%)	71 (60%)	1. €20,000 - €80,000	194 (55%)	1. 1-3 times GDP	30 (64%)
Justification:			Justification:			Justification:		Justification:
Sacristán [24]	54 (47%)	17 (14%)	None	23 (20%)	22 (18%)	RVZ [28]	41 (12%)	WHO [8]	17 (36%)
Other	12 (10%)	6 (5%)	George [4]	5 (4%)	10 (8%)	ZIN [2]	61 (17%)	Other	10 (21%)
None	12 (10%)	14 (11%)	Harris [37]	16 (14%)	8 (7%)	Other	36 (10%)	None	3 (6%)
			Other	31 (27%)	31 (26%)	None	56 (16%)
2. €20,000–€25,000	NA	27 (22%)	2. AUD28,000	NA	13 (11%)	2. Other	78 (22%)	2. Other	9 (19%)
3. Both €20,000–€30,000	NA	7 (6%)	3. Other	29 (25%)	20 (17%)	3. None	78 (22%)	3. None	8 (17%)
4. Other	15 (13%)	29 (24%)	4. None	12 (10%)	15 (13%)
5. None	22 (19%)	23 (19%)

CEAs cost-effectiveness analyses, CET cost-effectiveness threshold

In Spain, we identified 115 original CEAs published in the period from January 2014 to December 2015, i.e., prior to the first publication of the HOC estimate. Among them, most studies (68%) used the €30,000 figure as their CET. The most widely cited study for this value was Sacristan et al. [25], which was referred to in 52 studies (47%), but some papers (10%) did not use any reference to justify the use of the €30,000 value, and others (10%) used different studies as reference, such as De Cock et al. [36] and the UK NICE threshold [37]. In addition, 15 studies (13%) used different threshold values referring to alternative sources, such as a willingness-to-pay (WTP) study [38] and a study measuring the monetary value of a QALY in the context of traffic accidents [39]. Nineteen percent of identified CEAs in Spain did not use any CET. After the publication of the HOC estimate in January 2016, the percentage of CEAs citing the figure of €30,000 dropped to 32%, of which 17 studies (14%) cited Sacristan et al. [25] as their source. The range between €20,000 (or €22,000) and €25,000 is cited in 22% of CEAs, all referring to the HOC value estimated in Vallejo-Torres et al. study [16, 26], and an additional 6% of studies cited both Sacristan and Vallejo-Torres and used a range between €20,000 and €30,000. Finally, 29 studies (24%) used alternative CETs, such as the WHO rule of one to three times per capita GDP (about €25,000–€75,000 in Spain) and the UK threshold of £20,000–£30,000 per QALY [8, 37], while 23 studies (19%) conducted in Spain in this latter period did not refer to any CET.

Among the 116 CEAs identified for Australia in the period from January 2016 to December 2017, 75 studies (65%) used the CET figure of AUD50,000. The authors of these papers refer to different sources, including George et al. [4], Carter et al. [40], Harris et al. [41] and Neumann et al. [3], but many studies (20%) citing the AUD50,000 value do not provide any reference to justify it. An additional 29 papers (25%) use alternative threshold values and sources, such as a WTP study by Shiroiwa et al. [42], and the UK NICE threshold [37]. Only 10% of CEAs published in 2016 and 2017 in Australia did not refer to any CET value. With respect to CEAs conducted after the HOC estimate provided by Edney et al. [17] was published in Australia, the percentage of papers citing AUD50,000 fell to 60%, while the study by Edney et al. [17] was cited in 13 studies (11%). The percentage of papers using alternative CETs rather than the AUD50,000 or the AUD28,000 decreased from 25% to 17%, while 13% of CEAs after the publication of the HOC value in Australia did not refer to any CET.

In the Netherlands, the results are presented for the full period (2016–2020) because none of the empirical HOC estimates [18, 19] were cited by any of the identified studies. We identified 350 studies performing a CEA (42% after the publication of the HOC estimate). 194 studies (55%) used the official threshold values of €20,000–€80,000 per QALY, where 41 papers (12%) referred to the RVZ advice [30], 61 papers (17%) to the ZIN guidelines [2], 36 papers (10%) used other references, and 56 papers (16%) used the official threshold without any reference. Seventy-eight papers (22%) used a threshold different from the official CET, including references to the WHO rule (approximately €30,000–€90,000) [8], the UK threshold proposed by NICE [37], an alternative guideline for CEA commissioned by the Collective of Health Insurers (College voor de Zorgverzekeringen; CVZ) [43], and a WTP study by Bobinac et al. [44]. In 22% (78 papers), no threshold was used.

Studies identified in South Africa did not refer to the HOC value estimated by Edoka et al. [21], and are therefore also presented for the full period. Among the 51 CEAs conducted between 2016 and 2020 (only 11% of them after the publication of the HOC empirical value), the most widely used threshold was the one- to three-times country GDP per capita (approximately US$6,000–US$18,000), which was used in 64% of identified studies. Of these studies, 50% cited WHO recommendations as justification for their choice. Other thresholds used in identified studies include the cost per QALY threshold estimated by Woods et al. [45], the cost per DALY estimated by Ochalek et al. [46] and a disease-specific threshold identified by Meyer-Rath et al. [47]. In 17% of South African studies included in this review, no thresholds were used.

Regression Analyses Results

Given that empirical HOC estimates were not cited by any of the identified studies in the Netherlands and South Africa, regression analyses were performed using only studies conducted in Spain and Australia after the publication of the HOC estimate. Summary statistics of all variables are presented in ESM S2, alongside with the regression results of full models including all potential covariates.

Table 3 presents the results for the reduced logit model and reduced OLS model, which only kept covariates that were significant at a 10% significance level. In the binary model, the following variables were negatively associated with the use of the HOC value: if the study authors had a conflict of interest, if the intervention type was educational/behavioural, and if the study was based on observational data (the latter only weakly significant), while the following variables were positively associated with the use of the HOC estimate: a longer period between the time of the publication of the HOC estimate and the time the study was conducted, if the ICER of the intervention was below €20,000/AUD20,000, and if the study was conducted in Spain (the latter only weakly significant). In the linear model, only two variables were found to be significantly associated with the value of the CET used in the study; the ICER of the intervention, indicating that studies with higher ICER results used higher CETs, and if the condition under study was considered rare (the latter only weakly significant).

Table 3.

Summary statistics and regression analyses results of reduced logit and OLS models

Dependent variables	Mean (SD)	Coefficient (SE)
Logit model: HOC = 1; otherwise = 0	0.186 (0.390)
OLS model: CET continuous	38,957.3 (50,896.1)
Explanatory variables		Logit model	OLS model
Time since HOC publication (days)	726.2 (421.3)	0.001*** (0.001)
Conflict of interest
Yes	0.331 (0.471)	− 0.955** (0.429)
Rare disease = 1; = 0 otherwise			12,728.5* (6664.6)
Intervention type
Educational/behavioural	0.132 (0.339)	− 1.629** (0.800)
Methodology
Observational study	0.062 (0.242)	− 1.526* (0.863)
CEA result Spain/Australia
ICER under €20k/AUS$20k	0.397 (0.490)	0.755** (0.364)
ICER results (continuous)	39,943.2 (96604.0)	NA	0.015*** (0.0003)
Country
Spain	0.508 (0.501)	0.799* (0.483)
Constant		− 2.941*** (0.464)	34963.8*** (1089.3)
Observations	242	242	151

SD standard deviation, SE standard error, CEA cost-effectiveness analysis, CET cost-effectiveness threshold, HOC health opportunity cost, ICER incremental cost-effectiveness ratio

^***p value < 0.01; **p value < 0.05; *p value < 0.1

Models excluding studies submitted within only 6 months of the HOC study publication date and models defining the CET value as the midpoint when a CET range was reported yielded the same results as presented in Table 3 (results not shown but available upon request).

Discussion

The use of evidence-based CETs based on HOC estimates was uneven across the identified CEAs published in the scientific literature in the four analysed countries. HOC-based CETs are now cited in 28% of CEAs in Spain and referred to in 11% of studies conducted in Australia. However, they were not referred to in CEAs undertaken in the Netherlands or South Africa over the period of this review. In regression analyses, we observed that in Spain and Australia more recent studies are more likely to cite the HOC value, indicating an increasing acknowledgment of these figures. The presence of a conflict of interest by the authors conducting the study and ICER values above the HOC figure were found to decrease the likelihood of using HOC-based values, which is unsurprising given that in both countries the alternative CET values most widely cited are above HOC estimates. We also found that educational/behavioural interventions are less likely to be compared against HOC-based CETs, which might reflect that these types of interventions are commonly associated with costs that do not solely fall on healthcare systems, making the HOC-value less relevant. Some weak associations were found with respect to whether the study was based on observational data and with studies conducted in Spain. While an interpretation of the former finding can only be speculative, and potentially linked to the correlation between this variable and other study characteristics, the latter finding reflects the higher proportion of papers citing the HOC-based value in Spain as compared to in Australia.

In the linear model, we observed a significant association between the ICER value and the CET value selected by researchers as the benchmark; for every dollar increase in the estimated ICER, the CET increased by US$0.015. This finding is in line with but of a smaller magnitude than a previous study conducted in the USA that concluded that the CET was endogenous to the ICER of the technology being evaluated and showed that CEAs with higher ICERs cited higher CET figures; CETs were found in this study to grow by US$0.37 for each dollar increase in the estimated ICER [24]. Our modelling approach was slightly different to Padula et al. [24], who used data from 2000 to 2017 in the USA and applied year of publication fixed effects in their analyses. Instead, we controlled for the time elapsed since the publication of the HOC estimate because the year of publication varied by country. Including year fixed effects in our analyses did not show statistical significance and did not have an impact on our results. In this second regression, we also found a weak association between the CET value used and whether the disease was considered rare. This finding might reflect that some jurisdictions apply CET modifiers when appraising highly specialised treatments. For example, the UK have used a benchmark up to ten times higher than its standard upper limit for treatments of rare conditions, i.e., to up to £300,000 per QALY [48].

Analysing four countries from three different continents, differing not only with respect to their health system characteristics but also with respect to the role of CEA in decision-making, have offered us the possibility to explore the use of HOC values in the CEA empirical research undertaken in very different contexts. Our findings allow us to hypothesise about the factors that might explain the lack of adoption of HOC estimates as reference values to deem a technology cost-effective in the published literature. First, it is worth noting that the HOC of funding decisions might not be the most relevant information to draw conclusions on the cost-effectiveness of new technologies in every context. As mentioned earlier, in situations where costs do not solely fall on healthcare systems and when budgets are endogenous and can be increased to accommodate the costs of new interventions, information on the consumption value of health becomes more relevant to conclude on the cost-effectiveness of these interventions. This is because the opportunity costs of such funding decisions would not fall within the health system but across other alternative uses of public spending. This has been argued to be the case in the Netherlands, in which the goal is to maximize social welfare from a more flexible budget [10], and this might partly explain the lack of references to the HOC values. Secondly, and also related to the Dutch context, the presence of an official decision threshold value that is not adjusted in the light of empirical HOC estimations clearly prevents analysts from using HOC values as reference CETs. This might also be the case in England and Wales, given NICE’s position of maintaining the current £20,000–£30,000 threshold range.1 In such circumstances, authors might prefer to compare their results to the official approval norm to aid decision-making in the context where the CEA is to be applied. However, it has recently been suggested that when a discrepancy exists between the policy threshold and the empirical evidence regarding the HOC, CEA analysts should compare their results according to both [49]. Authors might still refrain from doing so if they are unconvinced by the available empirical studies or if HOC estimates are sufficiently close to official thresholds. The latter point has been argued in the case of the Netherlands, where the Dutch National Health Care Institute stated in a reply to the empirical research findings that, in their view, the empirical work validated the use of the €20,000–€80,000 per QALY as reference values [50].

Different issues arise when decision-making bodies do not have an explicit CET to inform decision-making. The case of South Africa shows a very strong alignment to the recommendation made by WHO; 64% of identified studies used one to three times per capita GDP as CETs. This finding is in line with Leech et al., who found in their review that 66% of CEAs conducted in LMICs from 2000 to 2015 cited the WHO recommendation [31]. For South Africa, the potential influential impact of the empirical analysis undertaken by Edoka et al. [21] might be seen in the course of time, as this was the most recent among the HOC estimates explored in this study. The in-depth analyses conducted for Spain and Australia offered us the possibility of exploring two jurisdictions with similar contextual factors and some disparities. In both countries, the level of discretion for analysts to select a value to which to compare the ICER of the technology under study is wide, and commonly cited CET values do not have any empirical grounding or institutional support (i.e., decision-making bodies do not reference a specific CET value). The observed differences between the two countries might shed light on the barriers faced for the adoption of HOC estimates in these settings. The lower adoption observed in Australia might be the result of, at least, three factors. One is the wider gap between the HOC estimate and the arbitrary values commonly cited in Australia (AUD28,000 vs. AUD50,000) as compared to that in Spain (€22,000–€25,000 vs. €30,000). This implies that the use of the HOC value in Australia might mean that more interventions that are judged to be cost-effective under arbitrary CETs would not be so under the empirical HOC, making the reference to the HOC value less convenient for authors with an a priori hypothesis to prove. Secondly, the time elapsed since the first publication of the empirical values was found to impact on the extent to which the HOC values are referenced. While in Spain the HOC estimation was first published as a report in early 2016, the HOC estimate was not published until the end of 2017 in Australia. And thirdly, the fact that the empirical work conducted in Spain was commissioned by the Ministry of Health might have provided researchers with a stronger incentive to use them.

Over and above the role of HOC estimates in the scientific literature, it is relevant to note that neither Spanish nor Australian authorities have formally adopted the HOC values as CETs. In the case of Spain, HTA reports incorporating a CEA and conducted by RedETS now often refer to the HOC range to draw recommendations for health authorities, but the assessment process of pharmaceutical products makes no reference to the use of a CET, which might reflect the still unclear and apparent minor role that cost-effectiveness evidence plays in decision-making in Spain. In Australia, on the other hand, the long and established history of using economic evaluations as an input in health decision-making may be a barrier to adoption of an explicit threshold value, based on the maxim “if it isn’t broke, don’t fix it”. While our study has focused on the influential impact that HOC estimates have had on the published economic evaluation literature, further research eliciting researchers’ and decision makers’ views on the validity of HOC-based values is required to assess the current and future role of HOC estimates in actual decision-making. This further research might take a mixed-methods approach, surveying authors of identified CEAs and interviewing decision and policy makers in various contexts. A review of policy reports and other grey literature not identified by our study might also complement such further research.

Finally, it is worth noting that empirical efforts have been directed towards offering an evidence-based grounding to allow the comparison of whether the expected gains of new health interventions are greater than the losses expected from diverting the resources required to fund them. We acknowledge that the scope of the gains and losses to be measured for each decision would depend on the decision makers’ perspective and their budget restrictions, and they might go beyond what it is captured by the HOC estimates discussed in this study [51]. Even then, the measurement of the opportunity costs continues to be inherently an empirical question and their values are not a political matter. Policy makers might decide to use decision thresholds that are not informed by available, context-relevant opportunity costs of funding decisions, but these policy thresholds should not be viewed as defining whether an intervention is cost-effective. Alternatively, HOC-based policy thresholds might be modified to reflect other objectives, for example, reflecting a trade-off between efficiency with respect to some equity criteria and other factors in decision-making. Such trade-offs should be explicit to avoid the confusion of deeming interventions as cost-effective when their ICERs are above the empirically measured opportunity costs. Furthermore, when applying CET modifiers (such as applying higher equity weights to specific population subgroups), a symmetrical account to the gains of the beneficiaries of the intervention and to the losses of the individuals bearing the opportunity cost who also meet the same special consideration should be applied, but this is often overlooked [52, 53]. For instance, in Spain, some researchers have recently recommended using a range between €25,000 and €60,000 to judge whether interventions are cost-effective [54], arguing that the lower limit corresponds to what has been empirically measured in previous studies in Spain and the upper limit is a discretionary suggestion based on modifiers applied to decision thresholds in other countries when accounting for additional factors. The authors do not indicate when the use of the upper limit is appropriate in the Spanish context, neither is the potential implication of using a CET nearly three times higher than the approximated HOC of funding decisions in Spain discussed—i.e. that for every unit of health gained with a new intervention, nearly three units of health would be lost elsewhere in the population (and potentially among individuals with similar equity considerations). However, based on the findings from this review, we might expect that this arbitrary range, and specifically the upper limit, will be referred to in future CEAs to conclude whether interventions are cost-effective, particularly by studies with unattractive ICER results.

In conclusion, empirical HOC values have been estimated in several countries as a vehicle to allow the comparison of whether the health gains of new health technologies are larger than the health losses expected when resources to fund them need to be diverted. In contexts where the overall aim is to maximise population health from the constraint resources assigned to the healthcare system, published CEA results should be discussed with reference to these available HOC-based CETs. We explored whether these values were used in the published CEA literature in four different countries, and our findings allow us to identify the potential factors that might prevent the adoption of empirical HOC values a CETs. Given the findings observed in our study and a previous analysis [24], suggesting that CETs might be endogenously selected to fit authors’ conclusions, we strongly recommend that claims for recommending a health technology with an ICER above a relevant HOC-based CET should be justified, for example, with explicit reference to other decision-making factors, but without disregarding the need to apply such weighting factors in a symmetrical manner to patients benefiting from new interventions and to—often invisible—patients bearing the opportunity costs. In addition, study protocols should specify the CET value or range that will be used, if any, to draw conclusions regarding cost-effectiveness to avoid the endogenous selection of CETs after ICER results are calculated.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Funding

Laura Vallejo-Torres would like to acknowledge the support of the researcher-led project RTI2018-096365-J-I00 funded by the Spanish Ministry of Science, the Spanish National Research Agency and the European Regional Development Fund (FEDER) and the research Grant from University of Las Palmas de Gran Canaria ULPGC2018-19.

Conflict of interest

None of the authors have any conflicts of interest to declare.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and material

Our data are available in the published literature through the Electronic Databases used in the search. We have included full details of the search strategy in the Electronic Supplementary Material (Table S1).

Code availability

Not applicable.

Author contributions

LVT, BGL and LCE conceived the study design. All authors contributed equally to paper selection and data extraction. LVT and BGL led the data analysis and LVT, BGL, IE, NS and JK drafted the initial manuscript. All authors commented on previous versions of the manuscript and approved the final manuscript.