Transferability of Economic Evaluations of Treatments for Advanced Melanoma

Abstract

Background

Differing methodological requirements and decision-making criteria are recognised as barriers to transferability of cost-effectiveness analysis (CEA) across jurisdictions.

Objective

We assessed the generic and specific transferability of published CEAs of systemic treatments for advanced melanoma to the Irish setting.

Methods

CEAs of treatments for melanoma were identified by systematic review. Transferability to the Irish setting was assessed using the EUnetHTA transferability tool for Economic Evaluation. We present a narrative discussion comparing the differences in key parameter inputs and the likely impact of these differences on the model outcomes and the reimbursement recommendation. Transferability is considered within the context of the Irish cost-effectiveness threshold, using the net monetary benefit (NMB) framework.

Results

No published CEAs (n = 15) aligned with the Irish reference case for CEA. Changes to key parameters were unlikely to change the conclusions of the CEA when the cost-effectiveness threshold was considered. Ten studies (19 pairwise comparisons) were compared with findings by the National Centre for Pharmacoeconomics (NCPE) using NMB. Without accounting for differences in the cost-effectiveness threshold, there was alignment between the study conclusions and NCPE recommendations in 73.7% cases. When the Irish cost-effectiveness threshold was applied in the estimation of NMB, there was agreement in 89.5% of cases.

Conclusions

Alignment in methodological requirements for CEA is important to facilitate joint health technology assessment (HTA) by regional collaborations in Europe. When parameter inputs are not exactly aligned, conclusions may still be comparable across jurisdictions. For international joint procurement initiatives, determining and implementing joint decision rules may be more important than trying to align rules regarding methodological and parameter inputs.

Electronic supplementary material

The online version of this article (10.1007/s40273-019-00860-y) contains supplementary material, which is available to authorized users.

Key Points for Decision Makers

Differing methodological requirements and decision-making criteria are recognised as barriers to transferability of cost-effectiveness analysis (CEA) across jurisdictions.

Even when parameter inputs and methodological assumptions are not exactly aligned, this case study demonstrates that conclusions of CEA may be comparable across jurisdictions.

For international joint procurement initiatives, determining and implementing joint decision rules may be more important than trying to align rules regarding CEA parameter input.

Introduction

There is increasing emphasis on collaboration between health technology assessment (HTA) agencies across Europe to manage increasing expenditure in the health sector. EUnetHTA, a network of more than 80 European HTA agencies across 30 countries, was established to create sustainable collaboration in HTA production and use in Europe [1]. In January 2018, the European Commission published a proposal for mandatory joint relative effectiveness assessments of new drug technologies in Europe, envisaging implementation within a 6-year timeframe from the passing of the regulations [2]. Separately, the Beneluxa initiative envisages more ambitious cooperation, proposing collaboration around joint HTA and pricing and reimbursement [3]. Ireland has recently joined the Beneluxa collaboration [4].

Cost-effectiveness analysis (CEA) is a component of HTA. In Ireland, the National Centre for Pharmacoeconomics (NCPE) performs CEA of new drugs on behalf of the national health payer, the Health Service Executive (HSE) [5]. These are conducted in line with the national reference case as outlined by the Health Information and Quality Authority (HIQA) in their Guidelines for Economic Evaluation of Health Technologies in Ireland [6]. There are two stated cost-effectiveness thresholds for pharmaceuticals in Ireland, €20,000 per quality-adjusted life-year (QALY) and €45,000 per QALY, as specified in the current agreement between the HSE and related government bodies, and the industry representative body [7]. There is no explicit use of higher thresholds for cancer or rare disease treatments, although it has been shown that factors other than the incremental cost-effectiveness ratio (ICER), such as quality of the clinical evidence, can influence NCPE recommendations for reimbursement decisions [8]. NCPE recommendations on cost effectiveness and relative clinical effectiveness are considered by the HSE when making their reimbursement decisions, alongside the additional criteria outlined in Schedule 3 Part 3 of the Health (Pricing and Supply of Medical Goods) Act 2013 [9].

The reimbursement process for pharmaceuticals in Ireland is different to that of the other Beneluxa countries, in terms of the specifics of the national reference case for CEA, but also in terms of cost-effectiveness thresholds for reimbursement [10–12]. Of interest is how these differences in the national reference case and cost-effectiveness thresholds may post barriers to joint CEA and decision making [13].

Cost-effectiveness estimates for pharmaceuticals vary between countries in western Europe [14]. Cost inputs for CEA vary greatly between jurisdictions, due to differences in both unit costs and resource use [15]. Additionally, it is recognised that outcome measurements for CEA, such as treatment effects and preferences for health, cannot be assumed to be constant across jurisdictions, but are shaped by factors such as baseline population risk, clinical management and cultural influences [16, 17]. The requirement for localised parameter inputs has long been recognised as a barrier to transferring CEA between regions or countries. Moreover, decision rules for drug reimbursement following CEA vary, with many countries having no explicit cost-effectiveness threshold for reimbursement decisions.

Transferability of CEA has been defined as “the ability to extrapolate results obtained from one setting or context to another” [18]. A distinction has been drawn between the ‘generic transferability’, a property of the study itself reflecting chosen methods and reporting, and ‘specific transferability’, a function of the decision context in a given jurisdiction [18]. Critical factors affecting the generic transferability of CEA include quality, transparency, accuracy of reporting and methodology [19]. Methodological requirements such as discount rates, time horizons and assessment of preferences, differ between jurisdictions and have been identified as key factors to consider when assessing generic transferability [18]. There is limited empirical information on how relevant these methodological choices are to reimbursement decisions in the context of cost-effectiveness thresholds; that is, the specific transferability.

We sought to assess the generic and specific transferability of published CEAs of systemic treatments for advanced melanoma skin cancer to the Irish setting. The aim was to determine if the outcomes and conclusions of these CEAs are transferable to the Irish setting. The disease area of melanoma was chosen as recent advances in treatment have increased survival for patients with advanced melanoma, but there is large geographical disparity in reimbursement decisions and patient access across Europe [20].

Methods

Data Identification and Extraction

CEAs were identified through systematic review [21]. De novo decision analytic models investigating the cost effectiveness of treatments for advanced melanoma were eligible for inclusion, regardless of line of treatment. Summary documents of models submitted to HTA agencies for reimbursement purposes were excluded as it was expected they would not contain sufficient information to assess transferability, and would potentially draw conclusions based on mark-down prices (drug prices to which a price reduction has been applied), which would not be considered transferable. Only pharmacological treatments were considered, including comparisons with best supportive care and no treatment. All studies with an outcome of cost per QALY, cost per life-year gained (LYG) or alternative measure of health outcome were included. The search was conducted in the Cochrane database, MEDLINE, Embase and EconLit in September 2018 [Appendix Table A1, see electronic supplementary material (ESM)] [21]. Data on model parameters, predicted total costs, incremental QALYs, incremental costs, ICERs, cost-effectiveness threshold and study conclusions were extracted by two authors independently (CG, LMcC). Total costs were inflated to 2017 values using the consumer price index for health and converted to US dollars using the purchasing power parity (PPP) method [22]. Comprehensiveness of reporting was assessed using the CHEERS checklist [23].

Assessment of Generic Transferability of Study Outcomes

Transferability to the Irish setting was examined using the Economic Evaluation domain of the EUnetHTA transferability tool [24]. This tool was developed as part of the EUnetHTA HTA Adaptation Toolkit, which focuses on three distinct elements for adaptation: relevance, reliability and transferability. The transferability domain consists of three questions. (i) How generalisable and relevant are the results and validity of the data and model to the relevant jurisdictions and populations? (ii) Are there any differences in the following parameters: perspective, preferences, relative costs, indirect costs, discount rate, technological context, personnel characteristics, epidemiological context, factors that influence incidence and prevalence, demographic context, life expectancy, reproduction, pre- and post-intervention care, integration of technology into the healthcare system and incentives? (iii) Does the evaluation violate the national guidelines for CEA? The tool is a qualitative instrument and no quantitative score for transferability is produced. We chose to focus on Sects. 2 and 3 of the tool to inform our consideration of transferability, as these sections posed defined questions that can be related back to the national reference case for CEA, and allowed the flexibility to record an overall impression of the transferability of the study also.

Two authors (CG, LMcC) completed the assessment independently, and differences were resolved by arbitration. Considerations on generic transferability were made with reference to the publicly funded healthcare system in Ireland, the current clinical management of advanced melanoma in Ireland [25] and the Irish national reference case for CEA, published in 2018 (Table 1) [6]. The extracted data is tabulated, and the authors’ overall conclusions regarding transferability of each study presented. Where differences were documented from the Irish reference case, the published deterministic sensitivity analyses were reviewed to determine the likely impact of these differences on the outcome of the CEA.

Table 1.

HIQA reference case for economic evaluation in Ireland [6]

Element of the technology assessment	Reference case
Evaluation type	Cost-utility analysis
Perspective on costs	Publicly funded health and social care system in Ireland
Perspective on outcomes	All health benefits accruing to individuals
Choice of comparator	Routine care in Ireland
Synthesis of effectiveness	Based on systematic review
Outcome measurement	Quality-adjusted life-years
Discount rate	Annual rate of 5% on costs and outcomes after the first yeara
Sensitivity analysis	Probabilistic and sensitivity analysis
Equity rating	Equal rating should be applied to the outcome measure

HIQA Health Information and Quality Authority

^aSince July 2019, the discount rate in Ireland was lowered to 4% on both costs and outcomes [52]

Studies were considered to have a relevant technological context if the interventions or comparators are part of current routine care for advanced melanoma in Ireland or were representative of routine care at the time of publication. Differences in pre- and post-intervention care were considered in the same manner. Where insufficient information to assess differences from the national reference case or Irish clinical practice was provided, the assessment was marked ‘unclear’. The template includes ‘Incentives’ as a relevant topic for consideration. This was interpreted to refer to financial incentive structures that may impact on clinical decisions. The national reference case specifies the perspective of the publicly funded health system in Ireland, and thus it was considered that studies in countries with predominately privately funded health insurance markets could have a difference incentive structure to the Irish setting. The template also specified ‘Relative costs’ for consideration, which was interpreted as likely differences in the unit costs of resource use and for the interventions. Studies that were clearly not generically transferable to the Irish setting were excluded from the assessment of specific transferability.

Assessment of Specific Transferability of Study Conclusions

For studies considered to demonstrate generic transferability to the Irish setting, the specific transferability to the Irish context was considered. The cost-effectiveness thresholds specified in the included CEAs were extracted, and converted to US dollars (2017) using the PPP method. In a comparable manner, the Irish thresholds of €20,000 and €45,000 per QALY were converted to US dollars. The extracted and inflated incremental costs and QALYs from the included CEAs for each comparison (pairwise ICER) were used to calculate the net monetary benefit (NMB) at (i) the cost-effectiveness threshold specified in the published CEA and (ii) the Irish cost-effectiveness thresholds of US$26,667 and US$60,000 per QALY. NMB was calculated according to the formula: NMB = (∆QALYs × λ) – Δ costs, where λ is the cost-effectiveness threshold. A positive value for the NMB implies that the product is cost effective at the chosen threshold. NMB was used in preference to ICERs to allow dominant strategies to be explored.

The pairwise comparisons where the NMB was positive or negative at the cost-effectiveness threshold specified in the CEA was determined, and the proportion considered to be cost effective (positive NMB) calculated. The NMB for each pairwise comparison was then recalculated using the Irish cost-effectiveness threshold (US$26,666 and US$60,000 per QALY). The proportion of pairwise comparisons where the NMB changed from positive to negative or vice versa, indicating a change in the decision when interpreted at the Irish cost-effectiveness threshold, was calculated.

The outcome from the NCPE HTA assessment for each pairwise comparison was extracted from the summary HTA reports published on the NCPE website [26–34]. Concordance between recommendations at the study CEA threshold, and when reanalysed at the Irish threshold, with the published NCPE recommendations was investigated.

Results

The systematic review identified 660 records, and 131 underwent full-text review [21]. Fifteen academic studies, published between 2000 and 2018, were eligible for inclusion (Appendix 1, Table A2, see ESM) [35–49]. Eight studies were from the US, five from European countries, one from Canada and one from Australia. No study was adjudicated to have perfect compliance with the reporting checklists. From the 15 included studies, 35 distinct pairwise comparisons were identified.

EUnetHTA Transferability Assessment

The outcomes of the transferability assessment are shown in Table 2. Most of the identified CEAs had the required perspective (payer, n = 10), and therefore the required approach to indirect costs [35, 37, 38, 41, 42, 44–47, 49]. Two CEAs adopted the societal perspective [36, 43], while three stated a societal perspective was taken but did not document the inclusion of indirect costs and were categorised as payer perspective [39, 40, 48]. Utility values were derived using either health state vignettes and the standard gamble method (n = 5), or from a generic utility instrument (EQ-5D, n = 7), while one study used a mixture of both [44]. Three studies did not apply or did not specify a discount rate [36, 39, 43]. One study applied a different rate to costs and QALYs [44]. Rates applied varied from 3 to 6%; only three studies applied the same discount rate as the Irish reference case of 5% [35, 42, 47]. The technological context for the management of advanced melanoma has moved swiftly in recent times. However, studies were considered transferable if they considered comparators that were relevant at the time of publication, and if the pre- and post-intervention care modelled in the CEA were considered a likely representation of care at that time in Ireland. Thus, no studies were excluded due to differences in the technological context or pre- and post-intervention care.

Table 2.

Conclusions of EUnetHTA transferability assessment

	Barzey et al. 2013 [37]	Bohensky et al. 2016 [35]	Curl et al. 2014 [40]	De Francesco et al. 2016 [38]	Delea et al. 2015 [42]	Hillner et al. 2000 [43]	Jensen et al. 2016 [36]	Oh et al. 2017 [48]	Pike et al. 2015 [49]	Shih et al. 2015 [39]	Miguel et al. 2017 [47]	Wang et al. 2017 [41]	Matter-Walstra et al. 2015 [44]	Kohn et al. 2017 [45]	Meng et al. 2018 [46]
Are there any differences in the following parameters?
Perspective  Publicly funded health system	Payer	Payer	Payera	Payer	Payer	Soc	Soc	Payera	Payer	Payera	Payer	Payer	Payer	Payer	Payer
Preferences EQ-5D-3Lb	SG	EQ5D	SG	EQ5D	EQ5D	NA	NA	SG	EQ5D	SG	EQ5D	EQ5D	EQ5D, SG	SG	EQ5D
Relative costs Irish costs where possible	Nat costs	Nat costs	Nat costs	Nat costsc	Nat costs	Nat costs	Nat costs	Nat costsc	Nat costs	Nat costs	Nat costs	Nat costs	Nat costs	Nat costs	Nat costsc
Indirect costs Excluded (Exc)	Exc	Exc	Exc	Exc	Exc	Incl	Incl	Exc	Exc	Exc	Exc	Exc	Exc	Exc	Exc
Discount rate 5% on costs and outcome	3%	5%	3%	3%	5%	NAd	NAd	3% costs only	4%	NAd	5%	3%	3% costs, 6% outcomes	3%	3.5%
Technological context Routine care in Ireland	No	No	No	No	No	No	No	No	No	No	No	No	No	No	No
Personnel characteristics Not specified in reference case	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely
Epidemiological context (including genetic variants) Relevant target population	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely
Factors which influence incidence and prevalence Relevant target population	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely
Demographic context Relevant target population	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely
Life expectancy Relevant target population	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely
Reproduction Not applicable	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA
Pre- and post-intervention care Routine care in Ireland	No	No	No	No	No	No	No	No	No	No	No	No	No	No	No
Integration of technology in health care system Not specified	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely	Unlikely
Incentives Publicly funded health and social care	Yes	Uncl.	Yes	No	No	Yes	Yes	Yes	No	Yes	No	Yes	Uncl.	Yes	No
Overall impressions
If differences exist, how likely is it that each factor would impact the results? In which direction? Of what magnitude?	Uncl.	Uncl.	Uncl.	Highly likely	Uncl.	Highly likely	Highly likely	Highly likely	Uncl.	Uncl.	Uncl.	Uncl.	Uncl.	Uncl.	Highly likely
Taken together, how would they impact the results and of what magnitude?	Uncl.	Uncl.	Uncl.	Signif.	Uncl.	Signif.	Signif.	Signif.	Uncl.	Uncl.	Uncl.	Uncl.	Uncl.	Uncl.	Signif.
Given these potential differences, how would the conclusions likely change in the target setting? Are you able to quantify this in any manner?	Uncl.	Uncl.	Uncl.	Likely	Un-likely	Likely	Likely	Likely	Uncl.	Un-likely	Uncl.	Uncl.	Un-likely	Uncl.	Likely
Does the economic evaluation violate your national/regional guidelines for health economic evaluation?	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes

CEA cost-effectiveness analysis, Exc excluded, Incl included, NA not applicable, NCPE National Centre for Pharmacoeconomics, Nat costs national costs used, SG standard gamble, Signif. significantly, Soc societal, Uncl. unclear

^aCEA where the societal perspective was specified, but no indirect costs included

^bNCPE guidance

^cTreatment acquisition costs were based on mark-down prices rather than the list price

^dNo discounting applied

Treatment acquisition costs were generally based on the publicly available price, with three exceptions where mark-down prices were used, which limits transferability to the Irish context [38, 46, 48]. Three studies did not provide a cost per QALY outcome [36, 43, 48], and were not considered transferable to the Irish context, as there is no specified cost-effectiveness threshold for alternative outcomes.

The EUnetHTA tool includes several health system and national factors when considering transferability: personnel characteristics, the integration of technology into the healthcare system, factors affecting disease incidence and prevalence and the epidemiological and demographic context. The reference case for HTA in Ireland does not provide specific directions relating to these topics, except to highlight that the inputs should be reflective of the relevant target population. The epidemiological context and factors that influence incidence and prevalence were considered likely to be different in the identified cost-effectiveness models, as Ireland has a relatively high incidence of advanced melanoma and a lower incidence of melanoma with the BRAF mutation compared with international norms [50]. Ireland also has a younger population than many other OECD countries, and a relatively high life expectancy [51]. However, in the absence of explicit modelling of baseline risk or national population demographics, these differences were considered unlikely to impact on the outcomes of the included cost-effectiveness models. Insufficient information was provided in the reviewed studies to determine if personnel characteristics or integration of technology into the health service were similar to Ireland, but it was considered unlikely that there would be significant differences. These factors were all marked as unlikely to affect transferability in this assessment.

Incentives were considered likely to be the same in five CEAs with publicly funded health systems, as they provide health care in a similar model to Ireland [38, 42, 46, 47, 49]. Studies conducted within the US health system were considered to have different incentive structures, due to differences in the model of health care provision [36, 37, 39–41, 43, 45, 48]. The incentive structure was considered unclear in two studies, set in Australia and Switzerland, due to the interplay of private and public provision and compulsory private health insurance, respectively [35, 44]. However, the incentive structure was considered unlikely to have a significant impact overall on the costs and QALY gain associated with each treatment, and so did not preclude any studies from the specific transferability assessment.

Where deterministic sensitivity analyses were published, the outcomes were reviewed in light of the specified parameters of the Irish reference case. Discount rates, when included, were not found to be important drivers of the ICER in any of the CEAs [35, 38, 47]. Time horizon was an important driver of the ICER in three studies [35, 37, 42]. In two of these studies, the time horizons used were significantly shorter than the recommended lifetime horizon of the Irish reference case, which could potentially limit the transferability of the outcomes to the Irish context [35, 42]. In both cases, the decision was unlikely to change with an increase in the time horizon. Health state costs were important in four studies in determining the ICER [35, 37, 38, 41], but would only have changed the decision in one study when varied to the upper limit in the deterministic sensitivity analysis [41]. Utility was an important determinant of the ICER in seven of the included studies, but only in two of those would it be likely to change the decision based on the tornado plots provided in the published manuscript [45, 48].

None of the identified CEAs were entirely generically transferable to the Irish setting, as no model was fully aligned with the Irish reference case. Alignment with the reference case was most commonly seen for perspective, treatment of indirect costs and the technological context. Disagreement was most commonly seen for the discount rate (n = 12). It was considered highly likely that the results would change or be irrelevant if transferred to the Irish setting for five of the CEAs, two of which did not provide a cost per QALY outcome, and three of which used mark-down prices to estimate the ICER. The remaining studies (n = 10) were included in the consideration of specific transferability.

Transferability of Conclusions in Accordance with Cost-Effectiveness Threshold

For the assessment of specific transferability, incremental costs, QALYs and cost-effectiveness thresholds were extracted from the included CEAs (n = 10) (Table 3). Twenty-five pairwise comparisons were considered from these studies. The NMB was calculated at the cost-effectiveness thresholds specified in the published studies, which varied from US$35,000 to US$200,000 per QALY (Table 3). Of the 25 comparisons, nine had a positive NMB at the cost-effectiveness threshold specified in the published study (36%) (Table 3), indicating that they were cost effective at the study threshold, and 16 (64%) had a negative NMB, indicating that they were not considered to be cost effective.

Table 3.

Net monetary benefit at study cost-effectiveness threshold and Irish cost-effectiveness threshold

Study	Intervention	Comparator	Incremental cost (2017 US$)	Incremental QALYs	Study threshold (2017 US$)	NMB at study threshold (2017 $US)	NMB at Irish threshold (US$60,000/QALY)	NMB at Irish threshold (US$26,667/QALY)
Ipilimumab
Kohn et al. 2017, US [45]	Ipilimumab 1L	Dacarbazine	5769	0.08	100,000	2231	− 969	− 3636
Barzey et al. 2013, US [37]	Ipilimumab 2L+	BSC	179,534	1.14	200,000	48,466	− 111,134	− 149,134
Pike et al. 2015, Norway [49]	Ipilimumab 1L	Dacarbazine	82,475	0.48	49,169	− 58,874	− 53,675	− 69,675
Pembrolizumab
Kohn et al. 2017, US [45]	Pembrolizumab 1L	Dacarbazine	− 19,630	0.12	100,000	31,630	26,830	22,830
Kohn et al. 2017, US [45]	Pembrolizumab every 2 weeksa 1L	Dacarbazine	169,012	0.17	100,000	− 152,012	− 158,812	− 164,479
Pike et al. 2015, Norway [49]	Pembrolizumab 1L	Dacarbazine	95,977	0.80	49,169	− 56,642	− 47,977	− 74,644
Miguel et al. 2017, Portugal [47]	Pembrolizumab 1L	Ipilimumab	61,963	0.98	66,666.67	3370	− 3163	− 35,830
Wang et al. 2017, US [41]	Pembrolizumab 1L	Ipilimumab	67,751	0.79	100,000	11,249	− 20,351	− 46,684
Nivolumab
Kohn et al. 2017, US [45]	Nivolumab 1L	Dacarbazine	26,083	0.28	100,000	1917	− 9283	− 18,616
Pike et al. 2015, Norway [49]	Nivolumab 1L	Dacarbazine	93,625	0.82	49,169	− 53,306	− 44,425	− 71,758
Bohensky et al. 2016, Australia [35]	Nivolumab 1L	Ipilimumab	42,159	1.30	35,000	3341	35,841	− 7492
Nivolumab in combination with ipilimumab
Kohn et al. 2017, US [45]	Nivolumab with ipilimumab 1L	Dacarbazine	61,159	0.30	100,000	− 31,159	− 43,159	− 53,159
Pike et al. 2015, Norway [49]	Nivolumab with ipilimumab 1L	Dacarbazine	139,825	0.81	49,169	− 99,998	− 91,225	− 118,225
Dabrafenib in combination with trametinib
Matter-Walstra et al. 2015, Switzerland [44]	Dabrafenib with trametinib 1L	Vemurafenib	126,829	0.46	68,000	− 95,549	− 99,229	− 114,562
Pike et al. 2015, Norway [49]	Dabrafenib with trametinib 1L	Dacarbazine	241,178	0.83	49,169	− 200,368	− 191,378	− 219,045
Vemurafenib in combination with cobimetinib
Pike et al. 2015, Norway [49]	Vemurafenib with cobimetinib 1L	Dacarbazine	240,069	0.89	49,169	− 196,309	− 186,669	− 216,336
Vemurafenib
Curl et al. 2014, US [40]	Vemurafenib 1L	Dacarbazine	165,963	0.42	100,000	− 123,963	− 140,763	− 154,763
Curl et al. 2014, US [40]	Vemurafenib followed by ipilimumab	Vemurafenib	109,417	0.20	100,000	− 89,417	− 97,417	− 104,084
Pike et al. 2015, Norway [49]	Vemurafenib 1L	Dacarbazine	81,180	0.31	49,169	− 65,938	− 62,580	− 72,913
Shih et al. 2015, US [39]	Vemurafenib 1L	Dacarbazine	38,815	0.11	100,000	− 27,965	− 32,305	− 35,922
Dabrafenib
Delea et al. 2015, Canada [42]	Dabrafenib 1L	Dacarbazine	63,915	0.21	159,872	− 31,061	− 51,585	− 58,435
Delea et al. 2015, Canada [42]	Dabrafenib 1L	Vemurafenib	− 33,580	0.05	159,872	41,350	36,496	34,876
Shih et al. 2015, US [39]	Dabrafenib 1L	Dacarbazine	26,080	0.16	100,000	− 10,430	− 16,690	− 21,907
Shih et al. 2015, US [39]	Dabrafenib 1L	Vemurafenib	− 12,736	0.05	100,000	17,536	15,616	14,016
Pike et al. 2015, Norway [49]	Dabrafenib 1L	Dacarbazine	81,120	0.35	49,169	− 63,911	− 60,120	− 71,787

1L first line, 2L+ second or later lines of treatment, BSC best supportive care, NMB net monetary benefit, QALY quality-adjusted life-year

^aNot a licensed regimen

Secondly, the NMB for these pairwise comparisons was recalculated using the Irish cost-effectiveness thresholds. When the higher Irish threshold (US$60,000 per QALY) was applied to the calculation of NMB, only four comparisons estimated a positive NMB (16%), a reduction of 20% compared with using the CEA-specific threshold (Table 3). Using the lower Irish threshold (US$26,667 per QALY), only three comparisons (12%) retained a positive NMB.

Data on incremental costs and QALYs for the pairwise comparisons were extracted from the published NCPE summary HTA reports and converted to US dollars (Table 4). The NCPE had not evaluated six of the pairwise comparisons for the Irish drug reimbursement process (mainly comparisons of the newest agents versus chemotherapy, and unlicensed regimens). Of the 19 pairwise comparisons that could be compared with the identified CEAs, in 14 (73.7%) instances the NCPE conclusion based on the ICER was in agreement with the study conclusions, and five were in disagreement (26.3%).

Table 4.

Extracted outcomes from NCPE summary HTA reports

Drug	Comparator	Extracted data			Yeara	Converted to US$
		ICER (€ per QALY)	Incremental costs (€)	Incremental QALYs		ICER (2017 US$)	Incremental costs (2017 US$)
Vemurafenib [31]	Dacarbazine	131,883	NR	NR	2012	177,233	NR
Dabrafenib [30]	Dacarbazine	84,473	113,613	1.35	2014	112,741	151,632
Dabrafenib [30]	Vemurafenib	Dominant	− 43,380	0.36	2014	Dominant	− 57,897
Ipilimumab 2L [27]	BSC	147,899	NR	NR	2011	202,168	NR
Pembrolizumab 1L [28]	Ipilimumab	Dominant	− 3093	0.42	2016	Dominant	− 4140
Pembrolizumab 2L [29]	BSC	85,766	72,280	0.84	2016	114,801	96,749
Nivolumab 1L [34]	Ipilimumab	101,282	NR	NR	2016	135,569	NR
Nivolumab 1L [34]	Vemurafenib	29,018	NR	NR	2016	38,842	NR
Nivolumab 1L [34]	Dabrafenib	46,276	NR	NR	2016	61,942	NR
Nivo + Ipi [32]	Ipilimumab	47,748	101,354	2.13	2016	63,912	135,666
Nivo + Ipi [32]	Nivolumab	Dominant	− 7,792	0.93	2016	Dominant	− 10,430
Nivo + Ipi [32]	Pembrolizumab	Dominant	− 143,751	0.94	2016	Dominant	− 192,416
Nivo + Ipi [32]	Dab + Tram	14,850	21,454	1.45	2016	19,877	28,717
Dab + Tram [33]	Vemurafenib	177,275	170,314	0.96	2017	236,367	227,085
Dab + Tram [33]	Dabrafenib	244,822	182,417	0.75	2017	326,429	243,223
Dab + Tram [33]	Pembrolizumab	126,128	56,299	0.45	2017	168,171	75,065
Vem + Cobi [26]	Vemurafenib	326,868	168,266	0.51	2017	435,824	224,355
Vem + Cobi [26]	Dabrafenib	324,192	189,936	0.59	2017	432,256	253,248
Vem + Cobi [26]	Dab + Tram	108,284	15,806	0.15	2017	144,379	21,075

1L first line, 2L second line, BSC best supportive care, Dab + Tram dabrafenib in combination with trametinib, HTA health technology assessment, ICER incremental cost-effectiveness ratio, NCPE National Centre for Pharmacoeconomics, Nivo + Ipi nivolumab in combination with ipilimumab, NR not reported, QALY quality-adjusted life-year, Vem + Cobi vemurafenib in combination with cobimetinib

^aYear of publication on NCPE website

When the study NMB for each pairwise comparison was re-estimated at the higher Irish threshold, 17 of 19 (89.5%) pairwise comparisons were in agreement with the NCPE conclusions based on the ICER. In the five comparisons where the conclusions changed from positive at the CEA-specific threshold to negative at the Irish threshold, three changed to be in agreement with the NCPE conclusions, whilst two that had previously agreed with the NCPE were now different [41, 47].

When the study NMB for each pairwise comparison was re-estimated using the lower Irish threshold, only three comparisons (12%) retained a positive NMB. The single additional study that changed conclusion at this threshold compared with the higher Irish threshold shifted to be in agreement with the NCPE conclusion, resulting in a total of 18 out of 19 comparisons (95%) being in agreement with the NCPE conclusions.

Discussion

Decision analytic models evaluating the cost effectiveness of treatments for advanced melanoma have been reviewed for their transferability to the Irish setting. The models varied in the structural assumptions underpinning the model outputs and conclusions, in terms of the perspective, model type, time horizon adopted, preferences and discount rate applied.

Assessment of transferability was difficult due to poor quality reporting [21]. Unsurprisingly, none of the identified CEAs demonstrated complete generic transferability to the Irish setting as none were aligned with the national reference case for CEA. Alignment with the reference case was most commonly seen for perspective, treatment of indirect costs and the technological context. Disagreement was most commonly seen for the discount rate (n = 12). The potential impact on study conclusion of differences from the Irish reference case was not always easy to predict. The parameters considered most likely to cause variation include the discount rate, time horizon and relative costs. However, close examination of the deterministic sensitivity analyses in the published CEAs suggested that variations in most of these parameters would rarely have changed the decision at the cost-effectiveness threshold employed in the CEA. Of the 15 identified CEAs, five were excluded from the assessment of specific transferability as they were deemed inapplicable to the Irish context; two did not contain a cost per QALY outcome and three used mark-down prices.

The EUnetHTA transferability tool does not explicitly consider the cost-effectiveness threshold. We considered the transferability of the study outcomes to the Irish context by estimating the NMB at the study threshold and at the Irish cost-effectiveness thresholds. Using the Irish cost-effectiveness thresholds of US$26,667 and US$60,000 per QALY to estimate the NMB of each intervention, there was a reduction of 24% and 20%, respectively, in the proportion of positive conclusions compared with when the study-specific threshold was used.

The study conclusions were also compared with the conclusions of the NCPE for the same comparisons. Without any adjustment for differences in the cost-effectiveness threshold, there was good agreement between the published CEAs and the conclusions of the NCPE summary HTA reports (74% aligned). Adjusting for the lower Irish threshold led to alignment in conclusions in 95% of cases, and using the higher Irish threshold, in 89.5% cases. Thus, despite differences in model inputs and methodological assumptions, the conclusions of the studies were consistent with the NCPE conclusions, and even more so when updated to consider the Irish cost-effectiveness threshold.

Guidelines for economic evaluation undergo periodic review. In Ireland, the HIQA Guidelines for Economic Evaluation were published in 2014, updated in 2018 [6] and again in 2019 [52]. The NCPE guidelines are updated frequently, in response to methodological advances. If joint HTA is a realistic expectation of the Beneluxa collaboration, then future guideline reviews should consider the guidelines from the Beneluxa countries, to determine if there are areas where closer alignment could be achieved.

Commentary regarding the potential for success of the Beneluxa collaboration has highlighted the challenges posed by the differing HTA assessment processes [13]. We have shown in this study that while parameter inputs may not be exactly aligned with the requirements for the national reference case, the conclusions may be comparable across jurisdictions. Accounting for differing thresholds across jurisdictions produces even greater convergence between study conclusions; up to 95% concordance. Thus, while these studies are not generically transferable to the Irish setting, the specific transferability was high, and increased when the Irish threshold was considered in the analysis. This suggests that for joint HTA and procurement as envisaged under the Beneluxa process, determining and implementing joint decision rules may be more important than trying to align rules regarding parameter inputs.

The consistency of outcomes demonstrated here using advanced melanoma as a case study may not be generalisable to other interventions. When price negotiations begin to lower the ICERs closer to the specified cost-effectiveness threshold, the differences due to parameter inputs and generic methodological assumptions will become more relevant, and are more likely to impact on the reimbursement decision. Additionally, for potentially ‘curative’ gene therapies or vaccine programmes, differences in approaches to discounting are likely to be highly relevant. However, for many high-cost cancer drugs and drugs for rare diseases, the ICERs can be many multiples of the national threshold, and changes in generic parameter inputs or the cost-effectiveness threshold are unlikely to change the decision outcome. In many instances, reimbursement is achieved, indicating that the willingness-to-pay of the payer may be higher that the stated cost-effectiveness threshold. Of note, the objectives of the Beneluxa collaboration are to ensure “timely access and affordability of medicines” [3], without reference to the strict efficiency maximising objective of the standard CEA framework. Thus, there may be a pragmatic approach taken that could allow collaborating partners to overcome these difficulties as decisions approach the chosen cost-effectiveness threshold.

There are limitations to our study. As models used for reimbursement decision making are rarely publicly available, a case study of published CEAs was used. Without access to the cost-effectiveness models, it was not possible to quantitatively assess the impact of generic model inputs such as the discount rate on the ICER or the NMB. The outcomes of the transferability assessment are limited in applicability to other jurisdictions with different reference case requirements to Ireland. Assessment of transferability is subjective, although it was easy to reach agreement between the two assessors. Assessment is constrained by the thoroughness of reporting of the CEA. The assessment of transferability does not explicitly review the quality of the included studies, which is an important metric for consideration.

Conclusions

Assessing compliance with the national reference case is essential when considering transferability of CEAs to a local setting. Additionally, consideration must be given to the local decision-making context including cost-effectiveness thresholds. Regional alignments of smaller countries such as the Beneluxa collaboration will encounter issues regarding generic transferability. However, the core issue of specific transferability with regards to the decision thresholds may be a more relevant consideration.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Data Availability Statement

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Compliance with Ethical Standards

No informed consent or ethics approval was required for this study. Claire Gorry, Laura McCullagh and Michael Barry have no conflicts of interest. Claire Gorry is enrolled on a funded PhD programme through the National Centre for Pharmacoeconomics, Ireland.