Largely ignored: the impact of the threshold value for a QALY on the importance of a transferability factor

Abstract

Recently, several checklists systematically assessed factors that affect the transferability of cost-effectiveness (CE) studies between jurisdictions. The role of the threshold value for a QALY has been given little consideration in these checklists, even though the importance of a factor as a cause of between country differences in CE depends on this threshold. In this paper, we study the impact of the willingness-to-pay (WTP) per QALY on the importance of transferability factors in the case of smoking cessation support (SCS). We investigated, for several values of the WTP, how differences between six countries affect the incremental net monetary benefit (INMB) of SCS. The investigated factors were demography, smoking prevalence, mortality, epidemiology and costs of smoking-related diseases, resource use and unit costs of SCS, utility weights and discount rates. We found that when the WTP decreased, factors that mainly affect health outcomes became less important and factors that mainly effect costs became more important. With a WTP below €1,000, the factors most responsible for between country differences in INMB were resource use and unit costs of SCS and the costs of smoking-related diseases. Utility values had little impact. At a threshold above €10,000, between country differences were primarily due to different discount rates, utility weights and epidemiology of smoking-related diseases. Costs of smoking-related diseases had little impact. At all thresholds, demography had little impact. We concluded that, when judging the transferability of a CE study, we should consider the between country differences in WTP threshold values.

Introduction

Transferability has been defined as the degree to which the results of a cost-effectiveness study performed in one jurisdiction are representative for another jurisdiction, frequently another country. The term is also used to express the amount of adaptation that is necessary to make the results applicable to another jurisdiction [1]. A study that requires just a few simple adaptations, like a change in unit costs or discount rate, is more easily transferable than a study that requires many complicated adaptations, like a change in resource use pattern or a change in the case mix of the study population.

The question arises to what extent we can assess transferability. Recently, several checklists have been developed which systematically check the transferability of CE studies between jurisdictions [2–4]. Such checklists contain a list of issues which must be satisfied before a study is considered transferable. Examples of such issues are whether the perspective and comparators are relevant for the country of interest, whether the disease epidemiology is comparable in the country of interest and whether discount rates were specified. Some of the checklists generate a summary transferability score, assigning either equal importance to each issue [2] or weighting the importance of each issue [3].

The advantage of such checklists is that they force the user to systematically think about transferability. The disadvantage is that the transferability scores are difficult to interpret. What does a particular transferability score really say? Moreover, in all of these checklists, there is one aspect that has been given little consideration, namely the threshold value for a QALY. Although some checklists encourage the user to discuss the generalizability of the results in the light of country-specific decision criteria, the checklists ignore the fact that the importance of a transferability issue depends on the threshold value for a QALY. Issues that are important contributors to between country differences in CE at a low threshold need not be of equal importance at a higher threshold. For example, a difference in the incidence of an infectious disease between two countries, which causes the incremental cost-effectiveness ratio (ICER) of a vaccination program to double from €12,000 in country A to €24,000 in country B, is less important when the threshold value is €50,000 per QALY but highly important when the threshold value is €20,000.

As we will discuss later in this paper, no country has an explicit threshold value for a QALY. However, this does not mean that there is no threshold. Since no country has infinite resources available for health care, decisions have to be made which interventions to reimburse and which not. At the same time, although heavily criticized, the QALY is widely regarded as a relevant outcome for decision-makers. Assuming both the existence of a country-specific threshold and the acceptance of QALYs as an outcome measures that is relevant to the decision-makers in our countries of interest, country-specific INMBs can be calculated for all relevant countries.

The aim of this paper is to assess whether the extent to which between country differences influence the country-specific INMB depends on the threshold value for a QALY. We ranked several possible causes (in this paper “transferability factors” or “factors”) of between country differences in CE of smoking cessation support, according to their impact on the INMB. By doing this for different levels of the threshold value for a QALY, we studied how the importance of a transferability factor depends on this threshold value. In an earlier study, we found that at a threshold value of €20,000, discount rates and the epidemiology of smoking-related diseases were important drivers of between country differences in INMB of smoking cessation interventions [5].

Methods

The BENESCO model

The economic evaluation of smoking cessation support was based on the Benefits of Smoking Cessation on Outcomes (BENESCO) model, using a lifetime horizon and a healthcare sector’s perspective [6]. The BENESCO model simulates the benefits of quitting smoking in terms of smoking-related morbidity, mortality and associated medical costs. Diseases included in the model are chronic obstructive pulmonary disorder (COPD), lung cancer, chronic heart disease (CHD) and stroke. The model is structured as a Markov model and follows a hypothetical cohort of current smokers making a single attempt to quit smoking at the beginning of the simulation. More information on the model can be found in the online appendix and several publications [7–9].

Smoking cessation support

If a person decides to quit smoking, (s)he can do this unaided, or using one of the available forms of smoking cessation strategies. In this paper, we compare three forms of pharmacological smoking cessation support: nicotine replacement therapy (NRT), bupropion and varenicline. NRT is the generic term for any form of smoking cessation aid which delivers a measured dose of nicotine to the person using it. Examples include the nicotine patch or nicotine gum. Bupropion is an antidepressant used to support smoking cessation [10]. Varenicline is designed to relieve symptoms of nicotine withdrawal including cigarette craving and block the reinforcing effects of continued nicotine use [11]. The reference case to which the results of other countries in this analysis are compared is an economic evaluation performed from the Dutch health care perspective. In this study, NRT has been shown to be cost-effective compared to unaided cessation, bupropion was dominant over NRT and varenicline was dominant over bupropion [7].

Cost-benefit

The primary outcome of the economic evaluation was the INMB. The INMB was chosen because it explicitly incorporates the threshold value for the willingness-to-pay (WTP), or the societal value for a QALY (λ). Hence, we could study the extent to which the importance of the transferability factors depends on the λ (see paragraph below). The INMB was calculated for different values of λ ranging from 0 to €50,000. Comparing two smoking cessation interventions, A and B, the INMB was calculated as:

A positive INMB indicates that the net benefits of intervention A are higher than the net benefits of intervention B. A negative INMB indicates that the net benefits of intervention A are lower than the net benefits of intervention B.

Factors affecting transferability

Within the BENESCO model, all variables except the risk ratio of getting a smoking-related illness were changed to calculate country-specific cost-effectiveness. These variables can intuitively be grouped in a total of nine transferability factors, which could potentially cause differences in the cost-benefit of smoking cessation support between countries. We investigated each of these factors. Each transferability factor consisted of a group of country-specific input parameters which were varied simultaneously. Demography included the total number of people older than 18 years of age in six age/gender classes. Smoking prevalence refers to the percentage of smokers, non-smokers and former smokers in each age/gender class. All-cause mortality was the percentage of the total number of people in each age/gender class that dies during a single year. The epidemiology of smoking-related diseases consisted of three elements: the incidence rates, prevalence rates and annual cause-specific mortality rates by age/gender class. The costs of smoking-related diseases were separated into first year costs and costs in subsequent years, except for COPD. The amount of resource use (i.e., medication and counselling) associated with the SCS and the unit costs of these resources were the two elements defining the intervention costs. The utility weights were defined both for the general population and for patients with a smoking-related disease. The discount rates for both costs and outcomes were set equal to the values recommended in 2007 in the national guidelines for economic evaluations. For each country, we adopted a health care perspective. Information on the main input factors can be found in the “Appendix”.

The importance of each transferability factor

Based on the hierarchy of effectiveness of the smoking cessation interventions, we calculated the INMB of NRT versus unaided cessation, bupropion versus NRT and varenicline versus bupropion in the Netherlands [7]. These results were used as the reference values. We then changed the Dutch input values of each of the nine factors individually to the country-specific values for Germany, Sweden, the UK, Belgium and France [5, 8, 12, 13]. This changes the INMB. The percentage of change in the INMB was then averaged over the three treatment comparisons. Each factor is then rank ordered from highest (1) to lowest (9) percentage of change in INMB. We compare these rankings for different threshold values, ranging from €0 to €50,000 per QALY, for each country separately.

Results

The INMB of NRT versus unaided cessation in the Netherlands at a WTP threshold of €20,000 per QALY was €1.42 million. The INMB of bupropion versus NRT was €0.39 million, and the INMB of varenicline versus bupropion was €0.90 million. Table 1 shows the ranking of factors at different threshold values for a QALY, after replacing the Dutch reference values with the German country-specific values. It is clear from this table that the threshold value has a considerable impact on the importance of each factor.

Table 1.

Univariate ranking of factors after changing to the German values, at different threshold values for the willingness-to-pay (WTP) for a QALY

Threshold value for WTP	Demography	All-cause mortality	Smoking prevalence	Epidemiology	Costs of smoking-related diseases	Resource use	Unit costs	Utility weights	Discount rates
€0	7	8	6	5	3	1	2	9	4
€100	7	8	6	5	4	1	2	9	3
€500	8	9	6	5	4	1	3	7	2
€1,000	8	9	7	5	3	1	4	6	2
€5,000	8	9	7	4	6	2	5	3	1
€10,000	8	9	6	3	7	4	5	2	1
€20,000	8	9	4	3	7	5	6	2	1
€50,000	7	9	4	3	8	5	6	2	1

Ranked according to the percentage change in INMB compared to the reference country, averaged over three pair-wise comparisons of smoking cessation strategies

This can be seen most clearly for the cost of smoking-related diseases, resource use, unit costs and utility weights. The INMB is calculated as the difference in QALYs multiplied by λ minus the difference in costs. Hence, the QALY gains are weighted with the λ before subtracting the additional costs. When λ decreases, the transferability factors that mainly affect the health outcomes are given less weight and become less important, whereas the transferability factors which mainly affect the costs become more important. Among the latter are the costs of smoking cessation support, which are driven by the resources used and the unit costs of these resources, and the costs of smoking-related diseases. When λ increases, the transferability factors with a large impact on the QALYs are given a higher weigh, increasing their importance as a cause of between country differences in net benefit. As a consequence, utility weights, epidemiology (mainly incidence and mortality) of smoking-related diseases and smoking prevalence become more important.

When replacing Dutch by German input values, the discount rate, which changed from 4% for costs and 1.5% for effects in the Netherlands to 5% for both costs and effects in Germany, led to the highest percentage of change in INMB at thresholds of €5,000 or higher. At lower thresholds, the relative importance of the discount rate decreases, with the resource use taking over as the most important factor. All-cause mortality and demography are relatively unimportant at all threshold values.

Similar tables result for the other countries. They are shown in the “Appendix”. For the other four countries, the same general overview can be seen, although there are individual differences. Tables on the outcomes of other countries can be found in the “Appendix”. In all countries, unit costs and/or resource use are amongst the two most important causes of between country differences in cost-benefit at low thresholds. At high thresholds, utility weights, discount rates and the epidemiology of smoking-related diseases are the three most important factors for all countries, except for Belgium which had almost the same discount rates as in the Netherlands.

Figure 1 shows the percentage of change in the INMB of NRT versus unaided cessation, when substituting the reference case input values to the German values, at various levels of λ. The results were comparable for the other pair-wise comparisons of smoking cessation interventions (not shown, available on request). At a λ of €0, substituting Dutch resource use by German resource use led to the greatest change in INMB, a decrease of 92%. The second most important transferability factor was the costs of smoking-related diseases, decreasing the INMB with 29%. At a λ of €500, Germany-specific resource use still caused the INMB to change most, but now followed by the discount rates. At a λ of €5,000 or above, the discount rates had the highest impact on the INMB, causing a decrease in INMB of 99%, followed by the utility weights, causing a decrease in INMB of 25%.

Fig. 1.

Percentage of change in the INMB of NRT versus unaided cessation, when replacing Dutch input values for each factor by the German input values, at different threshold values for the willingness-to-pay (WTP) for a QALY. F1: demography, F2: smoking prevalence, F3: mortality, F4: epidemiology, F5: costs of smoking-related diseases, F6: resource use, F7: unit costs, F8: utility weights, F9: discount rates

The closer the threshold value of a QALY was to the original ICER in the reference country—in this case €1,637 per QALY—the more the INMB was affected by a change to country-specific input data. This was true for every transferability factor, since the INMB of the base case comparison is close to zero with a λ close to the ICER. For most transferability factors, the sign of the impact on the INMB at a threshold of €1,000 or lower switched from positive to negative or vise versa at a threshold of €5,000 or higher. Only the factor discount rates for Sweden and the UK and the factor epidemiology for Belgium and Sweden show a slightly different pattern. The reason is that the INMB of the base case comparison is always positive with a λ above the value of the ICER, but negative with a λ below the ICER. A similar change in INMB due to a new country-specific variable value will therefore have a directly opposite effect.

Figure 2 shows, for various levels of λ, the change in INMB of NRT versus unaided cessation, when all reference case input values are simultaneously replaced by country-specific values. This change to the country-specific INMB is shown for each country separately. At lower levels of λ, the INMB changes most when adapting the Dutch input data to the UK input data. This is because the resource use and costs of NRT in the UK differ most from that in the Netherlands, i.e., NRT is considerably less expensive in the UK than in the Netherlands. At higher levels of λ, the INMB changes most when adapting the Dutch input data to the German input data. This is because of the great difference in discount rates.

Fig. 2.

Percentage of change in the INMB of NRT versus unaided cessation, when simultaneously replacing Dutch input values for all transferability factors by the country-specific input values at different threshold values for the willingness-to-pay (WTP) for a QALY. Demography, smoking prevalence, mortality, epidemiology, costs of smoking-related diseases, resource use, unit costs, utility weights, discount rates

Discussion

This paper has clearly shown that, when we transfer a foreign economic evaluation to the country of interest, the factor which should most carefully be adapted is driven by the threshold value for a QALY. We feel that this aspect has been given too little attention in the transferability debate until now. When discussing the transferability of smoking cessation evaluations in countries with low threshold values, we should pay most attention to the country-adaptation of the cost drivers, i.e., the costs of the interventions and the disease that are studied. In countries with high threshold values, we should pay most attention to the country-adaptation of the factors that affect the health outcomes. These include disease epidemiology and utility values. Although the results are specific for smoking cessation interventions, and indeed for the BENESCO model, we feel this conclusion is applicable to similar interventions of a preventative nature that require initial investments which by far precede the returns on these investments in terms of improved health outcomes and savings in the costs of health care utilization. In addition, we feel that based on this paper, the threshold value for a QALY should be an integral part of the investigation of transferability for all economic evaluations.

The approach taken in this paper assumes the acceptance of the QALY as a relevant decision-making outcome and the existence of threshold values, either explicit or implicit. The reason that the role of the threshold value has been largely ignored in the transferability discussion may be related to the fact that we are far from reaching a consensus on the maximum willingness to pay for a QALY. There is not a single jurisdiction where the threshold value is really fixed. With respect to the countries in our current study, NICE (National Institute for Health and Clinical Excellence) in the UK mentions the most explicit threshold value, but they too reject the use of a single, absolute threshold, instead preferring to make decisions on a case-by-case basis. NICE is unlikely to reject a technology with a ratio in the range of £5,000 – £15,000 per QALY solely on the grounds of cost ineffectiveness but would need special reasons for accepting technologies with ratios over £25,000 – £35,000 per QALY as cost effective [14]. In the Netherlands, a threshold of €20,000 per QALY is often cited. However, this threshold was obtained from economic evaluations of preventive interventions and is certainly not used consistently [15]. Currently, there is discussion in the Netherlands on increasing the threshold value depending on the burden of the disease of interest, with a maximum threshold of up to €80,000 for very severe diseases [16, 17]. In Germany, the IQWiG (Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen) has prepared guidelines for economic evaluation as part of the evaluation of the value of pharmaceuticals [18], but coverage does not depend on any externally set maximum standard. In 2008, the Belgian KCE (Federal Knowledge Centre for Health Care) started to produce methodological reports in order to help standardize the methodology used for health technology assessment [19]. One such report mentions that CE is now only rarely used as an argument for reimbursement. The budget impact is considered more important. This report was intended as a starting point for a discussion on the role of threshold values in Belgium [20]. In Sweden, no guidance has been given as to acceptable cost-effectiveness ratios, defined in terms of cost per QALY or otherwise, although cost-effectiveness is considered a crucial aspect in the Swedish reimbursement system [21]. Rather than apply a single threshold, there may be different (implicit) thresholds depending on the severity of the disease or an assessment of patient need [22]. In France, no threshold value is used when making health care decisions [23].

In a cost-utility analysis, an intervention is found to be cost-effective if the cost per QALY falls below the WTP for a QALY. This value is used as an external threshold value against which the ICER is compared. In the cost-benefit analysis, the societal value of a QALY is directly incorporated into the analysis. For an intervention to be cost-effective, the INMB (calculated as [QALY(A) – QALY(B)] × λ – [Costs(A) – Costs(B)]) needs to be positive. In the net benefit approach, it is apparent that the impact of a country-specific model parameter on the decision to adopt an intervention depends on the threshold value of the QALY. Though not immediately visible in the cost-utility analysis itself, the impact of a parameter on the probability that the ICER falls below the threshold value equally depends on the level of this threshold value. Whether or not the threshold value is explicitly incorporated into the analysis does to affect the importance of a country-specific parameter in the transferability discussion.

The consequence of our findings may be that we have to adjust the available transferability check lists to encourage checking whether there might be differences in the willingness to pay between the countries, next to checking how well a foreign study represents the circumstances in the country of interest and how much effort is required to adapt particular data inputs to the country of interest. For example, NRT, bupropion and varenicline are currently not covered by the basic health care insurance in the Netherlands, although the health insurance board CVZ has recently advised the Ministry of Health to reimburse them as part of an integrated smoking cessation program [24]. This implies a current willingness to pay of €0. Consequently, when future foreign CE studies of new smoking cessation interventions (e.g., a vaccine) become available, the transferability of the cost of this intervention to the Netherlands will be driving the reimbursement discussion. In contrast, in the UK, where NRT, bupropion and varenicline are already paid by the NHS, the discussion on the transferability of foreign cost-effectiveness studies of such a drug might focus more on the representativeness of the epidemiology of smoking-related diseases for the UK and the utility values of patients having a smoking-related disease.

Conclusion

When judging the transferability of a CE study from one jurisdiction to another, it is relevant to consider the between country differences in threshold values per QALY. Between country, differences in cost-effectiveness are determined by between country differences in unit costs, disease epidemiology, discount rates etc., but the importance of each of these is influenced by the threshold value for a QALY. Between country differences that are important at a low threshold value might be less important at a high threshold value and vice versa.

Appendix: Background information BENESCO model and additional results

The projections of the effects of smoking cessation were based on the BENESCO (Benefits of Smoking Cessation on Outcomes) model [7]. The BENESCO model simulates the consequences of smoking and the benefits of quitting in terms of smoking-related morbidity, mortality and associated medical costs in a population. The model is structured as a Markov model (cycle length 1 year) and follows a hypothetical cohort of current smokers making a single attempt to quit smoking at the beginning of the simulation. The cohort is followed from the time of their quit attempt until all members of the cohort have died. Individuals are classified into one of three smoking states, i.e., smoker, recent quitter (abstinent 1–5 years after successful quit attempt) or long-term quitter. Transition probabilities between smoking states in the first year depend on cessation rates of the interventions, while the probabilities after 1 year depend on relapse rates, which in turn depend on time since quitting. The model simulates the age, gender and smoking status-specific incidence and mortality of four major diseases for which smoking is a well-established risk factor: chronic obstructive pulmonary disease (COPD), lung cancer, coronary heart disease and stroke. Smoking state-specific incidence and mortality rates were calculated using relative risks [25, 26]. The incidence and mortality rates for recent quitters were calculated using the relative risks of former smokers versus never smokers, while the rates for long-term quitters were assumed to be the same as those of never smokers. Because COPD and lung cancer are chronic progressive conditions, these diseases were given hierarchical prominence over the other conditions with acute recurrent events. This means that individuals with COPD or lung cancer remain in this state until they die and cannot move to a CHD or stroke state, whereas individuals with CHD or stroke can move to the COPD or lung cancer state. A patient cannot have two diseases at the same time. The model calculates the total number of smokers and quitters that have one of the smoking-related diseases as well as the number of deaths (due to one of the smoking-related diseases and overall) over the time horizon of the simulation. Based on these numbers, the total health care costs associated with the different disease states and the total number of (quality adjusted) life years are calculated. The model uses three age bands: 18–34 years, 35–64 years and 65 years and older. Subjects alive in the model at age 99 years are all assumed to die in the next cycle. It is assumed that there is no smoking-related morbidity or mortality in the 18–34 years age class.