Cost–utility analysis: Use QALYs only with great caution

Is it cost-effective? This question is increasingly asked about health interventions as we attempt to achieve the maximum benefit from limited health care resources. As a result, cost–effectiveness analysis, which attempts to define exactly how much good is achieved for each dollar spent, is playing a greater role in shaping our health care system. One form of cost–effectiveness analysis, cost–utility analysis, allows the comparison of different health outcomes (such as prolongation of life, prevention of blindness or relief of suffering) by measuring them all in terms of a single unit — the quality-adjusted life-year (QALY). To do this, any state of health or disability is assigned a utility on a scale ranging from 0 (immediate death) to 1 (a state of perfect health). The outcome of any health intervention can then be calculated as the product of the increase in utility that it may cause and the time in years over which it may be enjoyed. When allocating scarce resources, those interventions that are expected to produce fewer QALYs for any given cost are given a lower priority.

The ability to compare directly the dollar cost of different health outcomes is attractive to the decision-maker. However, the use of the QALY for this purpose has severe limitations and these must be widely understood, above all by the decision-makers who use QALYs. Descriptions of cost–utility analysis can be found elsewhere.^1,2

What are these limitations? When should we question evidence based on this form of analysis? First, we should insist that the QALY, like any other unit of measurement, be meaningful (its significance understandable by the user), valid (measuring what it is intended to measure), reliable (repeatable by the same or different individuals) and relevant (applicable to the population affected by the policy in question). In practice, it is always hard and often impossible to be sure that these conditions are met. As a typical example, let us consider a study of the drug sildenafil, which was carried out explicitly to help insurers decide whether they should cover the cost of this drug.³

Sildenafil is a drug taken shortly before sexual intercourse to correct erectile dysfunction. There has been uncertainty as to whether insuring agencies, including Canadian provincial governments, should cover its cost. The study in question³ concluded that the cost–effectiveness (direct medical costs only) of taking sildenafil 5 times per month would be US$11 230 per QALY, and that this “compares favorably with other medical conditions, costing less than renal dialysis, cholesterol lowering medication, and coronary artery bypass grafting.”

From this conclusion, we are presumably to understand that experts have demonstrated, by some method that has been found acceptable by reputable medical journals, that to enable an individual to avoid erectile dysfunction 5 times per month would “cost less than” other interventions such as renal dialysis, which cost more per QALY (and thus be a better use of limited funds). To what extent does this use of the QALY conform to the criteria outlined earlier?

Is the QALY meaningful? Sildenafil caused no significant change in the duration of life. The critical estimate was the loss of utility associated with loss of erectile function. To many of us, it is misleading to report a change in the quality of life in units of length of life, namely “life-years” (adjusted). As stated by Russell in Gold and colleagues,⁴ cost–effectiveness analysis “can be used to evaluate interventions for treating schizophrenia and interventions for treating heart disease. But the health outcomes are so different that it is difficult to capture them in the same measurement system and direct comparisons of the QALYs created by the two kinds of interventions may not yet be possible.”

Is this use of the QALY valid? Does it measure what it is intended to measure? The utility on which the QALY is based is intended to reflect the level of preference for the outcome in question, as a basis for the commitment of societal resources. This it only partially achieves. As expressed by Nord,⁵ the assumption of “distributive neutrality” that underlies the QALY frequently violates societal concerns for fairness in the allocation of health care resources. For example, in general, society does not consider a unit of health gained by a severely ill individual to be of equal value to a unit of health gained by an individual who is less severely ill.⁵ Though as yet unstudied, a QALY gained through correction of erectile dysfunction by an otherwise healthy individual would probably not be considered equivalent to a QALY gained through life-prolonging dialysis by an individual about to die from renal failure. To ignore this and other differences in the societal value of the QALY could seriously mislead health policy decisions.

Is the QALY a reliable unit on which to base health policy decisions? Is it repeatable? Utilities are currently measured by different techniques, and the results vary according to the method used.^1,2,6,7 Indeed, as Gold and colleagues state, the “diversity in how preference weights are gathered markedly constrains the ability to credibly compare analyses where the effectiveness measure is presented in QALYs.”⁸ Even when the same investigators use the same methods, the repeatability of individual utility estimates, both within and between studies, can be very poor.^9,10 Thus, reliability cannot be assumed, especially when utilities are measured by different techniques.

Is the QALY relevant? Is the QALY measured in this study applicable to the population that will be influenced by the policy in question? Utility estimates vary according to who is making the estimate.² The increase in utility attributed to sildenafil in this study³ was based on a paper by Volk and colleagues⁶ who questioned 10 healthy men whose average age was 56 years about the importance they attached to erectile function. Using the time trade-off technique, these men estimated that, compared with perfect health, the reduction of utility they would experience with impotence would be 0.26 (i.e., they would forfeit 26% of their expected healthy life to avoid impotence!). However, when this was estimated by their wives, the result was remarkably different: 0.02. Physicians who treat prostate cancer have estimated the loss of utility attributable to impotence to be 0.05¹¹ and 0.15.¹² Whose estimate is relevant to the decision to cover the cost of this drug? There is as yet no unanimity as to whose viewpoint should be used when making societal policy decisions.^1,2

Second, in addition to these limitations to the use of cost–effectiveness analysis, we must also remember that the sole objective of carrying out a cost–utility analysis is to compare, explicitly or implicitly, one use of resources with other possible uses. When the studies with which the cost–utility analysis in question can be compared are not identified, the cost–utility analysis should clearly not be used in health policy decisions.

If we cannot use QALYs, how should we compare different health outcomes? Unfortunately, until we can find a measure that is universally meaningful, valid, reliable and relevant, we will have to return to reporting outcomes in natural units, such as years of blindness, headaches or erectile dysfunction prevented, and decision-makers will have to decide on their preferences for the use of resources without this aid. Although this may be more difficult, they will at least understand what they are doing.