In economic evaluation of healthcare interventions, the dominant practice is to calculate an incremental cost effectiveness ratio, usually based on the comparison of a new intervention against current practice. Canadian and UK health economists question the economic foundations of such an approach
Who could resist implementing the results of a study showing that using alteplase (tPA; tissue plasminogen activator) rather than streptokinase in treatment of acute myocardial infarction costs $32 678 (£21 340; €33 330) per life year gained, which the authors declare to be “cost effective by customary criteria”?1 Despite similar claims from several such studies, the impact of economic evaluation on setting of priorities remains unclear.2–4 Among the reasons given for this are that opportunities for reducing costs while maintaining quality still arise, and that cost effectiveness analyses do not take all factors into account.3
Achieving the same result more cheaply—a success for economics—represents a classic cost effectiveness approach. The possibility that not all factors have been considered suggests that other approaches may make economic evaluation more relevant. We contend that, beyond the classic approach, many studies labelled as cost effectiveness analyses of health care are not really that at all. At best, this mislabelling is confusing: at worst, conclusions drawn by the studies' authors could be harmful to patients' health. Thus, there are contraindications to the use of cost effectiveness analysis in health care, and an alternative economic approach is required.
In this paper we revisit the basic economic principles. Then we make the case that lack of adherence to such principles, through current practice of reducing everything to incremental cost effectiveness ratios, leads to contraindications.
Summary points
The theoretical basis of the use of incremental cost effectiveness ratios needs re-examination
These ratios have little to do with cost effectiveness, as they often imply a need for more resources, thus raising questions of broader resource allocation
Lack of recognition of this allocation aspect is likely to lead to disappointment with economic evaluation generally
Authors making economic evaluations should pay more attention to the type of efficiency question being considered and its implications for opportunity cost; they should not make recommendations if resource allocation issues are involved
Guidelines for economic evaluations should be revised to take account of concepts of efficiency
Economic evaluation
Basic principles
The basic principle of economic evaluations is opportunity cost: use of resources to meet a need incurs an opportunity cost, that being the benefit which could be obtained by next best use of those resources. Efficiency arises when benefits are maximised and opportunity costs minimised. To achieve efficiency, information on both resource use (costs) and benefits (often health gains) that would result from alternative approaches is needed. By deriving estimates of the costs and the effectiveness of a new procedure, and relating them to the status quo, it should be possible to determine whether the procedure is less costly and at least as effective as the status quo, in which case it would be judged to be better (more technically efficient); or more costly and more effective, in which case a judgment has to be made about whether the extra cost is worth the gains achieved (a question of allocative efficiency—if the number of patients treated remains the same, more resources would have to be allocated to this area of care at the expense of another group of patients).
Our main point is that cost effectiveness analysis alone cannot handle questions of allocative efficiency, although it is currently being asked to do so, but there are economic evaluation approaches that can. Data on effectiveness and cost can be brought together in a matrix, combining potential impacts on effectiveness and cost resulting from a change in care (figure). The optimum position is cell A1; costs are saved and greater effectiveness achieved relative to existing care. Interventions in cells A2 and B1 are similarly technically efficient and assigned a green light (“worthwhile”). Correspondingly, interventions in cells B3, C2, and C3 receive a red light (“not worthwhile”). In C1, a judgment is required as to whether the more costly new procedure is worthwhile in terms of additional effectiveness (the allocative efficiency question). Cell A3 also requires judgment, but that combination is unlikely to arise in practice. Cell C1 is where alteplase falls, along with the results of many other “cost effectiveness analyses.”
An old example: colon cancer screening
To illustrate these principles, economists often refer to the case of the “sixth stool guaiac.”5 In the mid-1970s, the American Cancer Society recommended that, when cancer of the colon is suspected, each stool sample be tested six times. The first part of a sample would be tested, and, if results were positive, the patient would have further confirmatory tests and, if necessary, treatment. If the first test was negative, the second part would be tested. If this was positive, the patient would have further confirmatory testing; if it was negative, the third part would be tested, and so on. A screened person would be declared negative only after all six parts had been tested. Neuhauser and Lewicki's analysis of this policy showed that about 66 of the expected 72 cases are detected after the first round of testing, at a cost of $1175 per case detected (table 1).5 The second round of testing ensures that almost all cases are detected, at an average cost of $1507 per case. Six rounds capture all cases, at a cost per case of $2451.
Table 1.
Cases of cancer detected in a 10 000 population by guaiac testing and costs ($) of screening with six sequential tests
| No of tests
|
Total cases detected
|
Total costs
|
Average costs
|
|---|---|---|---|
| 1 | 65.0465 | 77 511 | 1175 |
| 2 | 71.4424 | 107 690 | 1507 |
| 3 | 71.9003 | 130 199 | 1811 |
| 4 | 71.9385 | 148116 | 2059 |
| 5 | 71.9417 | 163 141 | 2268 |
| 6 | 71.9420 | 176 331 | 2451 |
A more revealing way to look at the data, however, is in terms of the extra costs and cases detected from each successive round of testing (table 2). Two rounds detects an extra 5.5 cases compared with one, the extra cost being $30 179, or $5492 per case. Six rounds over five provides little gain—at an extra cost per extra case detected of over $47 million. This intervention is definitely in cell C1 of the matrix in the figure.
Table 2.
Incremental cases detected and incremental (and marginal) costs ($) of screening with six sequential tests
| No of tests
|
Total cases detected
|
Total costs
|
Average costs
|
|---|---|---|---|
| 1 | 65.0465 | 77 511 | 1 175 |
| 2 | 5.4956 | 30 179 | 5 492 |
| 3 | 0.4580 | 22 509 | 49 150 |
| 4 | 0.0382 | 17 917 | 469 534 |
| 5 | 0.0032 | 15 024 | 4 724 695 |
| 6 | 0.0003 | 13 190 | 47 107 214 |
Referring back to our theoretical concepts, the $47m indicates that the opportunity cost of the $13 190 spent on having six rounds rather than five is too great. These resources could produce more benefit if used for another need elsewhere. In a cash limited healthcare system, we would contract this service at the margin (by having fewer rounds of screening, but not eliminating the service) and reinvest the resources elsewhere, producing more benefits overall.
Contraindications to cost effectiveness analysis
The term cost effectiveness has a specific meaning in economics. It has to do with organising inputs to production in the most technically efficient way, choosing the combination which minimises costs. Thus, cost effectiveness analysis is often introduced as a method for determining:
The least cost way of achieving a given output (or goal),6 often referred to in the health literature as cost minimisation analysis
Whether the same level of output be achieved with less of one input
The best way of spending a given budget for a group of patients.7
All these are consistent with being in cells A1, A2, or B1 (or their opposites) in the figure. (Of course, in establishing cost effectiveness or improved technical efficiency, resources may be saved, raising a further allocative question of how to best spend these savings. Thus, technical and allocative efficiency are intimately linked.)
At this stage, note two points. Firstly, in the above contexts, cost effectiveness analysis does not involve comparisons of groups of patients with different diseases. If a cheaper way of achieving a given health improvement is found, this can simply be substituted for the previous form of care. The same group of patients is treated by whichever intervention is more cost effective. Secondly, cost effectiveness analysis will not involve consideration of whether a budget allocated to treatment of a given group of patients (or population sub-group) should be expanded. If the budget were expanded, we would be in cell C1 of the figure, not A1, A2, or B1. The extra resources would have to come from some other activity which (contradicting the first point in this paragraph) would involve comparisons of groups in terms of whether the gains to be had by expanding the budget in one area are greater than potential gains elsewhere. Among other things, opportunity costs will depend on resources available to the healthcare funder. Given this, it is inappropriate for study authors to make claims that an intervention should or should not be implemented or to say whether it is cost effective when results fall in C1. Such recommendations could lead to inappropriate adoption or rejection of a new intervention. We must now consider what happens in cell C1.
Whither the incremental cost effectiveness ratio?
The most common output from an economic evaluation is the incremental cost effectiveness ratio, determined by measuring the incremental benefits (say, in life years) from a new intervention and dividing these by the incremental cost, usually relative to current practice.
The use of the incremental cost effectiveness ratio has two important features for the purposes of our argument. Firstly, as noted, cost and effectiveness are usually increased relative to the status quo. Secondly, it is common for results of studies that have used incremental cost effectiveness ratios to include recommendations. Our MEDLINE search of four major journals (New England Journal of Medicine, JAMA, Annals of Internal Medicine, and the BMJ) from 1 January 1999 to 22 August 2001 for the terms “cost effective” or “cost effectiveness analysis” found 132 articles. Review of the abstracts indicated that 86 articles were not relevant; 53 of these had no abstract, and the remaining 33 were guideline reviews or not fully worked up, despite the terms appearing in the abstract. Thirty two of the 46 relevant articles belonged in cell C1, yet the interventions in these studies were described as cost effective or were recommended for use. Table 3 presents a sample of these studies with their recommendations.6–16 To fund these more expensive treatments, interventions for some other group(s) would have to be sacrificed. Seven studies, also falling into cell C1, determined a ratio of costs and benefits without going on to present the results as cost effective, another seven presented results as incremental costs and benefits (with no ratio), and one made a specific recommendation for implementation. None of the 46 recognised that extra resources would be required to fund the “cost effective” option. The 32 studies falling into cell C1 clearly contravene the points we have made in this article; the other 14 were indeterminate.
Table 3.
Examples of the misuse of the incremental cost effectiveness ratio
| Comparators
|
Incremental cost effectiveness ratio
|
Authors′ comments/interpretations
|
|---|---|---|
| Combined antiretroviral therapy for HIV v no therapy | $13 000-23 000 per QALY gained | Treatment of HIV infection with a combination of three antiretroviral drugs is a cost effective use of resources10 |
| 3 days' hospitalisation following acute myocardial infarction v 4 days' hospitalisation | $105 629 per life year saved | Hospitalisation of patients with uncomplicated myocardial infarction beyond three days after thrombolysis is economically unattractive by conventional standards11 |
| Low molecular weight heparins v unfractionated heparin | $7820 per QALY gained | Low molecular weight heparins are highly cost effective for inpatient management of venous thrombosis12 |
| Screening for hereditary haemochromatosis v no screening | $508 per life year saved | HFE testing for the C282Y mutation is a cost effective method of screening relatives of patients13 |
| Colonoscopy v occult blood testing | $11 382 per case detected | At a higher total cost of screening, colonoscopy represents a cost effective alternative because additional life years are saved to justify additional costs14 |
| Specific mammography screening strategy in women over 70 v no screening | $66 773 per life year saved | . . . results in a small gain in life expectancy and is moderately cost effective15 |
| Sildenafil v papaverine-phentolamine for erectile dysfunction | 3639 per QALY gained | Treatment with sildenafil is cost effective16 |
| Systematic diabetic eye screening v opportunistic screening | 32 per true positive identified | Replacing existing programmes with systematic screening for diabetic eye disease is justified17 |
| Two view v one view mammography reading | 6589-6716 per case detected | Given limited resources, priority should be given to introducing double reading [as this] is more cost effective18 |
QALY=quality adjusted life year
Although many reviews have assessed how well studies comply with guidelines,17,18 no other reviews have scrutinised economic evaluations in this way. To our knowledge, only one set of guidelines and two short commentaries consider that many “economic” evaluations seem to ignore some of the basics of economics.19–21
Given that the incremental cost effectiveness ratio implies that more resources may need to be allocated to an area of care, it seems that the incremental cost effectiveness ratio has nothing to do with cost effectiveness, as it raises allocative (as opposed to technical) efficiency questions involving considerations of opportunity cost: where would these incremental resources come from, and what would have to be given up? Going back to the example of alteplase, the authors state that its implementation would cost the United States about $500m a year.1 In a comparison with other interventions, these resources might be better spent on HIV prevention or hip replacements, for example. Cost effectiveness analysis, as defined in the textbooks and in the figure, gives the impression that a more cost effective treatment can be substituted for a less cost effective one, with no sacrifice involved; this is clearly not the case with incremental cost effectiveness analysis.
It is no wonder that decision makers become frustrated with health economics. Perhaps health economists have not been clear about limitations of economic evaluation. Recommendations that ignore opportunity costs will either not be relevant to decision makers or, if blindly followed, may result in inappropriate adoptions or rejections of treatments. This is partly the result of the current decision making culture, which expects an unambiguous answer. Within the confines of a particular evaluation (when study results fall into cell C1 of the figure), however, that answer cannot be given, but only an insight on the magnitudes of the extra costs and benefits involved. Focusing on an incremental cost effectiveness ratio and whether it falls above or below implied thresholds of $50 000 or £30 000, or on vague comparison with “customary criteria,” detracts from thinking about opportunity cost, which is the basis of the economic approach. Making the adoption of the recommendations of the National Institute of Clinical Excellence compulsory adds a further threat to adherence to the opportunity cost principle.
Where now? Towards cost benefit analysis
Confusion abounds. Allocative efficiency questions are normally considered by cost benefit analysis, which has been excluded in many national and international guidelines,22 leaving cost effectiveness analysis to deal with questions of both technical and allocative efficiency. Yet cost effectiveness analysis as we have defined it is relatively simple for users of economic evaluation to interpret, and it generates relatively simple decision rules. So, beyond this, where do we go?
The first practical message is that when a new treatment is more costly than current practice, analysts should highlight the potential opportunity costs involved. A first step in this is to present an estimate of the extra costs of introducing the new treatment, as was done in the alteplase example. This is becoming known as budget impact analysis,23 although whether another label is needed is questionable. We already have cost benefit analysis, which highlights decisions about allocating more resources to an area of care, even if the benefits of doing so are not measured in monetary terms.
Secondly, if benefits are not measured in money, a balance sheet approach to cost benefit analysis can be taken.24 Costs and resource implications (whether negative or positive) are measured on one side, while benefits in terms of health and wellbeing are listed on the other. This framework is similar to the “cost consequence” type of analysis identified by some guidelines. Again, we question the need for adding yet another label to the economic evaluation repertoire. The same argument applies to “cost utility” analysis. If we keep to definitions based on whether we are dealing with a question of technical or allocative efficiency, we need only two types of evaluation label, cost effectiveness analysis and cost benefit analysis.
Thirdly, in the C1 type of situation (higher cost, higher effectiveness), study authors should not be making recommendations about what treatments to accept or reject.
Fourthly, guidelines for economic evaluation rarely talk about efficiency concepts. Different types of efficiency questions which could be considered should be outlined at the start of an evaluation, with the caveat that the economic evaluation may not give a clear cut result. Increased attention should be paid to defining efficiency in revisions of guidelines.
Conclusion
The perception that economic evaluation has been unsuccessful in health care may arise from expecting too much. In our view, classic cost effectiveness analysis has been successful. The concept of cost effectiveness does not apply where new treatments require more resources, and misuse of the incremental cost effectiveness ratio, often by researchers, may lead to inefficient treatments being adopted. These contraindications to cost effectiveness analysis should be considered when it is used.
Figure.
Effectiveness-cost matrix for comparison of new treatment with current care
Footnotes
Funding: CD is PPP Foundation professor of health economics at the University of Newcastle upon Tyne; he is also Canadian Institutes for Health Research senior investigator and Alberta Heritage senior scholar at the University of Calgary. GC is an Alberta Heritage population health investigator. CM is funded by the Canadian Health Services Research Foundation. The views expressed in this article are those of the authors, and not of the funders.
Competing interests: None declared.
References
- 1.Mark DB, Hlatky MA, Califf RM, Naylor CD, Lee KL, Armstrong PW, et al. Cost effectiveness of thrombolytic therapy with tissue plasminogen activator as compared with streptokinase for acute myocardial infarction. N Engl J Med. 1995;332:1418–1424. doi: 10.1056/NEJM199505253322106. [DOI] [PubMed] [Google Scholar]
- 2.Prosser L, Koplan J, Neuman P, Weinstein M. Barriers to using cost-effectiveness studies in managed care decision making. Am J Manag Care. 2000;6:173–179. [PubMed] [Google Scholar]
- 3.Litvak E, Long MC, Schwartz JS. Cost-effectiveness analysis under managed care: not yet ready for prime time? Am J Manag Care. 2000;6:254–255. [PubMed] [Google Scholar]
- 4.Hoffman C, Graf von der Schulenburg JM.for the EUROMET group. The influence of economic evaluation studies on decision making Health Policy 200052179–192. [DOI] [PubMed] [Google Scholar]
- 5.Neuhauser D, Lewicki AM. What do we gain from the sixth stool guaiac? N Engl J Med. 1975;293:255–258. doi: 10.1056/NEJM197507312930504. [DOI] [PubMed] [Google Scholar]
- 6.Weinstein M, Stason WB. Foundations of cost effectiveness analysis for health and medical practices. New Engl J Med. 1977;296:715–721. doi: 10.1056/NEJM197703312961304. [DOI] [PubMed] [Google Scholar]
- 7.Johannesson M, Jonsson B. Economic evaluation in health care: is there a role for cost-benefit analysis? Health Policy. 1991;17:1–23. doi: 10.1016/0168-8510(91)90114-d. [DOI] [PubMed] [Google Scholar]
- 8.Freedberg KA, Losina E, Weinstein MC, Paltiel AD, Cohen CJ, Seage GR, et al. The cost effectiveness of combination antiretroviral therapy for HIV disease. N Engl J Med. 2001;344:824–831. doi: 10.1056/NEJM200103153441108. [DOI] [PubMed] [Google Scholar]
- 9.Newby LK, Eisenstein EL, Califf RM, Thompson TD, Nelson CL, Peterson ED, et al. Cost effectiveness of early discharge after uncomplicated acute myocardial infarction. N Engl J Med. 2000;342:749–755. doi: 10.1056/NEJM200003163421101. [DOI] [PubMed] [Google Scholar]
- 10.Gould MK, Dembitzer AD, Sanders GD, Garber AM. Low-molecular-weight heparins compared with unfractionated heparin for treatment of acute deep venous thrombosis. Ann Intern Med. 1999;130:789–799. doi: 10.7326/0003-4819-130-10-199905180-00002. [DOI] [PubMed] [Google Scholar]
- 11.El-Serag HB, Inadomi JM, Kowdley KV. Screening for hereditary hemochromatosis in siblings and children of affected patients: a cost-effectiveness analysis. Ann Intern Med. 2000;132:261–269. doi: 10.7326/0003-4819-132-4-200002150-00003. [DOI] [PubMed] [Google Scholar]
- 12.Sonnenberg A, Delco F, Inadomi JM. Cost-effectiveness of colonoscopy in screening for colorectal cancer. Ann Intern Med. 2000;133:573–584. doi: 10.7326/0003-4819-133-8-200010170-00007. [DOI] [PubMed] [Google Scholar]
- 13.Kerlikowske K, Salzmann P, Phillips KA, Cauley JA, Cummings SR. Continuing screening mammography in women aged 70 to 79 years. JAMA. 1999;282:2157–2163. doi: 10.1001/jama.282.22.2156. [DOI] [PubMed] [Google Scholar]
- 14.Stolk EA, Busschabach JJ, Caffa M, Meuleman EJ, Rutten FF. Cost utility analysis of sildenafil compared with papaverine-phentolamine injections. BMJ. 2000;320:1165–1168. doi: 10.1136/bmj.320.7243.1165. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.James M, Turner DA, Broadbent DM, Vora J, Harding SP. Cost effectiveness analysis of screening for sight threatening diabetic eye disease. BMJ. 2000;320:1627–1631. doi: 10.1136/bmj.320.7250.1627. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Johnston K, Brown J. Two view mammography at incident screens: cost effectiveness analysis of policy options. BMJ. 1999;319:1097–1102. doi: 10.1136/bmj.319.7217.1097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Neumann PJ, Stone PW, Chapman RH, Sandberg EA, Bell CM. The quality of reporting in published cost-utility analyses, 1976-1997. Ann Intern Med. 2000;132:964–972. doi: 10.7326/0003-4819-132-12-200006200-00007. [DOI] [PubMed] [Google Scholar]
- 18.Garcia-Altes A. Twenty years of health care economic analysis in Spain: are we doing well? Health Econ. 2001;10:715–729. doi: 10.1002/hec.608. [DOI] [PubMed] [Google Scholar]
- 19.Drummond MF, Jefferson T.on behalf of the BMJ Economic Evaluation Working Party. Guidelines for authors and peer reviewers of economic submissions to the BMJ BMJ 1997313275–283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gafni A, Birch S. Guidelines for the adoption of new technologies: a prescription for uncontrolled growth in expenditures and how to avoid the problem. Can Med Assoc J. 1993;148:913–917. [PMC free article] [PubMed] [Google Scholar]
- 21.Palmer S, Byford S, Raftery J. Types of economic evaluation. BMJ. 1999;318:1349. doi: 10.1136/bmj.318.7194.1349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Gold MR, Siegel JE, Russell LB, Weinstein MC, editors. Cost-effectiveness in health and medicine. New York: Oxford University Press; 1996. [Google Scholar]
- 23.Trueman P, Drummond M, Hutton J. Developing guidance for budget impact analysis. Pharmacoeconomics. 2001;19:609–621. doi: 10.2165/00019053-200119060-00001. [DOI] [PubMed] [Google Scholar]
- 24.Donaldson C, Hundley V, McIntosh E. Using economics alongside clinical trials: Why we cannot choose the evaluation technique in advance. Health Econ Lett. 1996;5:267–269. doi: 10.1002/(SICI)1099-1050(199605)5:3<267::AID-HEC209>3.0.CO;2-X. [DOI] [PubMed] [Google Scholar]