Introduction
In the first article in this series (EBSJ, volume 3, issue 4), we described some basic terminologies related to health economic studies. In this second and final article, we describe
available tools for evaluating the quality of economic studies and
aspects of a high-quality economic study.
As with clinical studies, health economic studies vary in scope and quality. Their usefulness for decision and policy making is linked to their methodological rigor, accuracy, and generalizability. Thus, critical appraisal is important to put the results in the context of study quality.
Economic evaluations identify and compare appropriate alternatives of care, their incremental impact on health outcomes, and their incremental costs. As described in the first part of the series, there are several types of comprehensive medico-economic evaluations.
Cost-minimization studies consider the cost differences between alternatives of equal effectiveness. This assumes each alternative is equally effective, which is rarely true.
Cost-benefit studies consider both costs and benefits in monetary terms. Expressing benefits in monetary terms is, however, controversial.
Cost-effectiveness studies consider differences in costs and differences in effectiveness, but effectiveness is measured variably between studies. For comparisons involving the same disease, condition, or health state, this may be the most meaningful type of evaluation. It is less helpful for comparing studies involving different diseases.
Cost-utility studies consider differences in costs and outcomes for quality-adjusted survival, most often using the quality-adjusted life year (QALY). Cost-utility studies have the advantage of providing an incremental cost-effectiveness ratio (ICER) expressed as “cost per quality-adjusted life year” (cost per QALY) that eases comparison across multiple studies. This approach incorporates assumptions regarding how each alternative impacts patient's quality of life.
Each employs different methodologies, potentially complicating critical appraisal, but some common criteria can be assessed across studies. No standard, universally accepted method of critical appraisal of economic analyses is currently in use. Across the available methods, there is agreement on several points of methodological quality.1
Assessing the Quality of Economic Evaluations
Several organizations and research groups (e.g., Center for Health Economics, University of York [http://www.york.ac.uk/che/]; Tufts University Cost Effectiveness Registry [http://research.tufts-nemc.org/cear4/]) are dedicated to excellence in the conduct of health economic evaluation and provide detailed insights into many aspects of health economic analysis. The Tufts site contains a registry that allows one to search for cost-effectiveness analyses.
A variety of tools are available to assist new or experienced readers of economic evaluations, such as the Quality of Health Economic Studies (QHES) checklist,2,3 the Consensus Health Economic Criteria list (CHEC),4 and the British Medical Journal (BMJ) standards for quality of health economic evaluations.5 All share emphasis on critically appraising some common domains of economic evaluations. In one formal comparison of QHES, the BMJ, and CHEC instruments,6 the following domains were included for assessment in all three:
Study objective
Economic perspective
Time horizon and discount rate
Outcomes measurement
Outcomes valuation
Cost measurement
Incremental analysis
Handling of uncertainty
Appropriateness of the conclusion
At least two of the instruments included assessment of the following domains:
Economic study design
Population description
Description of alternatives
Outcomes choice
Cost valuation
Details of models
Limitations
Conflict of interest
Let us look at a few of the concepts related to these domains and how they apply to evaluation of study quality in a bit more detail.
What Is the Study Question or Objective of the Economic Evaluation?
Specification of the study question should set the stage for assessing the appropriateness of the economic evaluation design and methods. If the stated study question is to evaluate the cost-effectiveness of artificial disc replacement but only information on costs are presented and no comparison to viable alternative treatments is done, the quality of the study would immediately be suspected and the usefulness to policy makers is limited.
Does the Study Compare Costs and Outcomes of Meaningful and Appropriate Clinical Alternatives?
This is one of the first questions to consider. Studies that report only costs or do not formally compare clinical alternatives are not considered full economic evaluations. High-quality economic evaluation will clearly state all components of both intervention and comparator interventions. Meaningful incremental analysis in a full economic study is only possible if costs and outcomes for competing alternatives for care are reported. It is crucial that both intervention and comparator be realistic, appropriate reflections of actual use or care situations. A justification for why specific alternatives are evaluated should be stated. Studies that do not take such an approach or do not clearly state the alternatives being compared are not considered of high quality.
Does the Study Justify Its Perspective and Time Horizon Clearly?
In a high-quality economic evaluation, the perspective is clearly stated and the rationale for its selection justified. As you may remember from the first article in this series, economic evaluations can take one of several perspectives, which have direct bearing on the types of costs and outcomes considered, such as a patient, payer, provider, health system, or society. The types and sources of cost data, measurement of benefits, and various modeling assumptions are influenced by the choice of perspective. Whatever the choice of perspective, the authors' choice of parameters should reflect it. If a societal perspective is chosen, for example, our critical appraisal should address the extent to which all long-term costs (both direct and indirect costs such as lost productivity, recovery time, and direct costs to patients) and outcomes to health systems and patients are included in the model. The perspective must be consistent with the research question or objective. The data used must be consistent with the perspective.
Economic evaluations generally employ complex modeling techniques to estimate costs and outcomes over time. A high-quality study will use a time horizon of at least 10 years to include all relevant sequelae and relapse, reoperation, or recurrence. Shorter time horizons can also provide valuable information depending on the study objective and available data. For longer time horizons, preferences for future benefits over immediate ones should be described in the study assumptions and modeled appropriately, usually using discounting of costs and benefits beyond 1 year, typically between 3 and 5%.
How Appropriate Are the Data Sources?
Economic evaluations regularly include data from disparate sources such as clinical trials, previously published scientific data, surveys collected from patients, or health care cost and utilization databases. High-quality studies provide clear descriptions for all data sources, they are complete, justify their selection, and employ data that are least subject to bias.
Patient-level clinical and cost data collected alongside a randomized controlled trial (RCT) or high-quality meta-analyses for RCTs are examples of data with low potential for bias. If these data are not available, then data from high-quality observational studies (e.g., methodologically rigorous prospective cohort studies) can provide useful information for modeling.
Existing databases can provide valuable cost estimates as well, though should be critically appraised for their generalizability and their fit with the study's perspective. For example, data from a health plan cost-and-use database used in a study taking a societal perspective might have incomplete data on indirect costs to patients.
Expert opinion—providers' own estimates of clinical or cost estimates—is often biased and therefore considered the least reliable source of data.
Data relevant to all aspects of the economic model need to be described.
How Were Outcomes Measured?
The choice and measurement of the outcome should reflect the study objective. Instruments and/or clinical endpoints (e.g., survival) that are reliable and validated in the patient population of interest are preferred to surrogate or intermediate endpoints. There is an increasing trend toward the use of patient-important outcomes, ideally developed with input from the patient population.7 For spinal conditions, patient-relevant conditions might include pain relief, function, and productivity.8
In a cost-utility analysis, quality-adjusted survival is calculated using a measure of utility combined with the health outcome in question. A high-quality economic evaluation will use validated utility measures and state clearly how they were calculated, ideally from a sample of people affected by condition or from a general community population (clinician assessment of utility would not be the ideal method of assessment). Some universal measures used to estimate utility (reported as a number between 0 and 1, where 1 represents perfect health) include the EQ-5D, the SF-6D, or through patient weighting of hypothetical scenarios such as discrete choice analysis.9,10 Condition-specific measures are also available, such as the Oswestry Disability Index.11
Were Sensitivity Analyses Presented?
Economic evaluations employ a variety of inputs and assumptions, all of which may induce uncertainty and variability. Thus, a high-quality evaluation will employ sensitivity analyses to assess uncertainty and evaluate factors that most influence the results. Some of these techniques can be quite complex. One-way sensitivity analyses vary individual variables (such as reoperation rate, pain reduction, or procedure cost) across a range to see how the results are changed. Probabilistic sensitivity analyses employ bootstrapping or other statistical techniques to arrive at a sense of which variables are “driving” the model results. This can be useful to decision-makers who need to understand the most important components of the cost and how they impact the cost-effectiveness.12
What about Bias?
Just like in clinical studies, a high-quality economic study will discuss the potential, magnitude, and direction of potential bias. For example, leaving out indirect costs to patients in a study taking a societal perspective where indirect costs are higher in the intervention compared with the alternative might alter the cost-effectiveness ratio. Failure to account for the range of adverse events can induce bias. Various assumptions about modeling may introduce bias. For example, if assumptions regarding the modeling of risk of certain adverse events are from poor quality studies or data that do not reflect current practice, the results may inaccurately favor one of the alternatives.
How Transparent Were the Authors?
Increasingly, health economics practitioners are encouraged to provide the details of data sources as well as assumptions made regarding modeling of the data to encourage replication by other teams and so that major assumptions that influence the applicability of the model and study findings can be examined. The potential for conflict of interest has become increasingly important among policy makers. High-quality economic study will clearly state the funding sources provided to conduct the work and the role of the funder in analysis and reporting of the study.
Are the Conclusions Appropriate?
As in assessing any clinical study, the assessment of its conclusions can be somewhat subjective but should be based on clinical knowledge and can be somewhat of an art. In addition to the points presented earlier, a more “gestalt” level assessment can consider how well
the study objective matched the study design and methods employed; as well as
the strength and certainty of the justification of the conclusions given the types of data used, modeling assumptions, perspective, time horizon, and potential sources of bias.
In other words, if the conclusions of a study seem too good to be true, then they just might be.
Summary
There are many opportunities for bias in economic evaluations. Thus, critical appraisal of economic studies is very important but complex. No standard methodology for appraisal has been universally adopted. However, consideration of the common elements that constitute a high-quality economic study provides the clinician with the tools to understand when one might be dealing with a high-quality study, a deeply flawed one, or one that made the best use of the data available.
Studies of economic impact have become increasingly important in health policy decisions given the finite resources available for health care. While they should not be the sole basis of decision making, such studies provide potentially valuable information if well done. It is therefore important that high-quality economic studies be conducted and the results of economic studies be placed in the context of their quality.
Learn more about economic evaluation
Drummond MF, Sculpher MJ, Torrance GW, O'Brien BJ, Stoddart GL. Methods for the Economic Evaluation of Health Care Programs. New York, NY: Oxford University Press; 2005
What is an economic evaluation? Introduction to Economic Evaluation. Centers for Disease Control and Prevention. Available at: http://www.cdc.gov/owcd/EET/SeriesIntroduction/1.html
Health Economics Information Resources. A self-study course. National Information Center on Health Services Research and Health Care Technology (NICHSR). Available at: http://www.nlm.nih.gov/nichsr/edu/healthecon/beginningend.pdf
Boos N. The impact of economic evaluation on quality management in spine surgery. Eur Spine J 2009;18(Suppl 3):338–347
Holtz A, Nelson HD, Reid E. Our questions, our decisions: standards for patient-centered outcomes research. Preliminary Draft Methodology Report; PCORI Methodology Committee; Patient-Centered Outcomes Research Institute 2012; Available at: http://www.pcori.org/assets/Preliminary-Draft-Methodology-Report.pdf
Funding
This article was funded by AOSpine.
References
- 1.Hjelmgren J, Berggren F, Andersson F. Health economic guidelines—similarities, differences and some implications. Value Health . 2001;4(3):225–250. doi: 10.1046/j.1524-4733.2001.43040.x. [DOI] [PubMed] [Google Scholar]
- 2.Chiou C F, Hay J W, Wallace J F. et al. Development and validation of a grading system for the quality of cost-effectiveness studies. Med Care . 2003;41(1):32–44. doi: 10.1097/00005650-200301000-00007. [DOI] [PubMed] [Google Scholar]
- 3.Ofman J J, Sullivan S D, Neumann P J. et al. Examining the value and quality of health economic analyses: implications of utilizing the QHES. J Manag Care Pharm . 2003;9(1):53–61. doi: 10.18553/jmcp.2003.9.1.53. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Evers S, Goossens M, de Vet H, van Tulder M, Ament A. Criteria list for assessment of methodological quality of economic evaluations: Consensus on Health Economic Criteria. Int J Technol Assess Health Care. 2005;21(2):240–245. [PubMed] [Google Scholar]
- 5.Craig D Nixon J Armstrong N Glanville J Kleijnen J Drummond M NHS Economic Evaluation Database Research Team; Response of NHS Economic Evaluation Database Research Team New BMJ policy on economic evaluations BMJ 20033267386445, author reply 445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Gerkens S, Crott R, Cleemput I. et al. Comparison of three instruments assessing the quality of economic evaluations: a practical exercise on economic evaluations of the surgical treatment of obesity. Int J Technol Assess Health Care . 2008;24(3):318–325. doi: 10.1017/S0266462308080422. [DOI] [PubMed] [Google Scholar]
- 7.Gabriel S E, Normand S L. Getting the methods right—the foundation of patient-centered outcomes research. N Engl J Med . 2012;367(9):787–790. doi: 10.1056/NEJMp1207437. [DOI] [PubMed] [Google Scholar]
- 8.Copay A G, Glassman S D, Subach B R, Berven S, Schuler T C, Carreon L Y. Minimum clinically important difference in lumbar spine surgery patients: a choice of methods using the Oswestry Disability Index, Medical Outcomes Study questionnaire Short Form 36, and pain scales. Spine J . 2008;8(6):968–974. doi: 10.1016/j.spinee.2007.11.006. [DOI] [PubMed] [Google Scholar]
- 9.Lancsar E, Wildman J, Donaldson C, Ryan M, Baker R. Deriving distributional weights for QALYs through discrete choice experiments. J Health Econ . 2011;30(2):466–478. doi: 10.1016/j.jhealeco.2011.01.003. [DOI] [PubMed] [Google Scholar]
- 10.Sach T H, Barton G R, Jenkinson C, Doherty M, Avery A J, Muir K R. Comparing cost-utility estimates: does the choice of EQ-5D or SF-6D matter? . Med Care . 2009;47(8):889–894. doi: 10.1097/MLR.0b013e3181a39428. [DOI] [PubMed] [Google Scholar]
- 11.Mehra A, Baker D, Disney S, Pynsent P B. Oswestry Disability Index scoring made easy. Ann R Coll Surg Engl . 2008;90(6):497–499. doi: 10.1308/003588408X300984. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Andronis L Barton P Bryan S Sensitivity analysis in economic evaluation: an audit of NICE current practice and a review of its use and value in decision-making Health Technol Assess 20091329iii, ix–xi, 1–61 [DOI] [PubMed] [Google Scholar]