Quality of economic evaluations in health care

Economic evaluation is becoming established globally as one of the tools for decision making in health care.¹ Its rise in popularity is reflected by the increasing number of published economic evaluations. One source estimates that 1803 economic evaluations were published in medical journals in 1979-90,² rising to 2222 in 1991-6.³ This increase in both the availability of economic evaluations and willingness to use their results to allocate scarce resources reinforces the need for evaluations to be methodologically sound so that the consequent healthcare decisions are ethically defensible. Do our current economic evaluations meet the necessary methodological requirements?

In the early 1990s several systematic reviews cast doubt on the scientific reliability of some published evaluations. All advocated better standards of conducting and reporting economic evaluations.^4–7 A subsequent survey among editors of medical journals found that none had a coherent editorial policy for economic evaluations, and few had peer reviewers with knowledge of health economics.⁸ Thus one of the quality control mechanisms of the research community, peer review, was failing to ensure adequate methodological standards.

Following interest by researchers and editors, the BMJ defined and promulgated guidelines for editors and peer reviewers aimed at ensuring clear standards for both submission and editorial management of economic evaluations.⁹ The impact of the guidelines was evaluated shortly after their publication, but they have since been used only in editorial management.¹⁰ The BMJ guidelines came at a time of similar initiatives to address poor methodology in the reporting of studies, such as the CONSORT statement on randomised controlled trials and, later, the QUORUM statement on systematic reviews. While preliminary indications are that the use of CONSORT is effective in improving the quality of reports of randomised controlled trials,¹¹,¹² the effect of the BMJ and other guidelines on the quality of economic evaluations appears muted.

Several important economic methodological reviews have been published in the 1990s. Although coverage of economic evaluation has been limited and the tools used for quality assessment have varied, the overall conclusions show that there is a long way to go before economic evaluations can be regarded as good enough to justify their use in decision making.

For example, in an assessment of 228 cost utility analyses over the period 1976-97 Neumann et al found wide variations in quality of reporting, a modest improvement over time, and a tendency for better reporting in general medical journals than in specialist journals.¹³ In a similar assessment Gerard et al showed an improvement in clarity of comparator reporting and the nature of the evidence of effectiveness of interventions evaluated. However, incongruent perspectives and serious deficiencies in the estimation of costs, interpretation of results, and the use of patients for eliciting utility weights were recurrent areas of weakness.¹⁴ A review assessing the quality of 41 economic evaluations of antenatal screening published in 1991-9 found methodological problems with most studies, including major flaws in study design, data collection, analysis, and interpretation.¹⁵

An unusual glimpse into the quality of unpublished economic evaluations comes from an assessment of 326 pharmacoeconomic submissions made in 1994-7 by the drug industry to the Australian reimbursement authority. This shows that 218 submissions (67%) had major methodological problems, with 31 of these having more than one problem: 62% of problems related to the choice of estimates for effectiveness of the evaluated pharmaceuticals and 28.5% to methods of modelling and related clinical assumptions. In nine cases there were serious calculation errors.¹⁶ Perhaps the most alarming aspect of the findings is that such serious flaws came to light only after a very detailed assessment—which is beyond the means of routine editorial peer review.

Because improvement in methodological quality has been slow and uneven and initiatives to address the problem seem to have had only limited impact, we believe it is time for action. Journals, grant giving bodies, and regulatory agencies should adopt and enforce explicit peer review policies and use standardised tools for assessing economic submissions—action that has strong parallels with the successful adoption of CONSORT and QUORUM for methodological improvement of trials and systematic reviews. In addition, we need periodic methodological assessments of economic evaluations using adequate sampling frames. The assessments should be ongoing and publicly accessible. Unless swift action is taken low methodological quality risks bringing the practice of economic evaluation into disrepute—an outcome that is in no one's interest.