Pitfalls of prioritizing cost-effectiveness in the assessment of medical innovation: A comment on Wallis and Detsky guest editorial

We read with interest Dr. Wallis and Dr. Detsky’s comments on Health Quality Ontario (HQO)’s report on robotic-asisted radical prostatectomy (RARP).^1,2

We share the authors’ concerns with regard to the quality of the base case cost-effectiveness analysis (CEA) included in the report. The pivotal role of a single trial in parameterizing the model, the shortness of the time horizon, and the inadequate consideration of uncertainty in the evidence base are all highly problematic.

However, we are also concerned that Wallis and Detsky go on to promote further erroneous approaches to the economic evaluation of health technologies. High-quality healthcare resource allocation processes require the use of the best available evidence. Unfortunately, Wallis and Detsky’s suggestions would exacerbate many of the problems with HQO’s analysis rather than remedy them.

Appropriately characterizing uncertainty

The editorialists start from the position that “basing a firm recommendation on a single study requires it to be considered incontrovertible and definitive.” Such a position ignores the distinct approach to considering uncertainty in the evidence base that has been developed in health technology assessment (HTA) and in decision science more generally.

Any given study provides a central estimate of effect and a characterization of the uncertainty around that estimate. If the study is used to parameterize the inputs used in a decision analysis, this uncertainty can, in turn, result in uncertainty in the outputs of that analysis, including the determination of whether a technology is cost-effective. It follows that uncertainty in model inputs may result in the ‘wrong’ treatment option appearing cost-effective, which can have both financial consequences and implications for the health of patients.

Whether the evidence provided by any study is sufficient to support any particular analysis therefore requires an assessment of the risk of making a ‘wrong’ decision and the associated costs — both financial and in terms of diminished health — that would result from such a decision. In some cases, these costs may be substantial, such that a HTA agency would prefer to wait until more robust evidence is available before making a recommendation. However, in other cases, there may be greater costs associated with delaying a decision, and so a HTA agency may prefer to make a recommendation sooner rather than later, despite the potential costs associated with making a ‘wrong’ decision. Requiring decision-makers to wait for a ‘definitive’ study in all cases, regardless of cost, would impose an unnecessary burden upon healthcare systems and potentially diminish the health of patients.

A better approach would be to require that the models used by HTA agencies appropriately reflect uncertainty in input parameters, and also require that agencies consider the potential costs associated with this uncertainty through value-of-information (VOI) analysis.³ Such an approach is consistent with the latest guidelines for conducting economic evaluations published by the Canadian Agency for Drugs and Technologies in Health (CADTH), which require that parameter uncertainty be considering through probabilistic analysis and also that VOI analysis be conducted if there is the possibility of commissioning or conducting further research to reduce parameter uncertainty.⁴

Unfortunately, the HQO model for RARP did not adequately reflect parameter uncertainty, and no analysis was conducted to consider the potential costs associated with this uncertainty. Of particular note, HQO’s base case analysis assumed “no differences in functional and oncological outcomes between robot-assisted and open radical prostatectomy at one year post-surgery,” on the basis that “the results of the clinical review did not find high-quality evidence suggesting differences in long-term outcomes between robot-assisted and open radical prostatectomy.”² This is problematic, since an absence of “high-quality evidence suggesting differences in long-term outcomes” does not provide justification for assuming that long-term outcomes are equivalent with certainty, which is the assumption that HQO effectively made by not modelling differences in long-term outcomes in its base case analysis. A more appropriate assumption would be that it is uncertain whether there are differences in long-term outcomes. This would require modelling long-term outcomes, allowing for the possibility of differences between the treatment options to be considered in a probabilistic analysis. The results of a probabilistic analysis conducted under this approach would reflect this uncertainty.

It follows that HQO’s base case probabilistic analysis of RARP — which erroneously assumed that long-term outcomes were equivalent with certainty — underestimated the true uncertainty in the model results. Indeed, this is apparent from Fig. 19 of HQO’s report, which shows that the incremental quality-adjusted life years (QALYs) associated with RARP lie within just 0.01 QALYs of the mean estimate on every single Monte Carlo simulation.² If true, this implies that HQO’s estimate of the incremental QALYs associated with RARP is accurate to within one-hundredth of a QALY with certainty, a finding that clearly lacks face validity given the absence of high-quality evidence on long-term outcomes. A more realistic account of parameter uncertainty would increase the uncertainty around the estimated incremental costs and QALYs associated with RARP. This might, in turn, increase the uncertainty around whether or not RARP is cost-effective and increase the value of obtaining additional evidence, raising the possibility that HQO should have waited for more robust data to become available before making its recommendation. A more rigorous probabilistic analysis, in conjunction with a VOI analysis, would have increased the usefulness of HQO’s economic analysis to those responsible for recommending whether or not to fund RARP.

Budget impact considerations

Wallis and Detsky note that the estimated total annual budget impact of RARP “ranges from $0.4 to $1.9 million CAD, a small fraction of the total annual healthcare expenditures of $54 billion.” The implication of this comparison appears to be that considerations of cost-effectiveness are less relevant when the budget impact of a technology is a small proportion of the total healthcare budget.

Compared to a total budget of $54 billion, almost any new technology will have a relatively small budget impact, yet this is not a justification for downplaying the potential consequences of its adoption. Investments of $0.4–1.9 million are capable of producing substantial gains in health in many other areas of healthcare, so spending these resources on RARP has an opportunity cost that can be considered in terms of health forgone by other patients. Such an opportunity cost exists whenever resources are spent on new technologies, regardless of the size of the overall budget.

Maintaining equity in healthcare resource allocation also requires that decision rules be applied consistently across patients. The opportunity cost of re-allocating limited healthcare resources to RARP will be borne by other patients within the healthcare system, and a fundamental reason for conducting CEA is to take account of this opportunity cost. Deprioritizing the economic analysis — and hence the consideration given to potential health losses borne by other patients — on the specious reasoning that the budget impact is a small proportion of the overall budget not only risks diminishing population health, but also violates a fundamental principle of equity in resource allocation.

The remit of cost-effectiveness analysis

Wallis and Detsky uncritically cite a now-dated 2003 review of pharmaceutical funding recommendations in Ontario, noting that this paper “showed that cost-effectiveness estimates contributed substantially to the debate only in cases evaluating innovative drugs (those that offer clinically significant benefit with an increased cost).”

The problem with such an approach was described by Birch and Gafni over 20 years ago.⁵ It constrains the remit of CEA to only those technologies that are a source of upward budgetary pressure, rather than allowing it to also inform decisions that may bend the cost curve. As decision-makers become increasingly interested in disinvestment and increased efficiency, the value of CEA to inform decisions outside of this narrow remit should become evident. In any case, HQO was clearly justified in using CEA in its assessment of RARP, given the additional costs that RARP is expected to impose upon the healthcare system.

Time horizon and model sophistication

We agree with Wallis and Detsky that the 12-week time horizon adopted in the HQO model of RARP is inadequate for judging the value of a treatment for a long-term condition. Indeed, guidelines published by CADTH and similar HTA agencies internationally indicate that a lifetime horizon is required to construct an unbiased estimate of the long-term incremental costs and effects of technologies such as RARP.^4,6

However, despite advocating for a longer time horizon, the authors cite the 2003 review of funding recommendations in Ontario for its finding that “for drugs that had no incremental clinical benefit, as was assumed about RARP here, only cost estimates were needed, and most of these did not require sophisticated modelling.”⁷

This finding should not have been cited uncritically for at least two reasons. First, the assumption that drugs can be assumed with certainty to have “no incremental benefit” is not widely regarded as credible within the current HTA literature; indeed, it is for this reason that CADTH now recommends against using ‘cost-minimization analysis’ in its methods guidelines.⁴ Second, if “sophisticated modelling is not required” to derive cost estimates, then it is not obvious how uncertain short-term trial data should be extrapolated over a longer time horizon. Given that a funding recommendation will still have to be made in any case, the absence of formal modelling leaves decision-makers with little option but to informally extrapolate from the short-term evidence or to ignore long-term outcomes entirely. Neither approach supports transparent and procedurally just decision-making.⁸

Final thoughts

Wallis and Detsky rightly raise a number of issues with HQO’s economic evaluation of RARP, and we have outlined a number of additional problems with this analysis here, in particular the incomplete consideration of parameter uncertainty.

Unfortunately, Wallis and Detsky’s response to these issues has been to attempt to diminish the role that economic analysis should play in decision-making. We believe that a better approach is to use more appropriate methods for economic analyses so that these can support and enhance the recommendations of agencies such as HQO. The consideration of high-quality economic evidence is vital if we wish to ensure that the adoption of new technologies improves population health and promotes equity in the allocation of limited healthcare resources.