HTA – Algorithm or Process?

Abstract

Daniels, Porteny and Urrutia et al make a good case for the idea that that public decisions ought to be made not only "in the light of" evidence but also "on the basis of" budget impact, financial protection and equity. Health technology assessment (HTA) should, they say, be accordingly expanded to consider matters additional to safety and cost-effectiveness. They also complain that most HTA reports fail to develop ethical arguments and generally do not even mention ethical issues. This comment argues that some of these defects are more apparent than real and are not inherent in HTA – as distinct from being common characteristics found in poorly conducted HTAs. More generally, HTA does not need "extension" since (1) ethical issues are already embedded in HTA processes, not least in their scoping phases, and (2) HTA processes are already sufficiently flexible to accommodate evidence about a wide range of factors, and will not need fundamental change in order to accommodate the new forms of decision-relevant evidence about distributional impact and financial protection that are now starting to emerge. HTA and related techniques are there to support decision-makers who have authority to make decisions. Analysts like us are there to support and advise them (and not to assume the responsibilities for which they, and not we, are accountable). The required quality in HTA then becomes its effectiveness as a means of addressing the issues of concern to decision-makers. What is also required is adherence by competent analysts to a standard template of good analytical practice. The competencies include not merely those of the usual disciplines (particularly biostatistics, cognitive psychology, health economics, epidemiology, and ethics) but also the imaginative and interpersonal skills for exploring the "real" question behind the decision-maker’s brief (actual or postulated) and eliciting the social values that necessarily pervade the entire analysis. The product of such exploration defines the authoritative scope of an HTA.

Health technology assessment (HTA) can be seen as a form of policy research,¹ as a culturally specific construct² as a source of good evidence³ or, as I shall suggest, as a decision-making process that is embedded in further public and professional policy processes.

Daniels, Porteny and Urrutia (henceforth DPU)³ make a good case for the idea that public decisions ought to be made not only “in the light of” evidence but also “on the basis of” budget impact, financial protection and equity. HTA^[1] should accordingly, they say, be expanded to consider matters other than safety and cost-effectiveness^[2]. The phrases in quotation marks imply the necessity for conversation and deliberation amongst the deciders or their advisers. However, “most HTA reports…fail to develop ethical arguments and generally do not even mention ethical issues” [p. 1],³ a view shared by others.^6,7 At this (quantitative) point about “most” papers, a paper by Hoffman⁸ is cited which does not, unfortunately, state what a “HTA report” is and certainly does not cite or count any. If this is a reference to reports on technology assessments conducted by the likes of the National Institute of Health and Care Excellence (NICE)^[3] in England and Wales, it seems inaccurate as well as unfair, for NICE scopes its appraisals conscientiously, lays bare its social value judgments and has considerable public participation – all of which provide ample opportunity for ethical discussion. If it is a comment on specific journal articles dealing with the effectiveness or aspects of the cost-effectiveness of an intervention, it depends on whether the intention of the authors (or of those who commissioned the research in question) was to provide a broadly-based assessment and whether the study had any ethical content. Typically, such studies are partial and are sometimes exceedingly partial (for example, concerned exclusively with clinical efficacy). It is scarcely surprising, and hardly to be objected to, that a study does not address what it does not seek to address. It is the process of evidence assimilation and interpretation, along with the application of contextual judgments of both fact and value, which constitutes the assessment. An item of evidence, or a specific calculation (as in a specific algorithm), is but one input into the process and may have no ethical content as such or, if it has, may be well-enough comprehended by the cognoscenti for it to need no explicit mention.

Let us distinguish algorithms and processes. Both contain evidence. Both often also contain value judgments about what is good science, good ethics and about what is good for society. An algorithm specifies a series of calculations with given parameters that generates a quantitative result or output. The quality-adjusted life-year (QALY) is an example of an algorithm. HTA is better seen as a process through which evidence is gathered and appraised for quality and relevance, checked against criteria and where some of the parameters and structural characteristics of the calculation may themselves be up for discussion and decision^[4]. Plainly, it is possible to conduct an exercise which might be entitled “HTA” entirely in a research environment and with little or no direct interaction with any actual social decision-maker. This might especially be true if one were engaged in advocacy. But such exercises would lack authority (where would the value and other judgments embodied in the analysis have come from?) and may also lack relevance (how sure can one be that the problem tackled is one that engages the attention of decision-makers?). Such an approach pushes HTA in an algorithmic direction – and may result in the conclusions of the analysis being ignored. DPU would probably agree that HTA is a process and moreover that it is one that is itself nested in a wider political and professional set of processes: their opening paragraph refers to HTA as “one good source of evidence.” Where the edges of the issues and their resolution are assigned – within the HTA process or the broader one – is hardly a fixed line of demarcation. In many cases, the HTA process is embodied in an agency which in turn has a place in a wider decision-making and policy context. I imagine that in most cases the constitution and governance arrangements define the nature of the decisions or recommendations made by the agency and hence their authority and its accountability.

Analysts have two main tasks in relation to processes. One is to design a satisfactory process – for example, one meeting the requirements of “accountability for reasonableness.”⁹ It is not necessary – indeed it is unethical – to prescribe all the specific ethical judgments that may have to be made. That is not the job of analysts but of decision-makers and their advisers. The design job of analysts is to create a framework which ensures that all relevant, quantitatively and ethically significant issues can be considered by decision-makers and their advisers. This may often be usefully accomplished by direct challenges by analysts, enablers and others who organise the process and this, in turn, is a major advantage of a deliberative process over algorithms. The ethical views of analysts as to what is good for society are not, however, of interest. Their proper skill lies in eliciting decision-makers’ values not in asserting their own. Such a process is part of the scoping of an HTA.

The other job of analysts is to populate the HTA process with ideas and evidence. A well-designed process will also surface specific matters of ethics (or epidemiology, or economics) for explicit consideration, whether or not they are submitted in the package of evidence. It is unnecessary for ethical (epidemiological, economic, etc) matters to be raised that embody established principles or whose treatment is stipulated. Decision-makers may have already determined that a particular outcome measure be employed within a specific jurisdiction, or that a particular discount rate be used. There may be ethical considerations involved, and in making such rulings, all relevant ethical matters ought to have been borne in mind, but there is no expectation that they would be constantly revisited - though unusual or innovative ideas and evidence should be specifically addressed and ideally reported. Checks might be made that the assumptions, for example about the outcome measure, are indeed applicable in the case under consideration – a good reason for having patient representatives engaged in the process.

The extent to which DPU’s claimed failure of HTA to address ethical issues is inherent in HTA or the result of poor application or poor reporting is unclear. The apparent failure may simply be based upon a reading of a wrong part of the literature. Moreover, regarding the more granular components of an HTA process, such as the estimation of a price elasticity, there may be no ethical content at all. Most HTA reports may indeed not develop ethical arguments or mention ethical issues but whether this is a “failure” is less clear.

The ethical underpinnings of economic evaluation are readily available in dozens of economics textbooks on welfare economics; they fall into two main categories: welfarist and extra-welfarist,¹⁰ which differ markedly in the treatment and ethical content of both algorithms and processes, but which are hardly trade secrets. It is, or should be, well-known that major value assumptions are built into the algorithm known as the QALY (EQ-5D style and its variations) which include principles for the selection of dimensions, scaling them and weighting them. There is a huge literature covering all these aspects. It is excessive to demand that these elements be routinely exposed in all applications of the algorithm. Some competencies have to be assumed in the readership.

The Reference Case

A “Reference Case”^[5] is a good example of a set of decision principles for a technology assessment. The NICE in England and Wales developed its own Reference Case¹² and in the Methods for Economic Evaluation Project,^13,14 the Bill & Melinda Gates Foundation sponsored the development of a more comprehensive standard-setting Reference Case for HTAs and similar exercises conducted as a part of its programmes. This is being further developed for international use by the International Decision Support Initiative of NICE International and the Health Intervention and Technology Assessment Program (HITAP) in Thailand. Its 11 principles, to which it is hard to do brief justice, all entail explicitly ethical elements: transparency, selection of comparators, use of evidence, outcome measures, cost measures, time horizons, costs and effects outside health, heterogeneity (eg, population subgroups), uncertainty, impact on budgets and other constraints, equity and social justice.

What is of ethical significance is often contextual. For example, the value of the financial protection afforded by provision of a new intervention in societies where healthcare is either free of charge or subject to minor charges with many exemptions for the needy, may not be affected in any way by the introduction of a new technology. Where that is not true (for example, where the new intervention may be accompanied by a new user charge) such a valuation may be judged^[6] to be needed – but technical (and ethical) guidance on how to estimate it is only now being developed. Rarely considered is the financial consequence, and how it is to be reckoned, when interventions are deemed to be cost-ineffective and provided only in the private sector to those willing to pay. Similarly contextual is the “perspective” to be adopted. This will typically be determined by political authorities (whose accountability is to the public^[7]) so the perspective adopted in studies to support public decision-making may, therefore, not be a matter for the discretion of the analysts or researchers.¹⁵ Ethicists may then search in vain for a discussion of the ethics of the perspective.

A Reference Case militates against bad practice; standard-setting and guides to good practice can have impact^[8]. Frequently what is wrong is not the tool but its users or the environment that may constrain their proper use of it. Further solutions might lie in more thorough training in normative economics and related subjects, including moral philosophy. A good training in pharmacy or epidemiology with a few ad hoc courses in welfare economics, HTA and medical ethics will not do.

There is a notable lack of practical ethical tools for use in decision-making contexts.^16,17 The incorporation of health equity into HTA through the development of new algorithms and process design has, however, advanced in recent years^18-24 having been much impeded in the past by a multiplicity of ill-formulated concepts and values and a lack of a widely accepted normative source on which to build controversial choices. Research programmes are fortunately progressing in tackling the substantial theoretical and empirical difficulties in quantifying equity and financial impacts in ways suitable for integration in decision-making.^25-27

Efficiency and Ethics

DPU argue that “[s]ometimes reducing the unfair distribution of health in a population…should be given priority over making a health system more efficient (eg, by pursuing what is most cost-effective to maximize health)” [p1].³ This seems eminently reasonable but is actually problematic on grounds of the implied meaning of “efficiency.” The difficulty is rooted in the idea of “population health.” Somehow the “healths” of all the individuals in question have to be added up to find the total and the shares. But how should this be done? If our measure of health were the QALY, one might simply sum QALYs, thereby implicitly assigning a uniform weight to each. But that would be to build in a very specific distributional value judgment and doing so will almost inevitably result in a clash with ideas of interpersonal equity, most of which entail assigning higher weights to those who are deprived, have shorter life expectancies, or who have lower capacities to benefit from healthcare. It is better to understand efficiency as “being cost-effective”: ie, seeking to achieve any given distribution of outcomes at the lowest opportunity cost (the cost being again an appropriately weighted sum of the health gain lost through the proposed prioritisation). The false trade-off may arise from an inappropriate use of the economists’ concept of “allocative” efficiency, which characterises an alleged optimal allocation of outcomes across individuals according to marginal willingnesses to pay and which locates a specific distribution on the health frontier²⁸: one that is almost certainly inequitable on most criteria of equity. Replacing one inequitable rule for determining the ideal position on the health frontier with an arbitrary other will not do. It is better to stick with the idea of “productive” efficiency (ie, cost-effectiveness) and explicitly lay out the principles for selecting the equitable allocation. In this way, there is no conflict between efficiency and equity, and minds can then be focused on the variety of ethical desiderata that vie for attention but cannot simultaneously be fully achieved, partly because of resource limitations (including limited information) and partly because the ethical desiderata often conflict. The false dichotomy between efficiency and equity is made worse by propositions such as “equity trumps efficiency.”^28-30 In both cases, the contribution of efficiency to human welfare is denigrated.

The Competence of Health Technology Assessment

DPU are concerned that HTA might exceed its competence. An example is given of a situation in which a decision might hinge on non-quantified considerations. The implication is that HTA rests solely on quantified evidence and that it, therefore, needs to be embedded in a “fair, deliberative” process so that the other considerations may be brought to bear. But quantified evidence has never been the “base” on which “evidence-based” decisions are taken that is why the term “evidence-informed” decision-making is much to be preferred. Some of the evidence fed into the process, like cost-effectiveness studies, may be quantitative; other evidence may be qualitative. It will all be of variable quality and relevance, which must be judged. Some critical evidence will be missing. Some will be no more than opinion. The object of the process is to enable consideration of all types and to reach a credible and defensible judgment. DPU underline an important characteristic of a decision-making process, which is “legitimacy.” This requires “accountability for reasonableness” but also more. A good decision process involves its participants in deliberation. New ethical issues immediately arise: who should participate? Does participation also involve decision-making power? Are all participants to have equally influential roles? At what stages of the process do they participate? Do different participants participate at different stages? Do participating patient representatives agree that the QALYs used in the research evidence are adequate indicators of benefit for the purposes in hand? Do physician partners agree that the evidence on efficacy can reasonably be extended to effectiveness? In such ways, the process itself may reveal evidence that was previously lacking. To see HTA as a process for collective thinking, or a complex of processes and subprocesses (like having a selection panel to identify stakeholders) to support the thinking process, greatly enhances accountability, legitimacy, and credibility. The more complete it is the more accountability, legitimacy and credibility are enhanced.

Envoi

Two cheers only for DPU. They get some important things right. They are right to deplore shoddy practical analyses that uncritically embody dubious ethics (whether embodied in efficiency or distributive questions); they are right to advocate deliberative methods of social decision-making; they are right in that there is a way to go before some of the “broader” questions (especially of equity and financial protection) can be routinely and quantitatively embodied in processes of HTA. But their view that HTA reports “do not even mention ethical issues” is naïve: some reports contain no ethics; some are the outcome of many years of ethical enquiry, a litany of which is not needed; some ethical questions are not up for discussion in HTA, being settled by the sponsors of the process; and some are laid on the table by a good Reference Case for explicit discussion. Our focus as academic analysts ought to be on the design of competent processes, the training of competent analysts, the education of competent commissioners and users of HTA, the competent briefing of the public and professional media, and enabling the intelligent interpretation of HTA results by those who implement them – the healthcare professionals – and those on whose behalf HTA is done – the public.

So, should HTA reports really start to provide explicit quantitative information about budget impact, distributional impact, and financial protection impact? It depends on the context and whether these are important considerations for decision-makers. Reports may not need to include information about financial protection impact when such impacts are small, as is usually the case in high income countries with universal health systems and relatively generous systems of social protection. Nor need they tediously rehearse the ethics of HTA when the ethics are already embedded in processes. They probably do more often need to address health distributional issues in all jurisdictions. But one size will not fit all. What works well for the United Kingdom may not be right for South Africa.

Does HTA need “extension”? No – but specific studies (and so their reports) may sometimes need it badly!

Acknowledgments

The helpful comments of two anonymous referees.

Ethical issues

Not applicable.

Competing interests

Author declares that he has no competing interests.

Author’s contribution

AJC is the single author of the paper.

Citation: Culyer AJ. HTA – algorithm or process? Comment on "Expanded HTA: enhancing fairness and legitimacy." Int J Health Policy Manag. 2016;5(8):501–505. doi:10.15171/ijhpm.2016.59