WHO evaluates NICE

The National Institute of Clinical Excellence (NICE) has emerged in the brief period of its existence as an important influence on decision making and allocation of resources by the NHS. The recent release of a report by the World Health Organization on NICE's technology appraisal programme provides an international perspective on NICE's processes and impact.¹

By design, the report has important limitations. NICE commissioned it in response to concerns raised by the 2002 Select Committee Inquiry² regarding the scientific validity of its processes. NICE restricted WHO to consideration of the “methods and scientific robustness” of technology appraisal and resulting guidance. Other important roles NICE has, such as providing clinical guidelines and recommendations on audit methods, are not assessed. Wider questions about NICE's role as a de facto priority setting body for the NHS are acknowledged but not addressed. These include non-trivial issues—what impact does guidance from NICE have in practice on allocation of resources and health outcomes at the local level? Is it resolving postcode rationing or simply diverting attention from continuing important disparities in the availability of effective interventions that have yet to receive the imprimatur of NICE? Since decisions by NICE are made without consideration of a budgetary constraint but its guidance is mandatory for purchasers, do favourable decisions write open cheques for the NHS to honour? If that happens, what is sacrificed at a local level to provide it?

With these caveats in mind, NICE might be forgiven for being somewhat self congratulatory about this report. The review largely affirms NICE as a leading organisation internationally in the use of evidence about clinical and cost effectiveness to inform decisions in the health sector. WHO makes numerous sound recommendations about strengthening NICE but on the whole the report is a ringing endorsement of it.

The report's 28 recommendations are largely directed at process. They cover issues as diverse as the composition and reimbursement of committee members, contractual issues regarding the academic institutions that provide analytical input, the timing and nature of the clinical, cost effectiveness, and other evidence it considers, and the structure of its final reports. The report does not engage in specific methodological debates about the conduct of economic evaluations, but does recommend that there be more consistency both in the methods used in technical appraisal and in the way that results and decisions are reported. In an important bow to the challenges at the level of primary care trusts, WHO's report suggests that models be developed by NICE that will provide a resource for local authorities seeking to understand the implications for the budget of specific guidance.

Both WHO and the select committee voice concern about the conflict between transparency in decision making by NICE and its use of manufacturers' commercial evidence in confidence. The only problem with disallowing such evidence is that it might not then be made available; this is unlikely because manufacturers will continue to provide any evidence that is favourable to their claims. Moreover, the report points out and questions the duplication of effort seen in instances where technology assessments are produced by both manufacturers and academic centres. NICE does not accept manufacturers' evidence uncritically, recognising that it incorporates a marketing opportunity for the manufacturer and that its value is largely in offering viewpoints and evidence that the appraisal team may have missed. The report says that NICE should consider only one analysis, based on independent, industry, and other stakeholders' contributions, but this raises the question of who should pay for this—should the public pay most of the cost of acquiring the information that is used to sell them a product?

Like the select committee WHO notes the need to be more explicit about and to provide a proper justification for the “threshold” ratio for cost effectiveness it uses to make judgments about what is and is not acceptable value for money. NICE remains publicly coy on this issue—everyone WHO consulted, except Michael Rawlins the chair of NICE, was clear that NICE has a threshold and that is £30 000 per quality adjusted life year (QALY) gained. Our analysis shows a much more complex picture.³ Although NICE's commissioning of innovative research on this and other key issues such as the trade-offs between equity and efficiency signals its intent to provide a firm foundation for these aspects of its decisions in the future, we missed reading more tangible guidance as to the process by which such thresholds might be brought into question or determined.

WHO advises that NICE articulate the ethical and social value judgments that the appraisal committee uses. Surprisingly little mention is made in this context of the Citizen's Council, which was appointed at the end of 2002 and charged with helping in the development of social value judgments that should underpin NICE's guidance to the NHS.

On the whole WHO raises little that is new. One intriguing exception is the comparison of recommendations on three technologies by NICE and its counterparts in Canada, Australia, the Netherlands, and Italy. WHO says, “NICE recommendations seemed less conservative than those made by other authorities,” by which it means more permissive. WHO goes further noting that shifts occur from what looks likely to be a no at the appraisal consultation document stage, to a yes in the guidance. Although openness to further evidence and wider opinions is a positive feature of the appraisal process, others have pointed out the potential difficulties involved when patient advocacy groups or manufacturing interests appear to provide undue influence.⁴,⁵ If NICE is systematically softer in its guidance than other health technology assessment agencies this reinforces the need to be cautious about the way in which stakeholders are involved.

NICE can be congratulated not only for its achievements but also for its willingness to undergo publicly internal and external scrutiny. Important questions remain about the practical value and implementation of guidance from NICE—we look forward to further efforts that address these.