Returns on Research Funded Under the NIHR Health Technology Assessment (HTA) Programme

Abstract

The Department of Health (England) requested that RAND Europe conduct an economic analysis of the impact of the HTA Programme. This article describes the results of that work, which consisted of analysis of the potential economic benefits of a sample of HTA funded studies and comparison to programme costs, supplemented by a set of short case studies exploring the impacts of the HTA Programme on policy and practice. Based on our analysis, if 12 per cent of the potential net benefit of implementing the findings of this sample of 10 studies for one year was realised, it would cover the cost of the HTA Programme from 1993 to 2012. Drawing on the case studies and the economic analysis, we have also made a number of observations that could help ensure that the HTA Programme maximises the likelihood of findings being adopted.

The NIHR's Health Technology Assessment (HTA) Programme was established in 1993 and is the largest dedicated research programme for the National Health Service (NHS). The Health Technology Assessment (HTA) programme funds 'independent research about the effectiveness, costs and broader impact of healthcare treatments and tests for those who plan, provide or receive care in the NHS (NHS, 2016). The purpose of the programme is to ensure that high quality research evidence is made available on the effectiveness, costs and impact of health technologies to policymakers, practitioners and patients in a timely and efficient manner. The work of the HTA Programme covers both primary research and evidence synthesis. The research is either commissioned or researcher-led. The underlying principle of the HTA Programme is that clinical research should not only use the most rigorous techniques, but should be needs-led, with a clear benefit to patients and practitioners.

The Department of Health (DH) wants to know the benefits that would have been available to the health system, and the wider community, if the findings of HTA studies had been implemented. Because the implementation of these findings is outside of the scope of the HTA programme, considering the extent of adoption is not included in the economic analysis conducted here. Instead, we focus on measuring what the potential benefits of adoption could be. HTA-funded research can deliver economic benefits in a range of ways. In the economic analysis conducted here, we focus on the benefits of showing a new intervention should be implemented. In this context, benefit can be delivered in two ways:

• By demonstrating a new intervention improves health outcomes, measured in terms of quality-adjusted life years (QALYs), relative to the existing standard of care.

• By demonstrating that a new intervention offers the same health outcomes as the existing standard of care but at a lower cost.

In this study, these benefits are identified and monetised for a sample of HTA studies, and compared to the cost of the entire HTA Programme. The economic analysis estimates the direct benefits assuming all of the new interventions supported by HTA research evidence had been fully adopted, balanced against their costs. This assumption is appropriate because the role of the HTA is to provide relevant and reliable evidence for the NHS; it is beyond the remit of the HTA to ensure this evidence is used.

To make a broader assessment of the role of the HTA in entire health system it is important to consider what is known about the uptake of evidence. Some studies on the impact of HTA appraisals, such as NICE technology appraisals, on practice suggest significant impact of HTA appraisals and others report little impact. Studies on the implementation of NICE guidance show that the extent of implementation varies by location and the type of technology assessed (Garrido et al, 2008, Ch6; Drummond, 2006). According to Drummond (2006), no health system is particularly good at the implementation of the findings from HTA appraisals.

The economic analysis in this article focuses on the benefits from adopting new interventions—there are also other ways that HTA research can provide benefits to the NHS, such as showing that current interventions are not cost effective. To assess the wider range of benefits that can be provided by HTA research, we have also conducted ten case studies. The case studies also examine the extent to which the HTA research had an impact on policy and practice.

Economic Analysis

We suspected that a few studies would have provided most of the potential benefit from the HTA programme, therefore we selected HTA studies that were judged likely to have large potential benefits to give us a baseline estimate for the impact of the programme. Studies with likely high potential benefit were identified in two ways: suggestions from NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC), and scanning of the two most recent volumes of the HTA journal. Studies were screened to ensure they met the following criteria:

• The study shows that the intervention trialled is either cost-effective or cost-saving compared to the standard of care.

• The study reports health outcomes in QALYs.

• The study shows that the probability that the intervention is either cost-effective or cost-saving is greater than 60 per cent at the £20,000 threshold for QALY value.

We identified 27 possible studies, 10 of which met the inclusion criteria. To calculate the potential benefit of the studies we assumed the recommendations were fully implemented for one year (which, were interventions are implemented, is likely to be an underestimate).

We estimate the 10 studies analysed provided a potential net-benefit of £3.0 billion based on a value of £20,000 per QALY, and £5.0bn based on a value of £30,000 per QALY. According to NETSCC, the total research cost of the HTA Programme since 1993 was £317m, with the estimated overall cost of the HTA Programme £367m. The estimated overall cost of the HTA Programme includes the cost of NHS support for HTA research.

We therefore conclude that 12 per cent of the calculated potential net benefit would cover the total cost of the HTA Programme from 1993 to 2012.

To carry out this type of analysis it is necessary to make various assumptions. The key assumptions in this study are:

• That study findings are fully implemented in the NHS, and would not have been implemented without the HTA research (as discussed above);

• There are no other benefits (or dis-benefits) resulting from other HTA studies. This is likely to underestimate the benefits of HTA research given the small size of the sample (10/743), assuming HTA studies do no harm. In particular, thus excludes any studies which provide evidence for stopping ineffective treatment or treatment that is not cost effective;

• That each treatment, where implemented, is implemented for one year before it is superseded. This is likely to underestimate the impact of the programme;

• That the differences in approach to economic analysis in each study do not affect the overall estimate of impact, and that the base case scenarios from each study are good estimates;

• That results from the studies can be replicated in the general population;

• That prevalence and size of population affected has been consistent and that sources used to estimate number of cases are appropriate;

• That cost estimates provided by NETSCC are accurate;

• That the NHS budget is increased to allow for the increased costs of new interventions introduced. The analysis doesn't allow for “opportunity cost” of other treatments displaced.

It is interesting to note that our conclusion is consistent with three studies from Canada that examine the benefits of HTA research (Jacob and Battista, 1993; Jacob and McGregor, 1997; McGregor, 2006). In these three studies, the authors model hypothetical cost reductions resulting from HTA reports. Jacob and Battista (1993) found that the cost of producing HTA reports was 7% of the projected savings from implementations of the reports' recommendations. Jacob and McGregor (1997) found that implementations of 21 HTA reports' recommendations would result in cost savings of 16–27 million Canadian dollars and McGregor (2006) found the impact of the implementation of 18 HTA studies would result in cost savings of 3.1 million Canadian dollars annually.

Case Studies

In addition to the economic analysis, ten case studies were conducted to explore the wider range of benefits that could arise from HTA research and examine some of the issues around implementation of findings. They were based on document review and interview and selected pragmatically to cover HTA studies with a high potential for impact that could illustrate and explore a range of types of potential benefits of the HTA Programme. These case studies covered four of the projects included in the economic analysis, and six other HTA projects. Across the ten case studies:

• Three had a clear impact on policy through citation on guidance, with another expected to be included in guidance that is forthcoming.

• Three showed a clear impact on practice.

• A further three cases showed some evidence of changes in practice but attributing that to the specific study is more challenging, though it is likely the study played a role.

Overall, from the ten studies, four had a clear impact on either policy or practice or both. A further five showed some evidence of impact on either policy, practice or both. In these cases, either the impact had happened but it was less clearly linked to the research, or the impact was forthcoming (e.g. the research was expected to be included in forthcoming guidance) There was also evidence of wider impacts, such as developing the credibility of research in a particular field, or building capacity to transfer policy into practice.

We identify a number of further interesting observations from the case study set. A key observation is that it can be difficult to directly link changes in policy and practice to a single HTA-funded study (or, indeed, to any individual piece of research). Wider factors as well as other pieces of research may play an important role in both enabling and blocking implementation. This suggests that holding researchers or even the research programme as a whole, directly to account for whether particular research findings are taken up by the healthcare system is not a fair reflection of the wider environment.

This is illustrated by the role that perceptions of clinicians play in implementation. A number of the studies investigated suggested that a procedure should be removed from practice, as it costs money and does not provide any significant benefit. Here, in theory, putting the findings into practice should be straightforward since there is no “new” treatment to introduce. However, it is challenging to overcome the existing views and habits of practitioners. In these situations, clinicians are likely used to using these interventions, and may be convinced of their effectiveness based on personal experience. They may also face pressure from industry or from patients who want to see a particular treatment used. This highlights that clinical and cost effectiveness are not always the sole driving factors in whether a new treatment is recommended or adopted—even in NICE guidance, as illustrated in at least one case study. Wider factors, including patient preference and experience, can also be important.

Conclusions

Based on this analysis, if 12 per cent of the potential net benefit of implementing the findings of this sample of 10 studies for one year was realised, it would cover the cost of the HTA Programme from 1993 to 2012. Drawing on the case studies and the economic analysis, we have made a number of observations that could help ensure that the HTA Programme maximises the likelihood of findings being adopted:

• Consider the full range of costs of implementation. Conducting economic analysis to meet NICE requirements means that some types of costs are not included in most HTA studies. For example, training costs related to introducing a new intervention would not be included. Addressing these potential barriers to uptake in HTA studies could help to support implementation.

• Consider the importance of timing relative to the revision of relevant guidelines. There were examples from the case studies where guidance was published shortly before the HTA work was published. Communication with NICE and other guideline producers could improve timing and allow for the sharing of early results where appropriate.

• Continue to support things other programmes are less willing to support. Examples include long-term follow up of cohorts and meta-analyses of data across trials, both of which could support impact on policy and practice.

• Continue to identify unproven practice in use in the NHS as well as new interventions. There are practices used in the NHS that are not evidence-based, and where these are ineffective, removing them could lead to cost savings. Four of the case studies illustrated how HTA research can provide the evidence about such practices.

• Continue to require systematic reviews before primary research is commissioned. Ensuring that primary research is only conducted where there is a gap in the evidence and that it is appropriately powered to draw the necessary conclusions, based on a review of previous evidence, will help ensure the findings of HTA-funded studies are relevant to policy and practice.

• Improve meta-data on studies. The HTA has an ongoing project to develop meta-data across its clinical trials (NETSCC, HTA—08/117/01). Gathering meta-data of this type across the HTA portfolio would enable the results of relevant HTA studies to be identified, classified and used more readily. This could promote both better analysis of the HTA Programme, and better use of its study findings.

• Ensure consistency of economic analyses. Several case studies showed the importance of economic analyses to the overall impact of the study. The type and quality of economic analysis included in the HTA studies differs, however, it has become more standardised over time, partly due to changing NICE requirements.