Using HTA and guideline development as a tool for research priority setting the NICE way: reducing research waste by identifying the right research to fund.

Abstract

Background

The National Institute for Health and Care Excellence (NICE) was established in 1999 and provides national guidance and advice to improve health and social care. Several steps in the research cycle have been identified that can support the reduction of waste that occurs in biomedical research. The first step in the process is ensuring appropriate research priority setting occurs so only the questions that are needed to fill existing gaps in the evidence are funded. This paper summarises the research priority setting processes at NICE.

Methods

NICE uses its guidance production processes to identify and prioritise research questions through systematic reviews, economic analyses and stakeholder consultations and then highlights those priorities by engagement with the research community. NICE also highlights its methodological areas for research to ensure the appropriate development and growth of the evidence landscape.

Results

NICE has prioritised research questions through its guidance production and methodological work and has successfully had several research products funded through the National Institute for Health Research and Medical Research Council. This paper summarises those activities and results.

Conclusions

This activity of NICE therefore reduces research waste by ensuring that the research it recommends has been systematically prioritised through evidence reviews and stakeholder input.

Strengths and limitations of this study.

• Highlights the importance of reducing research waste and the prioritisation of research as a key step in that process.

• Describes the systematic prioritisation of research done by National Institute for Health and Care Excellence (NICE).

• Illustrates the methodological research undertaken from prioritised topics by NICE in recent years.

• This paper is limited to the prioritisation of research undertaken at NICE and it does not reflect all the work the nstitute does to reduce research waste.

Introduction

There is no doubt that the substantial public and private investment in biomedical research (estimated to be around US$240 billion in 2010)¹ has led to significant improvements to the quality of life of people. However, not all research yields such benefits, and though some waste is unavoidable due to the nature of science, many improvements to the way we conduct our work can be made to ensure that our investments are sound and the value of our research is increased.^{1 2} Getting the first step of the research cycle, that is, appropriate priority setting of research or identifying the correct research question to fund, is key to this being successful.¹

Previous research has shown that there is a huge mismatch between the research that is funded and the research needed by the end users. A study found that only 9 out of 334 research articles highlighted priorities relevant to patients or clinicians.² A second study found that while the majority of trials funded were related to drugs, the end users were much more interested (80%) in research about ‘education and training, service delivery, psychological interventions, physical interventions, exercise, complementary interventions, diet etc’, compared with research in drugs (20%).¹

Another problem noted is that often the new research ignores what is already known about the problem and does not take into account previous research and therefore can duplicate information and give little additional value.^{1 2} Moreover, such research may miss the true knowledge gaps that need answering, and this can often be compounded by the failure to publish negative research (publication bias) that can go undetected unless methodology such as funnel plots are used. Even if an appropriate research gap is identified, inappropriate design or statistical analyses or the problem of ‘effect to bias ratio’ (where the magnitude of the effect size and biases like selection or confounding, are similar) may mean that the validity of any results may be questionable.^{3 4}

The National Institute for Health and Care Excellence (NICE) is an independent organisation responsible for providing evidence-informed guidance on health and social care for England.^{5 6} NICE guidelines (clinical, public health and social care), quality standards, Health Technology Assessments (HTA) and other products help to deliver the best possible care within the limited resources available.^{5 6} The Science Policy and Research (SP&R) programme⁷ works together with various academic, charity and research funding organisations to enable the appropriate research to be done. It is now also working with life sciences companies, using European Commission funding through the Innovative Medicines Initiative (IMI) to take this work forward. In this paper, we summarise these processes and the work that has been undertaken through these mechanisms.

Research priorities identified through systematic reviews (guideline or HTA production)

NICE guidance is based on the synthesis of evidence primarily through the process of systematic reviewing and, if appropriate, health economic modelling and cost-effectiveness decision analysis that are presented using Grading of Recommendations Assessment, Development and Evaluation (GRADE) profiles that allow for a transparent representation of the confidence in the evidence available for decision making.⁸ The results of this work are discussed by independent advisory committees (consisting of members external to NICE), which can include a diversity of National Health Service staff, healthcare professionals, social care practitioners, commissioners and providers of care, patients, service users and carers, industry and academics. The committees, through a process of guided deliberation of the evidence, reach conclusions by formally making draft recommendations. During a period of public consultation, external stakeholders can comment on draft recommendations before they are finalised and eventually published.⁹ The evidence base supporting guidance recommendations are reviewed every few years to ensure that any results potentially demonstrating a change in practice or care are formally considered for updating recommendations.

This process explicitly describes the evidence base and identifies where there are gaps, uncertainties or conflicts, in the existing evidence. Many uncertainties identified may be interesting to answer but could have little impact on people’s care or NICE’s ability to make appropriate evidence-based and evidence-informed recommendations. Uncertainties can arise because there is no evidence available or due to the absence of relevant research or if research has been done but not yet published. Or there may be evidence available but there is still insufficient information on which to base a recommendation (eg, due to inadequate reporting or poor quality), or the research is out of date (eg, a systematic review that needs updating with recent trials or if clinical practice has changed). However, if these uncertainties could have an impact, and support future NICE guidance recommendations, it is important for the institute to make recommendations for research (see table 1).

Table 1.

Example format for research recommendation rationale to support prioritisation

Potential criterion	Explanation
Importance to patients, service users or the population	What would be the impact of any new or altered guidance on the population (eg, acceptability to patients or service users, quality of life, morbidity or disease prevalence, severity of disease or mortality)?
Relevance to NICE guidance	How would the answer to this question change future NICE guidance (ie, generate new knowledge or evidence)? How important is the question to the overall guidance? High: the research is essential to inform future updates of key recommendations in the guidance. Medium: the research is relevant to the recommendations in the guidance, but the research recommendations are not essential to future updates. Low: the research is of interest and will fill existing evidence gaps.
Relevance to the NHS, public health, social care and voluntary sectors	What would be the impact on the NHS, public health, social care and voluntary sector and (if relevant) the public sector of any new or altered guidance (eg, financial advantage, or effect on staff, strategic planning or service delivery)?
National priorities	Is the question relevant to a national priority area (such as a national policy or parliamentary paper)? If so, specify the document.
Current evidence base	What are the problems with the current evidence base? (That is, why is further research needed?) Is there any relevant ongoing research that may resolve the uncertainty?
Equality	Does the research recommendation have any relevance to equality? For example, does it focus on groups needing special consideration, or on a technology, intervention or service that is not available for use by people with certain disabilities? What is known about the impact of the intervention on the health gradient?
Feasibility	Can the proposed research be carried out within a realistic timescale? Would the sample size needed to resolve the question be feasible? Would the expense needed to resolve the question be warranted? Are there any ethical or technical issues?
Other comments	Any other important issues that should be mentioned, such as potential funders, outcomes of previous attempts to address this issue or methodological problems.

NHS, National Health Service; NICE, National Institute for Health and Care Excellence.

These recommendations are deliberated, discussed, agreed and formulated by the independent advisory committees and the external community (both specialist and public) that are then invited to comment and contribute.⁹ The final recommendations are identifiable in the guidance and are also collated into the publicly available NICE research recommendations database.¹⁰ NICE then liaises with the research community to ensure they are addressed as detailed in the paragraphs that follow. By making research recommendations, NICE is looking to steer the research community for them to generate new evidence to inform the future guidance update and review cycle. If gaps remain in the evidence base after several years, the committee may consider keeping, removing or archiving the research recommendations based on any new findings. In 2010 (updated in 2015), NICE established a formal guide to support this process and to provide a route to ensuring they are picked up as key topics for research funding.¹¹ The guide describes a step-by-step approach to identifying uncertainties, formulating research recommendations and research questions, prioritising them and communicating them to researchers and research funders (see figure 1).

Figure 1.

The NICE research recommendations process. NETSCC, NIHR Evaluation, Trials and Studies Coordinating Centre; NICE, National Institute for Health and Care Excellence; NIHR, National Institute for Health Research.

NICE works closely with National Institute for Health Research (NIHR; a government body that supports health and care research of national research priorities)¹² to prioritise, promote and commission its research recommendations. The identified topics are considered by a number of their different research programmes, particularly the HTA, Public Health Research and the Health Services & Delivery Research programmes. If they are found to be suitable and fulfil their criteria, they enter the commissioning process. This process has evolved over the last 10 years and since 2005, over £59 million worth of these research projects have been funded (see table 2). In 2015, NICE and NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) agreed a fast-tracked route for the most important research recommendations, to be flagged under the ‘NICE Key Priority’ designation. These research recommendations have the potential to have the highest impact on future guidance recommendations and people’s care.

Table 2.

Summary of the cost of National Institute of Health Research (NIHR)-funded projects (as of 31 March 2016)

	NICE priority topics		NICE database topics		Total
	No of NETS projects	Funded cost (£)	No of NETS projects	Funded cost (£)	No of NETS projects	Funded cost (£)
HTA programme	24	£24 369 414	22 (I TAR)	£15 805 225	46	£40 174 639
PHR programme	4	£1 764 186	26	£16 448 522	30	£18 212 708
HS&DR programme	NA	NA	2	£738 188	2	£738 188
Total	28	£26 133 600	50	£32 991 935	78	£59 125 535

Source: NIHR Evaluations, Trials and Studies (NETS) Coordinating Centre (NETSCC).

HS&DR, Health Services & Delivery Research; HTA, Health Technology Assessment; NA, not applicable; NICE, National Institute for Health and Care Excellence; PHR, Public Health Research.

Methodological research priorities: ‘research on research’

The Edinburgh Reducing Waste conference in 2015 highlighted the importance of being efficient in the ‘what’ but also the ‘how’ research is done. While the basic methodology of systematic reviews and cost-effectiveness analyses may be well established, there is still controversy about ‘best practice’ and challenges frequently arise in the context of specific technologies, particularly when decisions are negative. It is therefore vital that methodological research is undertaken. Inadequacies in the evidence base have required the exploration of newer methods such as indirect comparisons and the use of network meta-analyses that allow for comparisons across different drugs and interventions where those trials have not been undertaken.¹³ Different perspectives exist about what data constitute as evidence and about what should be used for decision making. Due to the paucity of the evidence base for some interventions, particularly in the public health and social care domains, NICE has always considered the ‘best available’ evidence to inform its decisions, which move beyond randomised controlled trials, using GRADE to determine the validity of the collective evidence.^{8 9 13} In order to ensure that the evidence NICE receives is fit for decision making, each of NICE’s guidance programmes have a methods guide that are regularly updated to incorporate new developments. NICE Scientific Advice works directly with commercial evidence producers on a fee-for-service basis to ensure the appropriate evidence is available for decision making.

In 2008, NICE had its first comprehensive review of the methodological research needed for future development of the work of the institute (methodological priorities for research), funded by the Medical Research Council (MRC) and NIHR joint Methodology Research Programme (MRP).¹⁴ The methodological review identified and prioritised methods research topics through a focused literature review, interviews, an email survey, a workshop and web-based feedback exercise. Participants were members of the NICE secretariat and its advisory bodies, representatives from academia, industry and other organisations working closely with NICE. The first prioritised project the MRP funded in 2008 was on the appropriate use of ‘only in research’ decision¹⁵ and a further nine projects were funded through another MRP call via their needs-led route in 2009 (over £2.3 million, see table 3).

Table 3.

Methodological research projects funded through Methodology Research Programme (MRP) in 2009 (approximately £2.3 million)

Research project	Principal investigator	Primary research institute	Project duration (months)	Value funded
Widening the spectrum of health outcomes used in Health Technology Assessment: integrated synthesis and mapping to QALYs	Professor A E Ades	University of Bristol	24	238 868
Use of generic and condition-specific measures in NICE decision making	Dr L Longworth	University of Sheffield	24	289 189
Preparatory study for the re-evaluation of the EQ-5D tariff	Dr A Tsuchiya	University of Sheffield	18	242 969
Economic modelling of diagnostic/treatment pathways in NICE clinical guidelines	Dr J Lord	Brunel University	24	284 471
Properties of statistical methods for indirect and mixed treatment comparison – a computer simulation evaluation	Dr F Song	University of East Anglia	12	96 810
Methodological search filter performance: assessment to improve efficiency of evidence information retrieval.	Ms C Lefebvre	NHS R&D Programme	24	95 833
Methods to estimate the NICE cost-effectiveness threshold	Professor M Sculpher	University of York	24	351 357
Methods for the Indirect estimation of health state utilities	Professor C McCabe	University of Leeds	24	194 949
Methods for strengthening evaluation and implementation: specifying components of behaviour change interventions	Professor S Michie	University College London	36	509 200

EQ-5D, EuroQol 5 dimensions; NICE, National Institute for Health and Care Excellence; QALYs, quality-adjusted life-years.

To ensure that NICE had robust processes for timely identification, prioritisation and communication of methodological research needs, an Internal Research Advisory Group (IRAG; formerly the internal methods advisory group) was established in 2010. This group was tasked to develop the institute’s methodological development needs in both the short term and long term by continuing the systematic identification and prioritisation of key research uncertainties on an ongoing basis and working with MRP to commission research.¹² The group consisted of representatives of the different guidance and non-guidance producing centres and programmes across NICE, who were supported as and when needed by external research advisors from academia who were methodological experts in the fields relevant to the work of NICE. Each programme provided their methodological research priorities, and 45 uncertainties were collated from across the Institute. These were clarified and reprioritised into themes that included modelling service delivery, using qualitative evidence to capture patients experience, extrapolating data for comorbid and paediatric populations and using observational data from large datasets for decision making. These were then highlighted by MRC through a NICE-specific funding call (see figure 2).

Figure 2.

Process of identifying methodological research priorities at NICE and relationship with Medical Research Council (MRC): example 2010–2012. NICE, National Institute for Health and Care Excellence.

Some of the prioritised topics did not receive any bids and other internal support mechanisms were used to fund them as small scoping projects. This led to more clarified research questions, of which one was undertaken internally by the SP&R programme at NICE on the use of colloquial evidence¹³ and the other to support the use of observational data in healthcare decision making was put forward as a highlight notice by the MRP in 2017.¹⁶ They also funded two more NICE priorities on developing a reference protocol for expert elicitation in healthcare decision making in 2016¹⁷ and going beyond health-related quality of life towards a broader quality-adjusted life-years measure for use across sectors in 2017.¹⁸

The SP&R programme has also participated in a project considering the role of social values in priority setting as part of an international collaboration to try and unpick how cultural contexts of well-being interacted with healthcare decision making.^{19 20} Additionally, the programme supported the establishment of a Research Support Unit in 2013 to undertake short projects identified through internal prioritisation by IRAG on methodological areas that included reviewing the literature on social value judgements relevant for NICE and identifying and appraising promising sources of UK clinical, public health and social care real-world data.²¹ Other arrangements for NICE to access research commissioning include the Decision Support Unit, the Technical Support Unit and the External Assessment Centres that facilitate methodological research for HTAs, clinical guidelines and medical technologies evaluation work, respectively. NICE has also recently partnered with Myeloma UK to explore how patient preferences could be captured and included in HTAs more readily.

NICE has also been looking at internationally important and relevant research priorities that will potentially have a global impact on health and care decision making. It has been successful in gaining international cooperation and European Commission funding through the IMI for some of the key priorities currently facing the field. These include the use of real-world data for early decision making (GetReal),²² the medicines adaptive pathways to patients activities to foster access to beneficial treatments for the right patient groups at the earliest appropriate time in the product life-span in a sustainable fashion (ADAPT-SMART)²³ and using big data for obtaining better outcomes for patients (BD4BO).²⁴ The Institute has also been actively involved with other European Commission funded projects such as EUnetHTA (an effective and sustainable network for HTA across Europe)²⁵ and DECIDE (patient and public focused strategies for communicating evidence-based recommendations).²⁶

In 2017 NICE published nine areas that it had, through the SP&R programme and the IRAG, identified as priorities for methodological research. These are: (1) real-world evidence; (2) data science; (3) adaptive pathways; (4) patient preferences; (5) improvements in cross-sector comparisons; (6) expert elicitation; (7) complex data visualisation; (8) precision medicine; and (9) implementation of NICE guidance. Methodological research in these areas will help NICE assess the need for improvement in the methods and processes it uses to produce guidance and anticipate and adapt to policy developments and changes in health and social care delivery in the next few years. By only highlighting methodological research areas to funders that were systematically prioritised, NICE ensures that only questions whose answers will impact its future work are studied, therefore doing its part to reduce research waste.¹

Discussion

Priority setting is an explicit method that considers what to fund by weighing the trade-offs between the various options in the process.²⁷ It is imperative to avoid research waste such that only key gaps in knowledge are fulfilled by undertaking new research that help build a more complete evidence landscape for future policy development and better clinical practice. Systematic approaches to research priority setting improve the transparency of research management and are often based on a ‘research cycle’ approach that includes the identification of research questions, ranking their priority, identifying existing research, and setting priorities for primary research, and involving relevant stakeholders at key points throughout the cycle.²⁸ It is therefore essential that a formal process of prioritisation of research be established within guidance-producing organisations, with the involvement of all legitimate stakeholders, to increase the ownership of the ensuing research and the likelihood of the results influencing practice and policy.²⁹

Creating research recommendations is part of the guidance production cycle, and they should:

• identify any uncertainties that may affect people’s care

• be developed using an appropriate technique to frame research question development, for example population, intervention, comparator and outcome (PICO) or evidence, population, intervention, comparator, outcome and time (EPICOT).

• go through a process of stakeholder consultation and be reviewed as part of the guidance review and update cycle.

This is the path that NICE has taken and therefore has a strong emphasis on reducing research waste. The NICE research recommendations are derived from the uncertainties identified through the NICE guidance development process and are an integral part of the guidance development and review cycles. The institute therefore reduces research waste by supporting the redirection of resources to more valuable activities.³⁰ By making its research priorities and funded research publicly available, it minimises the duplication of research. It also drafts research questions with appropriate detail (eg, type of study design and sample size required), such that the answers obtained from its commissioned research are meaningful. Publicly funded guidance development bodies have a responsibility to ensure that they are guiding appropriate future research based on identified need and gaps in the evidence base. This is also true for research funders, who also have the same responsibility. NIHR has shown that only 5 (11%) of the 47 trials funded over the 2-year period (2006–2008) were not based on identified gaps from systematic reviews, and they had valid reasons for doing so.³¹

Methodological uncertainties derived from NICE also go through a process of systematic identification and prioritisation and are promoted to funders to encourage research being funded into the various aspects of how to undertake the development of future guidance. Though this paper is limited to describing the prioritisation of research undertaken at NICE, we feel it is an important part of the role of any HTA or guideline-producing agency globally to undertake in order to reduce research waste.

In recent years, NICE has entered into a few international research collaborations. These are becoming increasingly important as sources of research funding, particularly within the current economic climate, are dwindling. Such partnerships support the identification and prioritisation of cross-cutting research needs and also potential joint funding routes. There should therefore be a shared responsibility between research partners to ensure that key research is undertaken and cooperation to allow sharing of information to avoid duplication and improve efficiency. NICE has partnered with The National Center for Biotechnology Information so that there will be a PubMed bookshelf dedicated to methods research, and the methodological research reports undertaken by organisations like NICE would also be indexed and accessible in the future.

A number of methods to identify priority areas for research have emerged. These include measuring the burden of disease or the expected return from research as well as estimates of the welfare losses resulting from variations in clinical practice.³² The value of information analysis is a more novel approach to prioritising research uncertainties that quantifies the expected net benefit from the results of the additional research to society, against the cost of conducting that piece of research and its implementation. Through this framework, the value of acquiring additional information to inform the decision problem helps alleviate some of the uncertainty had less definitive evidence been used instead.^{33 34} This method has a firm foundation in statistical decision theory and has been successfully used in other areas of research.³² It has also been employed for developing research recommendations from a number of NICE guidance but not yet been undertaken routinely.

Other successful initiatives to identify, prioritise and promote research uncertainties include the James Lind Alliance (now part of NIHR and carried out via NETSCC) Priority Setting Partnerships (PSPs). These PSPs bring together patients, carers and clinicians using transparent methods to prioritise ‘known unknowns’ that have been elicited primarily from evidence-based knowledge.³⁵ Until January 2016, these research priorities were collated and published in the UK Database of Uncertainties about the Effects of Treatments. This was launched in 2006 to collate uncertainties from reports of systematic reviews and clinical guidelines, protocols for systematic reviews (such as those published in the Cochrane Database of Systematic Reviews) or from registered information about ongoing clinical trials. Some of the treatment uncertainties came directly from patients or carers or from clinicians and cover a wide variety of health problems, for example, cancer, mental health and skin disorders.³⁵ Another good example can be seen from the field of international development where the use of evidence gap maps have been developed by the International Initiative for Impact Evaluation. Evidence gap maps are evidence collections that map out existing and ongoing systematic reviews or primary studies in a sector or subsector, such as maternal health, HIV/AIDS and agriculture. They present a visual overview of existing evidence and therefore highlight the areas of gaps within the evidence landscape.³⁶

Conclusions

It is important that HTA and guideline organisations use their systematic processes to identify research gaps and then subsequently link with national research funders to ensure they are addressed. NICE therefore supports the reducing research waste campaign by ensuring that the research it recommends has a beneficial impact on the health and care of the people, as it has been systematically identified as a genuine gap in the health evidence or a method that needs further clarification or development.