Establishing local priorities for a health research agenda

Abstract

To describe the two stage prioritisation process being used by the UK National Institute for Health Research’s Collaboration for Leadership in Applied Health Research and Care for the South West Peninsula (or PenCLAHRC) – a joint health service and university partnership and reflect on implications for the wider context of priority setting in health care research.

PenCLAHRC’s process establishes the priorities of stakeholders including service users across a regional health system for locally relevant health services research and implementation. Health research questions are collected from clinicians, academics and service users in Devon and Cornwall (UK) using a web-based question formulation tool. There is a two stage prioritisation process which uses explicit criteria and a wide stakeholder group including service users to identify important research questions relevant to the south west peninsula locality. To date a wide variety of health research topics have been prioritised by the PenCLAHRC stakeholders. The research agenda reflects the interests of academics, clinicians and service users in the local area. Potential challenges to implementation of the process include time constraints, variable quality of questions (including the language of research) and initiating and maintaining engagement in the process.

Shared prioritisation of local health research needs can be achieved between stakeholders from a wide range of perspectives. The processes developed have been successful and, with minor changes, will continue to be used during subsequent rounds of prioritisation. Engagement of stakeholders in establishing a research agenda encourages the most relevant health questions to be asked and may improve implementation of research findings and take up by service users.

Context

PenCLAHRC (the National Institute for Health Research’s Collaboration for Leadership in Applied Health Research and Care for the South West Peninsula (NIHR CLAHRC SWP) was established as a pilot on 1^st October 2008 alongside eight other collaborations across the UK¹(1). CLAHRCs are partnerships between higher education institutions and National Health Service (NHS) organisations which aim to improve patient outcomes through the conduct and application of applied health research. They were established by England’s National Institute for Health Research (NIHR) in response to Sir David Cooksey's Review of Health Research (2) and the report of the Chief Medical Officer 's High Level Group on Clinical Effectiveness (3). The former highlighted the “second gap in translation” between research outputs and clinical practice. The latter suggested that the NHS should better utilise the capacity of higher education to support initiatives to enhance clinical effectiveness by developing new models of partnership to encourage relevant research and greater openness to change (4). The vision for CLAHRCs is to make the NHS more sensitive to the findings of research and make academia more responsive to the needs of the health service.

PenCLAHRC’s overall objectives are driven by the principles of Evidence Based Practice (EBP) (5). These are to support the identification of research questions that address clinical concerns; to support and undertake research that tests treatments, diagnostics and new ways of working in specific clinical areas; and where effective interventions are identified, to support research into how NHS staff can incorporate them into routine practice. By doing this PenCLAHRC hopes to enable patients across the local community and beyond to receive a better standard of healthcare (http://clahrc-peninsula.nihr.ac.uk) (6).

PenCLAHRC’s distinctive features (in comparison to other CLAHRCs) are its belief in a more dynamic model of research prioritisation and conduct (rather than linear – see Figure 1), its engagement with services users (7) and its capacity building to enhance the involvement of all stakeholders across the research cycle. By involving a wide group of stakeholders (including service users) at all stages of the research process, from the inception of research ideas through to delivery of outputs, a portfolio of clinically relevant, locally tractable and patient-informed projects can be established. Further, as the process itself helps to identify existing evidence needs and barriers to implementation, it is believed that the research findings are more likely to be taken up effectively. Widespread and genuine involvement of the local health community in the process may generate a significant cultural change in attitudes to research. PenCLAHRC sees such engagement as crucial to secure the ultimate aim of a shift in the way applied health research is conducted and valued such that it is readily sustainable beyond the original five years of the pilot. The extent to which the Peninsula and other CLAHRCs meet their objectives is the subject of research originally sponsored by the NHS Service Delivery and Organisation programme(8) which has now been subsumed into the Health Services and Delivery Research programme (HS&DR http://www.netscc.ac.uk/hsdr/).

Figure 1.

The role of PenCLAHRC in a dynamic approach to research and research prioritisation

Research prioritisation is necessary when the resources required to answer the questions raised outweigh the resources that are available (9). Prioritisation can also be used to identify research questions with the potential for greater impact on patient care and outcomes, particularly in the context of localised priority setting. In this paper we describe the development of the PenCLAHRC prioritisation process and how it has changed over the four rounds completed to date and reflect on the implications it may have for a wider (national) context of priority setting in health care research. Other authors have reported on processes of priority setting in health care on a national scale (9–11); this article reflects how processes used on a national level can also be used at the local level.

Question Generation

Question identification began during the development of the bid to establish the CLAHRC with a simple letter emailed to NHS stakeholder organisations seeking questions regarding everyday practice. Over 100 responses were obtained via email of varying detail and structure. In many cases, respondents described an issue but no specific question was identifiable; effective prioritisation of questions in these cases was not possible. It became apparent that a structured format was necessary to capture clinical uncertainties in the form of a question with sufficient supporting information to facilitate prioritisation. The following strategies were developed to aid this process:

• -Web-based tool
• -Locality Leads
• -Workshops

The web-tool was incorporated into the PenCLAHRC website (http://clahrc-peninsula.nihr.ac.uk/submit-question.php)(6) to help question originators develop a clear and focused question. The web-tool provides two structures for questions: a PICO (Population, Intervention, Comparator, Outcomes) structure, which is appropriate for questions about therapies or diagnostic tools, and a structure for questions concerned with the implementation of change in the NHS related to practices for which good evidence already exists (e.g. prevention of venous thromboembolism)(12).

Locality Leads (part-time secondees to PenCLAHRC at consultant level, drawn from acute and primary care trusts) and the Public and Patient Involvement (PPI) Lead, encourage the submission of research questions to PenCLAHRC through workshops and meetings with clinicians and service users. Some of these workshops are specific Evidence Based Practice (EBP) training events designed to help clinicians and service users understand a common language of research in order to be able to develop their own research questions in the future. Since January 2010, over 2,900 people (including 277 service users). have attended these meetings and workshops across Devon and Cornwall

Prioritisation Stage 1 - Executive Group Prioritisation

The Executive Group consists of Peninsula CLAHRC’s director and deputies as well as a number of other academic and clinical positions (see Table 1).

Table 1. Members of the Executive Group.

Role	Number	Locality or Theme
Locality lead (clinical)	2	Cornwall
	2	Plymouth (with south and west Devon)
	2	Exeter (with north and east Devon)
Theme lead (academic)	1	Environment and Human Health
	1	Mental Health and Neurology
	1	Public and Patient Involvement
	1	Primary care and Improvement
	1	Ageing and Development and Implementation

The purpose of the prioritisation meeting is to refine the scope of the question and to prioritise the submitted questions using criteria which aim to highlight the most locally relevant and feasible questions:

• -Size of the health problem which may be addressed by the research
• -Local health care resource impact of the problem
• -Potential for health gain from change contingent on carrying out research
• -Feasibility of answering the question
• -Appropriateness of the local environment to address the question (e.g. local demography)

These criteria reflect the common categories of prioritisation criteria identified in a review conducted by Noorani and colleagues (13) which looked at practical approaches to priority setting in health care. To date (during five rounds over two years) over 290 questions have been considered at Stage 1 of prioritisation. Members receive a pack detailing all the questions in advance of the meeting to ensure that the discussion during the meeting is informed and the meeting productive. All topics are introduced by the Chair (one of the PenCLAHRC deputy directors) who summarises the question and states where particular adherence (or otherwise) to PenCLAHRC priorities is met. This instigates further discussion around local health priorities and prevalence of health problems locally which requires input from each of the 10-15 group members. The meetings last approximately four hours. Discussion continues until all members of the group feel they have sufficient information to rank the questions. A proportion of the highest ranking questions are selected to be taken forward to the next stage of prioritisation. The number is limited, by the agreed timescale and staff capacity, to no more than 12 topics for each Stakeholder meeting. A wide range of questions have been submitted, a selection of which can be seen in Appendix A. Figure 1 details the total number of questions to have come through the PenCLAHRC process from January 2009 to June 2012 (five rounds of prioritisation).

Prioritisation Stage 2 - Stakeholder Group Prioritisation

PenCLAHRC is comprised of the following major stakeholders:

• Higher education institutions

  • -
    University of Exeter
  • -
    Plymouth University
• Strategic Health Authority

  • -
    NHS South West
• Primary Care NHS Trusts

  • -
    NHS Cornwall and Isles of Scilly
  • -
    NHS Devon
  • -
    NHS Plymouth
• NHS Foundation Trusts

  • -
    South Devon Healthcare NHS Foundation Trust
  • -
    Royal Devon & Exeter Healthcare Foundation Trust
  • -
    Cornwall Partnership NHS Foundation Trust
  • -
    South Western Ambulance Service NHS Foundation Trust
• Acute NHS Trusts

  • -
    Northern Devon Healthcare NHS Trust
  • -
    Plymouth Hospitals NHS Trust
  • -
    Royal Cornwall Hospitals NHS Trust
• Mental Health Trusts

  • -
    Devon Partnership NHS Trust
• Primary Care Trusts

  • -
    Torbay and Southern Devon Health and Care NHS Trust

Each of the Stakeholder organisations and service users are invited to contribute to the prioritisation process at a face-to-face meeting held at a central location in the region and led by the same Chair. The aim is to discuss each topic in more detail, supported by additional information; focus questions in more depth using the expertise of the stakeholder group; and generate a ranked list of research ideas for further development within PenCLAHRC.

In preparation for this meeting, the Evidence Synthesis Team (which consists of three researchers and two information specialists) spend two to three days preparing Priority Briefings for the topics prioritised from Stage 1 to supplement and extend the information initially provided by the question submitter. In each case, the question originator is consulted to clarify and develop the question. A brief literature review is performed to identify pertinent guidelines, published and ongoing research, information to assist in assessing the size and impact of the issue, relevance to local health objectives and the potential benefits of resolution of uncertainty. Priority Briefings are not designed to be systematic reviews of the literature.

Service users are also represented at the Stakeholder meeting by two volunteers from PenCLAHRC’s patient involvement group (PenPIG). The group meets prior to the event to discuss the Priority Briefings. PenPIG is a diverse group of service users consisting of approximately 14 people who discuss each question and agree on their votes for the meeting. The service user representatives are able to alter their votes if appropriate following the discussion at the Stakeholder meeting. This process is unique to this particular group of stakeholders (although other stakeholders may consult colleagues informally). PenPIG also provides useful feedback on the accessibility of the priority briefings in terms of language and structure.

The Stakeholder meetings have been well attended with representatives from at least 13 of the potential 16 partner organisations in the collaboration plus two patient representatives, 15-20 voting participants in total (with dedicated support from PenCLAHRC staff) at each meeting. Each question is introduced by a member of the group (with no specialist knowledge of the topic area and is specifically chosen by the Chair in order to reduce any bias that prior knowledge might bring to the introduction of the question). No attempt is made to reach a consensus on the priority of each topic as the meeting progresses. However, the focus of questions may be altered by consensus if necessary. This meeting lasts about five hours. At the end of the discussion, the stakeholders vote for the topics they believe most worthy of further PenCLAHRC involvement according to the prioritisation criteria. The voting process is further described below.

Prioritisation method

The voting method used by PenCLAHRC in the Executive and Stakeholder prioritisation meetings is based on Rank Order Voting. Participants are each allocated a quota of individual votes (ten in the Executive meeting and six in the Stakeholder meeting), of these votes half are worth one point and half are worth two points. All stakeholders votes hold an equal weighting. Voting is anonymous. This system allows for individual priority setting (ranking) to be achieved without the group having to reach an overall consensus on each topic. Rank Order Voting was considered to be the best format on which to base prioritisation for the PenCLAHRC process as it enables a number of topics to be selected in a fair and practical manner. The reasons why this particular method was chosen and how it compares to other research prioritisation processes are discussed further in the reflections section.

Questions and topics

Since the first round of PenCLAHRC’s prioritisation process in 2009 questions have been submitted from range of health areas. Figure 3 shows what percentage of the questions submitted to the process and what percentage of questions supported by PenCLAHRC (over the first four rounds of prioritisation) fall into each International Statistical Classification of Disease and Related Health Problems (ICD 10) with a separate category for Service Delivery questions in the final column (SD). From this data we can see that the range of questions supported by PenCLAHRC is narrower than the range of questions submitted to the process. This might be expected due to the small proportion of questions that PenCLAHRC are able to support. For example, the proportion of questions submitted in the ICD10 V classification (Mental and Behavioural Disorders) is fairly balanced with the proportion that are represented in the questions supported by PenCLAHRC. The imbalance works in both directions and may reflect the local relevance and priority of those health areas, perceived by the PenCLAHRC stakeholders.

Figure 3. Proportion of questions submitted to– and supported by-PenCLAHRC by ICD 10 classification.

The types of questions submitted to– and supported by PenCLAHRC also varied. We identified four questions types: development - questions that require the development of an intervention or tool to be tested, intervention – questions about the effectiveness or cost effectiveness of an intervention that has already been developed, implementation questions regarding barriers and facilitators to interventions being implemented when there is already clear evidence on effectiveness, and modelling – questions regarding the arrangements of NHS services which can be addressed using operational research methods (for which PenCLAHRC has established a small research team in collaboration with the NHS). Figure 4 shows the percentage of the types of questions submitted to PenCLAHRC and Figure 5 shows the percentage of the types of questions supported by PenCLAHRC. The pie charts show that the proportion of development, intervention and modelling questions submitted to and supported by PenCLAHRC are relatively balanced.

Figure 4. Proportion of questions submitted to PenCLAHRC by question type – and supported by- PenCLAHRC by question type.

Figure 5. Proportion of questions supported by PenCLAHRC by question type.

Table 3 lists some examples of the topics that have been prioritised and how they have been taken forward by PenCLAHRC.

Table 3. Voting example.

Candidate/vote	30%	36%	34%
1	B	A	C
2	A	B	B
3	C	C	A

Reflections

PenCLAHRC is just one example of how mixed groups of stakeholders can be encouraged to develop relevant health care priorities. A review by Stewart and colleagues (14) found many reports of engagement of services users and/or clinicians in developing health related research questions. The James Lind Alliance (JLA) also facilitates collaboration between patients, carers and clinicians through priority setting partnerships. The JLA, much like PenCLAHRC, aims to initiate health care research which has more relevance and benefit to the service users and clinicians (15). However, JLA uses different methods of prioritisation, concentrating on one health area at a time and specifically excluding academics from the initial prioritisation processes aims for national coverage of clinicians and service users. Engaging with service users in priority setting is in line with the emphasis being placed on the role for service users in the ongoing identification, development and implementation of health research (http://www.involve.org.uk/).

It is interesting to note that although those carrying out the prioritisation in PenCLAHRC came from a wide range of backgrounds, there appears to be considerable convergence of priorities across groups, particularly for the highest ranked questions which were supported by most constituents. This can also be seen in other stakeholder group settings (16) and provides support for partnership working. However, topics (particularly implementation questions) that do not have the support of relevant NHS Trust cannot be expected to go far without a successful business case.

From the outset, a number of challenges were recognised that might influence the success of the PenCLAHRC structure and aims:

• Quality of questions - acquiring the resources and skills across the collaboration to develop the language of research and to develop specific research questions as opposed to general identification of issues.

• Time constraints - the timescale for delivery expected by stakeholders, particularly given the need to establish a process and resource it with appropriate staff.

• Engagement - potential for bias within the process towards known contacts within the local health community, leading to disenfranchisement within the collaboration. The wide range of stakeholders to be included, in terms of different organisations, professional groups and geographic areas. The process itself raising expectations amongst the clinical community which could not be met.

As PenCLAHRC has developed there has been growing clarity about these challenges. Feedback on the prioritisation process (gathered by an evaluation form presented at the Executive and Stakeholder meetings – Appendix B) has informed a number of thoughts and decisions which are discussed below.

Quality of questions

Concern was raised regarding the varying detail that was provided in question submissions for the Executive stage of prioritisation. It was felt that this variation may provoke bias in the selection of questions during the rapid prioritisation of topics by the Executive Group as those with more information may be favoured, disregarding the question itself. In response to this risk, the development of the web tool and the delivery of EBP workshops have helped to standardise the detail provided at the question generation stage. However, questions with little initial information in their submission have been prioritised in the past.

Interestingly, use of the web tool highlighted difficulties in usefully defining the difference between the question types described earlier; development, intervention and implementation (language of research). There are instances of questions being categorised as intervention questions when in reality they cover implementation. This may be resolved through further development of the EBP workshops and will improve over time as the language of EBP develops and becomes more widespread. These workshops will also improve the skills and resources available to produce more focussed research questions.

Another potential criticism of the PenCLAHRC process is that topics lost at the first stage of prioritisation do not benefit from the information gathering that happens for the Priority Briefing. This means there is a possibility that topics discarded at the first stage of prioritisation might have been preferred over those that progressed if they had been subject to more lengthy and careful information support. To reduce the potential for this to happen, question originators are advised to contact their Locality Lead for support and to submit the question in alliance with them. Furthermore, originators of discarded questions are sent feedback letters to encourage them to develop their question further or submit their question to other research organisations that may also be interested in the topic e.g. NIHR Health Technology Assessment (HTA) programme, the Database of Uncertainties about the Effects of Treatments (DUETS), and the local NHS Research Design Service (RDS).

Time constraints

Two concerns regarding the meeting organisation were the time taken during the Executive Group meeting to discuss each topic and the use of video-conferencing. Whilst many of the group reported the amount of discussion and information available to be adequate, there were concerns that tiredness was affecting concentration and therefore influencing the voting on topics discussed later in the meeting. To date there has been no evidence that this has been a problem. The initial use of video-conferencing during the Executive meeting was awkward and both prioritisation meetings (Executive and Stakeholder) are now held face-to-face. This also seems to help improve engagement in the meetings from all stakeholders.

Engagement

Geographically, the areas of Torbay and North Devon have been under-represented in the questions submitted to the PenCLAHRC process to date. This highlights the problem of bias towards known contacts in the local health community. Difficulties in engaging with these areas may also be attributed to the travelling distances and time involved and to a lack of personal contacts. This is also reflected in the clinical specialities that have been covered. Question generation activity is recorded and analysed on a regular basis to identify gaps in engagement.

It has been a particular challenge to engage clinicians and academics alike in the initial PenCLAHRC process once they have realised that no direct funding is available. However, disengagement at the Executive and Stakeholder meetings (resulting in a lack of or inappropriate representation of stakeholder groups) was rare. This occurred once, and the importance of engagement from all stakeholders was emphasised such it did not happen again. PenCLAHRC is able to provide support for research projects irrespective of whether the question originator has the time, resources or knowledge to lead on that project. If a project is supported by PenCLAHRC, a team of relevant researchers, academics, clinicians and service users is brought together by a designated project facilitator to take the project forward (including applying for external funding).

Voting method

The voting method used by PenCLAHRC in the Executive and Stakeholder prioritisation meetings is based on Rank Order Voting. This was considered to be the best format on which to base prioritisation for the PenCLAHRC process as voting is speedy, accurate and anonymous following a shared, informative discussion. Several other voting systems/methods could have been available to prioritise research suggestions, such as Nominal Group Process (17), the Hanlon Process of Prioritisation (18), Majority Rule (19) or Rank Order Voting (developed by Borda and referred to as the Borda Count) (19). Reasons for these methods were not taken up are described below.

Nominal Group Process provides a structured group discussion whereby input from all participants is balanced and makes the best use of experience and expertise without the risk of dominant individuals skewing the process or results. Ideas can be generated, problems solved and prioritising decisions proposed in a face-to-face non-threatening environment (18). The Hanlon Process of Prioritisation encourages members of the group to work together to ensure all participants have consistency in language and understanding. This is to help when discussing the variety of issues/programmes on the prioritisation agenda (18). Majority Rule works by taking into account the runner-up places of projects. Participants vote independently for their first, second, and third project preference. For example, if we look at the results of a vote in Table 3, in the Plurality Rule method the project who received the most votes as the most preferable project will win (i.e. candidate A), all other information/votes are discarded. In the Majority Rule process, however, the winner is the project that would beat each opponent in a pairwise comparison. So from Table 2 we can see that 64% of the voters prefer project B to project A, and 66% prefer B to C. Hence project B would win and A would be second with 66% preferring A to C and 36% preferring A to B, and C would come in last with only 34% preferring C to A or B (19). These methods of prioritisation were not chosen due to time restraints and the need to rank the topics rather than to seek one clear ‘winner’.

Table 2. Example projects resulting from Stakeholder prioritisation.

Topic	Project/Action
How can we get the most benefit from thrombolysis for acute ischaemic stroke in the south west peninsula	A modelling study to examine different configurations of stroke services to inform service changes.
Does the Health Compass programme promote healthy diet and physical activity in people with high cardiovascular risk	A protocol will be developed for funding for a multicentre cluster randomised controlled trial to examine the effect of a locally developed intervention
Is the Action for Rehabilitation from Neurological Injury Training technique effective in increasing functionality, improving activity and improving autonomy in people who are between 1 and 5 years after their stroke	A protocol will be developed for a trial of long term rehabilitation
Would a parent delivered programme of training for paediatric ward staff improve the effectiveness of communication with disabled children and their parents when they are admitted to hospital	An intervention will be developed and evaluated with service users.
What school level interventions would improve social and emotional behaviour among primary school children (specifically two programmes)	A systematic review will be conducted on teacher-oriented interventions to improve social and emotional well being in primary school children.
Would a mental health/developmental paediatric assessment of primary school children who are excluded from school lead to evidence-based management of underlying problems and hence improve the likelihood of a positive long term outcome?	A systematic review will be conducted to establish the prevalence of mental health problems in school-aged children and how children with problems are dealt with within the school environment.
Does a simple pre-clinic form improve patient satisfaction following a diabetes appointment?	An intervention will be developed and evaluated with service users.
How can patient initiated clinics (PIC) be implemented for rheumatoid arthritis given the demonstration of the acceptability, effectiveness and cost effectiveness of such service organisation?	A systematic review will be conducted on the acceptability, effectiveness and cost effectiveness of PICs across disease areas (including rheumatoid arthritis
What are the benefits and costs of providing peer support to parents of children with disability?	A systematic review and evaluation of a pilot scheme on the topic will be conducted.
What is the value of respite care for unpaid carers of people with dementia? (If there is some value, what forms of respite care are most beneficial?)	This will be developed into a full research proposal and funding will be sought.
In some situations, are telephone consultations a safe and effective method of providing routine follow-up in secondary care?	A systematic review on this topic will be conducted.

A number of other research organisations use similar methods of prioritisation (and identification). Although now undergoing some change, the NIHR HTA programme consults with stakeholders in the NHS and external organisations, encourages submissions from researchers and scans the medical literature to identify research questions. In the process up to 2013 HTA prepared briefings for those questions that made it through the first round of prioritisation, where each question is considered in relation to the research programme criteria. The briefings were then considered by expert panels in the second round of prioritisation. The Medical Research Council (MRC) (20) consults with a wide range of stakeholders including Government departments, the public sector, industry, charities and the general public. The NIHR SDO programme (now HS&DR http://www.netscc.ac.uk/hsdr/) engages a wide range of stakeholders and several levels of prioritisation in order to identify relevant areas of research. Other research funders in the U.S. such as the National Institute of Health (NIH) and the Centre for Disease Control and Prevention (CDC) also use a collaborative reviewing format, although the inclusion of service users is unclear and voting methods vary.

The Future

PenCLAHRC has developed into a team of over fifty people, with wide ranging skills including statisticians, decision modellers, systematic reviewers, information scientists and other methodologists. Around 25 research projects have been established through the prioritisation process and are either underway or the subject of major funding bids. So far two large funding bids have been successful total grants have amounted in over £3million for PenCLAHRC projects.

PenCLAHRC will continue to run cycles of question generation and prioritisation, incorporating appropriate changes and improvements according to internal evaluation. In addition, PenCLAHRC has developed more intensive EBP courses to help clinicians and service users in Devon and Cornwall improve their understanding of research, development of research questions and translation of research into practice. Such courses will empower attendees to construct and submit quality research questions to answer service and practice based queries. This will help build capacity for implementation and implementation research as advocated in the Chief Medical Officer’s High Level Group on Clinical Effectiveness (3) as part of a more sustainable approach to research (4). It is believed that engagement of stakeholders in establishing a research agenda increases relevance and may enhance the implementation of research findings (21).

Figure 2. Total number of questions through the prioritisation process (August 2009 - June 2012).