What does it mean to involve consumers successfully in NHS research? A consensus study

Abstract

Objective  To obtain consensus on the principles and indicators of successful consumer involvement in NHS research.

Design  Consensus methods were used. An expert workshop, employing the nominal group technique was used to generate potential principles and indicators. A two‐round postal Delphi process was used to obtain consensus on the principles and indicators.

Setting and participants  Participants were drawn from health, social care, universities and consumer organizations. A purposive sampling strategy was used to identify people who had experience and/or knowledge of consumer involvement in NHS research. Six researchers and seven consumers participated in an expert workshop. Ninety‐six people completed both rounds of the Delphi process.

Main outcome measures  Consensus on principles and indicators of successful consumer involvement in NHS research.

Results  Eight principles were developed through an expert workshop and Delphi process, and rated as both clear and valid. Consensus was reached on at least one clear and valid indicator by which to measure each principle.

Conclusions  Consensus has been obtained on eight principles of successful consumer involvement in NHS research. They may help commissioners, researchers and consumers to deepen their understanding of this issue, and can be used to guide good practice.

Introduction

The involvement of consumers is central to NHS Research and Development policy. ¹ , ² Current NHS guidance on research governance states that consumer involvement should exist at every stage of research where appropriate. ³ Consumers are said to bring unique perspectives to research, making it more relevant to their needs, and therefore to the NHS. ⁴ , ⁵ The value of consumer involvement in health research has been acknowledged both nationally and internationally, with many influential bodies, such as the Health Technology Assessment Programme, ⁶ the Medical Research Council, ⁷ the National Cancer Research Institute, ⁸ the Cochrane Collaboration, ⁹ and the Consumers’ Health Forum of Australia, ¹⁰ including consumers as partners in research.

Consumer involvement in health research is a relatively new concept for health professionals, with little empirical research to draw on. Most accounts to date have been descriptive or anecdotal with uncertain generalizability, and it is not clear how consumers can influence and benefit the research process. ¹¹ In 2000, Sir John Patterson, the Director of Research and Development at the Department of Health stated that: ‘No systematic evaluation has been undertaken to assess the impact of the involvement of consumers in the research process.’ ¹² Few studies have risen to this challenge, and there continues to be a lack of evidence of the effectiveness of consumer involvement in research. ¹³

Despite clear directives from the Department of Health and guidance from the Consumers in NHS Research (re‐named INVOLVE) Support Unit, ⁴ , ¹⁴ implementation of the policy has been sporadic, with limited understanding among health professionals about the meaning and implications of active partnership with consumers in research. ¹⁵ One survey of health researchers revealed that many were unsure if they had involved consumers or not in their research. ¹⁶

The present study set out to develop through consensus methodology, principles of successful consumer involvement in NHS research, and indicators by which the principles could be measured. It was anticipated that these would be of interest to commissioners, researchers and consumers; would assist in developing further understanding about the meaning of consumer involvement in NHS research; would provide guidance for good practice; and might also be a first step in developing robust ways of assessing the impact on research of consumer involvement.

Methods

Consensus methods were employed in two stages: (a) an expert workshop which utilized the nominal group technique (NGT), and (b) a two‐round postal Delphi process.

An expert workshop

An expert workshop was held in January 2002 with the aim of developing principles or standards of successful consumer involvement in NHS research. In order to evaluate whether the principles could be implemented, the intention was also to develop indicators by which to measure the principles. The criteria for inviting consumers and researchers to the expert workshop were: (a) some ‘standing’ in the field of consumer involvement in research. This was interpreted as at least one published article or report on the topic of consumer involvement in research, or membership of a relevant committee; (b) a willingness to share ideas with others.

The NGT was used to facilitate discussion at the expert workshop around the question ‘what is meant by the successful involvement of consumers in research?’ The NGT is a highly controlled small‐group process for generating ideas. ¹⁷ Typical applications of the technique are for the development of consensus guidelines or standards in areas where research‐based evidence is absent or inconclusive. ¹⁸ The key components of the method are: formulation and presentation of the nominal question; silent generation of ideas in writing; feedback from group members to record each idea in a succinct phrase; group discussion of each idea in turn for clarification and evaluation; individual voting on priority ideas; feedback of results; and further discussion and re‐voting. There was no prior collation of principles from the literature in advance of the expert workshop. Participants were given four examples of potential principles and indicators beforehand.

By using an NGT, a number of principles were proposed by the workshop participants. The retained principles were those considered important by 85% or more of the participants for assessing if consumers are successfully involved in NHS research. Participants were also facilitated to identify indicators by which the principles could be measured. Following discussions with the Advisory Group, a cut‐off of 85% was chosen in this study to set a high level of consensus. There does not appear to be agreement on what constitutes consensus when using consensus methods, and different criteria may be used for describing when consensus is reached. However the importance of clearly determining consensus before the method is used has been highlighted. ¹⁹

A two‐round postal Delphi process

Consensus was sought from a wider group of researchers and consumers with knowledge or experience of consumer involvement in research on the retained principles developed at the expert workshop, and their associated indicators. A two round postal Delphi process was used for this purpose. The Delphi process is defined as a method used to obtain the most reliable consensus by a group of experts, through a series of intensive questionnaires, interspersed with controlled feedback. ²⁰ Key characteristics of the Delphi process include a panel of experts who do not meet face‐to‐face, two or more rounds of questionnaires or interviews to develop ideas, and the systematic emergence of a consensus. ²¹

The number of people in the UK who have knowledge or experience of consumer involvement in NHS research is limited and we therefore used a variety of means to recruit as many Delphi participants as possible:

• • Each expert workshop participant was asked to pass on invitations to take part to up to five consumers known to them
• • UK authors of at least one journal article or report on consumer involvement in NHS research, and speakers at relevant national and regional conferences were invited to take part
• • An invitation to participate was published in the Consumers in NHS Research Support Unit Newsletter
• • Consumers who contributed to the Consumers in NHS Research Support Unit publication ‘Getting involved in research: a guide for consumers’ ¹⁴ were contacted care of the Support Unit.

Through these means, 131 people agreed to take part in the Delphi study.

In April 2002, the round 1 Delphi questionnaire was sent out. This questionnaire incorporated the retained principles developed at the workshop, and between one and five indicators for each principle. In this round, participants were invited to reword existing principles and indicators and to contribute further principles and indicators if they wished.

Delphi questionnaires in both rounds asked respondents to rate the principles on two nine‐point scales: clarity (the extent to which the principle is expressed in clear, precise and unambiguous language); and validity (the extent to which the principle is important for assessing if consumers are successfully involved in NHS research). Participants were asked to rate each indicator on three nine‐point scales: clarity (the extent to which the indicator is expressed in clear, precise and unambiguous language); validity (the extent to which the indicator is a good measure of the associated principle); and feasibility (the extent to which the data for this indicator is, or could be made, available and consistently recorded). The above definitions of clarity, validity and feasibility were included in the questionnaire. Panellists were also invited to contact the researchers for further clarification if necessary.

Completed round 1 questionnaires were then analysed by the research team in order to generate data to be fed back to participants at round 2. Information fed back related to: (a) the median rating of each principle and indicator on the clarity, validity and feasibility scales; (b) distribution data relating to each scale point on each scale; and (c) if consensus was achieved at round 1.

The level of consensus was set at 85% before the mailing of round 1. A principle was to be retained if 85% or more of the panel rated it between 7 and 9 on the nine‐point scale on both clarity and validity. An indicator was to be retained if 85% or more of the panel rated it between 7 and 9 on clarity and validity and feasibility.

The round 2 questionnaire was sent out in July 2002. At round 2, participants were requested to rate the original principles and indicators again, in the light of the provided median and distribution data relating to round 1. Participants were also invited to rate any new principles and indicators proposed in the first round. If consensus was achieved at round 1, participants were not invited to re‐rate at round 2.

Consumer involvement in this study

Three people who provided a consumer perspective at the expert workshop agreed to join the Advisory Group after the workshop and were consulted at different stages of the research, influencing the methodology and the interpretation of the results.

Expert workshop

Six researchers and seven consumers attended the expert workshop. Thirteen principles were generated, with 85% consensus on 12 of them (see Table 1). The principle that was not retained because of poor agreement stated that the process of involving consumers in research should be enjoyable. [One of the 12 principles (C) was later split into two principles, C and D, as a result of responses during the Delphi process, thus giving 13 principles in total]. There was not enough time to develop indicators of the principles on the day and they were subsequently produced by consulting the literature, using material from the expert workshop (post‐its, flip charts and notes of the discussions), postal consultation of members of the expert workshop, and by consulting Advisory Group members.

Table 1.

Clarity and validity ratings of the principles of successful consumer involvement in NHS research

Principle	Clarity		Validity
	Median	Consensus1	Median	Consensus1
A: The research will lead to benefits for consumers, in terms identified by the consumers themselves	7	No	8	Yes
B: Consumers are involved in every stage of the research, from identifying the research area through to sharing the research findings	8	Yes	8	No
C: Consumers’ expectations of being involved in the research are made clear to the researcher	7	No	8	No
D: The roles of consumers are agreed between the researchers and consumers involved in the research	8	Yes	8	Yes
E: Consumers have the opportunity to engage in research in the manner and at the level they wish, opting out of being involved in research at any time	8	Yes	7	No
F: Researchers budget appropriately for the costs of consumer involvement in research	8	Yes	8	Yes
G: Consumers are from sections of society and walks of life that are appropriate to the research	7	No	7	No
H: Researchers respect the differing skills, knowledge and experience of consumers	8	Yes	8.5	Yes
I: Consumers are offered training and personal support, to enable them to be involved in research	8	Yes	8	Yes
J: Researchers ensure that they have the necessary skills to involve consumers in the research process	8	Yes	8	Yes
K: Consumers are involved in decisions about how participants are both recruited and kept informed about the progress of the research	8	Yes	8	Yes
L: Consumer involvement is described in research reports	9	Yes	9	Yes
M: Research findings are available to consumers, in formats and in language they can easily understand	9	Yes	9	Yes

Retained principles are given in bold.

¹Consensus defined as 85% of panellists rating the scale item between the tertiles of 1–3 OR 4–6 OR 7–9. Yes, consensus reached; no, consensus not reached.

Description of the sample

Of the 131 people who agreed to take part in the study, first round questionnaires were received from 110, giving an attrition rate of 16%. Second round questionnaires were sent to these 110 participants, of which 96 were returned, yielding a round 2 attrition rate of 13%. Demographic data presented relate to those returning both rounds of the questionnaire (n = 96).

Most Delphi panellists were aged between 36 and 55 years, and more women (n = 59) than men (n = 37) participated. The ethnic origin of most of the sample was White UK (n = 85). Panellists described themselves as providing one of the following perspectives: consumer, researcher, consumer and researcher, other or consumer and other. The perspective rated most frequently was that of researcher (n = 33), followed by consumer (n = 29) then consumer and researcher (n = 26). Consumer perspectives were broken down further into five groupings taken from a list of options based on the definition of ‘consumer’ employed by Consumers in NHS Research: ¹⁴ an advocate/activist/consumer representative (n = 21), a patient or long‐term user of services (n = 15), an employee of an organization that is for consumers, e.g. a charity (n = 12), a member of an organization of consumers where the organization is managed by more than 50% of people with that experience or health condition (n = 9), and a carer (n = 8). Although only one was asked for, some panellists provided more than one consumer perspective.

Those identifying themselves as researchers were asked to provide information on the type(s) of health research with which they had been most actively involved. The most common was health services research (n = 53). Other kinds of research included clinical trials (n = 16), secondary research (n = 13), behavioural research (n = 10) and population‐based research (n = 10).

We asked all panellists to provide up to three health areas in which they had most active experience of consumer involvement in research. The five dominant health areas were mental health, physical and learning disabilities, cancer, pregnancy, childbirth and childcare, and the health of older adults.

Rating of the principles and indicators

Table 1 displays clarity and validity ratings of the 13 principles of successful consumer involvement in NHS research. Eight of these satisfied the initial retention criterion, and data relating to their associated indicators were then examined.

Table 2 displays the clarity and validity ratings of the indicators of the eight retained principles. Both 1, 2 display the results of round 2 only. At the suggestion of Delphi participants, a small number changes were made between rounds 1 and 2. These changes concerned: (a) splitting one principle into two separate principles; (b) suggesting indicators to existing principles (e.g. H3 and I4); and (c) refining the wording of existing principles and indicators. Space does not allow a detailed discussion of such changes; however, the following two examples are offered. At round 1 principle C read, ‘The roles of consumers are negotiated, and their expectations clarified’. Panellists considered that the principle covered two separate issues, so this principle was split into principles C and D at round 2 (see Table 1). At round 1, principle I read, ‘Consumers have access to training, mentoring and personal support, to enable them to be involved in research’. Some panellists expressed their dislike of the term ‘mentoring’ in the principle, while others argued that consumers may not actually want to have access to such things. Therefore, at round 2 the principle read, ‘Consumers are offered training and personal support, to enable them to be involved in research.’

Table 2.

Clarity and validity ratings of indicators of the eight retained principles of successful consumer involvement in NHS research

Principle and associated indicators	Clarity		Validity
	Median	Consensus1	Median	Consensus1
D: The roles of consumers are agreed between the researchers and consumers involved in the research
D1: Researchers’ expectations of what they wanted from consumers were recorded at the beginning of the research	8	Yes	7	No
D2: The roles of consumers in the research were agreed on an individual basis	8	Yes	7	No
D3: The roles of consumers in the research were documented	8	Yes	8	Yes
D4: Consumers’ roles in the research were reviewed during the research process	8	No	7	No
D5: Consumers were asked about how they wished to be involved in the research	8	Yes	8	No
F: Researchers budget appropriately for the costs of consumer involvement in research
F1: Researchers applied for funding to involve consumers in the research>	8	Yes	9	Yes
F2: Consumers were reimbursed for their travel costs	8	Yes	9	Yes
F3: Consumers were reimbursed for their indirect costs (e.g. carer costs)	9	Yes	9	Yes
F4: Consumers were offered reimbursement for their time	9	Yes	8	No
F5: Consumers negotiated the rate of reimbursement for their expertise	8	No	7	No
H: Researchers respect the differing skills, knowledge and experience of consumers
H1: Consumers’ skills, knowledge and experience were recorded at the beginning of the research	8	Yes	7	No
H2: The contribution of consumers' skills, knowledge and experience was included in research reports and papers	8	Yes	8	Yes
H3: Consumers’ skills, knowledge and experience were used to the full where relevant	7	No	8	No
I: Consumers are offered training and personal support, to enable them to be involved in research
I1: Consumers' training needs related to their involvement in the research were agreed between consumers and researchers	8	Yes	8	Yes
I2: Consumers had access to training to facilitate their involvement in the research	8	Yes	8	Yes
I3: Mentors were available to provide personal and technical support to consumers	8	Yes	8	Yes
I4: In order to provide peer support, at least two consumers were involved in the research at the same time	8	No	8	No
I5: Consumers’ experiences of their involvement were reviewed and responded to in the course of the research	8	No	7	No
J: Researchers ensure that they have the necessary skills to involve consumers in the research process
J1: Researchers assessed their own training needs in relation to involving consumers in the research	8	No	7	No
J2: Researchers ensured that their own training needs were met in relation to involving consumers in the research	8	Yes	8	Yes
K: Consumers are involved in decisions about how participants are both recruited and kept informed about the progress of the research
K1: Consumers gave advice to researchers on how to recruit participants to the research	8	Yes	8	Yes
K2: Consumers gave advice to researchers on how to keep participants informed about the progress of the research	8	Yes	8	Yes
L: Consumer involvement is described in research reports
L1: The involvement of consumers in the research reports and publications was acknowledged	9	Yes	9	Yes
L2: Details were given in the research reports and publications of how consumers were involved in the research process	8	Yes	8	Yes
L3: Consumers’ views of the research findings were explicitly documented in the research reports and publications	8	Yes	8	No
L4: The impact of the consumers’ involvement in the research was described in the research reports and publications	7	No	7	No
M: Research findings are available to consumers, in formats and in language they can easily understand
M1: The research findings were publicly available	8	Yes	8	No
M2: Research findings were disseminated to consumers involved in the research in appropriate formats (e.g. large print, translations, audio, Braille)	8	Yes	8	Yes
M3: The distribution of the research findings to relevant consumer groups was in appropriate formats and easily understandable language	8	Yes	8	Yes
M4: Consumers involved in the research gave their advice on the choice of methods used to distribute the research findings	8	Yes	8	Yes
M5: Consumers involved in the research were given the opportunity to distribute research findings to their own networks and/or contacts unless prevented by the research funding body	8	Yes	8	No

Retained principles and indicators are given in bold.

¹Consensus defined as 85% of panellists rating a scale item between the tertiles of 1–3 OR 4–6 OR 7–9. Yes, consensus reached; no, consensus not reached.

During the Delphi process, a number of people voiced their concern about the issue of feasibility (the extent to which data for the indicator is, or could be, made available and consistently recorded by research teams). Some found feasibility difficult to rate in the abstract: ‘feasibility was hard to assess without context’, others questioned why it should be measured, and several found the concept confusing. These observations raised doubts about the validity of the ratings on feasibility, and it was therefore decided to remove the issue of feasibility from the data analysis. The retention criterion for the indicators was therefore adjusted. An indicator was now retained if 85% or more of the panel rated it between 7 and 9 on the nine‐point scales of clarity and validity only. Table 2 shows that eight of the retained principles have at least one clear and valid measurable indicator as rated by 85% or more of the panel. These were principles D, F, H, I, J, K, L and M.

In summary, through an expert workshop and a two‐round Delphi process, eight clear and valid principles of successful consumer involvement in NHS research have been developed, each of which has at least one clear and valid indicator (see Table 3).

Table 3.

The principles and indicators of successful consumer involvement in NHS research

	Principle	Indicator(s)
1	The roles of consumers are agreed between the researchers and consumers involved in the research	The roles of consumers in the research were documented
2	Researchers budget appropriately for the costs of consumer involvement in research	Researchers applied for funding to involve consumers in the research
		Consumers were reimbursed for their travel costs
		Consumers were reimbursed for their indirect costs (e.g. carer costs)
3	Researchers respect the differing skills, knowledge and experience of consumers	The contribution of consumers’ skills, knowledge and experience were included in research reports and papers
4	Consumers are offered training and personal support, to enable them to be involved in research	Consumers’ training needs related to their involvement in the research were agreed between consumers and researchers
		Consumers had access to training to facilitate their involvement in the research
		Mentors were available to provide personal and technical support to consumers
5	Researchers ensure that they have the necessary skills to involve consumers in the research process	Researchers ensured that their own training needs were met in relation to involving consumers in the research
6	Consumers are involved in decisions about how participants are both recruited and kept informed about the progress of the research	Consumers gave advice to researchers on how to recruit participants to the research
		Consumers gave advice to researchers on how to keep participants informed about the progress of the research
7	Consumer involvement is described in research reports	The involvement of consumers in the research reports and publications was acknowledged
		Details were given in the research reports and publications of how consumers were involved in the research process
8	Research findings are available to consumers, in formats and in language they can easily understand	Research findings were disseminated to consumers involved in the research in appropriate formats (e.g. large print, translations, audio, Braille)
		The distribution of the research findings to relevant consumer groups was in appropriate formats and easily understandable language
		Consumers involved in the research gave their advice on the choice of methods used to distribute the research findings

Discussion

Using a systematic approach, consensus has been achieved among researchers and consumers on eight clear and valid principles of successful consumer involvement in NHS research. Previous authors have identified mismatches between the views and priorities of researchers and consumers, ²² , ²³ , ²⁴ but findings in this paper suggest there is concordance about fundamental aspects of successful consumer involvement in research. The principles address research process issues, and all have associated clear and valid indicators. We did not adopt a weighting system for the principles on the advice of the expert workshop members.

The principles address ethical, moral and practical issues, and are congruent with guidance offered by INVOLVE on methods of involving consumers in research ⁴ , ¹⁴ and key principles said to underlie the philosophy held by many service‐user researchers about involving them in research. ²⁵ The principles reflect similar work that addressed the importance of being clear about why researchers wish to involve consumers or service‐users in research, funding and training issues, and the need for research findings to be available in accessible formats.

The Toronto Group ²⁶ has presented a range of degrees of service user involvement in research along a number of dimensions ranging from high to low involvement, to initiate a theoretical debate about the changing role and relations of research for research practitioners. Eight research issues, including research funding, research design and process, and dissemination were given as examples for readers to informally assess their own research activities against a continuum. Using ‘dissemination’ as an example, four possible levels of service‐user involvement are listed, ranging from ‘research participants/service users and their organizations make the decisions about research dissemination and publication formats’, through ‘research participants/service users and their organizations are involved in the decision making process for this’ to ‘research participants are not involved in the process of dissemination and findings are not produced in accessible formats.’ It is possible to map some of the principles developed in this study within the schema offered by the Toronto Group, ²⁶ and where the principles can be located suggests that they reflect an intermediate level of service user involvement.

The value and utility of the principles and indicators have yet to be established. They will almost certainly need to be further developed and refined. Future work is needed to establish how transferable they are to different research methodologies and models of consumer involvement.

Health services research was by far the most frequently cited area from which our researcher participants were drawn, with consumer involvement more commonly reported in certain health topics (mental health, physical disability and learning difficulties, cancer, pregnancy, childbirth and childcare, and the health of older adults). These findings suggest that the policy on consumer involvement may be more developed in these research categories, and/or that these health areas attract more research activity.

The principles and indicators that emerged during this study may be more appropriate to models of research in which the researcher leads and invites the involvement of consumers in research. Whether or not the principles can be adapted for user‐led research has yet to be tested. Over one‐quarter of our Delphi respondents described themselves as ‘consumer‐researchers’, suggesting that researchers and consumers can have overlapping roles. ²⁷ We did not explore the complexities of the terms ‘consumer’ and ‘consumer‐researcher’ or how these categories may have been interpreted by respondents in this study.

Combining the expert workshop and the Delphi process enabled the identification and refinement of principles and indicators by a large number of people, who were knowledgeable or experienced in the area of consumer involvement in research. These methods have been successfully used together in a previous study that developed key priorities for a consensus statement on user involvement in cancer services. ²⁸ While the Delphi process is more frequently associated with the development of consensus on treatment decisions, it has also been employed to develop quality indicators for primary care mental health care, ²⁹ and in this study, it proved to be an effective way of generating consensus on the descriptors or standards of what are considered to be successful ways of involving consumers in NHS research.

As the aim was to target ‘expert’ participants, we used a systematic process to recruit as many people as possible from the small number of potential participants. The diversity of our participants was limited, and probably reflects that of the ‘experts’ in this field. Although we consulted a reference group of people from minority ethnic groups during the preparatory stages of this research, and one of the members continued to act as an advisor during the course of this study, we acknowledge the lack of ethnic diversity influencing this work. We did not specifically seek the opinions of people with disabilities during in this research, and recognize that this too is a limitation of the study.

In many cases, feasibility (the extent to which data for the indicator is, or could be, made available and consistently recorded) was consistently rated lower than that of clarity and validity. An example of an indicator that did not meet the 85% consensus level on feasibility is: ‘Consumers gave advice to researchers on how to recruit participants to the research’. Participants may have felt that it was too soon for the data on the indicators to be recorded in a consistent way. If so, it is not known if this concerns current or future feasibility. Is this an issue of low expectations that will change as people become aware of more examples of consumer involvement, or do people hold the view that it will never be feasible? This can be tested, and the authors are undertaking a national survey of NHS lead researchers to explore this issue. Another explanation is that other more appropriate measurable indicators of the principles should be identified. The Delphi questionnaire was a long and complex instrument, and several people told the research team that they found the concept of feasibility unclear, despite being given an explanation.

This study marks an early attempt to employ rigorous methods in an area where little empirical research has so far been undertaken. The main benefits of the principles are in guiding good practice, providing recommendations for commissioning research, and in deepening understanding about involving consumers in research. Having developed the principles, additional work is required to develop feasible indicators, in order to monitor and evaluate the principles. Findings from this study suggest that this task will be challenging.