Critical appraisal guidelines for assessing the quality and impact of user involvement in research

Abstract

Background  The involvement of service users in the design and conduct of health research has developed significantly in the UK in recent years. Involving service users is now seen as a core component of good research practice for all forms of health research. Given the important role that users have in health research, it is necessary to develop guidelines for their effective involvement. Whilst guidelines are currently being formulated, there remain no criteria with which to assess user involvement in published studies and funding applications.

Objective  This study offers guidelines for appraising the quality and impact of user involvement in published papers and grant applications.

Methods  Appraisal guidelines for user involvement have been developed on the basis of available literature and experiences from studies involving cancer patients and carers in the design and conduct of research.

Findings  Nine appraisal criteria have been developed. Criteria include issues such as ‘Is the rationale for involving users clearly demonstrated?’, ‘Is the level of user involvement appropriate?’, ‘Is the recruitment strategy appropriate?’, and ‘Is the nature of training appropriate?’

Conclusion  Generating and applying guidelines is vital if the impact of user involvement agenda in health research is to be understood.

Introduction

Service users are exerting increasing influence over the conduct of health research in the UK and internationally. ¹ Service user involvement challenges the sovereign status of professional expertise and typically generates polarized views, from those who view such involvement as a panacea ¹ , ² to those who perceive it as a trend that potentially jeopardizes the integrity of the research process. ³

The Department of Health recently recommended that all health research conducted in the UK should involve users to comply with principles of good research practice, regardless of methodological background. ⁴ Whilst guidelines have been developed on how to involve users successfully, little attention has been given to examining the quality of user involvement in published studies or funding applications and its impact on the quality of research. ¹ , ⁵ To address this gap, we present a framework for assessing the quality and impact of user involvement in published research and funding applications.

Background

The growth of user involvement

Surveys published by the Consumers in NHS Research Support Group in 2001 revealed a limited involvement of users in UK health research. ⁶ , ⁷ The Support Group was renamed ‘INVOLVE’ in 2003. Since then there has been a steady increase in the number of studies involving users and in the application of participatory and user‐led approaches in health research. ⁸ , ⁹

The revised Research Governance Framework for Health and Social Care (Department of Health, 2005) identified user involvement as a measure of good research practice, stating that ‘Relevant service users and carers, or their representative groups, should be involved wherever possible in the design, conduct, analysis and reporting of research’. ⁴ Similarly the NHS research ethics application form now requires all applicants to detail their user involvement activities, regardless of methodological or disciplinary standpoint. ¹⁰ User involvement is no longer the preserve of psychosocial research, but is a requirement of all forms of academic inquiry, including clinical trials, biomedical research and surveys. Involving users across this broad spectrum of research activity necessitates careful consideration of the rationale for involvement and the application of appropriate strategies for user engagement, as discussed later in this study.

Drivers for user involvement

There are three principal drivers for user involvement in research:

1. The political imperative for the engagement of service users in research

The Department of Health recognized the value of user involvement in research in the 1991 NHS R&D Strategy and has maintained its commitment to engaging users. ¹¹ The political imperative can generate problems where researchers engage with users out of necessity rather than any real or thoughtful commitment to user involvement. ¹²

2. The growth in patient and carer advocacy

Advocates of user involvement argue that there is an ethical and moral right for patients and carers to be informed about and engaged in research activity. ¹² The influence of patient advocacy is particularly pronounced in the US where groups such as the National Breast Cancer Coalition have had a direct impact on the research agenda. ¹³ Similarly, the Alzheimer’s Society and the Multiple Sclerosis Society in the UK have been effective in ensuring that service users are involved extensively in research. ¹⁴

3. The academic community

Members of the academic community who involve users in research are disseminating evidence of the benefits of such engagement. Advantages of user involvement include engaging with patients and carers in prioritizing topics for research ¹⁵ , ¹⁶ and facilitating the implementation of study findings. ¹⁶ Much of this evidence is anecdotal, however, and concerns have been raised over a lack of empirical evidence for the impact of user involvement. ¹¹

The need for appraisal guidelines

Boote et al. ⁵ have published criteria for successful user involvement in research. These criteria include the agreement of roles between users and researchers and appropriate budgeting for involvement activities. ⁵ Whilst these measures are important, attention also needs to be given to assessing the influence of user involvement on research quality in published research and funding applications. ¹⁷ Furthermore, an ideological argument has been proposed that service user involvement in health research involving human participants is a moral and democratic right. ¹² Sustaining a purely ideological approach can lead to tokenism, which whether inadvertent or deliberate undermines the real value and impact of appropriate and well‐targeted user involvement. Ideological concerns should not supersede pragmatic considerations. This study argues that service users should be involved in health research only where they generate real benefit, and the nature of user involvement must not be detrimental to either the user or research quality. There is a need, therefore, to consider pragmatic issues in order to ensure that effective involvement practice and the quality of research are not compromised.

Various strategies have been developed for assessing research quality. The Critical Appraisal Skills Programme (CASP) co‐ordinated by the Public Health Resource Unit in Oxford (http://www.phru.nhs.uk/Pages/PHD/CASP.htm) provides a useful framework for measuring research quality. A component of the programme includes the production of assessment tools designed to enable readers to assess the quality of published research. These assessment tools are developed by a multidisciplinary working group with appropriate expertise and are then piloted.

Informed by the CASP framework, an assessment tool has been developed to assess the quality and impact of user involvement in published research and funding applications (Table 1). The assessment tool has been designed as a collaborative exercise, bringing together commissioners of psychosocial research from Macmillan Cancer Support, researchers with experience of involving users in research, and user involvement expertise at a national level through the National Cancer Research Initiative Consumer Liaison Group and the UK Clinical Research Network. In a similar manner to CASP approach, the assessment tool was initially designed by combining research evidence with relevant expertise and reflecting on the application of each criterion to research, commissioning, and user involvement practice.

Table 1.

 Critical appraisal criteria for assessing the quality and impact of user involvement on health research

Research activity	Appraisal criteria	Write comments here
Planning and project design	1. Is the rationale for involving users clearly demonstrated? Consider the following:  (a) Have the researchers explained the rationale for user involvement?
	2. Is the level of user involvement appropriate? Consider the following:  (a) Have the researchers explained and justified the level of user involvement? (e.g. have they discussed whether the study involves user consultation, user collaboration or user control?) (b) Have the researchers discussed the nature of tasks users were asked to perform (e.g. identifying the research question, selecting the research method, commenting on information sheets, data collection, data analysis, dissemination?)
Recruitment and training	3. Is the recruitment strategy appropriate? Consider the following:  (a) Have the researchers explained how users have been identified? (b) Have attempts been made to involve a wide cross‐section of interests where appropriate (e.g. ethnic minorities, age, gender)? (c) Have the researchers discussed the ‘credentials’ of the users involved? (E.g. Do the researchers discuss why the users involved are appropriate to meeting the aims of the involvement activity?)
	4. Is the nature of training appropriate? Consider the following:  (a) Have the researchers discussed the nature of the training provided? (b) Is the nature and extent of the training justified by the researchers? (e.g. Do the researchers discuss how the training meets the needs of the users during the course of the study?) (c) Has an account been given of user involvement training for professional researchers, where necessary?
Data collection and analysis	5. Has sufficient attention been given to the ethical considerations of user involvement and how these were managed? Consider the following:  (a) Do the researchers discuss ethical issues relating to the involvement of users in research (e.g. fatigue, the emotional demands of data collection)? (b) Are there any discussions about the management of ethical issues (e.g. provision of adequate information about research tasks, peer supervision)?
	6. Has sufficient attention been given to the methodological considerations of user involvement and how these were managed? Consider the following:  (a) Have the researchers discussed methodological issues relating to user involvement in research (e.g. potential impact on the quality of the data)? (b) Do the researchers discuss how methodological issues are managed (e.g. how differences in interpretations of qualitative data are negotiated?)
Dissemination	7. Have there been any attempts to involve users in the dissemination of findings? Consider the following:  (a) Have users been involved in the writing of the publication / funding application? (b) Have the researchers described how the findings have been disseminated to participants and service users? (c) Are findings disseminated appropriately where necessary (e.g. translation of findings into different languages, provision of interim findings to participants in receipt of palliative care)?
Evaluation and impact assessment	8. Has the ‘added‐value’ of user involvement been clearly demonstrated? Consider the following:  (a) Do the researchers discuss what difference involving users in the design and conduct of the research has made to the research process? (I.e. Have the researchers considered whether the study and findings would look any different if users were not involved?) (b) Do the researchers support the claims for the benefits of user involvement with examples from the research project?
	9. Have there been any attempts to evaluate the user involvement component of the research? Consider the following:  (a) Have the researchers discussed the evaluation of the impact of user involvement on the research project (e.g. impact on the length of the study, the financial cost of involvement activities, cost‐benefit analyses)? (b) Do the researchers support claims about the impact of user involvement with examples from the evaluation?

Research evidence used to inform the assessment criteria included the activities of the two Macmillan Research Units. The Southampton Macmillan Research Unit conducted the Macmillan Listening Study, a UK wide research prioritization exercise involving cancer patients and carers as co‐researchers. ¹⁶ , ¹⁸ , ¹⁹ The Manchester Macmillan Research Unit worked closely with service users to identify research priorities, which formed the core of their research programme. ²⁰ The assessment tool has the following applications:

• 1 A set of guidelines to enable readers to assess the quality and impact of user involvement in published studies.
• 2 A set of guidelines that can be used by researchers to ensure effective strategies for user engagement are employed in research.
• 3 A set of guidelines for use by funding bodies to enable them to establish principles of effective user involvement in their own practice and to assess the quality of user involvement in applications. This is particularly important as evidence suggests that, with certain notable exceptions (such as the MS society, the Medical Research Council and the Alzheimer’s society), many funding organizations do not have effective user involvement processes in place. ¹⁴ , ²¹

Assessing the quality of research involving service users

This section provides detail and explanation of each criterion comprising the tool for assessing the quality and impact of user involvement in published research and funding applications (Table 1). Several issues should be noted to assist the application of the criteria. First, it is acknowledged that the term ‘user’ is contentious. The term ‘user’ here refers to members of the general public, patients, care‐givers, potential patients and public, community and voluntary organizations. Whilst the expertise cited to justify the criteria is derived predominantly from cancer and palliative care research, the term ‘user’ is used to refer to people engaged in all forms of health research. Second, the term ‘researcher’ in Table 1 is used to refer to any individual engaged in undertaking research activity, whether they are users or professionals, unless stated otherwise. For example, the term ‘researcher’ also refers to the user–researchers who are leading user‐controlled research.

Third, no equivalence in the importance of each criterion is implied. The relative importance of each criterion is dependent on the context of its application. For example, when assessing the quality of user involvement in research proposals, consideration of the rationale, and level of user involvement is more important than dissemination practices, which can be negotiated after contracting.

Similarly, it could be argued that the rationale of user involvement is more important than the quality of training given that training is of little relevance if service users are involved inappropriately. Furthermore, the relevance and applicability of each criterion is also influenced by the research method and level of involvement. Hence the training criterion is less relevant where forms of engagement are limited (e.g. commenting on information sheets compared with more developed involvement through action research).

The relative nature of the criteria demonstrates certain differences between applying the assessment tool to research proposals and applying the tool to published research. When assessing the quality of user involvement in funding proposals, the purpose of the criteria is to indicate the level of skill of the research team and the quality of the proposed framework for engaging with users. Hence there is potential for negotiation during the contracting process and, indeed, the criteria can be used by the commissioner to monitor the progress of the research in relation to user involvement. Conversely, there can be no negotiation once the research has been completed and published, and hence the criteria in Table 1 should be used as a one off assessment of the quality of user involvement. In this regard, the criteria are designed as guidelines to be applied flexibly to research papers and applications.

1. Is the rationale for involving users clearly demonstrated?

There is often an assumed benefit to user involvement in research with little consideration as to why and how patients and carers should be involved. The emphasis on involving users in ensuring good research practice and maintaining ethical standards has tended to result in perfunctory and tokenistic ‘tick‐box’ forms of engagement. ¹² This can be unsatisfactory for the service user, who is unclear about their role and feels poorly consulted, and for the research team who may not maximize the potential of user involvement in their study. There may also be a tendency for funding applicants to play the ‘user‐card’ to increase their chances of success. ²² Therefore, it is essential for research teams to be clear about why they wish to involve service users before approaching patients and carers, and to make this rationale clear in published papers and funding applications. Similarly, the rationale for not involving service users in research should also be documented.

2. Is the level of user involvement appropriate?

There are numerous strategies for involving users in research. Hanley et al. ¹ identified three levels of user involvement ranging from consultation (asking for users’ views to inform decision making), through collaboration (sustaining an active partnership with users throughout the research process), to user‐control (where the research initiative and decision making lies with service users rather than professional researchers).

It is important that the level of user involvement is appropriate for the aims of the research. The level of involvement inevitably influences the nature of the tasks users are expected to undertake, and these need to be proportionate to the aims of the study. For example, whilst it may not be feasible for users to be involved in data collection or analysis in basic science research, it may be appropriate to involve them in revising information sheets or assisting with dissemination. In addition, service users may be involved in research at a level that is inappropriately high (e.g. where data collection in palliative care research may be too physically demanding). The level and nature of user involvement should be considered carefully and made explicit.

3. Is the recruitment strategy appropriate?

It is important to understand where and how service users have been approached. Recruiting users via inappropriate avenues may generate undue influence on the nature, direction, and outcomes of the research process. It has been suggested that researchers tend to select ‘well behaved’ users who are unlikely to contradict their views. ³ It is necessary therefore that researchers detail the processes through which users have been identified, approached and involved.

Stevens et al. ²³ have commented on the tendency in cancer research to involve service users from pre‐existing support groups. Support group members, however, may not reflect the experiences and interests of study participants. Service users identified through support groups are typically white, middle‐class, middle‐aged people who are urban based, have more common conditions and are relatively fit. Hence those from diverse ethnic minority backgrounds, older or younger people and those with a lower socio‐economic status are likely to be excluded, as are those from rural locations, those with rarer conditions or those in the palliative phase of their illness. ²³ Attempts must be made therefore to involve diverse service users where appropriate. ¹⁸ In the Macmillan Listening Study, for example, there was recognition that there were no representatives from diverse ethnic minorities or people receiving palliative care services at the initial project meeting. Consequently, users from diverse ethnic minority backgrounds were specifically approached via community organizations and users who were in receipt of palliative care services were approached through participating hospices to ensure broad representation was achieved. ¹⁸ , ¹⁹

4. Is the nature of training provided appropriate?

Research activity is often technically demanding, requiring specific skills gained through experience. It is not surprising therefore that training should be listed as an important criterion in assessing success in user involvement. ⁵ It is essential that the nature of training provided to service users is developed carefully and made explicit. It is important to document, for example, whether the training provided has been accredited. How has the training been integrated into the running of the study? Is the training flexible and responsive to meet the changing needs and concerns of users as they gain experience during the course of a study? Similarly, the quality of training for professional researchers concerning user involvement needs to be made explicit, particularly where the user involvement expertise of the research team is unclear. This was noted as an important success criterion for user involvement by Boote et al. ⁵ User involvement training for professional researchers has an impact on the quality of research for, where the rationale for user involvement is clear, poor practices of user involvement from the research team can lead to suboptimal involvement, thus affecting research quality.

5. Has sufficient attention been given to the ethical considerations of user involvement and how these were managed?

Ethical considerations are as important for service users involved in research as they are to participants. However, it is surprising that these concerns are often overlooked. Table 2 illustrates the ethical issues collated through our experience of user involvement in cancer research. Many of these issues are similar to those of participants, for example, ensuring service users are fully informed and are able to leave the study at any time of their choosing. Other issues, such as the availability of peer‐supervision or counselling where necessary, are not specific to user involvement but should be available to all researchers as part of good research practice. It is necessary for any study to adhere to principles of good ethical conduct when involving service users and any ethical problems that occur and how they are managed should be documented.

Table 2.

 Ethical considerations of user involvement in research

1	Are service users fully informed about the study before agreeing to be involved?
2	Are service users able to ‘opt out’ of the study?
3	Are service users well enough to participate?
4	Are service users over‐committed with other research?
5	How will service users’ details be kept?
6	Will service users’ expenses be met by the project’s budget?
7	Will service users become distressed by taking part in research?
8	Will service users be provided with peer supervision/counselling support where necessary?

6. Has sufficient attention been given to the methodological considerations of user involvement and how these were managed?

Involving service users in research can place specific methodological demands on the research process. For example, what are the methodological issues generated by involving a patient as an interviewer or focus group moderator who may know the participant through the clinical setting? How can fatigue be managed during data collection to ensure the quality of the data is not compromised? In research involving people in receipt of palliative care services, for instance, it has been shown that a highly collaborative approach between user–researchers and professional researchers can assist in the management of fatigue. ¹⁹ This approach involved strategies such as the use of visual cues during data collection to indicate where additional support was required from the professional researcher. ¹⁹ These strategies clearly respond to an ethical requirement as well as enhancing the quality of data collection.

Methodological considerations also relate to data analysis. For example, if patients or carers are involved in data analysis, then how should differences occurring between their analysis and that of the ‘professional’ researcher be managed? These considerations need to be documented and reported in research grant applications and publications. ¹⁹

7. Has there been any attempt to involve users in the dissemination of findings?

Disseminating findings to users involved in research as participants or collaborators is recommended in the Research Governance Framework for Health and Social Care as a component of good research practice. ⁴ However, it is important that findings are provided appropriately. For example, MacNeil and Fernandez ²⁴ suggest that not all participants wish to receive a summary of findings of research they have been involved in. Participants from diverse ethnic backgrounds should have access to findings in their preferred language where feasible and appropriate. Participants near to the end of life may not be alive when the study finishes, and thus interim findings may need to be disseminated to this population group before data analysis is complete. ¹⁹

The dissemination of findings is also a consideration for editorial committees. It is often a requirement for journals that accepted papers do not report data that is already in the public domain. Given the time it takes for generating, reviewing, amending, proofing, and printing papers, it is an important consideration whether this requirement conflicts with the recommendation that research findings should be made available to study participants.

8. Has the‘added‐value’of user involvement been demonstrated clearly?

It is important that researchers report the impact user involvement has had on the research process. Reporting this information is vital as it enables evidence of the potential benefits and challenges of involving users to be shared. It has been shown, however, that researchers under‐report user involvement activity in their publications. ²⁵ This is often because user involvement activity is perceived to be of less importance in a paper than other concerns. Given the word limits imposed by journals, user involvement activity is therefore more likely to be omitted in favour of other details. This is significant as not reporting user‐involvement activity limits the potential of learning from practice. ²⁵

9. Have attempts been made to evaluate the user involvement component of the research?

As stated previously, evidence on the benefits of user involvement is often anecdotal with little attention given to evaluating service user activity. The effectiveness of how service users have been identified, trained and supported and the impact they had on the quality and outcomes of the research is key if the user involvement agenda is to be developed and supported in the future. It is important for such an evaluation to be an integral part of studies involving service users.

Discussion

The generation of criteria for assessing the quality and impact of user involvement in research and funding applications is distinct from similar activity reported in recent publications. Assessment criteria to date have typically focused on the quality of user involvement rather than the impact on the quality of the research. ⁵ Boote et al., for example, developed a range of criteria for user involvement, including ensuring the roles of consumers are agreed collectively, ensuring there is appropriate budgeting, and sustaining effective communication with service users over the course of a study. Contrasting criteria reported in Table 1 with those discussed by Boote et al. ⁵ reveal certain similarities, such as clarity of the role of the user in research, training for users and researchers, and dissemination. However, notable differences also exist. For example, the rationale for user involvement, the recruitment strategy, ethical considerations, and an assessment of methodological impact are not cited in Boote et al.’s criteria. ⁵ These differences can be attributed to a focus on assessing the impact of user involvement in research and funding applications is distinct as opposed to defining the success of user involvement. ⁵

The differences in criteria are more significant than that of differing aims and objectives. The criteria illustrated in Table 1 represent a particular ideological perspective that differs from the belief that user involvement is a moral and democratic right regardless of methodological impact. ¹¹ Many of the criteria discussed in this study are not relevant for those adopting a democratic ideological perspective, where issues of added value and rationale as defined in Table 1 are redundant. In this regard, the criteria reported in this study are appropriate for all models of user involvement – consultative, collaborative, or user controlled – where such engagement is viewed as a pragmatic, as opposed to an ideological, concern. ¹ This study presents a series of pragmatic concerns that have been developed to enhance effective user involvement practice and the quality of research.

Conclusion

User involvement in research has developed significantly in recent years. Whilst resistance to engaging with users remains, user involvement is now established as a core component of good practice for all research activity. Responding to the call for engaging users necessitates the generation of guidelines for effective involvement practices. This study develops previous work in the area by providing guidelines for assessing the quality and impact of user involvement in published studies and funding applications. Examining the impact of involving users in this manner is essential if the user involvement agenda is to be maintained and supported.

Conflict of interest statement

The corresponding author has had full access to all the data in the study and has final responsibility for the decision to submit for publication. There are no conflicts of interest for any of the authors in this paper.

Role of the funding source

The guidelines were formulated in part on the basis of studies and user involvement activities funded by Macmillan Cancer Support and the National Cancer Research Institute.