Abstract
Objectives
To identify the extent to which the Cochrane Collaboration involves consumers (patients, carers, patient and non‐patient members of patient/consumer organizations) as members of Cochrane Review Groups (CRGs); to explore the emphasis CRGs place on identifying and collecting information on outcomes identified by patients as being important indicators of quality and effectiveness of treatment and care (‘patient‐defined outcomes’).
Method
A postal questionnaire designed by The College of Health, a UK patient organization, was sent in January 1998 to all CRGs registered with the Cochrane Collaboration on 1 January 1998 (n = 42).
Results and conclusions
Replies were received from 35 CRGs, a response rate of 83% and 33 questionnaires (79%) were completed. The survey revealed that CRGs varied in the extent to which they had recruited consumer members: almost one third of respondents said their CRG did not have any consumer members. There was also no apparent consensus across CRGs on the importance attached to identifying and collecting information on patient‐defined outcomes or on integrating such information into their activities. It is hoped that differences between CRGs may inform discussions as to whether and how the Cochrane Collaboration might address the issue of patient‐defined outcomes in the future.
Keywords: Cochrane collaboration, consumer involvement, patient outcomes
Introduction
Providing a clinically effective service means drawing on the best evidence available to ensure that resources are used for treatments and care, which give the best possible outcomes for patients and are not wasted on ineffective procedures. 1 Difficulties sometimes arise however, in determining what constitutes ‘evidence’ and ‘outcomes’.
Outcomes, both desired and actual, can be defined by a range of different interests, including clinicians, managers, policy makers, researchers, patients, families and carers. Most primary research however, is undertaken from the perspective of clinicians and researchers. Whilst these groups may be good judges of success on the basis of precise clinical tests and measures, patients and clinicians sometimes differ in regard to what counts as effective treatment and what constitutes appropriate outcome. 2 , 3, 4, 5, –6 Differences between patients further complicate matters. Using angina as an example, a patient who is normally physically active may be more distressed by having mild angina than a sedentary patient with more severe symptoms who is just happy to slow down. Other factors may affect patients’ desired outcomes, for example whether the patient is in work, the kind of work, or differing levels of anxiety about the implications of having angina. 7
Alongside growing demands for improved access to up‐to‐date, evidence‐based information to guide clinical decision making, there has been a growing emphasis on involving health care consumers (patients, service users, carers, patient representatives and the public) in health care research. The scope for involvement is not limited to ensuring that patients and their families have access to information to help them make informed choices. Examples of consumer contributions also relate to identifying important questions and relevant outcomes, drawing up priorities for research topics, appraising protocols, recruiting and preparing information for participants, undertaking research and interpreting findings. 8
Providing consumers with opportunities to contribute to research agendas and programmes is seen as a way ‘to improve the way research is prioritized, commissioned and disseminated’. 9 Emphasis is on maximizing the input of patient, and sometimes carer, expertise to inform the definition of outcomes. Furthermore, it has been argued that ‘for patients to be able to make informed choices, they need access to full information on outcomes which in turn must be partly based on patient definitions of good and poor outcome’. 10 A distinction therefore emerges between patient‐centred outcome dimensions and measures that have been determined solely by health professionals and researchers and patient‐defined outcomes (outcomes identified by patients and service users themselves as being important indicators of quality and effectiveness of treatment 11 ). In a keynote speech in 1998, John Swales, Director of Research and Development in the UK Department of Health, described how clinical research now distinguishes between explanatory trials, which focus on identifying a specific effect of a drug, for example, and pragmatic trials which evaluate the impact of treatment in the setting where it is delivered. ‘The pragmatic trial’ he argued, ‘is now inseparable from evidence‐based care. Its development has led us to recognize the essential part played by patients and carers’. 12
The Cochrane Collaboration is an international network of individuals committed to preparing, maintaining and disseminating systematic, up‐to‐date reviews of health care. It has played a significant role in improving access to information about evidence‐based healthcare. The Collaboration has acknowledged the importance of involving consumers in all aspects of its work, seeking to avoid professional/researcher bias, to ensure that Cochrane reviewers consider issues of significance to consumers and to identify gaps where consumer issues have not been addressed. 13
Within the Cochrane Collaboration, Cochrane Review Groups (CRGs) are responsible for identifying randomized controlled trials relevant to their area of interest and producing systematic reviews of the evidence. Cochrane Review Groups seeking registration are required to have explicit plans for consumer involvement. This paper reports on a survey of CRGs which aimed:
• to identify the extent to which consumers participate as members of CRGs;
• to explore the emphasis CRGs place on identifying and collecting information on outcomes identified by patients as being important indicators of quality and effectiveness of treatment and care (‘patient‐defined outcomes’).
Methods
A questionnaire was designed at the College of Health, a national UK charity set up to represent the interests of patients and promote patient‐centred care. The project was carried out by a senior researcher with a nationally recognized background in consumer involvement issues. 14 , 15 Following comments received from the UK Cochrane Centre (UKCC) and piloting with one CRG, some minor amendments were made.
The following information was requested from all respondents:
The name of the review group;
The number of consumer members on the CRG;
The type(s) of consumers recruited as members of the CRG. Respondents were asked to indicate which of the following categories best described their consumer member(s):
• patient with a condition relevant to the scope of the CRG,
• carer of a patient with a relevant condition,
• patient member of a relevant patient/consumer organization,
• non‐patient member of a relevant patient/consumer organization,
• other.
Six yes/no questions asked respondents to indicate whether the CRG had undertaken work in any of the following areas:
• discussed collecting information on patient‐defined outcomes,
• carried out a search literature on patient‐defined outcomes,
• identified key literature on patient‐defined outcomes,
• produced a bibliography, summary or review on patient‐defined outcomes,
• incorporated literature on patient‐defined outcomes into wider systematic reviews,
• liaised with relevant patient representatives, patient groups or organizations to collect information on patient‐defined outcomes.
Where respondents ticked ‘yes’, they were asked to provide details with space provided for free text responses.
Respondents were also asked to provide details of the CRG’s intention to carry out any future work on patient defined outcomes (free text responses);
Respondents whose CRGs had addressed patient‐defined outcome issues were asked:
• to indicate who had main responsibility for collecting information in this area (yes/no answers were requested for each of four response options: all CRG members working together; a subset of CRG members; individual CRG member(s) working independently; external agency commissioned to carry out this work),
• to rate the part played by consumer representatives in collecting information on patient‐defined outcomes (response options: lead role; major role; minor role; no role),
• to indicate (yes/no) if patient organizations had provided information on this issue to indicate the usefulness of different sources of literature on patient‐defined outcomes: databases such as MEDLINE; hand searching; literature identified by specialist libraries, professional experts, patient representatives or groups, telephone helplines; other sources. A three‐point rating scale plus ‘source not used’ option was provided for each item).
Respondents from CRGs with no plans to review literature or collect information on patient‐defined outcomes were asked to explain why (free text responses)
A letter explaining the project and a copy of the questionnaire was mailed by UKCC administrative staff to the CRG coordinator of all formally registered CRGs known to the UKCC in January 1998. One reminder letter and questionnaire was sent in April 1998 to each CRG which did not respond to the first mailing. Completed questionnaires were returned to the College of Health and responses were analysed by the author. Yes/no and rating scale answers were quantified. Open responses to questions requesting details of activities as well as unsolicited free comments made on the questionnaire were listed by the author. Where made, these have been used in the results section to illustrate the different views expressed by respondents.
Results
Sample and response rates
Questionnaires were sent to 42 CRGs and responses were obtained from 35 (83%). Two respondents said their groups were too recently convened to be able to answer the questions, leaving 33 completed responses, a questionnaire completion rate of 79%. This includes one respondent who addressed the issues raised in the questionnaire by letter.
Consumer membership of CRGs
The numbers of CRGs with consumer members are shown in Table 1. Ten CRGs (almost one third) had no consumer members. However, additional unsolicited information provided by some respondents in this category revealed that one of these CRGs was actively looking for consumer editors and three expressed an intention to recruit ‘in the future’. Whilst two of the groups without consumer membership had only registered with the Cochrane Collaboration in 1998, allowing little time for recruitment of consumer members, the other eight CRGs had been registered for 12–51 months (mean 32.1 months). For CRGs with consumer members, the number ranged from 1 to 10 (mean 3.6). Six CRGs (30% of all respondents) had one or two consumer members and 12 CRGs (36%) had three or more.
Table 1.
Consumer membership of Cochrane Review Groups
CRG work on patient‐defined outcomes
Respondents were asked what consideration they had given to collecting information on patient‐defined outcomes:
• 19 respondents (58%) indicated that they had discussed the issue;
• Five respondents (15%) had carried out a search for literature on patient‐defined outcomes and three of these (9% of the total) had produced a bibliography, summary or review. One of these and four other CRGs had incorporated literature on patient‐defined outcomes into wider systematic reviews.
• 12 respondents (36%) said their CRG had liaised with patient representatives, patient groups or organizations to identify or collect information on patient‐defined outcomes and two more CRGs had identified relevant organizations to contact about this.
Eight of the 12 respondents whose CRGs had liaised with consumers to identify or collect information on patient‐defined outcomes rated the importance consumers had played. Six felt that consumer representatives had played a major role and two were hoping that their role would develop over time. Reported contributions included informing the methodology, development and reporting of reviews, participation in working groups, suggesting outcomes and/or identifying areas of interest that patients would like the CRG to address.
Free comments
Most respondents made free comments on the questionnaire and these fell into three broad categories:
• The committed in practice: respondents whose CRGs had already started work on looking at patient defined outcomes (four CRGs, 12%);
• The committed in theory: respondents who expressed an interest in pursing work on patient‐defined outcomes in the future (22 CRGs, 66%);
• The unconvinced: respondents who had no intention at present to carry out any specific work on patient defined outcomes and/or had reservations about the relevance of this work (six CRGs, 18%).
The committed in practice category
Four respondents said their CRG had already carried out work identifying and collecting information on patient‐defined outcomes. The ‘Consumer and Communication’ group was in this category: it is policy for their reviewers to identify how they are going to assess patient‐defined outcomes as part of their methodology and the group has had consumer input to a list of outcomes incorporated into its scope. Three condition‐specific CRGs also provided examples of work carried out:
[The CRG has] commissioned a hospital consumer network to collect information on patient‐defined outcomes; sought consumer views on the development and reporting of reviews; is looking for grants to support a study testing different models of getting consumer involvement and patient‐defined outcomes.
[We have] discussed collection of patient‐defined outcomes with [a national patient organization]; individual reviewers liaise with patient groups on ad hoc basis; [we are] developing an international panel of consumer representatives to help identify patient‐defined outcomes at the outset and as part of the editorial process.
[The CRG has] held a one day workshop with consumer groups; meets regularly with consumer representatives and support groups; intends asking consumer representatives to tell us what important patient‐defined outcomes there are in their particular disease/condition.
The committed in theory category
Twenty‐two CRGs expressed an intention to carry out work on patient‐defined outcomes although some had reservations. Two respondents queried the questionnaire’s focus on patient‐defined outcomes, arguing that the brief for CRGs was not to search for outcomes but to identify all randomized controlled trials relevant to their topic areas. Another respondent queried the need to make specific attempts to collect information on patient‐defined outcomes, suggesting they would be picked up in the course of a general search.
Three respondents were unsure how to tackle the issue of patient‐defined outcomes and hoped to get some ideas from the survey and two more planned to discuss how to take this forward at a future meeting.
Other CRGs were already considering possible strategies for incorporating patient‐defined outcomes into their work. As a minimum, this usually involved discussing the issue with consumers (usually patient organizations) but some groups had more detailed plans:
We intend to provide basic training of patients and consumers to understand the work of Cochrane and [the] aim of reviews. Then to determine patient‐defined outcomes in [our] field and incorporate them, as standard, into reviews produced by CRG thereafter.
[We] plan to contact relevant organizations for input into specific reviews. Also to communicate activity of the group to these organizations via newsletters. No detailed plans as yet for literature searching.
Our CRG business plan includes:
• establishing links with consumer, carer and patient groups and their representatives;
• having a consumer/lay editor for all reviews (became group policy in October 1997);
• incorporating a consumer agenda to the CRG business plan.
A colleague is developing a patient‐defined quality of life measure. We intend to conduct a clinical trial using this tool.
Five of the 14 CRGs committed to the concept of incorporating patient‐defined outcomes in their reviews expressed concerns about locating information, about handling information and most fundamentally, about its availability in the first place:
So far we have resorted to asking patient support groups for their views at protocol stage. [We intend] to search for any published literature on patient defined outcomes – any helpful tips on search terms and which databases are likely to be fruitful would be gratefully received.
Patient organization has suggested different areas of interest to patients which they would like to see addressed – I am not sure how to set about developing a relevant search strategy in order to identify the literature on patient‐defined outcomes. Any suggestions?
We seek to include any RCT that contains a measured variable…. One of the major difficulties we have encountered is the lack of standardization in patient‐centred and patient‐defined outcomes. Lack of standardization means that qualitative data are extremely difficult to handle.
[We have] liaised with patient organizations. Recognized the need for patient‐defined outcomes but have concerns about availability of information. If it is not available this will be made clear in the discussion section [of the review]. A major problem is lack of collection of patient‐defined outcomes in completed published trials. We need to point out the lack of data and make sure future trials collect as necessary – in some cases it is too late.
We are very keen to get consumer input and are planning some discussion groups – it may be that we can use this forum to address issue of patient‐defined outcomes. We are somewhat influenced by the outcomes that original RCTs looked at. You need patient involvement in planning RCTs and the outcomes measured.
Not aware of existence of literature on patient‐defined outcomes. Possibly easier to recruit a sample population and ask them directly about patient‐defined outcomes.
The unconvinced category
Respondents from six CRGs, four of which had no consumer members, explained why they had not made any attempt to collect information on patient‐defined outcomes:
Not sure about its future relevance.
Individual reviewers are likely to pick up some of the issues but we are not making an issue of this at present – [working] flat out with a thoroughly stretched infrastructure.
Resource limitations dictate we confine our work to finding RCTs rather than reviewing literature on patient‐defined outcomes.
No plans to carry out work on patient‐defined outcomes at present: lack of resources; lesser priority.
We search for diseases rather than outcomes so all outcomes, patient‐defined or not, would be picked up.
We have decided on our long‐term outcomes.
Discussion
The relevance of much research that has been driven solely by professionals and academic interests is being questioned with a corresponding emphasis on the role healthcare consumers can play in providing insights and expertise that complement, sometimes even counterbalance, those of healthcare professionals and researchers. 8 By systematically reviewing clinical trials, the Cochrane Collaboration has a crucial role to play in evaluating the quality of primary research on which those reviews are based. Consumer input can help ensure that reviewers not only consider outcomes identified by clinicians and researchers but also consider the extent to which primary research has addressed outcomes identified by patients as being important indicators of quality and effectiveness of treatment and care.
The Cochrane Collaboration has always expressed a willingness to collaborate with consumers. It has a ‘consumer network’ which aims to support consumer participation and has already addressed issues relating to consumer involvement in the structures of the Collaboration. 13 Problems facing the Collaboration are similar to those of other organizations wanting to improve consumer involvement. They include when and how to involve consumers as well as how to achieve a strategic and consistent approach across a range of activities within the Collaboration.
The survey described in this paper aimed to provide some insight into these issues. Two specific components of consumer input were explored: consumer membership of CRGs and the extent to which outcomes identified by patients as being important indicators of quality and effectiveness of treatment and care (‘patient‐defined outcomes’) inform the work of CRGs.
The survey revealed that CRGs vary in the extent to which they have recruited consumer members. Given the Collaboration’s commitment to consumer participation, it was disappointing that almost one third of CRGs in the survey did not have any consumer members.
The survey revealed that more than half of CRGs had at least discussed the issue of patient‐defined outcomes (meaning that almost half had not done so). Respondents also expressed different views on devoting time to patient‐defined outcomes. At one end of the scale, some CRGs had searched for literature on patient‐defined outcomes and produced reports or incorporated the information into wider systematic reviews. At the other end of the spectrum, respondents queried the relevance of looking for patient‐defined outcomes.
Amongst respondents who did attach importance to patient‐defined outcomes, there were some reservations about the existence of published literature on patient‐defined outcomes and a concern about the lack of standardization where data were collected. More than a third of respondents viewed consumers as an important source of information on patient‐defined outcomes. Where approached, consumer contributions were viewed positively by respondents.
The survey results suggest that there is currently no consensus amongst CRGs on the importance attached to identifying and collecting information on patient‐defined outcomes or on integrating such information into their activities. However, any plans for future work might usefully be informed by different methods already used by CRGs described as ‘committed in practice’ and by some of the ideas proposed by those ‘committed in theory’.
This survey provides no definitive answers as to whether or how to tackle patient‐defined outcomes. In reviewing current thinking and activity by CRGs on this issue, the findings provide some food for thought on different ways in which work in this area might progress in the future. On the basis of the Collaboration’s own criteria for consumer involvement, two recommendations for early consideration by the Collaboration and its CRGs are proposed:
Opportunities for consumers to influence the agenda and activities of CRGs would be greatly improved if all CRGs were to recruit consumer members;
The Collaboration should develop a more consistent approach to the issue of patient‐defined outcomes, including consideration of whether or not: (a) to attempt to identify patient‐defined outcomes to inform the review process; (b) to point out, where relevant, the lack of data on patient‐defined outcomes in completed trials; and (c) to encourage future trials to collect information on patient‐defined outcomes.
It is hoped that decisions about future work in this area would themselves be taken in consultation with consumers.
Acknowledgements
The author would like to thank Iain Chalmers for his support for this survey, Barbara Meredith for her useful comments on the manuscript, Cath Milwain for mailing questionnaires and Kate Oldfield for help in piloting the questionnaire. Funding for the author’s post is provided by the Department of Health.
References
- 1. National Health Service Executive . Promoting Clinical Effectiveness: a Framework for Action in and Through the NHS Leeds: NHS Executive, 1996.
- 2. Williamson C. Whose Standards: Consumer and Professional Standards in Health Care Buckingham: Open University Press, 1992.
- 3. Byrne M. Cancer chemotherapy and quality of life. British Medical Journal, 1992; 304 : 1523 1524. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Godfrey M. User and carer outcomes in mental health. Outcomes Briefing (Nuffield Institute for Health, University of Leeds), 1996; 8 : 17 20. [Google Scholar]
- 5. Greenhalgh J. (The role of) User outcomes within diabetes care. Outcomes Briefing (Nuffield Institute for Health, University of Leeds), 1996; 8 : 10 16. [Google Scholar]
- 6. Jachuk SJ, Brierly H, Jachuck S, Wilcox PM. The effect of hypotensive drugs on the quality of life. Journal of the Royal College of General Practitioners, 1982; 32 : 103 105. [PMC free article] [PubMed] [Google Scholar]
- 7. Hlatky MA. Patient preferences and clinical guidelines. Journal of the American Medical Association (Editorial), 1995; 273 : 1219 1220. [PubMed] [Google Scholar]
- 8. Entwistle V, Renfrew M, Yearley S et al. Lay Perspectives: advantages for health research. British Medical Journal, 1998; 316 : 463 466. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Standing Advisory Group on Consumer Involvement in the NHS Research & Development Programme . Research: What’s in it for consumers? Leeds: NHS Executive, 1998.
- 10. Neuberger J. The public face of outcomes. Health Service Journal, 1993; 103 (5341): 19 19. 10126010 [Google Scholar]
- 11. Dixon P & Long AF. Broadening the criteria for selecting and developing health status instruments. Outcomes Briefing (Nuffield Institute for Health, University of Leeds),1995; 6 : 4 14. [Google Scholar]
- 12. Swales J. Keynote Speech: Consumer Involvement in Health Research – the Research Perspective. Research, What’s in It for Me? Conference Report. London: Department of Health, 1998.
- 13. Bastian H. The Power of Sharing Knowledge: consumer participation in the Cochrane Collaboration Oxford, UK: Cochrane Centre, 1994.
- 14. Kelson M. User Involvement: a Guide to Developing Effective User Involvement Strategies in the NHS. London: College of Health, 1997.
- 15. Kelson M. Promoting User Involvement in Clinical Audit: Practical Guidance on Achieving Effective Involvement. London: College of Health, 1998.