Consumer involvement in systematic reviews of comparative effectiveness research

Abstract

Background  The Institute of Medicine recently recommended that comparative effectiveness research (CER) should involve input from consumers. While systematic reviews are a major component of CER, little is known about consumer involvement.

Objective  To explore current approaches to involving consumers in US‐based and key international organizations and groups conducting or commissioning systematic reviews (‘organizations’).

Design  In‐depth, semi‐structured interviews with key informants and review of organizations’ websites.

Setting and participants  Seventeen highly regarded US‐based and international (Cochrane Collaboration, Campbell Collaboration) organizations.

Results  Organizations that usually involve consumers (seven of 17 in our sample) involve them at a programmatic level in the organization or in individual reviews through one‐time consultation or on‐going collaboration. For example, consumers may suggest topics, provide input on the key questions of the review, provide comments on draft protocols and reports, serve as co‐authors or on an advisory group. Organizations involve different types of consumers (individual patients, consumer advocates, families and caregivers), recruiting them mainly through patient organizations and consumer networks. Some offer training in research methods, and one developed training for researchers on how to involve consumers. Little formal evaluation of the effects of consumer involvement is being carried out.

Conclusions  Consumers are currently involved in systematic reviews in a variety of ways and for various reasons. Assessing which approaches are most effective in achieving different aims of consumer involvement is now required to inform future recommendations on consumer involvement in CER.

Background

Consumer involvement has been placed high on the comparative effectiveness research (CER) agenda in the United States: The Institute of Medicine (IOM) recently recommended that CER should ‘fully involve consumers, patients and caregivers’ ¹ (p. 18). The Patient Protection and Affordable Care Act defined that the Patient‐Centered Outcomes Research Institute (PCORI) will have consumer and patient representatives on the Board, as well as on expert advisory panels, which will identify research priorities and establish a research agenda. ² The Agency for Healthcare Research and Quality (AHRQ) is undertaking a major effort to establish a Community Forum to expand and systematize broad citizen and stakeholder engagement in its CER initiative. ³

Many influential institutions worldwide advocate for direct consumer involvement in health research. ⁴ Within the National Health Service (NHS) in the UK, researchers have been encouraged to involve the public and service users at all stages of the research process since the late 90s. ⁵ Australia’s National Health and Medical Research Council endorsed a strategy on consumer and community participation in health and medical research in 2001. ⁶ Internationally, the Cochrane Collaboration has advocated for the direct involvement of consumers since the collaboration’s foundation in 1993. ⁷

A variety of rationales have been put forward to support consumer involvement. ⁴ Prominent lines of reasoning are that those who are ultimately affected by health research and who pay for it should have a say in the research process ⁸ and that consumer involvement has the potential to improve the quality, relevance and impact of health research – for example, by ensuring that the questions or outcomes addressed in health research are important to consumers. ⁴ , ⁹

In the United States, the Recovery and Reinvestment Act of 2009 allocated $ 1.1 billion to CER, i.e. ‘research assessing the comparative effectiveness of health care treatments and strategies’. ¹⁰ The 2010 Patient Protection and Affordable Care Act went a step further and established the PCORI to conduct comparative clinical effectiveness research, along with a new funding stream for CER. ² CER includes both the generation and the synthesis of research comparing clinical outcomes and effectiveness of items, procedures and services for prevention, diagnosis or treatment. ¹⁰ Systematic reviews, which address a particular research question by identifying, selecting, critically appraising and synthesizing data from original research studies following systematic and explicit methods, ¹¹ are therefore an important component of current investments in health research in the United States.

In this context, the call for consumer involvement in CER raised the question on how consumers can be involved in the systematic review process as a part of the synthesis of information that, in turn, frequently sets additional research agendas. Internationally, the literature on consumer involvement in systematic reviews is still relatively sparse and consists largely of case examples how consumers have been involved in individual review projects. ¹² , ¹³ , ¹⁴ To inform on‐going CER activities, we explored current practices of selected US‐based and key international organizations that conduct or commission systematic reviews with regard to consumer involvement.

Methods

Study design

We conducted semi‐structured, in‐depth interviews with key informants of US‐based and international organizations commissioning or engaged in systematic reviews regarding current practices of consumer involvement. Our work was commissioned by the IOM to inform a report developed by the Committee on Standards for Systematic Reviews of Clinical Effectiveness Research. ¹⁵

Definitions of the term ‘consumer’

Definitions of the term ‘consumer’ are numerous, as are the attempts to classify them, ⁴ and other terms (e.g. service users or the public) may be preferred. ¹⁶ In this study, we used the term ‘consumer’ to include patients, families of patients, lay caregivers, consumer advocates and members of the general public. We were interested in direct involvement of consumers in the systematic review process as opposed to indirect methods of including consumer views, for example, by drawing on previously published surveys among patients.

Selection of sample

We aimed to capture current practices of consumer involvement across a range of organizations and groups (in the following referred to as ‘organizations’) that either conduct systematic reviews or commission them. Our goal was to learn from different types of highly regarded organizations how consumers are currently involved, to obtain in‐depth understanding about their processes, and not to survey a representative sample of all organizations potentially involved in systematic reviews. We included in our sample federal agencies, payer and provider organizations as well as a selection of private and university‐based organizations. Our choice of organizations was informed by the interest of the IOM Committee and also based on the authors’ knowledge of organizations in the United States involved in systematic review production. Additionally, we included groups that had been selected by the IOM to represent the professional societies’ perspective at a Committee meeting in January 2010. We also included the Cochrane and Campbell Collaborations to add an international perspective and specifically interviewed two Cochrane review groups actively involving consumers (Cochrane Musculoskeletal Group, Cochrane Pregnancy and Childbirth Group), in addition to interviewing a Steering Committee Co‐Chair about policy and practice in the Collaboration as a whole (see Table 1 for a list of surveyed organizations).

Table 1.

Organizations contributing data in in‐depth interviews

Organizations commissioning systematic reviews (outsourced or in‐house)1

Agency for Healthcare Research and Quality (Effective Health Care Program; all outsourced) with the Center for Evidence‐based Policy2

Professional societies

American Academy of Pediatrics (in‐house/outsourced)

American College of Chest Physicians (outsourced)

Provider/payer/research organizations

Centers for Medicare and Medicaid Services (in‐house/outsourced)

Office of Medical Applications of Research, National Institutes for Health (outsourced)

US Department of Veterans Affairs (outsourced)

Organizations conducting systematic reviews

US‐based organizations

Johns Hopkins Evidence‐based Practice Center (EPC)

Oregon Evidence‐based Practice Center (EPC)

ECRI Institute (non‐EPC)3

Blue Cross and Blue Shield Association, Technology Evaluation Center (non‐EPC)3

Kaiser Permanente, National Guideline Program4

Hayes, Inc.

Mayo Clinic, Knowledge and Encounter Research Unit

International organizations

Campbell Collaboration5

Cochrane Collaboration (Steering Group)6

Cochrane Musculoskeletal Group

Cochrane Pregnancy and Childbirth Group

¹outsourced = the conduct of the systematic reviews is usually commissioned to external organizations, in‐house = systematic reviews are conducted by people internal to the organization.

²2 interviews were held – one with key informants from Agency for Healthcare Research and Quality (AHRQ), the other one with a key informant of the Stakeholder Engagement Team of the Scientific Resource Center based at the CEbP, which provides scientific support for AHRQ.

³Although these institutes serve as EPCs for AHRQ, the interviews focused on the processes for systematic reviews by non‐federal commissioners as these are the majority of systematic reviews carried out at these organizations. For those systematic reviews commissioned under the EPC‐funding, AHRQ’s processes for consumer involvement apply.

⁴At Kaiser Permanente, systematic reviews are also commissioned.

⁵The information provided in the interview mainly refers to processes at the former Nordic Campbell Center (SFI Campbell).

⁶For the Cochrane Collaboration as a whole, the interview was carried out with a key informant of the Steering Group.

Data collection and analysis

We developed the semi‐structured interview protocol based on questions asked by the IOM Committee. In our study, we were mostly interested in questions related to ‘how are consumers currently involved in systematic reviews conducted or commissioned by selected organizations?’ For the majority of questions we asked, we solicited factual information about procedures used: approaches to involve consumers, the organization’s rationale for involving consumers, the types of consumers involved, recruitment and compensation of consumers, how potential conflicts of interest were handled, training and education offered for consumers and researchers, and formal evaluation of consumer involvement. Other questions drew on key informants’ personal experiences and views, for example their experiences with different types of consumers and their personal impression of the impact of involving consumers. Questions were modified slightly to improve interview flow and more details about consumer involvement after initial interviews. A complete list of topics covered in the interviews is provided in Appendix S1. We distinguished four main stages of the review process where consumers can be involved: (i) topic identification and prioritization; (ii) protocol development; (iii) review conduct; and (iv) translation of the results into a consumer‐friendly language and dissemination. We considered organizations to usually involve consumers if the key informants reported that their organization makes proactive attempts to involve consumers in the systematic review process on a regular basis. Key informants who stated at the beginning of the interview that they do not usually include consumers were asked their views on reasons for not doing so.

One investigator (JK) carried out all interviews and communication with key informants. Before carrying out interviews, we reviewed the selected organizations’ websites for information on consumer involvement and the names of potential key informants, typically top officials from these organizations or people in senior positions who were directly responsible for and very knowledgeable about consumer involvement in their organization. Between January and April 2010, we contacted the selected key informants, or a more appropriate contact to whom we were referred, to arrange a semi‐structured telephone interview. We sent a guide to the key informants before the scheduled interview to give them an opportunity to prepare some of the interview questions, and they gave oral consent to be interviewed. Interviews took up to 90 min, were audio‐recorded and transcribed verbatim by an independent transcription service.

One investigator (JK) used information from the transcripts to present findings from each interview using a table format. For the description of procedures used, we summarized key concepts discussed in the interview for each organization, based on our predefined categories of the main review stages. For the description of an organization’s rationale as well as key informants’ personal experiences with different groups of consumers and their impression of the impact of consumer involvement, we extracted verbatim quotations from the transcripts. Where appropriate, information from the interviews was supplemented by information from the websites. Questions that surfaced during the analysis of the interviews were clarified with the key informants via e‐mail. Completed summary tables, prepared for a comprehensive internal background paper to the IOM, were sent to the key informants for the verification of fact and editing of the summaries and verbatim quotes as necessary. To improve readability and presentation in this article, we further condensed and rearranged the information on the organizations’ procedures from the original tables (see Supporting information). The presentation of informants’ personal impressions focuses on illustrating key points that were raised in the interviews and that were supported by selected verbatim quotations.

The study was funded by the IOM and the Commonwealth Fund although the latter did not have a role in the conduct or reporting of this study. The IOM Committee on Standards for Systematic Reviews of Clinical Effectiveness Research was involved in approving the organizations sampled and the semi‐structured interview protocol. It did not have a role in the analysis and interpretation of data, in writing of the paper, or in the decision to submit the paper for publication. We submitted the study protocol and interview guide to the Johns Hopkins Institutional Review Board, and it was classified as ‘exempt’.

Results

Of 20 organizations approached, two organizations reported that they neither commission nor conduct systematic reviews, and representatives of one organization were not available for an interview in the time frame of the project (see Appendix S2). We therefore collected data from 17 organizations (see Table 1).

Although a majority (10/17) of organizations surveyed do not usually involve consumers in the process of commissioning or conducting systematic reviews, some of them engage consumers occasionally in the systematic review process or they involve them regularly in other parts of their processes (e.g. when making coverage decisions or as public input in consensus conferences). Asked about reasons for not involving consumers, some key informants pointed out, for example, that they did not have the time or resources to do this. Others raised the concern that involving consumers may have a negative impact on the scientific rigour of the review, and some key informants were unsure about how to find the ‘right’ consumer to involve in this process.

In the following, we focus on those organizations (7/17) that usually involve consumers in the systematic review process: the Agency for Healthcare Research and Quality’s Effective Health Care (EHC) Program, including the Center for Evidence‐based Policy (CEbP), which housed the Stakeholder Engagement Team for this programme, the Oregon Evidence‐based Practice Center [EPC], the Johns Hopkins EPC, the Cochrane Collaboration, the Cochrane Musculoskeletal Group, the Cochrane Pregnancy and Childbirth Group, and the Campbell Collaboration. Note that the Campbell Collaboration has a broader concept of consumers than other organizations (see Appendix S3).

Consumer involvement occurs both at a programmatic level and at the level of the individual systematic review (Fig. 1). However, we did not identify consistent or uniform approaches to consumer involvement within or across organizations. In fact, the interviews revealed that the organizations and their policies are intertwined to a considerable extent (e.g. the Cochrane Collaboration Steering Committee sets out policies for the individual review groups) and complement each other across organizations (e.g. for a single systematic review, the EPCs may be responsible for involving consumers at one stage and AHRQ responsible at another stage). Within organizations, approaches to consumer involvement may differ depending on the commissioner of the specific review. Even with the same organization and commissioner approaches evolve constantly, for example because of (loss of) funding or advancement in the methodology. Therefore, we did not make explicit comparisons among organizations and instead used examples to illustrate approaches we found in these organizations.

Figure 1.

 Different forms of consumer involvement in systematic reviews across selected organizations, both one‐time and continuous. AHRQ, Agency for Healthcare Research and Quality; CPCG, Cochrane Pregnancy and Childbirth Group; EPC, Evidence‐based Practice Center.

Approaches for involving consumers at a programmatic level

The Cochrane Collaboration accords consumers two elected seats on the Steering Committee, which is responsible for high‐level organizational decision making (see Fig. 1 and Appendix S4). Within the Cochrane Musculoskeletal Group, consumers are represented on the Editorial Board, the decision‐making body for Cochrane review groups. At AHRQ, consumers are represented in the Stakeholder Group for the EHC Program, a group of 15–20 individuals who represent a variety of constituents, including healthcare providers, payers and consumers. The Stakeholder Group meets regularly and provides input into the programme (e.g. on the topic nomination or selection process).

Approaches for involving consumers in individual systematic reviews

Consumers have provided input at specific stages of an individual review, and they have been integrated into the entire process of producing a review (see Fig. 2). We first describe different approaches of involving consumers in specific stages before we describe how consumers have been integrated across different stages. As no organization covered all possible areas of consumer involvement, we will describe one or more examples of that which appeared to be most comprehensive and structured in each of the categories.

Figure 2.

 Approaches for consumer involvement in different stages of a systematic review at selected organizations (examples). The main stages of a systematic review are Topic identification and prioritization, Protocol development, Review conduct and Dissemination. Consumers are involved in all of these stages – either as a one‐time involvement or as a continuous involvement, for example by being on the author team or on an advisory group. For the full range of approaches applied in the organizations of our sample, see Appendix S4. AHRQ, Agency for Healthcare Research and Quality; CMSG, Cochrane Musculoskeletal Group; CPCG, Cochrane Pregnancy and Childbirth Group; EPC, Evidence‐based Practice Center.

Topic identification and prioritization

Consumers have been involved in the identification of topics in different ways (see Fig. 2 and Appendix S4). AHRQ provides the opportunity for consumers as well as healthcare professionals, researchers, policy makers or others to nominate research topics through the EHC Program’s website, and these may result in a systematic review. The Cochrane Musculoskeletal Group has carried out two ‘brainstorming’ workshops with consumers to identify priorities in musculoskeletal health.

Protocol development

Opportunities for consumer involvement exist both at the beginning and at the end of protocol development (see Fig. 2 and Appendix S4). The Oregon EPC involves consumers as key informants in the development of key questions for a protocol’s analytic framework. The researchers carry out interviews, or sometimes focus groups, to ensure that a review addresses questions that are relevant to consumers. Once a protocol has been drafted by an EPC, AHRQ publishes it on its website for 4 weeks and sends out notifications via an e‐mail list to inform those potentially interested about the opportunity for comment. Within the Cochrane Musculoskeletal and Pregnancy and Childbirth Groups, members of the Consumer Network and Consumer panel, respectively, peer‐review all draft protocols and reviews produced within the Groups.

Review conduct

AHRQ requests public comment using website postings, and Cochrane policy suggests consumer involvement in peer review (see Fig. 2 and Appendix S4). The Johns Hopkins EPC and Cochrane groups in our sample each involve consumers as peer reviewers for completed draft reviews. Consumers have also been involved in the review conduct as authors (see below: continuous forms of involvement across review stages).

Translation and dissemination

For reviews conducted as a part of the AHRQ EHC Program, the Eisenberg Center carries out focus groups to develop and test consumer guides, which summarize review results and put them into context for a lay audience. The Cochrane Collaboration has involved consumers in the development of the format of one section of Cochrane reviews, the ‘plain language summary’. The Cochrane Musculoskeletal Group has actively involved consumers in disseminating the findings and the work of the review group, for example by presenting at conferences (see Fig. 2 and Appendix S4).

Continuous forms of involvement across review stages

Consumers who act as key informants for AHRQ‐sponsored reviews can also be members of the Technical Expert Group, which gives advice to the review team and carries out peer review (see Fig. 2). Some review groups within the Cochrane Collaboration involve consumers in hand searching specialty journals to identify relevant studies for their reviews. The Cochrane Pregnancy and Childbirth Group involves consumers as co‐authors of systematic reviews in some cases. Consumers serving as co‐authors are usually ‘consumer facilitators’ (see below: groups of consumers), whose main role is to represent the consumer perspective by collecting feedback from other consumers and working on ways to reflect that feedback in the review. SFI Campbell (the former Nordic Campbell Center) forms advisory groups for their reviews, which include members of relevant interest organizations in the field (e.g. homeless people), but also frontline practitioners. Facilitated by a moderator, this group meets up to three times with the researchers to discuss the draft protocol and the draft review and to work on a summary of the review results in consumer‐friendly language.

Rationale for involving consumers

Key informants reported a range of rationales for involving consumers in conducting systematic reviews (see Box 1 for a selection of verbatim quotes). These include increasing the relevance or accessibility of the review, increasing accountability of the research process, complementing the perspective of professionals and increasing the acceptance of review results.

Table Box 1.

Selected verbatim quotations on rationales for consumer involvement

[The main rationale of involving consumers is] to get the questions right so that we can inform relevant and real‐life decisions, and to set the context.

Our main rationale is to make the final product, the systematic review, more relevant to consumers and to real‐life problems.

Our highest priority in terms of involving consumers is to make sure that the results of the review are presented in the plain language summary in a format that is accessible for consumers.

In providing care and allocating resources, we must in some fundamental way be accountable and responsible to the people who receive care.

I think it just helps us to make sure that we understand how patients view the issues and sometimes that’s different than the way clinicians view the issues.

The other equally important reason for user involvement is to pick key stakeholders, who you know when the results hit the fan, then whatever they say is going to be hugely important for the non‐academic impact of that particular systematic review, i.e. on policy and practice. Therefore, it is important to educate them into potential future ambassadors for the review, or at least to become more knowledgeable about systematic review and research generally.

Who is involved as consumers?

Most of the interviewees reported that they involve consumers with direct personal experience of the target condition or treatment under review – be they representatives of patient organizations or individual patients (see Appendix S5). Sometimes, families or informal caregivers are also involved, for example, when there is a wish to complement the patient perspective, when patients cannot speak for themselves, or when the intervention also directly affects others (e.g. partner support during pregnancy). Asked whether it makes a difference what type of consumers they involve, several key informants stressed that the contribution mainly depends on individual personality traits and skills, and some suggested that consumers from patient organizations and individual patients may bring in different perspectives (see Box 2 for selected quotations to illustrate these points).

Table Box 2.

Selected verbatim quotations on experiences with different types of consumers

It depends on an individual’s traits, just as it is with researchers.

You could involve two different people, one of whom works out very well and is very reliable and conscientious and has the time and has a good way of communicating issues. Another person, however, may have difficulty responding in a timely manner, may be less skillful in communicating or may feel intimated by the subject matter and have trouble building confidence. There are many reasons why people would do a great job or may have some challenges.

I can’t say that there is a general difference that I can make between people who are, let’s say, more sophisticated consumers and who are part of a larger consumer network or a local consumer group, and those who are just sort of individuals who want to help out or who come to us because they read it on the internet or heard from their doctor or their friend about it. So I really can’t make a generalization on their involvement. They each make their own contribution for the same degree of success I think.

I think we have probably a preference for involving people who are interested as individuals rather than interested as representatives of various groups. I think everybody has something that they can contribute, so I’m not sure it makes a big difference.

The researchers find it more challenging when the consumer ends up representing an organization or perspective with an agenda of their own, which may or may not be the same as, or in harmony with an individual patient experience or the focus of the systematic review.

I do think it does make a difference who you involve. I think representatives of an advocacy organization bring a really different perspective, particularly if they have a particularly passionate mission that they are really trying to push. I think they bring a perspective that may be valuable, but it is different than an individual patient with a particular experience. I think those voices may all have their own place. But it is important to identify what the objectives are, and to match the kinds of perspective and opportunity for involvement.

Within the Cochrane Collaboration, consumers can contribute to reviews as individuals, working with one or more review groups or fields, or as members of the Cochrane Consumer Network (CCNet; see Fig. 1). CCNet oversees a moderated e‐mail listserv, both to recruit consumers for peer review and to facilitate communication among consumers and others actively involved in the Collaboration. Topics of discussion include Cochrane reviews as well as more general issues related to evidence‐based health care. Consumer networks can also exist in individual review groups. The Cochrane Musculoskeletal Group, for example, hosts a network of consumers with personal experience of musculoskeletal diseases who are involved in the group’s activities. A similar structure (‘consumer panel’) is in place at the Cochrane Pregnancy and Childbirth Group.

Consumer recruitment

While some approaches to consumer involvement in individual systematic reviews (topic suggestions via a website, public comments) rely on self‐selected consumers, other approaches involve active recruitment of consumers (see Appendix S6). The approaches pursued by AHRQ and its EPCs include contacting clearing houses such as the Stakeholder Engagement Team of the Scientific Resource Center located in Oregon, or the US Cochrane Center’s Consumers United for Evidence‐based Health Care (CUE), to identify relevant patient organizations and to contact individual members. Also, the EPCs have found it useful to contact local clinics or doctors to identify relevant patient organizations or, if none such are available for the particular question, individual patients. To recruit participants for focus groups, local advertising is also used. At the Cochrane Collaboration, consumers for specific reviews are usually recruited from the existing consumer networks. Often, review authors ask for peer reviewers via CCNet’s e‐mail listserv.

Consumer compensation

Compensation for time and resources invested by consumers varies across, and within, organizations, depending on the consumers’ role, and is often not provided (see Appendix S7). AHRQ compensates consumers financially in a manner comparable to others for their work as peer reviewers and as members of focus groups, but usually not in their role as key informants. The individuals we spoke with in the Cochrane Collaboration and the Campbell Collaboration did not report that consumers are compensated financially for their time commitment. A modest number of stipends are available for travel expenses to the annual Cochrane Colloquium, for consumers who have been involved in contributing to Cochrane reviews.

Dealing with potential conflicts of interest

We found considerable variation in how organizations deal with potential conflicts of interest (see Appendix S8). While all consumers who are involved in reviews carried out for AHRQ have to declare potential conflicts of interest, this was not reported by those we interviewed associated with the Campbell and the Cochrane Collaborations. At the latter, consumers have to declare potential conflicts of interest when they serve as review authors, but no clear guidance is available for those groups we spoke with for other forms of involvement (e.g. peer reviewers).

Provision of training

Training in research methods is offered to consumers in some organizations, and one informant reported that her organization is developing training for researchers on how to involve consumers (see Appendix S9). AHRQ developed basic information material to orientate stakeholders, including consumers, to their role in the review process, in order to create realistic expectations. To prepare researchers for the involvement of consumers and other stakeholders in the review process, CEbP has developed a web‐based training programme on behalf of AHRQ. The Cochrane Collaboration and its various review groups offer training and education for consumers in the methods of systematic reviews and evidence‐based health care. Training formats include local, national and international workshops of different lengths and intensities, a freely available online course on evidence‐based health care, and mentoring systems.

Evaluation of the impact of consumer involvement

We asked key informants about their personal impression concerning the benefits of involving consumers, both with regard to systematic reviews themselves and beyond, and we provide illustrative quotations in Box 3 , Box 4 . All key informants were positive about the impact of involving consumers, especially concerning the potential beneficial effect on the relevance and usefulness of the reviews. Some reported concrete cases where the involvement of consumers had made a difference (e.g. led to re‐shaping the review question), others found that the benefits are not always easily tangible and one key informant alluded to a gap between the potential for impact and the actual evidence supporting impact, based on formal evaluations. With regard to possible secondary benefits of consumer involvement, our key informants mentioned positive effects for consumers (e.g. acquiring knowledge and skills with the evidence‐based approach, benefits from taking part in discussions with clinicians about the condition that affects them), for researchers (e.g. the feeling that their research actually makes a difference) and for the organization (e.g. lending credibility and trust to the programme, establishment of a culture of knowledge exchange between researchers and consumers).

Table Box 3.

Selected verbatim quotations on personal impressions on the impact of consumer involvement on systematic reviews

I think it [involving consumers] makes the outcomes more relevant and it is an additional quality check in terms of the usefulness of the finished product. It’s an effort in that direction. For example, the patients will or the consumers will identify outcomes that are important to them and question outcomes that are not as important to them.

We think it improves it [the systematic review] absolutely, so that this is a much more relevant product for consumers; it’s much more relevant work if we involve consumers.

I do think it [involving consumers] helps us to keep the reports targeted on the issues that are most important to patients. I think it probably helps with the overall credibility and usefulness of reports.

Definitely [it does improve the product]. […] I think it is very helpful to have a – translated if you will – summary of the review. And I think that’s not only for consumers…I’m looking at the plain language summary for a quick overview of what the review found. So I think that’s been a big improvement.

It is all really subtle, what the actual effect is. I am absolutely certain that they have asked, ‘Why don’t you look at this and this outcome?’or‘If you look at this intervention, this is not really how that intervention classically is played out in practice–it is usually, for example, paired with this other intervention’. It is more in a small scale, for examplesometimes sharpening the researcher’s ability to argue his or her point.

We have not been able to formally measure what improvements actually get incorporated. […] I would say that most definitely consumers have great potential for making important contributions. Whether or not the input is actually taken on or not, I do not know. Without doubt, it is more useful if people who develop and use reviews are able to understand the needs and concerns of consumers better. It is more useful if the review measures outcomes that matter to consumers. It is more useful if it is clearer and easier to understand, with a strong, informative background section. It is more useful if possibly insensitive language has been removed. It is more useful if it has a really clear meaningful plain language summary. There is potential all across the board.

Table Box 4.

Selected verbatim quotations on personal impressions on secondary benefits of involving consumers

It helps increase the pool of people that understand the value of these evidence reports, and I think it can help consumer groups as an additional tool in their own advocacy work. […] I think if they’re advocating for something, it’s helpful for them to know what the evidence is.

When consumers become more confident and more knowledgeable and develop critical appraisal skills, they can in turn use them when they read about new research in the media or continue to use the health care system. […] There are great benefits to increasing the health literacy in the general population about these matters.

It is a very good experience for patients to hear a discussion. In other words, instead of just doing a key informant interview with that patient, it is important, and useful, and beneficial for them to hear a discussion that also has all these sort of clinical experts in it. […] It is very eye‐opening for patients to hear what the doctors and other clinicians are saying.

On the researcher’s side or on the translator’s side it can provide a sense of accomplishment or a feeling that you’re doing the right thing, that you’re really being helpful, useful. That your research is going to make a difference.

One of the interesting things that the editors of the review group have told us over the years is that there has been a feedback process whereby the professionals grow in their knowledge and understanding of consumer perspectives and start to anticipate some of the things that they had not previously seen as important.

Of those two major goals with user involvement, I think we succeeded best in educating the future ambassadors and kind of recruiting people for the evidence‐based policy and practice cause. I think that worked really, really well.

It lends credibility to the programme, and enhances trust in the programme. It certainly helps with the translation and the dissemination.

I think there are some benefits to the organization. To be seen as involving consumers and responsive to the consumers. The aims of the organization are very different for those that involve consumer perspectives and those that don’t. […] I think it’s given us a broader, more rounded approach to issues. I think it’s made us a more inclusive organization […].

I think it has been a sort of culture and knowledge exchange. To the extent that investigators are willing and able to see consumers as experts in their own right, there is an exchange of knowledge and culture that is new and good for everybody.

However, little formal evaluation within organizations has been carried out and published (see Appendix S10). One exception is the Cochrane Collaboration, which conducted an external review of CCNet in 2009, using collaboration‐wide surveys and interviews with managing editors of the review groups and consumers. ¹⁷ The review reported that just over half of the 36 review group respondents felt they were benefitting as expected by involving consumers (e.g. increased readability and/or quality of reviews, usefulness of summaries in a consumer‐friendly language) and that all six consumers interviewed found it ‘very hard to comment on whether their involvement had had any impact’ (p. 3).

Discussion

Remarkably little is known about consumer involvement in systematic reviews in general and how to involve consumers in the best possible way. To begin providing guidance on how to tackle this issue, we carried out a study of organizations that commission or conduct systematic reviews. Among the organizations in our sample that usually involve consumers in the systematic review process, we saw considerable variability in the approaches they pursued. Organizations involved consumers at a programmatic level in the organization or in individual reviews through one‐time consultation or on‐going collaboration. Across organizations, consumers were involved in all steps of systematic reviews from topic suggestion to serving as co‐authors, and the rationale for their involvement as well as the degree of contribution to the various steps of the systematic review process differed. Furthermore, limited evaluation of the impact of consumer involvement has been carried out in the organizations we surveyed.

The surveyed organizations involved consumers for various reasons. The aim described most frequently in our interviews was that involving consumers is an important way to ‘get it right’ (i.e. ensure that reviews address the questions and outcomes that are relevant from a consumer perspective). An expectation associated with consumer involvement is that reviews become more accessible for a lay audience while increasing the acceptance of their results. In addition, the interviewees suggested that involving those who are ultimately affected by the research increases the accountability of the research process. However, at the current time, good quality evidence is lacking about the extent to which involving consumers allows review authors to achieve these expectations, i.e. whether or not consumer involvement has an impact on outcomes such as relevance and accessibility of a review or acceptance of the results. Anecdotal evidence provided in the interviews suggests, for example, that consumer input has been essential for identifying the right questions for reviews, which may otherwise have been academically sophisticated, but potentially irrelevant for the healthcare decisions patients have to make. However, there has been no formal evaluation of the impact, and one would expect that different approaches (e.g. providing the opportunity for public comment, involving consumers as key informants or forming an advisory board with consumers) have different effects and that different approaches may be adequate, depending on which aim is pursued.

The strength of this study is that we explored strategies for consumer involvement of highly regarded organizations. Until now, authors have primarily focused on examples of consumer involvement in single systematic reviews ¹² , ¹³ , ¹⁴ , ¹⁸ or in one organization ¹⁹ , ²⁰ (for a recent overview of case examples, see Boote et al. ²¹ ). Our study provides additional information about how organizations conducting systematic reviews involve consumers and depicts a variety of approaches applied.

A challenge we encountered in our study was that only a few organizations contributed information about how consumers can be involved in systematic reviews. In fact, a majority of the organizations in our sample reported that they do not usually involve consumers directly in the process. In addition, our findings may reflect a smaller range than we would find if we had surveyed all organizations performing systematic reviews. Therefore, it is likely that additional strategies are applied by other organizations. For example, a Spanish review team has reported that they used a Delphi method to consult patients via e‐mail about relevant treatments and self‐perceived health problems. ¹⁸ A further challenge was that key informants reported that the strategies applied by their organizations depend on the commissioner of the review and that strategies are constantly evolving and may change over time. For this reason, this report focuses on depicting the range of strategies used rather than comparing differences among organizations. In addition, as we selected senior officials rather than consumers as key informants, the study mainly reflects a ‘professional’ perspective on this topic. However, consumer feedback was received at the planning stage of the study, as the IOM Committee includes a consumer.

It would be premature to suggest firm standards for involving consumers in systematic reviews. The variability we observed highlights that many questions remain open (see Box 5). Best strategies and processes for consumer involvement may also depend on the goals of the respective organization. ²² For example, answers to questions about which types of consumers ideally should be involved, how they should be recruited, and to what extent consumers and researchers should receive training, may vary by context. Furthermore, it is also unclear whether consumers should represent a group’s view or whether they should represent themselves and their own experience. Similar issues have recently been identified for the involvement of patients in development of clinical guidelines. ²³ It is also unclear whether it is sufficient to elicit views from a limited number of consumers or whether systematic reviewers need to draw on formal research studies of consumer views, when they are available, to learn about the patient perspective. ²⁴ This highlights that there is a need to clarify the relationship between direct engagement processes and scientific evidence on patient views as complementary or alternative ways of taking into account the patients’ perspective. ²⁵

Table Box 5.

Sample of questions remaining open regarding consumer involvement in systematic reviews

Which types of consumers should be involved?

Should it be consumer advocates who can refer to the shared experience of their constituency or should it be individual patients who draw exclusively on their own personal experience? When should informal caregivers or family members be involved, and should members of the public play a role, such as in the Community Forum that is currently being established 3 ?

Who should consumers represent?

Should they be representing a group’s view, or should they talk about their own experience?

When is it sufficient to involve a limited number of consumers directly in the process to represent the patients’ interests and when are more representative data required?

How should consumers be selected?

Should this process be open to everybody or is it legitimate for researchers to draw on a selection of consumers they worked with successfully before?

What degree of training should be provided for consumers and researchers?

Can training in research methods help consumers to understand the researchers’ language, or, conversely, should researchers be trained to work together with consumers?

For certain aspects of conducting systematic reviews, however, we believe it is appropriate to make some recommendations, and these partially overlap with recommendations from the NHS. ²⁶ For example, we need clear policies on disclosure of potential conflicts of interests for consumers, just as for other contributors to systematic reviews. We also think it is fair to consider setting standards for compensating consumers for their time and expenses related to involvement. In addition, we suggest that organizations involving consumers in the review process explicitly report their aims and approaches for doing so – not only for reasons of transparency but also to broaden the knowledge base. Finally, the approaches used to involve consumers should be accompanied by impact evaluation.

In conclusion, we have found that consumers are involved in systematic reviews in a variety of ways and for various reasons. Assessing which approaches are most effective in achieving different aims of consumer involvement is now required to inform future recommendations on consumer involvement in CER.

Source of funding

Financial support for this project was provided by the Institute of Medicine, Washington, D.C., United States, and the Commonwealth Fund, New York, NY, United States. The views presented here are those of the authors and should not be attributed to The Commonwealth Fund, The Institute of Medicine, or its respective directors, officers or staff.

Potential conflicts of interest

Julia Kreis is employed by the Institute for Quality and Efficiency in Health Care, which involves consumers in the process of conducting systematic reviews. Milo A. Puhan does not have any conflicts of interest in relation to this manuscript. Holger J. Schünemann has several functions within the Cochrane Collaboration. Kay Dickersin was a member of the IOM Committee on Standards for Systematic Reviews of Clinical Effectiveness Research. She is involved in a number of consumer groups and has advocated for consumer involvement in research. She is the director of the US Cochrane Center, which founded and hosts Consumers United for Evidence‐based Healthcare (CUE).

Supporting information

Appendix S1. Items covered in the interviews

Appendix S2. Complete list of organizations approached in the study

Appendix S3. Description of organizations that reported to usually involve consumers in systematic reviews

Appendix S4. Approaches for consumer involvement that have been used

Appendix S5. Types of consumers involved in systematic review processes

Appendix S6. Recruitment strategies

Appendix S7. Financial compensation of consumers

Appendix S8. Handling of potential conflicts of interest of consumers

Appendix S9. Training provided to consumers or researchers

Appendix S10. Formal evaluation of consumer involvement

Supporting info item