Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement

Abstract

Aims

Stakeholder engagement is fundamental to comparative effectiveness research (CER), but lacks consistent terminology. This paper aims to define stakeholder engagement and present a conceptual model for involving stakeholders in CER.

Materials & methods

The definitions and model were developed from a literature search, expert input and experience with the Center for Comparative Effectiveness Research in Cancer Genomics, a proof-of-concept platform for stakeholder involvement in priority setting and CER study design.

Results

Definitions for stakeholder and stakeholder engagement reflect the target constituencies and their role in CER. The ‘analytic-deliberative’ conceptual model for stakeholder engagement illustrates the inputs, methods and outputs relevant to CER. The model differentiates methods at each stage of the project; depicts the relationship between components; and identifies outcome measures for evaluation of the process.

Conclusion

While the definitions and model require testing before being broadly adopted, they are an important foundational step and will be useful for investigators, funders and stakeholder groups interested in contributing to CER.

Comparative effectiveness research (CER) encompasses both the synthesis of existing evidence and the generation of new evidence that compares alternative approaches to the prevention, diagnosis or treatment of a health condition or the delivery and provision of healthcare services [1]. While a broad range of study designs such as systematic reviews, observational studies and randomized controlled trials are included in CER, what distinguishes this field of research is its particular purpose. The purpose of CER, as stated by the Institute of Medicine (IOM), is to “assist consumers, clinicians, purchasers, and policy-makers to make informed decisions that will improve healthcare at both the individual and population levels” [1]. This is a critical distinction as the traditional construct of health research often places stakeholders as passive audiences for research results, rather than directly informing priority areas and study design [2,3]. Thus, in traditional research, scientists and experts often drive the research focus with a narrow or incomplete understanding of the information needs of end users, resulting in research findings that are poorly aligned with the information needs of real-world decision-makers [4,5].

The practice of stakeholder engagement in CER seeks to eliminate this divide by actively involving stakeholders across the phases of the research process to ensure the utility and relevance of research results for decision-makers. As such, stakeholder engagement is a fundamental, and perhaps defining, aspect of CER. The impact of CER depends on developing effective processes and support for the meaningful participation of stakeholders throughout the research continuum, from setting priorities to study design, to research implementation and the dissemination of results [6,7].

While there is little disagreement regarding the potential for stakeholders to share valuable perspectives to help shape research, at an operational level there are several challenges, including varying expectations regarding what ‘engagement’ entails, a lack of shared understanding regarding what tools and methods are available to conduct stakeholder engagement and an absence of information to document whether the process has achieved its stated aims [8–12]. In this article we propose definitions of the terms ‘stakeholder’ and ‘stakeholder engagement’ in the context of CER and offer a conceptual model for involving stakeholders in the CER process. Our framework incorporates contributions from both qualitative and quantitative research methods, describes relationships among key components of the process and proposes outcome measures to assess the effectiveness of stakeholder engagement [8].

Materials & methods

The overall approach to defining stakeholder and stakeholder engagement in the context of CER, and developing a conceptual model, involved three components: a literature search; practical experience with an existing stakeholder engagement process in an ongoing CER project; and review and revision by an expert panel.

Review of stakeholder engagement literature

A search of the literature on stakeholder engagement was conducted to identify prior definitions and conceptualizations of stakeholder engagement in research. This literature search was structured to examine key articles from both biomedical and social sciences, and was focused on four main topics: definitions of stakeholder, rationales for stakeholder engagement, stakeholder engagement practices and conceptual models for stakeholder engagement. Articles were identified using keywords that were related to the established topics and confined to PubMed, Scopus and the Web of Science. These keywords were assigned to three domains: who (stakeholder, public, patient), what (engagement, deliberation, participation) and purpose (priority setting, health technology assessment, CER). This list was supplemented by recommendations from colleagues with expertise in the field and a search of bibliographies from an initial list of key review articles [8–10,13]. The initial search revealed several hundred articles relevant to practices in stakeholder engagement in the areas of environmental risk, science policy and health technology assessment. It is important to note that the search conducted was not a full systematic review, but rather a scan of articles relevant to the theory or practice of stakeholder engagement. Detailed quality appraisal and data extraction were not performed and thus the literature reviewed lies heavily upon the content available in the abstracts reviewed by one researcher. Included articles were tagged using keywords such as stakeholder, patient, public participation, CER and deliberative methods. Articles were also tagged for fields such as environmental policy, oncology and health technology assessment [14–18]. The remaining literature was then reviewed to extract information regarding theory and practices that would be relevant to stakeholder engagement in CER.

Stakeholder engagement: the Center for Comparative Effectiveness Research in Cancer Genomics experience

The Center for Comparative Effectiveness Research in Cancer Genomics (CANCERGEN) is a multidisciplinary collaborative consortium, which includes the Fred Hutchinson Cancer Research Center (WA, USA), University of Washington (WA, USA), the Center for Medical Technology Policy (CMTP; MD, USA) and Southwest Oncology Group (MI, USA), one of the largest National Cancer Institute-supported cancer clinical trials cooperative groups in the USA [101]. One of the primary aims of CANCERGEN is to develop and implement a stakeholder engagement process to shape the selection and development of CER studies in the area of cancer genomics. An External Stakeholder Advisory Group (ESAG) was established for the CANCERGEN project and was central to efforts to prioritize a subset of genetic tests for subsequent design of prospective CER studies [19]. The ESAG was comprised of individuals selected to represent the perspectives of a diverse range of constituencies related to cancer genomic technologies. A total of 13 stakeholders were recruited from six key groups: two patients/consumers, three healthcare payers, three practicing clinicians, two policy-makers, one regulator; and two from the pharmaceutical and diagnostic industry. Staff at CMTP managed communications with the ESAG and led the stakeholder engagement activities for the project.

Development using an expert panel

The results of the literature review and the CANCERGEN project experiences related to the ESAG informed the development of definitions of stakeholder and stakeholder engagement as well as the conceptual model for stakeholder engagement. The definitions and model were reviewed by CMTP’s internal staff and the organization’s expert panel, the Patient and Consumer Advisory Council. This panel is a standing committee of eight members with expertise in patient advocacy and engagement who advise the CMTP board of directors and staff to ensure the patient and consumer perspective is given adequate and appropriate weight in all of the Center’s initiatives. The members represent a balance of disease-specific organizations with a more general patient and consumer perspective. Additional external experts, including representatives of the NICE Patient and Public Involvement Program and Citizens Council in the UK also provided feedback on the conceptual model based on their professional experience.

Evaluation of conceptual model: application to CANCERGEN

The conceptual model was designed for use with CER-related projects generally; however, the CANCERGEN project was an ideal first opportunity to pilot test its real-world utility in an iterative fashion. Given that there were at least three distinct stakeholder engagement activities planned over the course of 2 years with the same ESAG, we refined our conceptualization of stakeholder engagement over time based on practical experience and adjusted the model accordingly.

Results

Defining stakeholders in CER

Definitions of stakeholders in the published literature differ in terms of who is included within the stakeholder rubric. Burton defined stakeholders to include groups who have expert knowledge that should be taken into account, will be essential to the implementation of resulting policies, and/or have an interest in the outcome of the work [20]. Other definitions emphasize the stakeholder’s potential to influence the actions of an organization, project or policy direction [21]. More limited interpretations define stakeholders as patients or primary caregivers and clinicians [22]. For these categories of stakeholders, the emphasis is placed on individuals with experiential knowledge of the disease or condition to be investigated [23]. However, references to stakeholders with personal experience with a health condition often involve inconsistent use of terminology, including ‘patient’, ‘consumer’, and ‘user’. This terminological problem, as Boote and colleagues described, relates to the different preferences among people and cultures of how the relationship between an individual and the healthcare system are described [3,6]. Descriptions also included within this category of stakeholder are individuals who speak for patients/consumers through support and advocacy groups, usually with the contingency that they have had personal experience with the disease or condition [24]. Although the categories can appear ambiguous and overlapping, the critical distinction for our purposes is that the terms ‘public’ or ‘citizen’ are best reserved for individuals without a direct interest in a particular healthcare issue. Stakeholders have been distinguished from ‘the public’ because they have self-interest in a given issue; therefore, their involvement in a topic is seen as both rational and more likely to contribute to the quality and legitimacy of subsequent actions [14,25]. We found no similar debate in the health policy literature for other categories of stakeholder perspectives such as payers, scholars or policy-makers, presumably because of implicit clarity regarding their direct interests in topics such as health technology priority setting [26].

Based on the literature review and the goals of CER, we developed a draft definition that was then reviewed by our panel of experts. The resulting definition for stakeholders in CER is presented in Box 1. The definition is intended to be broadly applicable to all stakeholder groups that may be involved in CER, understanding that their information needs and eventual application of CER results may differ. The distinction between organizations and communities within the definition stemmed from the view that communities represent a broad demographic of organizations that are linked by a shared purpose. For example, there are a variety of patient organizations with different organizational missions. The patient community, for example, can be seen as the collective whole of these organizations united by the shared purpose of improving healthcare access and decision-making for patients. The inclusion of both process and outcomes in the definition emphasizes the need for stakeholders to not only be concerned with the process of designing research, but also the outcomes of this research [27].

Box 1. Definitions for stakeholder and stakeholder engagement in the context of comparative effectiveness research.

• Stakeholder

  • Individuals, organizations or communities that have a direct interest in the process and outcomes of a project, research or policy endeavor

• Stakeholder engagement

  • An iterative process of actively soliciting the knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interests in a particular issue, for the dual purposes of:

    • Creating a shared understanding

    • Making relevant, transparent and effective decisions

A related foundational step was to also gain agreement regarding the general categories of stakeholders that are essential to the design and implementation of CER. Stakeholder categories that should be routinely considered for inclusion in CER with brief descriptions and examples are listed in Figure 1. Based on discussions with our expert panel and in our experience with specific projects, we recognize these categories may not be exhaustive or mutually exclusive. For example, life sciences industry stakeholders (e.g., pharmaceutical companies) have their own category but also frequently fund research activities. In addition, we recognize residual ambiguity exists regarding the use of the terms ‘patient’ and ‘consumer’ when used to classify individuals who have personal experience with a health condition [3]. We believe this topic requires a broader discussion to gain consensus in the USA for what term is ultimately applied, as even within our expert panel of patients and consumers, agreement on terminology was not reached. For the purposes of this exercise we have grouped them together.

Figure 1.

Stakeholder categories in comparative effectiveness research.

Our goal was to select categories that should always be given thoughtful consideration for inclusion in CER activities. The final selection of appropriate stakeholder representatives from the categories should be customized to the individual project goals ensuring that the individual selected is knowledgeable to speak on the topic and is clear regarding the role expectations. We do not propose a ‘one-size-fits-all’ approach, recognizing that each project will be unique in its specific stakeholder perspective requirements. Researchers should consider the intent of the research activity and what type of input would be most beneficial. For example, identifying topics for future research requires individuals with a broad understanding of research needs in a given field, whereas designing research studies may emphasize evidence gaps from patient, practicing clinician and payer perspectives.

Defining stakeholder engagement

The literature is replete with descriptions of why stakeholders should be involved in the research and decision-making process, encompassing both ethical and practical rationales [3,6,102]. From an ethical standpoint, the case has been made that patients and their representatives have a right to be included in decisions that may impact their health and well being, or that of future patients [3,22,23]. It has also been suggested that patient involvement has a positive impact on the transparency and accountability of research organizations [2,20].

More practical considerations include the experiential knowledge that stakeholders bring to the process [2,23]. In addition, those who participate in the research process may have more confidence in the outcomes of research [20], which can lead to better dissemination and implementation of research findings. An overarching theme throughout the literature on stakeholder engagement is that patients and others with personal experience of a disease or condition offer a unique perspective that, if explicitly incorporated, will lead to research that is more relevant and translatable [6,27,103,104]. Research in the area of priority setting has demonstrated differences in the priorities of researchers and patients, as well as patient identification of topics that had not been previously considered [22,28]. While much of the healthcare-related literature on stakeholder engagement has emphasized a patient or consumer perspective, the need to include other stakeholder groups, such as payers and practicing clinicians, in research activities has also been recognized as important to ensuring the usefulness of CER results for decision-making [22,29,30].

Community-based participatory research (CBPR) is a particular model of involving community members in full partnership with researchers that has many similarities to the goals of involving stakeholders in CER. In CBPR, the focus is on eliminating health disparities and using research for social action and change, and the key stakeholders are community representatives who act as full partners with researchers to design, implement and evaluate research for the benefit of the community [31,32]. Characteristics of successful CBPR efforts and common challenges should be evaluated to add velocity to stakeholder engagement efforts in CER.

Approaches to obtaining stakeholder input can vary widely in terms of the level and intensity of stakeholder involvement [3,31,33]. Experts in the field have distinguished three categories of engagement: communication, involving a one-way flow of information from sponsor to participant; consultation, in which information is conveyed from the participant to the sponsor; and participation, in which information is exchanged between sponsors and facilitators [33]. Consultation is similar to input obtained through key informant interviews or focus groups with stakeholders. Through these modes of input, the research team obtains information from stakeholders but discussion and deliberation do not occur. Only at the level of participation, where there is bidirectional communication among stakeholders, can there be the opportunity for reciprocal learning and shared decision-making, a necessary feature of engagement in the CER context. The collaboration defined by the category of participation is characteristic of stakeholder engagement in CER, as the ultimate goal of the process is partnership between stakeholders and researchers [3,105]. Other models include more intensive levels of stakeholder control such as ‘user-led’ research, a term coined in the UK for research that is essentially designed and conducted by the public [11]. This approach further advances the potential spectrum of stakeholder involvement by placing a greater emphasis on ‘users’ initiating and leading research efforts, but one that has not typically been applied in the multistakeholder context of CER.

Despite the stated importance of involving stakeholders in CER-related activities, the concept of stakeholder engagement in this context has not been specifically defined. From our previous work with stakeholders and from reviewing the literature, we understood the importance of explicitly stating why individuals were selected, the types of information to be exchanged and that the process was intended to lead to a decision while also creating new insights or a different experience because of the group interactions. This led us to develop a definition of stakeholder engagement as presented in Box 1.

The expert panel reviewed and agreed that our definition was useful for CER for several reasons: it clarified that stakeholder engagement is an interactive process that involves not only empiric evidence, but also judgment and values in a manner that is intended to result in participants reaching a shared understanding (not necessarily consensus) regarding a topic. The panel also concurred that the definition was helpful in stating that the second purpose of stakeholder engagement is for decision-making, and that those decisions have certain characteristics that can be objectively assessed. In the context of CER, decisions must be relevant (meet the information needs of patients, clinicians and payers); transparent (the process must ultimately be perceived as trustworthy by all participants); and effective (the decisions must be actionable with a clear implementation plan). Finally, the process is typically iterative, as CER topics are complex and reaching a shared understanding among all stakeholders requires dialog and reciprocal learning [32].

Developing a conceptual model for stakeholder engagement in CER

Stakeholder engagement explicitly includes deliberative methods to reflect experience, knowledge and judgment where there are questions of scientific uncertainty and disagreement that would benefit from the broader perspectives of a multistakeholder group. For example, questions regarding research priorities or relevant research outcomes needed to support policy decisions are not trade-offs that can be answered solely through scientific decision-making, as they also involve value-based assessments.

Our review of the literature identified a comparable framework originating from the National Research Council’s description of an ‘analytic-deliberative process’ for informing risk decision-making [34]. While this original work focused on applying the model in the setting of environmental risks in order to help stakeholders (including the public) understand both the facts and the uncertainties of environmental hazards, we felt this approach had important generalizable lessons for our work on CANCERGEN. This model views analysis and deliberation as complementary approaches to gaining knowledge regarding an issue, and has been adapted by others in specific attempts to operationalize cooperative discourse in a manner that seeks to reconcile expert-driven and ‘citizen-centric’ approaches [35]. The relationship between analysis and deliberation is bidirectional; analysis can provide information to be used in deliberation, and deliberation can be used to determine what needs to be analyzed. Deliberation in this context refers to more than an informal discussion of issues. It is distinguished from other types of group activities by an emphasis on considering different points of view and coming to a reasoned decision, and it often involves formalized procedures to ensure adequate representation and exchange of stakeholder views [13]. In the context of CER, the analytic-deliberative model emphasizes the dual importance of evidence collection and deliberation by stakeholders in arriving at decisions and recommendations. Our adaptation of the analytic-deliberative model for stakeholder engagement in CER is shown in Figure 2.

Figure 2. Conceptual model for stakeholder engagement in comparative effectiveness research.

CER: Comparative effectiveness research.

It is important to note that inputs in this model are not solely based on research or evidence from the literature; rather, they are a combination of values, research and professional experience. A special distinction is made for knowledge and experience of patients and consumers, both in keeping with the mission of the recently formed Patient-Centered Outcomes Research Institute (DC, USA) and the fact that this group represents the ultimate end-users of healthcare research [36]. Utilizing their knowledge and first-hand experience from the beginning of and throughout the CER process helps ensure that the research outcomes are relevant to their decision-making needs.

The methods highlighted in the conceptual model are techniques for combing the various inputs to arrive at a decision. Traditional consensus methods such as nominal group technique and Delphi processes are often successful methods for engagement [10]. Technology-enabled approaches such as online surveys and audience response rankings are also included as possible methods to synthesize information and make decisions. However, these techniques are often paired with less formalized face-to-face interactions (e.g., workshops directed by a neutral facilitator), in order to promote rich dialog, as well as generate opinions and discussion that may not have been considered using online voting or surveys alone [37]. More recently, techniques such as value of information (VOI) analyses have been adapted for use with stakeholder groups to assist with decisions such as priority setting or study design tradeoffs, as a complementary strategy to inform decision-making [106].

The bidirectional arrows in the model between the first two components are intended to indicate the iterative nature of the process of gathering and synthesizing evidence and values. In our experience these activities typically occur over time with an evolving body of information and methods that are chosen based on the specific project requirements. Also, we attempted to underscore the analytic and deliberative nature of the framework by equally weighting the various types of inputs as well as indicating that both qualitative and quantitative methods are used to combine the inputs.

The outputs of the conceptual model represent the decisions made by the stakeholder group. Those highlighted in the conceptual model include likely end points for stakeholder engagement in CER. The generation of potential research topics, establishment of research priorities, design of relevant and effective studies, and setting of evidentiary thresholds for clinical and health policy decision-making are all key components of CER that require stakeholder involvement. These are the immediate products of the stakeholder engagement process and highlight that its fundamental purpose is to make decisions. This distinguishes the effort from exercises such as focus groups or key informant interviews, where the purpose is simply to collect stakeholder input on a topic.

The outcomes of stakeholder engagement are divided into process outcomes and CER outcomes in the model and represent two aspects of stakeholder engagement that are implied but rarely explicitly stated or measured in the CER context. They are separated because these are quite different measures, both temporally and in terms of how they are linked to the rationale for stakeholder engagement. Process outcomes such as stakeholder ratings of effective engagement, changes in knowledge or project decisions, and increased pathways for implementation represent the immediate outcomes of the stakeholder engagement process. Another example of the direct product of the deliberative methods used to engage stakeholders would be concepts we are classifying as ‘meta-criteria’, because they are outcomes of the process that are intangible, but measurable concepts that are reflective of an effective process. Examples of meta-criteria that have been reported in the evaluation literature for stakeholder engagement include: trust, legitimacy, fairness, accountability, respect and competence [8,13,14,38–47]. There are a variety of methods for evaluating process outcomes [38], including exit questionnaires for stakeholder meetings, follow-up interviews and qualitative analysis of stakeholder meeting transcripts [107].

More importantly, but also more difficult to attribute to the stakeholder engagement process, would be the assessment of CER outcomes. These presumably are the primary reasons why the CER project was undertaken, and include intermediate outcomes such as more useful evidence for clinical and policy decision-making because study results are more actionable and informative for patients and other stakeholders. They also include long-term objectives such as improved health outcomes and more efficient use of healthcare resources. While there is no single approach for measuring the level of success in meeting both the intermediate and long-term objectives, ideally researchers will want to consider how one could demonstrate the relative contributions of stakeholder engagement to meeting these goals.

Evaluation of the conceptual model: stakeholder engagement in CANCERGEN

We mapped our experience with the CANCERGEN stakeholder engagement process to the proposed conceptual model to determine its utility and application to CER. The overall application of the stakeholder engagement process within CANCERGEN as it relates to each element of our conceptual model is depicted in Figure 3. While the stakeholders represented within the CANCERGEN ESAG were consistent with the proposed definition of stakeholders in CER, we confirmed that both the groups and the individual stakeholder representatives had a direct interest in the process and outcomes of CANCERGEN from a micro (the project itself) to macro (policy implications) level by sharing the definitions during a stakeholder meeting.

Figure 3. Application of the conceptual model to stakeholder engagement in the Center for Comparative Effectiveness Research in Cancer Genomics.

AHRQ: Agency for Healthcare Research and Quality; CER: Comparative effectiveness research; VOI: Value of information.

Inputs

The first step in generating evidence for the stakeholder engagement process was a landscape analysis to identify candidate genetic tests to include in ranking and deliberation. After establishing priority setting criteria based in part on the Agency for Health Research and Quality’s Effective Health Care Program criteria for CER, the list of criteria was reviewed, revised and subsequently approved by the ESAG. The ESAG was provided with background briefs and test target profiles for each of the genetic tests and asked to rank the tests using an online survey. An 8-h meeting of the ESAG was held in-person which involved presentations by patient/consumer representatives and an ethics expert and presentations by investigators on each of the tests under consideration to supplement the stakeholders’ existing knowledge and ensure that ESAG members had an adequate level of shared understanding.

Methods

The in-person meeting incorporated both qualitative and quantitative methods to prioritize genetic tests. As demonstrated in Figure 3, quantitative data obtained from independent participation of online ranking of genetic tests was combined with in-depth facilitated discussions that included the opportunity for stakeholders to interact with oncology investigators and ask technical and clinical questions regarding the various genetic tests. Following this face-to-face discussion, there was a second opportunity to vote (using a show of hands). As not all ESAG members were in attendance at the meeting, a final, online priority setting vote to choose the top three genetic tests occurred 2 weeks later, after there was an opportunity to brief the remaining ESAG members. This initial priority setting exercise was followed several months later by VOI analyses of the selected tests, facilitated discussion, and re-ranking of candidate tests based on electronic voting.

Following the re-ranking of the candidate tests investigators developed study concepts for the top two tests and circulated these concepts to the ESAG. Quantitative and qualitative methods were used to evaluate and refine these study concepts. Stakeholders provided initial feedback on the study concepts via an online survey that utilized a five-point Likert scale to determine stakeholder opinion and preference. A subsequent full-day in-person meeting was held with the ESAG to generate additional feedback through in-depth deliberations.

Outputs

The iterative stakeholder engagement process led to two primary outputs: the prioritization of candidate genetic tests from an initial list of six tests to three tests and the development of CER study designs. The study designs developed through the stakeholder engagement process created prospective, real-world study protocols (draft phase) for prioritized genomic tests in lung and breast cancer.

Outcomes

A stakeholder-driven process led to the final prioritization of genomic tests and the design of CER study protocols. The ESAG had an opportunity to combine the information presented to them with their own knowledge and experience in order to prioritize the genomic tests through a deliberative process. As outlined in the conceptual model, both process outcomes and CER outcomes are important to assess. An evaluation to assess process measures of a fair and effective process as experienced by the stakeholders themselves is currently underway. The stakeholder engagement process developed CER study designs that have the potential to significantly impact healthcare decision-making; however, CER outcomes of CANCERGEN are indeterminate at this early stage of study design and implementation.

Discussion

This paper defines terms for stakeholder and stakeholder engagement in the context of CER and proposes a conceptual model for involving stakeholders throughout CER activities. The definitions clarify the intent of stakeholder engagement as well as the different perspectives that should be included as part of the research process. The conceptual model, informed by our work within a multiyear stakeholder project, offers a useful framework for conducting stakeholder engagement as part of CER activities.

The conceptual model organized our engagement activities and linked these processes to specific decisions. Stakeholders in particular were interested in understanding how their input contributed to project outputs and investigator decision-making. Providing feedback is critical in maintaining stakeholder involvement and interest over a multiyear project, as well as for demonstrating the value of stakeholder engagement to sometimes skeptical researchers. The conceptual model made the process explicit and diminished concerns that stakeholder engagement is an informal, unverifiable process. In particular, mapping the distinction between process outputs (decisions) from process and CER outcomes was valuable for describing how the intended effects of stakeholder engagement extend beyond specific project-related decisions. The model also helped to develop criteria that were used to evaluate the process at various critical stages, such as when a novel technique such as VOI theory was incorporated into the ESAG process and again at the end of the overall project.

Given that stakeholders have a direct interest in the process or outcomes of the project, they will (by definition) have the potential for conflicts of interest. The engagement process is designed to select for disparate perspectives, yet we fully anticipate that these conflicts of interest can potentially interfere with our simultaneous goals of creating a trustworthy, fair and legitimate process. Recognizing and managing these conflicts of interest is an area of intense interest in the field of stakeholder engagement and procedures are in development by our team and others to address these conflicts during the recruitment phase of stakeholders to new projects.

In our cumulative experience leading stakeholder engagement efforts in CER, we have identified confusing terminology and the lack of a shared conceptual framework as impediments to implementation. This perception was recently confirmed by a review of stakeholder engagement practices among experts within the Agency for Healthcare Research and Quality funded Evidence-Based Practice Centers who involve stakeholders in future research prioritization activities. Investigators found that even among individuals with extensive experience engaging stakeholders, terminology was inconsistently used to define stakeholders and there was a compelling need for definitions and greater consistency and clarity in the stakeholder engagement process [103].

We approached the CANCERGEN project as an opportunity to define key concepts, and test the applicability of the conceptual model within a multiyear stakeholder project that involved both priority setting and study design tasks. Comparison of our definition of stakeholders with one more recently presented at meetings by the Agency for Healthcare Research and Quality on the topic of CER (“individuals or organizations who have an interest, personal or professional, in the topic”) shows some similarities [103]. Our position is that one of the most important lessons is to define stakeholders and the relevant stakeholder groups at the very beginning of the project and make these decisions clear to all members of the project team. Thus, if particular groups are excluded, there is a clear rationale provided that is supported by the particular project objectives.

We also recognize that within a multiyear project it is possible that the research objectives may evolve. As such, it is important to routinely assess the make-up of the stakeholder group to ensure appropriate perspectives are still mapping to current project needs. For example, within the CANCERGEN project, tests that were prioritized for subsequent CER studies were in specific therapeutic areas such as breast cancer that required additional consultation with breast cancer specialists and advocacy groups to address specific study design questions. Clearly articulating to stakeholders what perspective is being sought and the views the stakeholder should represent (their individual perspective vs the perspective of the community) will ensure stakeholder engagement outputs and outcomes are aligned with current project goals.

To the best of our knowledge, no other groups have specifically defined stakeholder engagement in the context of CER. While there is related literature predominately based in the social sciences, the methods are largely unfamiliar to many researchers applying for CER funding or working in industry, where the workforce tends to include more clinical researchers. Recent examples of similar normative steps occurred in defining ‘outcomes research’ [48], and most recently ‘patient-centered outcomes research’ [49]. The important aspects of our definition of stakeholder engagement are that we underscore the intentionality of seeking inputs such as experience, preferences and values in addition to scientific expertise, in a manner that seeks to make all stakeholders feel that they see a role for themselves in the process. In addition, we differentiate stakeholder engagement from focus groups and town hall meetings by emphasizing the bidirectional nature of the communication (shared understanding) for the purpose of decision-making. We go one step further by characterizing those decisions as ‘relevant, transparent and effective’, thereby establishing a threshold for the quality of the stakeholder engagement process itself and providing criteria for evaluating the process.

Several groups have moved immediately to identifying tactical solutions to near-term problems such as whether to differentially weight stakeholder input or the need to build trust, without clearly describing the key components of the process and how they relate to each other.

The CANCERGEN project provided the opportunity to test this model in a 2-year priority setting and research design project for genomic applications in cancer, an area of high unmet need for evidence of clinical and comparative effectiveness but (like many areas of medicine) confronted with limited research funding. The CANCERGEN project developed a proof-of-concept platform for launching successive stakeholder-driven projects within Southwest Oncology Group to prioritize and guide CER studies. However, as with any multistakeholder project, the process can be complex and controversial. Clear definitions and a conceptual framework can be useful tools for communicating, training and also evaluating the effectiveness of the engagement process [26].

What remains unresolved is how to measure whether the stakeholder engagement process leads to improved CER outcomes. At this point, greater experience needs to be gained with stakeholder engagement while still making the plausible connections with the overarching goals of improved patient outcomes. Given that we only obtained feedback regarding our definitions from a small panel of experts, the generalizability of the results are limited. While we had broad representation from the patient and consumer advocacy perspectives, some of the other stakeholder categories (e.g., policy-makers, industry and payers) were not represented on the expert panel. The conceptual model was developed based on the literature and organizational experience and only applied to a single project. CANCERGEN serves as a successful example of a stakeholder-driven process, but more than one project is needed to evaluate the usefulness of the model in the face of complexities such as conflicts of interest, evolving methods for effective engagement and the current lack of evidence of impact. While the results of our evaluation at the end of the CANCERGEN project will inform the model, its utility is limited pending further evaluation in a broader range of CER projects.

Conclusion

Much attention has been paid to the importance of CER in moving the US healthcare system forward and ensuring that investments in clinical research yield real-world solutions for decision-makers [49,50]. Incorporating the views of relevant stakeholder groups is widely perceived as essential to meeting this goal [6,51]. The proposed definitions and conceptual model of stakeholder engagement offers a starting point for investigators and funders to use in the prioritization, design and implementation of CER. While this work merits further testing with a larger group of stakeholders and CER projects, the application of the definitions and conceptual model to the CANCERGEN project provides a useful example of how they can contribute to advancing the very important role of stakeholder engagement.

Future perspective

There are major efforts within AHRQ, the NIH (e.g., The Clinical and Translational Science Awards) and most recently PCORI to encourage multistakeholder collaborations in research priority setting, study design and dissemination. For example, PCORI’s recent funding announcement for their Pilot Project Grants explicitly required stakeholder involvement in the research plan. Specifically in the area of oncology, the IOM recently recommended that the NCI Cooperative Group Program foster the expanded participation of stakeholders as part of their efforts to improve the efficiency and relevancy of clinical studies [52]. Such investments and recommendations will likely further shift the traditional research paradigm to become more collaborative with patients, clinicians, policy-makers and other stakeholders.

Consensus regarding terminology, definitions and the conceptualization of stakeholder engagement is essential for understanding the contributions of stakeholders to CER and evaluating the effectiveness of engagement practices. Only then can we measure and evaluate the influence of stakeholders on particular CER projects and use these results to inform process improvements and conceptual model refinements. Through such efforts we will be able to assess the impact of involving stakeholders early and throughout the research process, as we all pursue the goal of developing evidence more closely aligned with decision-makers’ needs.

Executive summary.

• The practice of stakeholder engagement in the context of comparative effectiveness research (CER) is hampered by a lack of a shared understanding of terminology, participant roles and engagement methods.

• The proposed definitions and conceptual model of stakeholder engagement offers a starting point for investigators and funders to use in the prioritization, design and implementation of CER projects.

• The descriptions of relevant stakeholder categories clarifies which groups should routinely be considered for inclusion in stakeholder-guided CER projects and emphasizes the need for customizing stakeholder selection to groups that have a direct interest in the particular topic.

• Our definition of stakeholder engagement distinguishes the interactive nature of the process and makes clear that shared, effective decision-making is the critical output.

• Informed by a 2-year CER project experience, the conceptual model provides an initial framework for explicitly linking stakeholder inputs and deliberative methods to decisions as well as process and CER outcomes.

• This work merits further testing with a larger group of stakeholders and CER projects, with the specific intent of examining the impact of stakeholder engagement on CER outcomes.