A Systematic Review of Stakeholder Engagement in Comparative Effectiveness and Patient-Centered Outcomes Research

ABSTRACT

OBJECTIVES

We conducted a review of the peer-reviewed literature since 2003 to catalogue reported methods of stakeholder engagement in comparative effectiveness research and patient-centered outcomes research.

METHODS AND RESULTS

We worked with stakeholders before, during and after the review was conducted to: define the primary and key research questions; conduct the literature search; screen titles, abstracts and articles; abstract data from the articles; and analyze the data. The literature search yielded 2,062 abstracts. The review was conducted on 70 articles that reported on stakeholder engagement in individual research projects or programs.

FINDINGS

Reports of stakeholder engagement are highly variable in content and quality. We found frequent engagement with patients, modestly frequent engagement with clinicians, and infrequent engagement with stakeholders in other key decision-making groups across the healthcare system. Stakeholder engagement was more common in earlier (prioritization) than in later (implementation and dissemination) stages of research. The roles and activities of stakeholders were highly variable across research and program reports.

RECOMMENDATIONS

To improve on the quality and content of reporting, we developed a 7-Item Stakeholder Engagement Reporting Questionnaire. We recommend three directions for future research: 1) descriptive research on stakeholder-engagement in research; 2) evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research; and 3) development and validation of tools that can be used to support stakeholder engagement in future work.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-014-2878-x) contains supplementary material, which is available to authorized users.

BACKGROUND

Better stakeholder engagement has been proposed to help realign healthcare research with the needs of clinicians, patients, policymakers, and payers. Advocates for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR) have been especially strong proponents of this recommendation, on the basis that stakeholder engagement may improve the relevance of research questions, increase the transparency of research activities, and accelerate the adoption of evidence into practice.^1–6 These entreaties could result in a new era of stakeholder-engaged research, and could lead to an important benchmark for patient-centered research: that it “is useful to clinicians and patients—and is used.”⁷

Because little is known about the methods that have been used for engaging stakeholders in the research process, we undertook a systematic review of the peer-reviewed literature to catalogue reported methods of engagement. In prior work,⁸ we defined stakeholder as an individual or group who is responsible for or affected by health-and healthcare-related decisions that can be informed by research evidence. We defined engagement as a bi-directional relationship between stakeholder and researcher that results in informed decision-making about the prioritization, conduct and use of research. We based our work on the 7Ps Framework for Stakeholder Engagement⁸ and the Six Stages Model for CER and PCOR. The 7Ps Framework, developed through use of a deductive-inductive method,¹⁰ suggests that stakeholders may be organized into seven types: patients and the public, providers, purchasers, payers, policy makers, product makers, and principal investigators. The Six Stages Model, developed by researchers affiliated with the Tufts Clinical and Translational Science Institute (CTSI), suggests that CER and PCOR may be organized conceptually into six types: evidence prioritization (identification of research priorities), evidence generation (trials and observational studies), evidence synthesis (systematic reviews and meta-analyses), evidence integration (simulation modeling, cost effectiveness analyses and other methods to address gaps left by trials, observational studies, meta-analyses and systematic reviews), evidence dissemination and application (dissemination and implementation research, and feedback and assessment (evaluation of the research program).

The 7Ps Framework and six Stages Model provided a structure by which we were able to review stakeholder engagement that was reported in the past 10 years of CER and PCOR.

METHODS

We used a five-step process to conduct the systematic review: definition of key questions; literature search; title, abstract and full text screening; data abstraction; and analysis. As described further below, we engaged stakeholders (Table 1) during each step.

Table 1.

Stakeholders Participating in this Review

Category Subcategory	Name	Affiliation
Patients and the public
Consumer	Grant P. Thompson	Consumer reports health ratings center
Consumer	Gerald Rasmussen	Consumer reports health ratings center
Providers
Provider organization	Judy Bradford	Fenway Institute/Fenway health
Payers and purchasers
Self-insured employer	Lawrence Becker	Xerox corporation
Policy Makers
Federal	JoAnne Grunbaum	Centers for disease control & prevention
Product makers
Pharmaceutical	Eleanor Perfetto	Pfizer, Inc.
Principal investigators
Health services	Julie Lynch	University of Massachusetts, Boston
Clinical	Radley (Chris) Sheldrick	Tufts Medical Center

This table presents the 7P Framework category, a subcategory, name, and affiliation of stakeholders who participated in this systematic review. Stakeholders are also mentioned in the acknowledgements

Definition of Key Questions

Our primary question was: how have U.S. researchers reported engagement with stakeholders in the prioritization, conduct and use of comparative effectiveness and patient-centered outcomes research since 2003? Aside from this primary question, we set forth seven key questions for exploration (Text Box 1). In addition to these a priori research questions, we present findings about the impact of stakeholder engagement on research outcomes.

Text Box 1. Research Question and Key Questions

Research Question How have U.S. researchers reported engagement with stakeholders in the prioritization, conduct and use of comparative effectiveness and patient-centered outcomes research since 2003? Key Questions  1. With which types of stakeholders have researchers reported engagement?  2. In what stages of research have researchers reported engagement with stakeholders?  3. How have researchers reported engagement with stakeholders?  4. What level of engagement have researchers used?  5. What modes of engagement have researchers reported?  6. What mechanisms of engagement have researchers reported?

7. Was a special provision reported for the engagement of patients?

This table presents the 7P Framework category, a subcategory, name, and affiliation of stakeholders who participated in this systematic review. Stakeholders are also mentioned in the acknowledgements

Literature Search

We searched peer-reviewed published literature that described stakeholder engagement in PCOR and CER since 2003, the year that the Medicare Prescription Drug Improvement and Modernization Act established the Effective Healthcare Program as part of the Agency for Healthcare Research and Quality. We first searched MEDLINE and Cochrane Central Trials Registry databases from 2003 to May 2012 for English-language studies that reported on stakeholder engagement in the prioritization, conduct or use of PCOR and CER. In addition, we searched the Cochrane Database of Systematic Reviews to identify relevant reviews on the topic. In this stage, we sought to capture publications that: 1) reported on health-related research and programs in any of the six stages of CER and PCOR; (2) reported engagement with stakeholders; and 3) were described as comparative or patient-centered. A wide scope of keywords and MESH terms was used. A preliminary search conducted in November of 2011 yielded several hundred abstracts that were used to inform the search strategy. Terms from these abstracts were added to the final search, followed by consultation with experts in automated search strategies, and with stakeholders who have expertise in making healthcare decisions. The final search strategy included 76 terms (Online Appendix A).

Title, Abstract, and Full Text Screening

Following the search, we conducted a preliminary screen of article titles and abstracts, using double review by trained readers and adjudication of disagreements by the principal investigator. Explicit inclusion and exclusion criteria were developed prior to screening. To be included and advanced to full text screening, abstracts had to indicate that stakeholders were engaged in any one (or more) of the Six Stages of Research (prioritization, generation, synthesis, integration, dissemination & implementation, feedback & assessment) as investigators, staff, partners or consultants. Publications were also considered to be stakeholder-engaged if they reported collaboration with site-based staff who were practitioners in healthcare delivery, payment or policy organizations. Stakeholders had to plausibly belong to one of seven groups of individuals and organizations with an interest in the outcomes of research (patients and the public, providers, purchasers, payers, policy makers, product makers, and other principal investigators).⁹ Abstracts were excluded and did not proceed to full text screening if stakeholders and related terms were not mentioned, if the abstract was a report of planned research (we sought reports of completed research and programs), or if stakeholders were engaged as subjects of research only.

Full text screening involved a repetition of this process on the full text of the article. Our unit of analysis was the published research report. Thus, surveys of patients or clinicians as subjects of research, but not as fully-fledged stakeholders, did not qualify on their own for inclusion, and we did not try to infer if a survey led to further engagement with its respondents as stakeholders after the report was published. However, if a report described both a survey and use of respondents to inform priorities or conduct further research, it qualified for full text review.

Data Extraction

Data were extracted from full text articles into an electronic questionnaire and database built specifically for this project. Staff members conducting extractions used the electronic questionnaire (Online Appendix B) to populate an excel spreadsheet on which later analyses could be run. The electronic questionnaire and linked database were developed by the full team and used carefully crafted non-skip rules for required responses, skip patterns to follow question logic, drop down response options for uniform data gathering, free text fields to supplement uniform data gathering, and documentation to track progress on the reviews. The research question and seven key questions described in Table 1 formed the major sections of the electronic survey instrument and provide a framework for reporting our results. Randomly selected articles were re-reviewed by team members to check for accuracy. Free text fields were coded and reported after all extractions were completed.

Analysis

We adhered as closely as possible to address recommendations in the Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA).^9,10 Because our synthesis focused on what authors have reported about stakeholder engagement, no syntheses of any outcome measure were included in our analysis. This meant that formal tests of risk of bias in individual studies or across studies were not needed. Most data are reported categorically as a number or percent of publications.

Stakeholder Engagement

We engaged with stakeholders to assist in the design, conduct, and interpretation of our review (Table 1). During our planning stages, we engaged with two patients and one stakeholder from five of the remaining six categories described in the 7Ps framework. For this review, we combined payers and purchasers into a single category, since their standpoints on engagement in research are similar. The target numbers we identified for each stakeholder type assured that our panel was over-weighted with patients and was not dominated by individuals or organizations with a commercial interest, income, or organizational conflict of interest resulting from the sale of healthcare interventions or products. Stakeholders were identified through personal and professional networks prior to conducting the evidence synthesis, and all shared an interest in stakeholder engagement in research. All invited individuals agreed to participate.

We held two stakeholder meetings: the first to articulate relevant research and key questions and adapt the study design; the second to review preliminary results in the analysis. With respect to the study design, seven stakeholders and experts in the conduct of systematic review from Tufts Medical Center suggested data elements for inclusion in the electronic extraction form. Stakeholders also helped us identify effective ways to communicate the findings in tables and figures for this manuscript. All stakeholders were invited to participate by email and phone throughout the research, including a review of the manuscript. Stakeholders have been asked to assist in dissemination of the results. We did not conduct any formal evaluation of the impact of stakeholder engagement during the course of the review, but have included this project in a slate of stakeholder-engaged research that will be evaluated and whose results will be presented elsewhere.

RESULTS

The search yielded 2,062 abstracts for initial screening. Abstract screening yielded 275 articles for full text screening. Full text screening yielded 157 articles, and of those, 62 reported on research or programs in non-U.S. settings and 95 reported on research or programs in U.S. settings. We further classified the 95 articles into reports of research (n = 36), reports on programs (n = 34), and syntheses of stakeholder engagement (n = 25). For the purposes of this project, reports of research describe and address a health question for which the answer is not already known, and reports of programs describe the implementation of a health-related intervention without addressing a related health question. A PRISMA diagram^11,12 is depicted in Fig. 1. This review presents descriptive data on the reports of research projects or programs (n = 70) (Table 2),^13–82 since it was not possible to extract from syntheses of multiple stakeholder engagement projects any detailed information on the individual projects.

Figure 1.

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) diagram. This diagram presents the number abstracts (n = 2,062) and articles (n = 275) retrieved for screening, the number of articles reporting on research or programs in U.S. settings (n = 95), and the number of articles included in the review (n = 70).

Table 2.

Article Characteristics

	n	%
Article setting (n = 157)
U.S.	95	61
Non-U.S.	62	39
Type of article (U.S. only, n = 95)
Research	36	38
Programmatic	34	36
Synthesis of research	25	26
Selected characteristics (Research only, n = 36)
Research stage
Evidence prioritization	6	17
Evidence generation	26	72
Evidence synthesis	0	0
Evidence interpretation and integration	0	0
Dissemination and Application	3	8
Feedback and Assessment	1	3
Type of methodology used
Qualitative	11	31
Quantitative	4	11
Mixed methods	14	39
ND*	7	19
Study setting
Community	28	78
Health care setting	6	17
NA*	2	6
Unit of analysis
Current patients	7	19
Population (Public health), including community individuals	19	53
Providers and provider organizations (hospitals, health centers…)	5	14
Other**	5	14

This table presents characteristics of 157 U.S. and non-U.S. research articles, program reports, and syntheses of research and programs that were included in the full text review. ^*NA and ND responses include articles in Stages 1 and 6, where a formal research may have not been presented. ^**Other includes miscellaneous responses, such as members of a collaborative, geographic/administrative boundaries, etc.

Of the 70 articles included in this review, topics included health behaviors (diet, sexual health), mental health, disparities, violence prevention, chronic diseases, health insurance enrollment, and others. Articles addressed the needs of current patients (commonly patients with HIV, diabetes, or depression), minority populations (including African Americans, Hispanics, Native Americans, and Asians) and service providers (commonly clinicians or nurses).

Stakeholder Categories

Of the seven stakeholder categories in the 7Ps Framework, patients and the public were most frequently engaged in peer reviewed research and program reports (Fig. 2a). Eighty percent of articles (n = 56) reported engagement with this group. Providers were the next most frequently cited group (n = 32); policy makers (n = 16) and principal investigators (n = 11) were next. Payers (n = 6), product makers (n = 4), and purchasers (n = 0) were rarely or never mentioned. Only 9 % of research and program reports described strategies to balance views of stakeholders with competing viewpoints. Stakeholders were typically recruited through personal and professional networks (61 %). About a quarter of the articles did not report on the method of recruitment, and there were no articles reporting use of sampling methods to recruit stakeholders from a specific population.

Figure 2.

Who are the stakeholders and in what stages of research are they engaged? Panel A presents the frequencies of articles mentioning engagement with each of the seven stakeholder categories in the 7Ps Framework. Because articles might report engagement with multiple stakeholder groups, the denominator for each bar in this histogram equals 70. In Panel B, the share of each bar presenting engagement with a stakeholder group represents the probability of engagement with that group within the stage of research. Because an article could span multiple stages of research, the total reports of engagement in the six research stages equals more than 70 (n = 107).

Stage of Research

Stakeholder engagement was more common in earlier than in later stages of research (Fig. 2b). The first two stages in the Tufts CTSI six-stage model of CER, the prioritization (n = 31) and generation (n = 45) of evidence, were by far the most common type of research or program activity for which stakeholder engagement was reported. Engagement was less common in the process of dissemination and application of evidence (n = 9) and in evaluating research activities (n = 10), and it was rare in research projects that involved the synthesis, integration, or interpretation of evidence.

Much of the engagement occurred before the research or program was underway, either by helping to define a research question (34 %) or by defining some key aspect of the population, intervention(s), or comparators (44 %). Engagement to monitor and oversee a project after it was up and running was also fairly common (36 %). Slightly less frequent were engagement activities that involved interpreting, disseminating and applying the results of a project (9–10 %), echoing the relative lack of engagement in research projects that focused on these activities.

Methods and Modes of Engagement

The activities of stakeholders were highly variable across research and program reports. Approximately half of the articles reported engaging with stakeholders as consultants to their projects, while one-third reported engagement at the level of co-principal investigator and one-third reported engaging stakeholders as staff on the project (more than one response per project was possible). One in five reports indicated that stakeholders were engaged as the subjects of the research or program. Stakeholders were engaged through surveys and key informant interviews about 15–20 % of the time and in focus groups and panels 20–30 % of the time.

Analysis and Impact of Stakeholder Engagement

Nearly half (44 %) of the articles did not report how the views of stakeholders were synthesized and used in the project. About one-quarter (26 %) reported qualitative synthesis of written notes or transcripts and more than one-third (36 %) reported that stakeholders and investigators engaged with each other in joint-decision making.

Although it was not possible to validate claims of the benefits and challenges related to stakeholder engagement, several common themes were reported. About one in five articles reported that stakeholder engagement improved the relevance of research, increased stakeholder trust in research and researchers, enhanced mutual learning by stakeholders and researchers about each other, or improved research adoption. A smaller number reported that engagement improved the transparency of research (6 %) and increased understanding of the research process (9 %). The most common challenge reported was that stakeholder engagement is time consuming (19 %). Other challenges were reported by between 3 and 6 % of articles, including that stakeholder engagement requires researcher flexibility, trust among researchers and stakeholders, commitment from both the researchers and stakeholders to maintain contact and participation, difficulty establishing stakeholder representativeness throughout the course of the research program, increased ethical concerns in some institutional review boards (IRB), stakeholder distress while participating (particularly with patients and family members), and difficulty overcoming cultural differences between stakeholders and researchers.

DISCUSSION

Reports on CER and PCOR are highly variable in the types of stakeholders who are engaged. We found frequent engagement with patients, modestly frequent engagement with clinicians, and infrequent engagement with stakeholders representing other key decision-makers across the healthcare system. The frequency of engagement with patients is encouraging, as it reflects a growing opinion that this ultimate decision-maker is the most salient stakeholder in research. This view was emphasized by the stakeholder panel, is reflected in the call for patient-centered medicine in initiatives across the Federal government,^83–87 and is explicitly emphasized in guidance on Patient-Centered Outcomes Research Institute (PCORI)-funded research.^88,89

The relatively modest frequency of engagement with clinicians and infrequency of engagement with payers, purchasers, policy-makers, product-makers and other principal investigators is discouraging. These groups, though not salient for every research project or program evaluation, represent a broad range of critical decision-makers in the healthcare system. Many high profile cases have demonstrated that ignoring these groups during research activities can lead to unwarranted challenges: the research can result in policy and other decisions that are ill-fitted to the interests, opinions, and needs of these groups, and can result in fierce opposition that derails well-intentioned decision-making. One of the best known of such cases recently was the U.S. Preventive Services Task Force (USPSTF) guideline on mammography screening, in which patients, patient advocates, radiologists, oncologists, and others lined up to oppose the guideline after it was issued for public comment.⁸⁸ Inclusion of clinicians and other stakeholders has been embraced by PCORI,^89,90 and has been identified as a priority for dissemination and implementation of CER and PCOR findings.⁹¹

The inclusion of a broad range of stakeholders in research is not without its challenges. These may range from logistical (e.g., scheduling and funding) to process-oriented (e.g., balancing competing interests, addressing implicit power differentials, and managing conflict). Indeed, the main reported barriers to engaging stakeholders in CER and PCOR were time and the need for logistical support and funding flexibility. We did not, however, see reports of difficulties in managing the decision-making process. Stakeholders may have conflicts with each other and perceived or direct conflicts of interest, but well-developed mechanisms for managing these conflicts exist, including public disclosure, non-voting participation, and recusal. Where conflicts of interest cannot be managed, intentional exclusion may be the right choice, but this decision should be made explicitly and transparently before the research begins.

We also saw variable reporting on engagement by stage of research. Engagement is most common during evidence prioritization and generation, and is infrequent or rare in latter stages: synthesis, integration, dissemination and evaluation. The infrequency of engagement in these latter stages could reflect just an omission in reporting, but we suspect it reflects actual lack of engagement. In order to maintain bi-directional relationships between researchers and stakeholders, we believe that uninterrupted engagement over the lifecycle of research is necessary. To accomplish full and uninterrupted engagement, researchers and stakeholders may need to invest time and resources for training and support. Furthermore, one potential benefit of engagement during methodologically challenging stages of synthesis and integration would be to improve the transparency of research activities. If stakeholders understand and can articulate how findings were established, they are far more likely to become effective ambassadors during dissemination and implementation efforts.

To improve on the quality and content of reporting, we developed a 7-Item questionnaire for reporting on stakeholder engagement in research (Text Box 2). The questionnaire includes items on the types of stakeholders engaged, whether target numbers were established for each type of stakeholder before recruitment began, how a balance of stakeholder perspectives was considered and achieved, and the methods used to identify, recruit and enroll stakeholders in engagement activities. We also recommend reporting on whether engagement occurred: 1) before research, in priority setting, topic development, question development, and research design; 2) during research, including enrollment of patients, conduct of data collection, analysis, and interpretation of findings; and 3) after research, including dissemination and implementation of findings, and evaluation of the research itself. Finally, we suggest reporting on the modes and methods of engagement, and we recommend an assessment of the impact of engagement on the relevance, transparency, and adoption of research.

Text Box 2. A 7-Item Questionnaire for Reporting on Stakeholder Engagement in Research

1. What types of stakeholders were engaged? 2. What were the a priori target number(s) for each type of stakeholder? Were targets met? 3. How was balance of stakeholder perspectives considered and achieved? 4. What methods were used to identify, recruit and enroll stakeholders in research activities? 5. Did engagement occur:  a. before research began, during priority setting, topic development, question development, and research design;  b. during research activities, including enrollment of patients, conduct of data collection, analysis, and interpretation of findings; and  c. after research was concluded, including dissemination and implementation of findings, and evaluation of the research itself? 6. What were the intensity, methods and modes of engagement?

7. What, if any, was the impact of stakeholder engagement on:  a. the relevance of research questions;  b. the transparency of the research process; and  c. the adoption of research evidence into practice settings?

This figure presents a list of questions that may be used by researchers to guide future reporting on stakeholder-engaged research. These questions were co-developed with a stakeholder panel

We recommend three directions for future research: 1) descriptive research on stakeholder-engagement in research; 2) evaluative research on the impact of stakeholder engagement on the relevance, transparency and adoption of research; and 3) development and validation of tools that can be used to support stakeholder engagement in future work. First, future research is needed to describe the nature and extent of stakeholder engagement. For example, to understand in more detail how extensively stakeholders have been engaged in clinical trial research or evidence syntheses, a review team might draw a random sample of peer reviewed articles from a limited set of top-ranked journals within a limited time frame, and contact corresponding author(s) for structured interviews aimed at learning whether, to what extent and how stakeholders were engaged in the project. Second, future lines of inquiry could evaluate the benefits of different approaches for engaging stakeholders with respect to the relevance of research questions, transparency of research activities, and adoption of evidence into practice. Third, future research should be organized to develop training materials, methods and tools for advancing stakeholder engagement in healthcare research. Hands-on training opportunities and user-friendly tools and methods for stakeholder identification and recruitment, engagement, and evaluation are needed to help investigators embrace stakeholder engagement in their research projects. Prior work in the field of Community-Based Participatory research (CBPR) has addressed some of these needs.⁹²

This review of stakeholder engagement faced a key limitation. First, though our search strategy cast a wide net by deploying a comprehensive set of terms in multiple databases, we recognize that many research programs and projects may engage with stakeholders without reporting on engagement in the published article. This limitation, however, also underscores the primary finding: that reporting on stakeholder engagement activities is highly variable in quality and content.

In summary, we found substantial variation in the quality and content of reporting about stakeholder engagement in research. To address some of these shortfalls, we developed a 7-item questionnaire for reporting on stakeholder engagement in research (Text Box 2). Finally, we recommend new descriptive research on stakeholder engagement; new evaluation studies to establish whether engagement influences the relevance, transparency and adoption of research; and new training, method and tool development to support best practices in future stakeholder-engaged research activities.