The PCORI Engagement Rubric: Promising Practices for Partnering in Research

Abstract

PURPOSE

Engaging patients, caregivers, and other health care stakeholders as partners in planning, conducting, and disseminating research is a promising way to improve clinical decision making and outcomes. Many researchers, patients, and other stakeholders, however, lack clarity about when and how to engage as partners within the clinical research process. To address the need for guidance on creating meaningful stakeholder partnerships in patient-centered clinical comparative effectiveness research, the Patient-Centered Outcomes Research Institute (PCORI) developed the PCORI Engagement Rubric (Rubric).

METHODS

PCORI developed the Rubric drawing from a synthesis of the literature, a qualitative study with patients, a targeted review of engagement plans from PCORI-funded project applications, and a moderated discussion and review with PCORI’s Advisory Panel on Patient Engagement.

RESULTS

The Rubric provides a framework for operationalizing engagement to incorporate patients and other stakeholders in all phases of research. It includes: principles of engagement; definitions of stakeholder types; key considerations for planning, conducting, and disseminating engaged research; potential engagement activities; and examples of promising practices from PCORI-funded projects.

CONCLUSIONS

PCORI designed the Rubric to illustrate opportunities for engagement to researchers interested in applying for PCORI funding and to patients and other stakeholders interested in greater involvement in research. By encouraging PCORI applicants, awardees, and others to apply the rubric, PCORI hopes to shift the research paradigm from one of conducting research on patients as subjects to a pursuit carried out in collaboration with patients and other stakeholders to better reflect the values, preferences, and outcomes that matter to the patient community.

INTRODUCTION

Engaging patients and other stakeholders as partners in research increasingly is recognized as a promising approach to generate evidence that is trusted, meaningful, and useful to clinicians, patients, and their families when making health care decisions. The evidence base for stakeholder engagement in clinical research is growing; it shows that engagement is associated with increased recruitment and retention of study populations; more patient-centered and culturally appropriate methods; and greater relevance of research questions and outcome measures.^1–3

The Patient-Centered Outcomes Research Institute: Fostering Engaged Research

The Patient-Centered Outcomes Research Institute (PCORI), authorized and funded in the 2010 Patient Protection and Affordable Care Act, funds comparative clinical effectiveness research for the purpose of generating evidence that helps patients and their health care providers better understand their diagnostic and treatment options and make more informed clinical decisions.⁴ Active and sustained engagement of patients and other stakeholders in setting research priorities, reviewing research applications for funding; designing, conducting, and disseminating research; and evaluating PCORI’s progress are all central to PCORI’s mission.

PCORI is deeply committed to patient-centeredness and patient, caregiver, and other stakeholder engagement as organizing principles guiding its governance and operations. These commitments build on the rich history of partnership between traditional powerholders (eg, government officials, scientific, or health professionals) and the intended beneficiaries of programs and services to better understand and address key issues facing communities.^5,6

Community-oriented approaches to conducting health research emerged from seminal movements such as the use of “action research,” developed by Kurt Lewin in the 1940s, and its derivatives: participatory action research and community-based participatory research. These approaches were critical in advancing the role of patients and communities from “subjects” or “objects” of research to empowered “co-experts” throughout the research process.^7–10 The active inclusion of empowered participants in the generation of evidence and determination of actions for change drives participatory action research and community-based participatory research approaches.¹¹ By illustrating the potential of patient and other stakeholder involvement to improve the relevance and use of research to guide patient-centered care, these movements helped lay the foundation for PCORI’s efforts to advance research partnerships.^12,13

Since its earliest application cycles, PCORI provided guidance and resources to support researchers in conducting scientifically rigorous patient-centered and stakeholder-engaged research (Figure 1). The “PCORI Methodology Standards” specify expectations for patient-centeredness and 10 other topic areas to assist researchers in developing methodologically rigorous clinical effectiveness research.¹⁴ PCORI’s “Merit Review Criteria” for research applications also serves as a resource for researchers and reviewers in assessing, among other things, each project’s patient-centeredness and engagement plans.¹⁵

Figure 1.

PCORI guidance for patient-centeredness and engagement in research.

Note: The criteria have been refined over time. This figure reflects the PCORI criteria as of publication of this manuscript.

During PCORI’s earliest funding cycles, however, many researchers interested in applying for PCORI funding expressed uncertainty about how to operationalize engagement of patient and stakeholder partners as referenced in the Methodology Standards and review criteria. Similarly, patients and other stakeholders were seeking clarity on their role as partners in patient-centered clinical effectiveness research. Using adult learning principles, PCORI developed a framework for operationalizing stakeholder engagement in research that highlights promising and innovative examples from its funded projects. This article describes the development of the framework, hereafter called the PCORI Engagement Rubric (Rubric), and its intended application in research practice.

APPROACH TO DEVELOPING THE RUBRIC PCORI’s

Early Foundational Work That Informed the Rubric’s Development

PCORI’s early efforts to define and advance understanding of patient and other stakeholder engagement in research focused on guiding principles, which laid the groundwork for the development of the Rubric. In 2011, PCORI commissioned 2 systematic literature reviews to synthesize the evidence of how to engage patients in research and identify the benefits, harms, and barriers associated with engagement. In addition, PCORI funded a qualitative study using focus groups of patients from hard-to-reach populations (ie, those that typically do not participate in research due to cultural, socioeconomic, physical, or cognitive barriers) and interviews with clinicians or others caring for these patients.^1,2,16 This work led to suggestions for 10 standards for engaging hard-to-reach patient populations and the establishment of Patient Centered Outcomes Research (PCOR) Engagement Principles to guide its funded research (Figure 2).

Figure 2.

PCOR engagement principles.

Phases of Development

The Rubric was developed in 2013 following an evidence-driven, iterative consensus-building process falling roughly into 3 phases. The first phase consisted of a targeted review of research applications to identify exemplar patient engagement practices to guide development of a draft Rubric. Phase 2 included moderated small-group discussions with PCORI’s Advisory Panel on Patient Engagement to review and refine the draft Rubric. The Advisory Panel on Patient Engagement is a standing panel comprised of 21 patient, caregiver, patient advocate, industry, clinician, and researcher representatives created to ensure the highest standards of engagement and a culture of patient-centeredness in all aspects of PCORI’s work. In the final development phase, PCORI senior engagement staff reviewed the Advisory Panel’s recommendations, then refined the Rubric and obtained feedback from PCORI leadership. On an ongoing basis, PCORI updates the Rubric as advancements in engagement practices are identified.

Targeted Review of Engagement Plans

PCORI staff conducted a targeted review of all 150 funded research project applications from PCORI’s first 3 award cycles to identify applications with novel and promising engagement activities that could potentially direct the study to be more patient-centered. PCORI’s early focus in developing the Rubric was on engagement of patients, family members, caregivers, and their advocacy organizations. Of particular interest were engagement practices that exemplified the PCOR Engagement Principles. Two senior engagement experts (S.S. and S.S.) developed an abstraction template for the review and independently categorized the engagement plans by research focus area, engagement activities, research phase(s), stakeholder characteristics, and potential effects of engagement activities on the study. The engagement experts then met to review and reach consensus on the data and collaboratively classify the collected engagement activities within themes, which provided the structure for the initial framework. Subsequently, PCORI mapped each engagement activity in the draft Rubric back to the PCORI Methodology Standards and the PCOR Engagement Principles to ensure consistency.

Moderated Working Group Discussion with Advisory Panel

PCORI staff shared the initial draft with the Advisory Panel on Patient Engagement during in-person moderated working group discussions. PCORI facilitators led the discussions. PCORI divided the Advisory Panel into 7 breakout groups, each with diverse representation. Using the Rubric as a guide, each group evaluated engagement plans from the same 3 research project applications based on their perceptions of the meaningfulness of the proposed engagement activities. In doing so, the Advisory Panel assessed the Rubric itself, as well as its potential for integration into the merit review process. PCORI engagement experts jointly reviewed the discussion notes from the 7 breakout groups and debriefing notes and then summarized salient themes and patterns related to the Advisory Panel‘s perceptions of the Rubric. PCORI then revised the Rubric by: (1) separating descriptions of engagement practices that exemplified the PCOR Engagement Principles under a set of “global principles” instead of sprinkling examples throughout the Rubric; (2) categorizing engagement activities into 3 broad categories of study phases (ie, planning, conduct, and dissemination); and (3) including as many concrete examples as possible of engagement activities from successful projects to illustrate ways to engage stakeholders throughout the 3 study phases.

PCORI Internal Expert Review and Ongoing Revisions

PCORI’s scientific program directors then reviewed the revised Rubric and recommended specific changes, including that prescriptive language be modified. PCORI’s Board Committee on Outreach, Engagement and Communication approved the Rubric and supported the Rubric’s public release in February 2014 because of its potential to help researchers interested in applying for PCORI funding by providing concrete, actionable examples on how to engage patients and families throughout the phases of research. It was released under the name “Patient and Family Engagement Rubric.”

PCORI’s methodology committee, board members, science and engagement directors, and the Advisory Panel on Patient Engagement continue to refine the Rubric by ensuring it remains relevant and evolves with the field. For example, in 2014, the Rubric was expanded to include examples of engagement of all relevant stakeholders and not solely patients, family members, caregivers and their advocacy organizations; it was renamed the “Engagement Rubric.” A 2016 revision, described in this paper, refined definitions, added additional guidance, and expanded key concepts to better assist applicants and potential partners in preparing to engage in PCOR.

THE PCORI ENGAGEMENT RUBRIC

The Engagement Rubric describes when and how engagement can occur throughout the phases of clinical research.¹⁷ It includes definitions of the PCOR Engagement Principles (Figure 2), definitions of key concepts, key considerations for engaged research, and guidance in the form of potential engagement activities within the 3 broad phases of research, as well as vignettes highlighting engagement practices from PCORI-funded projects (See Figure 3 for selected excerpts).

Figure 3.

PCORI Engagement Rubric.

The Rubric defines 2 groups expected to be contributing members of the research team. “Patient partners” include patients (ie, individuals with the lived experience of conditions under study), their family members and caregivers, and organizations that represent patients and caregivers. “Stakeholder partners” include clinicians, researchers, purchasers, payers, health care industry, hospitals and health care systems, policy makers, and training institutions.

The vignettes from PCORI-funded projects corresponding to the engagement activities provide examples of how stakeholders have contributed to improving the efficiency, relevance, and patient-centeredness of the research and more effectively disseminating the findings to those who would benefit. The key considerations section of the Rubric addresses things like the need to financially compensate patients and other stakeholder partners, provide bi-directional training opportunities to engaged research teams, and clearly define roles and decision-making authority for patients and other stakeholder partners.

DISCUSSION

The Engagement Rubric was first incorporated into PCORI’s spring 2014 cycle of funding announcements and has since become a cornerstone for guidance on engagement across the organization. The Engagement Rubric is not intended to be comprehensive or prescriptive. Instead, it provides a practical resource that distills and prioritizes information from promising practices intended to systematically bring stakeholders into the research process in the most impactful way. The Rubric defines and embraces patient, caregiver, and stakeholder engagement as occurring from generation of potential research questions to dissemination of research results and guides applicants in proposal writing and patients and other stakeholders interested in partnering in PCOR. Researchers, patients, and other stakeholder partners can work together, using the PCOR Engagement Principles as a guiding beacon, to determine which of the activities, as well as any additional innovative approaches, best fit their projects given the population or condition to be studied, the nature of past research and incorporation of patient and other stakeholder views into that work, and the gaps to be addressed by the work. The Rubric can also be helpful for evaluating applications for research funding, developing PCOR training materials, and monitoring and supporting research teams in successfully executing their engagement plans.

Limitations of the Rubric

As the Rubric evolves, it is both catalyzing and capturing nascent practices and lessons learned. As such, several limitations should be noted. First, while extensively grounded in literature and evolving practice, the Rubric was developed primarily based on engagement practices applied in PCORI-funded projects. As a result, it is not yet representative of patients and other stakeholder engagement practices in all PCOR studies or in other engaged research. Additionally, more research is needed to document the effects of these engagement practices on how studies are carried out. Second, the Rubric is oriented toward researcher-driven research in that it primarily reflects engagement from the perspective of researchers who are leading the research team, creating partnerships with patients and other stakeholders, and seeking PCORI funding, rather than the perspectives of patients or other stakeholders who are initiating and driving the research and seeking partnerships with researchers, such as in the Patient-Powered Research Networks.¹⁸ Finally, the current Rubric does not address the “pre-engagement” phase of research— the relationship-building efforts that lay the foundation for partnered work on a specific research project.

Implications for the Future

The Engagement Rubric is a living document that will evolve as practices for patient, caregiver, and other stakeholder engagement in research advance. The Rubric is based on the premise that stakeholder input is important at key stages in the process to produce research that matters and evidence that will be applied by patients, families, clinicians, and payers and other relevant stakeholders. The value of the Rubric lies in its influence on the research community to embrace all stakeholders as true partners whose involvement is essential in the research process. PCORI is planning to evaluate researchers’ as well as patient and stakeholder partners’ use of the Rubric and how it is influencing their approach to engaged research. PCORI is also actively investigating how engagement practices are affecting study design and processes (eg, research questions, outcomes selected, recruitment rates, research relevance and quality) and ultimately, the uptake of study findings and impact of those findings on quality health decisions, health care, and health outcomes. PCORI will use what is learned to improve the Rubric and related resources and to help to build a stronger science of patient and other stakeholder engagement. Through this integration of pioneering engagement practices with the stakeholder engagement evidence base and its’ application to PCORI funded projects, PCORI hopes to shift the paradigm of research from one of conducting research on patients as subjects to a pursuit carried out in collaboration with patients and other stakeholders to better reflect the values, preferences, and outcomes that matter to the patient community.