Researcher and partner perspectives on the relationship between engagement in research and three uses of patient-centered comparative clinical effectiveness research study findings

Abstract

Background

The uptake of research findings into clinical practice is critical to providing health care that improves health outcomes for patients. This study explored how Patient-Centered Outcomes Research Institute (PCORI) awardees perceive the relationship between engagement of patients and other partners in research and three uses, or applications, of patient-centered comparative clinical effectiveness research (CER) study findings, which may lead to uptake in clinical practice: (1) Integration into clinical practice guidelines, recognized point-of-care decision tools, or documents that may inform policy; (2) Implementation beyond the study, including at sites outside of the study setting or patient populations; and (3) Active dissemination of findings to specific audiences by parties external to the study team.

Methods

This exploratory qualitative study examined awardee and partner perceptions of what led to each use of study findings and how engaged partners contributed. We purposively selected PCORI-funded research projects with documentation of each use and conducted virtual interviews with 42 individuals (15 PIs or project leads, 2 research team members, and 25 partners) from 17 projects. We conducted thematic analysis of individual projects or project sets, across projects within each use case, and across the three uses.

Results

Participants described three primary activities in which engaged partners made contributions before, during and after CER studies that facilitated the use of study findings: (1) generating relevant study findings, (2) distributing study findings strategically, and (3) making connections to people or organizations outside the study team. In addition, engagement continued to facilitate the use of study findings during subsequent PCORI-funded implementation and dissemination-specific projects, with partners adapting interventions and creating and tailoring dissemination messages and products. Finally, participants described attributes of teams’ engagement approaches that may have supported partner contributions, including early and ongoing engagement, leveraging partners’ connections and understanding of community needs, and using multiple engagement approaches.

Conclusion

This study identified examples of how engagement can help facilitate the use of CER study findings, especially when engagement contributions occur in meaningful ways. Findings from this study suggest a framework for future research on the relationship between engagement in research and uptake of study findings into clinical practice.

Plain language summary

This study looked at how PCORI awardees describe the relationship between engagement in research and three uses of study findings: (1) Citing findings in clinical practice guidelines or other documents that support health policy; (2) Putting findings into practice after the study; and (3) Sharing findings with specific audiences by people not on the study team. We selected PCORI-funded research projects that had signs of each use. Then, we interviewed 42 people, including 15 lead investigators or project leads, 2 research team members, and 25 partners. We summarized findings for each project and then across the three uses.

Partners helped move forward each of the three uses of study findings. They helped to plan and conduct the studies, making findings relevant. Partners also helped to plan and then share study findings. They made connections to people and organizations for the use of study findings. In addition, partners helped to put findings into practice after the study and tailored and shared findings in their communities.

Finally, the way project teams engaged partners may have supported partner contributions. Project teams had early and ongoing engagement in their study. Partners shared their understanding of community needs and used their connections to share findings. Also, projects often had more than one way to work with partners.

Findings from this study highlight ways engagement in research can contribute to use and potentially uptake of study findings in clinical practice. The study also suggests topics for future research on how engagement affects the use of findings.

Manuscript Text.

Background

The uptake of evidence-based research findings into clinical practice is critical to the provision of more effective healthcare delivery that provides maximal benefit to patients and improves health outcomes. Many individual and system-level factors influence the uptake of research findings including continuous education, organizational culture and connections between knowledge producers and end users [1]. As such, there is an estimated 17-year time lag between the generation of evidence-based research and its integration into clinical practice [2].

Engaging patients, clinicians, and other partners in the design and conduct of a research study and subsequent dissemination or implementation projects may be one way to facilitate and accelerate uptake of research into clinical practice. Engagement in research is an active partnership between researchers, patients, caregivers, clinicians and other partners throughout the entire research process, where partners are involved in planning, conducting and disseminating the research. A growing body of literature documents the benefits of patient and other partner engagement in research studies, including active influence by partners on study protocols and study enrollment rates [3–7]. These contributions have resulted in impacts on studies’ acceptability, feasibility, rigor, and relevance [8, 9]. For dissemination efforts, patients and other partners can serve as ambassadors of research findings, expanding audiences beyond peer-reviewed journals and facilitating uptake of results into the community and healthcare setting [3, 10]. Further, in the field of implementation science, engagement of diverse partners and communities, especially in the planning, implementation, and sustainment of interventions, is increasingly recognized as a core component to successful implementation and translation of research findings [11, 12].

However, research directly examining the role of engagement in the uptake of study results in clinical practice is limited. Challenges to assessing this relationship include the length of time between when a research study on health or health care starts and the generation of findings, subsequent dissemination and implementation efforts, and eventual uptake. Such studies typically require several years to complete and several more years for interventions recommended as evidence-based practice in systematic reviews, guidelines, or textbooks to be fully implemented [13, 14]. Also, validated measures related to the processes and impacts of engagement or to evidence use are limited [15].

To advance the understanding of the relationship between engagement in research and the uptake of study findings in clinical practice, we conducted an exploratory qualitative study with Patient-Centered Outcomes Research Institute (PCORI) awardees and their patient and other partners. PCORI requires awardees to engage diverse partners, which may include patients, caregivers, clinicians, and others in the health and healthcare community, in patient-centered comparative clinical effectiveness research (CER) studies as well as subsequent projects to disseminate and implement results [16]. This exploratory study examined the relationship between engagement in research and uses, or applications, of study findings that may contribute to uptake in clinical practice. These uses are often considered as components in the adoption of research evidence or as steps on a pathway to the uptake of research in practice and its impact on health care and health outcomes [16–20]. The aim of this study was to explore awardee and partner perceptions of what led to each use of study findings and how engaged partners contributed to the process. We defined study findings as the results of primary outcomes from the PCORI-funded research projects and explored three uses of these findings:

• Integration of CER study findings into clinical practice guidelines, recognized point-of-care decision tools such as UpToDate^®, or other documents that may inform policy such as government-commissioned literature reviews. Translating study findings into actionable guidelines and incorporating findings into point-of-care decisions tools are recognized as important steps toward uptake of evidence-based clinical practices to improve patient outcomes [21, 22].

• Implementation of findings beyond the CER study, including at sites (e.g., hospitals, clinics, or other delivery settings) outside of the CER study setting or for different patient populations. The intentional and iterative application of study findings in clinical decision-making supports the adoption of evidence into practice and sustained behavior change based on evidence that can benefit populations served [16].

• Active dissemination of CER study findings to specific audiences by parties external to the original CER study team. Active dissemination and communication strategies have been shown to be more effective for increasing awareness, understanding, and application of study findings among diverse populations than passive dissemination tools alone (e.g., publication in a journal) [23–25].

Methods

We applied and adapted use case and multiple case study methodology to explore the relationship between partner engagement in research and the three uses of CER study findings [26–29]. Use cases are helpful for identifying specific attributes or features of a product or service and how it is or could be used to address a specific challenge or in various scenarios. For this formative study, we developed use cases to identify ways partner contributions in engaged research could facilitate the use of CER study findings along with real-world examples described by researchers and partners. We conducted one use case for each of the three uses.

We intentionally chose these specific uses because they can be considered interim steps toward uptake in clinical practice [16–20], were feasible to examine at the start of this study given that the first results of PCORI-funded projects only became available in July 2017, and had documentation of use collected by PCORI that enabled us to identify a pool of candidate research projects. To improve validity, each use case included multiple PCORI-funded projects and triangulated multiple data sources for each project (i.e., document review, interviews with researchers and partners).

PCORI obtained input from patient, clinician, and other partners in multiple ways over the course of several years to develop the focus, aims, and overall approach to the examination of the relationship between engagement in research and the use of study findings. They worked with their multi-stakeholder Board of Governors and multiple PCORI advisory panels including the Patient Engagement Advisory Panel (PEAP). They had strategic conversations with others in the healthcare community such as other funders and institutions. The team also shared and discussed early findings from these use cases with members of the PEAP.

The American Institutes for Research’s Institutional Review Board (IRB) reviewed the protocol. The core team included the project lead (MEM) and six analysts (ML, THB, KF, MH, TD, DA). All had prior training and experience conducting qualitative research and were familiar with engagement in research.

Sample

We used purposeful sampling to select PCORI-funded research projects for which PCORI had documentation of one of the three uses of CER study findings.

Integration of CER findings

For this use, projects included PCORI-funded research projects with CER study findings cited in two or more ways: clinical practice guidelines, UpToDate^® point-of-care decision tool, and documents that may inform policy. PCORI collected information about these citations but did not fund efforts to cite findings in these policy documents.

Implementation of findings beyond the CER study

For this use, projects included PCORI-funded research projects that received a PCORI-funded implementation award [30]. PCORI funds these awards to actively facilitate the uptake and integration of evidence from PCORI-funded CER projects, in the context of related evidence, into real-world practice.

Active dissemination of CER findings by external parties

For this use, projects included PCORI-funded research projects whose CER study findings were disseminated through a PCORI Engagement Award: Dissemination Initiative project [31]. PCORI funds Dissemination Engagement Awards to support organizations and communities in actively communicating CER study findings to specific audiences in ways that encourage use of this information in decision making. To indicate interest in the findings by external parties, the Project Lead of the Engagement Award differed from the Principal Investigator (PI) of the original CER study.

We excluded projects whose findings would not apply to clinical practice (e.g., from the advancing research methods portfolio), whose research project team members had participated or were participating in other similar data collection activities for PCORI, or whose original research projects had less comprehensive engagement based on an assessment by PCORI. Based on these criteria, PCORI identified 29 candidate research projects across use cases. The project lead (MEM) prioritized outreach to the final list of projects, aiming to balance variation in how findings were used within each use case (e.g., type of expansion within implementation), as well as the diversity of original research projects in terms of PCORI’s research agenda, partners engaged in original CER study (e.g., patients, caregivers, clinicians, payers), and characteristics of the original CER study (e.g., study design, health condition, intervention, study population).

Analysts conducted email outreach to recruit PIs from the CER study and, if applicable, the PI or project lead of the implementation project or Dissemination Engagement Award project. We required at least two interviews for each project or project set for enrollment. (A project set is the original CER study and its subsequent implementation project or Dissemination Engagement Award project. The integration use case included the original CER study only.) After obtaining a list of recommended participants (e.g., researchers, partners) from the PI(s) and/or project lead, we contacted those individuals via email or phone to assess interest, availability, and schedule interviews.

Across use cases, we contacted PIs of 17 research projects: 3 research project PIs declined to participate (2 in the integration use case and 1 in the active dissemination use case) and 4 research project PIs did not respond to initial or follow-up email requests (2 in the integration use case and 1 each in the implementation and active dissemination use cases). Table 1 shows the numbers of candidate and contacted research projects as well as the final number of included projects by use case.

Table 1.

Number of candidate and contacted research projects and final number of included projects by use case

Documentation of use	Number of candidate research projects	Number of research projects contacted	Final number of included projects
1. Integration of CER findings. Funded-research projects with primary comparative effectiveness results cited in two or more ways: clinical practice guidelines, UpToDate® point-of-care decision tool, and documents that may inform policy	12	7	3 research projects
2. Implementation of findings beyond the CER study. Funded-research projects that have received PCORI implementation awards	10	4	3 project sets of a research project and implementation project (6 projects total)
3. Active dissemination of CER findings by external parties. Funded-research projects that have received PCORI Dissemination Engagement Awards, with a Project Lead that differs from the original study Principal Investigator (PI), indicating external parties are actively disseminating study findings	7	6	4 project sets of research project and a Dissemination Engagement Award project (8 projects total)

Data collection

For each project or project set, one analyst conducted the document review and interviews. The analyst reviewed interim progress reports submitted to PCORI, publications of CER study findings, and other information specific to each use, and extracted information about the study findings, how they were used, and engagement contributions to the project. Analysts used a core set of questions related to each use for all interviews, tailoring the questions based on the document review (Table 2). Topics included confirming the findings and use of findings, critical factors that led to use of findings, how researchers and partners perceived the relationship between engagement and the use of study findings, and partner contributions throughout the research study and subsequent project, if applicable.

Table 2.

Core interview questions for each use case

Use case	Questions
All use cases	• Which study findings were cited in [document name]/ the focus of this implementation project/ were disseminated? • What made people interested in these findings? What made the findings relevant to these audiences? How did you know that the findings were relevant? • What were the critical factors that led to this use of study findings? • How, if at all, did engagement influence these study findings during the original award? Who was involved? When and how were they involved?
Integration of CER findings	• What was the process for integrating the findings into guidelines/ UpToDate / policy document? • What did you do, if anything, to facilitate that process during the original study? After the study? • What did study partners do, if anything, to contribute to this process?
Implementation of findings beyond the CER study.	• What led to implementation of the study findings at sites beyond the original study? • How, if at all, were partners involved in implementation?
Active dissemination of CER findings by external parties	• What initiated interest in these study findings? How did your team [Engagement Award team members] hear about the study findings? • How, if at all, were partners involved in disseminating findings to intended audiences?

Analysts conducted interviews on a rolling basis between October 2021 and March 2022. Interviews lasted 60 min and took place via a video conferencing platform, either with video or by phone only. Participants received an electronic gift card for their time. All interviews (except for one) were in English and were audio-recorded and professionally transcribed. For the interview in Spanish, a bilingual Spanish-speaking interviewer translated the recruitment email, and consent form, and conducted the interview in Spanish. A Spanish-speaking analyst prepared detailed interview notes in Spanish, which were translated into English following the interview and referenced during analysis.

Data analysis

We systematically organized primary and secondary data and employed thematic analysis to identify core themes and patterns at three levels: [1] individual projects or project sets [2], across projects within each use case, and [3] across the three use cases.

Project or project sets

For each individual project or project set, analysts reviewed data from the document review and interviews and abstracted information into a structured project analysis memo. The analysis memo included deductive fields aligned with the study aims and interview guide, as well as data that emerged inductively during review. The memo captured document review outputs including project details based on administrative data, the list of the documents reviewed, a study overview including CER study information, engagement practices, and descriptions of how study findings were used. The memo also included interview details (number of interviews conducted and characteristics of interviewees), and an episode profile that summarized how interviewees for each project described CER study findings, partner engagement and the perceived relationship between engagement and use of study findings. We shared a brief summary of the memo with the PI and project lead to confirm findings and incorporated any edits or clarifications.

Use case

To compare data across projects or project sets and inform each use case, two analysts per use case developed data analysis tables to synthesize information about how CER study findings have been used, what led to the use of study findings, how engagement facilitated use of study findings, and external factors influencing the use of study findings. Then, analysts prepared a report for each use case that summarized facilitators that interview participants perceived had led to the use of study findings and how engagement contributed to the use of study findings. The project lead (MEM), use case analysts (THB, ML, KF), and PCORI team members (LF, RM, KLC) reviewed all use case summary reports for clarity and relevance to the study aims.

Cross-use case

To synthesize findings across use cases and projects or project sets, the project lead (MEM) and three use case analysts (ML, KF, THB) used multiple techniques. First, we drafted a memo that reviewed conclusions across the three use cases to understand common versus unique aspects of engagement and the use of CER study findings. Then, each team member prepared a visual representation of the findings. The entire team discussed the memos, visual representations, and initial findings. The team met weekly to discuss emerging findings, share personal reflections on how the data aligned with or differed from expectations and test assumptions, biases, and conclusions. PCORI team members (LF, RM, KLC) reviewed iterative summaries of the emerging findings. This manuscript presents the synthesized results across use cases.

Results

Sample description

The final sample included 10 projects or project sets (the original CER study and subsequent project if applicable), for a total of 17 projects. In total, we conducted interviews with 42 people from 10 research projects, 3 implementation projects, and 4 Dissemination Engagement Award projects. Across the 17 projects, interviewees included 15 PIs or project leads, 2 research team members, and 25 partners.

The 10 research projects varied in terms of the study design and health condition of the original CER study. Six research projects used a randomized design (e.g., randomized controlled trial, cluster randomized trial) and four research projects used an observational design (e.g., cohort study). Four research projects focused on mental health, four focused on chronic conditions, three focused on youth, and two projects on geriatric care. Findings from the original CER studies also varied, with some findings demonstrating effectiveness of one intervention compared to another and others demonstrating no statistically significant difference between interventions. Table 3 shows the variation within each use (e.g., type of document for the integration use case or intended audience for the active dissemination use case). Among the projects included in this study, research projects were conducted between 2013 and 2019; implementation projects were conducted between 2018 and 2024; and Dissemination Engagement Award projects were conducted between 2019 and 2022.

Table 3.

Characteristics of uses of study findings by use case

Characteristic	Number of projects
Use case 1. Integration of CER findings ( n  = 3 research projects)
Guidelines	2 research projects (findings cited in 3 guidelines)
UpToDate articles	3 research projects (findings cited in 6 articles)
Document(s) that could inform policy	2 research projects (findings cited in 3 documents)
Use case 2. Implementation of findings beyond the CER study ( n  = 3 implementation projects)
Expansion from one setting to another	1 implementation project
Expansion to additional sites within same setting	1 implementation project
Clinical finding to program	1 implementation project
Use case 3. Active dissemination of CER findings by external parties ( n  = 4 Dissemination Engagement Award projects)
Intended audience
Patients or Community members	3 Dissemination Engagement Award projects
Professional societies/ member organizations	2 Dissemination Engagement Award projects
Clinicians	1 Dissemination Engagement Award project
Dissemination methods
Direct distribution to communities	3 Dissemination Engagement Award projects
Social media	3 Dissemination Engagement Award projects
Email newsletters, radio ads/podcasts, virtual webinars, posting materials on websites	2 Dissemination Engagement Award projects each
In-person events	1 Dissemination Engagement Award project

Table 4 provides a summary of engagement approaches as well as contributions that interview participants reported had facilitated the use of study findings by project or project set in each use case. We then summarize the main themes from the cross-use case analysis.

Table 4.

Engagement contributions identified as facilitating the use of study findings by use case project or project set

Project	Engagement approach	Engagement Contributions identified as facilitating use of study findings
Use case 1: Integration of CER study findings
Integration Research Project 1	Multiple perspective advisory group; intentionally hired community member as research staff	• Improved response to study outreach • Facilitated dissemination by connecting PI to a local community group; this group started sharing the materials at regular, community events. • Continued to work with the project team to implement study findings and develop a new intervention for caregivers
Integration Research Project 2	Single perspective advisory group; Input from social media survey and online focus group	• Recommended how to modify the approach for identifying cohort members, which increased the cohort size by about 30% • Prioritized which data and outcomes would be most meaningful to report out for the community • Recommended journals that would have the largest visibility • Clinical partner facilitated getting findings cited in two sets of forthcoming clinical guidelines after CER study
Integration Research Project 3	Multiple perspective advisory group	• Recommended focusing study aims on patient-clinician communication • Suggested both phone and mail surveys for data collection to reach study participants who may be hard of hearing, thus expanding the reach of the study • At least one advisory group member co-authored every publication • Co-authors prioritized which secondary data analyses would be most meaningful for publication • Directly advocated for the use of intervention at meetings with government agency
Use case 2: Implementation of findings beyond the CER study
Implementation Project Set 1
Research Project	Single perspective advisory group; Ad hoc engagement with clinicians	• Helped develop and then adapt a feasible and easy-to-access program prior to and during the research project
Implementation Project	Multiple perspective advisory group, working groups	• Helped adapt the intervention, including broadening it for different diagnoses • Connected researchers with advocacy organizations • Helped problem solve challenges (e.g., low completion rates)
Implementation Project Set 2
Research Project	Multiple perspective advisory group	• Helped to develop a feasible and easy-to-use program prior to research project • Adapted the intervention for the study population • Helped to problem solve when the study encountered low survey completion rates
Implementation Project	Multiple perspective advisory group	• Helped determine next steps, based on study findings, for the project • Developed messaging and outreach strategies for implementation sites • Connected researchers with contacts at implementation sites • Adapted in-person training materials to an online training format
Implementation Project Set 3
Research Project	Patient co-investigators; professional society; input from interviews	• Patient co-investigators helped write proposal, determine study design, develop the survey, and problem solve when there were recruitment and retention issues • Added quality of life as an outcome based on interviews
Implementation Project	Patient co-investigator; professional society; site-based patient representatives	• Helped to determine next steps for findings and provided input on program to improve prescribing practices • Adapted program and developed educational modules for implementation project
Use case 3: Active Dissemination of CER findings by external parties
Dissemination Project Set 1
Research Project	Multiple perspective advisory group; intentionally hired community member as research staff; input from interviews and focus groups	• Developed intervention prior to research project based on input from interviews and focus groups with community members • Improved feasibility by advising on recruitment and approaches • Reviewed and adapted all study materials including the intervention • Interpreted study results • Co-developed publications
Engagement Award	Patient co-investigator; multiple perspective advisory group; partner-led work groups	• Adapted and created new materials for the intervention • Actively distributed educational materials using a variety of in-person and virtual methods
Dissemination Project Set 2
Research Project	Multiple perspective advisory group	• Tailored intervention materials to better address the needs of patients • Advised on the relevance of the intervention
Engagement Award	Included patients as team members; representatives from patient organizations, professional societies, and member organizations	• Assisted in conducting focus groups which informed the dissemination strategy • Adapted and posted educational materials • Disseminated materials via email outreach
Dissemination Project Set 3
Research Project	Multiple perspective advisory group	• Provided guidance and oversight of the project during its implementation • Assisted with decision making and helped to resolve problems during CER study
Engagement Award	Multiple perspective advisory group	• Convened a planning committee of clinical leaders to provide guidance and input on the conference sessions • Involved patients in presentations which gave a real-world perspective on the nuances of implementation • Drafted a research framework for advancing the evidence generation necessary for widespread, population-based implementation strategies
Dissemination Project Set 4
Research Project	Patient co-investigator; Multiple perspective advisory group; input from town hall	• Developed intervention prior to research project based on input from community town halls • Developed culturally tailored educational materials • Used their connections to help recruit hundreds of community ambassadors • Interpreted study findings
Engagement Award	Hired community member as research staff; representatives from organizational partners	• Adapted materials from the research project for dissemination • Actively distributed educational materials using a variety of in-person and virtual methods

Across projects, the most common engagement approach was an advisory group, either an advisory group with members representing multiple perspectives (e.g., one group with patients, clinicians, and insurers) or an advisory group with members representing a single perspective (e.g., a group with patients). Other engagement approaches included: partner co-investigators; hiring community members as research team staff; working groups that integrated different perspectives and had responsibility for specific project activities; ad hoc engagement with professional societies or other organizations; and input approaches such as interviews, focus groups, or town halls.

Perceived partners’ contributions to the CER study that facilitated the use of study findings

Across use cases, participants described three primary activities in which engaged partners made contributions that facilitated the use of study findings:

• Generating relevant study findings.

• Distributing study findings strategically.

• Making connections to people or organizations outside the study team.

Below, we describe each facilitator and the partner contributions on the CER study that researchers and partners reported as helping to lead to the uses of study findings. Table 5 highlights examples of these contributions by activity.

Table 5.

Examples of partner contributions on the CER study that help to facilitate use of study findings

Activity	Examples of partner contributions on CER study
Generating relevant study findings	During study planning and conduct, partners: • Provided input on aims, research questions, and outcomes • Co-wrote proposal • Recommended a process for better identifying the population in medical records for an observational cohort study • Designed recruitment materials to improve enrollment • Led study recruitment • Developed, tested, and helped to adapt the intervention • Suggested ways to increase participation in the intervention • Helped to resolve challenges with retention of study participants • Collected data for the study • Provide input during data analysis • Helped to interpret findings • Prioritized which findings were most relevant to intended audiences
Distributing study findings strategically	During the study, partners: • Suggested well-known journals or publications • Co-authored manuscripts • Presented at conferences • Shared findings via personal and professional networks
Making connections to people or organizations outside the study team	At the end of or after the study, partners: • Were part of committees or organizations producing clinical guidelines (Integration of CER findings use case only) • Shared findings to interested parties for dissemination (Active dissemination of CER findings by external parties use case only)

Generating relevant study findings

Across projects, researchers and partners reported that relevance drove use of findings, and partner engagement during study planning and conduct facilitated the generation of relevant findings. When asked what made study findings relevant, interview participants said that their findings provided an innovative solution to bridge a care or knowledge gap, addressed health inequities, or had unique or clear findings within the context of the evidence base. In addition, interviewees in the implementation use case responded by describing intervention characteristics that facilitated their use more broadly. Specifically, interventions were perceived to be useful, low cost, easy to use, and accessible by various users, such as patients or clinicians – characteristics that would enable their use in health delivery settings.

Partner contributions improved relevance by helping to focus the CER study on research questions and outcomes that matter to intended audiences and by contributing to analysis and interpretation. For example, a researcher said:

They [the partner co-investigators] helped us write the proposal, think about the outcomes, design the surveys, and then when we collected the data, helped us interpret [the results].

To inform and refine relevance of research aims, research questions, and study measures, partners collaborated as co-investigators or provided guidance as advisory board members. Research teams also collected broad input from partners prior to or early in the study through interviews, focus groups, and town halls.

In addition, interview participants identified ways partners contributed to analysis and interpretation that improved the relevance of study findings, for example by prioritizing results. One researcher described:

By the time we had to do the analysis, we had so much data that it was impossible to write it all up. So that was guided by stakeholders and their advice to us in terms of what should be the number one priority.

Finally, partner contributions improved study feasibility, or the ability to complete study activities, as well as the user-centeredness and acceptability of study materials, by reflecting the needs and preferences of patients, clinicians, and other partners. These contributions occurred during intervention design and adaptation, recruitment and enrollment, and data collection and helped projects complete the studies, achieve their recruitment goals, and ultimately generate the study findings.

Distributing study findings strategically

Interview participants said releasing findings through well-known journals or conferences and sharing findings through various networks was an important and necessary step toward the use of study findings.

Partner contributions helped to ensure findings were distributed strategically. Partners co-authored manuscripts, determined where and how findings were shared, and informally and formally distributed findings through their own personal and professional networks. One researcher noted:

Because we had a robust stakeholder group and stakeholders as the co-authors of that paper, the word got out into the community as well. There’s both formal and informal ways to disseminate this.

Another researcher described their collaboration with community partners:

An example of one thing that’s been a really great collaboration is that several of the people on the advisory committee are on [a community workgroup]. I was invited onto that workgroup through them and asked to present…We presented our findings there, and people were really excited about it…One of the things that wound up happening just this last year is that we engaged the Latinx community again through the [community workgroup]. They wound up putting on several Spanish speaking groups that included our easy to read [interventions and other tools].

Making connections to people or organizations outside the study team

In three projects in two use cases, interviewees noted how personal connections helped to inform the use of study findings at the end or after the CER study. For two projects in the active dissemination use case, partners on the CER study facilitated connections to people interested in study findings who then applied for Dissemination Engagement Awards. In the integration use case, one research project had a clinical partner who was the chair of two organizations responsible for creating guidelines and facilitating the citation of the CER study findings in forthcoming guidelines.

Partner contributions to implementation and dissemination efforts after the CER study

Interviewees described how partners continued to facilitate the use of study findings by contributing to subsequent implementation and dissemination projects. (This theme was not relevant to the integration use case since a subsequent project or award was not associated with that use.)

Influencing early stages of implementation projects

Partners helped to inform the focus of two implementation projects. In all three implementation projects, partners contributed to preparing interventions, either by developing an intervention based on CER study findings, refining elements of a tested intervention, or adapting interventions to specific contexts. Interview participants on two implementation projects said partners supported implementation site identification and recruitment by developing outreach messaging and connecting researchers with contacts. For example, one researcher said:

A stakeholder advisory member directly linked us to [implementation sites] and individuals within [sites] that she thought would be interested in the intervention. She facilitated our connecting with them.

Because all implementation projects in this use case were still in process at the time of the interviews, interview participants could not report the ultimate impact of this engagement.

Influencing every aspect of Dissemination Engagement Award projects

Interview participants described engagement as essential at every step for the Dissemination Engagement Award projects. Interviewees described how partners created dissemination products; adapted messaging about CER study findings to meet the needs and goals of communities, including making them culturally and linguistically appropriate; and communicated findings to intended audiences. In two projects, partners further adapted or tailored educational materials to align with the cultural needs of the specific populations being served. For example, one community partner said:

Our whole script was set up and that’s why I say cultural tailoring is really important. You have to meet people where they are, and that’s just not physically, but emotionally, through language and making it [easier] for people to understand… When I would talk to the group, we would talk about that and use terms and words that people like the regular lay person would understand. Everything that we had was set up like that. Then we knew how to translate it. Through my experience, I knew how to translate it to make them… It was called messages for us by us.

Also, across Dissemination Engagement Award projects, partners actively distributed materials to specific audiences using in-person and virtual methods, including distributing directly to community members, posting materials on social media and partners’ websites, emailing information in newsletters, sharing messages via radio ads and podcasts, leading virtual webinars, and holding in-person events.

Observed attributes of engagement supporting partner contributions across projects

In our analysis, we observed three attributes of engagement across projects that may have supported partner contributions, including early and ongoing engagement, leveraging partners’ connections and understanding of community needs, and use of multiple engagement approaches. Although not directly related to the primary study aim, this theme emerged inductively during analysis, and provided insight into project teams’ approaches to engagement that may affect the ability of partner contributions to facilitate use of study findings.

Teams had early and ongoing engagement, often with long-standing or positive working relationships

In all 10 projects or project sets, interview participants described early and ongoing engagement through pre-existing relationships, work together on multiple projects (before and after the projects in this study), or positive working relationships.

Relationships between partners and researchers often developed prior to the CER study, and partners often worked with researchers on multiple projects funded by PCORI and others. For example, one researcher said:

We continue to collaborate to this day, even though the research project’s been over for three or four years now… We had a separate project and a continuation study that’s still ongoing. Our stakeholders are, not all of them, of course, but we keep in touch with the ones who want to be engaged. I imagine that there’s more work that’s going to happen in the future with their help and engagement.

Across projects, these relationships varied in terms of how many relationships were pre-existing (e.g., with one partner or an established group), length of relationship (e.g., recently met at a conference or worked together for years), and in how they had worked together (e.g., conversations, worked together on a previous project).

In three projects, interviewees explicitly noted that positive working relationships facilitated engagement contributions that then helped to generate relevant CER study findings. For example, on one project, the team built trusting relationships by collaborating with community members from the inception of the work, making sure it made sense to them and incorporated their vision of success. The team identified strategies that would reach the intended audiences, letting the community drive the approach and remaining flexible. They evaluated actions to see if what was applied had benefit or required modification. As another example, patient advisory group members discussed how the team did not function with power imbalances (e.g., giving more deference to the opinion of clinicians or researchers). One partner said:

I had a great time feeling that we were really engaged…There was nothing about somebody being a senior doctor or we had to treat somebody with more respect. There were no hidden agendas whatsoever, we just worked.

Teams leveraged partners’ connections and understanding of their communities

Across the three use cases and in at least 8 of the 10 project sets, interview participants described leveraging their partners’ connections and their understanding of community needs. These connections and understandings often facilitated the uses of CER study findings described above. For example, partners connected the PI with local community groups to support dissemination of findings; partners facilitated connections between the research team and professional contacts to support messaging development for implementation sites; and partners facilitated connections between the original CER study and project leads for the Dissemination Engagement Awards.

Teams used multiple approaches to support engagement (e.g., co-investigators, advisory groups)

Of the 17 projects, 11 reported using multiple engagement approaches: 5 projects used three engagement approaches, and 6 projects used two engagement approaches. Engagement approaches ranged from those that enabled input (e.g., interviews, focus groups) to those that enabled consultation and collaboration (e.g., advisory boards, co-investigators). For example, the engagement approach on one research project (Dissemination Project Set 4) included a patient co-investigator and an advisory group with members representing multiple perspectives; this project had also previously elicited input from town halls with community members to develop the intervention. No projects solely used approaches that only elicited input from partners (e.g., interviews, focus groups).

Discussion

Although PCORI and other organizations have required or undertaken engagement in research in part with the intent that it would facilitate use of study findings, the field is still at an early stage in understanding the relationship between engagement and use and uptake of study findings. This exploratory qualitative study used a systematic approach to identify concrete examples of how engagement can facilitate use of study findings, especially when engagement contributions occur in meaningful ways. Figure 1 provides a visual representation of the findings from this exploratory study and suggests a framework for the influence of engagement throughout the research lifecycle, starting with planning and conducting a CER study, continuing to subsequent dissemination and implementation efforts, with the ultimate goal being to facilitate uptake of study findings into clinical practice.

Fig. 1.

Visual representation of findings: how meaningful engagement throughout the research lifecycle facilitated uses of CER study findings

The influence and impact of engagement on study planning and conduct has also been demonstrated in other studies, including engagement’s influence on determining study outcomes and measures and data collection [8, 9]. The current findings also provide new insights into engagement contributions at later study stages, such as analysis or distribution of findings, which can inform subsequent use of findings. Further, the findings highlight how partner contributions continue in dissemination and implementation efforts.

Although many of the factors that can lead to use of study findings are beyond the control of study teams, the current findings suggest engagement approaches that, with further examination, may be found to help teams facilitate such use. Specifically, the attributes of engagement identified in this study may affect the extent to which partners’ contributions have a meaningful influence on study planning and conduct, which in turn facilitates use and ultimately uptake of study findings.

First, interview participants described early engagement (e.g., relationships developed prior to the CER study) and sustainable partnerships throughout and beyond studies as important in facilitating use of study findings, specifically with regards to improving relevance of study findings and supporting dissemination and implementation efforts. Others have noted the challenge of sustaining relationships [32], especially without funding to compensate partners, since this work often happens outside or between funded projects. To address these challenges, funders or academic institutions can consider allocating resources to support partnerships before or between studies. Second, researchers can be intentional about identifying and conducting outreach to partners early in the study, and specifically to partners who are from or have direct connections to communities affected and would be most likely to use study findings. Finally, project teams used multiple approaches to collaborate with and get input from partners. Engagement approaches in these projects, such as co-investigators and advisory boards, often enabled collaboration and shared leadership. No projects relied solely on approaches with unidirectional input. When designing an engagement strategy, teams can consider applying multiple engagement approaches, especially those where partners are in positions of leadership on the study. Other studies have identified strategies for effective partner engagement [32–35] or have connected partner engagement with increased research relevance and applicability of the results [36, 37]. Future research, especially prospective studies, can further explore the connection between these attributes of engagement, the influence and impact of partner contributions, and the use and ultimate uptake of study findings.

This study has some limitations. This formative study was exploratory and used a purposeful sampling approach to select a small number of projects. These projects represented three uses of study findings that were feasible for PCORI-funded research projects to achieve since PCORI funding started and where PCORI had documentation related to the use. While this sampling approach allowed us to gain an in-depth understanding of specific projects, it limits the generalizability of the findings. Further, more than two years had passed between the completion of the 10 research projects and the time of these interviews; interview participants may have not recalled all the engagement contributions or factors that led to the uses of study findings. As possible, we conducted interviews with multiple partners across projects. However, we did not conduct interviews with all types of partners involved in each project.

Even with these limitations, this formative study highlighted ways engagement in research contributes throughout the research lifecycle from study planning to uses of study findings on the pathway to the uptake in clinical practice. It also raises additional research questions that can inform how research funders and institutions can better facilitate incorporation of evidence into practice by supporting engagement in research. Specific questions that could be explored are: How does engagement in research facilitate integration of findings into clinical practice? Which engagement approaches (e.g., behaviors, techniques, practices) during and after CER studies facilitate uses of study findings, and under which conditions? How does engagement in research contribute to other uses of study findings not examined in this study?

Conclusions

The findings from this formative study identified concrete examples of ways engagement in and after CER studies can facilitate use of CER study findings. Future examination of these topics can continue to demonstrate the value of engagement in research as well as move toward an understanding of the context and conditions to improve the quality and impact of engagement. Engagement in research, using strategies with demonstrated effectiveness, will continue to be a critical component leading to more patient-centered and usable CER that can inform decision making and improve health and healthcare outcomes.

Author contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by MEM, ML, THB, KF, and DA at the American Institutes for Research. MEM, ML, and THB at the American Institutes for Research wrote the first draft of the manuscript and responded to reviewer comments. All authors reviewed and commented on iterative versions of the manuscript. All authors read and approved the final manuscript.

Funding

The Patient-Centered Outcomes Research Institute (PCORI) contracted with the American Institutes for Research to conduct this research (PCORI Task Order #3 under IDIQ-TO#3-AIR-ENG-AOSEPP_2017.10.26, which ends in October 2024). Staff at AIR led the conceptualization, design, prioritizing candidate projects and contacting projects about participation, data collection and analysis, and preparation of the manuscript. Staff at PCORI had a role in the conceptualization, design, identifying candidate projects for inclusion, data analysis, decision to publish, and preparation of the manuscript.

Data availability

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Declarations

Ethics approval and consent to participate

The American Institutes of Research Institutional Review Board reviewed this study and consent procedures.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.