“Natural Experiments for Translation in Diabetes 2.0 (NEXT-D2) Network Engagement In Action: Stakeholder Engagement Activities to Enhance Patient-Centeredness of Research”

Abstract

Background:

Stakeholders (i.e., patients, policy makers, clinicians, advocacy groups, health system leaders, payers and others) offer critical input at various stages in the research continuum, and their contributions are increasingly recognized as an important component of effective translational research. Natural experiments in particular may benefit from stakeholder feedback in addressing real world issues and providing insight into future policy decisions, though best practices for the engagement of stakeholders in observational studies are limited in the literature.

Methods:

The Natural Experiments for Translation in Diabetes 2.0 (NEXT-D2) network utilizes rigorous methods to evaluate natural experiments in health policy and program delivery with a focus on diabetes-related outcomes. Each of the eight partnering institutions incorporates stakeholder engagement throughout multiple study phases to enhance patient-centeredness of results. NEXT-D2 dedicates a committee to Engagement for resource sharing, enhancing engagement approaches, and advancing network-wide engagement activities. Key stakeholder engagement activities include Study Meetings, Proposal Development, Trainings & Educational Opportunities, Data Analysis, and Results Dissemination. Network-wide patient-centered resources and multimedia have also been developed through the broad expertise of each site’s stakeholder group.

Conclusions:

This collaboration has created a continuous feedback loop wherein site-level engagement approaches are informed via the network and network-level engagement efforts are shaped by individual sites. Emerging best practices include: incorporating stakeholders in multiple ways throughout the research, building on previous relationships with stakeholders, enhancing capacity through stakeholder and investigator training, involving stakeholders in refining outcome choices and understanding the meaning of variables, and recognizing the power of stakeholders in maximizing dissemination.

Introduction

Engagement in research is defined as the meaningful involvement of nontraditional partners throughout the research process, including planning for, conducting, and disseminating study results.¹ Stakeholders (i.e. patients, policy makers, clinicians, advocacy groups, health system leaders, payers and others) play a critical role in accelerating the transition of new findings from laboratory discovery to clinical and community implementation. By offering input at various stages in this research continuum, stakeholder contributions are increasingly recognized as an important component of effective translational research.^2–4 Many well-designed, peer-reviewed research studies have not resonated with potential end-users or created the intended effects on policy or practice, due in part to limited stakeholder engagement before and during the research process. Stakeholders are often motivated to engage with researchers in order to translate meaningful findings to patients, with quicker dissemination and greater relevance than usually possible with traditional research approaches.^2–7 Though stakeholders contribute as partners in various types of research designs, natural experiments in particular may benefit from stakeholder feedback in addressing real world issues.⁸ Natural experiments are real-world programs and policies that are not specifically designed by researchers, but can be carefully studied with stakeholder-engaged research to provide insights into future policy decisions.⁸

Through a unique consortium of eight academic institutions and three funding agencies, the Natural Experiments for Translation in Diabetes 2.0 (NEXT-D2) network utilizes rigorous methods to evaluate natural experiments in health policy and program delivery with a focus on diabetes-related outcomes.⁹ Initially launched in 2010 through funding from the Centers for Disease Control and Prevention (CDC) and National Institutes of Health (NIH), the Patient-Centered Outcomes Research Institute (PCORI) now also supports the NEXT-D2 network by funding comparative effectiveness research (CER) to identify programs and policies that are most helpful to patients with diabetes and prediabetes. Each institution incorporates stakeholder engagement throughout multiple study phases to enhance patient-centeredness of results by having the wealth of voices and expertise inform key study decisions.¹ The following 8 institutions comprise NEXT-D2: Harvard University, the Icahn School of Medicine at Mount Sinai (ISMMS), Northwestern University, Oregon Health & Science University (OHSU), Penn State Health Milton S. Hershey Medical Center (PSU), Tulane University, University of California Berkeley (UC Berkeley), and the University of California Los Angeles (UCLA). Institutions’ research project descriptions are reported in prior publications^9–23 as well as this Supplement; briefly, each study focuses on measuring the effects of interventions in the following categories: health insurance expansion, value-based healthcare and financing models, and innovations in care coordination.

Compared to other research designs, natural experiments present distinct challenges to stakeholder engagement in that they lack phases during which stakeholders are typically engaged, such as recruitment and data collection, and often involve large, complex data sets. There’s also a scarcity of benchmarks in the literature for effective engagement of stakeholders in natural experiments. Recognizing these limitations, the NEXT-D2 network dedicated an Engagement Committee to develop engagement practices among studies and to incorporate network-level activities. The Engagement Committee originated not as a funder requirement but rather an organic response to the growing interest in identifying successful approaches when partnering with stakeholders. Although only three awardees were required to engage patients and stakeholders in their studies (ISMMS, PSU, and Tulane), early meetings demonstrated that all consortia members engaged stakeholders in their projects in various ways. Through the Engagement Committee, sites have seen potential value in additional techniques of stakeholder engagement and further developed engagement plans post award. This collaboration has benefited from continuous feedback that informs both site and network-level engagement approaches. This paper outlines individual project engagement activities, which vary across the engagement spectrum, as well as the network-wide activities proposed and underway. Further, we offer insight into recommendations for future studies to incorporate our emerging and promising engagement practices.

Methods

Engagement Committee Governance

The Engagement Committee was created to provide an opportunity for monthly sharing of site-level engagement activities and lessons learned across the NEXT-D2 network. As sites had varying familiarity with stakeholder engagement at baseline, a key goal of the Committee is to share knowledge and experience across all eight partnering institutions. The PCORI-funded sites (ISMMS, PSU, and Tulane) lead efforts and share guiding resources from their funder to the other partner institutions. Specifically, the Patient-Centered Outcomes Research (PCOR) Guiding Principles (reciprocal relationships, co-learning, partnerships, transparency, honesty, and trust) from PCORI’s Engagement Rubric²⁴ have served as the network’s framework to both form and sustain meaningful and authentic engagement. The Committee relies on these principles to ensure their site’s study stakeholders are treated with respect and as equal contributors to their team; are encouraged to share their experiential or expert voices; are compensated appropriately for their time and efforts; and are empowered to share in study decisions through an open and transparent dialogue.

Investigators, project staff, and engagement coordinators attend Committee meetings remotely to discuss engagement strategies, challenges, and successes, overseen by two co-chairs (JK, VM). Additionally, network-level engagement activities are operationalized and executed during these monthly meetings. Network-wide in-person meetings, held three times each year, provide an opportunity to highlight engagement activities and their impact and to conduct network-wide engagement activities. Individual sites and the network benefit from knowledge and resource sharing of specific best practices in stakeholder-engaged research. For example, the Committee discussed human subjects protection training programs that are layperson-friendly during an in-person meeting and shared related materials among institutions.

Site Level Stakeholder Recruitment and Governance

Partnering institutions recruited stakeholders of various types that shared an interest and knowledge in each study’s focus (e.g., patients, clinicians, government agencies, national professional organizations, local community organizations, data vendors, health insurance industry leaders and program/policy implementation stakeholders) (Table 1). Varying approaches to stakeholder recruitment took place; in some cases, patients were referred to projects by serving on other advisory boards or through clinician referrals. For example, ISMMS formed a group of patients, caregivers, and clinician providers from individuals who collaborated on the New York City Clinical Data Research Network (CDRN)²⁵ and then recruited additional members with relevant expertise to form a diabetes-focused “accelerator” (a group of stakeholders from diverse backgrounds who collaborate to generate new research questions, ideas, approaches, and projects).²⁶ PSU recruited patient partners from clinician referrals and participation in prior related studies. Northwestern’s team relied upon a Patient and Community Advisory Committee (PCAC) created for the Chicago Area Patient-Centered Outcomes Research Network (CAPriCORN). PCAC members were chosen from across the region and included individuals with personal experience in a priority health condition, health professionals caring for such patients, community members advocating for patient perspectives and representatives from disease-specific patient advocacy/voluntary health organizations.²⁷ Based on these experiences, recommendations to future teams include leveraging existing connections within their professional networks to identify appropriate stakeholders.

Table 1.

Types of Stakeholders at Each Partnering NEXT-D2 Institution.

Academic Institution	Types of Stakeholders Engaged
	Patients	Clinicians	Government Agencies (i.e. public health officials, payers)	National Professional Organizations (i.e. The Obesity Society)	Local Community Organizations (i.e. PA Nutrition Education Network)	Data Vendors*	Health Insurance Industry Leaders	Program/Policy Implementation Stakeholders
Harvard University		•	•			•	•
Icahn School of Medicine at Mount Sinai (ISMMS)	•	•	•		•			•
Northwestern University	•	•			•
Oregon Health & Science University (OHSU)	•	•				•
Penn State Hershey Medical Center (PSU)	•	•	•	•	•
Tulane University	•	•					•
University of California Berkeley (UC Berkeley)			•					•
University of California Los Angeles (UCLA)			•	•			•

^*Harvard’s data vendor sells health insurance claims data from a large national health insurer over approximately 2000 – 2019. OHSU works with OCHIN (Oregon Community Health Information Network) to access clinical data and partner on data extraction and analysis.

To support meaningful and sustainable partnerships across a diverse group of collaborators, sites draw from the PCOR Engagement Principles^24,28 when planning and conducting engagement activities. Study teams do not move forward with a decision until all voices have been considered and consensus among stakeholders has been met, upholding the reciprocal nature of this collaboration. For example, when PSU was developing their patient-centered study website, they sought feedback from patient partners throughout development and provided opportunities for voting of specific features and resources for website inclusion. Sites also maintain open and transparent communication by detailing study decisions through follow-up documentation and providing trainings as necessary. Further, stakeholders are treated as equal members of the research team and are reimbursed for any study related time or travel. Finally, stakeholder evaluations present an opportunity to monitor the quality of engagement and identify areas for improvement. Both PSU and Tulane’s engagement evaluations specifically address adherence to PCORI’s engagement principles by including questions that measure levels of trust, understanding of the research process, stakeholder influence, etc. Several partnering sites (OHSU, PSU, ISMMS, Tulane) have dedicated an engagement coordinator to serve as a liaison between the research team and stakeholders. The engagement coordinator is responsible for executing the site’s engagement plan and ensuring meaningful engagement throughout the research process, a critical role that we encourage future groups to identify and support.

Given the diverse professional and experiential repertoire of stakeholder partners, sites offer opportunities for stakeholders to engage at multiple time points and with varying levels of time and effort commitment. PSU developed a Needs Assessment for patient partners to elucidate prior experience in research, goals of involvement, and learner preferences; results guided their engagement plans specifically with regards to training needs and meeting frequency to optimize engagement. They meet monthly with patient partners to execute more day-to-day research operations, whereas other study stakeholders are convened quarterly to troubleshoot study challenges and plan future directions. Similarly, while all of Tulane’s stakeholders (patients, payers, clinicians) are included in monthly conference calls, they also hold separate bi-monthly calls with just the patient partners and engagement team. This allows the Tulane team to address any points that require further clarification for patient partners or gather additional input from patient partners that they may not have the opportunity or confidence to share with the whole group. The ISMMS team includes a board whose members commit several hours per month and contribute to all stages of the study. However, the team also engages with patients and care management leaders through single one-hour focus groups and interviews. This allows more individuals an opportunity to participate not just as research participants, but also as contributors to research design, outcome choice, and dissemination plans.

We encourage future teams to delineate through a collaborative process the roles and responsibilities of each stakeholder so that expectations and time commitments are identified and revisit these periodically to avoid tokenism and ensure meaningful involvement. For example, OHSU originally planned to meet with patient stakeholders monthly but experienced challenges with having enough appropriate content for these meetings. Convening fewer meetings resulted in richer feedback generation of network-wide and site level engagement activities.

Site Level Engagement Activities

Though engagement activities are specific to a site’s particular research context, there are several similar approaches employed by studies across the NEXT-D2 network that are highlighted below (see Table 2 and Figure 1).

Table 2.

Engagement Activities, Partnering Institution Examples and Resultant Study Impact.

Engagement Activity	Partnering Institution Examples and Resultant Study Impact
Study Meetings: Web-based quarterly meetings and annual in-person meetings offer opportunities to provide research and engagement updates, discuss methods, data analysis, and policy implications. The patient experience as well as stakeholder expertise influence study decisions made at meetings and are instrumental in shaping study direction. More frequent meetings are encouraged for stakeholders more intensely involved. *PCOR Guiding Principles Applied: Transparency, Honesty & Trust. Reciprocal Relationships, Co-learning, Partnerships	In addition to monthly stakeholder calls, Tulane holds separate bi-monthly calls with just the patient partners and engagement team to address any points that require further clarification and gather additional input from patient partners that they may not have the opportunity or confidence to share with the whole group. This has opened lines of communication and created a safe environment for patient voices to be heard.
Proposal Development: Engaging stakeholders during the proposal stage provides an opportunity to build a rapport of trust and mutual respect among the research team and stakeholders that is beneficial in sustaining engagement throughout project life. Patient and other stakeholder feedback strengthen proposals by ensuring relevancy when identifying study populations and encouraging research teams to explore research areas they feel are important, thus resulting in more patient-centered outcomes. PCOR Guiding Principles Applied: Transparency, Honesty & Trust, Reciprocal Relationships, Co-learning	Engagement with study stakeholders at Harvard during proposal development facilitated buy-in from data partners whose expertise were critical to project success. Engagement with payer stakeholders at study conception was critical to identify the specific intervention targeting high-cost, high need Medicaid beneficiaries for study as a natural experiment, and provided preliminary insurance claims data to strengthen UCLA’s study proposal. Tulane’s stakeholders encouraged the research team during proposal development to investigate not just exposure to chronic care management services, but also the content of such services. This additional focus was incorporated into the qualitative research plan for the project. UC Berkeley engaged their advisory group in an expert panel process during proposal development to prioritize implementation foci and strategies by State Innovation Model (SIM) states that would have the most impact on patient outcomes.
Trainings: Offering onboarding, research training, and relationship building opportunities help build research teams that cultivate trust, mutual respect, and continuous learning, thus upholding the reciprocal nature of this partnership. Sites have presented their training programs at network meetings for other partnering institutions to incorporate into their training programs where applicable. PCOR Guiding Principles Applied: Reciprocal Relationships, Co-learning, Partnerships	Patients and stakeholders at Northwestern participated in an on-boarding process that included introductory meetings, the importance of stakeholder engagement, and an operations manual. Patient partners at PSU are offered timely study-related trainings based on selected topics of interest to prepare them for the current research phase (e.g., data extraction, data analysis, dissemination). Applying adult learning principles to these foundational trainings have enhanced subsequent engagement in these research phases. PSU has shared their trainings throughout the network for uptake across sites.
Data Analysis: Partner institutions utilize the various skillsets of their teams to enhance patient-centeredness of study results by working together to troubleshoot data issues, use their lived or professional expertise to identify variables of interest, and prioritize the analysis plan. PCOR Guiding Principles Applied: Transparency, Honesty & Trust, Reciprocal Relationships, Co-learning	PSU engaged stakeholders in data discussions when preliminary analyses revealed a low uptake of obesity counseling among patients with diabetes. Stakeholders provided feedback on possible barriers to implementation within primary care and their input prioritized future analytic directions. OHSU engages a clinician stakeholder in interpreting study data and providing key clinical guidance to several project manuscripts, including interpretation of data, methods and outcomes from their perspective as Medical Director of a county Public Health Department. Project stakeholders at UC Berkeley provided formative feedback, assessed the face validity of research results, and aided in the interpretation of quantitative results. Patient partners at PSU conducted a review of Patient Reported Outcomes (PROs) collected at institutions within their Clinical Data Research Network (CDRN) and offered feedback on which measures they felt were most important to include during study analyses based on their lived experience.
Study Results Dissemination: Study results dissemination is accelerated through the extensive connections within each site’s stakeholder body. PCOR Guiding Principles Applied: Reciprocal relationships, Co-learning, Partnerships	Stakeholders at OHSU are engaged in the dissemination of research aims through presentations, interviews, and blogging. Patient and clinician stakeholders work with study staff to write and publish, attend key project meetings with the study team, participate as equal co-authors in manuscripts, and disseminate study findings to clinicians, patient advocates and policy groups (OHSU, PSU, ISMMS).

^*Patient-Centered Outcomes Research

Figure 1.

Key Engagement Activities Across Partner Institutions.

Study Meetings

Partnering institutions report open, bidirectional communication as the cornerstone for effective engagement. Therefore, each site secures regular study meetings to provide progress updates, incorporate stakeholder feedback to advance study aims, and plan for future milestones with their diverse and geographically dispersed stakeholder groups. The patient experience as well as stakeholder expertise influence study decisions and are instrumental in shaping study direction. For instance, the UC Berkeley research team has created an engagement structure to support their work on the state-level State Innovation Model (SIM). Their team facilitates web-based advisory group meetings quarterly, which involve brief presentations of research updates, discussion of methods, results, and policy implications among stakeholders, and stakeholder communication about activities and developments in states that inform their analyses. Tulane leverages their broad stakeholder connections through partnership with the Research Action for Health Network (REACHnet) to bring the important expertise of people who live with diabetes and who provide diabetes care as lay caregivers or clinicians to these study discussions. OHSU, PSU, and Harvard also utilize meetings to review data extraction, security and troubleshoot data issues. For example, when preliminary analyses revealed a low uptake of obesity counseling among patients with diabetes, PSU sought input from stakeholders at a study meeting to understand barriers to implementation within primary care and utilized their feedback to prioritize future analytic directions. Meetings at ISMMS provide an opportunity to collaborate on recruitment plans and troubleshooting, data collection tool development, and dissemination plan development as well as a forum for review of study progress and analyses. Study meetings also provide an opportunity for patient and other stakeholder partners to review and summarize key manuscripts in development and discuss how engagement contributes to the science of these papers (e.g., more patient-centered outcomes).

In addition, meetings are used to generate feedback on the NEXT-D2 network-wide initiatives and activities. For example, the development of the NEXT-D2 website offered an opportunity for translating research terminology such as “natural experiment” used in study descriptions into lay language, which was led by stakeholders across partnering sites. Though most sites hold meetings quarterly, stakeholders more intensely engaged meet more frequently (e.g., weekly at UCLA; monthly at PSU). This has been a valuable opportunity, particularly for patient partners at PSU, to form a comradery and to participate in more day-to-day study operations. An annual in-person study meeting is also hosted by most study sites to strengthen relationships and advance study aims in a face-to-face collaborative setting.

Proposal Development

Recognizing the importance of engaging stakeholders as early as possible, several sites incorporated stakeholder feedback during proposal development to help shape study design and refine study outcomes. For UCLA, engagement with payer stakeholders at study conception was critical to identify the specific intervention targeting high-cost, high need Medicaid beneficiaries for study as a natural experiment, and provided preliminary insurance claims data to strengthen their proposal. Similarly, Harvard began their engagement with study stakeholders during this initial research phase to facilitate buy-in from data partners whose expertise were critical to project success. PSU and ISMMS sought feedback from patient partners on study aims during their proposal development, resulting in outcomes that were more patient-centered. For example, patient partners at PSU felt weight was just as important an outcome measure as A1c given it is both a strong predictor of diabetes and monitored regularly during diabetes management. Since the diabetes accelerator already existed at ISMMS, they were able to move swiftly to engage a transdisciplinary group in proposal development who already had mutual trust and respect, and were ready to contribute their diverse perspectives into the proposal. UC Berkeley engaged their advisory group in an expert panel process to prioritize implementation foci and strategies by SIM states that would have the most [or the greatest/most significant] impact on patient outcomes. Finally, Tulane’s stakeholders encouraged the research team to investigate not just exposure to chronic care management services, but also the content of such services during proposal development. This additional focus was incorporated into the qualitative research plan for the project.

Trainings & Educational Opportunities

In order to build research teams that cultivate trust, mutual respect, and continuous learning, sites offer onboarding, research training, and relationship building opportunities. Northwestern engaged patients and other stakeholders in an on-boarding process that included introductory meetings, the importance of stakeholder engagement, and an operations manual for their advisory committee. PSU similarly met with patient partners during onboarding to outline study objectives and identify their role(s) throughout the project. For investigators with limited experience in stakeholder-engaged research, sites like ISMMS and PSU provide trainings on how to effectively conduct patient-centered research.

Given the various professional and educational backgrounds of study stakeholders, research trainings may be necessary to overcome knowledge gaps and enhance stakeholder engagement in key study phases. Sites like PSU and OHSU offer patient partners timely study-related trainings based on selected topics of interest to prepare them for the current research phase (e.g., data extraction, data analysis, dissemination). PSU applies adult learning principles to their training programs; created for the adult learner, content is presented in short (15–20 minutes), interactive segments that include group discussion, scenarios, and role-play to encourage meaningful discussion and enhance knowledge acquisition. Given the importance of data analysis in natural experiments – which is arguably one of the most challenging elements with which to engage community stakeholders - PSU offered patient partners a two-part mini-series training to provide them with 1.) an introductory overview of how data is prepared for analysis (i.e. sample topics included how to handle missing data, errors in dataset) and 2.) basic statistical tests and terminology as well as outcomes specific to PSU’s study. Partners have anecdotally shared that they benefited by learning about foundational topics in data analysis in a supportive learning environment and it enhanced their participation in data discussions. Sites have presented their training programs at network meetings for other partnering institutions to incorporate into their training programs where applicable. Training resources have also been shared with PCORI through their online resource repository, PCORnet Commons, to increase uptake of training as a critical component of engagement.

Data Analysis

Natural experiments provide an opportunity to analyze various types of data, unlocking another opportunity for stakeholder input. Partnering institutions utilize the skillsets of their teams to enhance patient-centeredness by working together to troubleshoot data issues and prioritize the analysis plan. For example, at PSU patient partners conducted a review of Patient Reported Outcomes collected at institutions within thr CDRN and offered feedback on which measures they felt were most important to include during study analyses based on their lived experience (i.e., they emphasized the strong link between heart health and diabetes and, therefore, encouraged the inclusion of cardiovascular self-report measures in secondary analyses). OHSU engages a clinician stakeholder in interpreting study data and providing key clinical guidance to several project manuscripts, including interpretation of data, methods and outcomes from their perspective as Medical Director of a county Public Health Department. Public health officials and payers as well as program/policy implementation stakeholders at UC Berkeley provide formative feedback, assess the face validity of research results, and aid in the interpretation of quantitative results focused on hospitalization rates, 30-day readmissions, and behavioral health outcomes. The expert panel was integral to the creation of their taxonomy used to classify SIM implementation variation. Using a 3-round modified Delphi expert panel process, they asked stakeholders to rank the most important factors that differentiate SIM states with respect to implementation and outcomes. The process resulted in criteria used to create a taxonomy of state SIM implementation based on emphasis on behavioral health in payment and delivery system reforms, multi-payer alignment and depth of testing of value-based payment reforms, and SIM funding per capita. A health system stakeholder at ISMMS who serves on the advisory board for the Health Home (the program under study) was able to provide critical insight into the meaning of variables present in administrative data sets and point out insightful interpretation of the data. When preliminary analyses at Tulane revealed that some patients receiving non-face-to-face care coordination services for chronic care management had good glycemic control even prior to receiving the services, their patient partners encouraged the research team to explore whether this approach helped with glycemic control specifically for those with elevated A1c prior to receiving services. The patient partners were particularly interested in how the program impacts those who need help with their diabetes. As a result of this input, the research team conducted sub-group analyses focusing on patients with poor glycemic control a priori. These examples demonstrate how stakeholders can partner in this phase to enhance patient-centeredness and ensure relevance of findings.

Study Results Dissemination

Study results dissemination is accelerated by leveraging the connections of each site’s stakeholders. Several sites (PSU, OHSU, UC Berkeley) have encouraged and facilitated stakeholder participation in scientific manuscript co-authorship and conference presentations as well as community-facing platforms (i.e. websites, interviews, blogs).²⁹ Patient and clinician stakeholders at OHSU work with study staff to attend key project meetings with the study team, participate as co-authors in manuscripts, and disseminate study findings to clinicians, patient advocates and policy groups. Sites like OHSU, PSU, and ISMMS leverage clinician stakeholder expertise to offer important insight in how study findings can be effectively disseminated to patient and clinician populations, as well as other relevant platforms from a clinical perspective. Additionally, patient partners at PSU contribute to the development and dissemination of study newsletters and a patient-facing study website to increase access to study resources and updates. Finally, study stakeholders from relevant government and professional organizations are well positioned to transfer study findings into action through legislation or improved practice. UCLA’s dissemination of study results to stakeholders had the additional benefit of resulting in future research opportunities. Specifically, their stakeholder was appreciative of the analyses and, as a result, offered the opportunity for the team to evaluate two similar programs. In this way, long-term stakeholder relationships can result in expanded research opportunities.

Engagement Evaluations

To determine the effectiveness of approaches and improve efforts, partnering institutions offer opportunities for stakeholders to evaluate site engagement and adherence to PCOR guiding principles. PSU offers opportunities for the evaluation of engagement efforts twice each project period; a qualitative interview conducted by a non-team member (1:1 with patient partners, group interview with other study stakeholders) to elucidate how successfully the research team has incorporated their voice and expertise and a quantitative evaluation to shed light on engagement activities that were most meaningful to partners and prioritize activities they are most interested in participating in the next project period. Similarly, ISMMS’s engagement coordinator facilitates periodic check-ins with board members to gain their feedback on activities. Tulane uses a 360-degree Engagement Assessment Tool to optimize engagement based on the PCOR Engagement Principles. The evaluation is administered to all investigators and study stakeholders and used to create an anonymous report that highlights strengths and weaknesses in relationships between stakeholders and the research team. This tool is used to facilitate discussion about areas for improvement and approaches for strengthening stakeholder engagement, as needed. Results from evaluations have strengthened engagement processes. For example, at PSU, a patient partner offered that she would like more time to connect with the investigative team as monthly meetings are led by the engagement coordinator only. To address this feedback, patient partners now meet quarterly with study investigators to engage in both study operations and engagement activity discussions.

Network Level Engagement Activities

The NEXT-D2 Engagement Committee values the importance of monthly study meetings as a space for resource sharing, enhancing approaches, and advancing network-wide endeavors. For example, PSU shared their community research ethics training with other interested sites to use as well, minimizing duplicative efforts and increasing uptake of effective resources. Sites also benefit by learning through others’ experiences overcoming engagement barriers in real-time and through supportive feedback.

Meetings are also used to execute network-wide engagement activities and draw from the various expertise of each site’s stakeholder group. For example, partnering institutions have solicited input from their diverse patient and stakeholder groups to develop lay public research definitions and lay abstracts of research studies as resources on the NEXT-D2 network website, https://uclahealth.org/nextd2/. The Committee discussed the most effective way to leverage stakeholder voices in this activity and shared best practices to ensure real engagement. Additionally, stakeholders across all sites have contributed to the development of a video directed towards patients that explains the importance of the NEXT-D2 network in identifying programs or policies that are most helpful to patients with or at risk of type 2 diabetes, using lay language to convey the importance of natural experiment evaluations. Stakeholders from various sites have provided input on the script and visual layout, resulting in a patient-centered resource that will also be housed on the network website. Sites have also lent their stakeholder expertise to studies experiencing data challenges, which has increased opportunities for stakeholder engagement across sites. For example, UCLA had established ties with stakeholders from UnitedHealth, and Harvard depends heavily on UnitedHealth for certain NEXT-D2 aims so connections were made between sites to inform Harvard’s study objectives and assist with data hurdles.

The NEXT-D2 Central Coordinating Center (CCC) hosts three in-person network-wide meetings each year. In addition to partnering institutions and their stakeholders, the CCC invites methodologists and health policy and program leaders with expertise relevant to the evaluation of natural experiments in diabetes care and prevention. Meeting structure includes panel sessions featuring invited guest speakers, presentations of research findings from NEXT-D2 sites, discussion of engagement approaches, and feedback on study design, analysis, and generalizability.

The CCC facilitated engagement of health policy and program leaders by inviting 15 stakeholders to the in-person network-wide meeting held in January 2019. Attendees included representatives from national funding organizations (NIH, PCORI, CDC) and stakeholder organizations (insurers, national policy organizations). The meeting included 2 panel sessions where invited guests shared details on their current activities and perspectives on the “state of the science” in Health Insurance Design, Lowering Pharmaceutical Costs, Best Practices in Diabetes Care and Prevention, and Addressing Social Determinants in Diabetes Care. In addition, the invited stakeholders provided feedback on findings from the NEXT-D2 network, commenting on study design, analysis, relevance and generalizability.

Conclusions

The NEXT-D2 network has provided infrastructure and support to studies utilizing CDRNs and other large secondary data sets to advance our understanding of diabetes policy and program effectiveness. The network also supports the pursuit of patient-centered outcomes by recognizing the importance of stakeholders as partners in research. This collaboration among eight academic institutions has created a continuous feedback loop wherein site-level engagement approaches are informed via the network and network-level engagement efforts are shaped by individual sites. Though partnering institutions developed engagement plans specific to their project needs and funding mechanism requirements, studies have made enhancements to their approaches through resource sharing and troubleshooting barriers with other sites. Similarly, the network has the advantage of gaining abundant stakeholder feedback on network-level engagement efforts by leaning on the robust and diverse stakeholders from each partnering site. This synergistic collaboration on both the micro and macro levels is a promising approach to enhancing patient-centered outcomes across the evaluation of different natural experiments.

The importance of stakeholder engagement in research is increasingly recognized.^6,30 However, there is limited information in the literature regarding engaging patients and other stakeholders in observational research using large data sets. PCORI-funded CDRNs have described the importance of stakeholder engagement to the development and governance of these resources.^29,31–34 Getting stakeholders on board and sustaining authentic partnerships throughout project life (and beyond) is critical in natural experiments due in part by the role reversal; instead of researchers developing the design of traditional randomized-controlled trials, natural experiments are in essence designed by stakeholders, who often control access to data that is needed to complete the evaluation of such experiments, and who are in the trenches to inform and lead researchers through the field. The experience of the NEXT-D2 network presented here adds a new example of stakeholder engagement in action across eight natural experiments. We have identified several emerging practices for this work, including: incorporating diverse stakeholders in multiple ways at time points throughout the research, building on previous relationships with stakeholders, enhancing capacity through stakeholder and investigator training, involving stakeholders in refining outcome choices and understanding the meaning of variables, and recognizing the power of stakeholders in maximizing dissemination. Future directions for engagement efforts of the NEXT-D2 consortium include further refining best practices in stakeholder engagement in observational studies, particularly in the areas of sustainability of partners throughout project life, engagement evaluation, results interpretation, and dissemination. Additionally, we hope to further utilize the network website and other dissemination vehicles to increase uptake of stakeholder engagement practices and resources.