Integration of Stakeholder Engagement From Development to Dissemination in Genomic Medicine Research: Approaches and Outcomes from the CSER Consortium

Abstract

PURPOSE:

There is a critical need for genomic medicine research that reflects and benefits socioeconomically and ancestrally diverse populations. Disparities in research populations persist, however, highlighting that traditional study designs and materials may be insufficient or inaccessible to all groups. New approaches can be gained through collaborations with patient/community stakeholders. Although some benefits of stakeholder engagement are recognized, routine incorporation into design and implementation of genomics research has yet to be realized.

METHODS:

The NIH-funded Clinical Sequencing Evidence-Generating Research (CSER) consortium required stakeholder engagement as a dedicated project component. Each CSER project planned and carried out stakeholder engagement activities with differing goals and expected outcomes. Examples were curated from each project to highlight engagement strategies and outcomes throughout the research lifecycle from development through dissemination.

RESULTS:

Projects tailored strategies to individual study needs, logistical constraints, and other challenges. Lessons learned include starting early with engagement efforts across project stakeholder groups and planned flexibility to enable adaptations throughout the project lifecycle.

CONCLUSION:

Each CSER project utilized more than one approach to engage with relevant stakeholders, resulting in numerous adaptations and tremendous value-added throughout the full research lifecycle. Incorporation of community stakeholder insight improves the outcomes and relevance of genomic medicine research.

INTRODUCTION

Clinical genomics research has demonstrated benefits through identifying diagnoses and enabling genome-guided care ^1,2. Despite these successes, however, the preponderance of research participants are from majority populations ^3,4. Inadequate representation of people of color and individuals from lower socioeconomic status creates gaps in medical knowledge about effective approaches for screening, diagnosis and treatment in these populations. These gaps impede equity in research benefit from clinical genomics research and contribute to disparities.

Numerous factors contribute to lack of diversity in study populations, challenging researchers to examine and address potential barriers that may hinder access or willingness to participate ⁵. Potential logistical barriers may include limited means of transportation to research appointments, competing demands for time, and the affordability of missing work^6,7. Historically grounded mistrust of medical genetics and researchers ^8,9 compounded with personal experiences of inequities in health systems, and poor communication with healthcare providers can further erode trust and contribute to skepticism of medical research. Moreover, research teams frequently lack the diversity of the communities they hope to enroll.

The Clinical Sequencing Evidence-generating Research (CSER) consortium is addressing disparities in research participation via a concerted effort to recruit greater than 60% of participants from underrepresented and underserved populations defined by each site based on factors such as racial, ethnic, geographic or insurance status data ¹⁰. Thus, an additional directive at the programmatic funding-level for CSER was to engage with stakeholders to inform our diversity and inclusion efforts¹⁰. The clinical settings and aims for each of the CSER sites are described in Table 1.

Table 1:

CSER Project Descriptions. Each CSER site is unique with different aims, processes, enrollment locations and targeted enrollment populations.

Study Namea	Institution	Patient Enrollment Settingsb	Target Enrollment Populations (Sub-populations)b	Key Outcomes
CHARM	Kaiser Permanente Northwest	Outpatient primary care clinics from two health systems: Kaiser Permanente Northwest (Portland, Oregon), and Denver Health (a system of Federally Qualified Health Centers in Denver, Colorado)	Adults (18–49 years) at risk for hereditary cancer; (Racial/ethnic minority, low income, low health literacy, Medicaid/ Medicare or uninsured, Spanish speaking)	Assesses the utility of clinical exome sequencing and how it affects care in diverse populations of adults at risk for hereditary cancer syndromes.
ClinSeq	NIH / NHGRIc	National Institutes of Health Clinical Center (Bethesda, Maryland)	Adults, no specific phenotype; (African American, Afro-Caribbean, African)	Conducts genetic sequencing amongst healthy volunteers in order to study the impact of returning their individual genetic results and to build a resource for genotypedrive research.
KidsCanSeq	Baylor College of Medicine	Academic and non-academic medical centers, outpatient clinics in Texas: Texas Children’s Hospital, MD Anderson Cancer Center (Houston); University of Texas Health Science Center at San Antonio, Children’s Hospital of San Antonio (San Antonio); Cook Children’s (Fort Worth); Vannie Cook Children’s Clinic (McAllen)	Children with cancer and their parents; (Medically underserved, Hispanic/Latino, African American, Asian, Spanish speaking)	Studies the utility of genome-scale testing, compared with more targeted methods, in diverse pediatric cancer patient populations and diverse health care settings in Texas.
NCGENES2	University of North Carolina at Chapel Hill	Outpatient pediatric genetic and neurology clinics at academic medical centers; community hospital in North Carolina: University of North Carolina Chapel Hill (Chapel Hill), Mission Health (Asheville), East Carolina University (Greenville)	Children (< 16 years) and caregivers presenting as new patients with suspected genetic conditions (developmental disabilities, dysmorphology, neuromuscular disorders); (African American, Hispanic/Latino, Medicaid or uninsured)	Assesses the utility of clinical exome sequencing compared to standard of care testing in diverse pediatric populations presenting for initial genetic evaluation. Also assesses the impact of pre-clinic preparatory materials on measures of caregiver-provider engagement and care.
NYCKidsSeq	Icahn School of Medicine at Mount Sinai & Montefiore Medical Center	Academic medical centers, private practice in New York City, New York: The Mount Sinai Hospital, Mount Sinai Doctors Faculty Practice, Mount Sinai Kravis Children’s Hospital, Mount Sinai West Hospital (Manhattan); Montefiore Medical Center, The Children’s Hospital at Montefiore (Bronx)	Children (ages 0–21) with suspected neurologic, immunologic, and cardiac genetic conditions; (African American, Hispanic/Latino, Medicaid, Spanish-speaking)	Assesses the clinical and economic utility for use of genomic medicine for underserved children. Also assesses family understanding and satisfaction.
P3EGS	University of California, San Francisco	Academic medical center, outpatient clinics, neonatal intensive care unit in pediatric intensive care unit and community hospital in California: UCSF Benioff Children’s Hospital Mission Bay, UCSF Fetal Treatment Center, Zuckerberg San Francisco General Hospital, (San Francisco); UCSF Benioff Children’s Hospital Oakland (Oakland); Fresno Community Health Center (Fresno)	Infants and children with severe developmental disorders, with or without congenital anomalies (pediatric); parents whose fetus has a structural anomaly (prenatal); (Underserved by census tract, Medicaid, Asian, Hispanic/Latino, African American)	Assesses the utility of whole exome sequencing as a tool for diagnosing infants and children with serious developmental disorders. Also assesses providing genetic information to parents when a prenatal study reveals a fetus with a structural anomaly.
SouthSeq	HudsonAlpha Institute for Biotechnology	Academic medical center and community neonatal intensive care units, academic maternal fetal medicine outpatient clinics; Children’s Hospital of New Orleans (New Orleans, Louisiana); University of Alabama at Birmingham (Birmingham, Alabama; University of Louisville (Louisville, Kentucky; University of Mississippi Medical Center, Woman’s Hospital (Jackson, Mississippi)	Newborns with suspected genetic conditions; (African American, underserved, rural)	Performs whole-genome sequencing on newborns suspected to have genetic disorders. Assesses return of results mechanisms to expand access to genetic testing to diverse, especially historically underserved, communities.

^aAll projects have study materials (including consent forms, education materials, and surveys) available in both English and Spanish, some of which is publicly available at https://cser-consortium.org/cser-research-materials.

^bAdapted with permission from Amendola et al., 2018 ¹⁰ and Goddard et al., 2020²¹

^cClinSeq completed enrollment at the start of the extramural studies and thus did not assess stakeholder engagement-related variables as in other CSER projects.

Community Stakeholder Engagement

Stakeholders are defined as representatives from the group(s) responsible for, or affected by health and/or healthcare decisions informed by the research¹². In clinical genomic research this includes diverse patients, parents, research participants, health care providers, payers, policy makers, advocacy groups and community representatives ¹³. Herein, we will focus primarily on engagement with community and patient/caregiver stakeholders. Such stakeholders can serve as cultural brokers that advocate for the needs and concerns of their community and build bridges between their communities and researchers ¹⁴.

The engagement approaches and methods employed with community stakeholders are dependent on multiple factors. These include desired study outcomes, time commitments, relevance of issues, and experience of both the stakeholders and researchers seeking to engage with them. Successful engagement necessitates researchers be adaptable to change, willing to commit the time and resources required, begin engagement early in the research process (preferably during idea generation), and integrate feedback into research processes and outputs. This complexity and commitment may be daunting, but the benefits essential. To encourage integration of stakeholder engagement, we highlight engagement strategies applied across the CSER projects framed by purposeful consideration of the approach and insights gained.

ENGAGEMENT ACROSS THE RESEARCH LIFECYCLE

Key to engaged research is the application of strategies throughout the life of the research project. Figure 1 depicts the research lifecycle and potential opportunities for stakeholder engagement throughout the stages.

Figure 1.

Stakeholder engagement throughout the research lifecycle enables continued value and insight.

Stage 1: Pre-Award / Planning

During the pre-award period study questions are defined and hypotheses formulated into specific aims and outcomes to propose for funding. At this early stage, having complete shared decision making with stakeholders would align with the concepts of Community-Based Participatory Research ¹⁵. That may be challenging in highly specialized fields, and when there is limited time to respond to funding announcements. These challenges can be addressed through cultivating long-term relationships with stakeholders (e.g., patients, advocates, and clinicians) such that readiness and capacity are already present in a standing advisory board. This affords ample opportunity to co-develop research questions, aims and strategies with groups who are likely to be impacted by the research outcomes.

Stage 2: Post-Award/Pre-Enrollment

Once a proposal has received funding (or other initiation), preparation to conduct the study moves forward. Prior to enrollment, the study protocols, recruitment materials, consent processes, and educational materials will need to be finalized. This period offers robust opportunities to engage with stakeholders through a variety of mechanisms as are described in the examples below.

Stage 3: Ongoing Enrollment

During this phase, the study is actively enrolling participants who are progressing through study processes. As challenges inevitably arise, there are unique opportunities to gain input from stakeholders to address emergent issues that could result in modifications to study protocols and materials.

Stage 4: Data Analysis

Data analysis may be the most underutilized phase to incorporate stakeholder engagement in a genomic medicine research study. Some aspects may require specialized training and would not be practical to expect community stakeholders to perform. However, stakeholders may seek different questions from our data that can guide correlative analysis or examination for trends. In addition, stakeholder input into analysis can guide the framing of findings and development of key messages for future dissemination.

Stage 5: Dissemination

Dissemination efforts may have diverse audiences including the scientific community, clinicians, policy makers, funders, and organizational leaders as well as research participants, and broader communities who may be impacted by current and future research efforts. Community stakeholder insight is essential to identify and prioritize key findings, guide lay language descriptions, and present opportunities to reach various community groups with broad and/or tailored messages. Their guidance can also shape professional community dissemination including contributing to and co-authoring manuscripts ^16–18 and co-presenting at professional conferences.

METHODS

Engagement methods are distinct from qualitative research as they are purposely not intended for the generation of new data. Activities typically include trust building with a target community, gaining insight into concerns relevant to the intended study population, development of culturally responsive research questions, materials, and protocols ^19,20 and troubleshooting research issues to identify alternate approaches that may be more responsive to the targeted participant population. Each CSER site is unique with different aims, processes, and targeted enrollment populations as described in Table 1 ^10,21. Therefore, each project planned and carried out different stakeholder engagement activities with differing goals and expected outcomes (Figure 2). Approaches were selected from each of the active enrollment sites to demonstrate a variety of methods. Case examples describe how and why an approach was used and highlights some outcomes.

Figure 2.

The frequency of stakeholder engagement approaches used by CSER project groups across the research life cycle.

†Frequency refers to how many of the 6 active enrollment CSER sites indicated that they used that approach for activities in the particular stage.

‡The Consortium-wide Stakeholder Committee provided broad insight across the consortium and is therefore represented across each of the stages after award. In additional, they were used by KidsCanSeq for project-specific work during Ongoing Enrollment. CSER, Clinical Sequencing Evidence-Generating Research.

RESULTS

COMMUNITY ENGAGEMENT STRATEGIES

1. Advisory Boards/Action Committees ideally forge a partnership between researchers and various individuals who may have different roles in their community ²². Consisting of community stakeholders, such as potential or previous research participants, advocates for patient groups and health conditions, social service professionals, and other types of community leaders (e.g., faith leaders, attorneys, public service officials, community developers, and educators), they can provide valuable insights, iterative guidance, and recommendations as the research advances.

   NYCKidsSeq utilized a standing Genomics Stakeholder Board to enable partnership during the pre-award / planning phase. The standing board is focused on implementing translational genomics in diverse NYC populations and includes patients, advocates, and clinicians, as well as researchers, funders, and entrepreneurs ²³. NYCKidSeq research leadership met with the standing board beginning in the planning phase to develop the formative research questions and guide recruitment of new members specific to the goals of the later funded research project.

   Once awarded, NYCKidsSeq developed a project-specific action group including standing board members and newly recruited stakeholders to address the specific goals of the now funded project. In particular, this board helped develop qualitative interviews and quantitative surveys for parents and clinicians, as well as GUIA, a low literacy, Spanish-English tool for genetic counselors to facilitate results disclosure ²⁴.

   NYCKIDSeq provided regular updates to their board about enrollment and retention by site and by recruiter. This enabled continual insight to help investigators address challenges and remain accountable regarding enrollment of various population groups. Study alterations included adding evening and weekend hours for study contact and conducting mock sessions to address reasons potential participants were hesitant to enroll. The on-going relationship was also instrumental to the team when rapidly and effectively pivoting to remote engagement due to COVID-19 clinical shutdowns.

   CHARM had two geographically distinct enrollment locations with different populations (Denver, CO, and Portland, OR). They developed a Patient Advisory Committee (PAC) at each location during the post-award/pre-enrollment period to reflect the local population. PAC members included: community members with low literacy skills, English and Spanish speakers, recruitment site staff, and study site staff engaged at the beginning of study. Each PAC team held an initial in-person group meeting. For English-speaking members, subsequent meetings were also in person. Conflicts in schedules for Spanish-language PAC members prohibited group meetings; thus, all subsequent contacts with this group were conducted through targeted feedback via telephone. PAC feedback influenced the development of recruitment and other participant-facing materials as well as study processes such as consent and return of results.

   NCGENES recruited and engaged a community consult team (CCT) early in the post-award period to guide numerous elements of the project including the development of a clinic visit guide and question prompt list intended to impact caregiver-clinician engagement. Study participant families were randomized to receive/not these materials and clinic visit transcripts from both arms were coded by research team members to explore question asking as one factor of engagement. NCGENES coding team partnered with the CCT to analyze the coded data and explore the potential impact of the materials the CCT had developed. Their perspective provided insight into the complex nuance of information seeking, giving, and receiving that can occur during a specialist visit. CCT feedback altered analysis and has been incorporated into discussions and message framing for dissemination.

2. Consortium-level Community/Patient Stakeholder Groups provide feedback and insight into the planning and implementation of research across a consortium. The CSER-wide community and patient stakeholder group consisted of 1–2 representatives recruited by each CSER project to represent their stakeholders. This group provided early perspective to the CSER consortium to inform research planning and partnership with participants.

   Key points included:

   • Keep information clear and simple. Do not overburden patients/caregivers who may already be overwhelmed
   • The distinction between research and clinical care may not be clear to participants
   • Participating in research requires building relationships and development of / maintaining mutual trust
   • Patients/caregivers want to work with researchers. Treat them as partners, not menu items.
   • Capacity building is not just for the community/patient stakeholders. Researchers also need to gain skills to appropriately engage.

   The CSER-wide group provided a means for sites and consortium workgroups to engage with motivated, informed stakeholders, through web-based conferencing. It also spread the potential challenge of recruiting and maintaining a group across multiple sites. This approach relied on a CSER project liaison to connect their stakeholder representative to potential opportunities.

   KidsCanSeq initially planned to utilize a standing Advisory Committee for development and refinement of study materials, however, situational circumstances impeded that plan. The study team utilized the CSER-wide stakeholder advisory group for feedback about a specific process, that to return variant of uncertain significant (VUS) results via letter. Two virtual feedback groups (one in English and one in Spanish) were held to review types of result uncertainty and approaches to aid families’ understanding of VUS. The summary points included: preparing families for possible uncertainty; outlining what is known; and providing clear next steps. Providing resources and take-aways was encouraged. Using these points, an education card was developed to include with results packets for participants who have VUS identified.

   A CSER consortium workgroup project aimed to describe a series of results discussion sessions collected from the on-going CSER projects. Data analysis focused on themes regarding genetic counseling challenges mapped back to practice-based competencies. The work group project team engaged the CSER-wide stakeholder group through two virtual feedback discussions. Exemplar cases were discussed, and possible counselor strategies were elicited and then paired with genetic counselor practiced-based competencies. Stakeholder input highlighted practical strategies the team had not considered such as planning additional appointments to break-up discussion of complex topics and management ²⁵.

3. Feedback Groups are small groups of stakeholders who represent specific experience or demographic characteristics for targeted feedback regarding a research project idea, recruitment plans or challenges, data collection methods, relevance and format of an intervention, and feasibility of proposed research workflows. Guided by a facilitator, and often using predetermined questions based on project-specific agendas, researchers can hold multiple feedback groups focused on the same agenda and/or at multiple points throughout the study. Community Engagement Studios (CES) are a type of structured feedback group that typically utilizes an independent moderator not part of the research team and may be conducted in a different location with participants selected to have characteristics similar to the project enrollment needs ^26,27. In both forms, research team members may choose to be present and listen to the discussion.

   NCGENES utilized feedback groups prior to enrollment to gain insight into ways the research team could encourage enrollment by supporting future participant families on long clinical days. Caregiver participants from a prior genomics study were recruited to represent the target sociodemographic groups for the current study. These caregivers who had experience navigating specialty care with a complex needs child and prior research participation shared numerous practical suggestions. These included: valet parking, substantial snacks, tablets with loaded content to entertain children, and participant ‘thank you’ gifts for the child participant (e.g. hats, t-shirts and balls). Another outcome was insight into potential challenges that could arise for future participant families including sickness, disruption in childcare, and transportation difficulties that could result in late or missed appointments. This emphasized the need for enrollment teams to have flexibility in scheduling research visits.

   SouthSeq convened two community engagement studios (CES) prior to enrollment. Parents of children who had previously been admitted to the NICU at Children’s Hospital of Alabama were recruited through community outreach that included social media posting, flyers, and phone calls. Using CES, the SouthSeq investigators received feedback on the Genome Gateway survey platform, the format and content of the genomic testing result letters, and the educational materials for parents, aimed at improving knowledge about genomic testing. The CES were conducted by trained moderators from Vanderbilt university, neutral to the project team. Based on the feedback obtained from parents the research team changed the format of the genomic testing results template and revised educational materials.

4. Targeted / One-on-one Feedback seeks out individuals with specific knowledge, experience or insights about a research topic,²⁸ who may be geographically distant or have other barriers to regular participation in groups^29,30. As such, it may be particularly useful to engage under-represented populations.

   P³EGS faced several challenges recruiting and convening an advisory board that represented their study population with a high proportion of low-income and non-English speaking participants. Barriers to attending scheduled groups meetings included a lack of childcare support, inadequate transportation, and irregular work schedules. To better accommodate participating families and community stakeholders, the team pivoted to solicit one-on-one input on an as-needed basis. Feedback sessions yielded valuable information including the need for and best way to provide incentives for completing study surveys as a way of respecting families’ time.

   KidsCanSeq utilized targeted feedback as a straight-forward means to gain perspectives about their study process for return of results via mail for participants with no significant findings. The first 15 participants to receive results in this manner were contacted by phone and engaged for targeted feedback. Feedback led to a second effort to explore materials to help participants understand uncertainty stemming from genetics evaluation.

5. Deliberative Engagement/Democracy encompasses a range of approaches to seek public perspectives for the development of governing policies such as those that guide health access or biomedical research ³¹. Typically organized as a facilitated group or series of groups, this approach can engage diverse community stakeholders in deliberation with policy or situational experts with the aim of developing an informed consensus opinion, or set of recommendations to guide policy or practice development ^32,33.

   SouthSeq has planning in progress to convene a multi-stakeholder group including parents who with their newborns participated in the research, members of the SouthSeq Community Advisory Board, clinicians, and health system executives from the five medical centers participating in the project, and the investigators. The meeting will take place at the end of the project when study results are available for deliberation. A primary goal of the group is to produce informed stakeholder opinions, commentary, and feedback on the findings from SouthSeq to include in publications and other dissemination products. A secondary goal is to generate recommendations on the implementation of routine sequencing in Neonatal Intensive Care Units.

6. Human-Centered Design is a process that engages intended downstream users or beneficiaries in the design of a specific product or service ³⁴. It emphasizes multidisciplinary collaboration, creative brainstorming, rapid prototyping and testing of solutions, with the people who are most knowledgeable about a practice, service or device that is in need of change ³⁵.

   P³EGS received supplemental funding to support the development of an innovative, human-centered design strategy for communicating aggregate study results to participants. Human-centered design is an approach to developing solutions that directly incorporates the values and priorities of intended beneficiaries ³⁶. The P³EGS supplement project is in progress and involves a multi-stage collaboration to a) identify qualitative and quantitative study results that are of interest to participants, b) develop simple, visually appealing materials in Spanish and English, and c) disseminate results and assess their acceptability and relevance. The broader goals of the project are to increase public trust in scientific research and ensure that research participants share in the benefits of research by receiving information that is responsive to their needs and preferences.

DISCUSSION

For the benefits of genomic medicine to be accessible for all people, it is critical that clinical genetics research and outcomes be reflective of our diverse populations. While the need for diversity is well supported, how to achieve that goal is intricately complex and potentially daunting for researchers. Patient and community stakeholders are a source of novel strategies and perspectives that are invaluable to enable the research shifts that are needed. Routine integration of stakeholder feedback throughout a research project is not the norm, however. Often the term engagement is narrowly used to describe recruitment and retention efforts. While research participants are certainly stakeholders in the research, we have purposely chosen to emphasize the added value of patient/community stakeholder engagement across the full research project. This means defining research questions, protocol planning, enrollment and troubleshooting, data analysis, and dissemination of key findings. This manuscript aims to provide working examples to encourage and empower researchers to incorporate stakeholder engagement into their research. Our intent is to demystify stakeholder engagement and shed light on potential obstacles, alternative approaches, and lessons learned. Brief comparison of the approaches highlighted are presented in Table 2.

Table 2:

Considerations for Stakeholder Engagement Strategies. A comparison of the stakeholder engagement strategies utilized by the CSER project groups highlighting some benefits and challenges for each approach.

Approach	Benefits	Challenges
Advisory Committees	•  Partnership between researchers and community •  Long term relationship throughout the research project facilitates broader ideas and applications	•  May not be representative of target population •  Requires significant time commitment from research team and stakeholders over course of the full project or as standing committee •  Requires transparency and shared decisionmaking
Targeted One-on-One Feedback	•  Time and method can be tailored for convenience of the stakeholder •  Reduces constraints of transportation and childcare •  Increased privacy of discussion may enable deeper sharing •  Recruitment on an individual level can refine representativeness	•  No group dynamic •  Limited number of stakeholders •  Feedback may be more individual compared to community/broader level •  Feedback may not be representative of target
Feedback / Pilot Groups	•  Group dynamics can introduce new and converging ideas •  Can enables perspective from larger numbers of stakeholders via separate groups •  Allows further exploration of responses through repeat meetings	•  Discussion can be dominated by stronger personalities •  Need to coordinate space/time to meet needs of group (e.g., evenings, weekends) •  Need to consider other supports to enable participation (e.g., travel, food, childcare)
Community Engagement Studios	•  Facilitated by a neutral moderator •  Enables teams to tap into readily available expertise and processes	•  Representative populations recruited by moderator may be different than target study population •  Consulting fees for facilitators may be higher than internal teams
Consortium Stakeholder Group	•  Representation across the consortium projects •  Enables broader discussion on shared themes •  Increases diversity and/or generalizability of perspectives •  Interaction with/influence on consortium leadership and funders	•  Scheduling/communications and budgeting logistics across a broad geographic area •  Issues and/or feedback may not be applicable to a specific project •  Consortium level issues may not be as relevant to the local project stakeholder •  Consortium-level issues may not be as relevant to the local project stakeholder leading to less feedback
Deliberative Democracy	•  Includes community voice from the population studied •  Includes relevant subject matter experts on the target issue •  Aims to propose policy or opinion about a defined issue through consensus building •  Resulting policy/proposal reflects the informed opinion of the stakeholders	•  Significant time commitment from research team and stakeholders over a defined period of time •  May involve large numbers of stakeholders representing multiple perspectives •  Can require a team of facilitators to coordinate •  Often includes significant capacity building through informational presentations to enable informed deliberation
Human-Centered Design	•  Iterative process that involves direct collaboration with stakeholders •  Participatory design results in solutions that are more relevant, meaningful, and useful to stakeholders	•  Significant time commitment from research team and stakeholders •  Stakeholders may not be representative of overall participant population

Identifying Stakeholder Groups

Research projects have numerous stakeholders. Although we have focused on community/patient stakeholder groups, it is important to recognize the existence and importance of engaging a broad range of groups who are involved in or impacted by clinical research. These include research team members, research participants (and caregivers), institutional review boards (IRB), health care administrators, organizational leaders, payers and funders. For example, some CSER sites faced barriers to implementation of their research due to difficulty obtaining IRB approval--a challenge that has been documented previously ³⁷ and likely exacerbated by the increasingly complex nature of genomic testing and research. CHARM researchers analyzed how interactions with IRB stakeholders shaped their project ¹⁷ and NYCKidSeq stakeholder members, engaged their IRB to revise “standard” consent language to enhance understanding and clarity. In addition, there may be a need to identify individuals and groups who can represent community/patient stakeholder perspectives that may be difficult to obtain directly. For example, some populations may be too emotionally vulnerable to be involved beyond participation as a research subject. This was faced by the research teams at P³EGS, KidsCanSeq and SouthSeq where their target stakeholder group were caregivers of often very sick infants and children with potentially life-limiting conditions. Efforts to engage with vulnerable individuals requires sensitivity and consideration of the appropriateness of recruiting emotionally vulnerable stakeholders to participate in an advisory capacity. Engagement in an advisory capacity may be a positive experience for some participants and overwhelming for others, so researchers may need to consider alternative groups who can represent these key stakeholders. This can include caregivers who have previously experienced the situation or advocates for these families such as clinic social workers, and community support leaders.

Learning to be flexible

When attempting to engage with stakeholders, there is a need for flexibility. Community stakeholders are often busy, working adults who live in various locales. Some CSER projects amended plans to enable evening or weekend meetings or provide an option to call into meetings. Some sites found that rotating the location to different areas ensured the travel burden was reduced and equally shared. Sites also worked to build relationships in virtual group meetings held through web-based meeting platforms.

The importance of flexibility in approaching stakeholder engagement emerged as a key theme among all of the consortium sites. Research teams often tailored engagement strategies to meet study needs and facilitate stakeholder participation (Figure 2). All CSER sites also endeavored to engage stakeholders from underserved and underrepresented populations. To successfully engage with stakeholders, research teams cannot utilize a one-size-fits-all approach and may need to adjust planned methods. For example, the P³EGS study initially planned to convene a traditional community advisory board but found that convening such a group in person presented challenges for families with unpredictable work hours and caring for a child with special needs. Instead the P³EGS team pivoted to utilize one-on-one discussions with stakeholders, which also facilitated inclusion of participants who did not speak English ³⁸.

The concept of flexibility also applies to the broader research timeline. Research timelines are critical. Competing research priorities and the typical hiccups common to clinical research can diminish the resources available and value placed on stakeholder engagement activities. Emphasis in translational genomics research can be hyper-focused on enrollment, retention/attrition, and numbers of laboratory tests completed. The additional time and budget required to establish robust stakeholder relationships can be seen as overly burdensome especially if it means delaying enrollment or other key metrics required in quarterly reports. To be successful, research team leadership needs to value stakeholders as integral to the research timeline, recognize how stakeholders can help meet the deliverables (i.e., recruitment) and plan for the appropriate time and flexibility to ensure stakeholder insight throughout the project.

More than just talk- Integrate

Stakeholder engagement is not a one-way street in which researchers explain what they are doing to their advisory group. Researchers must be willing to amend protocols, reconsider data questions and actively partner on message framing and dissemination. CSER projects incorporated stakeholder feedback into protocols and materials. Sites that engaged advisory boards continued to incorporate that feedback throughout their studies. Actually incorporating stakeholder feedback and sharing changes made with the broader research community demonstrate authenticity and show stakeholders their input is critical and meaningful.

Conclusion

Each CSER project utilized more than one approach to engage with relevant stakeholders, resulting in numerous adaptations and tremendous value-added throughout the full research lifecycle. Early and continual engagement offers the opportunity for research questions, aims and strategies to be influenced by the groups who are likely to be impacted by the research outcomes. Mid-project engagement affords the integration of stakeholder insight into study logistics such as participant-facing materials (e.g., consents, recruitment, etc), enrollment protocols, and troubleshooting for protocol or enrollment-based challenges that inevitably arise. Notably, each project, successfully met or exceeded their goals to enroll > 60% research participants from underrepresented populations. Continued engagement throughout data analysis and dissemination enables stakeholder perspectives to influence analysis of findings, highlight key messages, and inform dissemination to diverse audiences who may be impacted by the research.

In conclusion, the incorporation of community stakeholder insight throughout research projects can improve medical genetic research and outcomes. For our research to benefit all populations, it is imperative that stakeholder engagement efforts be recognized, valued, and supported as integral, not supplementary, to medical genetics research.