Increasing the genomic workforce through research capacity building: Designing evaluation plans for maximum impact

Summary

More interventions are needed to address the need for workforce diversity and research capacity building (RCB) in genomics. In 2023, the National Human Genome Research Institute and the National Institute on Minority Health and Health Disparities of the National Institutes of Health funded the Diversity Centers for Genome Research Consortium to address this critical gap. The NIH program staff designed a prospective evaluation plan and developed common data elements (CDEs) that capture common evaluation outputs, synergize and streamline reporting, and facilitate continuous quality improvement. We created five CDEs: genomics programs and equipment, scientific productivity, scientific collaboration, community engagement, and workforce development. The prospective development of an evaluation plan based on CDEs facilitates the ongoing evaluation, reporting, and adjustment of RCB interventions to enhance the diversity of the genomics workforce.

Prospective evaluation plans for research efforts are required for ongoing evaluation, quality improvement, and adjustment of implementation strategies. By sharing our methods for creating an a priori evaluation plan, we provide a framework for others to consider as they work to increase diversity and build research capacity in the genomics workforce.

Introduction

Research capacity building (RCB) is the process of individual and institutional development that strengthens skill levels and abilities to develop, implement, and sustain high-quality research efforts that can be translated into practice and improve outcomes¹ (https://guides.hsl.virginia.edu/VRL/researchcapacity). Few programs address the need for workforce diversity and RCB in genomics. In September 2023, the National Human Genome Research Institute (NHGRI) and the National Institute on Minority Health and Health Disparities (NIMHD) of the National Institutes of Health (NIH) funded the Diversity Centers for Genome Research (DCGR) Consortium to address this critical gap.² Three centers were initially funded: the Diversity Center for Genome Research at Meharry Medical College, the Pacific Center for Genome Research at the University of Hawai’i at Mānoa, and the Diversity Center for Genome Research at the University of Texas Rio Grande Valley (UTRGV). Each center comprises an administrative core, a workforce development core, a community engagement core, and two to three innovative genomics research projects.

Increasing diversity in genomics through RCB

The DCGR Consortium (from now on called the Consortium) addresses several areas of NHGRI’s strategic vision for improving health at The Forefront of Genomics³ and the four goals of NHGRI’s Action Agenda for Building a Diverse Genomics Workforce: (1) developing and supporting initiatives to provide early exposure to careers in genomics, (2) creating training programs for undergraduate and graduate education in genomics, (3) fostering career development and research transition programs, and (4) evaluating progress toward greater diversity in the genomics workforce.⁴ The specific goals of the Consortium are (1) to strive for diversity in all aspects of genomics research through a commitment to systematic inclusion of diverse and underrepresented communities, (2) to embrace interdisciplinary team-oriented genomics research, (3) to maximize the utility of genomics for all members of the public, and (4) to champion a diverse genomics workforce. The Consortium aims to train genomic researchers at all career levels and support the development of innovative genomic research projects through infrastructure building and the formation of interdisciplinary research teams at United States (US)-based, resource-limited institutions.

The call to strengthen RCB through funding mechanisms to increase institutional and community partnerships has encouraged a steady growth in RCB consortia and programs.^5,6 These consortia typically involve multidisciplinary collaborations of academic and community investigators engaged in research, training, and professional development activities aligned with the goals of the institutions and the funder.^7,8,9 However, more data are needed on the implementation, monitoring, and evaluation of RCB consortia and their effectiveness in achieving the associated learning goals and best practices.^5,9,10,11 Furthermore, a review of the literature on RCB focused on genomic research yielded only one paper focused on equitable partnerships and capacity building.¹²

Evaluation of RCB efforts

Evaluating consortia’s outcomes should include understanding the processes and factors that drive them, particularly for complex RCB programs involving multiple partners and approaches.^5,13,14 However, several challenges exist related to measuring the impact of RCB initiatives, such as consortia, and more evidence is needed to inform RCB design, implementation, evaluation, and standardized metrics that measure outcomes and impact.^11,15,16 Three NIH-funded research consortia have implemented retrospective efforts to standardize evaluation across centers several years after the NIH established these consortia. The Research Centers for Minority Institutions (RCMI), the Clinical and Translational Science Institutes (CTSI), and the Comprehensive Partnerships to Enhance Cancer Health Equity (CPACHE) each used a variety of methods and frameworks to guide their evaluation plans and the development of standardized evaluation methods and instruments.^7,17,18

Lessons learned from these long-established consortia support the need to implement evaluation plans of RCB initiatives at the initiation of consortia. Consortia evaluation plans should consider the complexities of a given consortium and align with funder requirements. RCB initiatives should strategically identify desired outcomes through a priori needs assessment informed by program, participant, and community goals. This approach will allow meaningful data collection to inform monitoring and evaluation.^16,19 Efficient evaluation of consortia requires support platforms for cross-institution evaluation and resource sharing, which include quantitative and qualitative data collection.¹¹ The latter is critical and will yield information that will deepen and illuminate an understanding of RCB implementation and its benefits. Investigators or evaluators should construct short- and long-term program evaluation plans for self-initiated monitoring and reporting to assess implementation efforts and allow for continuous quality improvement.^11,20

This manuscript describes the methods used to create a prospective evaluation plan and develop common data elements (CDEs) in a diverse genomic RCB consortium in the US. The CDEs will enable the Consortium to track outputs, ensure consistent evaluation reporting, facilitate continuous quality improvement, and adjust strategies to meet consortium goals. We describe our process and methods using terminology relevant to a US context. This report may provide direction for other RCB consortia in developing evaluation plans.

Methods

The Consortium aims to train genomic researchers (defined and referred to as “trainees” going forward) at all career levels through “mentors” (defined as any person at any level who supports and instructs trainees) and support the development of innovative genomic research projects through infrastructure building and the formation of interdisciplinary research teams at US-based, resource-limited institutions. NIH program staff designed a prospective evaluation plan to enable the reporting of Consortium-wide outputs and outcomes based on the development of CDEs. Developing the CDEs began with a review of the individual milestones required by each funded center. Each center develops its programs and runs them independently. Therefore, center milestones are unique to each center. Center milestones reflect the NIH program staff’s review, revision, and approval of center milestones before funding is released to each center.

The initial analysis of center milestones and subsequent creation of CDEs occurred over 6 months after the Consortium was established. We applied content analysis methods²¹ and the translational science benefits model, a framework designed to link the impact of scientific activities and outcomes to downstream health programs and policies in developing the CDEs.²² The content analysis was iterative, led by the lead author (K.J.R.), and employed checkpoints throughout the analysis to solicit input from NHGRI program staff (W.O. and E.B.M.). This approach ensured the identified content areas represented all the centers' milestones. The team identified common content areas from each center’s milestones to generate cross-consortium metrics and inform the creation of the CDEs. The NHGRI team and Consortium evaluation working group members reviewed the final CDEs to ensure they included all data from each center’s milestones. Figure 1 depicts the CDE development process.

Figure 1.

Process map for CDE creation

NIH needs monthly reporting of centers' milestones to assess each center’s progress. In consultation with the NHGRI information technology branch staff, we developed an efficient and automated process to ensure that monthly reporting would be manageable for center grantees. Based on this consultation, the team chose Qualtrics software (https://www.qualtrics.com) to collect milestones and monitor progress from each center through monthly surveys. Monthly surveys will be completed electronically by each center and housed in an internal Consortium database (Consortium members portal) managed by NHGRI.

NIH program staff will collate these monthly milestones to report individual center and Consortium CDEs. This process will allow NHGRI to gather data associated with each center’s milestones, run timely reports, compare progress and outcomes between centers, and generate and report CDEs. CDEs will be used for monthly and yearly tracking during the Consortium’s 5- to 7-year funding cycle and long-term program evaluation. In addition to serving as a central location for reporting milestones by each center, the Consortium members portal enables communication between NHGRI program staff and centers, allows uploading of published manuscripts and presentations, and facilitates resource sharing between the centers.

Results

The final CDEs identified are genomic programs and equipment, scientific productivity, scientific collaboration, community engagement, and workforce development. Final CDEs with definitions and individual metrics can be found in Table 1.

Table 1.

Common data elements

Common data element and definition	Individual metrics
Genomics programs and equipment
Genomic educational activities such as seminars, courses, curriculums; genomic lab space and equipment	# of courses developed and conducted # of curriculums developed and conducted # of seminars developed and conducted # of new lab spaces # of new genomics equipment
Scientific productivity
Genomic-focused research and dissemination, including publications, presentations, and grant applications	# of grant applications that leverage center resources # of grant applications that leverage Consortium resources # IRB applications # of manuscripts submitted for each site # of manuscripts accepted for publication for each site # of manuscripts published for each site # of abstracts submitted for each site # of oral presentations completed for each site # of poster presentations completed for each site # of cross-consortium manuscripts submitted # of cross-consortium manuscripts accepted for publication # of cross-consortium manuscripts published # of cross-consortium abstracts submitted # of cross-consortium presentations completed # of cross-consortium poster presentations completed
Scientific collaboration
Genomic research project collaborations and partnerships (internal and external)	# of partnerships or collaborations (internal) # of partnerships or collaborations (external) # of meetings with internal collaborators # of meetings with external collaborators (Consortium meetings, steering committee, working groups, and meetings with those outside of the Consortium, including community partners) # of internal advisory board members # of institutions represented on internal advisory board # of meeting internal advisory board meetings # of external advisory board members # of institutions represented on external advisory board # of meeting external advisory board meetings
Community engagement
Community members, CAB members, healthcare partners, and other community entities/institutions and activities involving community members	# of CAB meetings # of CAB members # of institutions/community entities represented # of research projects, which include community members # of educational programs that include community members
Workforce development
Genomic research training activities, including research methods and analyses trainees complete, trainee/mentor numbers, and demographics	# basic/science/experimental/# and level of trainees involved # bench to beside and # and level of trainees involved # implementation studies and # and level of trainees involved # translational studies and # and level of trainees involved # animal models used and # and level of trainees involved # human models used and # and level of trainees involved # other methods/models and # and level of trainees involved types of research methods types of research analyses type of equipment trained on # of trainees per educational level # of trainees by self-identified race/ethnicity # of trainees by self-identified gender # of trainees by self-identified disability # of trainees by disadvantaged background # of mentors per educational level # of mentors by self-identified race/ethnicity # of mentors by self-identified gender # of mentors by self-identified disability # of mentors from a disadvantaged background # of graduate students completed training # of postdoctoral students completed training # of graduate/postdoctoral students hired for an academic position # of graduate/postdoctoral students hired for an industry position
Trainee/mentor demographics
Educational level (K-12, baccalaureate, graduate, PhD, postdoctoral, faculty)
Race/ethnicitya
Genderb
Disability statusc
Disadvantaged background statusd

All demographic data are self-reported. Internal, at a center’s home institution; external, across the Consortium and outside of the Consortium, including community partners; CAB, community advisory board.

^aRace and ethnicity: American Indian or Alaska Native, Asian, Black or African American, White, and Hispanic or Latino.

^bGender: male, female, non-binary/third gender, transgender, other, and prefer not to answer.

^cDisability: a physical or mental impairment that substantially limits one or more major life activities, as described in the Americans with Disabilities Act of 1990, as amended.

^dDisadvantaged background: an individual is considered to be from a disadvantaged background if he or she meets two or more of the following criteria: (1) were or currently are homeless, as defined by the McKinney-Vento Homeless Assistance Act (definition: https://nche.ed.gov/mckinney-vento/); (2) were or currently are in the foster care system, as determined by the Administration for Children and Families (definition: https://www.acf.hhs.gov/cb/focus-areas/foster-care); (3) were eligible for the Federal Free and Reduced Lunch Program for 2 or more years (definition: https://www.fns.usda.gov/school-meals/income-eligibility-guidelines); (4) have/had no parents or legal guardians who completed a bachelor’s degree (see https://nces.ed.gov/pubs2018/2018009.pdf); (5) were or currently are eligible for Federal Pell grants (definition: https://www2.ed.gov/programs/fpg/eligibility.html); (6) received support from the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) as a parent or child (definition: https://www.fns.usda.gov/wic/wic-eligibilityrequirements); and/or (7) grew up in one of the following areas: (a) a US rural area, as designated by the Health Resources and Services Administration (HRSA) Rural Health Grants Eligibility Analyzer (https://data.hrsa.gov/tools/rural-health), or (b) a Centers for Medicare and Medicaid Services designated Low-Income and Health Professional Shortage Areas (qualifying zip codes are included in the file). Only one of the two possibilities in #7 can be used as a criterion for defining a disadvantaged background.

Genomics programs and equipment

We defined the genomics programs and equipment CDE as genomic educational and research activities, genomic equipment, and laboratory space. Educational activities may include formal academic modules, workshops, courses, and curricula or may focus on community education. Center investigators may target community genomic education to school children (K-12), healthcare partners, or other vital partners identified by the community engagement core at each center. Centers have proposed various innovative ways to engage students in primary school education down to the lowest level (kindergarten or age 5 in the US). Activities and educational programs are designed with developmental and academic levels in mind and may be delivered in the school or community setting. Similar strategies and innovations for genomic education and programs have been proposed for healthcare workers, partners, and community-based settings. Each center and its community partners will jointly lead the strategic implementation of these programs.

Scientific productivity

The scientific productivity CDE includes publications, presentations, and grant applications related to genomics that result directly from or leverage individual center or Consortium resources. Presentations may consist of traditional scientific conference platforms and poster presentations or informational sessions for the community. Centers will report grant activity for all funders, including local philanthropic funding, research foundations, and federal grants. Centers will report authorship of peer-reviewed articles for types of authors (e.g., scientists, community members, post-docs, graduate students, and undergraduate students). Centers will be encouraged to include community members and trainees in the research process so that they merit authorship on the resulting papers. This approach consists of all key interested parties. It builds into the sustainability and success of each center and the DCGR through objective measures of scientific productivity to the center’s institutions, the NIH, other funders, and community partners.

Scientific collaboration

The scientific collaboration CDE includes new genomic research project collaborations and partnerships within each center, across the Consortium, and with others outside the Consortium developed because of resources associated with the program. Each center must have an internal and an external advisory board to broaden areas of scientific collaboration. The timely updating and reporting of research projects on the Consortium members portal will facilitate collaborations across and outside the Consortium that may not have occurred otherwise. Having these data accessible to investigators reduces barriers to finding collaborators whose research aligns or complements center initiatives and builds capacity through leveraging Consortium-wide data, resources, and expertise.

Community engagement

The community engagement CDE will capture and quantify the number of community members involved in research projects (as participants and research team members) and educational programs. We defined community members as those on the community advisory board (CAB), healthcare providers, and other local entities. Centers will report the number of CAB members and meetings. This CDE is essential to ensure that center and Consortium-wide projects are aligned with community-identified needs for genomic education, research, and capacity building. Including community engagement in the Consortium structure is an exemplar for other research contexts to consider, which will enhance sustainability.

Workforce development

The workforce development CDE will collect and report information regarding the types of genomic research training activities and experiences, including research methods and analyses. Additionally, we will capture trainee/mentor demographics, including educational level (K-12, baccalaureate, graduate, doctoral, postdoctoral, and faculty), self-reported race and ethnicity, gender, disability status, and if the trainee/mentor is from a disadvantaged background. The reporting of this CDE will allow the sharing of genomic educational training, research design, methods, and analyses being conducted at each center with the Consortium. The availability of these resources to Consortium members will reduce barriers and facilitate RCB in the genomic workforce.

Discussion

This manuscript describes the methods for creating a prospective evaluation plan and developing CDEs for a diverse genomics RCB consortium. The prospective development of CDEs provides the Consortium with a mechanism to capture outputs, ensure accurate and consistent reporting across centers, facilitate continuous quality improvement, and adjust strategies to meet Consortium goals. We hope to provide a framework and strategic approach that future RCB consortia can use to develop their evaluation plans and metrics to increase research capacity and workforce diversity.

Though RCB consortia have been previously implemented,^23,24 the strategic focus on increasing the diversity of the genomic workforce through training opportunities in state-of-the-art genomic research in the context of established administrative, workforce development, and community engagement cores is innovative. The development of each center’s milestones and CDEs allows NIH program staff to track the progress of individual centers and the overall Consortium and identify areas for improvement concurrently. Training students from different academic levels in innovative genomic research and technologies is of high value and timely, as there is a need to ensure equitable and diverse perspectives in genomic science. Rigorous and well-planned evaluation plans are crucial to demonstrate the value of these efforts for long-term success, reproducibility, and sustainability. Our prospective evaluation plan is intended to be ongoing and iterative. It will allow trainees to provide feedback and insights into factors that may enhance their academic and professional development, thereby improving training strategies and outcomes. The systematic and prospective creation of CDEs, grounded in a rigorous theoretical framework for evaluation, is a model for other consortia and large-scale efforts seeking to increase RCB to consider.

Annual progress reporting by centers is a condition of NIH funding and provides center-specific reports of scientific progress, financial status, and compliance. Consortium investigators suggested that the monthly reporting of center milestones should align with the required annual research performance progress reporting. However, the milestones must be directly mapped to the documentation necessary to generate yearly progress reports. Monthly reporting of the center’s milestones through the Consortium members portal efficiently collects the data required for CDE reporting and intra-institutional and overall consortium progress. Monthly milestone reporting will inform but not replace the annual progress reports used for research fiduciary oversight. Creating CDEs prospectively and obtaining input from all end users (i.e., grantees, NIH program staff, and NIH IT support) simplifies monthly reporting while ensuring that the evaluation results have utility (https://www.eval.org/About/Competencies-Standards/Program-Evaluation-Standards). The design of the monthly surveys is such that data will be carried forward from month to month. Any updated information regarding a particular field, for example, a manuscript or grant that moves from "in progress" to "submitted," can be addressed with a click. Additionally, if fields are irrelevant that month, for example, no meetings have been conducted with community partners, then there is an option to choose not applicable or none or skip depending on the data being collected. When the annual individual center research progress reports are due, the centers can include summative information from their monthly reports.

Strengths and limitations

The process of prospectively developing CDEs for consortium-wide evaluation at the Consortium’s initiation is a strength of this work. This proactive approach allowed for a smoother process and alignment with the second round of center grantees’ (Carver Genomic Research Center, Tuskegee University; FIU Diversity Center for Genome Research, Florida State University; and Center for Genomic Diversity at Morehouse School of Medicine) milestones submission and approval. An additional strength of this work was the iterative and collaborative approach between NIH program staff, NIH IT support, and grantees in developing CDEs.

We identified limitations that impacted the CDE development and reporting process. Current funding does not allow for a dedicated coordinating center outside of the NIH, and the Consortium member portal used to collect each center’s milestones go-live date was delayed. It took several months after the three original centers received funding to build the Consortium member portal and begin to collect monthly surveys. The Consortium funding is limited to US-based resource-limited institutions. Therefore, the language that describes the CDEs may not translate to other global contexts, reflect granular details of an individual trainee’s country of origin, or apply to international consortia. The current iteration of CDEs may not measure important details of genomic training, for example, if genomic training methods use publicly available datasets, the modality of continuing education (online, self-directed, or asynchronous), trainee feedback, activities conducted by the CAB members, or if standard reporting of genomics education being used. The Consortium’s evaluation working group will consider these limitations as they provide direction moving forward. The reporting item standards for education and its evaluation in genomics (RISE2 Genomics) may be a tool for this group to adopt in their efforts to add a more robust evidence base in genomic education and evaluation across diverse settings.²⁵

Next steps

Subgroups of Consortium members will monitor selected CDE reporting to ensure that CDEs reliably measure Consortium outcomes and impact. These subgroups include NIH program staff, evaluators from each center, and Consortium evaluation working group members. This approach will maintain the integrity of evaluation, ensure that continuous quality improvement is prioritized and sustained, and provide a better understanding and elucidate areas where efforts are needed to meet the Consortium’s goal of increasing the diversity of the genomic workforce. Each center’s evaluator will collect qualitative data from their center to gain feedback on the monthly reporting process and provide recommendations for improving it.

Future evaluation planning will focus on the impact and sustainability of tracking the trainees' professional development and career trajectories. These data are needed to document the long-term effects on the genomics workforce, the scientific career trajectories of the trainees, and the overall scientific impact of the Consortium. For example, genomic RCB impacts should consider the experiences of the Consortium evaluation working group, which notes that their students often live in and desire to remain in the communities where they completed their education. This may be a consideration for future RCB evaluation efforts to understand the challenges in finding career opportunities in genomics research while continuing to grow and support the genomics workforce in and around where trainees live and study.

Conclusion

Creating CDEs that align with Consortium goals and center milestones will help ensure that ongoing evaluation allows for optimizing the RCB activities delivered by the Consortium. The prospective development of an evaluation plan was an innovative approach to enhancing the diversity of the genomics workforce and a model to replicate in other significant RCB efforts.

Acknowledgments

The authors would like to thank Ann Pennington, Ronald Carter, Jack Eidsness, Sam Cassedy, Tesh Haile, Karl Unsworth, and Melody Sahar-Khiz of the information technology branch at NHGRI for developing the DCGR Members Portal and for their considerable help and advice on the technical implementation of the surveys and Patricia Cooper from the National Cancer Institute for her help with the Qualtrics platform. This project was supported by the NHGRI-NIH-ACMG Fellowship in Genomic Medicine Program Management and the following grants from the NIH: 1UG3HG013248-01, 1U54HG013243-01, 1U54HG013247-01, 1UG3HG013553-01, 1U54HG013595-01, and 1UG3HG013615-01.

Author contributions

Conceptualization, K.J.R. (lead), A.S. (supporting), and E.B.M. (supporting); data curation, K.J.R. (lead), W.O. (supporting), A.S. (supporting), S.W.-B. (supporting), L.V.S. (supporting), and E.M. (supporting); formal analysis, K.J.R. (lead); methodology, K.J.R. (lead); writing – original draft, K.J.R. (lead); writing – review & editing, W.O. (equal), A.S. (equal), S.W.-B. (equal), L.V.S. (equal), E.M. (equal), R.D.C. (equal), S.B.F. (equal), and E.B.M. (equal); project administration, W.O. (lead); supervision, E.B.M. (lead).

Declaration of interests

The authors declare no competing interests.