Developing the Process and Tracking the Implementation and Evaluation of the National Institutes of Health Strategic Plan for Women’s Health Research

Background

The National Institutes of Health (NIH) released Advancing Science for the Health of Women: The Trans-NIH Strategic Plan for Women’s Health Research, 2019–2023 (Strategic Plan) in February 2019. The NIH Office of Research on Women’s Health (ORWH) engaged staff members from across NIH to develop recommendations on the implementation and evaluation of the Strategic Plan. Objective: This paper describes the process used to develop recommendations for tools and approaches that NIH Institutes, Centers, and Offices (ICOs) could apply when implementing and evaluating the Strategic Plan. Methods: A Trans-NIH Strategic Plan Implementation and Evaluation Guidance Development Team conducted meetings and individual interviews with 69 NIH staff members knowledgeable about research on the health of women and sex and gender differences and met with 11 Advisory Committee on Research on Women’s Health Strategic Plan Evaluation Working Group members. The purpose of these stakeholder meetings and interviews was to obtain recommendations for implementing the Strategic Plan and identify measures for evaluating implementation success. A thematic analysis was performed to synthesize and map the recommendations to the Strategic Plan goals and objectives. Results: The process resulted in the Guide for Implementing and Evaluating the 2019–2023 Trans-NIH Strategic Plan for Women’s Health Research Across NIH Institutes, Centers, and Offices (the Guide), which includes both a conceptual and logic model for implementation and evaluation. Conclusions: The Guide offers methods, tools, and suggestions that ICO planning and evaluation staff, as well as national and international entities, can choose from when determining how to implement the Strategic Plan through ICO activities, programs, and research initiatives and how to evaluate their efforts in the context of their unique mission.

Introduction

Research on the health of women is an integral part of the National Institutes of Health (NIH) mission “to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability.” Inherent in this mission is the concept that research ultimately yields a return on taxpayer investment by benefiting everyone. Despite progress in biomedical research, women in the United States continue to experience high rates of illness and physical disability compared with women in other high-income countries. Research to identify scientifically significant sex and gender differences is an important priority for NIH. NIH envisions that (1) the biomedical research enterprise will thoroughly integrate sex and gender influences, (2) every woman will receive evidence-based disease prevention and treatment tailored to her own needs and circumstances, and (3) all women in science careers will reach their full potential.¹

Continuing its long-standing commitment to research on the health of women, NIH released Advancing Science for the Health of Women: The Trans-NIH Strategic Plan for Women’s Health Research, 2019–2023 (Strategic Plan) in February 2019. NIH’s Office of Research on Women’s Health (ORWH) coordinated the development of this Strategic Plan, working closely with other constituent NIH Institutes, Centers, and Offices (ICOs). ORWH, established by an act of Congress in 1990, has as its mission to promote research on the health of women within NIH and in the larger biomedical community; ensure that women are appropriately represented in biomedical and biobehavioral research studies funded by NIH; and support the recruitment, retention, reentry, and advancement of women in biomedical careers. Strategically positioned within the NIH Office of the Director to collaborate with other ICOs, ORWH is responsible for coordinating trans-NIH initiatives and identifying scientific opportunities to advance research on the health of women, improve understanding of how sex and gender influence health, and increase research in populations of women that are underrepresented in clinical research.

The Strategic Plan was shaped by the 21st Century Cures Act (Public Law No. 114-255) of 2016 (Cures Act), which was intended to spur innovation and bolster discovery, development, and delivery throughout the nation’s biomedical research enterprise. The Cures Act reaffirmed the NIH commitment to furthering research on the health of women by (1) endorsing the importance of including women in clinical research and assessing sex differences in human, animal, tissue, and cell studies; (2) requiring that people of all ages be represented in clinical research; and (3) expanding sex- and gender-based and race- and ethnicity-based results reporting requirements. The Cures Act mandates that all NIH ICO strategic plans integrate the principles conveyed in Advancing Science for the Health of Women: The Trans-NIH Strategic Plan for Women’s Health Research, 2019–2023 into their activities and research agendas. The law requires annual consultation between ICO directors and the ORWH director to ensure that ICO objectives and strategic plans address the health of women and women’s health disparities.

The Strategic Plan responds to the provisions of the 21st Century Cures Act and NIH policy updates on the inclusion of women and minorities (NOT-OD-18-014) and people of all ages (NOT-OD-18-116) in clinical research. The Strategic Plan emphasizes that breakthroughs in research on women’s health will be generated by moving beyond the inclusion of women in clinical studies toward the intentional integration of sex and gender considerations at every stage of the biomedical research continuum. Moreover, the Strategic Plan provides a framework for coordinating NIH efforts to enhance the evidence base for women’s health research in basic, preclinical, clinical, and translational studies and for spurring new interdisciplinary initiatives to respond to priorities in women’s health. The Strategic Plan’s three guiding principles provide an overarching frame of reference for rigorous research that aims to understand sex and gender influences, improve the health of women, and reduce health disparities (Figure 1).

Figure 1.

Guiding Principles for the Strategic Plan.

The Strategic Plan’s three guiding principles emphasize the importance of (1) considering the complex interactions of multiple factors that influence the health of women, (2) including different NIH-designated U.S. health disparity populations in research, and (3) engaging diverse perspectives in the biomedical research enterprise. These guiding principles are based on the idea that the health of women extends beyond reproductive health to encompass a broad spectrum of diseases and conditions influenced by such factors as sex, gender, race, ethnicity, and age. Those factors interact with social determinants―such as behavior, environment, and policies―across a woman’s life course to influence her health. All these factors interact with a woman’s unique biological characteristics at the genetic, molecular, cellular, and physiological levels to determine her health, longevity, and disease and disability outcomes.

The Strategic Plan presents the NIH commitment to fill gaps in knowledge about the health of women. It contains the following five strategic goals¹:

1. Advance rigorous research that is relevant to the health of women.

2. Develop methods and leverage data sources to consider sex and gender influences that enhance research for the health of women.

3. Enhance dissemination and implementation of evidence to improve the health of women.

4. Promote training and careers to develop a well-trained, diverse, and robust workforce to advance science for the health of women.

5. Improve evaluation of research that is relevant to the health of women.

Between November 2018 and May 2019, a Trans-NIH Strategic Plan Implementation and Evaluation Guidance Development Team (Guidance Development Team) composed of staff from ORWH and outside expert consultants collected and analyzed stakeholder input with the goal of generating recommendations for implementing the five strategic goals and assessing implementation success. The result of these efforts, the Guide for Implementing and Evaluating the 2019–2023 Trans-NIH Strategic Plan for Women’s Health Research Across NIH Institutes, Centers, and Offices (the Guide), provides methods and metrics that NIH staff members can use to implement the Strategic Plan through ICO activities, programs, and research initiatives and to evaluate their efforts.

The remainder of this article describes the process employed to develop the Guide, the content of the Guide itself, and the purpose and recommended use of this important document.

Developing Guidance for the Implementation and Evaluation of Strategic Plan Activities Across NIH

The development of the Guide was a systematic and participatory process. ORWH began this process by forming a Guidance Development Team given two charges:

1. Engage, through a rigorous and reliable process, key stakeholder groups across NIH to obtain advice.

2. Develop the Guide with a focus on generating tools for ICOs.

The Guidance Development Team began by gathering and reviewing information relevant to the implementation and evaluation of federal plans and programs. The information reviewed included pertinent evaluation definitions and models,^2–5 outcome metrics currently used by other federal agencies,^6,7 key elements of policies and plans to advance research on women’s health,^8–10 and the ORWH biennial report, Report of the Advisory Committee on Research on Women’s Health, Fiscal Years 2015–2016.¹¹ Information derived from these sources informed the next step in the process, soliciting input from stakeholders.

Solicitation of Input From Stakeholders

The process for selecting stakeholders ensured participation by representatives from a diverse group of ICOs. Participants had a wide range of subject-matter expertise relevant to the Strategic Plan’s goals and guiding principles.

The Guidance Development Team used purposive sampling approaches to identify individuals with an understanding of research on women’s health, including representatives from ORWH; the Advisory Committee on Research on Women’s Health (ACRWH); the NIH Coordinating Committee on Research on Women’s Health (CCRWH); and the co-chairs and members of the five teams that worked on developing the five goals of the Strategic Plan. The ACRWH is composed of scientists, clinicians, and other health professionals from the scientific community who advise the ORWH and make recommendations on priority topics affecting women’s health and sex/gender differences research. The CCRWH is composed of ICO directors or their senior designees who serve as liaisons between ORWH and NIH ICOs and provide guidance and support to ORWH (see Figure 2).

Figure 2.

Feedback Collection and Analysis Process and Stakeholder Roles in Developing the Guide.

Members of these committees and Strategic Plan team co-chairs were asked to name colleagues with a long-standing record of championing the health of women, which resulted in the identification of an additional 12 NIH stakeholders who agreed to provide input. Additional participants were members of the following bodies:

• the Trans-NIH Sex as a Biological Variable (SABV) Working Group, which was created to inform the development and implementation of policy requiring SABV to be factored into research designs and analyses for studies using vertebrate animals and humans

• the Inclusion Governance Committee, which monitors inclusion in clinical research by sex/gender, race/ethnicity, and age

• the NIH Working Group on Women in Biomedical Careers, which is a trans-NIH effort to develop innovative strategies to promote entry, recruitment, retention, and sustained advancement of women in biomedical and research careers

• the Center for Scientific Review, which organizes peer review groups or study sections that evaluate the majority of research grant applications sent to NIH

• the Dissemination and Implementation Research component of the NIH Office of Disease Prevention, which provides frameworks, theories, and models to help researchers systematically understand, use, and evaluate evidence-based interventions

• the NIH Office of Evaluation, Performance, and Reporting, which provides leadership and coordination of NIH efforts to better capture, communicate, and enhance the value of biomedical research

Table 1 describes the distribution of participants in the interviews and meetings, the consultation methods employed, and the time of data collection. A total of 80 individuals participated.

Table 1.

Stakeholder Groups and Details on Consultations.

Stakeholder Group	Number of Participants	Consultation Type	Time Frame
ORWH leadership and staff members	19	In-person group brainstorming meeting	November 2018
ACRWH Implementation and Evaluation Working Group members	11	Feedback form and conference call	February–March 2019
CCRWH members	27	ICO input form	March 2019
Strategic Plan co-chairs	11	Individual or group telephone interviews	April–May 2019
Other stakeholders	12	Individual telephone interviews	April–May 2019
Total participants	80

Stakeholder Input Collection and Thematic Analysis

Stakeholder input collection occurred between November 2018 and May 2019. ORWH formed the ACRWH Implementation and Evaluation Working Group to provide advice to the Guidance Development Team, review early drafts of analysis reports, and suggest implementation activities and evaluation metrics.

During the first phase of the data collection and analysis process, a brainstorming session with ORWH leadership and staff yielded input on (1) what successful Strategic Plan implementation would look like and (2) the questions that would need to be answered to evaluate implementation. The Guidance Development Team compiled, analyzed, and synthesized the input; mapped it to the Strategic Plan’s goals and objectives; and created (1) a draft conceptual model for Strategic Plan implementation and evaluation and (2) a draft overall Strategic Plan evaluation logic model.

Members of the ACRWH Implementation and Evaluation Working Group and the CCRWH provided written feedback. Specifically, ACRWH Implementation and Evaluation Working Group members were asked to provide feedback on the following: (1) appropriate implementation activities for NIH ICOs and other stakeholders to ensure the goals of the Strategic Plan are achieved; (2) metrics of NIH ICO progress toward achieving the Strategic Plan goals and objectives; (3) short-, medium-, and long-term outcomes for measuring the implementation of the Strategic Plan; (4) metrics for demonstrating that outcomes have been achieved; (5) assumptions that need to be captured when evaluating progress toward Strategic Plan goals; and (6) external factors that could affect the implementation of the Strategic Plan. Two Guidance Development Team members performed independent thematic analyses of the ACRWH Implementation and Evaluation Working Group’s input. After analysis and summarization of the feedback, Guidance Development Team members convened a conference call with this working group to present and clarify the first-phase input from members of the ACRWH Implementation and Evaluation Working Group. During the call, the following recommendations were generated as well as an Implementation Plan roadmap:

• Educate and disseminate to increase awareness of the Strategic Plan among NIH staff.

• Promote adoption of the Strategic Plan within and outside of NIH.

• Include the Strategic Plan in IC-specific strategic planning.

• Sustain or create processes to support implementation of the Strategic Plan.

• Provide funding and co-funding and stimulate partnerships.

• Hold and advertise conferences, workshops, symposia, and webinars related to the Strategic Plan.

• Make sure there is a system/process in place to monitor adherence to SABV and inclusion policies.

• Promote career equality for women at NIH.

• Create opportunities for women scientists.

• Develop publications.

The recommendations were summarized and presented at the ACRWH meeting on October 23, 2019. All ACRWH members discussed the feasibility of the proposed implementation activities, evaluation metrics, and collaboration strategies and agreed to provide further guidance.

During the second phase of the data collection and analysis process, the Guidance Development Team used a feedback form to collect information from CCRWH members regarding ICO-specific plans and considerations for implementing the Strategic Plan and how implementation of the Strategic Plan should be evaluated. The form asked CCRWH members representing various ICOs about (1) planned activities to implement the Strategic Plan at their ICOs, (2) names of staff at the ICO who would be involved in their implementation process, (3) outcomes they hoped to achieve at their ICO as a result of implementing the Strategic Plan, (4) how they planned to measure the effectiveness of this implementation process at their ICO and what metrics and data sources would be used for that purpose, (5) how they planned to ensure that the Strategic Plan implementation activities contributed to their ICO’s mission, (6) steps they will take to ensure the Strategic Plan implementation at their ICO promotes NIH’s vision for the Strategic Plan, and (7) NIH policies and regulations for which ICO compliance will be improved as a result of the Strategic Plan implementation. The form also asked CCRWH members to identify other NIH staff who were champions of research on women’s health and sex differences.

The Guidance Development Team members also conducted semi-structured interviews with the co-chairs of each team that worked on the five strategic goals. These co-chairs recommended additional NIH stakeholders with expertise in women’s health research to interview. The co-chairs and other stakeholders were asked about the following: (1) approaches for implementing the Strategic Plan; (2) relationship of the Strategic Plan to NIH’s vision, mission, and policies; (3) questions to ask to evaluate the implementation of the Strategic Plan; (4) anticipated short-, medium-, and long-term outcomes of a successful implementation process; (5) evaluation methods, metrics, and data sources; (6) challenges to the effective implementation of the Strategic Plan and evaluation of that process; and (7) strategies for overcoming those challenges.

Three Guidance Development Team members independently reviewed stakeholder input, conducted a thematic analysis of this input, synthesized stakeholder recommendations and mapped them to the Strategic Plan goals and objectives, and generated a list of potential activities that ICOs could implement to achieve the goals and objectives of the Strategic Plan.

Results of the Input Collection and Analysis Process and Development of the Guide

The results of the collection and analysis of input from NIH stakeholders and the ACRWH Implementation and Evaluation Working Group―combined with a review of methods, metrics, and models used by other federal agencies―provide the basis for the content for the Guide. The Guide offers tools and suggestions that ICO planning and evaluation staff and scientists can use when determining how to approach implementing the Strategic Plan and evaluating its implementation. It also offers a resource to help NIH ICOs understand the context of the Strategic Plan, how NIH is promoting it, and what they can do to implement and evaluate activities and initiatives.

The Guide suggests six overarching implementation activity categories for ICOs:

1. Include the Strategic Plan in ICO-specific strategic planning.

2. Sustain or create processes to support implementation of the Strategic Plan.

3. Provide funding or co-funding and stimulate partnerships.

4. Promote the implementation of the Strategic Plan by convening and advertising conferences, workshops, and symposia, as well as through publications.

5. Encourage and monitor adherence to the SABV and inclusion policies.

6. Promote gender diversity of the biomedical workforce and create career development opportunities for women scientists.

ICOs are free to choose the types of women’s health-related activities to incorporate into their own strategic planning efforts. Evaluation of the implementation of these activities is encouraged but not required.

The Guide includes several tools that ICOs can use to develop plans for implementing and evaluating activities relevant to the Strategic Plan goals and objectives. ICOs are free to choose the tools they deem most useful. Tools in the guidance document include the following:

• a conceptual model illustrating the relationship among all efforts to promote the implementation of the Strategic Plan and evaluation of the effectiveness of those efforts (Figure 3)

• a brief discussion of terminology to ensure that users of the Guide work from common definitions of measurement terms as they relate to the Strategic Plan

• an evaluation framework to assess the Strategic Plan’s implementation

• an overarching evaluation logic model (Figure 4)

• recommended data sources and analytical methods to facilitate evaluation

• a logic model, evaluation questions, and a table of potential implementation activities and evaluation metrics for each strategic goal

• a table listing challenges that ICOs might face, as well as stakeholder recommendations for responding to those challenges

Figure 3.

Conceptual Model for Implementing the Strategic Plan and Evaluating Its Implementation. The conceptual model was developed by Dr Samia Noursi, Laura Sharon, Dr Dawnkimberly Hopkins, Dana Simms, and Dr Ching-yi Shieh.

Figure 4.

Overarching Strategic Plan Evaluation Logic Model.

Challenges and Limitations

The process for developing this guide required the collaboration of many individuals and groups across the NIH, involving many interactions to ensure that ORWH connected with appropriate stakeholders within the organization. Identifying appropriate stakeholders to ensure adequate representation across NIH was challenging. A large number of NIH staff are involved in activities relevant to women’s health research and it was impossible to interview them all. ORWH therefore decided to begin by interviewing several leaders within NIH, including members of NIH-wide working groups, senior staff who have worked on strategic plans, and chairs of various NIH committees. These individuals were asked, in turn, to identify and recruit appropriate interviewees. This process required political savviness and consideration of all involved to ensure that no one that needed to be involved was overlooked. Once the list of interviewees was generated, a second challenge was scheduling interviews with the large number of NIH staff members on this list who tended to have full schedules. Finally, transcribing the content of the interviews and analyzing the feedback was time-consuming and required many hours of staff time. Despite all of these challenges, ORWH was able to secure the necessary resources to accomplish its Strategic Plan Guide information collection goals.

Conclusions

NIH cannot fully realize its mission of “turning discovery into health” unless the biomedical findings and innovations it generates apply to both women and men. The Advancing Science for the Health of Women: The Trans-NIH Strategic Plan for Women’s Health Research, 2019–2023 offers a road map for moving closer to a biomedical research enterprise that thoroughly integrates sex and gender influences, provides every woman with evidence-based interventions, and facilitates the attainment of women’s full potential in science careers. Through their research, programs, and activities, all NIH ICOs—individually and as part of NIH’s interdisciplinary approach—advance our understanding of scientifically important sex and gender influences and of the many factors that contribute to the health of women. The guidance document described in this article and the systematic and collaborative process that led to its creation will help ensure that the vision for research on the health of women, NIH’s overall mission, and the missions of ORWH and other ICOs are aligned to advance the goals and objectives of the Strategic Plan. By providing ICOs with recommended activities and tools for Strategic Plan implementation and evaluation, NIH ensures that individual strategic plans and collaborative efforts across NIH address the health of women and women’s health disparities. The adoption of the Strategic Plan into the ongoing work of ORWH and other NIH ICOs—as well as continued engagement with external stakeholders in the biomedical research community—will turn discovery into health by reducing health disparities, improving medication safety, and enhancing health care as part of personalized medicine with the ultimate result of safe and effective diagnostics, therapeutics, and preventive strategies that are sex- and gender-informed.

Despite progress in biomedical research, women in the United States continue to experience high rates of illness and physical disability compared with women in other high-income countries. Women in low-income countries also suffer devastating consequences as a result of illness, physical disabilities, and other health conditions. NIH prioritizes improving women’s health both across the United States and globally; supporting women’s health research worldwide; promoting the career development of women scientists around the world; participating in government-wide efforts to end gender-based violence; and partnering with international organizations to improve global health equity and empowerment for women, girls, men, and their communities. The information provided in the guidance document can be adapted to strategic planning and evaluation of women’s health research efforts in other settings, both nationally and internationally. We are hopeful that the Guide will be a useful resource within the United States and other countries for increasing focus on women’s health research to ultimately improve the lives of women everywhere.