Building capacity and equity in implementation science: evaluation of a national mentored training program

Abstract

Background

As implementation science evolves, it is essential to expand training capacity to build intellectual capital continually. The demand for training in implementation science far outstrips the current supply. This paper presents the methods and findings from the Institute for Implementation Science Scholars (IS-2) national training program (2020–2024).

Methods

The IS-2 was a US-based, two-year training program that provided mentored training for early- and mid-career researchers interested in applying implementation science principles to reduce the burden of chronic disease disparities. Scholars attended two annual, 2.5-day intensive training sessions, received ongoing remote and in-person mentoring, and were supported by other activities (e.g., pilot funding, networking events, mock grant reviews). A quasi-experimental (pre/post) design evaluated IS-2 on skill building, mentoring, and networking. We used descriptive and inferential statistics to characterize the sample and analyzed primary outcomes and networks.

Results

A majority of the 59 scholars were female (86%), white (61%), and assistant professors (61%). Forty-three implementation science competencies were assessed; all skill categories increased from baseline to 10 months and from 10 to 22 months post-enrollment. The relative change was largest for advanced competencies. Scholars rated their assigned mentors as highly competent across all mentoring competencies. A vibrant mentoring network was established, with the highest number of network ties in 2023, facilitating manuscript publication and joint research. Under-represented scholars (n = 21) had similar skill gains relative to scholars not-under represented, yet were less likely to hold network ties in 2024. After accounting for other predictors, sharing a mentoring relationship within the previous two years was a strong positive predictor of forming collaboration ties between network members in 2024 (odds ratio = 9.66; 95% confidence interval = 6.34–14.74). IS-2 showed multiple impacts of practice and societal relevance (e.g., improving intervention reach, building cost data in patient decision aids).

Conclusions

The approaches used in IS-2 effectively helped mentees gain skills in implementation science, experience mentorship for career development, and establish collaborative networks. The results demonstrate how the field can develop and utilize a mentoring program to reach diverse scholars, incorporate equity into curricula, and conduct high-quality mentoring to address critical implementation science topics.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-025-01446-3.

Contributions to the literature.

• This study is among the few published evaluations of a comprehensive, national implementation science training program.

• Positive outcomes included skill-building (particularly for advanced skills), mentoring, and rapid growth of mentee-mentor and peer-peer collaborations.

• Network experiences of scholars under-represented by race/ethnicity may differ from scholars not in this category.

• Assessments also provided evidence that implementation science training can impact policy, practice, population health, and health equity.

Introduction

For dissemination and implementation science (hereafter referred to as “implementation science” [IS]) to continue maturing and showing impact, human and intellectual capital must be continually developed to generate new knowledge and narrow the research-to-practice gap. There is growing emphasis at the US National Institutes of Health on addressing this translation gap [1–3]. Capacity-building in IS includes improvements in knowledge, skills, networks, and resources among scholars [4–6]. Multiple reviews have pointed out the limited range of capacity-building activities to grow the field of IS [7–11]. Capacity building for IS has occurred in multiple formats, including university degree or certificate programs, summer training institutes, workshops, and conferences [9, 12]. 

There has been continued growth of training programs in 13 countries [11], yet significant gaps remain [9, 11, 13]. In particular, despite multiple calls for a greater focus on health equity in IS [14–17], few training programs explicitly focus on health equity. Equitable IS provides “explicit attention to the culture, history, values, assets, and needs of the community—are integrated into the principles, strategies, frameworks, and tools of implementation science” [18]. Training programs in equitable IS focus not only on the training content but also on who provides the training and who is being trained (i.e., health equity includes training a more racially/ethnically diverse pool of researchers [19, 20]). Additionally, while multiple training programs include a mentoring focus, only a few programs to date have documented a significant relationship between mentoring and scientific outputs, including initiating new research, submitting new grant proposals, presenting research results, and publishing peer-reviewed papers [21, 22]. IS training programs often focus on the standard academic products (e.g., grants, publications, scientific presentations) themselves, with less emphasis on impacts outside of academia that are likely to be of higher societal relevance (e.g., practice or policy changes), and few existing training programs have used systems science methods, such as social network analysis, to examine the scope and value of networking. Finally, many of the current training programs have focused on specific topics, such as mental health and cancer [13], suggesting the need to expand attention to other priority health conditions (e.g., diabetes).

Here, we describe briefly the key elements of a national training program that attempts to address the aforementioned content gaps in training in the still relatively young IS field—The Institute for Implementation Science Scholars (IS-2) at Washington University in St. Louis, which specifically sought to: 1) build human capital for equitable IS via skill building and evidence-informed mentoring, and 2) develop a collaborative network of IS scholars and faculty.

Background: the IS-2 training program

IS-2 was a two-year training program that recruited and engaged three overlapping scholar cohorts between February 2020 and December 2024. The program provided mentored training for early-stage and mid-career investigators in using IS methods and strategies to reduce the burden of chronic disease disparities. Eligibility requirements included a doctorate and a full-time appointment in a research or clinical setting where research is conducted, as well as a stated interest in health equity. Special efforts were made to identify underrepresented scholars (URS), as defined by race/ethnicity (i.e., Black, American Indian, or Hispanic), to enrich the pool of trained IS scholars in terms of ethnic and sociocultural perspectives and experiences.

Faculty were drawn from 16 research institutions across the United States. The IS-2 faculty brought multiple core skills and experiences to this training program, including 1) a strong background in developing and conducting training programs, 2) broad knowledge in IS and development of IS competencies, 3) skills and experiences in integrating and addressing health equity in IS, and 4) proven experience mentoring early career scientists. Faculty mentors had diverse backgrounds, including behavioral science, epidemiology, social work, health psychology, and medical anthropology. Formal training in structured mentoring approaches was available to any faculty member who was interested.

To recruit scholars, we implemented a comprehensive plan to identify a large pool of qualified and diverse scholars, utilizing multiple communication platforms, including targeted email announcements, social media, research center and interest group listservs, and IS conferences. Our efforts included leveraging the networks of our faculty, who have a strong track record of training researchers from various backgrounds. Additionally, partnerships with NIH Centers, the CDC Prevention Research Centers, St. Jude Global, and the Council on Black Health enhanced our ability to attract and retain a diverse group of scholars, including those from underrepresented groups. For each cohort, capacity and resources permitted the enrollment of between 19 and 21 scholars (selected from a pool of 47 to 64 applicants per cohort). All three cohorts were active and could interact at the program's height (see Fig. 1).

Fig. 1.

Timeline and components of the IS-2 program

As shown in Fig. 1, curriculum refinement was done as an initial IS-2 activity. Although the overall training approach was based on previously developed IS competencies [23], the curriculum for IS-2 was augmented with additional equity-focused content. The team conducted a modified Delphi process with researchers and practice leaders to identify novel IS competencies related to increasing health equity and the speed of research translation [9]. This expanded list of equity-focused IS competencies provided the foundation for the IS-2 curriculum. Examples of equity-focused content in IS-2 included community engagement with marginalized communities, equity and adaptation in IS, integrating equity topics in grant writing, and equity-focused case studies.

Several elements of the IS-2 program were built upon successful approaches from previous IS training programs [9]. Key Summer Institute activities and other program elements are described in Table 1. The kick-off activity was the Summer Institute held at Washington University in St. Louis. Each scholar in each cohort attended two of these annual 2.5-day trainings, which provided didactic, group, and individual instruction, balancing didactic coursework with opportunities for networking and engagement. Evidence-informed structured mentoring was the centerpiece of IS-2 [24–26]. Based on research interests, each scholar was assigned a faculty mentor, with each mentor having one to two mentees in each cohort. Following each Summer Institute, qualitative and quantitative data were collected to solicit input on the effectiveness of training approaches and potential improvements for future sessions (see Fig. 1). Other IS-2 opportunities listed in Table 1 included product-oriented workgroups. Sample workgroup topics included policy implementation and hybrid designs [27], measurement and health equity [28], equity and telehealth [24], and IS and pediatric diabetes [25].

Table 1.

Opportunities for scholar capacity building

Training Program Elements
Summer Institutes	- 2.5 day on-site training activities (except cohort 1 due to COVID-19), including  – Plenary presentations on methods and applications of implementation science  – Fishbowls– small group peer and mentor feedback on research projects or manuscripts  – Speed-mentoring –short and focused one-to-one meetings on various career and implementation science topics
Structured mentoring meetings (virtual)	- Virtual mentee meetings with assigned mentor (e.g., 1 to 3 mentees per mentor) at least monthly for the rest of the year after the summer institute (virtual) to review research ideas or grant specific aims, input on manuscripts, presentations, academic and career development, and peer-to-peer advice and support
Pilot Funding	- Pilot funding: $1,000 to support scholar research activities flexibly, e.g., to hire research assistants, collect data, and travel to conferences
Webinars	- Quarterly webinars on topics that scholars chose to build skills and foster networking. Sample topics included writing specific aims pages, economic evaluation, complex data analysis, and designing for dissemination and sustainability. All webinars were recorded and uploaded to the IS-2 Scholar Portal
Networking Events	- The IS-2 team hosted multiple networking events, including meeting at the Annual Conference on the Science of Dissemination and Implementation in Health (in Washington, DC) and other professional meetings and convened cohort graduation celebrations that included peer-to-peer networking
Mock Grant Reviews (virtual)	- Scholars were offered mock grant reviews for their proposal submissions. The mock grant review, hosted by the scholar’s mentor, included feedback from Core Faculty Members and other scholars
Product-oriented Workgroups	- With support from Core Faculty, scholar-led optional working groups were offered to facilitate knowledge exchange, collaboration through product-focused activities, and support “cross-pollination” between scholar cohorts
Virtual Office Hours	- Virtual office hours with any Core Faculty for feedback on various research issues, additional training ideas, or career advice

Methods

Evaluation design

We used a quasi-experimental (pre/post) design to evaluate IS-2 featuring three main data collection components developed for this program: the Skills Survey (scholars’ skill assessment and mentoring perceptions), the Mentoring Competency Assessment (scholars’ perceptions of mentoring received), and the Network Survey (scholar and faculty reported collaborations and mentoring relationships).

The Institutional Review Board at Washington University in St. Louis approved all research activities in this project.

Measures

Three surveys were used for data collection to assess skills, networks, and mentoring.

The Skills Survey assessed achievement for 43 IS competencies [9, 23]. The survey used a 5-point Likert agreement scale to assess skills within four main domain areas (Additional file 1):

• A)Definitions, background, and rationale (n = 10 competencies; e.g., IS terminology, what is/is not IS research, the range of expertise needed);
• B)Theory, frameworks, and approaches (n = 7; e.g., the range of IS models/frameworks, core elements of effective interventions and implementation strategies, organizational and contextual factors);
• C)Study design and analysis (n = 14; e.g., core components of external validity, common measures, analytic strategies, mixed methods designs); and
• D)Practice-based considerations (n = 12; incorporating multi-level partner perspectives, use of participatory methods, sustaining partnerships, community engagement).

C) Study design and analysis (n = 14; e.g., core components of external validity, common measures, analytic strategies, mixed methods designs); and

D) Practice-based considerations (n = 12; incorporating multi-level partner perspectives, use of participatory methods, sustaining partnerships, community engagement).

Using card-sorting methods [23], skills had been previously defined as “Beginner,” “Intermediate,” and “Advanced” to delineate the various steps from understanding a concept to applying specific principles (Table 2). Scholars were asked to self-report their level of expertise in IS as “Beginner,” “Intermediate,” or “Advanced.”

Table 2.

Examples of beginner, intermediate and advanced skills

Skill category	Domaina	Sample competency
Beginner	A	• Define and communicate implementation science terminology
	B	• Identify appropriate conceptual models, frameworks, or program logic for implementation change
Intermediate	B	• Identify and articulate the interplay between policy and organizational processes in implementation
	C	• Apply common implementation measures and analytic strategies relevant for your research question(s) within your model/framework
Advanced	C	• Incorporate methods of economic evaluation (e.g., implementation costs, cost-effectiveness) in implementation study design
	D	• Use evidence to evaluate and adapt implementation strategies for specific populations, settings, contexts, resources, and/or capacities

^aDomains: A) Definitions, background, and rationale, B) Theory and approaches, C) Design and analysis, D) Practice-based considerations

The Network Survey assessed three core social network variables: frequency of contact, receiving or providing mentoring, and type of scientific collaboration [21]. Participants were asked to report on contact (within the last 12 months) with individuals they know and, on average, the frequency of direct contact (yearly, monthly, weekly). Contact was used as a filtering question to assess only those with contact for mentoring and collaboration. For the mentoring domain, participants were provided a list of individuals and asked to indicate individuals who had either mentored them or been mentored by them in the previous 12 months. For the collaboration domain, participants were asked to report engagement in activities within the last 12 months: 1) engaged in a joint research project or “Research,” 2) published or wrote a scholarly manuscript or “Manuscript,” 3) engaged in joint grant writing or “Grant Funding,” 4) co-taught a course or training or “Teaching,” or 5) presented research or “Presenting.” Collaboration networks were undirected, meaning that if member A nominated member B, there was a bidirectional tie between A and B, regardless of whether B nominated A. Mentoring networks were directed, requiring mutual nomination.

The Mentoring Competency Assessment (MCA) [26] for which scholars rated the skills of their mentors utilizing a validated skill inventory covering six areas of mentoring competencies using a 7-point Likert scale: maintaining effective communication, aligning expectations, assessing understanding, working effectively with mentees with different backgrounds, fostering independence, and promoting professional and career development (Additional file 1).

Impact data

As part of annual surveys of scholars, we collected data on IS-2's “real world” impacts in the program’s later years (up to four years post-enrollment). These data were categorized according to the expanded Translational Science Benefits Model (TSBM) [29–31]. These questions were intended to assess impact beyond traditional academic measures and indices. Impact domains were tools and methods for IS, clinical and medical, community and public health, economic, and policy and legislative.

Timing of data collection

IS skills were assessed at baseline (Time 1), 10-months (Time 2), and 22-months (Time 3). Mentoring skills were evaluated yearly at 10 months post-institute for faculty with active scholar mentors. Network surveys were sent to faculty and scholars (current and past cohorts) each year from 2020 through 2024. All surveys were administered by Qualtrics (Additional file 1) [32].

Statistical analysis

We first compared average skills ratings at each timepoint and calculated relative percent change at each time increment. We then compared pre- and 22 months post-enrollment data for the entire sample and stratified by URS status (as defined by race/ethnicity; i.e., Black, American Indian, or Hispanic) and disease focus area, calculating the mean difference, effect size (Cohen’s d) and relative percent change. Finally, we estimated the overall change in scholars’ skills across the three time periods with a linear mixed-effects model. Kenward-Rogers approximations were used to determine the significance of time on each skill and by skill category [33].

We analyzed network data using descriptive and visual methods. We used exponential random graph models (ERGMs) to explore predictive models of network collaboration among IS-2 scholars and faculty. ERGMs are used to model the probability distribution of a network graph based on a set of observed network statistics and structural features.

We compared any changes in scholars'ratings at 10 and 22 months for mentoring competency with paired t-tests. We also explored competency ratings at each time point by scholar characteristics (e.g., URS status as defined by race/ethnicity, gender, Scholar’s disease focus) with two-sample t-tests or one-way ANOVA.

All data were managed and analyzed in R with packages specifically designed for analyzing survey and network data [34–38].

Results

Scholar characteristics

From a total pool of 169, we enrolled 61 scholars. Of this group of 61, 59 completed IS-2 training. Most scholars were female (86%), and a majority were White (61%) (Table 3). Scholars represented 25 US states and the District of Columbia, plus 1 scholar from Australia. Thirty-six percent of scholars were under-represented (URS) by race/ethnicity. The highest percentage of scholars (61%) were university assistant professors. At the program's start, more than half (60%) had self-reported intermediate experience in IS, and 71% had access to a local mentor. The most common disease focus area was diabetes, followed by cancer.

Table 3.

Scholar characteristics reported at baseline

Characteristic	N = 59
Cohort
2020	20 (34%)
2021	21 (36%)
2022	18 (31%)
Gender
Male	8 (14%)
Female	51 (86%)
Race
White	36 (61%)
Black or African American	12 (20%)
Asian American	7 (12%)
American Indian or Alaska Native	1 (1.7%)
Other	3 (5.1%)
Prefer not to answer	2 (3.4%)
Ethnicity
Hispanic	10 (17%)
Non-Hispanic	49 (83%)
Position
Assistant Professor	36 (61%)
Associate Professor	9 (15%)
Research Scientist	4 (6.8%)
Postdoctoral Researcher	4 (6.8%)
Professor	1 (1.7%)
Other	5 (8.5%)
Implementation science self-rated experience level
Beginner	21 (38%)
Intermediate	33 (60%)
Advanced	1 (1.8%)
Disease or risk factor focus area
Diabetes	12 (20%)
Cancer	11 (19%)
Other	11 (19%)
Overlapping risk factors	25 (42%)
Current access to a local mentor
Yes	42 (71%)
No	7 (12%)
Not Sure	10 (17%)

We achieved a 100% response with the three cohorts for skill and mentoring assessment surveys at baseline. An average of 86.2% (range 73.7 to 100) was obtained at 10 months and 83.1% (range 65.0 to 94.7) at 22 months. Response rates for the social network surveys ranged from 71.0% to 93.9%.

Skills

All skill categories increased from baseline to 10 months and from 10 to 22 months post-enrollment (Fig. 2). Beginner skills had the highest mean rating at baseline (M = 3.16, SD = 0.81) and continued to be rated higher than both intermediate and advanced skills at 10 months (M = 3.94, SD = 0.60) and 22 months (M = 4.26, SD = 0.50) post-enrollment (relative change from baseline to 22 months = 35%). Intermediate skills increased from an average rating of 2.72 (SD = 0.77) to 3.52 (SD = 0.65) at 10 months and 3.90 (SD = 0.59) at 22 months post-enrollment (relative change = 43%). Advanced skills had a baseline mean rating of 2.25 (SD = 0.78) and increased to 2.98 (SD = 0.75) at 10 months and 3.46 (SD = 0.78) at 22 months post-enrollment (relative change = 54%).

Fig. 2.

Skills change over time among IS-2 scholars, by skill level

After accounting for within-participant variation, participants gained more than half a point in skills following each Institute. Beginner skill items showed an average increase of 0.57 points per time unit on a 5-point Likert scale (SE = 0.04, p < 0.001). Both intermediate and advanced skill items increased by 0.62 points per time unit (SE = 0.05, p < 0.001, and SE = 0.06, p < 0.0001).

Comparing all paired fellow data from pre- to 22 months post-enrollment, we found the highest relative percentage change (57.8%) for advanced skills (Table 4). The largest effect size was found in intermediate skills (mean difference = 1.25, d = 1.88). At 22 months, URS had skill gains similar to non-URS. Across the research focus areas, diabetes focus had the largest relative percentage change and largest effect sizes for intermediate (Cohen’s d = 1.46) and advanced skills (Cohen’s d = 1.52). Additional file 2 provides skill data stratified by URS status and research focus area.

Table 4.

Mean skill ratings pre and 22-months post-enrollment, stratified by under-represented Scholar status and disease focus area, 2020–2024

	Beginner				Intermediate				Advanced
	Time 1 Mean (SD)	Time 3 Mean (SD)	Relative % Change	Mean difference (Cohen’s d)	Time 1 Mean (SD)	Time 3 Mean (SD)	Relative % Change	Mean difference (Cohen’s d)	Time 1 Mean (SD)	Time 3 Mean (SD)	Relative % Change	Mean difference (Cohen’s d)
Full Sample
All IS-2 Scholars (N = 45)	3.11 (0.76)	4.25 (0.51)	36.6	1.14 (1.70)***	2.64 (0.73)	3.89 (3.90)	47.5	1.25 (1.88)***	2.19 (0.76)	3.46 (0.78)	57.8	1.27 (1.64)***
Stratified
URS status
URS (N = 15)	3.06 (0.89)	4.25 (0.48)	39.0	1.19 (1.55) ***	2.67 (0.87)	3.95 (0.56)	47.8	1.28 (1.61) ***	2.09 (0.81)	3.28 (0.84)	56.7	1.19 (1.44) ***
Non-URS (N = 35)	3.14 (0.71)	4.25 (0.53)	35.5	1.11 (1.75) ***	2.63 (0.66)	3.87 (0.61)	47.3	1.24 (1.95) ***	2.24 (0.74)	3.55 (0.75)	58.3	1.31 (1.75) ***
Research Focus
Diabetes (N = 10)	3.15 (0.61)	4.31 (0.56)	36.8	1.16 (1.97) **	2.64 (0.70)	4.10 (0.39)	55.3	1.46 (2.56) **	2.14 (0.59)	3.66 (0.57)	71.0	1.52 (2.63) **
Cancer (N = 9)	3.00 (1.00)	4.24 (0.51)	41.5	1.24 (1.20) **	2.49 (0.87)	3.68 (0.78)	47.7	1.19 (1.43) **	1.96 (0.61)	3.24 (0.80)	65.9	1.29 (1.80)*
Overlapping Risk Factors (N = 15)	3.25 (0.65)	4.19 (0.61)	29.0	0.94 (1.48) ***	2.74 (0.66)	3.81 (0.65)	39.1	1.07 (1.63) ***	2.27 (0.77)	3.39 (0.79)	49.4	1.12 (1.43) **

^*0.05, **0.01, ***0.001, p-value from paired t-test comparing pre and 22 months with Bonferroni correction for multiple tests

Mentoring

Scholars rated their assigned mentors highly competent across all MCA items (mean 6.64 [range 6.48 to 6.80] at 10 months and mean 6.71 [range 6.59 to 6.85] at 22 months) (Additional file 3). Just one competency (Aligning his/her expectations with your own) differed significantly from 10 to 22-month ratings, increasing from a mean of 6.59 (SD 0.70) to 6.74 (SD 0.53) p = 0.03. No significant differences were found between URS and non-URS in their total mean MCA (mean of all MCA items) ratings at 10 and 22 months. However, at 10 months, the following items were rated significantly higher by URS (p < 0.05): Coordinating effectively with other mentors with whom you work (7.00 vs. 6.42), Working with you to set research goals (6.87 vs. 6.50), Helping you develop strategies to meet research goals (6.87 vs. 6.50), Building your confidence (6.88 vs. 6.53), Understanding his/her impact as a role model for you (6.93 vs. 6.63), and Helping you acquire more resources (e.g. grants) (7.00 vs. 6.39).

Networking

The network size, or the total number of scholars and faculty (nodes), in 2020 (N = 33), 2021 (N = 62), and 2022 (N = 82) varied as new cohorts of scholars were added (Table 5). Total collaboration represents any collaboration relationship (tie) between two members in the network across any of the activities (i.e., Research, Manuscript, Grant Funding, Teaching, Presenting). The 2023 collaboration network had the largest number of ties (N = 291) and an average number of connections per member (average degree = 7.10). Mentoring networks were directed, and all mentoring ties (received and given) were highest in 2024 (N = 482), with an average of nearly 12 mentoring relationships per member (including both faculty-scholar and scholar-scholar mentoring). Year-over-year growth in mentoring relationships among faculty and the three cohorts of scholars is depicted in Fig. 3. In the network figures, the nodes are sized by the number of members to whom they provide mentoring or the number of members with whom they are connected.

Table 5.

Description of total collaboration and mentoring networks, 2020–2024

Network	Size	Ties	Densitya	Average degreeb	Betweenness centralizationc	Role modularityd
Total collaboration
2020	33	71	0.13	4.30	0.13	0.04
2021	62	147	0.08	4.74	0.28	0.00
2022	82	288	0.09	7.02	0.17	0.00
2023	82	291	0.09	7.10	0.13	0.01
2024	82	273	0.08	6.66	0.17	−0.00
Mentoring
2020	33	43	0.04	2.61	0.03	0.02
2021	62	207	0.05	6.68	0.14	0.06
2022	82	410	0.06	10.00	0.20	0.01
2023	82	416	0.06	10.15	0.23	0.05
2024	82	482	0.07	11.76	0.20	−0.01

^aThe proportion of observed to total possible ties

^bThe average number of connections

^cThe extent to which a network depends on one or a few individuals to keep everyone connected; ranges from 0 to 1

^dThe pattern of connections between faculty and scholars; a measurement of whether connections tend to exist within categories (high values) or between them (low values); ranges from −1 to + 1

Fig. 3.

Mentor networks among IS-2 scholars and faculty, 2020–2024

Betweenness centralization measures the extent to which a network's ties are influenced by a few central nodes (individuals), based on the number of shortest paths passing through each node (scale 0 to 1- closer to 1, more influential nodes) and is best used as a comparison across networks of the same size. For example, looking at years 2022–2024, where all scholars and faculty are included, the mentoring networks have moderate betweenness centralization measures (range 0.20 to 0.23), which are slightly larger than the total collaboration networks during the same years (range 0.13 to 0.17), indicating the mentoring networks may have more influential members with more ties.

Modularity, when measured based on an attribute like IS-2 role (scholar/faculty), quantifies how well the network can be divided into communities with more connections between nodes of the same role than between nodes of different roles. Modularity is typically measured on a scale from −1 to 1. A value close to 1 indicates that nodes within the role are highly interconnected and have few connections between different roles. A value close to 0 suggests no significant community structure. In contrast, a value close to −1 indicates that the network has fewer intra-community connections than inter-community connections, which is usually not observed in practical networks. Collaboration and mentoring networks'role modularity (range −0.01 to 0.05) suggested no community structures.

Figure 4 presents findings on total collaboration regardless of the type of collaborative activity. All types of collaboration increased over time except for Research and Manuscript from 2023 to 2024, Teaching and Presenting 2022 to 2023, and Manuscript 2020–2021 (Table 6 and Fig. 5). Research and Manuscript were the most common types of collaboration (range 132–138 ties and 128–161 ties, respectively) in the last 3 years (2022–2024) with all cohorts included. The majority of members (50 or more from the 82 total, or > 60%) had at least 3 types of collaboration (2022–2024).

Fig. 4.

Total collaboration networks among scholars and faculty, 2020–2024

Table 6.

Collaboration types description, 2020–2024

	2020	2021	2022	2023	2024
Types of collaboration
Research	31	85	132	149	138
Manuscript	36	29	128	167	161
Grant funding	25	46	70	74	79
Teaching	31	39	60	23	34
Presenting	18	39	58	52	76
Number of different types of collaboration
1 type of collaboration	24	55	77	73	78
2 types of collaboration	21	42	72	65	66
3 types of collaboration	17	32	54	50	57
4 types of collaboration	14	22	42	38	43
5 types of collaboration	11	12	26	21	24

Fig. 5.

Collaboration networks by type among scholars and faculty, 2020–2024

The average number of collaborations varied among subgroups within the network (Table 7). The 2022 cohort started as the most networked cohort, with a about 2 collaboration ties (M = 2.32, SD = 2.98) in their first year (2022). Faculty, likely already connected to at least each other, had the highest average number of collaborations at the start of the program (M = 9.31, SD = 3.61) and remained most networked throughout and in the final year (M = 11.64, SD = 8.20). The 2020 and 2021 cohort saw their largest number of collaborations after the conclusion of their second year (24 months) (M = 5.40, SD = 3.19 in 2022 and M = 6.05, SD = 4.78 in 2023, respectively), while the 2022 cohort saw a more rapid increase in collaborations by the conclusion of their first year (M = 6.11, SD = 4.94 in 2023).

Table 7.

Average degree^a in total collaboration network by group membership and year

	2020	2021	2022	2023	2024
	N = 33 Mean (SD)	N = 62 Mean (SD)	N = 82 Mean (SD)	N = 82 Mean (SD)	N = 82 Mean (SD)
Characteristic
Role
2020 Scholar	1.05 (1.23)	2.65 (1.79)	5.40 (3.19)	3.50 (3.12)	3.15 (2.56)
2021 Scholar	NA (NA)	1.62 (2.25)	4.52 (3.76)	6.05 (4.78)	5.71 (4.84)
2022 Scholar	NA (NA)	NA (NA)	2.32 (2.98)	6.11 (4.94)	5.63 (3.96)
Faculty Mentor	9.31 (3.61)	9.86 (6.44)	14.95 (9.28)	12.23 (9.11)	11.64 (8.20)
URSb
No	4.92 (5.02)	5.76 (5.92)	8.80 (8.27)	8.51 (7.32)	8.25 (6.70)
Yes	2.00 (2.77)	2.06 (2.79)	3.41 (2.85)	4.22 (4.29)	3.41 (3.21)
Disease risk focus area
Diabetes	1.50 (1.73)	2.17 (3.35)	6.27 (5.20)	5.93 (4.68)	5.13 (4.82)
Cancer	5.67 (4.93)	5.69 (3.45)	6.24 (4.60)	6.18 (4.52)	6.41 (4.24)
Other	2.14 (3.24)	2.67 (2.06)	5.08 (4.09)	4.46 (4.63)	4.31 (3.20)
Overlapping risk factors	5.44 (5.38)	6.07 (7.08)	8.38 (9.69)	8.92 (8.44)	8.22 (7.79)

^aThe average number of connections

^bURS Under-represented scholars based on race/ethnicity

Under-represented scholars had fewer collaborations throughout the program’s five years. Full network data from 2022–2024 show that on average, non-URSs had an additional 4–5 collaborations compared to URS. Individuals focusing on cancer had more collaborative ties than those working on diabetes in the first two years of network data collection (from 2.6 to 3.8 times more collaborations); however, these differences were minimal in later years.

Modeling the relationship between mentoring and collaborations

We tested the hypothesis that sharing mentoring relationships with stochastic models will build future collaborations within the IS-2 network. The null model includes only edges (constant term), which provides simulations of networks of the same size and density as the IS-2 network in 2024 (Table 8). In model 1, we added a term to capture the homophily or the likelihood of shared ties among nodes with the same disease risk factor research focus. We included the IS-2 role and URS status as node factor predictors. We also included terms to help understand the roles of transitivity (Geometrically Weighted Edgewise Shared Partner [GWESP]) and node degree distribution (Geometrically Weighted Degree [GWDegree]) in forming network structures. In Model 2 [34], we additionally included mentoring relationships from the two years before (2022–2023) as a relational predictor.

Table 8.

Exponential Random Graph Model (ERGM) results predicting collaboration in 2024 for scholars and faculty

	Null Model			Model 1: Node and structural predictors				Model 2: Node, structural and relational predictors
Term	Est.a	SEb	p	Est.	ORc	SE	p	Est.	OR	SE	p
Edges (constant)	−2.41	0.06	0.000	−3.84	-	0.31	0.000	−3.83	-	0.32	0.000
Same disease risk factor focus				0.44	1.55	0.11	0.000	0.39	1.47	0.12	0.001
Role (Faculty reference)
2020 Scholar				−0.59	0.56	0.12	0.000	−0.72	0.48	0.13	0.000
2021 Scholar				−0.30	0.74	0.09	0.001	−0.37	0.69	0.10	0.000
2022 Scholar				−0.23	0.79	0.09	0.007	−0.20	0.82	0.10	0.036
URSd				−0.37	0.69	0.09	0.000	−0.37	0.69	0.10	0.000
Geometrically weighted edge-wise shared partners (gwesp)				1.04	2.84	0.12	0.000	1.00	2.71	0.12	0.000
Geometrically weighted degree (gwdeg)				0.38	1.47	0.38	0.320	0.32	1.38	0.40	0.421
Shared mentoring tie present in previous two years (2022–2023)								2.27	9.66	0.22	0.000
AIC (model fit)	1889			1539				1435

^aEstimated parameter or coefficient

^bSE standard errors for the model’s estimated parameters (measures the precision of the estimated coefficient)

^cOR odds ratios (exponent of the estimate)

^dURS Under-represented scholars based on race/ethnicity

In models 1 and 2, non-URS status, faculty status, and sharing the same disease risk research focus with members all increased the likelihood of forming a tie within the IS-2 network.

Relative to faculty members, all cohorts of scholars (2020, 2021, and 2022) exhibited a lower likelihood of forming ties within the network. In Model 2, the likelihood of scholars from the 2022 cohort forming ties was approximately 71% higher than that of scholars from the 2020 cohort (ORs of 0.48 for the 2020 cohort vs. 0.82 for the 2022 cohort).

In addition, the significant GWESP term indicated a positive triad closure effect. In Model 2, each additional shared partner between two nodes resulted in a 2.71-fold increase in the odds of forming additional ties (p < 0.001). GWDegree improved the overall model fit but was not significant in either model 1 or 2, suggesting the degree distribution may capture some network characteristics, but its specific influence on tie formation is not statistically strong.

Model 2 shows that, after accounting for other predictors, sharing at least one mentoring relationship within the previous two years was a strong positive predictor of forming collaboration ties between IS-2 network members in 2024 (odds ratio = 9.66 (95% confidence interval = 6.34–14.74), p < 0.001).

Similar to logistic regression, we can calculate predicted probabilities of an outcome. In ERGMs, we predicted a tie in our network between two members. Therefore, in using the logistic transformation to estimate the probability range, we must make dyadic assumptions. For Model 2, if we assume that one member of the dyad is a Faculty Mentor and one is a 2022 Scholar, they both share the same disease area focus, both are Non-URS and where the presence of the tie produces no change in the value of either GWESP or GWDegree, the probability of a collaboration tie in 2024 is 2.54% without a shared mentoring tie (2022–2023) and 20.1% with a shared mentoring tie.

Goodness-of-fit measures were examined to determine how well our simulated models fit our observed network data. Overall, Model 2 demonstrated a good fit, evidenced by the convergence of the mean difference scores between the sampled and observed networks. Additional file 4 depicts the distribution of these difference scores, showing that the mean difference remains near 0 across the terms in the sampled simulated models, suggesting a good fit.

Impacts beyond academia

Scholars identified key translational benefits of their work using the TSBM (see Table 9) that supplement traditional academic impacts (e.g., publications, grants) [11]. These impacts were categorized across five domains: 1) IS, 2) clinical and medical, 3) community and public health, 4) economic, and 5) policy and legislative. The impacts reflect a focus on bridging research with practice and policy, benefits focusing on health equity in reaching disadvantaged populations and building clinical and community partnerships. Some impacts were immediate (e.g., developing a new course or disseminating evidence-based treatments), while others are proximal to longer-term policy changes (e.g., developing and disseminating a policy brief). The impacts in Table 9 are examples selected from 11 different Scholars.

Table 9.

Selected longer-term impacts derived from the IS-2 training^a

Domain	Example activity
Implementation science
	- Contributed to a review of equity-focused implementation science frameworks for the dissemination-implementation.org webtool
	- Contributed to the development of the first implementation science course in the Doctorate of Public Health (DrPH) Leadership course at [university X]
	- Equipped scholar to train and mentor our PhD students in implementation science as their academic department grew its core offerings
	- Used frameworks including PRISM, CFIR, and FRAME-IS to standardize our evaluation strategies and to select and test implementation strategies as we implement and disseminate child obesity programs into the real world
Clinical and medical
	- Developed Healthy Weight Clinic program, now recognized by the CDC and AAP as 1 of 7 evidence-based family healthy weight programs (goal to reach 14.4 million children)
	- Contributed to changing HIV treatment delivery in one of the largest (if not THE largest) HIV service organizations in [state X]. The reach/penetration to date has been modest, but it is growing, and we are designing for sustainability
	- Disseminated training in evidence-based treatments for eating disorders through a training initiative with the [state X] Eating Disorders Council
Community and public health
	- Contributed to developing a state-driven initiative for improving the utilization of federal nutrition support for low-income families
	- Identified and tested strategies to increase SNAP (also known as food stamp) outreach and enrollment in Eastern [state X]
	- Partnered with the National Eating Disorders Association to disseminate an online evidence-based screener for eating disorders, which over 200,000 respondents per year are now completing
	- Developed partnerships with the community, including forging relationships with the local school district and national organizations to address food insecurity in childhood using implementation science strategies
Economic
	- Incorporated direct and indirect costs in patient decision aids to help reduce exposure to unanticipated costs associated with care
Policy and legislative
	- Authored and disseminated COVID-19 Research Briefing for the US Senate Committee on Veteran Affairs
	- Authored and disseminated a policy brief for government and nonprofit partners post-COVID to advocate for the WIC program to remain remote after the pandemic was officially “over”

^aBased on the expanded Translational Science Benefits Model [29]

Discussion

Training and workforce development in IS are high priorities for our field [10, 39]. In IS-2, we trained 59 scholars who work in various content areas, settings, and populations. The approaches used in IS-2 effectively improved IS skills, reached racially diverse scholars, built equity in curricula, conducted high-quality mentoring, and showed the value of a national learning network of scholars and faculty.

Past and current IS training programs have several common elements. Nearly all training programs feature a multi-disciplinary faculty [40], which is a core tenet of effective team science [41]. Some programs use a systematic process to develop and apply IS competencies [9, 23, 42–44]. Experiential learning is often a centerpiece of large-scale training programs in IS [6], taking into account the needs and learning processes of adult learners (e.g., individuals motivated to learn by the need to solve problems) [45, 46]. The value of building a mentoring network for IS is prominent in several previous training programs [40, 47–49]. IS-2 addressed all of these elements.

Regarding improvements in skills, only one previous training program (Mentored Training for Dissemination and Implementation Research in Cancer [MT-DIRC]) had comparable data [49, 50]. MT-DIRC, like IS-2, showed statistically significant increases for all competencies. The largest gains in skills were observed for beginner competencies in MT-DIRC, whereas in IS-2, the largest increases were noted for advanced competencies. This may be because IS-2 scholars began with a higher level of beginner competencies, as supported by their local mentors.

Mentoring has long been shown to have numerous benefits (in particular, research productivity and career success) [51–53]. Mentoring benefits span the individual being mentored [54] and the mentor [55]. Feldman and colleagues found that early-career faculty with mentors had greater satisfaction with time allocation at work and higher academic self-efficacy than those without mentors [56]. Mentoring benefits from access to a collaborative learning network in which individuals receive ongoing mentoring over time to support the personal and professional growth, development, and success of partners through career and psychosocial support [52].

Mentoring networks have particular value in the team approaches used in IS [57], as in IS-2. Mentored networks in IS have multiple benefits, including multi-directional mentoring (i.e., mentor–mentee, peer-peer, home institution), increased professional reach of scholars, development of leadership skills, and collaborations that last beyond the life cycle of the official training period [57–59]. However, mentoring networks within training programs are rarely rigorously evaluated. In the IS-2 network, sharing a mentoring tie in the program's previous two years was associated with a nearly tenfold increased likelihood of collaboration in the final year of IS-2.

In our evaluation, subgroups showed differences in skill gains, networking, and mentoring. Skill gains among URS and non-URS were similar at 22 months. Regarding networking, URS had consistently lower average degree values in all social network data collection waves. The average degree values were 2 to 2.8 times higher in non-URS than among URS. In a related study of IS-2 scholars, Parrish and colleagues found that URS have a different networking experience than non-URS. For example, URS may be less satisfied with internal networks in their home institutions yet more satisfied with external networks [59], even in light of fewer network ties. This suggests a need to balance network quantity with quality [60]—high-quality mentors may help offset fewer total network ties. In IS-2, six mentoring competencies were rated more highly for URS, suggesting that their mentoring needs or institutional contexts may differ from non-URS. Scholars focusing on diabetes had fewer collaborative ties than others in the early years of IS-2, but showed a pattern in ties similar to other groups in later years. Researchers studying diabetes showed higher Cohen’s d values than other groups. This finding suggests that IS-2 activities worked well in building diabetes connections. The growth of connections in diabetes may reflect increased emphasis on IS at the US National Institute of Diabetes and Digestive and Kidney Diseases [61].

In our ERGM analyses, scholars in the earliest cohort were less likely to report collaborative ties in 2024. This is probably due in part to the effects of COVID-19. Due to COVID-19, the planned in-person Summer Institute was adapted into a series of multi-day virtual sessions. In addition, Scholars in the first cohort may have been affected more than other cohorts due to childcare or other responsibilities. Therefore, the 2020 scholars did not benefit from in-person learning and networking.

Despite its importance [14, 15, 62], few training programs in IS focus explicitly on health equity [63]. Equity-focused training includes multiple parts [15], including the ability to recruit scholars and faculty interested in health equity, whether equity is featured in core training curricula, how principles of community engagement are featured, and methods for evaluating progress toward equity. We sought to address these training elements in IS-2 in our Scholars, Core Faculty, course content, and training outputs.

Few studies have documented the longer-term impacts of IS training programs that go beyond the typical academic outputs of grants, publications, and scientific presentations. This is partly due to short-term follow-up periods on training program evaluations or a lack of resources for intensive evaluations. Despite a relatively short follow-up period, IS-2 shows multiple impacts of practice and societal relevance (e.g., improving reach and enrollment in nutrition interventions, and building cost data in patient decision aids). Several of the impacts reported by scholars focus on equity in methods, implementation strategy, or marginalized populations. To be more effective in enhancing and documenting these real-world impacts, scholars can benefit from methods for designing for dissemination, in which the products of research (e.g., an intervention, service model, policy, guidelines) are co-developed from the start with end-user input [64, 65]. In the Implementation Research Institute, which focuses on mental health, a set of helpful case studies on the impacts of training has been published [66].

Our IS-2 evaluation has strengths and weaknesses. It is one of the few national-scale training programs with longitudinal data across multiple domains of importance (skills, mentoring, networking). We had a focus on training for equitable IS (e.g., the diversity of scholars being trained, the focus of training). Along with these strengths, some weaknesses should be noted. Perhaps most importantly, while based on standardized instruments, our evaluation data rely on self-report and may be subject to social desirability bias. We lack a comparison (counterfactual) group to determine possible selection bias among scholars. We did not explore in-depth the quality and quantity of scholars’ mentoring at their home institutions, which likely contributed to specific metrics in our evaluation (particularly skill building).

The IS-2 program and the existing literature [7, 8, 11] suggest several priorities and future needs for training programs in IS.

Expand and scale-up training programs. The demand for training in IS far exceeds the current supply. For example, among six major IS training programs from 2011 through 2023, there was a combined acceptance rate of 28% [67].

Find the “special sauce” for each training program. Every IS training program needs to be tailored to the specific needs of scholars, funders, and the communities they serve. In IS-2, scholars benefited from formal training and peer-to-peer interactions, product-oriented workgroups, and mock grant reviews.

Focus more explicitly on equity. Implementation researchers report a high level of motivation to promote health equity in their research, yet few have the information or skill sets to do so [68]. Training programs that focus on a range of equity domains and dimensions throughout all stages of capacity building are likely to be the most useful.

Address training barriers. Barriers to success in training programs such as IS-2 include inconsistent mentoring, the lack of mentoring capability in one’s home institution, and challenges in extended training for people with caregiving responsibilities [58].

Expand to lower resource settings. In a review of 165 capacity-building programs in IS [11], only 1 was based in a low- and middle-income country (LMIC). Effective training programs in LMICs begin with core competencies [43], and feature the differing contexts within which implementation occurs [69].

Expand international networks. Some training programs, such as the Knowledge Translation Research and Knowledge Translation Summer Institute, have successfully built international networks [40]. Most US training programs focus only on US citizens, partly due to funding streams that limit participation from outside the United States.

Focus on advanced skills. Few training programs have focused on advanced skills in IS, even though these advanced skills have been articulated [9, 23, 67]. Our IS-2 data show that on a 5-point scale, scholars score about 1 point lower on advanced skills than on beginner skills.

Standardize measures. Few attempts have been made to standardize evaluation approaches for training programs in IS. While contextual differences need to be accounted for in each program evaluation, more standardized methods and metrics would allow for comparisons and pooled analyses.

Learn more about training sustainment. While sustainability is a key topic overall in IS [70, 71], the core factors for sustaining training programs over many years will likely differ from those that shape the continued delivery of evidence-based interventions and implementation strategies.

Focus more on implementers, not only academic researchers. There is a sizable challenge to broaden the content and scale of training for implementers (e.g., practitioners), to further the practice of implementation [72, 73]. Progress in implementation practice will require building both individual skills and organizational capacity [72, 74]. Some programs show promise in training implementers and leaders across many countries, including LMICs [73, 75, 76].

Conclusion

Implementation science enhances the likelihood that research findings will be successfully implemented in real-world settings, yielding tangible health benefits for communities and populations. A growing and more diverse pool of trained scholars is essential to realize the full promise of IS. Investment in training programs such as IS-2 not only advances clinical and population health but also enhances the science of research training—effectively building skills, fostering networks, and amplifying impact. IS-2 showed the largest changes in advanced IS skills. Through mentoring-focused learning collaboratives such as IS-2, scholars develop scientific collaborations, self-efficacy, and practice connections within an established network. Programs such as IS-2 serve as valuable models that others can contextualize and improve upon as they develop future training initiatives.

Disclaimer

The findings and conclusions in this article are those of the authors and do not necessarily represent the official positions of the National Institutes of Health or the Centers for Disease Control and Prevention.

Abbreviations

COVID-19

Coronavirus disease 2019

ERGMs

Exponential random graph models

GWDegree

Geometrically Weighted Degree

GWESP

Geometrically Weighted Edgewise Shared Partner

IS-2

The Institute for Implementation Science Scholars

LMIC

Low- and middle-income country

OR

Odds ratio

URS

Under-represented scholar

Authors’ contributions

RCB, SJK, and DHJ conceived the study. SJK coordinated the logistics of the program, the implementation of the evaluations, and the data analysis. RRJ assisted with developing survey instruments and data collection, led the data analysis, and drafted manuscript sections. LJC, DAC, GDC, GMC, KME, REG, ABH, TKH, LMK, SKK, RS, RCS, and RGT provided faculty and mentor support for the program and provided scientific input throughout the study. All authors provided edits on article drafts and approved the final version of the manuscript.

Funding

This work was supported in part by the National Cancer Institute (P50CA244431, P50CA244432, P50CA244433, P50CA244690, P50CA244693), the National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092950, R25DK123008), the National Institute of Mental Health (R25MH080916), the National Center for Advancing Translational Sciences (1UM1TR004929), the National Institute on Drug Abuse (K24DA045244), the Centers for Disease Control and Prevention (U48DP006395), and the Foundation for Barnes-Jewish Hospital.

Data availability

The datasets analyzed during the current study are available from the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

Ethics approval was obtained from the Washington University in St. Louis Institutional Review Board (#202002051; #202007186).

Consent for publication

No identifying information on any individual’s data is presented in this paper.

Competing interests

GMC, REG, and ABH are on the Editorial Board at Implementation Science. ABH is an Associate Editor and RCB, REG, and RGT are on the Editorial Board at Implementation Science Communications. All other authors declare they have no conflicting interests.