Building National Capacity for Research Mentor Training: An Evidence-Based Approach to Training the Trainers

This article describes the development, implementation, evaluation, and impact of a train-the-trainer workshop designed to promote widespread dissemination of an evidence-based research mentor training curriculum.

Abstract

Research mentor training (RMT), based on the published Entering Mentoring curricula series, has been shown to improve the knowledge and skills of research mentors across career stages, as self-reported by both the mentors engaged in training and their mentees. To promote widespread dissemination and empower others to implement this evidence-based training at their home institutions, we developed an extensive, interactive, multifaceted train-the-trainer workshop. The specific goals of these workshops are to 1) increase facilitator knowledge of an RMT curriculum, 2) increase facilitator confidence in implementing the curriculum, 3) provide a safe environment to practice facilitation of curricular activities, and 4) review implementation strategies and evaluation tools. Data indicate that our approach results in high satisfaction and significant confidence gains among attendees. Of the 195 diverse attendees trained in our workshops since Fall 2010, 44% report implementation at 39 different institutions, collectively training more than 500 mentors. Further, mentors who participated in the RMT sessions led by our trained facilitators report high facilitator effectiveness in guiding discussion. Implications and challenges to building the national capacity needed for improved research mentoring relationships are discussed.

INTRODUCTION

Strong mentorship has been linked to enhanced mentee productivity, self-efficacy, and career satisfaction; it is also an important predictor of the success of researchers in training (Palepu et al., 1998; Garman et al., 2001; Ramanan et al., 2002; Steiner et al., 2004; Sambunjak et al., 2006; McGee and Keller, 2007; Raggins and Kram, 2007; Bland et al., 2009; Feldman et al., 2010; Cho et al., 2011; Shea et al., 2011; Fleming et al., 2012). Mentored students are more likely to make decisions leading to academic persistence (Gloria and Robinson Kurpius, 2001), with positive mentoring being cited as the most important factor in degree attainment (Solorzano, 1998). For members of underrepresented minority (URM) groups, mentorship has been shown to enhance recruitment into biomedical research and related career pathways (Nagda et al., 1998; Hathaway et al., 2002). Incorporating mentorship of junior faculty members into faculty-development programs can improve retention in academia (Daley et al., 2006; Ries et al., 2009). Despite its critical importance, mentors generally do not receive training. Rather, they typically learn by example, trial and error, and peer observations (Keyser et al., 2008; Silet et al., 2010).

The Entering Mentoring (EM) curriculum (Handelsman et al., 2005) was originally designed to improve the effectiveness of graduate and postdoctoral mentors working with undergraduate researchers (mentees). It has since been adapted for research mentors who work with mentees at various career stages across the science, technology, engineering, and mathematics (STEM) disciplines and in medicine and public health (Sorkness et al., 2013; https://researchmentortraining.org; https://mentoringresources.ictr.wisc.edu). The foundation of the training is a process-based forum wherein mentors learn core mentoring competencies, experiment with various mentoring strategies, and solve mentoring dilemmas within small peer groups. Training sessions are typically offered as a series of 1- to 2-h interactive sessions facilitated by one to two faculty members, staff members, or postdoctoral trainees. The six competencies from the EM-based curricula are: 1) maintaining effective communication, 2) establishing and aligning expectations, 3) assessing mentees’ understanding of scientific research, 4) addressing equity and inclusion within mentor–mentee relationships, 5) fostering mentees’ independence, and 6) promoting mentees’ professional career development (Pfund et al., 2012a, 2013).

Quantitative and qualitative data indicate that, compared with untrained mentors, mentors who participated in EM-based training communicate with their mentees more effectively (Pfund et al., 2006). Undergraduate mentees indicated that they had better experiences with the trained mentors, compared with their previously untrained mentors (Pfund et al., 2006). Recently, the EM curriculum was adapted for faculty mentors of junior faculty members and postdoctoral trainees engaged in clinical and translational research (Pfund et al., 2012a, 2013). This curriculum was tested as part of a randomized controlled trial (RCT) at 16 academic sites, including 15 National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) institutions. Mentors were randomized into experimental or control groups at each institution. Both mentors and their mentees reported a positive impact on mentoring knowledge, skills, and behavior (Pfund et al., 2013). Specialized curricula tailored for mentors of biomedical, clinical and behavioral, and community-engaged researchers have subsequently been developed and tested (Asquith et al., 2014; House et al., 2014; Pfund et al., 2014a).

EM was intentionally designed as an easy-to-follow manual for those interested in implementing research mentor training (RMT), since curricula with detailed instructional notes have been reported to be effective for broad implementation (Smith et al., 1993). Each chapter includes clear learning objectives, activities, comprehensive training materials, detailed facilitator notes, and links to relevant online resources. The modular design of the curricula allows trainers to mix and match competencies and related activities to fit the needs of their mentors and their local context. To date, all of the adapted curricula have been made freely available online (https://researchmentortraining.org; https://mentoringresources.ictr.wisc.edu). These websites include supporting resources as well as build-your-own options, so users can customize curricula for their own purpose and download selected materials and accompanying facilitator notes as PDFs. In addition, several of the curricula have been published in print as part of the Entering Mentoring series (Handelsman et al., 2005; Pfund et al., 2012a, 2014b).

Since 2005, the EM series curricula have been used to train thousands of mentors across the country, including those mentoring undergraduates, graduate students, and postdoctoral trainees across STEM and medicine. However, dissemination of this evidence-based practice has not reached its full potential. In some cases, predictable barriers such as limited resources, rewards, and time are cited as the reasons for lack of implementation (Henderson and Dancy, 2007; American Association for the Advancement of Science [AAAS], 2011; D’Avanzo, 2013). However, more often than not, the faculty members, staff members, training grant directors, and undergraduate research program directors who wish to implement RMT lack the confidence to facilitate training on their own, despite the availability of the curricula and detailed facilitation notes. For example, many report that they lack content expertise, despite years of mentoring experience, while others cite a lack of small group–facilitation experience. Still others explain that they are simply more comfortable bringing in an “expert” facilitator to implement the training. This lack of confidence is not surprising; it has been cited as a common barrier to widespread dissemination and implementation (Hutchinson and Huberman, 1994; Henderson et al., 2011). However, dependence on external, expert trainers limits scalability and relies on a business model that can lead to inequitable access. Therefore, overcoming this confidence barrier is critical to the dissemination of RMT, especially as federal agencies call for training programs to include evidence-based mentoring practices and to incorporate effective ways for mentors to promote the professional development of their mentees, including the use of individual development plans (Hobin et al., 2012; Rockey, 2013; NIH, 2014).

To address the confidence barrier among potential users and empower them to build the needed local capacity for RMT, we developed an extensive train-the-trainer workshop for those interested in facilitating RMT. The train-the-trainer model is one means of dissemination and capacity-building that has been used across multiple contexts, including K–12 teacher development, professional development, and clinical training (Guskey, 2002; Guskey and Yoon, 2009; Pfund et al., 2009; Godfrey et al., 2004; Rubak et al., 2008; Brownson et al., 2012; Stevens and Hoskins, 2014). Previous literature indicates that multifaceted, interactive, train-the-trainer workshops may be an effective means of disseminating and implementing professional development curricula (Pearce et al., 2012). However, strong evidence of its effectiveness is still needed. In this paper, we report on our iterative approach to developing a train-the-trainer workshop (hereafter referred to as facilitator training [FT] and the effectiveness of our approach in increasing dissemination and implementation of the evidence-based RMT curriculum, Entering Mentoring (Handelsman et al., 2005; Pfund et al., 2006, 2014c; Sorkness et al., 2013). Our FT was based in part upon the successful training developed for facilitators of the National Academies Summer Institute (SI) on Undergraduate Education in Biology (Pfund et al., 2009; Wood and Handelsman, 2004; www.academiessummerinstitute.org). We report the previously unpublished evaluation results from the original FT implemented at the SI and the subsequent iterations of an FT workshop to prepare facilitators of RMT. Data are included from both the participants who attended FT and the mentors who subsequently engaged in the training they led. We highlight our development of the described FT across three phases and its effectiveness in increasing dissemination of an evidence-based approach to RMT on a national scale.

METHODS

Overview

The iterative nature of this research design is multifaceted and interactive, yielding an effective means of disseminating and implementing RMT. The RMT train-the-trainer workshop was originally based on the FT developed for the National Academies SI on Undergraduate Education in Biology and was subsequently revised through three subsequent phases: the RCT (phase 1), preparing for scale-up (phase 2), and national scale-up (phase 3).

Development of FT for the SIs

Our FT was based on a workshop we developed for the National Academies SI on Undergraduate Education in Biology (Wood and Handelsman, 2004; Pfund et al., 2009; www.academiessummerinstitute.org). The SIs are 5-d workshops for faculty and staff members aimed at developing evidence-based teaching skills in order to improve teaching in undergraduate science classrooms across the country. During the week, SI participants work in small groups to design instructional materials for their college biology courses. Each working group of five to eight participants is guided by a trained facilitator who participated in a previous SI and attended 6 h of FT. The FT agenda is shown in Table 1. The goals of this FT are to 1) increase facilitator knowledge of the SI curriculum, 2) increase facilitator skill and confidence in leading SI working groups, and 3) review the logistics of the SI.

Table 1.

SI FT agenda (2010)

Session topic	Description
Introductions	Attendees choose a picture that represents how they view their role as a facilitator and share it with the other attendees.
Orientation to the work spaces	Attendees explore the spaces in which they will work during the week.
Overview of the week	Attendees work with a partner to answer the challenge questions about the SI using the participant materials to learn the week’s objectives, curriculum, resources, and materials. Answers are discussed in large group.
Connecting the week’s activities and terms	Attendees work in groups of three to create a concept map of the vocabulary used during the SI. Attendees share their concept maps with the larger group and discuss the varied uses of these terms and how they can become working definitions for the week.
Questions about the week	Attendees ask any remaining questions they have about the SI and the week ahead.
Facilitation practice	Attendees read a scenario about a frustrated SI participant and identify ways to effectively facilitate this difficult situation.
Tools for dealing with group behaviors	Attendees engage with two resources about group dynamics (Constructive and Destructive Group Behaviors and Five Stages of Group Development), learn the roles and responsibilities of group facilitators, and draw on their previous experience as SI participants (or facilitators, if relevant) to brainstorm strategies to promote productive working group behaviors.
Strategies for establishing group norms and values	Attendees share strategies they have used to establish working group norms and values.
Planning time	Attendees work with a partner to design the opening 10 min of the first small-group work session, identify its objectives, and request resources.

SI FT Attendees

All facilitators attended the 6-h FT before leading a working group at the weeklong SI. Two surveys were distributed to determine the effectiveness of the SI FT and the SI facilitators: a follow-up survey for SI facilitators to rate the quality of the FT experience and a postevent survey for SI participants to evaluate the event and the quality of facilitation.

A onetime SI survey to evaluate the FT was deployed in Fall 2009 for everyone who facilitated a working group at an SI between 2006 and 2009. In this survey, facilitators selected from a 13-item list all the aspects of the FT they used when working with their groups at the SI. They also identified the extent to which the FT was sufficient to help them meet a series of SI and facilitator goals (scale: 1 = sufficient, 2 = not sufficient, 3 = don’t remember). In addition, the facilitators answered open-ended questions to determine what additional FT would have been helpful and what materials and resources they would need if they were to implement their own FT at their own regional SI.

SI Participants

The SI participant survey was launched annually from 2006 to 2010, ∼1–2 wk after completion of the SI. The questions were identical each year, so we report here the aggregated data across the 5 yr. Participants received an email link to complete an electronic survey in which they rated the general value of “having a trained facilitator for group work,” using a Likert-type scale with values of 1 = not at all valuable, 2 = somewhat valuable, 3 = extremely valuable, and N/A. In the aggregated 5-yr data, value responses of 2 and 3 comprise a single “valuable” category. In addition, participants rated their agreement with a series of statements specific to their facilitators, using a scale of 1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree, and N/A; agreement responses of 1 and 2 comprise a “disagree” category, and agreement responses of 3 and 4 were aggregated into an “agree” category. They also identified their facilitators by name and offered open-ended comments and suggestions for the facilitators and the SIs overall.

Development of FT for RMT: The RCT (Phase 1)

On the basis of feedback and the success of the SI FT, we adapted this approach, maintaining the most highly rated elements, to develop a 1.5-d FT for those who were preparing to implement RMT as part of a multisite RCT to test the effectiveness of one of the EM series curricula, Mentor Training for Clinical and Translational Researchers (Pfund et al., 2012a, 2014c). The goals of this workshop were to 1) increase facilitator knowledge of the RMT curriculum, 2) increase facilitator confidence in implementing the curriculum, 3) provide a safe environment to practice facilitation of activities from the curriculum, and 4) review logistics for implementation and evaluation. The specific learning objectives were for attendees to gain the ability to 1) establish constructive group dynamics, 2) facilitate a functional small-group discussion, 3) draw out diverse perspectives in a group discussion, 4) deal with challenging behaviors within a small group, and 5) implement the mentor training curriculum at their institutions. The training activities that address these learning objectives are outlined in the FT agenda in Table 2. Some notable additions to this FT, compared with the SI version, are the inclusion of three new sessions: one on learning through diversity (Pfund et al., 2012b; www.cirtl.net/CoreIdeas/learning_through_diversity), a second in which attendees were afforded the opportunity to practice their facilitation skills and receive peer feedback, and a third session on how to implement RMT at their home institutions.

Table 2.

RCT FT agenda

	Description of activities
Day 1: Topic and objective
Introductions and overview	Attendees engage in an introductory activity from the curriculum and discuss ground rules for the workshop.
Overview of facilitation	Attendees break into small groups and discuss what facilitation is and what it is not. Attendees consider how facilitation compares with teaching.
Establishing group dynamics	Attendees select their most constructive and destructive group behaviors, break into small groups, and discuss their chosen behaviors. In the large group, attendees discuss how facilitators can use this activity to establish group norms and address problematic group dynamics.
Learning through diversity	Attendees work individually to identify visible and invisible forms of diversity and discuss in the larger group the benefits and challenges of drawing out diversity for the benefit of all without creating feelings of tokenism among individuals who are in the minority.
Planning your first mentor training session	Attendees work with a partner to plan the activities for the introductory session. A template is provided with a list of example activities, many of which were modeled earlier in the training.
Lunch and mentor training curriculum challenge	Attendees enjoy lunch with colleagues and complete the curriculum challenge, which asks them to answer questions based on the provided curricular materials.
Curriculum overview	Attendees become familiar with the four-session curriculum plan, materials, and available support for implementation. Attendees ask questions in the large group.
Facilitation practice session	Attendees engage in the core activities across the curriculum. Individuals rotate in the role of facilitator during the session to give and receive feedback.
Day 2: Topic and objective
Cofacilitation	Attendees discuss the benefits and challenges of cofacilitation and work with their cofacilitators to establish roles and responsibilities.
Facilitation practice session	Attendees engage in the core activities across the curriculum. Individuals rotate playing the role of facilitator during the session to give and receive feedback.
Practice session debrief	Attendees share with the larger group their experiences in the practice sessions as participants or facilitators. Attendees raise challenges and solutions to issues that arose in the practice sessions.
Implementing the curriculum on your campus	Attendees learn about the process of implementing the curriculum and what supports are available to attendees before, during, and after implementation, including evaluation instruments.
Evaluation of the workshop	Attendees complete a short survey evaluating the training workshop.

At the end of the workshop, attendees were asked to rate individual components of the FT on a Likert-like scale with values of 1 = not at all valuable, 2 = somewhat valuable, and 3 = very valuable. Attendee responses to the open-ended question “What could the planners do to improve this workshop?” were also analyzed. Attendees also were asked to rate their confidence retrospectively, comparing their confidence before FT and at the end of FT. To evaluate the effectiveness of our trained facilitators, mentors who engaged in their RMT sessions across the 16 sites rated both the effectiveness of the training overall and the effectiveness of their facilitators (Pfund et al., 2013).

Adaptation of FT for RMT: Preparing for Scale-Up (Phase 2)

On the basis of the success of the FT described above, we worked to develop a shorter, streamlined version that would allow for more effective and efficient national dissemination. The criteria for this modified training included:

• Shortened training time not exceeding 6 h for 1-d implementation

• Flexibility for trained facilitators to use any of the existing (and future) RMT curricula and easily tailored to fit diverse audience needs

• Simpler instructions for ease of training and implementation

The topics included in the current FT agenda, resulting from several iterations of implementation and feedback, are shown in Table 3. There were three principal changes to the workshop content, compared with the one initially developed for the RCT. First, activities were added to help facilitators navigate the logistics of implementing RMT at their institutions. Second, the curriculum’s challenge and learning through diversity sections were removed as separate components, as these were regularly rated as the least valuable. However, participants were given the opportunity to walk through the curriculum in the Curriculum Overview, and all were exposed to the mentor training materials focused on equity and inclusion during the practices sessions. Third, in a final iteration, a “mini” mentor training session was modeled for attendees. Below we describe evidence of the effectiveness of this modified approach across a range of audiences with diverse interests and needs. FT workshops including the components in Table 3 were conducted at five institutions between August 2012 and May 2013 (the Medical College of Wisconsin, University of Cincinnati, University of Maryland–College Park, University of Pennsylvania/Children’s Hospital of Pennsylvania, and Vanderbilt University). Attendees at all five FTs rated all components of the workshop (Table 3).

Table 3.

Preparing for scale-up mean satisfaction ratings of each component of the modified FT workshop^a

Workshop component	Average rating	SD
Introductions and workshop overview	2.75	0.44
Model of RMT sessionb	N/A	N/A
Curriculum overview	2.61	0.49
Facilitation basics (overview)	2.68	0.45
Practice facilitation and debrief	2.81	0.39
Implementing the curriculum	2.49	0.55
Evaluation	2.61	0.50

^aWorkshop components were rated on a Likert-like scale with 1 = not at all valuable, 2 = somewhat valuable, and 3 = valuable; n = 38, with 79% reporting.

^bThis component was added after successful use in four FT workshops and is now standard.

Adaptation of FT for RMT: National Scale-Up (Phase 3)

On the basis of evaluation results during phase 2, we decided to scale up our efforts and offered four FT workshops between May and November 2013. We invited anyone interested in implementing RMT for mentors working with mentees across a broad range of disciplines and career stages, with particular emphasis placed on reaching a more diverse audience. Each FT was offered the day preceding or following a national meeting, either onsite or near the conference. The four conferences were selected due to their history of success in targeting participants from underrepresented groups across a broad range of career stages and disciplines and engaging many training-grant program leaders. To capture the diversity of FT attendees, we collected demographic data from attendees before they attended the workshop. For race and ethnicity (see Table 4), attendees could choose more than one option, allowing for inclusivity of attendees’ backgrounds. The workshop components implemented are shown in Table 3, with some minor rearrangement of activities required to accommodate varied start and end times and addition of the mini-RMT component. Importantly, attendees in these FTs had the opportunity to preselect which EM series curriculum they wanted to focus on in the practice facilitation sessions.

Table 4.

National scale-up demographic data from attendees in four FT workshops^a

			Gender		Race/ethnicity (check all that apply)
National venue	n Trained	Overall response rate	Male	Female	White	Black	American Indian	Hispanic/Latino	Other
Boston University, Boston, MA; during American Public Health Association meeting	21	90%	8	9	14	1	0	1	3
Health Equity Leadership Institute, Madison, WI	29	86%	11	14	17	6	3	4	5
Society for the Advancement of Chicanos and Native Americans in Science, San Antonio, TX	17	65%	1	10	8	0	1	7	2
Annual Biomedical Research Conference for Minority Scholars (ABRCMS), Nashville, TN	45	64%	8	23	9	20	0	4	4

^aDemographics are reported for attendees who completed the postworkshop surveys.

Attendees in FT (Phases 1–3)

Evaluation surveys were administered in paper format immediately after each FT. Attendees rated individual components of the FT workshop on a Likert-like scale with 1 = not at all valuable, 2 = somewhat valuable, and 3 = very valuable. An additional point of “valuable” was inserted into the scale for some surveys and as indicated in the Results to allow for variability across respondents. Attendees also retrospectively rated their confidence in facilitation skills, comparing their confidence before and after the FT workshop on a Likert-like scale with 1 = no confidence, 2 = low confidence, 3 = some confidence, and 4 = much confidence (Allen and Nimon, 2007). Evaluations contained open-ended questions regarding attendees’ intent to implement RMT, what additional resources might be needed for RMT implementation, and what improvements could be made to the FT workshop.

Following the final RMT session, facilitators completed an electronic survey to assess their experience. Specifically, facilitators were asked to rate the workshops overall on a five-point Likert scale from poor to excellent. They were also asked to rate the effectiveness of each session (1 = not effective, 2 = somewhat effective, 3 = effective, and 4 = very effective), whether they felt adequately prepared to facilitate RMT (1 = yes and 2 = no), and whether they would facilitate the training again (1 = yes and 2 = no). Open-ended questions also elicited responses on how to improve the mentor training and allowed facilitators to elaborate on their experiences.

Follow-up electronic surveys were administered to all attendees trained at the FT workshops between Fall 2010 and Summer 2014 to assess whether or not they had implemented RMT at their respective institutions. Surveys were originally sent to facilitators trained in phase I of the RCT at 1-and 2.5-yr post-FT. Surveys are now sent at a minimum of 3-mo post-FT and roughly every 6 mo to gather implementation updates.

To complement the analyses of the quantitative survey data and to inform ways to improve future FT workshops, we qualitatively analyzed responses to the open-ended questions in the surveys using a thematic analysis (Boyatzis, 1998). With regard to the RCT FT evaluations, the two question responses analyzed were: 1) “What could the planners do to improve this workshop?” and 2) “Please describe any additional support or resources you need in order to implement the mentor training curriculum effectively at your institution.” For the national scale-up FTs, the following three items were analyzed using the same method: 1) “What could the planners do to improve this workshop?,” 2) “Do you intend to implement research mentor training? If so, when do you intend to implement?,” and 3) “Please describe any additional support or resources you need in order to implement the mentor training curriculum effectively at your institution.” All responses to each item were compiled into one file to easily identify patterns and themes. Repeated statements or ideas resulted in coding of the responses to the themes identified. Themes with a frequency of four or greater are discussed in the Results. For the coding of all items, each attendee response was treated as one text segment for coding.

RMT Participants (Phases 1–3)

Facilitator effectiveness also was assessed from the perspective of the mentors trained in their respective RMT groups. Electronic surveys in which participating mentors rated their facilitators’ effectiveness in guiding discussion on a Likert-like scale (1 = very ineffective, 2 = ineffective, 3 = neither, 4 = effective, and 5 = very effective) were administered immediately after the final session of RMT. Mentors also indicated whether they would recommend the training to a colleague and whether attending RMT was a valuable use of their time (see Pfund et al., 2013, and https://mentoringresources.ictr.wisc.edu/TrainingEvaluation for measures used).

RESULTS

SIs

Between 2006 and 2010, 35 facilitators from 23 institutions attended the SI FT and subsequently facilitated a working group at one or more SI. Fifteen of the 35 facilitators (43%) responded to the 2009 FT survey. Eighty-seven percent of respondents indicated they found the most helpful aspects of the FT to be the detailed list of expected outcomes for each day of the SI, the “constructive and destructive” activity about individual behaviors in a group setting, and the daily facilitator debrief sessions during the SI. Other aspects that facilitators identified as most helpful included advice from experienced SI facilitators (73%), the “five stages of group development” activity, and consideration of the roles of a facilitator versus a teacher (68%). When asked what training materials or resources they would need to lead their own SI, 68% indicated that they already had all they needed but requested access to the documents plus a support network of experienced facilitators.

Participants in the SIs between 2006 and 2010 overwhelmingly noted the importance of having a facilitator work with them: 97% rated “having a trained facilitator for group work” as valuable (n = 159 respondents of 212 total participants). The SI participants reported that their facilitators were effective across a range of facilitator roles. Most notably, 99% of participants agreed the facilitator was a valuable resource. In addition, nearly all participants agreed their facilitators gave regular, useful feedback (95%); enhanced the group process (94%); kept the group on time and on task (94%); and were involved with the group just the right amount (92%). Positive comments about the facilitators dominated the open-ended responses (73%), such as “We could have been a dysfunctional group, but she kept us on task and cooperating” and “amazing job keeping us working together and toward our final product.”

The RCT (Phase 1)

Thirty-five individuals from 16 different institutions attended the Fall 2010 FT (Pfund et al., 2013; 2014c). Twenty-two attendees were female and 13 were male, with the majority being white (88%) faculty members (94%), including 44% MDs, 47% PhDs, and 8% other. The FT overall received a mean rating of 2.73, with all attendees indicating the training as either somewhat valuable (n = 8) or very valuable (n = 22, 86% responding). The three components rated most valuable were the introductory activity, overview of facilitation, and practice facilitation, with ratings of 2.77, 2.81 and 2.77, respectively. Of the 31 responses to the open-ended question “What could the planners do to improve this workshop?,” the most frequent themes were having more time to practice the facilitation, more modeling of facilitation techniques, and more focus on the content of the curriculum.

At the end of the 1.5-d training, attendees reported significant gains in their confidence across five major learning objectives for the FT (Table 5). Thirty-one surveys were completed with 20 attendee responses to the question, “What additional support do you still need?” Of the 20 responses, 45% noted they could not identify any additional support or resources to implement the training. A smaller number (n = 4) replied they would like to have packaged PDFs of the activity materials (e.g., copies of case studies) to use with each of the four mentor training sessions. The latter need has since been addressed with the development of the research mentoring website. Users can now freely download the full curriculum or select PDFs of specific chapters and activities.

Table 5.

RCT self-reported gains in confidence before and after FT^a

	Before			After
Learning objective	n	Mean	SD	n	Mean	SD	Difference*
Ability to establish constructive group dynamics	31	3.16	0.523	31	3.55	0.506	p = 0.001
Ability to facilitate a functional small group discussion	31	3.39	0.495	31	3.68	0.475	p = 0.003
Ability to draw out diverse perspectives in a group discussion	31	3.06	0.680	31	3.58	0.502	p < 0.001
Ability to deal with challenging behaviors within a small group	31	2.90	0.651	31	3.48	0.508	p < 0.001
Ability to implement the mentor training curriculum at your institution	31	2.39	0.761	31	3.55	0.506	p < 0.001

*Significant statistical difference in ratings before and after FT was determined by Wilcoxon signed-rank test.

^aMean retrospective confidence gains in facilitation skills before and after FT designed for trainers involved in the RCT (n = 31). Confidence was rated on a four-point Likert-like scale with 1 = no confidence, 2 = low confidence, 3 = some confidence, and 4 = much confidence.

Following the 1.5 d of FT, attendees returned to their home institutions to implement RMT with groups of six to 12 mentors. Although the order in which RMT curricular content was delivered was uniform, the length and spacing of the sessions varied. The 8 h of RMT were most commonly delivered as four 2-h sessions (n = 13); others offered two 4-h (n = 2) or one 2-h and two 3-h (n = 1) sessions. The sessions occurred over a period ranging from 2 d to 11 wk; the average was 5 wk (Pfund et al., 2014c). Following the final RMT session, facilitators completed a survey to assess their experiences with the facilitation process. Ninety-seven percent (n = 34) of the facilitators reported they felt adequately prepared to facilitate RMT, and 94% (n = 34) reported they would facilitate the training again. The two facilitators who reported they would not facilitate RMT again indicated time commitment and compensation as barriers to facilitation.

To evaluate the effectiveness of our trained facilitators, mentors who engaged in their RMT sessions across the 16 sites rated both the effectiveness of the training overall and the effectiveness of their facilitators. We previously reported that >88% of mentors who participated in RMT at these 16 sites found the training a valuable use of time (n = 112) and would recommend the session to a colleague (n = 114). They also found their facilitators effective (Pfund et al., 2013), with 96% (n = 123) rating their facilitators as effective or very effective.

Preparing for Scale-Up (Phase 2)

FT workshops including the components in Table 3 were conducted at five institutions between August 2012 and May 2013 (see Acknowledgments for full listing). A total of 48 attendees were trained with an average of 10 per site. Importantly, the sites differed in which of the EM-based curricula they used for the FT as follows: mentor training for clinical and translational researchers (one), biomedical researchers (two), community-engaged researchers (one), and clinical and behavioral researchers (one) (Pfund et al., 2012a, 2014a; Asquith et al., 2014; House et al., 2014; https://mentoringresources.ictr.wisc.edu).

The workshops overall received a mean rating of 4.51 on a five-point Likert scale from poor to excellent with all attendees indicating the training as either good (n = 4), very good (n = 14), or excellent (n = 27), with 94% reporting. For one site, a 2-h mini-RMT was modeled, consisting of the introductory session to the curriculum and activities from the Maintaining Effective Communication session, which was integrated to help effectively train facilitators who had not yet experienced RMT themselves. At the site in which the mini-RMT workshop was included, all attendees found it either somewhat valuable (n = 1), valuable (n = 1), or very valuable (n = 5), with 100% reporting.

Of the 48 trained, 13 launched mentor training at their respective sites following FT, resulting in trainings at all five sites. All institutions reported plans for future implementation with the other trained attendees. Although the order in which the curricular content was delivered was uniform, flexibility in the length and spacing of the sessions was essential to accommodate schedules and maximize attendance. The 8 h were most commonly delivered as four 2-h sessions (n = 4); another offered one 2-h and two 3-h sessions (n = 1). The sessions occurred over a period ranging from 5 to 10 wk. Following implementation, 80% of those reporting indicated they felt adequately prepared to facilitate RMT (n = 8), and 100% of the facilitators reported they would facilitate again (n = 10, five sites). Evaluation of their knowledge and skill gains following the FT and implementation showed gains in confidence in both their general facilitation skills and in their ability to implement the curriculum (unpublished data). Mentors (n = 63 across five institutions) who participated in the RMT sessions led by these trained facilitators rated the effectiveness of the facilitators as high, with 63% and 32% reporting their facilitators were very effective (n = 40) and effective (n = 20), respectively.

National Scale-Up (Phase 3)

The location, participation numbers, and demographics of the 112 FT attendees are shown in Table 4. Attendees represented a wide range of career stages and paths, with 22% full professors, 15% associate professors, 14% assistant professors, and 29% other (i.e., scientists, postdoctoral trainees, instructors, graduate students, deans, department chairs, directors); 30% also noted their single or dual role in administration. Additionally, the attendees represent 66 academic institutions with 83% of attendees (n = 88) working in the life sciences. As with previous FT, attendees rated each component of the workshop (Table 6) and their confidence, retrospectively, at the end of the FT (Table 7); all components resulted in significant gains (p < 0.01).

Table 6.

National scale-up satisfaction with FT workshop components^a

Workshop component	Mean	SD
Model of RMT session	3.63	0.53
Overview of facilitation	3.64	0.57
Overview of the curricula	3.53	0.59
Practice facilitation sessions and debrief	3.66	0.55
Implementing the curriculum on your campus	3.44	0.66
Evaluation measures of RMT	3.51	0.59

^aWorkshop components were rated on a Likert-like scale with 1 = not at all valuable, 2 = somewhat valuable, 3 = valuable, and 4 = very valuable. Means and SDs are reported for 83 respondents.

Table 7.

National scale-up self-reported confidence gains before and after FT workshop^a

	Before			After
Learning objective	N	Mean	SD	N	Mean	SD	Difference*
To utilize the available RMT curricula and supporting resources	85	2.35	0.812	85	3.62	0.511	p < 0.001
To describe evidence to support the effectiveness RMT	85	2.04	0.906	85	3.48	0.590	p < 0.001
To facilitate RMT using the process-based approach	85	2.21	0.832	85	3.41	0.563	p < 0.001
To recruit mentors to participate in training	83	2.31	0.869	83	3.17	0.640	p < 0.001
To implement RMT at your home institution	85	2.09	0.854	85	3.29	0.737	p < 0.001
To use metrics and tools to assess the effectiveness and impact of RMT	84	2.04	0.813	84	3.29	0.632	p < 0.001

*Significant statistical difference in ratings before and after FT was determined by Wilcoxon signed-rank test.

^aMean confidence gains in facilitation skills among attendees retrospectively after FT at four national Facilitating Entering Mentoring workshops (n = 85). Self-confidence was measured using a Likert-like scale with 1 = no confidence, 2 = low confidence, 3 = some confidence, and 4 = much confidence.

Of the 62 respondents to the question, “What could the planners do to improve this workshop?,” 24% wanted more time for the workshop. Of those attendees who specified time for a particular activity in their response, four specifically mentioned more time to practice facilitation. Eleven percent suggested more clarity regarding the workshop goals, content, or workshop instructions; four attendees specifically recommended more preworkshop preparatory assignments.

Attendees were also asked to describe any additional support or resources they needed in order to implement the mentor training curriculum effectively at their institutions. Of the 49 respondents to this question, 41% indicated institutional support would be essential to their being able to implement mentor training. Multiple components of support were identified, with five attendees specifically mentioning they would need to identify a cofacilitator at their institutions. Other aspects of institutional support included the buy-in of their deans or other institutional leaders, financial resources, and staff support. Five attendees specifically mentioned the importance of providing incentives to recruit mentors to participate in the training, as expressed in one response: “The faculty mentors need tangible value for mentoring, time or money—really true value.” Despite these concerns, 85% (n = 76) reported intent to implement RMT at their institutions in some form, and of those, 13% reported they would implement within the next year.

All of the attendees in FT discussed in this paper have since been surveyed regarding their planned or actual implementations of RMT at their home institutions. Because the attendees have engaged in FT across a 3.5-yr period, they have had differing amounts of time for implementation. Of the 195 attendees who participated in FT since Fall 2010, 44% reported scheduled and actual implementations (58% reporting). Survey responses indicate that RMT in some form has been implemented at 39 different institutions, collectively training more than 500 mentors. Of those who did not implement training, facilitators at 10 additional institutions planned to implement in the next year. A compilation of the implementation data are shown in Table 8, which includes data on the full 8-h and shortened forms of implementation in the last row. Examples of shortened implementation include 1- to 3-h workshops focusing on one or more competencies and semester-long series of workshops using one or more activities or readings from multiple competencies but not implementing entire sessions. Institutions implementing the full versions of curricula varied from a onetime implementation to implementing several times across 2 yr. Actual curricula implemented varied across the cohorts; however, the most commonly curriculum used was Mentor Training for Clinical and Translational Researchers, followed by Mentor Training for Clinical and Behavioral Researchers, Mentor Training for Biomedical Researchers, Mentor Training for Community Engaged Researchers, and, finally, Mentor Training for Undergraduate Researchers (https://mentoringresources.ictr.wisc.edu).

Table 8.

Implementation data of trained facilitators (phases 1–3)

National impact by institution
FT cohort and impact	RCT FT (phase 1)	Prepping for scale-up FTs (phase 2)	National scale-up FTs (phase 3)	Total
Number of attendees (by number of institution and attendees)	16 institutions; 35 attendees	6 institutions; 48 attendees	66 institutions; 112 attendees	88 institutions; 195 attendees
Number of institutions with no plans to implementa	0	0	2	2
Number of institutions planning to implement but nothing is scheduled a	0	0	10	10
Number of institutions with scheduled or already implemented workshopsa (number of independent full 8-h RMT; number of shorter workshops)b	16 (22; 7)	6 (17; 1)	17 (11; 14)	39 (50; 22)

^aPlanned or actual implementation of RMT as reported by attendees via survey or email exchange. Data collected from 100% of institutions who participated in the RCT and adapted FTs; 44% of institutions in national meeting FTs.

^bSome institutions have implemented multiple times.

DISCUSSION

EM was developed to make the process of learning to be a good research mentor more effective and efficient. Studies indicate that a process-based approach is effective in improving mentors’ ability to 1) communicate, 2) align expectations, 3) assess understanding, 4) address equity and inclusion, 5) foster independence, and 6) promote professional development—skills that are needed by mentors, research or otherwise. In this paper, we report on the development, implementation, and evaluation of train-the-trainer workshops designed to support widespread dissemination and implementation of the EM series curricula. The need for methods supporting implementation of evidence-based approaches to mentoring is timely, as federal agencies have called on training programs to include postdoctoral mentoring plans, evidence-based mentoring, and individual development plans into their program activities (www.nsf.gov/pubs/policydocs/pappguide/nsf10_1/gpg_2.jsp#fn28; http://publications.nigms.nih.gov/trainingstrategicplan/theme2.htm; http://grants.nih.gov/grants/guide/notice-files/NOT-OD-13-093.html). Most recently, the NIH funded the National Research Mentoring Network to focus on mentoring as a means to increase the numbers of individuals from diverse backgrounds in biomedical sciences (http://commonfund.nih.gov/diversity/Initiatives).

Despite the need for evidence-based approaches to mentoring, there are many barriers to widespread dissemination and implementation, even among the most effective interventions. These include limited time and resources, a lack of rewards, and lack of expertise (Henderson and Dancy, 2007; AAAS, 2011; D’Avanzo, 2013). However, a lack of confidence to implement approaches such as RMT is also a significant barrier to widespread dissemination and implementation (Hutchinson and Huberman, 1994; Henderson et al., 2011). These barriers have been explored at length by those engaged in dissemination and implementation research (for an overview, see Brownson et al., 2012) and teaching professional development. For example, the faculty-reported barriers to the uptake of active learning in their STEM classrooms include lack of experience, confidence, time, institutional rewards, financial rewards, and classroom structure as well as misalignment with their disciplinary expectations and their identity as faculty members (Henderson and Dancy, 2007; Henderson et al., 2011; Brownell and Tanner, 2012).

Interestingly, RMT as an intervention has had some advantages in overcoming some of these barriers, most specifically in the area of lack of experience with the content. First, facilitators of RMT are able to draw upon their experiences as research mentors and/or mentees, thus addressing the issue of lack of experience with the content. In the FT described here, it is stressed that facilitators are not expected to be mentoring content experts, but rather, it is more important that they be effective facilitators, guiding participants through the learning objectives and activities of the process-based curriculum so that participants may best learn from one another’s experiences and expertise. Importantly, the curriculum is not intended to have “all the answers”; rather, it promotes learning that can only be gleaned through conversation with others about their respective mentoring experiences and challenges. Second, faculty members may find that efforts to improve their mentoring are aligned with their overall professional identity as researchers and expectations in their discipline(s) to train the next generation of researchers. Thus, they may be more likely to engage in training or be facilitators of RMT to improve their own ability and that of their peers in fulfilling this role effectively.

Despite a reduction in two of the noted barriers, other barriers to widespread dissemination and implementation include a lack of confidence in facilitation skills, a need for easy-to-find resources, and doubt that effective mentoring will be valued by their institutions. In this paper, we report the process of developing an FT that is effective in raising the confidence of those interested in implementing RMT. This multifaceted FT was based on a successful FT developed for the SIs. It was designed to give attendees a chance to experience RMT, become familiar with the curriculum, practice facilitation, design an implementation plan, and connect to resources to support implementation (see Table 6 for ratings of these training components). Formative data from each iteration of FT informed the final length, structure, and content of the FT, with room for flexibility and contextualization.

Our FT has been implemented with nearly 200 people, and the data support its effectiveness in building confidence and promoting actual and planned implementation. We note that the data are self-reported by FT attendees and acknowledge the limitations of those data, as individuals tend to overestimate their abilities on intellectual tasks (Kruger and Dunning, 1999; Zell and Krizan, 2014). However, we also note the high ratings of the mentors who subsequently took RMT from these trained facilitators. Moreover, a fair percentage of those who have attended FT went on to implement RMT, which is the best measure of confidence gain.

Despite evidence supporting the effectiveness and national impact of the FT described here, attendees still reported needing support and resources, with almost half indicating that institutional support would be essential. Examples of such support included the need to identify a cofacilitator at their institutions, buy-in from their deans or other institutional leaders, financial resources, and staff support. Despite these concerns, 85% still reported intent to implement RMT at their institutions in various ways, and of those, 13% reported they would implement within the next year. We plan to follow up with these attendees to learn about the success of implementation efforts as well as to better understand the barriers to launching and sustaining RMT.

The larger goal of research mentoring programs is to change institutional culture to explicitly value mentoring. This program aims to impact that objective by increasing the number of trained mentors who recommend the training to their colleagues, by establishing that RMT is a valuable use of time and results in skill gains, and, additionally, by building capacity to disseminate training through trained, empowered facilitators. Collectively, these efforts may offer tangible reasons for institutions to invest in the infrastructure to train mentors, reward good mentors in their promotion criteria, and provide a model for dissemination and implementation of professional development (Steiner, 2014).

Future Directions

Our FT is an important step toward realizing broad dissemination and implementation of evidence-based RMT. We recognize that challenges still exist and need to be addressed. Although the FT has been successful in diminishing the confidence barrier, one of the most pressing challenges is the need to train cohorts of diverse “master trainers” who can lead FT across the country. We plan to train 12 such master trainers in 2015 to build national capacity. We have also partnered with core staff members at the American Physical Society (APS) and the American Society for Microbiology (ASM), thereby building sustained capacity within their organizations to offer RMT at their annual meetings. The APS launched its first RMT session in June 2014; ASM will launch its workshop in Spring 2015. We hope to engage additional organizations. Finally, we recognize the challenges of quality control, variation, and dosage as the RMT is implemented broadly. We have developed validated metrics to measure the impact of RMT and have created a centralized data-collection Web portal for those using these assessment tools; work is underway for a common tool for use with the mentors of undergraduates (Fleming et al., 2012, 2013; Byars-Winston et al., personal communication). These centralized data hubs will provide the opportunity for collecting a national data set to test RMT in diverse contexts.