Promoting STEM Trainee Research Self-Efficacy: A Mentor Training Intervention

Abstract

Self-efficacy, or an individual’s belief in his or her ability to successfully complete a given task, is a significant predictor of outcome expectations, interests, career aspirations, and persistence among undergraduate students in STEM fields. Despite the central role that efficacy beliefs play in STEM career choice and persistence, few training opportunities have used theoretical models like social cognitive career theory (SCCT) to help mentors learn how to support trainee research self-efficacy. To address this gap, a mentor training intervention was developed to translate the research and theory behind self-efficacy and into the practice of mentoring in STEM. Evaluation data from mentors who participated in (N = 166) and facilitators who implemented (N = 7) a training based on SCCT were used to assess the effectiveness of such an intervention. Mentors reported high satisfaction and significant retrospective skill gains related to promoting trainee research self-efficacy. Mentors also reported changes that they intended to make in their mentoring. Facilitators with varying levels of familiarity with self-efficacy were able to implement the module effectively and provided additional suggestions for further improvement of the training.

Less than 40% of undergraduate students who intend to pursue a science, technology, engineering, or mathematics (STEM) major actually complete a STEM degree (PCAST 2012). Some individuals leave because they find their passion in another discipline; however, others may lack the confidence in their capability to succeed. Examining this lack of confidence, or self-efficacy (Bandura, 1997) is an important step towards increasing persistence in STEM. Self-efficacy is a powerful predictor of persistence across many academic and non-academic domains (Bandura, 1997) and has been shown to be a significant predictor of success in STEM.

Previous research has revealed a relationship between self-efficacy and outcome expectations (i.e., beliefs about the consequences of a given action), interest, career aspirations, and persistence among undergraduate students in STEM, particularly for trainees from historically underrepresented minority groups (URM; Adedokun et al. 2013; Byars-Winston et al. 2010; Lent et al. 2005). Consequently, promoting trainee self-efficacy may increase interest and persistence in STEM career pathways.

Mentored undergraduate research experiences provide the opportunity for trainees to gain confidence in their ability to do research (i.e., research self-efficacy) and persist in STEM fields (Laursen et al., 2010). Research mentors are well-positioned to influence undergraduate student research self-efficacy, yet few mentor training interventions have used this framework to assist mentors in promoting trainee research self-efficacy. In this paper, we report on the effectiveness of a training intervention designed to help research mentors promote their trainees’ research self-efficacy. Using social cognitive career theory (SCCT; Lent, Brown & Hackett, 1994), we developed a mentor training intervention that introduced mentors to the concept of self-efficacy and four ways to promote trainee research self-efficacy (i.e., sources of self-efficacy) within the context of a mentored research experience.

Sources of Self-Efficacy

In order to promote research self-efficacy, mentors must learn about the different types of experiences that can raise or lower their trainee’s self-efficacy. Self-efficacy is informed by mastery experiences, vicarious experiences, social persuasion, and an individual’s physiological state (Bandura, 1997). Each source has the potential to raise or lower trainees’ research self-efficacy. Mastery experiences, or performance accomplishments, refer to past successes or failures that shape individuals’ views of their abilities. Vicarious experiences refer to the activities or achievements of others that can influence an individual indirectly through observation or role modeling. Social persuasions refer to messages received from others about one’s effort or ability. Physiological state refers to negative or positive physiological or affective arousals that individuals experience in different situations.

The four sources of self-efficacy have been studied in many academic subject areas (e.g., science, mathematics, and engineering). Across subject areas, mastery experiences have been shown to be the strongest contributors to individuals’ self-efficacy (Bandura 1997; Usher & Pajares 2008). However, each of the four sources can contribute independently or conjointly to efficacy beliefs. In fact, prior research suggests that STEM students of different racial, ethnic, and gender groups may differentially weight the sources of self-efficacy in gauging their efficacy beliefs. Social persuasion (i.e., encouragement from teachers), mastery experience (i.e., conducting research), and vicarious experience (i.e., social comparison and competition) have each been identified as positive contributors to URM students’ self-efficacy and science identity (Hurtado et al. 2009). Usher and Pajares’ (2008) literature review on the sources of self-efficacy reported several studies citing differences in how the four sources contribute to mathematics self-efficacy among different racial groups. More recently, Byars-Winston et al. (2016) found significant differences between Black/African American and Hispanic/Latino(a) research trainees’ responses on measures of vicarious learning, social persuasion, and affective emotional arousal (i.e., physiological state). Gender may also influence how individuals weight the sources of self-efficacy. Findings from a new meta-analytic study revealed a stronger source-to-self-efficacy link for men than for women in STEM disciplines (Byars-Winston, Diestelmann, Savoy, & Hoyt, 2017). Other researchers have noted vicarious experiences and social persuasions may be more salient for successful women in the sciences (Sawtelle et al. 2012; Zeldin & Pajares 2000), while mastery experiences may be important contributors to men’s self-beliefs (Sawtelle et al. 2012; Zeldin et al. 2008). Together, these findings suggest that individuals differentially experience and weight the sources of self-efficacy in a variety of academic contexts (Bandura, 1997). As such, mentors and trainees may differentially experience and weight the sources of research self-efficacy within the context of their mentoring relationships. Equipping mentors to recognize and support each of the four sources of self-efficacy may increase their ability to target and positively affect the research self-efficacy central to academic success and career persistence.

The Need for a Theoretically-Informed Mentor Training Intervention

Mentors can play an important role in promoting trainee research self-efficacy and persistence in STEM (Carpi et al. 2016; Feldman et al. 2010; Garman et al. 2001; Ghee et al. 2016) and researchers have offered strategies on how educators can support the development of students’ self-efficacy (e.g., Artino 2012; Margolis & McCabe 2006). Despite the evidence establishing the central role that self-efficacy plays in STEM academic persistence and career choice (Pajares, 1996, Byars-Winston et al. 2015) and the strategies suggested in these reports, few training opportunities have been developed for mentors to learn how to support research self-efficacy development in their trainees. Several programs and interventions designed to retain URM scientists have been developed (e.g., Ghee et al. 2016; Matsui et al. 2003), yet few have focused on training research mentors, and even fewer have based their professional development curricula on a theoretical model (Byars-Winston et al. 2011). Ironically, many STEM retention efforts target self-efficacy as an outcome variable of interest, but the interventions used in these efforts are not often grounded in or informed by SCCT. Without this theoretical framework, researchers and interveners (i.e., faculty and program staff) may not be able to identify the underlying mechanisms of the intervention that give rise to trainees’ self-efficacy. With this in mind, we developed a theoretically informed mentor training module to educate research mentors in STEM disciplines about research self-efficacy.

Training Mentors to Support Trainee Research Self-Efficacy

This module is designed to be incorporated into the Entering Mentoring (EM) curriculum (Handelsman et al. 2005, Pfund et al. 2015). Entering Mentoring (EM) is a mentor training intervention that uses a process and evidence-based approach to facilitate the development of research mentoring competencies. It incorporates a variety of activities, case studies, and links to resources. The curricular materials are organized as modules with participant and facilitator materials and each module addresses one mentoring competency (e.g., Aligning Expectations; Effective Communication; Fostering Independence). The EM curriculum has been shown to benefit mentors by increasing the likelihood of discussing expectations with trainees, considering the role of diversity in mentoring relationships, and increasing self-reported gains in skill compared to mentors who do not participate in training (Pfund et al. 2006; Pfund et al. 2014). This new intervention broadens the scope of the EM curriculum to include Promoting Trainee Research Self-Efficacy.

The Intervention: Promoting Trainee Research Self-Efficacy

In this module, mentors explore the concept of self-efficacy and how the four sources of self-efficacy can be supported by mentors in the context of their research experiences. The module was developed by a team of four researchers (Byars-Winston et al. 2013) and was based on pre-service and in-service teacher efficacy research (e.g., Ross, 1998) and the EM research mentor training curriculum (Handelsman et al. 2005; Pfund et al. 2015).

A list of learning objectives and a description of the activities included in the research self-efficacy module are provided in Table 1. In brief, the module familiarizes mentors with the four sources of self-efficacy and provides them with a “self-efficacy tool box” adapted from materials developed by Mathisen and Bronnick (2009), including practical techniques to support each of the four sources of trainees’ research self-efficacy. Mentors are provided opportunities to: (a) reflect upon and share their own research training experiences; (b) talk about positive research attributes and skills and practice their efficacy-building strategies through role play activities; (c) practice skills learned in the module via case studies and shared experiences; and (d) develop their confidence to build their trainees’ research self-efficacy. Mentors are introduced to each of the four sources of self-efficacy via different activities. For example, in the introductory activity “Anatomy of a Successful Research Experience,” mentors are invited to reflect on and share their own significant research experiences and identify which of the four sources that contributed to those experiences. Role-playing and practice through multiple case studies and scenarios permit mentors to experiment with promoting trainee research self-efficacy using the different sources of self-efficacy. For example, in the case study “William: To be of Not to Be in Research?” mentors explore how opportunities for trainees to experience success (i.e., mastery experiences), verbal feedback (i.e., social persuasion), modeling (i.e., vicarious experience), and mitigating anxiety (i.e., physiological state) can each influence the research self-efficacy of a trainee who is considering pursuit of a graduate degree in STEM.

Table 1.

Learning objectives for self-efficacy module.

Learning objectives Mentors will:	Description of core activities
1. Define self-efficacy and its four sources	Introduce the four sources of self-efficacy through discussion of critical incidents supporting mentors’ research self-efficacy.
2. Articulate their role in fostering mentees’ research self-efficacy	Identify strengths that mentors have observed in their mentees. Identify a skill or goal mentors want their trainees to obtain and identify ways in which the mentor can support the trainee using the four sources of self-efficacy.
3. Identify signs of self-efficacy in relation to research related tasks	Case Study, “William: To Be or Not to Be in Research?” and/or “Family Ties.” Discussion of how self-efficacy and its sources operate in different contexts.
4. Practice strategies for building mentees’ self-efficacy in research	Role Play exercise: Mentors play the role of the graduate student mentor in “The Case of the Slob” to practice strategies to build the trainee’s self-efficacy.

Note: The facilitator guide and participant materials for the self-efficacy mentor training module are available in the appendix for this manuscript.

Pilot tests of early versions of the module were conducted with the National Research Mentoring Network’s (NRMN) Master Facilitators (i.e., facilitators with previous experience implementing EM). Their feedback about ease of delivery and clarity of module content along with feedback from module participants is summarized in the changes described in Table 2 (IRB protocol # 2015–1330). This feedback was used to revise the ordering and presentation of content in the module to create version 4, the version presented and tested in this paper (see Appendix for a complete copy of the final training module).

Table 2.

Revision history of the self-efficacy mentor training module.

Version of curriculum	Date changes implemented	Changes reflected in this version
Version 1	N/A	N/A
Version 2	November 2015	• Changed order of activities • Modified case study of “William: To Be or Not To Be in Research?” for clarity • Revised handouts
Version 3	February 2016	• Added an alternate case study, “Family Ties” to more directly address the intersection of cultural identity and self-efficacy
Version 4	April 2016	• Added discussion questions for “Family Ties” case study • Revised sources of self-efficacy handout

Evaluating the Intervention

In this evaluation research paper, the self-efficacy mentor training module and the evaluation process used to examine the effectiveness of it are described. Evaluation data collected from 11 implementations of the module facilitated by 17 facilitators at 7 sites across the country were used to investigate several evaluation questions focused around three of Kirkpatrick’s (2016) levels of evaluation: reaction, learning, and behavior. Reaction examines the level of satisfaction with the training as reported by participants. Learning evaluates the knowledge or skills gained as a result of the training. Behavior considers the impact of the training on participants’ behaviors and the extent to which they are applying what they have learned. In this study, we evaluated participants’ intended behaviors. With respect to each of these levels, we were interested in answering the following questions:

1. Reaction: How likely were mentors to recommend this training to other mentors? Which activities were perceived of as most and least engaging by facilitators? How effective were the facilitators in implementing this training? How prepared did facilitators feel to implement this module?

2. Learning: What were the self-reported retrospective skill gains of mentors in their ability to assess and promote trainee’s research self-efficacy? How did these skill gains differ by implementation, by implementation type, or by prior mentoring experience?

3. Intended Behavior: How did participation in the training module affect the mentor’s perceptions of the quality of their own mentoring? How many mentors were likely to make changes in their mentoring practice as a result of attending this workshop? What changes did mentors plan to make?

Methods

Implementations of the Module

The self-efficacy mentor training module was implemented in three different formats: as a standalone in-person workshop, as part of an in-person EM training (Pfund et al. 2006; Pfund et al. 2015), and as part of a synchronous online EM training (McDaniels et al. 2016). The module was implemented at four different institutions, two disciplinary conferences, and across several institutions online. The length of the training varied from a 90-minute training workshop that focused specifically on promoting trainee research self-efficacy to a 10-hour training seminar that used the self-efficacy module with other modules included in the EM curriculum. A full description of each implementation is provided in Table 3. The length of time spent on this training module ranged from 60 to 90 minutes across the 11 implementations. Table 4 describes the number of facilitators who implemented each of the activities presented in the self-efficacy module. All facilitators used the self-efficacy toolbox, which provides specific strategies that mentors can use to support each of the four sources of self-efficacy. Most facilitators also implemented the significant research moment activity, having mentors recall a specific moment in their own training or research experience that has made them more confident as a researcher. The remaining activities were used less consistently across implementations.

Table 3.

Training module implementation sites.

Implementation
Number	Description	Participants N	Responded to Survey N (%)	Consented to Research N
1	1.5-hour workshop at disciplinary conference.	4	4 (100%)	1
2	4-hour mentor training at a university in the Northeastern US.	17	5 (29%)	5
3	1.5-hour workshop at a university in the Midwestern US.	75	55 (73%)	52
4	8-hour mentor training at an HBCU in the Southeastern US.	27	20 (74%)	18
5	13 hour online synchronous training.	10	7 (70%)	7
6	10 hour training at a university in the Midwestern US.	15	10 (67%)	9
7	10 hour training at a university in the Midwestern US.	15	11 (73%)	11
8	10 hour training at a university in the Midwestern US.	11	10 (91%)	10
9	10 hour training a university in the Midwestern US.	16	5 (29%)	5
10	8 hour training at a university in the Southeastern US.	29	24 (83%)	24
11	1.5-hour workshop at disciplinary conference.	35	28 (80%)	24
Total		254	179 (69%)	166 (93%)

Note. All of the trainings that took place at a university in the Midwestern United States (US) were implemented at the same institution. Abbreviations: HBCU = Historically Black College or University.

Table 4.

Description of self-efficacy module activities implemented and facilitator ratings of most and least engaging activities.

Description of core activities/materials	Facilitators who implemented each activity n (%)	Facilitators who rated activity n (%):
		most engaging for participants	least engaging for participants
Sources of self-efficacy handout	7 (100%)	2 (29%)	1 (14%)
Significant research moment (critical incident relating to mentors’ research self-efficacy)	6 (86%)	---	2 (29%)
Identify a skill or goal mentors want their mentees to obtain and identify ways in which the mentor can support the mentee using the four sources of self-efficacy.	2 (29%)	1 (14%)	1 (14%)
Case Study and Discussion: “William: To Be or Not To Be in Research?”	2 (29%)	2 (29%)	---
Case study and discussion: “Family Ties”	4 (57%)	1 (14%)	1 (14%)
Role Play exercise: “The Case of the Slob”	1 (14%)	---	---

Changes and improvements to the module.

Facilitators reported any modifications or adaptations made to the self-efficacy module. One facilitator described a slight modification to how the significant research moment activity was introduced: “I have had better success asking for all participants to think of a time that they struggled in research and then talk about how they overcame this struggle (and how self-efficacy plays into their eventual success).” Another facilitator mentioned a new reading that could be used with the module in lieu of the current suggested reading. When asked about new facilitation techniques, one facilitator reported that s/he incorporated the self-efficacy module with the EM module Fostering Independence (Pfund et al. 2015) and said that this pairing allowed for participants to explore the relationship between building self-efficacy and independence as a researcher. Four facilitators offered suggestions on what they might do differently the next time that they implement this module. One suggested having scales available for those participants who are interested in formally assessing research self-efficacy. Another proposed that they would link the self-efficacy module with the fostering independence module. Additional suggestions included spending more time preparing and using a different case study in future implementations.

Participants

Participants were research mentors in STEM (N = 254) who attended one of 11 research mentor trainings in Spring, Summer or Fall 2016. Most mentors in our sample were mentoring trainees at various career stages, ranging from high school students to senior faculty. Seventy percent (70%) of the mentors in our sample were mentors of undergraduate trainees; 24% were mentors of trainees at other career stages, and 6% reported that they were not currently mentoring any trainees. Of the 254 mentors who participated in these trainings, 179 (69%) completed surveys and 166 (93%) consented for their surveys to be used in research. Demographic information for each implementation site is provided in Tables 5 and 6. The majority of participants identified themselves as faculty/instructors (43%), graduate students (34%), or postdoctoral researchers (13%). Sixty percent (60%) of mentors self-identified as White; 49% self-identified as Female.

Table 5.

Participant career stage information.

	Implementation Site
	1	2	3	4	5	6	7	8	9	10	11	Total
N	1	5	52	18	7	9	11	10	5	24	24	166
Career Stage n (%)
Graduate Student	---	3 (60%)	28 (54%)	---	---	8 (89%)	8 (73%)	6 (60%)	4 (80%)	---	---	57 (34%)
Postdoctoral Researcher	---	1 (20%)	9 (17%)	---	3 (43%)	---	2 (18%)	4 (40%)	1 (20%)	---	2 (7%)	22 (13%)
Faculty/ Instructor	1 (100%)	---	8 (15%)	18 (100%)	3 (43%)	---	1 (9%)	---	---	21 (88%)	20 (83%)	72 (43%)
Researcher/Scientist	---	---	3 (6%)	---	---	---	---	1 (10%)	---	4 (17%)	4 (17%)	12 (7%)
Administrator	---	1 (20%)	5 (10%)	1 (6%)	1 (14%)	---	---	---	---	1 (4%)	2 (8%)	11 (7%)
Other	---	---	---	1 (6%)	---	---	---	---	---	---	1 (4%)	2 (1%)
Not Reported	---	---	4 (8%)	---		1 (11%)	---	---	---	---	---	5 (3%)

Note: Participants were permitted to select multiple options for this item. As a result, percentages in columns may add up to more than 100%.

Table 6.

Participant demographic information.

	Implementation Site
	1	2	3	4	5	6	7	8	9	10	11	Total
N	1	5	52	18	7	9	11	10	5	24	24	166
Racial/Ethnic Identity n (%)
American Indian/ Alaska Native	---	---	---	---	---	---	---	1 (10%)	---	---	1 (4%)	2 (1%)
Asian	---	3 (60%)	5 (10%)	3 (17%)	---	---	3 (27%)	2 (20%)	1 (20%)	1 (4%)	3 (12%)	21 (13%)
Black/African American	---	---	1 (2%)	7 (39%)	1 (14%)	1 (11%)	---	2 (20%)	---	3 (13%)	4 (17%)	19 (11%)
Native Hawaiian/Pacific Islander	---	---	---	---	---	---	---	---	---	---	---	0 (0%)
White	1 (100%)	1 (20%)	37 (71%)	3 (17%)	6 (86%)	7 (78%)	7 (64%)	6 (60%)	3 (60%)	14 (58%)	15 (63%)	100 (60%)
Other	---	1 (20%)	3 (6%)	1 (6%)	---	---	---	---	---	---	1 (4%)	6 (4%)
Hispanic/ Latino(a)	1 (100%)	1 (20%)	4 (8%)	1 (6%)	---	1 (11%)	---	1 (10%)	---	1 (4%)	2 (8%)	12 (7%)
Not Reported	---	---	3 (6%)	2 (11%)	---	2 (22%)	1 (9%)	1 (10%)	1 (20%)	7 (29%)	1 (4%)	18 (11%)
Gender Identity n (%)
Male	1 (100%)	2 (40%)	15 (29%)	8 (44%)	3 (43%)	3 (33%)	4 (36%)	5 (50%)	---	6 (25%)	16 (67%)	63 (38%)
Female	---	3 (60%)	32 (62%)	6 (33%)	4 (57%)	4 (44%)	6 (55%)	4 (40%)	4 (80%)	12 (50%)	7 (29%)	82 (49%)
Transgender	---	---	---	---	---	1 (11%)	---	---	---	---	---	1 (<1%)
Intersex	---	---	---	---	---	---	---	---	---	---	---	0 (0%)
Other	---	---	1 (2%)	---	---	---	---	---	---	---	---	1 (<1%)
Not Reported	---	---	4 (8%)	4 (22%)	---	2 (22%)	1 (9%)	1 (10%)	1 (20%)	6 (25%)	1 (4%)	20 (12%)

Note: Participants were permitted to select multiple options for these items. As a result, percentages in columns may add up to more than 100%.

Facilitators who led the implementation of the module provided feedback on the module through electronic surveys. Of the 17 facilitators who facilitated or co-facilitated the 11 implementations presented in this paper, 7 facilitators (41%) from 6 different implementations (i.e., Implementations 2, 4, 7, 8, 11, and 12) responded to the invitation to complete the survey and consented for their surveys to be used in research.

Evaluation of self-efficacy training module

The module was evaluated using a survey that included both Likert-type and openended question prompts (IRB protocol # 2016–0458)¹. Surveys were administered to mentors immediately following the training, and were either completed on paper or online, depending upon the preferences of facilitators and the training sites. In the case of online surveys, mentors were provided a link to the survey immediately following the training and given up to 4 weeks to complete their evaluation. The measures and procedures used for each approach are described below.

To measure participants’ satisfaction with the module (i.e., reaction), participants were asked “How likely are you to recommend this workshop to other mentors?” with response options ranging from 1 (very unlikely) to 5 (very likely). To assess reaction from the perspective of the facilitator, facilitators were asked to report from their perspective which activities were the most and least engaging for participants and their efficacy in engaging participants with the training module. They were also asked to report on their prior knowledge of self-efficacy and how well prepared they felt to implement the module based on the implementation instructions provided.

To measure learning, participants retrospectively assessed their skill level before and after training on a scale from 1 (not at all) to 7 (extremely) on five items related to building and assessing trainees’ research self-efficacy (see Table 7). To measure intended behavior, participants were asked whether they intended to make any changes in their mentoring as a result of attending the training and to describe any such changes.

Table 7.

Self-reported skill gains from self-efficacy module participants.

	Before M (SD)	Now M (SD)	M diff	t (df)	p	d z
Defining the sources of self-efficacy	3.17 (1.61)	4.92 (1.23)	1.75	12.75 (100)	<.001	1.26
Building mentees’ confidence for research	4.03 (1.39)	5.22 (1.09)	1.18	13.20 (152)	<.001	1.07
Employing strategies for building mentees’ self-efficacy in research	3.57 (1.35)	5.09 (1.07)	1.52	15.80 (152)	<.001	1.28
Assessing mentees confidence for research	3.75 (1.40)	4.80 (1.27)	1.05	11.42 (151)	<.001	0.93
Recognizing deficits in mentees’ confidence for research	3.62 (1.30)	4.81 (1.12)	1.19	11.98 (130)	<.001	1.05

Note: Skill gains for defining the sources of self-efficacy exclude Implementation 3. Due to a survey error, 23 cases were removed for analysis of “recognizing deficits in mentees’ confidence for research, resulting in a final sample size of 131. Possible responses ranged from 1 (not at all skilled) to 7 (extremely skilled). Cohen’s d_z is a measure of the effect size of the standardized mean difference. It was calculated using the formula provided by Rosenthal (1991): $d_z=t/\sqrt{n}$

Analyses

Reaction.

To examine mentors’ reaction to and satisfaction with the training module, the mean response to the question “How likely are you to recommend this workshop to other mentors?” was calculated. An independent samples t-test was conducted to determine if responses varied significantly by implementation type (i.e., between standalone self-efficacy workshops and research mentor training workshops including the self-efficacy module with EM modules). Due to the relatively low sample size of participants in online research mentor training (n = 7) this group was excluded from analysis. Levene’s test of equality of variances revealed that the homogeneity of variances assumption was tenable, p = .27. Means and frequency counts were calculated to describe facilitators’ assessment of most and least engaging module activities, self-reported efficacy in engaging participants with the training module, prior knowledge of self-efficacy, and level of preparation to implement the module.

Learning.

Dependent samples t-tests were run to determine if significant skill gains were reported, retrospectively, by mentors who attended the self-efficacy mentor training. To determine if skill gains reported varied significantly by implementation, by implementation type, or by prior mentoring experience, a skill gain variable was calculated for each of the five skill gain items by taking participants’ self-rated skill level before the workshop and subtracting it from their self-rated skill level after the workshop. Implementation 1 was removed from this analysis because it had only one participant who consented to research. For the item assessing gains in recognizing deficits in trainees’ confidence for research, participants from two implementations (n = 23) were removed prior to analysis due to a question error that resulted in unusable data. An Analysis of Variance (ANOVA) was conducted to determine if skill gains differed significantly across the 11 implementations. In all cases Levene’s Test of the Equality of Variances was not significant (ps > .05), indicating that the assumption of homogeneity of variances was tenable. Significant differences among implementations were found for defining sources of self-efficacy, F (9, 142) = 3.94, p < .001. Upon reviewing the post hoc results, it appeared that Implementation 3 had significantly higher skill gains in this area compared to other implementations. As a result, this implementation was removed prior to further analysis and considered separately. The subsequent ANOVA run without Implementation 3 was not significant, F (8, 91) = 1.92, p = .07, suggesting that the skill gains for the remaining implementations could be examined together. No other statistically significant differences in skill gains across implementations were detected for the remaining skill gain items.

Next, differences in skill gains by implementation type (i.e., standalone self-efficacy workshop compared with research mentor training workshops with self-efficacy module included) were examined using independent samples t tests. Finally, the relationship between the number of years of mentoring experience and skill gains reported as a result of attending the indicated research mentor training was explored using bivariate correlations. On the survey, mentors responded to the question, “How many years of experience do you have as a formal research mentor?” Mentors who responded with less than 1 or .5 were coded as .5 years. Responses that included “~” or “+” next to the number were reported as that number (i.e., ~5 was coded as 5; 15+ years was coded as 15).

Intended Behavior.

Self-reported changes in the overall quality of mentoring as well as intent to make any changes in mentoring as a result of attending the workshop were examined to assess intended changes in behavior. Descriptive statistics and dependent samples t-tests were conducted to examine whether significant changes in the quality of mentoring were reported by mentors. Prior to running this analysis, an ANOVA was conducted to determine if significant differences in reported changes in quality of mentoring were found across different implementations. The homogeneity assumption was found to be tenable based on the results of Levene’s test of equality of error variance (p = .78) The resulting F statistic was not significant, F (9, 137) = 1.59, p = .12, therefore all implementations were assessed together. Descriptive statistics were used to examine the percentage of participants who indicated that they intended to make changes because of the workshop and open-ended responses from mentors describing these changes were examined by one author to identify key themes. Responses to this question for participants in mentor trainings with multiple training modules (n = 66) were first assessed to determine whether they had any reference to self-efficacy. Those responses that included references to self-efficacy (14%, n = 9) were then reviewed for key themes relating to the four sources of self-efficacy. Responses from participants in a standalone self-efficacy workshop (n = 62) were also reviewed for key themes using this same approach.

Results

Reaction: Mentors are Highly Satisfied with the Self-Efficacy Training Intervention

The mean rating across all implementations was 4.27 on a scale of 1 (very unlikely) to 5 (very likely; SD = .841), with 81% of mentors indicating that they were either likely or very likely to recommend this workshop to other mentors. The independent samples t-test revealed no significant differences by implementation type, t(145) = 0.96, p = 0.34, indicating that the likelihood of participants to recommend the standalone workshop (n = 77; M = 4.19, SD = 0.71) as well as the module integrated into the longer research mentor training (n = 70; M = 4.31; SD= 0.81) were statistically equivalent.

Reaction: Facilitators Can Effectively Implement this Module

Facilitators were invited to report which activities they perceived to be most or least engaging for their participants (see Table 4). There was no clear consensus on a particular activity that was the most or least engaging for participants. The self-efficacy toolbox and a case study describing a mentor’s interaction with an undergraduate trainee considering graduate school (“William: To be or not to be in Research?”) were each rated as most engaging by two facilitators. Two facilitators reported that the critical research moment reflection and discussion was least engaging for their participants.

Most of the facilitators who responded to our survey (5 out of 7; 71%) rated their implementation as either “effective” or “very effective” at engaging their participants; two facilitators (29%) did not respond to this question.

Facilitators were asked to report how much they knew about self-efficacy prior to implementing this module: two facilitators (29%) responded “none”; three (43%) responded “some”; and two (29%) responded “a great deal.” Six of seven facilitators agreed that the facilitation guide was helpful in preparing to implement the module; one individual did not respond to this question. The mean response to the item “I felt well prepared to implement this module” was 6 (agree) on a scale ranging from 1 (strongly disagree) to 7 (strongly agree).

Learning: Mentors report significant skill gains across all implementation types and across the range of mentoring experience

Table 7 presents the results of the dependent samples t-tests examining skill gains across all training participants. Participants retrospectively reported significant skill gains across all five areas, p < .001, with effect sizes ranging from 0.93 to 1.28. In other words, participants reported retrospective skill levels after the training that ranged from .93 to 1.28 standard deviations above the skill level that they believed that they had prior to the training.

Skill gains were examined to determine if they differed significantly by implementation type or by participant level of research mentoring experience. To test the former, independent samples t-tests for each skill gain were conducted to determine if skill gains were significantly different between standalone workshops on self-efficacy compared to when the module was implemented as part of a longer research mentor training. Given the significant differences found for skill gains on defining the sources of self-efficacy for Implementation 3, this implementation was again excluded from analysis. No significant differences in skill gains were found by implementation type, indicating that self-reported skill gains were statistically comparable between implementations of the module as a standalone workshops and when it was implemented as part of research mentor training.

To determine whether skill gains were affected by the amount of research mentoring experience, we examined the correlation between the number of years of research mentoring experience and each self-reported skill gain to determine if there was a significant relationship. No significant relationships were found between years of mentoring experience and any of the five skill gains, ps > .05.

Intended Behavior: Mentors report intended changes in their mentoring as a result of training

Retrospective changes in overall quality of mentoring were assessed using a dependent samples t-test comparing mentors’ ratings of the quality of their mentoring before the training and after the training as assessed in the post-training survey (see Table 8). Overall, mentors retrospectively reported a significant increase by one standard deviation in the perceived quality of their mentoring after the workshop compared to the quality of their mentoring before the workshop, t (147) = 13.27, p < .001, d_z = 1.09.

Table 8.

Mentors’ self-reported retrospective assessment of the overall quality of their mentoring, before the workshop and after the workshop.

	Before M (SD)	After M (SD)	M diff	t (df)	p	dz
Quality of mentoring	4.36 (1.16)	5.40 (0.88)	1.04	13.27 (147)	< .001	1.09

Note: N = 148. Cohen’s d_z is a measure of the effect size of the standardized mean difference. It was calculated using the formula provided by Rosenthal (1991): $d_z=t/\sqrt{n}$. Participant responses could range from 1 (very low) to 7 (very high).

Mentors were asked to respond to a yes or no question indicating whether they planned to make any changes as a result of this workshop; 84% of participants indicated that they planned to make changes; 6% indicated that they did not intend to make any changes; and 10% did not respond to the question. Of those who planned to make changes as a result of the workshop, 128 individuals provided descriptions of those changes. Fourteen percent (14%) of EM training participants and 60% of standalone self-efficacy workshop participants directly referred to self-efficacy when discussing the specific changes they intended to make in their mentoring relationships; yielding a total of 45 responses that were further analyzed for themes relating to self-efficacy. Responses were coded based on references to any of the four sources of self-efficacy as well as references to self-efficacy in general (i.e., assessing self-efficacy or discussing self-efficacy with trainees). A full description of the codes assigned, along with example responses, frequencies, and percentages, appears in Table 9. Most participants (49%) provided examples of general strategies that they plan to implement to boost their trainees’ self-efficacy; comparatively fewer participants provided examples citing specific sources of self-efficacy, such as how they might support mastery experiences (24%), vicarious experiences (18%) provide social persuasions to their trainees (18%), and address emotional/physiological state (9%).

Table 9.

Participants’ open-ended responses to the prompt “please describe any changes you plan to make as a result of this workshop.”

Code	Frequency of codes assigned n (%)	Examples from participant responses
General	22 (49%)	• “I plan to include more discussions with my trainees to assess their self-efficacy” • “Trying to promote self-efficacy identifying differences between the mentee and myself.”
Mastery Experience	11 (24%)	• “Be more careful about identifying tasks to assign at the right level of challenge.” • “Assessing the difficulty of tasks, monitoring how students perceive their success”
Vicarious Experience	8 (18%)	• “Go out of my way to provide models for mentees” • “Recognizing the importance of vicarious experiences especially in relation to what individuals deem as valuable”
Social Persuasion	8 (18%)	• “I will revamp my annual review to include evaluation of self-efficacy, and I will work to give mentees that struggle with this more support. I’m also going to incorporate sending encouraging emails as a part of my mentoring. I can be too tough and demanding.” • “More evaluations for both the students and myself; recognizing to give more feedback regarding student’s contribution to the lab and skills that they already have”
Emotional/Physiological State	4 (9%)	• “Paying more attention to social and emotional states and what [mentees] might need” • “Listen more, be aware of emotional state”

Note: n = 45.

Discussion

Valantine and Collins (2015) recently highlighted several challenges to diversifying the biomedical workforce; among them was the need to identify psychological and social factors that can mitigate barriers to persistence and ultimately workforce diversity. Self-efficacy is one psychological factor that research suggests can begin to address this challenge (Chemers et al. 2011; Estrada et al. 2011, Byars-Winston et al. 2015, Byars-Winston et al. 2016). The evidence-based training module presented in this paper uses psychological theory to better inform mentor practices and improve the experiences of all trainees. This effective, easy to implement, and scalable intervention offers mentors the opportunity to learn strategies that support the four sources of self-efficacy.

Results from this evaluation study demonstrate that this novel intervention can help mentors be better equipped to promote their trainees’ confidence in their research capabilities. Given that sources of self-efficacy may be internalized and experienced differently across racial and ethnic groups and by gender (Byars-Winston et al. 2016, 2017; Usher & Pajares, 2008), we postulate that this mentor training intervention can be used to address the larger mission of diversifying the STEM workforce.

Overall, participants were satisfied with the self-efficacy intervention described here. The self-reported satisfaction in conjunction with self-reported skill gains support that this mentor training offers practical strategies to mentors on the specific things they can say or do to promote trainees’ research self-efficacy.

Several implementation venues were evaluated and the majority of participants were likely or very likely to recommend their mentor training to other mentors, regardless of the implementation venue. Similarly, retrospectively assessed skill gains in defining, recognizing, assessing, and promoting trainee research self-efficacy were significant across all implementations of the module. Self-reported skill gains did not differ significantly by implementation type or by mentoring experience. Together, these results suggest that mentors will likely report significant skill gains that are similar in their magnitude regardless of the format in which this self-efficacy module is delivered, and that both inexperienced and experienced mentors alike may benefit from participating in this module.

In addition to the statistical significance of this self-reported change in skill, the practical significance of change as indicated by the effect sizes suggests that mentors experience a substantial change of almost one standard deviation in their level of skill in understanding and promoting trainee research self-efficacy. Participants’ retrospective assessments of their skill gains in defining the sources of self-efficacy had the largest effect size. This result is further supported by feedback that we received from qualitative interviews conducted with mentors who experienced previous versions of this module, who noted that the information presented in the module provides them with a name for and a framework to describe what they did with their trainees (i.e., promoting self-efficacy).

Mentors who participated in the self-efficacy mentor training module reported a significant change in the overall quality of their mentoring. However, it should be noted that this question asked mentors to think retrospectively about the quality of their mentoring before the workshop and now, after the workshop; i.e., no time elapsed between the workshop and the survey that would have allowed mentors to interact with their trainees and assess real gains in the quality of their mentoring. A more accurate interpretation of these results may be mentors’ anticipations of the gains in the quality of their mentoring. Revising the wording of this question to ask about intentions to improve the quality of their mentoring after the workshop, or following up with mentors a few weeks after the workshop to assess changes in behaviors would extend our current understanding on the benefits of this module. The majority of participants indicated that they intended to make changes in their mentoring as a result of attending the mentor training. Of those mentors who indicated that they intended to make changes relating to promoting trainee self-efficacy, many expressed a desire to formally assess the self-efficacy of their trainees and discuss the topic of research self-efficacy with their trainees. Mentors who are interested in assessing the research self-efficacy of their trainees can utilize validated research self-efficacy instruments (e.g., Byars-Winston et al. 2016) or design their own measures of research self-efficacy using established, theoretically informed guidelines (Bandura, 2006). In addition, a training focused on assessing research self-efficacy may provide mentors already familiar with self-efficacy with a valuable tool to assess and track trainees’ research self-efficacy as they progress in their research experience.

All facilitators reported feeling prepared to implement the training module, despite varying levels of familiarity with the concept of self-efficacy. This suggests that the facilitator guide provides sufficient information and supporting materials to allow facilitators to implement the module with minimal preparation. Self-reported skill gains did differ by implementation venue on one item, with participants in Implementation 3 reporting significantly higher skill gains in defining the sources of self-efficacy compared to other Implementations. This was likely a result of the backgrounds of the Implementation 3 facilitators, who were SCCT researchers, rather than natural scientists. They may have focused more on the theoretical underpinnings of the module. Researchers should further investigate the benefits of additional facilitator training in SCCT on participant learning gains, as it may be beneficial to provide background information on SCCT to facilitators with different disciplinary backgrounds. However, it is equally important to note that all participants experienced significant gains across implementations, and that skill gains as measured by the other four items did not vary significantly by implementation. An important finding from our evaluation study is that facilitators with limited or no background in psychology or SCCT can (and do) successfully implement this module, as evidenced by participants’ self-reported satisfaction and perceived skill gains. Similar to the EM curriculum, a facilitator can use the facilitation guide to implement this self-efficacy mentor training with minimal preparation. The facilitator guide that accompanies this module was designed with the intent that any researcher or administrator with an interest in training mentors to more effectively promote trainee research self-efficacy can effectively deliver this module at his or her institution.

The ease of the module’s implementation makes it appealing to both mentors and facilitators, resulting in a curriculum that has the potential to impact a wide range of mentors and trainees. Facilitators reported using different activities across the six implementations surveyed for this study. Taken together with our findings of similar skill gains across implementations, this suggests that the activities available in the self-efficacy module can be mixed and matched to suit the needs of participants without significant losses in self-reported skill gains. A few facilitators also described the benefits of pairing the self-efficacy module with another EM mentor training module focused on fostering independence in trainees; further research should investigate these hypotheses more systematically to explore whether different combinations of EM curriculum modules offer unique benefits. The concepts introduced in this training module also have the potential to benefit individuals serving as mentors in other research contexts, such as course-based undergraduate research experiences (CURES).

We acknowledge several limitations to this study. The data presented do not permit us to investigate whether what mentors learned actually improved the self-efficacy of their trainees. However, interviews with mentors conducted after earlier versions of this module were implemented preliminarily suggest that the concepts and techniques discussed in the workshop are recalled and implemented by mentors several months after the workshop. Many mentors noted that they were more aware of how their trainees experienced the sources of self-efficacy as a result of participating in this mentor training module. For example, one mentor expressed that she became more aware of the messages that she was sending her trainee:

We were recruiting families to participate in our studies, and just the fact of having to go up to a stranger and asking them and telling them about the work they’re doing can be very difficult for some students. I think there was some frustration on my part [when the trainee] wasn’t very comfortable doing it [because] it affects how many participants we get. But you know, there’s multiple ways of me expressing my frustration. I could just say look you need to suck it up, and like I know you’re mad at this and you’re just going to have to do it because this is what we have to do. And I think in the past I have expressed frustrations that way without realizing that that could actually set us back even more because [of the impact] my words have on the confidence of the student.

Such responses suggest that mentors’ heightened awareness of the ways in which they can influence trainee self-efficacy will ultimately improve the self-efficacy of trainees who are supervised by trained mentors. This hypothesis can be tested in future work. An expanded study measuring trainee gains in research self-efficacy among trained and untrained mentors could lend further support to the conclusions reached in this paper. Measuring trainee research self-efficacy and its sources (e.g., Byars-Winston et al. 2016) in trainees of trained mentors would also yield further insight into the impacts of this training intervention.

The results presented in this paper suggest that theoretically informed, evidenced-based interventions build the capacity of research mentors to increase their perceived mentoring effectiveness. More specifically, an intervention designed to inform mentors about self-efficacy and its sources aids in the translation of motivation theories like SCCT into mentoring practices that have the potential to increase persistence in STEM. Pilot testing of the “Promoting Trainee Research Self-Efficacy” module has concluded and the module is available for use as part of a mentor training curriculum and can be obtained from the Center for the Investigation of Mentored Experiences in Research (CIMER, http://cimerproject.org) as linked from the National Research Mentoring Network (NRMN, http://nrmnet.net). A trainee training module that focuses on the same learning objectives from the trainee’s perspective has also been developed and is currently undergoing pilot testing as a module and as part of Entering Research (Branchaw et al. 2010), an undergraduate and graduate-level trainee training curriculum targeting five areas of trainee development. Other modules that address psychosocial factors, such as motivation, resilience, and identity development are currently in development with NRMN partners.