A randomized controlled trial of an intervention to increase cultural diversity awareness of research mentors of undergraduate students

Abstract

Cultural diversity variables like race and/or ethnicity influence research mentoring relationships, but mentors may not know how to address such variables with their mentees. Using a randomized controlled trial design, we tested a mentor training intervention to increase mentors’ awareness and skill in addressing cultural diversity in research mentoring relationships, documenting its impact on mentors and their undergraduate mentees’ ratings of mentor effectiveness. Participants were a national sample of 216 mentors and 117 mentees from 32 undergraduate research training programs in the United States. Mentors in the experimental condition reported greater gains than those in the comparison condition regarding the relevance of their racial/ethnic identity to mentoring and their confidence to mentor students across diverse cultural identities. Paired mentees of mentors in the experimental group rated their mentors higher at respectfully broaching and creating opportunities to address race/ethnicity matters than those with mentors in the comparison group. Our results support the efficacy of culturally focused mentorship education.

Mentorship education with cultural diversity awareness content facilitates culturally aware mentorship practices in science.

INTRODUCTION

Mentorship matters in the success of individuals in science, technology, engineering, and mathematics (STEM) (1, 2). Defined as a working alliance between individuals aimed at mutual growth and success through both career and psychosocial support (2), mentorship is particularly critical to the development of undergraduate (UG) students’ research and academic skills, helping them refine their research career goals and make informed decisions about pursuing graduate degrees (3–6). Authors of the National Academies of Sciences, Engineering, and Medicine (NASEM) report (2), The Science of Effective Mentorship in STEMM, documented that demographic variables like race and ethnicity interact with mentees’ perceptions of their mentored research experience and what they value in a research mentor, especially for those from historically excluded racial/ethnic groups in STEM (i.e., Black, Indigenous, and Hispanic/Latinx). However, some research mentors may not see the relevance in acknowledging such variables in their mentoring relationships (7, 8). Others may be hesitant to address social identities, including race/ethnicity, in mentoring relationships because of concerns of being misunderstood as being prejudiced or otherwise offending their mentee (9). McGee et al. (10) described how racialized STEM education can compromise the experience and success of students. In contrast, one study showed that UG STEM students from historically excluded groups whose research mentors addressed the link between their cultural backgrounds and their student experience as well as validated their racial and academic identities reported greater involvement in research, higher science identity and belonging in research, and higher STEM degree graduation rates than those whose mentors did not do so (11). On the basis of this evidence, the NASEM report (2) authors concluded that recognizing and responding to cultural diversity can contribute to mentors’ effectiveness and that there is an urgent need to build this capacity in research mentors (12).

To effectively recognize and respond to cultural diversity variables like race and/or ethnicity, mentors need cultural competence. The first step in cultural competence is developing cultural diversity awareness (CDA) (13). Thus, we assert that enhancing research mentors’ CDA to (i) recognize their own culturally shaped beliefs, perceptions, and judgments and (ii) be cognizant of cultural differences and similarities between themselves and their mentees (13, 14) through training will increase their mentoring effectiveness. Specifically, they will be alert and responsive to opportunities and challenges that may arise from race and ethnicity in their mentoring relationships and in the research training experiences of their research mentees.

Purpose of the study

The purpose of this study was to test the effectiveness of an established mentor training curriculum with its original equity and inclusion (E&I) module compared to the same curriculum with a unique module designed to increase research mentors’ CDA (see Table 1). We investigated the comparative impact of these two interventions on both mentors’ and their UG mentees’ ratings of their mentors’ effectiveness. Specifically, we conducted a randomized controlled trial (15) comparing the effects of Entering Mentoring (EM) (16, 17), an evidence-based mentor training curriculum, with its original E&I module compared to the EM curriculum with the E&I module replaced by the enhanced cultural awareness (ECA) module that we designed for this study. The ECA module is described in detail in another paper (18). In a previous randomized controlled trial, Pfund and colleagues (19) found that mentors in the intervention group who received EM training (with E&I module) self-reported statistically significant higher gains in all six mentoring competencies compared to mentors in the control group who received no training. The mentoring competency addressing diversity within the mentoring relationship showed the smallest gains for mentors in that study. House et al. (20) conducted follow-up interviews with participants from that study and found that while some increased their CDA and made changes to their mentoring practice, most did not. On the basis of these findings, we sought to further develop and test this aspect of the EM training.

Table 1. Mentor training curriculum by session.

EM, entering monitoring; E&I, equity and inclusion; ECA, enhancing cultural awareness.

Session 1
Learning objectives. Mentors will:	Description of module activities
Reflect on group dynamics and ways to make the group functional	Identify constructive and destructive behaviors that affect group dynamics
Review expectations of their mentee, consider how personal factors may impact expectations; provide constructive feedback	Analyze a case scenario related to misaligned expectations, mentee disengagement
Clearly communicate expectations for the mentoring relationship	Review mentoring compacts
EM competencies: Aligning expectations, maintaining effective communication
Session 2a: Standard addressing E&I
Learning objectives. Mentors will:	Description of module activities
Increase understanding of E&I in mentor-mentee interactions	Identify mentor-mentee differences and potential impact on their relationship
Recognize impact of biases and assumptions on mentoring relationship, acquire response strategies to manage them	Identify stereotypes associated with a list of occupations
EM competency: Addressing E&I
Session 2b: ECA
Learning objectives. Mentors will:	Description of module activities
Expand understanding of cultural diversity in mentoring relationships	Debrief iCAM online module; label and discuss ways underrepresented students react to discrimination
Recognize impact of biases and assumptions on mentoring relationship and acquire skills to manage them	Watch animated video that illustrates the experiences of minoritized individuals; discuss research about pros and cons of “colorblind” or color-evasive ideology
Awareness of cultural diversity in oneself and others	Complete broaching styles self-assessment related to race; work in pairs to critically review a challenging, racially salient interaction with a trainee
Communicate effectively across dimensions of cultural diversity, consider power dynamics	Discuss case scenario in which mentor observes racial discrimination in lab; role play responses with peer feedback
EM competency: Not applicable
Session 3
Learning objectives. Mentors will:	Description of module activities
Learn how to assess mentee understanding and provide feedback for mentees, consider diversity factors in assessing understanding	Discuss case scenarios related to mentee falling short of performance expectations; identify barriers to effective understanding between mentor and mentee
Identify reasons for a lack of understanding, including expert/novice difference	Discuss common factors that contribute to mentor-mentee misunderstandings
Define self-efficacy and its four sources; articulate their role in fostering research self-efficacy	Review presession materials about self-efficacy; discuss personal relevance
Identify signs of mentee self-efficacy and understand mentor role in fostering research self-efficacy	Generate a list of their mentees’ strengths and strategies to support them through challenging tasks
Practice strategies for building mentees’ self-efficacy in research	Discuss case scenario related to a mentee with doubts about being successful
EM competencies: Assessing understanding, self-efficacy
Session 4
Learning objectives. Mentors will:	Description of module activities
Define, assess, and evaluate researcher independence at different career stages	List research skills and competencies expected of mentees across career stages
Use various strategies to build mentee research self-efficacy and foster interdependence	Discuss case scenario about mentee seeking affirmation and approval for research skills
Identify the roles mentors play in the overall professional development of their mentees	Develop list of the roles mentors are responsible for in mentee professional development
Practice strategies for supporting mentee professional development	Discuss and then role play a case scenario related to mentee that is having trouble with the writing process
EM competencies: Fostering independence, promoting professional development

In the present study, the experimental group received EM + the ECA module instead of the original E&I module, and the comparison group received EM with the original E&I module. We focused on retrospective pre- to postintervention change scores on measures of CDA and mentoring effectiveness. CDA was assessed using Byars-Winston and Butz’s scale (21) that measures research mentors’ CDA related to race/ethnicity in research mentoring relationships and includes two versions. The mentor version assesses attitudes, behaviors, and confidence, and the mentee version assesses attitudes and mentor behaviors. Mentoring effectiveness was assessed with measures of participants’ mentoring skills using the Mentoring Competency Assessment (MCA) (22), mentoring self-efficacy, overall mentoring quality, and culturally aware mentoring (CAM) skills items from a previous study (23).

We tested two hypotheses. First, we hypothesized that mentors who received EM + the ECA module would report greater CDA than mentors who received EM + the original E&I module. Second, we hypothesized that mentees with mentors who received EM + the ECA module would rate their mentors’ CDA and overall mentoring effectiveness more highly than those whose mentors received EM + the original E&I module. Decades of research on the responses of K-12 teachers to multicultural education diversity training demonstrates that their attitudes toward diversity influence the effectiveness of the diversity training received (24). Therefore, we also posed an exploratory question: Do mentors’ attitudes toward the relevance of cultural diversity to mentoring before the mentor training intervention interact with how they benefit from the intervention (i.e., mentoring effectiveness)?

We report the results of our study with a national sample of research mentors predominantly from racial/ethnic groups well represented in STEM (i.e., white/Caucasian, and Asian) working with UG mentees largely from historically excluded racial/ethnic groups. All mentors and mentees were engaged in summer research programs. Participant recruitment procedures and the study measures are detailed in Materials and Methods.

RESULTS

The study population consisted of a national sample of 216 research mentors [faculty or co-mentors (e.g., postdoctoral fellows and graduate students)] and 117 UG students (any stage/year) participating in STEM research–oriented summer programs. Informed consent was obtained from all participants following approval from the University of Wisconsin-Madison Institutional Review Board. Mentors were randomized into the experimental or comparison groups to receive four 2-hour sessions (8 hours in total) of mentor training delivered synchronously online. After the mentor training intervention was complete, mentors and mentees completed previously validated measures administered through electronic survey platforms.

Characteristics of participants

There were no statistically significant differences found between mentors at baseline by treatment groups on demographic and background characteristics (see Table 2). Most mentors identified as women (54%) and white (68%). The majority was either in faculty positions or were graduate students. The groups were comparable in terms of mentoring experience and background with mean years of mentoring experience being 6.3 and 8 for the experimental and comparison groups, respectively. Similarly, no statistically significant differences were found between treatment groups for the paired mentees at baseline on their demographic characteristics. Most identified as women (67%) and as white (53%) or Hispanic (37%).

Table 2. Demographic characteristics of mentors and mentees.

None of the between-group P values were ≤ 0.05; percentages are calculated on the basis of the number of participants who responded to each item. Participants self-reported their gender and race/ethnicity, and some may have self-identified as more than one race or chose to not report. Of the 216 enrolled mentors, 197 received the intervention. UG, undergraduate.

Characteristic	Mentors (N = 197)		Mentees (N = 117)
	Experimental (N, % of 110)	Comparison (N, % of 87)	Experimental (N, % of 53)	Comparison (N, % of 64)
Gender (2)
Men	45 (41)	38 (44)	18 (34)	21 (33)
Women	61 (55)	45 (42)	35 (66)	43 (67)
Race/ethnicity
White	72 (65)	63 (72)	26 (49)	36 (56)
Asian	13 (12)	12 (14)	14 (26)	9 (14)
African American	10 (9)	3 (3)	7 (13)	11 (17)
Other	8 (7)	7 (8)	6 (11)	8 (12)
Hispanic	13 (12)	11 (13)	21 (37)	21 (33)
Title/position
Faculty member	49 (45)	31 (36)	–	–
Graduate student	34 (31)	31 (36)	–	–
Postdoctoral	14 (13)	13 (15)	–	–
Scientist	7 (6)	6 (7)	–	–
Other	6 (5)	6 (7)	–	–
Currently mentoring UG in research context	83 (75)	65 (75)	–	–
	Experimental (mean, SD)	Comparison (mean, SD)	Experimental (mean, SD)	Comparison (mean, SD)
Mentoring experience
Previous mentor training	1.79 (0.41)	1.81 (0.40)	–	–
Years of mentoring experience	6.31 (6.1)	7.99 (9.0)	–	–

Effects of the intervention

As shown in Table 3, between-group scores for mentors on the full measures of CDA attitudes, CDA confidence, mentoring skills as measured by the MCA, mentoring self-efficacy, overall mentoring quality, and CAM skills were not statistically different at baseline. Results from t tests analyses (see Fig. 1 and Table S1) revealed that the mean change in pre- to postintervention CDA attitudes and confidence and mentoring skills (MCA scores) was not statistically different across the treatment groups. There were also no statistically significant differences by treatment condition for gains in mentoring self-efficacy, overall mentoring quality, and CAM skills. That is, mentors in both treatment groups reported statistically significant pre- to postintervention gains in scores on all study measures.

Table 3. Comparison of mentor outcome measures at baseline by treatment group.

No statistically significant differences between treatment groups on study measures at baseline. Mann Whitney U tests were used to compare means of the baseline study measures for mentors by treatment group. CDA, cultural diversity awareness; MCA, mentoring competency assessment; CAM, culturally aware mentoring.

	Comparison		Experimental
	n	mean (SD)	n	Mean (SD)	z	α
CDA attitudes	84	3.47 (0.691)	108	3.39 (0.734)	−0.547	0.828
CDA confidence	84	3.13 (0.735)	109	3.13 (0.869)	−0.031	0.889
Mentoring skills (MCA)	62	4.49 (0.793)	84	4.43 (0.826)	−0.608	0.955
Mentoring self-efficacy	62	4.53 (0.1.02)	82	4.44 (1.05)	−0.686	–
Overall mentoring quality	61	4.64 (0.797)	82	4.44 (0.904)	−1.45	–
CAM skills	60	3.68 (1.33)	80	3.47 (1.27)	−0.974	0.903

Fig. 1. Study outcomes before and after gain scores by treatment group.

CDA, cultural diversity awareness; MCA, mentoring competency assessment; CAM, culturally aware mentoring.

Primary hypotheses

In support of our first hypothesis, item level comparisons using Mann Whitney U tests revealed that the experimental group reported statistically significant higher pre- to postintervention gains in the CDA attitudes item, which asked mentors how much they agreed that their racial/ethnic identity is relevant to their mentoring relationships, compared to the comparison group [0.795 (experimental) versus 0.301 (comparison); Z = −2.64, P < 0.01]. To test our second hypothesis regarding a treatment effect for mentees’ ratings of their mentors, we also conducted Mann Whitney U tests for individual items comparing the paired mentees’ perception of their mentors’ CDA behaviors. We found support for this hypothesis in that the paired mentees of mentors in the experimental group rated their mentors higher compared to those in the comparison group on the CDA behaviors item asking whether their mentor approached race/ethnicity topics in a respectful manner [3.94 (experimental) versus 3.28 (comparison); Z = −1.98, P < 0.05] and on the CDA behaviors item asking whether their mentor created opportunities for them to bring up issues of race/ethnicity as they arose [2.98 (experimental) versus 2.39 (comparison); Z = −2.14, P < 0.05) (see Fig. 2).

Fig. 2. Mentees’ ratings of mentors’ behaviors by treatment group.

Frequency of CDA behaviors: 1 = never and 5 = all.

Exploratory research question

To answer the question “are mentors’ preintervention CDA attitudes related to how they benefit from the intervention,” we examined bivariate correlations between the study variables for the total sample, since there were no group differences by treatment condition in the total scale scores. Spearman rank correlations indicated that preintervention CDA attitudes were positively correlated with postintervention CDA confidence [correlation coefficient (r) = 0.33, P < 0.01] and CDA behaviors (r = 0.38, P < 0.01). In the correlations between the gains scores on our study measures (see table S2), we observed the strongest (in absolute value) positive correlation coefficients among gains in mentoring skills, mentoring self-efficacy, overall quality, and CAM skills.

DISCUSSION

The goal of our mentor training intervention was to build the capacity of mentors of UG mentees in STEM summer research programs to be more aware of and responsive to cultural diversity in their mentoring relationships. The results of our experimental study confirm that mentorship education is effective. Across both treatment conditions of our mentor training intervention, mentors improved in their self-ratings of CDA attitudes, confidence, and behaviors as well as in their mentoring skills as measured by the MCA. Mentors also reported gains in their overall mentoring self-efficacy and mentoring quality. Our findings add support to the evidence summarized in the NASEM report (2), The Science of Effective Mentorship in STEMM, that mentorship is a learned skill and that mentorship education such as the EM (17) curriculum increases mentors’ self-reported mentoring competence and confidence. Building on the foundational work of EM, we found limited but encouraging support for the contribution of a unique mentorship education module, ECA, to mentor training outcomes. We discuss three main findings and their implications for advancing effective, culturally responsive mentorship in STEM fields.

First, we found partial support for the hypothesized effect of the standard EM curriculum with our ECA module to research mentor outcomes above the standard EM curriculum with the original E&I module. That is, this effect was found specifically for the CDA attitudes item assessing mentors’ belief that their own racial/ethnic identity is relevant to their mentoring relationships is particularly informative. It highlights that the activities in the ECA module are useful in advancing research mentors’ linking of their personal racial/ethnic identity to their practice of mentoring.

This finding is also noteworthy because it is difficult for most people, especially those from well-represented racial/ethnic groups, to see what they do as being connected to race (25), coupled with the tendency for STEM disciplines to be presented as neutral to cultural diversity factors (26). However, evidence from educational research indicates that one cannot address racial/ethnic disparities, like STEM participation of individuals from historically excluded groups, without addressing the matter of race/ethnicity differences; race neutrality does not work (27). Research mentors’ being cognizant of the ways in which their racial/ethnic identity interfaces with their mentorship practices may help them be more alert to disaffirming or inequitable mentoring practices and be more supportive of mentees’ racial/ethnic identity development as they pursue STEM fields. In their study on youth mentoring, Pfeifer et al. (28) found that mentors’ own ethnic identity exploration was positively associated with ethnocultural empathy; that is, empathetic responsiveness to others’ ethnic identity and concerns. The ECA module may help mentors to reflect on or strengthen their own racial/ethnic identity and subsequently bring a more culturally empathic lens to their mentoring relationships. Future research may investigate the impact of other relevant cultural diversity measures for research mentors, such as their racial/ethnic identity and ethnocultural empathy, on mentorship effectiveness. Assessing mentor readiness for doing the cultural self-reflection involved in these types of interventions may also be useful as noted below.

Second, as we hypothesized, mentees of research mentors who received EM with the ECA module rated their mentors more favorably on two specific CDA behaviors than did mentees of mentors who received EM without the ECA module: respectful approach to race/ethnicity topics and creating opportunities to discuss such topics. That the mentees noticed the difference is an encouraging finding, suggesting that our ECA module did not just support personally enlightened, culturally aware research mentors but that such CDA was detectable in their mentoring behaviors. Byars-Winston et al. (9) reported that some research mentors in STEM express difficulty and a lack of confidence in mentoring students from historically excluded groups and the need for further training in doing so. Our findings support that interactive, facilitated adult learning opportunities in which mentors can discuss, generate, and practice, with their mentor peers, strategies for engaging race/ethnicity topics in their mentoring can build their confidence and behaviors to respectfully lean into and initiate relevant matters of racial/ethnic topics with their mentees. Longitudinal research is needed to investigate whether there are differences in mentee academic or career development behavioral outcomes as a function of mentors with high CDA confidence and behaviors.

Last, our findings of positive correlations between preintervention CDA attitudes with postintervention CDA confidence and CDA behaviors lend support to the facilitative role of attitudinal disposition in participants’ response to or benefit from mentor education. Our data showing that cultural diversity attitudes are a factor in the outcomes that mentors report after intervention are consistent with similar findings in studies of pre-service teachers' responses to cultural diversity education (29). It may be that our intervention benefits research mentors who have already been thinking about or who value racial/ethnic diversity matters. Our previous study with research mentors from well-represented racial/ethnic groups captured varied attitudes toward addressing cultural diversity in their mentorship, including viewing cultural diversity as irrelevant to doing science and viewing addressing cultural diversity as a vulnerability factor for being perceived as prejudiced (9). Individuals are not easily characterized by a single attitude, but rather a combination of attitudes or dispositions. A future study applying linear modeling to examine whether CDA attitudes or other measures that assess attitudes toward diversity predict mentor education outcomes would be illuminating. Other investigators could also examine potential variations in the impact of our interventions by mentor demographic groups such as function of years of previous mentoring experience or gender.

Overall, we note that there were strong positive correlations between gains in CAM skills and mentoring skills for the total sample, which indicates that both the EM + original E&I module and the EM + ECA module increase mentors’ cultural awareness and their mentoring competency. Black et al. (18) recently found support for the effectiveness of the ECA module in promoting mentors’ intention to be culturally aware. Mentorship education that includes structured, culturally focused content to facilitate mentor CDA is effective in increasing mentors’ perceived skill and confidence to enact culturally responsive mentoring behaviors (30).

Limitations

We acknowledge the limitations of self-reported data. We included mentee data regarding mentor skills and behaviors to provide evidence of the mentor education impact beyond what mentors stated about themselves. While these data are informative and encouraging, we understand that there is a power dynamic between mentor and mentee and that dynamic could have influenced mentee responses. Although assurances of anonymity were given to mentee and mentor participants and they were both blinded to the mentors’ treatment condition, they were in formal mentored research programs designed to facilitate UG students’ success in STEM pathways. Research mentors from these programs often provide letters of recommendation for graduate degree or employment applications, and thus mentees may have felt compelled to favorably rate their mentors.

Mentors have substantial power and influence to shape mentored research experiences and the research training environment. Hence, interventions that change mentor behavior have the potential to change the research training system, which has not yet realized the ideals of a truly diverse and inclusive science. Changing mentor behavior is one strategy for realizing and advancing scientific workforce diversity. Results from our study can inform programs aimed at mentor professional development to include evidence-based curricula, like EM (17), to promote the attitudes, confidence, and behaviors needed to be culturally aware research mentors. EM with the addition of an ECA module holds promise as an effective intervention to facilitate culturally aware mentorship practices in STEM.

MATERIALS AND METHODS

Experimental design

We investigated the comparative impact of two mentor training interventions on both mentors’ and their UG mentees’ ratings of their mentors’ effectiveness. Specifically, we conducted a randomized controlled trial (15) to test the effectiveness of an established mentor training curriculum with its original E&I module compared to the same curriculum with the unique module designed to increase research mentors’ CDA.

Mentor training intervention

EM is a well-studied mentor training curriculum that has been tested and shown to be effective at increasing mentoring effectiveness (16, 17). EM includes six foundational mentoring competencies delivered as modules: (i) maintaining effective communication, (ii) establishing and aligning expectations, (iii) assessing mentees’ understanding of scientific research, (iv) addressing E&I within mentoring relationships, (v) fostering mentees’ independence, and (vi) promoting mentees’ professional career development. EM includes experiential activities and case scenarios for group discussion related to the competencies. The training was designed to be implemented with mentors who are in the process of mentoring, either in eight 1-hour sessions or four 2-hour sessions over 4 to 12 weeks. It has also been implemented in one full day or two half-day sessions.

Module learning objectives and activities

The ECA module, described in detail in another paper (18), is conceptually based on the multicultural competency (31) model. Three domains of competence are specified in the model as being necessary for culturally aware practice: attaining knowledge of diverse groups, increasing awareness of one’s own cultural beliefs and norms, and developing skills for effective practice. These three domains informed key elements included in the ECA module. The ECA module has four learning objectives: (i) expand understanding of cultural diversity in mentoring relationships (knowledge), (ii) recognize impact of biases and assumptions on mentoring relationship (knowledge), (iii) raise awareness of cultural diversity in oneself and others as well as consider power dynamics between mentors and mentees (awareness), and (iv) generate strategies to communicate effectively across dimensions of cultural diversity (skills) (Table 1, session 2b).

The ECA module is a 2-hour session. It includes experiential activities to guide mentors’ self-reflection on their cultural identity and understanding how their cultural experiences, especially related to race and ethnicity, influence their mentoring relationship and play a role in their mentees’ academic development. These activities include short videos, small group learning through guided discussion of case scenarios, role plays, and sharing of CAM practices and resources. Before the experimental condition with EM + ECA module, participants in this study completed an online self-directed, asynchronous training component on cultural diversity called iCAM described in detail in previous publications (18, 23). The iCAM training provides an introduction to the role of cultural diversity in STEM mentorship.

Implementation of intervention

The mentor training was delivered over the course of four online sessions. The four sessions were spaced 2 weeks apart so that participants met roughly twice a month for 2 months. Three of the four sessions of the curriculum were identical in content for the experimental and comparison groups, were each 2 hours long, and drew directly from the EM curriculum (Table 1). These sessions included an additional module built upon the EM framework called “promoting research self-efficacy” (32). The module tested in this study was introduced in the second training session. The comparison group received the addressing E&I module (session 2a) from EM, and the experimental group received the ECA module (session 2b) in place of the E&I module. Session 2a had 1 hour and 40 min of activities, leaving a 20-min flexible discussion time, and session 2b was 2 hours long. We delivered the cultural diversity content in the second session to help mentors gain the tools and skills needed for forming effective cross-cultural mentoring relationships while they were engaging in mentoring activities during the course of the study (see fig. S1).

Online platform

Because study participants were located across the United States, we designed the mentor training intervention to be implemented completely through an online platform. We have previously shown that EM can be delivered effectively in an online platform (33), and no statistical differences were observed between online and face-to-face delivery using propensity score matching (34). We chose the web conferencing tool, Blackboard Collaborate Ultra (hereafter referred to as Blackboard). Blackboard allows for virtual engagement using synchronous communication with the following features: sharing live or recorded audio/video lectures; screen, application, and presentation sharing; whiteboard; interactive polling; virtual breakout rooms for small group discussion; public and private text chat; and participant call-in options for audio. Blackboard was supported by the educational institution of the authors and available to employees and students at no cost. We also used Moodle, an open-source online learning platform that is private and secure (instructor access and login required), to warehouse all training materials including assignments, mentor biographies, and links to mentoring tools, resources, and suggested readings.

Procedure

We designed the study to control for unplanned variation in key study parameters as much as possible. Of particular concern to the study team were: (i) “contamination” due to participant contact outside of the instructional setting, (ii) the impact of program characteristics [i.e., predominantly white versus minority-serving institution (MSI)] on participants’ mentoring experiences, and (iii) the impact of differences in facilitator expertise and style. To reduce the likelihood that participants would discuss the curriculum with someone randomized into the other condition, we randomized participating programs rather than individual participants to ensure that all research mentors from a particular program were in the same treatment condition, either experimental or comparison group. All participants were in groups that had mentors from different programs in them.

Because the experiences of research mentors in programs at MSIs could potentially be different from those of mentors at non-MSIs, we distributed the programs at MSIs as evenly as possible between the two study conditions. Participating programs were divided into two lists (programs at MSIs and programs at non-MSIs), each group of programs was numbered in the order that the applications were received, and online random number generators were used to randomize the programs in each group (described below in the “Randomization” section).

Facilitator variation was controlled in two ways. First, all facilitators received extensive study-specific facilitator training with detailed scripts for each training session that they were instructed to follow closely. Second, we formed eight pairs of facilitators such that the pairs’ combined experience with the curriculum, facilitation expertise, and online learning technology was roughly equivalent. We also considered disciplinary and racial/ethnic diversity within each pair. These pairs taught together throughout the trial unless an absence was unavoidable, in which case a single designated substitute, an expert facilitator who was not otherwise facilitating any of the sections, stepped in. Each facilitator pair was assigned to lead two comparison sections and two experimental sections of the training to further reduce the impact of facilitator variation.

Settings and participants

The study population consisted of research mentors [faculty members or co-mentors [e.g., postdoctoral fellows and graduate students)] who worked closely with a UG student (any stage/year) participating in a summer STEM research–oriented program (see the Supplementary Materials). Mentors self-selected to participate in four 2-hour sessions (8 hours in total) of mentor training. Mentors were eligible if they were actively mentoring a UG student researcher during June and July of 2017 and if their relationship with their mentee began after 1 January 2017.

Sixteen sections of online training were offered; half were the comparison condition, and half were the experimental condition. Mentors indicated their availability for eight different sections offered across a range of days of the week and time of day, beginning the week of 22 May 2017 and ending the week of 17 July 2017. Each training section had its own Blackboard Ultra “classroom” and Moodle site where their session materials were posted. In addition to the facilitator pair leading each 2-hour training session, there was a technology support person online to help participants resolve any technical issues encountered during the session.

Recruitment

Recruitment was a two-stage process. First, summer UG research program directors interested in having their mentors enroll in research mentor training were identified. Second, an email was sent to the program mentors inviting them to participate in the training, and the program directors were enlisted to encourage them to register.

To recruit summer UG research programs, the study team retrieved contact information for potential programs from the National Science Foundation Research Experiences for Undergraduates (REU) website, resulting in 150 potential Biology REU sites, 58 potential Physics REU sites, and 78 potential Chemistry REU sites. Another 53 potential sites from the National Institutes of Health (NIH) Maximizing Access to Research Careers (MARC) programs were identified via the NIH website. Details for the REU and MARC program searches are provided in the Supplementary Materials.

Of the 343 programs invited to participate, 49 applied. Ten were excluded, because they did not meet the inclusion criteria (n = 3), later declined to participate (n = 3), or other reasons (n = 4). A total of 39 programs were included in the randomization process, and 32 programs successfully recruited research mentors who participated in the training (see Fig. 3). No monetary incentives were offered to programs or participating mentors. To assist program directors with recruiting research mentors, the study team provided a template flier and email text to send to potential participants. Mentors were informed that participating in the training and study would require them to: (i) be actively mentoring a UG researcher during June and July of 2017, (ii) have started working with the student after 1 January 2017, (iii) complete brief assignments in preparation for the training and participate in activities during the sessions, (iv) complete a survey about their research mentoring experience at the end of the summer, and (v) have access to technology to participate in online training. Once mentors applied to participate in the training, they were sent a brief online registration form and asked to indicate all of the training sections for which they would be able to attend all four sessions.

Fig. 3. Consort diagram for program enrollment.

Mentor recruitment resulted in 246 applications. Of these, 216 mentors were enrolled, and 30 were excluded, because they did not meet inclusion criteria (n = 5), canceled their registration (n = 17), or follow-up issues were encountered (n = 8) (see Fig. 4).

Fig. 4. Consort diagram for participant enrollment.

Randomization

We conducted two separate simple randomizations, because the stratifying characteristic (MSI, n = 7 programs) was disproportionate, making stratified randomization unfeasible. Using a random number generator (http://stattrek.com/statistics/random-number-generator.aspx) resulted in 3 MSI programs and 16 non-MSI programs assigned to the comparison treatment condition and 4 MSI programs and 15 non-MSI programs assigned to the experimental treatment condition. One additional non-MSI program joined the study after the initial randomization process and was randomly assigned to the comparison condition for a final count of 20 programs in the comparison condition and 19 programs in the experimental condition.

On the basis of the condition assigned to each participating program, the 216 enrolled mentors were allocated to the comparison condition (n = 96) and to the experimental condition (n = 120) as shown in Fig. 4. Attrition of 19 people before the first training session resulted in a total of 197 participants: comparison condition, n = 87 (92% retention) and experimental condition, n = 110 (91% retention). We were unable to conduct intent-to-treat analyses for the 19 people who left the study before the first training session when baseline survey data were collected.

Blinding of participants to assignment

Program directors and participating mentors were blinded to their group allocation throughout the trial. Mentees were not informed that there was mentor training and thus, by default, were unaware of the group allocation of their mentors. All data were anonymized by a study team member who did not participate in the trial in any other capacity so that the data analysts were blinded to the identification of the participants.

MEASURES

Data collection

Data were collected using the measures described below. In all cases, follow-up requests were sent as needed to improve participant response rate. For the paired mentees, response rates for evaluation surveys completed about their mentors were 48% (53 of 110) for the experimental group and 74% (64 of 87) for the comparison group. All data were collected through two electronic survey platforms (Qualtrics and the CIMER Assessment Platform; https://cimerproject.org/cimer-assessment-platform/). As shown in Table 3, high Cronbach’s α coefficients of >0.80 were observed for all study measures, indicating good internal consistency of items in the measures.

Cultural diversity awareness

To assess mentor’s self-reported changes in CDA related to race/ethnicity, we used the mentor CDA scale (19). It was validated with research mentors and UG researchers in STEM. Mentors received the mentor version of the CDA scale that includes 17 items comprising three subscales to assess attitudes about, behaviors supporting, and confidence to implement CAM practices in research mentoring relationships. These attitudes and confidence subscales were administered to mentors both pre- and postintervention, and the behavior subscale items were administered postintervention only. Mentees were administered the mentee version of the CDA scale, which includes nine items comprising two subscales that capture mentees’ perception of the relevance of CAM (attitudes) and their perception of the frequency of their mentors’ CAM behaviors (behaviors). These were administered to the mentees postintervention. Responses to the CDA measure are made on a five-point Likert-type scale ranging from: 1 (strongly disagree) to 5 (strongly agree) for the attitudes subscale, 1 (not at all confident) to 5 (completely confident) for the confidence subscale, and 1 (never) to 5 (all the time) for the behaviors subscale.

Mentoring skills

We used the MCA developed by Fleming et al. (22) to assess mentoring skills. The MCA is a validated measure of mentor skills, developed specifically for postsecondary STEM research contexts. Previous evidence of validity for this measure includes those mentoring graduate, professional, and postdoctoral trainees, but it has not yet been validated with mentors of UGs. The 26 items measure mentoring skills across six competencies: maintaining effective communication (six items), aligning expectations (five items), assessing understanding (three items), fostering independence (five items), addressing diversity (two items), and promoting professional development (five items). Mentors were asked to rate their perceived level of mentoring skills before the training (retrospectively) and after the training, using a seven-point Likert-type scale (1, not at all skilled; 4, moderately skilled; 7, extremely skilled) for each of the 26 MCA items.

Mentoring self-efficacy

This was assessed with a single item asking participants to rate their confidence in their ability to mentor effectively before the training (retrospectively) and after the training, using a seven-point Likert-type scale (1, very low; 4, average; and 7, very High).

Overall mentoring quality

This was assessed with a single item asking mentor participants to rate the overall quality of mentoring that they provided before the training (retrospectively) and after the training, using a seven-point Likert-type scale (1, very low; 4, average; and 7, very High).

CAM skills

Five items used in a previous study (21) asked participants to rate how skilled they were before the training (retrospective) and after the training, using a seven-point Likert-type scale (1, not at all skilled; 4, moderately skilled; and 7, extremely skilled). Specific items were: (i) intentionally creating opportunities for mentees to bring up issues of race/ethnicity when they arise, (ii) encouraging mentees to think about how the research relates to their own lived experience, (iii) going outside of my comfort zone to help mentees feel included in the lab, (iv) respectfully broaching the topic of race/ethnicity in my mentoring relationships, and (v) making a plan to increase my CAM skills.

Statistical analyses

We first summarized and compared mentor and mentee background characteristics at baseline by treatment group using mean group differences with 95% confidence intervals. We also compared baseline study measures across treatment groups. To account for the nonnormal distribution of the data, we used nonparametric tests for data analyses. We used Mann Whitney U tests to compare mentors’ mean baseline scores between the experimental and comparison groups for CDA attitudes, CDA confidence, mentoring skills, mentoring self-efficacy, overall mentoring quality, and CAM skills. Next, we analyzed changes in our study measures from pre- to posttreatment by calculating mean scores with 95% confidence intervals from the retrospective pre- and posttest assessments. We tested for differences between groups in the pre- to posttest change using Wilcoxon rank sum tests. We also examined mean differences in posttest study measures, including mentors’ CDA behaviors, by treatment group as well as mean differences in gains from pre- to posttest on study measures by treatment group using Mann Whitney U tests. The full scale and individual items were compared. We next used Mann Whitney U tests to compare mean scores of paired mentees’ ratings of mentors’ CDA behaviors by treatment group. The full scale and individual items of the CDA behaviors scale were compared. Last, we conducted a series of Spearman rank correlations to examine relationships among mentors’ CDA attitudes, confidence, and behaviors and mentoring skills.

Supplementary Materials

This PDF file includes:

Tables S1 and S2

Fig. S1