Internal Consistency and Application of a Mentee Survey to Assess Mentor Competencies in an Academic Medical Center across Demographic Groups

Abstract

The National Academies of Science stresses the importance of research mentoring. We assessed the internal consistency and application of a novel 33 item mentor evaluation survey and explored differences across subgroups. The survey was administered annually to mentees. The response rate was 17.8% for a sample of 710 respondents. The survey exhibited strong internal validity with Cronbach Alpha > 0.89 for each subscale. Overall scores across the three domains were high. Basic Science trainees scored their mentor significantly lower than those in Translational or Clinical Science across domains (0.11–0.25 points). Underrepresented Racial Ethnic Groups (UREG) trainee scores were significantly lower in academic guidance and personal communication. Women had lower scores in 4 out of 5 domains. The survey is a modified instrument to assess mentee experience, although further validation against mentee outcomes is needed.

Introduction

Prior studies have demonstrated the importance of effective mentoring for the career development and long-term success of trainees in the health sciences (Hathaway, 2002; Pfund et al., 2006; Shen et al., 2022; Smyth et al., 2022; Williams et al., 2023). In particular, the National Academies of Sciences has highlighted the critical importance of high-quality mentoring to support the retention of women and individuals from groups underrepresented in research in the research workforce (National Academies of Sciences & Medicine, 2019). In recognition of this need, academic institutions are increasingly investing in mentor training and other supports to improve the research mentoring experience of their trainees and junior faculty.

However, measuring the quality of mentorship received by trainees is challenging. The inherent power dynamic in the mentor-mentee relationship may discourage mentees from reporting concerns or providing constructive feedback about their mentor. Mentor self-assessment of their performance across core competencies has exhibited ceiling effects in prior studies (Pfund et al., 2014). Qualitative data collection (i.e., focus groups, interviews) is time consuming and expensive. Survey instruments are an alternative to obtain feedback from research trainees. A valid and efficient means to measure mentor competencies and research trainee experiences will allow institutional leadership to 1) identify specific development needs to address with institution level programs for trainees (e.g., negotiation skills, grant writing); 2) identify specific development needs to address with institution-level programs for faculty mentors (e.g., promoting self-efficacy, providing feedback); 3) identify departments or programs that would benefit from additional training or resources to improve the institutional culture of mentoring; and 4) ensure that the needs of trainees from groups underrepresented in research are being met equitably.

Several mentor assessment tools have been reported in the literature (Abedin et al., 2012; Anderson et al., 2012; Berk et al., 2005; Fleming et al., 2013), focus on specific groups of trainees rather than all mentees within the School or unit (e.g., medical students) (Schäfer et al., 2015), or are not specific to research mentorship (e.g., encompass resident training and clinician educators) (Yukawa et al., 2020). The validated Mentoring Competency Assessment can be completed by mentees or mentors themselves to assess their skills in 6 mentoring competencies. This tool was designed to measure change following a mentor training program targeting those competencies, and items query mentorship skills at a high level (e.g., “Motivating mentees”), without drilling down to specific behaviors and actions. The purpose of this study was to test the psychometric properties of a survey instrument designed for any research trainee to rate the quality of the mentorship received by their primary research mentor, describe its use over 3 years in an academic institution, and test whether there were significant differences in mentee ratings of mentorship quality by gender, race/ethnicity, and research type to inform subsequent programming. Relative to other surveys of mentor behaviors, our instrument focused on the critical needs of mentees in the health professions at the graduate and post-graduate training levels.

Methods

Survey Instrument

We identified an existing survey instrument on a Clinical and Translational Sciences Institute website that was attributed to the NIH/National Heart, Lung and Blood Institute (NIH/NHLBI) with a version date of February 2003, which encompassed mentoring competencies previously identified as critical to research trainees (Abedin et al., 2012). This tool was selected because it probed specific research mentorship skills that are modifiable with additional training and/or institutional policies and supports. We were unable to identify any publications that had used the tool nor any studies validating it. In email communications, the current NHLBI Education Director was not able to provide additional information about its development or provenance.

To adapt the survey for this study, we expanded the Likert scale from 4 to 5 items to include a “not applicable” option (e.g., “strongly agree”, “agree”, “disagree”, “strongly disagree”, “not applicable”). Additionally, after internal pilot testing with our advisory board and several trainees, we made minor wording changes to enhance the clarity of questions (e.g., “Helps me to envision a career plan” was changed to “Helps me to develop a detailed individual plan”), and “Provides guidance in development and presentation of research projects for outside review groups” was replaced with “Sets clear expectations for each of our roles and responsibilities.” An option to provide free text comments was added for each domain. With these modifications, the survey included 33 items that evaluated five domains of the mentee experience measured as the following subscales pre-specified in the instrument: 1) Intellectual Growth and Development (6 items), 2) Professional Career Development (6 items), 3) Academic Guidance (8 items), 4) Personal Communication (7 items), and 5) Role Modeling (5 items). Specific questions are found in Figure 1.

Figure 1.

Overall mentee responses to survey questions by domain.

The survey instrument was deployed by email to all current graduate students in the School of Medicine (i.e., Masters, PhD, M.D./PhD), post-doctoral trainees, NIH K awardees, R38, and T32 research trainees via a link to the Research Electronic Data Capture (REDCap^®) data management system (Harris et al., 2009). Surveys were distributed annually through the respective offices serving trainee groups, T32 principal investigators, and directly to each K awardee. Respondents were assured that only departmental leadership would see survey responses to preserve anonymity. In the first and second years of survey deployment, respondents were asked to specify the names of their mentors. Due to concerns raised by mentees, the requirement to name their mentors was eliminated in year three, and instead respondents selected their mentor’s department. Respondents were asked to self-identify gender, race/ethnicity, and if their research was basic, translational, or clinical, but these questions were optional. Consequently, data was collected in a way that blinded us from matching respondent demographics to their responses in order to maintain privacy and prevent the identification of individual respondents. Three email reminders were issued over 2 and 3 months after survey deployment in years 1 and 2 respectively to encourage further participation further. In year 3, a drawing for modest prizes (branded sweatshirts, coffee mugs) was offered to encourage survey completion.

Statistical Analysis

The internal consistency of the subscales was evaluated using Cronbach’s Alpha coefficient. Characteristics of the respondents were described using univariate proportions of self-reported gender, race/ethnicity, and type of research they were engaged in. Of primary interest was exploring the associations of gender and race/ethnicity with the five subscales. The subscale scores were defined as the mean item response for the non-missing items. For analytic purposes, the gender comparison was restricted to self-reported males and females, and for race/ethnicity to self-reported White, Asian, and underrepresented in medicine groups (UREG), defined as Black/African American, Hispanic/Latino, or more than one race. Training stage was not collected due to concerns about maintaining mentee anonymity and could not be used in the analysis. For group comparison, analyses were conducted using Ordinary Least Squares analysis of variance, with Tukey’s Honestly Significantly Different pairwise tests used for the 3 level Race/ethnicity group comparisons. Due to the exploratory nature of the analysis, uncorrected p values of 0.05 were the threshold for significance. As an additional exploratory analysis, a multivariable linear regression was conducted of the average item score combined over all five domains as the dependent variable with the independent variables being gender, race/ethnicity, and type of research.

Results

Over three years, surveys were sent to 3993 individuals, and 710 responses were received for an overall response rate of 17.8%. Characteristics of the 688 individuals who completed the optional demographic questions are in Table 1. Respondents’ demographic characteristics were similar to the overall demographics of trainees at our institution, with 53% self-reported Female, 48% White, 34% Asian, and 12% UREG respondents.

Table 1.

Demographic Characteristics of Respondents

Characteristic	N (%)
Gender identity
Female	367 (53.3)
Male	295 (42.9)
Trans, Non-binary, Nonconforming	4 (0.6)
Prefer not to answer	22 (3.2)
Race/Ethnicity
White	342 (48.2)
Asian	244 (34.4)
Black/African American	34 (4.8)
Hispanic/Latino	26 (3.7)
More than 1 race/ethnicity	21 (3.0)
Prefer not to answer	43 (6.1)
Type of Research
Basic	258 (69.0)
Translational	76 (20.3)
Clinical	40 (10.7)

^*N = 688, demographic questions were optional

Psychometric Testing

The five pre-specified domain subscales demonstrated excellent internal consistency, with Cronbach’s Alpha standardized scores for all three years for each subscale ≥ 0.90; Intellectual Growth and Development 0.93, Professional Career Development 0.94, Academic Guidance 0.91, Personal Communication 0.93 and Serves as a Role Model 0.92.

Survey Responses (Overall)

The distribution of responses by domain subscale and item are displayed in Figure 1 A–E. Mentee responses were most favorable for items in the Intellectual Growth and Development and Personal Communications subscales, with the proportion of “Disagree” or “Strongly Disagree” below 10% for all items. Responses for items in the Academic Guidance and Serves as a Role Model subscales were more variable, with >10% of respondents reporting “Disagree” or “Strongly Disagree” on items related to modeling appropriate work life balance, helps me develop good negotiation skills, and involves me in peer review. The highest proportion of disagreement was observed in the Professional Career Development subscale, with 16% Disagree” or “Strongly Disagree” on the “Maintains balance between supporting his/her own Research and developing my own career”, 18% on “Sets clear expectations for each of our roles and responsibilities,” 20% on “Provides training in the skills needed to mentor others,” and 25% on “Helps me to develop a detailed individual career plan.”

Survey Responses by Pre Specified Subgroups

Trainees who indicated they were working in the Basic Sciences scored their mentor slightly lower than those in Translational or Clinical Science (0.04–0.21 points lower on subscale average scores). This was statistically significant for the Career Development subscale.

Subgroup analyses by self-identified gender are described in Table 2. Trainees identifying as female had slightly lower average responses compared to those identifying as male on all subscales and overall (3.42 vs. 3.50, p=0.08), but was statistically significant for the Professional and Career Development subscale (3.24 vs. 3.36, p=0.03). Further examination of the specific items within this subscale showed this difference to be driven by responses on the development of negotiation skills, individual development plans, and skills needed to mentor others. There were insufficient responses from trainees identifying as transgender, nonconforming, or other gender to allow comparisons.

Subgroup analyses by race/ethnicity are also described in Table 2. Respondents identifying as Asian did not rate their mentors differently than those identifying as White. However, trainees identifying as belonging to a group considered UREG rated their mentors significantly lower in the Academic Guidance and Personal Communication subscales than did trainees identifying as White, Asian, or both groups. Examining specific items within the Academic Guidance subscale, UREG respondents reported lower scores for “working effectively with others” (p = 0.024), development of “good negotiation skills” (p = 0.025), and “involvement in peer review of abstracts and manuscripts” (p = 0.017) compared to White respondents. In the Personal Communication subscale, UREG scores were significantly lower for the “reviews progress, proposed timelines, and milestones” (p = 0.004), “does not take advantage of my time and abilities” (p = 0.025), “encourages effective teamwork and frequent self-assessment” (p=0.034) items.

A multivariable regression of overall average item score with gender, race/ethnicity and type of research in the model did not yield any significant effects. (Table 3).

Table 3.

Dependent Variable - Average Item Score of Total Scale

Source	DF	Sum of Squares	Mean Square	F Value	Pr > F
Model	4	2.9089547	0.7272387	2.24	0.0641
Error	357	115.7847107	0.3243269
Corrected Total	361	118.6936654
R-Square	Coeff Var	Root MSE	Total Scale Ave Mean
0.024508	16.21973	0.569497	3.511137
Source	DF	Type III SS	Mean Square	F Value	Pr > F
Race3	2	1.71841874	0.85920937	2.65	0.0721
Male	1	0.53025512	0.53025512	1.63	0.2019
Basic_Science	1	0.62630464	0.62630464	1.93	0.1655

Discussion

Academic medical centers need validated instruments to assess the quality of mentorship their research trainees receive. We confirmed strong internal consistency for a modified NIH/NHLBI survey instrument for trainees to rate the quality of their research mentorship in five domains. The instrument holds promise as a valuable tool for efficiently measuring mentee perception of mentorship behaviors over time and to identify mentor and mentee development needs at both the departmental and institutional levels. The objective focus on mentor behaviors allowed us to identify modifiable skills that could be addressed through strategic programming to facilitate mentor skill development. As the overarching goal is to improve the institutional culture of mentoring and overall mentor quality, accurate assessments of the mentored experience in research are needed to inform the development and implementation of mentor education activities. However, further validation is required to determine whether survey results are associated with short and long term mentee outcomes such as time to program completion, publications, funding, and career success.

Our institution has utilized the survey results in several meaningful ways. Results were shared with leaders in the Graduate School, Office for Faculty, the Clinical Translational Science Institute, and Office of Physician Scientist Development to inform professional development offerings for both faculty and trainees. We implemented programmatic enhancements focused on four identified areas of need: 1) individualization of the mentee training experience, 2) additional mentor training opportunities to develop skills in managing difficult interpersonal dynamics, 3) individualized training for SOM divisions with specific needs, and 4) identification of key constituencies in the SOM that need career development support. To support mentors in individualizing the mentee training experiences, we enhanced our resources describing mentor compacts and individual development plans (IDPs) on our office website and emphasized their use in our Center for the Improvement of Mentored Experiences in Research (CIMER) training sessions (e.g., communication across differences, modeling work life balance) (Butz et al., 2018). Additionally, we host an annual mentor/mentee symposium, which provides mentors and their trainees the opportunity to receive guidance and support in establishing healthy and productive relationships through evidence based training and practice. To support mentors in their ability to navigate difficult interpersonal dynamics in the professional environment, we established our Crucial Conversations and Advanced Mentoring symposia. These optional training opportunities expand upon discussions mentors have in the CIMER “Entering Mentoring” training series to emphasize skill development in effective communication strategies and maintenance of a productive and hospitable work environment. We used survey findings to identify SOM Divisions that needed additional mentor training and support and provided mandatory training experiences to address identified areas of need. High level summaries of survey results were sent to research faculty annually, along with registration links for relevant professional development opportunities. Departmental level scores and thematic summaries of qualitative comments were provided to Chairs and Research Training/Program Directors to be used in their internal evaluation and improvement processes. Departments with lower scores were offered additional mentor training opportunities and are connected with central support services for their trainees (e.g., the Ombuds office). Departments with particularly positive results were celebrated for their strong mentorship climate with faculty and trainees. Finally, to address the career development needs of key constituencies highlighted in the survey (e.g., post-doctoral trainees), we revised our current mentor training curricula to emphasize the evolution of mentor/mentee relationships throughout the training experience and provided assistance in developing effective strategies for supporting trainees as they achieve independence. The postdoctoral training office added trainee workshops in negotiation, the Graduate School and post-doctoral training offices now require the use of individual development plans, all postdoctoral trainees are invited to the institution’s mentor training workshops, and one basic science department decided to deploy mentoring compacts for all new trainees. We plan to use these results over time to track progress as we develop and implement new programs and initiatives to improve the quality of mentorship and trainee experience at our institution.

It is interesting to note the consistently lower ratings from mentees in basic science compared to those in translational and clinical research. This may be due to differences in career stage; trainees in the basic sciences are likely earlier in their career on average (i.e., more graduate students in basic science vs. postdoctoral fellows and early career faculty in clinical research). Another hypothesis is that basic science mentorship occurs more often in dyads than in teams, or that the power dynamics inherent in the mentoring relationship are magnified when the trainee works within a single laboratory setting. This observation requires further exploration from qualitative data.

Of particular concern were the small but statistically significant differences observed in the responses of women and UREG trainees, consistent with similar studies in academic medicine (DeCastro et al., 2014; Dimitriades et al., 2022; Loftin et al., 2012; Spangle et al., 2021; Steele et al., 2013). While these inequities are longstanding and complicated to address, it is clear that targeted efforts to eradicate systemic inequities in the mentorship experience and to transform institutional culture are required. Approaches may include peer and group mentorship programs targeted to individuals in these groups; for example, a systematic review reported that mentorship programs designed for women, regardless of model, are met with high satisfaction and can help promote and retain women in academic medicine (Farkas et al., 2019). In addition, training for mentors to become more culturally aware and promote equity and inclusion in their mentoring practices may be useful (Byars-Winston et al., 2023; Pfund et al., 2022). The Culturally Aware Mentoring Program from the Center for the Improvement of the Mentored Experience in Research is currently being tested for its impact in this area (Womack et al., 2020). Our survey findings may inform specific examples or areas to highlight in such training. For example, intentionally supporting trainees in becoming integral members of the research team and scheduling regular time for feedback and self-assessment may address the particular needs of trainees from underrepresented groups.

Limitations

This study has several limitations. While our sample was adequate for psychometric evaluation, the low response rate and self-selection of respondents prevented us from having a complete picture of the mentorship climate in the institution. Based on qualitative analysis of the write in comments, it is likely that mentees at the extremes of satisfaction (i.e., those who were especially happy or frustrated with their mentors) were motivated to complete the survey. Current efforts to improve response rates include combining it with other required documentation (e.g., thesis committee meetings, research training compliance certification) and messaging through their local program leadership. We also provided a report with a summary of results back to all trainees, along with a description of how the School of Medicine is using the results and programmatic changes resulting from their feedback. Further, the Likert type responses are limited and need to be combined with targeted focus groups or other qualitative data to understand the mentored research experience of trainees. Nevertheless, the results have proven useful in informing programs and policies in our institution. Future work should establish the tool’s utility beyond our single institution, determine whether mentor training has a measurable impact on mentee perceptions, and quantify the association between mentee perceptions and subsequent outcomes. We aggregated responses from trainees from any UREG to obtain a sufficient sample size for statistical comparison, but we recognize that this is a heterogeneous group, and there may be important differences in experience not captured by our analysis. We did not have any responses from trainees self-reporting as American Indian/Alaskan Native or Native Hawaiians/Pacific Islanders. We are unable to examine differences by career stage as this data was not collected to preserve the anonymity of respondents.

Summary

In summary, this mentor assessment tool employed in Duke’s School of Medicine has strong psychometric properties and may be useful for institutions in assessing the research mentoring climate across Departments and over time. Continued efforts to address disparities in mentee experience for women and individuals from groups underrepresented in medicine are needed to bridge the gaps among trainee experiences and their paths to long term success.

Biographies

Dr. Gentzon Hall is an Assistant Professor of Medicine and Vice Chief for Diversity, Equity and Inclusion in the Division of Nephrology. He serves at the Co-Lead for the CIMER Mentor Training Program in the Office of Faculty Research at Duke and Director of the Duke/HBCU Mentored Intership Program; a newly established partnership that pairs talented undergraduates from North Carolina Central University (an HBCU in Durham, NC) and research faculty at Duke for an immersive, 1-year research and clinical shadowing experience. His primary research is focused on the genetics of familial nephrotic syndrome in African Americans.

Dr. Leonor Corsino is an Associate Dean for student affairs, an Associate Professor in the Department of Medicine in the Division of Endocrinology, and an Associate Professor in the Department of Population Health Science. She is the former Associate Director for the School of Medicine Office of Faculty Mentoring and was part of the team implementing the NRMN Mentoring training at Duke. She is an NRMN Certified Facilitator. Her work in health professions education and diversifying the workforce includes leading and developing innovative programs to expose students to science and health professions.

Dr. Michelle Mack is an Associate Director of Research Operations for the Duke Office of Clinical Research (DOCR) and has served as the Program Director for the Office of Faculty Mentoring for several years. She serves as a facilitator for the NRMN Mentoring training in the Duke University School of Medicine. She has worked with the Faculty Mentoring office and as part of funding from the Clinical and Translational Science Award (CTSA) on grant writing programs for junior faculty and investigators.

Dr. Rasheeda Hall is a nephrologist and clinical-investigator who holds a rank of Assistant Professor of Medicine in the Division of Nephrology at Duke University and serves as a Staff Physician at the Durham Veterans Affairs Healthcare System. Dr. Hall’s research objective is to improve geriatric problems in adults with kidney disease.

Richard Sloane, MPH has worked as an applied biostatistician for almost 30 years, most of which has been spent in the Duke Center for the Study of Aging and Human Development. He has collaborated with a variety of investigators providing statistical and project support while also mentoring medical students, research fellows and junior faculty. He also contributes critical review and writing in support of manuscripts.

Dr. Beth Sullivan is James B. Duke Professor of Molecular Genetics and Microbiology and Professor of Cell Biology. She is also Associate Dean for Research Training in Duke School of Medicine. In partnership with the Office for Biomedical Graduate Education (OBGE), she oversees PhD education, aiming to develop and sustain student-centered graduate education, promote inclusion and equity in the experience, and prepare students for diverse careers. Dr. Sullivan is an NRMN-trained facilitator. Her primary research is focused on the genetics and genomics of human chromosome abnormalities associated with birth defects and cancer.

Dr. Holly Hough is the Director of the Office of Physician-Scientist Development at Duke. She was formerly a Research Program Leader in the Duke Office for Clinical Research (DOCR). She co-developed CRISP, or the Clinical Research Internship Portal, in collaboration with CTSI Workforce Development and previously designed and launched the integrative medicine program for patients at the North Carolina Cancer Hospital.

Dr. Kevin Thomas is a Professor of Medicine and the Vice Dean for Equity, Diversity and Inclusion and Chief Diversity Officer for Duke School of Medicine. Dr. Thomas serves as a key advisor and partner to the Dean of the School of Medicine to advance the School’s goal of cultivating an inclusive, welcoming, respectful and supportive working and learning environment. Dr. Thomas and his team collaborate with staff, faculty and students across the school to implement the School’s Moments to Movement Strategic Plan to Dismantle Racism and Advance Equity, Diversity and Inclusion for the School of Medicine and larger Duke Health community.

Dr. Cathleen Colón-Emeric is Professor of Medicine, Chief of the Division of Geriatrics, Associate Dean for Research Mentoring, and a Senior Fellow for the Center for Aging and Human Development at Duke University. She is a clinical researcher focusing on fracture prevention in older adults, particularly after hip fracture and in the long-term care setting. Her current funded projects include a randomized trial of health services interventions to improve osteoporosis care for men, development and testing of an Injury Liaison Service for Nursing Homes, and translational studies of Physical Resilience in older adults after acute illness or injury.

Appendix

Table 2.

Mean responses (and standard deviation) for all domain items by Gender and Race/Ethnicity.

Domain	Race/Ethnicity				Pairwise differences by Race/Ethnicity Difference (95% CI Tukey’s HSD) ***p<0.05
	White	Asian	URN	p-value	Asian-White	Asian-URN	White-URN
Intellectual Growth and Development	3.56 (0.54)	3.58 (0.52)	3.53 (0.61)	0.71	0.0216 (−0.085, 0.128)	0.053 (−0.100, 0.2060	0.031 (−0.116, 0.178)
Professional Career Development	3.31 (0.66)	3.33 (0.71)	3.16 (0.80)	0.12	0.019 (−0.120, 0.158)	0.168 (−0.032, 0.367)	0.149 (−0.042, 0.341)
Academic Guidance	3.43 (0.56)	3.45 (0.59)	3.27 (0.65)	0.02*	0.027 (−0.088, 0.143)	0.188 (0.023, 0.354)***	0.161 (0.002, 0.320)***
Personal Communication	3.50 (0.58)	3.52 (0.56)	3.35 (0.65)	0.04*	0.023 (−0.092, 0.138)	0.172 (0.007, 0.337)***	0.149 (−0.009, 0.307)
Serves as Role Model	3.55 (0.60)	3.52 (0.60)	3.41 (0.70)	0.15	−0.036 (−0.157, 0.086)	0.104 (−0.071, 0.278)	0.139 (−0.028, 0.307)
Total scale	3.47 (0.55)		3.34 (0.63)		0.012 (−0.099, 0.122)	0.139 (−0.020, 0.298)	0.127 (−0.025, 0.280)