The Role of Gender in Neurosurgical Residency Applicants’ Letters of Recommendation

Abstract

BACKGROUND

Letters of recommendation (LORs) are one of the most important components of the neurosurgical residency application. Studies in other fields and surgical subspecialties have found gender bias.

OBJECTIVE

To determine whether neurosurgical LORs contain significant linguistic gendered differences.

METHODS

We performed a retrospective review and linguistic analysis of all LORs submitted on behalf of applicants offered an interview invitation to a single neurosurgical residency program at an academic medical center between 2015-2016 and 2018-2019.

RESULTS

A total of 599 letters from 156 applicants (120 males and 36 females) were included. Background demographics, including United States Medical Licensing Examination (USMLE) Step 1 score and publications, were not significantly different between applicants. Male faculty authored 93% of all letters. Female faculty were more likely to write letters for female applicants than male applicants (12.1% vs 5.5%, P < .001). Letters for women were significantly longer (334 words vs 277 words, P < .001). Overall, 1754 agentic terms and 854 communal terms were coded. Letters for men contained significantly fewer agentic terms (10.7 vs 13.1, per applicant, P < .01) and communal terms (5.2 vs 6.4, P < .034). This difference, however, is nonsignificant when word count was used as a covariate. Female applicants were more significantly likely to be labeled “Outstanding” (2.4 vs 1.6 mentions per applicant).

CONCLUSION

Overall, there are more similarities than differences between genders in LORs for neurosurgical applicants. This finding is at odds with what has previously been reported in most other surgical specialties.

ABBREVIATIONS

ERAS

Electronic Residency Application Service

LOR

Letters of recommendation

SEM

standard error of the mean

USCMLE

United States Medical Licensing Examination

WINS

Women in Neurosurgery

Letters of recommendation (LORs) play an important role within the neurosurgical residency application. Multiple surveys of neurosurgeons have found that they are an integral component to the interview invitation and compiling the National Resident Matching Program rank list,^1,2 ranking just behind United States Medical Licensing Examination (USMLE) Step 1 scores. With the upcoming conversion of Step 1 to a pass/fail examination, LOR may become the foremost tool in the ranking process outside the interview.

Within academia, it has been demonstrated that differences exist between the language used to describe male and female job applicants and that it may affect hiring.^3,4 There is underlying societal bias. More recently, research in surgical residency and fellowship programs has demonstrated significant gendered differences in the terminology used to describe female and male applicants.^5-10 Agentic terms or languages are often used to describe an applicant's ability and refer to goal achievement or task functioning, whereas communal phrases are used to describe their concern for or relationships with others. In other medical specialties, male applicants are more commonly described with agentic terminology, whereas female applicants are more likely to described using communal language.

Within the field of neurosurgery, female applicants represent a small minority (approximately 14%) of total applicants despite being over 50% of graduating medical students. Furthermore, female gender has been shown to decrease the odds of matching in neurosurgery by nearly 50% in a multivariate analysis that controlled for USMLE Step 1 score, medical school rank, and alpha omega alpha status.¹¹ This disparity has been acknowledged by the national societies.^12-14 In 2008, there was a call to action in organized neurosurgery to attain a 20% female residency complement and a 20% female faculty complement by 2020.¹² However, in 2020, females made up only 10% of practicing academic neurosurgeons in the United States. This disparity is furthered by the lack of women in leadership roles within our field; as of 2020, only 4 women have served as chair of neurosurgical departments at United States medical schools, and women neurosurgeons have held just 1% of officer positions within the top 7 United States neurosurgery organizations in their 89-yr history.¹⁵

We sought to analyze LOR for neurosurgery applicants to determine if there were any gendered differences that have been seen in other specialties that may be reflective of a wider issue of implicit bias in residency selection.

METHODS

A retrospective review of applications submitted via the Electronic Residency Application Service (ERAS) to a single academic institution between 2015-2016 and 2018-2019 was performed. The project was given exempt status by the Institutional Review Board. The Association of American Medical Colleges allows for the use of ERAS application data by residency programs for noncommercial purposes such as research.¹⁶ Our interview selection process involves 2 faculty reviewing each application to make a determination. When consensus is not reached, the program director makes a final decision. We excluded applications of those who were not offered interview invitations. Applicant's name and other identifying data were removed. Deidentified demographic information was collected about applicants, including gender, clerkship grades, peer-reviewed journal publications, and USMLE Step scores. Within the ERAS applications from this period, the only response options for gender were “male” and “female”; applicants selected 1 of these 2 options, and thus, such a binary division was used for the current analysis. The authors realize that no such binary exists, and self-reporting of gender would better reflect the current understanding that gender is not a binary.

Authorship information was collected about the author for each LOR, including gender, institution, and job title. The LORs were scanned into plain text format using FreeOCR (http://www.paperfile.net/). Salutations, closings, and Family Education Rights and Privacy Act statements were removed. Letters were then proofread by the authors and typographical corrections were made when appropriate. A dictionary of agentic and communal codes was created based on previous research within the field (Table 1).^4,9 In addition, the authors identified a number of terms that were frequently used throughout the letters and created a separate word bank for these terms. The LORs were coded using QDA Miner v5 (Provalis Research, https://provalisresearch.com/products/qualitative-data-analysis-software/). Over 10% of the codes were reviewed by both authors to assure consensus in coding. A final review was performed of every code by NF. Quantitative and descriptive analysis was performed in QDA Miner and SPSS Statistics for Windows, Version 25 (IBM Corp, Armonk, New York).

TABLE 1.

Dictionary of Agentic and Communal Terms

Agentic terms	Communal terms
Superb	Affectionate
Excellent	Sympathetic
Outstanding	Nurturing
Assertive	Warm
Dominant	Thoughtful
Forceful	Delightful
Exemplary	Compassionate
Confident	Friendly
Leader	Kind
Strong	Husband/wife/spouse
Efficient	Children
Problem solver	Empathetic
Intelligent	Team player
Solid	Easy to work with
Bright	Well-liked
Excel	Communication skills
Exceptional	Conscientious
Rising star	Honest
Superior	Humble
Well-rounded	Calm
Bright future	Congenial

Reprinted from J Surg Education, 76(2), Hoffman A, Grant W, McCormick M, et al, Gendered differences in letters of recommendation for transplant surgery fellowship applications, pp. 427-432, Copyright 2018, with permission from Association of Program Directors in Surgery.

RESULTS

A total of 156 applicants from the 2015-2016 to 2018-2019 ERAS cycles, 120 males and 36 females, were included in the study. Between them, 599 LORs were collected and analyzed. Female applicants averaged 3.9 LORs per application and male applicants averaged 3.8 LORs. Overall, the male and female applicants were extremely well matched. The mean age was 27 (standard error of the mean (SEM) = 0.2) and 26.9 (SEM = 0.3) yr for men and women, respectively; USMLE Step 1 score was 244.1 (SEM = 1.2) vs 244.2 (SEM = 1.9), USMLE Step 2 score was 250.1 (SEM = 1.2) vs 250.6 (SEM = 1.9), and total publications were 4.3 (SEM = 0.6) vs 4.0 (SEM = 0.8). The complete results are summarized in Table 2.

TABLE 2.

Characteristics of Neurosurgical Residency Applicants

Variable	Gender	N	Mean	SEM	P-value
Age	Male	120	27.0	0.2	.864
	Female	36	26.9	0.3
AOA	Male	120	0.3	0.0	.843
	Female	36	0.3	0.1
First author publications	Male	120	1.5	0.3	.317
	Female	36	1.0	0.2
Total publications	Male	120	4.3	0.6	.838
	Female	36	4.0	0.8
Clerkship honors	Male	120	2.9	0.2	.984
	Female	36	2.9	0.4
Step 1	Male	120	244.1	1.2	.954
	Female	36	244.2	2.0
Step 2	Male	93	250.1	1.2	.844
	Female	34	250.6	1.9

AOA, Alpha omega alpha.

Overall, male faculty authored 557 LORs (93%), whereas female faculty wrote 42 LORs (7%). There were 23 unique female authors (4 department chairs and 12 full professors) and 251 unique male authors (113 department chairs and 192 full professors). Letter writers for women applicants were more likely to be female, whereas applicants who were men had statistically fewer LORs from female neurosurgeons (12.1%, n = 17 vs 5.5%, n = 25; P < .001; Table 3). Authors for female applicants were also more likely to be from the applicants’ home institution (47.9% vs 39.1%, P < .001). The authors’ titles were similar between male and female applicants. Department chairs wrote 64% of LORs for female applicants and 61% of LORs for male, program directors wrote 8% and 13%, and research faculty wrote 4% and 2%, respectively. Other faculty (primarily neurosurgeons acting as applicant mentors) wrote 24% of LORs for both men and women. LORs written for female applicants were significantly longer than those written for their male counterparts (334.3 words vs 277.5 words, P < .001). This trend held true regardless of the authors’ gender (Table 4).

TABLE 3.

Characteristics of Letter Authorship

Applicant gender	Author gender	No. of letters, % of gender	Mean length (words)	SEM	P-value
Male	Male	434, 95%	276	6	.063
	Female	25, 5%	325	39
Female	Male	123, 88%	321	17	.034
	Female	17, 12%	425	48

TABLE 4.

Coding Frequency by Gender

Code	Instances	Average	STD	SEM	P-value
Agentic
Total	1754
Male	1281	10.7	4.4	0.4	.009
Female	473	13.1	6.2	1.0
Communal
Total	854
Male	623	5.2	3.0	0.3	.034
Female	231	6.4	3.1	0.5

Overall, there were 1754 instances of agentic terms coded and 854 instances of communal terms coded. Letters for men contained significantly fewer agentic terms (10.7 vs 13.1, per applicant, P < .01) and communal terms (5.2 vs 6.4, P < .034) than letters for women. However, this difference was not sustained once difference in word count was accounted for (Table 5). A number of the individual codes had statistically significant differences between male and female applicants, including “Outstanding” (1.6 average instances per male applicant vs 2.4 average instances per female applicant), “Rising Star” (0.1 vs 0.3), “Communication Skills” (0.2 vs 0.4), and “Humble” (0.2 vs 0.4) (Table 5). The additional terms identified by the authors, “Hard Worker,” “Professional,” and “Work Ethic” showed no significant difference between genders. A secondary analysis comparing the terminology used by male and female authors, regardless of recipient gender, was also performed. Male authors were significantly more likely to use the communal term “well-liked” (121 instances vs 2, or 4.7% of total terms used vs 1.1%, P = .012) and female authors were more likely to use the communal term “thoughtful” (11 instances vs 54 or 5.9% vs 2.1% of total terms used, P < .001). There were no significant differences in the use of agentic terms by male or female authors.

TABLE 5.

Statistically Significant Codes by Gender (Greater Than 20 Total Codes)

Code	Instances (M, F)	Average (M, F)	P-value
Outstanding	187, 88	1.6, 2.4	.005
Rising Star	10, 10	0.1, 0.3	.008
Communication Skills	21, 14	0.2, 0.4	.028
Humble	20, 15	0.2, 0.4	.012

An additional analysis was performed to compare the language used in LORs from home and away institutions. LORs from home institutions were more likely to describe applicants as “leaders” and “exceptional” (P < .05); however, when we factored in applicant gender, we did not see statistical differences in either of these terms.

DISCUSSION

Our study is the first to evaluate gender differences in neurosurgical residency applicants’ LORs. Contrary to the findings in most other surgical specialties, we found little evidence of systemic male gender bias between the language used in LORs for female and male applicants. In fact, LORs for female neurosurgical applicants contained significantly more usage of both agentic and communal terms when compared to their male counterparts; however, these differences were small. The only commonly utilized agentic term (16% of all instances of agentic terms) that was used with a statistically significant higher frequency in LORs for female applicants compared to males was “Outstanding.” Although other terms showed significance, their overall use was low. Additionally, LORs written for female applicants were significant longer, a finding that is only replicated in the orthopedic surgery literature.¹⁷ These differences were seen across both male and female LOR authors. One possible explanation is that neurosurgery letter writing faculty are more cognizant of an implicit gender bias and have sought to mitigate it. This explanation may also account for the longer LORs written for female applicants.

We did observe other small differences between the LOR sources. Female applicants were more likely to receive letters from their home institution than their male counterparts, and their LORs were also more likely to be written by female authors. Both of these findings may be the result of female applicants selecting female mentors within the neurosurgical field, which has been championed by Women in Neurosurgery (WINS).¹⁴ Despite this, less than half of female applicants received letters from female mentors. Medical students are commonly advised to seek a letter from the department chair while on away rotations. Given the paucity of WINS leadership positions and ERAS letter submission limits, this practice may disadvantage female applicants who wish to include letters from female mentors at their home institution. Future research looking at the differences between letters written for female applicants by home and away institutions may be beneficial.

The first evaluation of gender bias in LORs in medicine was performed by Trix and Psenka.¹⁸ They found that the LORs written for female medical school faculty applicants were shorter and more likely to include negative language. Gender bias in LORs has previously been evaluated in urology, radiology, otolaryngology, emergency medicine, general surgery, transplant surgery, and pediatric surgery.^{5,6,8-10,19-22} In 2008, Messner and Shimahara²² extended the study evaluating faculty letters to otolaryngology residency applicants. The authors found an increased likelihood of gender term usage in letters for male applicants as well as increased likelihood of commenting on a female applicant's appearance when the letter writer was male. Urology LORs for male applicants have been shown to contain significantly more “power words,” references to drive, power, and work constructs, compared to letters for female applicants. The authors concluded that the gender-specific linguistic differences represented an example of implicit or unconscious gender bias that extends to the application process as a whole.⁵ LORs for male transplant surgery fellowship applications were found to be significantly more likely to contain agentic terms (superb, intelligent, and exceptional), whereas letters for female transplant applicants were more likely to contain communal terms (compassionate, calm, and delightful).⁹ In general surgery, LORs for female applicants were significantly shorter than those for male applicants and comments related to ability and leadership were found more significantly in letters for male applicants. Superlatives were used more frequently in LORs for male applicants and doubt raisers and physical description were mentioned more frequently in female LOR.¹⁰ An analysis of LOR for pediatric surgery found that more agentic terms were used for men than for women.⁸ However, most recently, a study of LOR in orthopedics found little difference between the language used to describe male and female applicants. The authors felt this was due to a conscious effort on the part of letter writers.¹⁹

Previous work has shown that when information gaps exist within a job candidate's application, the hirer will unconsciously use gender stereotypes to manufacture the missing details.²³ Methods to decrease implicit gender bias within the hiring process exist. A randomized trial at the University of Wisconsin used a gender-bias-habit-changing workshop and found that participants were more likely to engage in gender equity promoting behaviors.²⁴ Recently introduced by academic neurosurgery in response to the COVID-19 pandemic, standardized LORs, and additionally structured interviews, have been shown to reduce the opportunity to insert bias.^20,25-27

Limitations

Our study does contain several limitations. We only evaluated LORs of applicants who applied to and were subsequently offered an interview invitation at a single residency program, a methodology consistent with the literature.^6,10 It is possible that we are missing a subset of applicants who did not apply to our geographical region (northeast) or that our program has an underlying bias in our applicant selection progress. Overall, during the study period, our female faculty comprised 11% to 27% of the department. Future studies may benefit from evaluating every applicant from a single application cycle by collaborating with other residency programs. Additionally, the time frame of our study allows us to draw recent conclusions, but it does not allow us to look for trends over a longer period. ERAS limits the availability of prior application data to applicants who applied to the requesting program during the preceding 5 yr. It is possible that letter authors have biases that have changed over the past decade that we cannot account for or identify. In addition, such a small proportion of female applicants may make minor differences difficult to identify, although our sample (19% women) is higher than that of all incoming neurosurgical residency classes during our study period.²⁸ Finally, although all letters were meticulously reviewed, it is possible that some errors in letter scanning and analysis persisted through the quality checks, which led to some terms being missed, although we would expect these errors to be equally distributed across genders.

CONCLUSION

The language of neurosurgical residency applicant LORs contains more gendered similarities than differences. Overall, the differences are small and likely do not play a major role in the evaluation of candidates. However, these changes do represent a shift when examined in the context of the medical literature, which previously demonstrated a consist gender bias. Future research may indicate whether this is a conscious change on the part of letter writers in response to the overall strategic plan to address gender inequity. Further, the inclusion of match outcome data may demonstrate gender and linguistic differences between applicants who do or do not match.

Funding

This study did not receive any funding or financial support.

Disclosures

The authors have no personal, financial, or institutional interest in any of the drugs, materials, or devices described in this article. Dr Pilitsis is a consultant for Boston Scientific, Nevro, TerSera, Medtronic, Saluda, and Abbott and receives grant support from Medtronic, Boston Scientific, Abbott, Nevro, TerSera, NIH 2R01CA166379-06, and NIH U44NS115111. She is also a medical advisor for Aim Medical Robotics and Karuna and has stock equity.