Current Orthopaedic Residency Letters of Recommendation Are Not Biased by Gender of Applicant

Background:

Letters of recommendation (LORs) are highly influential in the residency selection process. Differences in language and length of LORs by gender have been demonstrated for applicants applying to surgical residencies and fellowships. This had yet to be studied in orthopaedic surgery. Given the gender disparity in the field, we sought to investigate the impact of gender on orthopaedic residency applicant LORs. We hypothesized that differences in length and language would be present for women applicants as compared to men.

Methods:

LORs for 2019 to 2020 applicants who applied to a single academic institution were selected for review. Female and male applicants were matched by medical school attended and United States Medical Licensing Examination Step 1 score. LORs were analyzed using both qualitative and quantitative analyses. Letters were evaluated for their word count, presence of language terms, and frequency of language terms. A similar subgroup language analysis was performed for standardized LORs (SLORs).

Results:

Six hundred fifty-six applicants met the initial screening criteria—126 women and 530 men. After matching, 71 female applicants were paired with 111 male applicants. Word count was, on average, longer for female applicants. LORs for female applicants were more likely to contain language terms that characterized their ability, achievement, participation in athletics, awards received, fit, leadership, and personality traits. Of these terms, ability and participation in athletics were also found more frequently in LORs written for women. In addition, language characterizing technical skills was found more frequently in LORs of female applicants. Similar codes were found to be statistically significant in the SLOR subgroup analysis.

Conclusion:

This study highlights that current orthopaedic surgery residency LORs do not appear to be biased by applicant gender. LORs were longer for female applicants and described them more positively. Future female orthopaedic residency applicants should be assured that current female candidates are applying with at least similar if not greater subjective qualifications to their male counterparts based on the findings of this study.

Letters of recommendation (LORs) are highly influential in the residency selection process because they are ranked by surgery program directors to be the second most important factor used to select applicants for interviews—United States Medical Licensing Examination (USMLE) scores are first¹. Therefore, of the subjective (e.g., Dean’s letter and LORs) and relatively objective (e.g., USMLE scores and clerkship grades) components of the Electronic Residency Application Service (ERAS), LORs are the most highly weighted subjective piece of the application. Because of the subjective nature of LORs, the presence of gender bias is a real possibility and should be of concern. Studies in academic medicine have drawn on the social role theory of sex differences to evaluate gender bias in LORs^2-5. This theory highlights how men are generally described in agentic terms (i.e., descriptions of assertiveness, independence, and self-confidence) and women in communal terms (i.e., descriptions of sympathy, sensitivity, and nurturance). Previous work has shown that communal characteristics have a negative relationship with hiring decisions⁴. In addition to language differences by gender, studies have also revealed difference in letter length, which is relevant because longer letters have been associated with higher favorability in the trainee review process^6,7. Differences in language and length by gender have been demonstrated for applicants applying to surgical residencies and fellowships—general surgery, otolaryngology, and transplant surgery—outside the field of orthopaedic surgery^2,3,6,8,9.

The lack of gender diversity in orthopaedic surgery has been an increasing topic of discussion among the profession. With women comprising nearly 50% of medical students in the recent years¹⁰, orthopaedic surgery remains the medical specialty with the lowest proportion of female residents at around 14%¹¹. Over the years, the percentage of women in orthopaedics has increased, yet this percentage increase has lagged behind other male-dominated specialties¹¹. Barriers to improvements in gender diversity in the field are multifactorial but are thought to be related to the lack of female mentors in academic orthopaedic surgery and a lack of early exposure in the field^12,13. There is also the possibility that gender bias plays a role in widening the gender gap.

In this context, the purpose of this study was to evaluate LOR differences in length and language by gender for orthopaedic surgery residency applicants. Based on previous literature, we hypothesized that differences in length and language would be present for women applicants as compared to men.

Methods

We performed this study using data from the ERAS for the 2019 to 2020 application cycle. Candidates applying through ERAS to a single academic institution during the 2019 to 2020 application cycle were screened for their eligibility to be included in the study. Applicants were included if they were first-time applicants to orthopaedic surgery residency through ERAS, received a USMLE Step 1 score of 200 or greater, and were enrolled in a US medical school. Those who did not meet these criteria were excluded. ERAS provides 2 gender choices (male or female) for applicants; therefore, gender was considered a dichotomous variable in this study. After the initial screening, female applicants were matched with male applicants by medical school and USMLE Step 1 scores within 5 points. If there were several male applicants who attended the same medical school as a female applicant with a Step 1 score in the required range, they all were added to the analysis.

The letters from these matched applicants were downloaded from ERAS in portable document format (pdf) and then converted to Microsoft Word documents to remove applicant names, letter writer names, Association of American Medical Colleges numbers, and ERAS numbers from the LORs. Removal of applicant and letter writer information also included the removal of salutation and signature components from the letters. After the removal, LOR word count was determined using Microsoft Word’s Word Count function. Letters were reviewed and corrected for any conversion errors. Deidentified documents were imported into Dedoose version 8.3.17—web application for managing, analyzing, and presenting qualitative and mixed method research data (Los Angeles, CA: SocioCultural Research Consultants, LLC www.dedoose.com, 2020).

The following categories of language terms (codes) were selected a priori based on literature review and entered into Dedoose: ability^2,14,15, academic background², achievement^2,15, agency^2,4, authority established², awards², career choice², clinical skills^2,7, communal^2,4, community service², doubt raisers^2,6, family², fit², fund of knowledge^2,7, future promise², grindstone adjectives^2,14,15, hardship², initiative², judgment^2,7, leadership², legacy², personality traits^2,7, physical description², personal but general terms², presentations², receptive to feedback², recruitment², research^2,14,15, scholarship², standout adjectives^2,14,15/superlatives², teaching^2,14,15/mentoring², teamwork², technical skills^2,7, and work habits^2,7/work ethic². Qualitative codes were refined while reading through the LORs, and the codes that emerged were the following: athletics, strong support, top student, and ranked highly were added to the recruitment category. Codes are defined in Appendix 1.

Letters were evaluated for their presence of code use (dichotomous variable) and frequency of code use (continuous variable). They were reviewed and coded by 2 researchers (S.A.L. and N.E.G). Discussion and review of 10% of cases allowed for consensus in coding—a similar process to what has been described in other published studies on this topic². An identical language subgroup analysis was performed on letters associated with SLORs.

Descriptive statistics was used to evaluate letter writer factors. T-test was used for parametric data, and Wilcoxon rank sum test was used for nonparametric data. Chi-square test was performed for dichotomous outcomes. Stata Software (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC) was used to perform all statistical analysis. The study was determined to not require Institutional Review Board (IRB) approval after submitting it for internal IRB review.

Results

Applicants

Six hundred fifty-six applicants met the initial screening criteria—126 women and 530 men. After matching by medical school and USMLE Step 1 scores, 71 female applicants were paired with 111 male applicants. Statistical analysis showed no difference between mean USMLE Step 1 scores for men and women, 249.85 and 249.93, respectively (p-value 0.37). The number of scholarly studies was similar between the 2 groups. These included counts of peer-reviewed journal articles/abstracts, poster/podium presentations, and the composite of these scholarly studies (p-values 0.14, 0.13, and 0.12, respectively).

Letter Writers

There were 650 LORs from 51 institutions that were written in support of these applicants. Two hundred forty-six letters were written for female applicants and 404 for male applicants. For faculty, 73 (11.23%) letters were written by female faculty, 525 (80.77%) letters were written by male faculty, and 52 (8%) letters were written by a group. The breakdown of letter writers by faculty rank and role can be found in Table I. Of the 650 LORs, 458 (70.46%) are associated with a standard letter of recommendation (SLOR)—178 of these letters were written for women and 280 for men.

TABLE I.

Number of LORs by Faculty Gender, Role, and Rank

	No. of Letters (n = 650)
Letters written by female faculty by rank/role	n = 73 (11.23%)
Rank
Professor	20 (27.40%)
Associate professor	25 (34.25%)
Assistant professor	27 (36.99%)
No specified faculty rank	1 (1.37%)
Role
Chair	3 (4.11%)
Program director	10 (13.70%)
Letters written by male faculty by rank/role	n = 525 (80.77%)
Rank
Professor	216 (41.14%)
Associate professor	167 (31.81%)
Assistant professor	125 (23.81%)
No specified faculty rank	17 (3.24%)
Role
Chair	118 (22.48%)
Program director	68 (12.95%)
Group letters	n = 52 (8.00%)

Letters

The average number of LORs for men and women was similar at 3.70 and 3.74, respectively (p-value 0.83). Average word count was significantly longer for female applicants than their male counterparts—a mean of 316.83 for women and 288.45 for men (p-value 0.04).

Certain differences in language were found when comparing LORs for male and female applicants (Tables II–V). When looking at language codes for all LORs from a dichotomous standpoint (presence vs absence in a letter), ability, achievement, participation in athletics, awards, fit, leadership, personality traits, and presentations were more likely to be mentioned in LORs for female applicants than male applicants (Table II). In addition, ability, athletics, leadership, and technical skills were mentioned more frequently for female applicants (Table III).

TABLE II.

Codes Counted in All LORs by Applicant Gender Listed by Presence of Code

Code	Total Frequency Count	Applicant		p
		Female Frequency Count (n = 178)	Male Frequency Count (n = 280)
Ability	403	188	215	0.04
Academic background	135	60	75	0.10
Achievement	189	84	105	0.04
Agency	174	77	97	0.32
Athletics	98	54	44	<0.001
Authority established	31	11	20	0.69
Awards	142	72	70	0.03
Career choice	6	4	2	0.33
Clinical skills	284	127	157	0.16
Communal	193	92	101	0.10
Community service	80	28	52	0.64
Doubt raiser	28	11	17	0.89
Family	2	1	1	0.75
Fit	46	26	20	0.04
Fund of knowledge	198	78	120	0.79
Future promise	380	160	220	0.37
Grindstone	439	190	249	0.12
Hardship	13	7	6	0.28
Initiative	129	62	67	0.07
Judgment	5	2	3	0.96
Leadership	150	71	79	0.04
Legacy	10	4	6	0.73
Personality traits	231	110	121	0.04
Physical description	7	2	5	0.57
Positive but general	380	157	223	0.42
Presentations	131	65	66	0.008
Receptive to feedback	39	20	19	0.18
Recruitment/ranked highly	239	108	131	0.22
Research	326	125	201	0.64
Scholarship	147	56	91	0.46
Standout adjectives/superlatives	387	178	209	0.11
Strong support	236	94	142	0.96
Teaching/mentoring	55	20	35	0.64
Teamwork	164	71	93	0.41
Technical skills	130	65	65	0.10
Top student	156	72	84	0.24
Work habits/ethic	213	80	133	0.66

TABLE III.

Codes Counted in All LORs by Applicant Gender Listed by Frequency

Code	Total Frequency Count	Applicant		p
		Female Frequency Count (n = 246)	Male Frequency Count (n = 404)
Ability	641	283	358	0.03
Academic background	216	87	129	0.44
Achievement	295	126	169	0.05
Agency	237	107	130	0.09
Athletics	142	78	64	<0.001
Authority established	49	19	30	0.89
Awards	221	105	116	0.05
Career choice	10	5	5	0.78
Clinical skills	400	177	223	0.06
Communal	269	122	147	0.15
Community service	113	40	73	0.79
Doubt raiser	36	15	21	0.91
Family	2	1	1	0.72
Fit	63	31	32	0.09
Fund of knowledge	278	109	169	0.86
Future promise	577	233	344	0.56
Grindstone	646	267	379	0.16
Hardship	21	13	8	0.07
Initiative	193	88	105	0.07
Judgment	8	4	4	0.78
Leadership	209	98	111	0.04
Legacy	15	6	9	0.47
Personality traits	338	150	188	0.11
Physical description	7	2	5	0.61
Positive but general	571	235	336	0.23
Presentations	205	93	112	0.05
Receptive to feedback	48	25	23	0.07
Recruitment/ranked highly	341	152	189	0.08
Research	486	185	301	0.78
Scholarship	238	89	149	0.26
Standout adjectives/superlatives	582	260	322	0.06
Strong support	369	139	230	0.94
Teaching/mentoring	93	41	52	0.34
Teamwork	223	95	128	0.26
Technical skills	180	92	88	0.007
Top student	220	97	123	0.26
Work habits/ethic	321	116	205	0.54

TABLE IV.

Codes Counted in SLORs by Applicant Gender Listed by Presence of Code

Code	Applicant		p
	Female Letter Count (n = 178)	Male Letter Count (n = 280)
Ability	102	146	0.28
Academic background	60	74	0.10
Achievement	67	78	0.03
Agency	59	80	0.30
Athletics	50	39	<0.001
Authority established	11	20	0.69
Awards	46	49	0.03
Career choice	3	2	0.33
Clinical skills	94	133	0.27
Communal	64	83	0.16
Community service	27	47	0.65
Doubt raiser	9	15	0.89
Family	1	1	0.75
Fit	23	20	0.04
Fund of knowledge	64	106	0.68
Future promise	116	172	0.42
Grindstone	113	153	0.06
Hardship	6	5	0.28
Initiative	51	61	0.10
Judgment	2	3	0.96
Leadership	52	59	0.05
Legacy	3	6	0.73
Personality traits	80	99	0.04
Physical description	2	5	0.57
Positive but general	99	142	0.31
Presentations	57	58	0.007
Receptive to feedback	16	16	0.18
Recruitment/ranked highly	87	125	0.38
Research	104	169	0.68
Scholarship	36	66	0.40
Standout adjectives/superlatives	93	131	0.25
Strong support	76	119	0.97
Teaching/mentoring	20	27	0.58
Teamwork	57	80	0.43
Technical skills	54	66	0.11
Top student	54	72	0.28
Work habits/ethic	69	111	0.85

TABLE V.

Codes Counted in SLORs by Applicant Gender Listed by Frequency

Code	Applicant		p
	Female Letter Count (n = 246)	Male Letter Count (n = 404)
Ability	149	221	0.14
Academic background	86	129	0.43
Achievement	96	126	0.04
Agency	81	107	0.08
Athletics	71	60	<0.001
Authority established	19	30	0.89
Awards	64	79	0.05
Career choice	3	6	0.78
Clinical skills	129	185	0.10
Communal	87	126	0.27
Community service	39	67	0.81
Doubt raiser	12	19	0.92
Family	1	1	0.72
Fit	28	30	0.09
Fund of knowledge	91	149	0.98
Future promise	166	261	0.45
Grindstone	153	232	0.23
Hardship	10	7	0.07
Initiative	70	92	0.10
Judgment	3	4	0.78
Leadership	70	86	0.04
Legacy	4	10	0.47
Personality traits	107	151	0.12
Physical description	2	5	0.61
Positive but general	145	219	0.24
Presentations	81	105	0.06
Receptive to feedback	21	20	0.07
Recruitment/ranked highly	122	178	0.17
Research	150	255	0.58
Scholarship	53	106	0.18
Standout adjectives/superlatives	135	199	0.16
Strong support	114	185	0.90
Teaching/mentoring	33	44	0.33
Teamwork	76	109	0.28
Technical skills	77	89	0.009
Top student	73	105	0.31
Work habits/ethic	94	161	0.68

When comparing the subgroup of letters associated with SLORs for male and female applicants, achievement, participation in athletics, awards, fit, personality traits, and presentations were more likely to be mentioned in SLORs (Table IV). In addition, achievement, participation in athletics, leadership, and technical skills were found to be mentioned more frequently for female applicants (Table V). All other language terms were not significantly different in their gender distribution.

Discussion

LORs are highly influential in the orthopaedic residency selection process¹. Studies in academic medicine have shown differences in language and length by gender^2-4,6,8,9. There have been no studies in the field of orthopaedic surgery that have evaluated the impact of applicant gender on LORs; therefore, we sought to evaluate the influence of applicant gender on residency LOR length and language.

In our study, we found the LOR word count to be longer for female applicants. We also found significant differences in the language used to characterize male and female applicants. LORs for female applicants were more likely to contain language terms that characterized their ability, achievement, participation in athletics, awards received, fit, leadership, and personality traits. Of these terms, ability and participation in athletics were also found more frequently in LORs written for women. In addition, language characterizing technical skills was found more frequently in LORs of female applicants. When looking at these language terms in the SLOR subgroup analysis, many of the same codes were found to be significant in both presence and frequency.

The SLOR subgroup analysis was performed to determine if there were meaningful differences when the SLOR was evaluated outside of traditional LORs. SLORs were recommended for orthopaedic surgery applications by the American Orthopaedic Association Council of Orthopaedic Residency Directors in 2017 in an effort to standardize the assessment and allow for a more meaningful comparison of orthopaedic applicants. Given their recent introduction into the application process, there have been few studies evaluating their effectiveness⁸. There has yet to be any comparison between traditional LORs and SLORs in the orthopaedic literature. A study in the otolaryngology head and neck surgery literature compared SLORs with traditional LORs for language and gender, and they found that female applicants were less likely to be described as “bright” and more likely to have their appearance mentioned in traditional letters than SLORs⁸. For these reasons, we performed a subgroup analysis, which resulted in very similar findings to that of LORs as a composite (Tables IV and V).

Regarding language, the results of our study show some similarities and notable differences when compared with related studies in academic medicine^2-4,6,8,9. A study by Madera et al. evaluated the differences between LORs for junior faculty members applying to a single institution⁴. This study drew from the social role theory of sex differences in that men are generally described in agentic terms and women in communal terms. Their study supported their hypothesis and found that women were more described as communal and less agentic than men and that communal characteristics had a negative relationship with hiring decisions in academia⁴. In the current literature evaluating trainee LORs written for applicants to various surgical specialties, most studies have found no differences in the use of agentic or communal terms based on gender^2,16. Similarly, our study found no difference in the use of these terms. In addition, the intent to recruit an applicant to stay at the writer’s institution for residency—viewed as one of the most positive factors in surgical LOR⁷—appeared equally in letters for men and women. This finding was similar to a study by Turrentine et al. who compared general surgery applicant LORs and gender².

Unlike most studies evaluating surgical trainee LORs and gender, our study found differences in language that overall characterized women positively for ability, achievement, awards received, and leadership. This contrasts with the study by Turrentine et al. who found that male applicants were more likely to be described by these terms. The same study also found that women were more likely to be described by their physical description. There have been several other studies in the surgical literature, showing that female applicants are more likely to be described by their physical characteristics than male applicants^2,8,9. To the contrary, we found no difference between the 2 groups regarding physical description.

Importantly, descriptions of a good fit were more likely to be mentioned in letters for female than male applicants although overall counts were similar. This may be of particular importance given the current times where medical students will be participating less in away rotations for the foreseeable future secondary to the global pandemic. The literature on orthopaedic away rotations has highlighted that program directors value away rotations for finding a “good fit”¹⁷. In times like these, where personal interactions are limited, descriptions of fit in LORs could become more important—particularly when an applicant does not represent the majority, making assumptions of good fit less likely to be made.

For letter length, our study found that residency LORs for women were longer. Turrentine et al. observed that general surgery residency LORs were on average longer for men, whereas French et al. found no difference^2,15. Previous work has associated longer letter with higher favorability in the review process^6,7.

Although the reason behind longer letters and more positive language used to describe female candidates during this application cycle is beyond the scope of this study, this may either accurately reflect the applicant pool or represent more attention being paid to equity for female applicants. In relation to the former, it is very plausible that imposter syndrome plays a role for female applicants applying into orthopaedic surgery. This phenomenon is when an individual doubts his/her competence and has persistent fear of being exposed as a fraud despite objective evidence that he/she is capable of. In particular, for women, the literature has shown that women are less likely to view themselves as qualified despite being similarly qualified to their male counterparts^18,19. In this sense, the orthopaedic applicant pool could be affected by imposter syndrome.

In addition to the inability to identify reasons for language differences, this study has several other limitations. We analyzed LORs from a single application cycle to a single academic institution and matched the candidates, which does exclude some applicants. Despite this, we do feel that our sample was relatively representative of the applicant pool. We used a low threshold for our initial screen (USMLE Step 1 score of 200) to try to minimize the exclusion of applicants. Matching was used to limit confounding although we do realize that this could also lead to selection bias. There were several factors that could have been chosen for matching applicants, but we specifically chose medical school and USMLE Step 1 score because these are more objective factors than clerkship grades, Alpha Omega Alpha status, and the Medical School Performance Evaluation, which have all been suggested to be influenced by bias^15,20-22. We did not control for scholarly endeavors, such as publications or presentations, a priori; nevertheless, these were found to be similar between the 2 groups in our analysis. Pronouns were not removed from the letters, so coder bias is a also consideration; however, this does not appear to be the case, as the letters were reviewed by 2 female authors, and the findings are actually contrary to what the authors hypothesized based on previous literature. The authors also recognize that there are individuals who do not identify with the binary gender system of male and female participants. The binary approach to gender was used in this study given that ERAS only provides these 2 gender choices for applicants and letter writers.

For future direction, a study evaluating the perceived qualifications to apply into orthopaedic surgery and differences in this perception based on gender could be conducted. This study would be interesting to compare with the general body of literature on gender and job applications. In addition, an analysis of the impact of gender on LORs over several years would be interesting for the evaluation of trends in language and word count.

Conclusion

This study highlights that current orthopaedic surgery residency LORs do not appear to be biased by applicant gender. LORs were longer for women and described female applicants positively for their abilities, achievements, leadership, and good fit in orthopaedic surgery, to name a few. This is unlike previous studies in academic medicine that have shown female applicants to be characterized by communal terms^3,4 and their physical characteristics^2,8,9. Future female applicants to orthopaedic surgery residency should be assured that current female candidates are applying with at least similar subjective qualifications to their male counterparts based on the findings of this study.

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A274). This content was not copy-edited or verified by JBJS.