Editor’s Spotlight/Take 5: Are There Gender-based Differences in Language in Letters of Recommendation to an Orthopaedic Surgery Residency Program?

Most of us, I believe, value similar qualities in the young people who apply to orthopaedic residency programs. Apart from tight adherence to important norms of our profession—things like honesty and integrity—we might seek someone who is dependable, someone who possesses a strong work ethic, and someone with enough confidence that he or she may someday have the chutzpah to cut into the flesh of another person.

The problem is that when we hang traits around the necks of people of different genders, the same traits are not always similarly becoming [17]. Words like “assertive”, “ambitious”, “direct”, and “confident” may look different to some evaluators when applied to young men than they do when applied to young women. Perhaps more importantly, we have a tendency to choose different words when describing men and women. He might be “driven”, but she “hard-working”; he might be “a real leader”, and she might be just a little “too strong-willed” (or worse, “bossy”). And even some traits that may sound good, like “helpful”, “reliable”, “delightful”, have consistently been associated with gender stereotypes [14, 15] that decrease women’s chances of workplace success.

This month’s Clinical Orthopaedics and Related Research® features a large symposium called Diversity and Disparities in Orthopaedic Surgery, in which authors explored a range of musculoskeletally relevant sex and gender differences. Covered topics include sports injuries [10], pain patterns [16], professional compensation [3], participation on the editorial boards of leading journals [13], and many others, including some that evaluate critical healthcare disparities unrelated to sex or gender [19].

But one paper in particular struck me as both important and surprising. The title of the paper (“Are There Gender-based Differences in Language in Letters of Recommendation to an Orthopaedic Surgery Residency Program?”) all-but predicted its own answer to be “yes”, as might a close review of the numerous earlier studies in other specialties on the same topic [2, 4, 5, 11, 14]. But this thoughtful project [6] from a multidisciplinary group led by Casey Jo Humbyrd MD—her second appearance in the Editor’s Spotlight section of CORR® [9, 18]—delivered an unexpected answer: Other than some small and subtle differences, some of which favored women over men, letters of recommendation written for men and for women were fairly similar.

Casey Jo Humbyrd MD

Their work was based on a robust linguistic analysis of nearly 2300 letters for more than 700 applicants to the Johns Hopkins orthopaedic residency program during a recent application cycle. While one may wonder whether the findings apply more broadly, I believe they probably do. Most candidates now apply to a large number of orthopaedic programs in a match cycle, so many of the same letters are seen by many, many programs.

This was one of a few papers in the Diversity and Disparities symposium that suggested that some problems related to gender differences in our specialty have been ameliorated [6, 13]. Pointing to areas of improvement is as important as pointing out areas of persistent unfairness, and I honor those authors who did both in this symposium. But it would be naïve to think that sexism has been banished from the residency selection process.

Please join me in the Take 5 interview that follows with Dr. Humbyrd, in which we go behind the discovery to see what is working well and what areas of unfairness persist in the residency match.

Take 5 Interview with Casey Jo Humbyrd MD, senior author of “Are There Gender-based Differences in Language in Letters of Recommendation to an Orthopaedic Surgery Residency Program?”

Seth S. Leopold MD: Congratulations on this well-conceived and well-executed study. The results surprised me; the ratio of men to women in orthopaedic surgery is more lopsided than it is in any other specialty [8]. How do we reconcile that with the findings of your study?

Casey Jo Humbyrd MD: Thank you for the kind compliments on the paper. We agree that women are underrepresented in orthopaedic surgery; in 2019, women represented only 15.2% of orthopaedic residents [1]. In our study population, the proportion of women was greater than that (about 20% of our applicant pool was women). This, in fact, supports previous studies showing that women and minority applicants seek out programs with better representation [7]. Certainly, this has been the case at our institution. While this hasn’t been the focus of our study, I think the pipeline issue remains—that is, even if the selection process isn’t biased, we don’t have enough women applying to the programs, because we don’t have the specialty pipeline many other fields have.

Dr. Leopold: Even on the specifics, some of the differences you identified—including the length of the letters, and the number of “achieve” words—favored women, while there were no differences in things like “grindstone” words (terms that focus on a candidate’s willingness to work or support, but that avoid commenting about a candidate’s ability), all of which contrasts with a great deal of prior research about this kind of sexism, including recent research in surgery [14]. To what degree should findings like yours and some others in this Symposium [13] cause orthopaedic surgeons to congratulate ourselves for having solved an important problem in our specialty?

Dr. Humbyrd: We were somewhat surprised as well that the results of our study did not show differences in word frequencies that have been seen before. The findings from our analysis showed that language used to describe women and men applicants was largely similar even when stratified by author gender or academic rank. We theorized that there may have been a homogenization of letters in orthopaedic surgery that may be driven by a desire to combat the known gender disparity, or a desire to focus on more-objective attributes. It is also possible that letters have become more “cookie cutter,” and that there is less content specific to the particular applicant. I think we should proceed cautiously in overinterpreting the results of our study. Word frequency is only one dimension of language and does not fully encompass all aspects of gender inequality. Therefore, it’s possible that using other methods of language analysis may uncover other differences in language.

Dr. Leopold: Writing the letter is only half the battle; the other half is reading it. Sometimes, the very same adjectives and descriptors don’t come across as favorably when applied to a woman as they might when describing a man. How can letter writers avoid this trap so as to represent their candidates well?

Dr. Humbyrd: We agree, and this further supports the notion that language and bias are complicated, nuanced issues. For example, there’s evidence that the word “ambitious” is seen as complimentary for a man while being a critique of a woman [12]). We believe that letter writers should continue to focus on specific objective attributes (“she pre-rounded on patients”) about a candidate rather than overly relying on subjective measures (“she has a nice personality”, “she is hard working”) of a candidate. That being said, we approach a fine line as letters of recommendation are supposed to add that “third dimension” to an applicant’s file, and so readers may desire some of these more-subjective kinds of comments. But we need to read them with an understanding that they may contain inherent bias.

Dr. Leopold: Can you give an example of a short paragraph from a letter that you’ve seen, and how you’d rewrite it for better effect in this regard?

Dr. Humbyrd: Here is an example of a paragraph that, while complementary, quotes others and does not really provide a detailed picture of the candidate: “Jordan is a fantastic student with endless potential. She served as a valuable team player and has an excellent work ethic. She is even keeled and well-liked by everyone. She has her head on straight and is very adaptable to any situation.”

By contrast, this paragraph paints a much more-specific picture of the same candidate by providing supporting examples: “Jordan is a fantastic student. She made herself a valuable member of the team by taking it upon herself to come to the hospital early and pre-round on patients to help facilitate morning rounds. The nursing staff told me about how she seemed ever-present in the operating room and available to help facilitate getting patients into the operating room and positioned for surgery.”

Dr. Leopold: You say that “it would be a mistake to solely rely on standardized scales to describe an applicant’s characteristics. Important information can be gleaned from letters of recommendation and perhaps efforts should be dedicated to mandating the format of letters more rather than leaving it up to authors on whether to use a fully standardized or fully narrative format.” I agree with you that 1–10 scales don’t do candidates justice (especially in this era of grade inflation), but I’m not sure how tighter formatting of narrative content will do the job. What really can the National Resident Matching Program do at the systems level to solve this without completely rendering letters of recommendation so generic as to be meaningless?

Dr. Humbyrd: We still believe that letters of recommendation can provide valuable insight into an applicant. There is definitely value in the narrative. At a systems level, we could be more explicit about useful letters or content to highlight in good letters of recommendation. Even more fascinating would be if letter writers could receive some feedback on the letters that they write. Many faculty members write a large number of letters year after year; most don’t get any systematic feedback (or any feedback at all). It would be interesting to see if you could submit sample letters of recommendation and receive language-analysis feedback. Clearly, there is more we can do to explore how best to support residency applicants through letters of recommendation. Finally, we would strongly advocate that letter writers avoid copying and pasting their letters. We are reviewing a different year’s letters, and we have found more than one letter writer who wrote the same letter for five applicants. Word for word. Often, they didn’t correct the gender or the name. This isn’t fair to our students, so in terms of low-hanging fruit, we would recommend our colleagues avoid doing this.