Letters of Recommendation: How Do They Fit Into the Modern Application?

In this issue of the Journal of Graduate Medical Education, Saudek et al¹ shed light on the issue of letters of recommendation in graduate medical education. A survey of pediatrics residency and fellowship directors posed questions regarding the content and impact of recommendation letters as part of the application process. The survey tool had 3 sections—content and style of the letter, items listed in regard to applicant qualities, and examples of summary statement verbiage—that were rated as either positive, neutral, or negative. The questions were not redundant and explored the areas well. The survey had a reasonable 43% response rate. The authors reported that the program directors felt the letters had value and that certain phrases in the summary statement (“the code”) were important. In addition, they note that letter writers may benefit from some training to know the typical types of summary statements made in the final paragraph and the degree to which these influence the reader. The authors caution, appropriately, that the results of this survey cannot be extrapolated to other disciplines.

I came away impressed that the authors examined an activity that takes a lot of time, is a required part of the application process (usually 3 letters), and yet very few of us really understand all that well.

I should disclose that I am the long-standing director of a surgical residency program, and I was also a division head for a decade. I have trained people for others to hire and hired people others have trained. Being on both sides of the issue has been valuable. In addition, it seems there is some, but not complete generalizability in this area from one specialty to another. I am part of a program director panel for our fourth-year students, to discuss frequently asked questions during the recruitment season. This is instructive, and there is a wide difference of opinion regarding letters of recommendation, from choosing writers who know the student very well and can speak authoritatively to his or her attributes, to choosing writers with a strong reputation, who are nationally well-known. I suspect the latter option is much more prevalent in the academic institutions where I have spent my career. I must admit that, although I have read hundreds of letters of recommendation over the years, I am not a savvy interpreter of the various codes at letter summation when compared to some of my more senior colleagues. I also have written a reasonable number of recommendations myself—primarily for our own residents wishing to advance their training.

The volume of items currently in an applicant's electronic file continues to grow. Much of what is listed in a typical letter of recommendation is often already available in other parts of the application. Most program leaders who have to sort through a 5-to-1 (or greater) ratio of applications to interviewees resort to some sort of screening scoring system. In that process various aspects of each application translate into points. Letters of recommendation do not easily convert to a number scale, although I have seen systems where points are assigned only for chair letters in this prescreening.

In general, there is close scrutiny of an application before and after a candidate has accepted an interview. Use of points and other filtering systems are a fact of life that most program directors know, yet applicants and faculty not intimately involved in the process may not.

The work by Saudek et al¹ confirms the code we all believe to exist in the recommendation letter summary comments, and brings attention to the topic of letters of recommendation in general. What is the future of this time-honored activity? It is interesting that subspecialty fellowship directors listed letters of recommendation as the most important factor in deciding to offer an interview and the third most important item in ranking applicants.² This makes sense, because residency is very much on-the-job training, where there is ample time to observe and report activities that directly relate to fellowship tasks. In contrast, the limited experience of students in resident roles makes accurate assessment by medical school faculty much more difficult. A resident can be observed running a service, caring for ill patients, and managing all the stress that entails. A student cannot even write orders.

I agree with the authors that standardized letters of recommendation merely add to grade inflation. Everyone writing recommendation letters has an inherent bias—schools want and need their students to match, and program directors and faculty leaders want their residents to get top fellowships. Some standardized forms try to mitigate this bias by requiring the writer to include applicant weaknesses as well as strengths. Many job applications, in a wide variety of disciplines, ask for a list of references that the potential employer can call to have a conversation about the applicant. This has the same weaknesses as letters, in that applicants can choose to select people who will best advocate for them. However, both as a program director and as a division head, it is my opinion that personal conversations with references are far more valuable than letters. To do this on a large scale is impractical but might be an option after other items in applicants' files had narrowed down the volume of candidates. This approach appears to work in other disciplines. When we hire a new residency coordinator, his or her curriculum vitae has a list of references and contact information rather than letters.

If the applicant is a superstar in all aspects, a letter of recommendation pointing that out is a bit redundant, in my opinion. Most frequently, I use letters of recommendation to find the occasional potential star who does not stand out, with typical metrics, in the application. My all-time favorite letter of recommendation provides an example. A senior surgeon with a reputation for brevity and no nonsense wrote the following about an unmatched student who we recruited for a preliminary surgery spot. “On paper she is not the best student from our school going into surgery this year. But she is. You should take a careful look at her.” We did and he was right. She won intern of the year and soon gained a categorical spot when we had attrition in her class. She went on to graduate our program and finish a fellowship at another academic program, where she was hired as faculty.

In summary, Saudek et al¹ have drawn our attention to letters of recommendation as an important aspect of the application process. Letters of recommendation are difficult to score and to quantify as an input in evaluating candidates. However, it is hard to imagine the process without them.