Writing More Informative Letters of Reference

Abstract

Writing a meaningful and valuable letter of reference is not an easy task. Several factors influence the quality of any letter of reference. First, the accuracy and reliability of the writer's impressions and judgment depend on how well he knows the individual being described. Second, the writer's frame of reference, which is determined by the number of persons at the same level that he has worked with, will impact the context and significance of his beliefs and estimations. Third, the letter-writing skills of the person composing the letter will naturally affect the letter. To support the other components of a candidate's application, a letter of reference should provide specific examples of how an individual's behavior or attitude compares to a reference group and should assess “intangibles” that are hard to glean from a curriculum vitae or from test scores. This report offers suggestions that should help physicians write more informative letters of reference.

Twenty years ago, Dr. Richard Friedman published an article entitled “Fantasy Land” in the New England Journal of Medicine in which he described the “fantasy land of letters of recommendation” as “a wondrous place…where people have excellent interpersonal skills and about a tenth of the inhabitants are among the finest I have ever worked with, and almost all are in the upper quarter.”¹ The author of “Fantasy Land” was describing his experience the previous night as he reviewed the folders of candidates applying to the internal medicine residency program at his institution.

Although now such letters are posted electronically (in ERAS, The Electronic Residency Application Service), little else has changed in 20 years with respect to the content of these letters or their tales of fantasy, despite the publication of “Fantasy Land” in one of the most influential medical journals in the world. This is perhaps most evident when one examines the situation with the dean's letter, a document that programs rely on heavily in their assessment of medical students. In 1989, the Association of American Medical Colleges (AAMC) distributed its “Guide to the Preparation of the Medical School Dean's Letter,” in which it was noted that “the Dean's letter (a letter of reference) is not a letter of recommendation; it is a letter of evaluation.”² A study published several years ago found that deans’ letters did not contain important negative information about a student (e.g., failing or marginal grades, leave of absence, or need to repeat a year of medical school) in 34% of cases in which such information was present in the transcript.³ Recognizing that the 1989 guidelines had not been fully implemented, a second Dean's Letter Advisory Committee was appointed by the AAMC in late 2000, resulting in the recommendation that the dean's letter should be renamed the medical student performance evaluation (MSPE).⁴ It was decided that this evaluation should be written with standard content and format, including a graphic representation of the student's grades.

Despite the fact that physicians are regularly asked to write letters of reference for colleagues and trainees, several factors can make this a tricky proposition. First, while physicians have received training for many of the tasks they routinely perform, letter writing is not a skill that physicians are taught. Second, writing letters may be seen as burdensome, because there is no clear reward for this activity and writing anything more than a perfunctory letter takes time. Third, physicians are often asked to write letters about individuals whom they may have worked with months or years earlier, making it challenging to recall specific examples and details that would make the letter more meaningful. Fourth, the encumbrance of knowing that one's reputation, as well as that of the institution, may be judged based on the accuracy of the assessment can be stressful. Finally, legal and confidentiality issues (including defamation and intentional misrepresentation) need to be carefully considered.

Previous articles that have been published about letters of reference have consisted almost exclusively of opinion pieces wherein experienced “reference letter writers” share their insights. While our initial intent was to write an evidence-based review related to this topic, the real absence of significant empiric work in this area precluded this possibility. As such, in addition to summarizing many of the most reasonable suggestions made by previous authors, we have decided to make some suggestions, with the hope that some new ideas and approaches may help to move letters of references in the right direction. The goal of this manuscript is to propose that reference letters become more standardized in their content; in particular, there should be an assessment of how an individual's behavior or attitude compares to a reference group and commentaries should be made about core characteristics (such as integrity or motivation) that are hard to glean from a curriculum vitae.

GUIDANCE FROM THE LITERATURE

Searching the Literature

Thirteen databases (PubMed, PsychINFO, ERIC, ArticleFirst, Periodicals Abstracts, Academic Search Elite, ISI Citation Indexes, Business Source Premier, ABI Inform, Index to Legal Periodicals and Books, LexisNexis, and HealthSTAR)c were searched for articles related to letters of reference. The bibliographies of the retrieved articles were also examined for relevant articles and a Google search was also performed. Forty-three articles were found, eight of which were studies involving data collection and interpretation.

Insights from Published Reports

Studies.

Two different aspects of reference letters have been investigated. First, in a small study of undergraduate seniors posing to their faculty advisors as prospective applicants to graduate schools, the effect of confidentiality was examined.⁵ Analysis of the responses showed significant differences between the confidential and nonconfidential letters, with students rated lower in most areas in the confidential letters. Second, a handful of studies have attempted to assess the validity of reference letters by looking at their predictive value.^6–11 Some investigators have found a lack of correlation between letters and the actual performance of house officers,^6,7 whereas others have found reference letters to be more reliable predictors of resident performance.^8,9 These studies attempting to assess the validity of reference letters have all been relatively small and have been limited by the manner in which the letters were rated and how the house officers’ overall performance was assessed.

Expert Opinion About Writing Letters of Reference

Many reports were identified wherein authors shared their opinions and experiences about writing high-quality letters of reference.^12–34 The highlights of these papers, grouped under the headings of “preparing to write the letter,”“writing the letter,” and “before sending the letter,” are listed in Table 1.

Table 1.

Recommendations from the Published Literature About How to Write Letters of Reference

Recommendations
Preparing to write the letter
•Address and send the letter only to individuals named by the trainee
•If you feel that you will be unable to write a favorable letter about the individual, you should inform him and give him the opportunity to decide if he prefers to ask someone else
•Commit enough, but not too much, time to the writing of the letter
•Remember the responsibility that you have to the profession of medicine
•Define whether you are writing a letter of reference (a genuine evaluation) or a letter of recommendation (a commentary only on strengths of an individual; weaknesses that have been identified are purposefully omitted)
•Consider meeting with the individual requesting the letter to review the person's career goals and other aspects that are important in order to create an honest picture of the candidate. Meeting with the individual can also serve to refresh one's memory about specific experiences that have occurred with the person and about their performance and character
Writing the letter
•Maintain a formal approach in the formatting of the letter
•Choose your words carefully
•Use common sense
•Keep in mind that honesty and authenticity are paramount
•Use specific, objective data when possible and supplement that with interpretations or explanations
•Provide the information that you would want to know if you were on the receiving end of the letter
•Confer concerns and weaknesses explicitly
•Keep it short and clear
•Make sure that the information shared reflects fairness and good faith
•Be certain to discuss the skills or characteristics that are most relevant and germane. These frequently include many of the following: personal qualities (especially integrity and motivation), professionalism, communication skills, relations with others, clinical competence, medical knowledge, technical skill, administrative ability
•Offer for the recipient to call you if clarification of any of the content is needed
Before sending the letter
•Critically read the letter with careful appraisal of the language used and the information that has been disclosed

Commentaries About the Legal Perspectives

Fourteen manuscripts focused on the legal perspectives that should be considered related to letters of reference.^25–37 Readers wishing additional detail on this topic beyond the overview described below may wish to review articles 31 to 34 from the reference list.

Given the nature of the profession of medicine, it would seem most appropriate to comply with the ethical obligation to follow the Golden Rule and to share any adverse information about an individual that one would hope to receive if the tables were reversed.²⁵ Although there do not appear to be any industry standards in this regard, courts have granted a qualified privilege to certain communications made in good faith between employers who share a common interest that make such communications reasonable in light of their relationship. Defamation requires proof of: 1) a false and derogatory statement about an individual, 2) communication of this to another person (if written: libel; if spoken: slander), and 3) actual harm to the individual resulting from the statement.³⁵ Because of the risk of defamation suits and the potential for negative comments in reference letters to influence hiring decisions, physicians should carefully review negative comments to make sure that they are accurate and factual.^36,37

Several of the articles described lawsuits that resulted from information conveyed in reference letters. Two such cases are described here for illustrative purposes. First, in 1981, three senior physicians on the faculty at the Brigham and Women's Hospital in Boston wrote enthusiastic reference letters for an anesthesiologist colleague without mentioning the fact that he had recently been convicted of rape, about which they were aware. Hospital officials who employed the physician were angry when it was discovered that a convicted felon had been hired. The Massachusetts Medical Society censored the three physicians and placed them on one year's probation for “failing to reveal a deficiency in a fellow physician's character.”³⁸ Second, in 1980, a professor and chairman of an orthopedics department received an inquiry about a former resident from a hospital in another state.³⁷ The chairman described his performance as “well below average” adding that “he was not suited to orthopedic surgery.” The former resident sued the professor for defamation. When the professor asked the federal district court to dismiss the action, he prevailed in the trial court but the court of appeals reversed this decision. Just before the court was to hear the case, the plaintiff dropped it due to mounting legal costs.

The following recommendations seem to be prudent based on the literature^31–34:

1. Do not disclose information about an individual without his written permission.

2. Ask the individual to waive his right of access to the reference letter. According to the Family Education Rights and Privacy Act of 1974, this limits the potential for liability.

3. Only share statements that are factual, truthful, and made in good faith.

4. Private information (such as medical information) should never be disclosed, unless you are given permission to do so.

5. If there is pertinent negative information that you believe needs to be conveyed in the letter, advise the individual of this and confirm that he still would like you to write the letter.

AUTHORS’ IDEAS ABOUT INNOVATIONS TO IMPROVE LETTERS OF REFERENCE

It is fair to wonder whether suggestions about letters of reference are even worth writing. After all, one can now download ready-made templates for such letters from the Internet in rich text format that is compatible with most word processors. Sample templates are available for employees of different levels, such as “average to slightly above average.”³⁹ Nevertheless, after reviewing the literature and reassessing many reference letters that we have written or received, as well as discussing various ways to advance the standards of reference letters, the following changes may have the potential to rejuvenate and enhance the value and confidence we place in them.

The Incorporation of Dee Hock's Criteria for Hiring Individuals

Dee Hock, the founder and CEO Emeritus of VISA International, is a well-respected leader and innovator in the fields of business and management. He believes that individuals should be hired and promoted first on the basis of integrity; second, motivation; third, capacity; fourth, understanding; fifth, knowledge; and last and least, experience.⁴⁰ It is worth noting that this hierarchy appears to be inversely correlated with what can be objectively documented. Knowledge and experience, for example, are easily demonstrated in a curriculum vitae or by scores on standardized tests. By contrast, integrity and motivation must typically be attested to by the opinions of others. Mr. Hock has said: “Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind. Experience is easy to provide and quickly put to good use by people with all the other qualities.”

If reference letters carefully documented the writer's assessment of the individual's integrity, motivation, capacity, and understanding, and only briefly alluded to an individual's knowledge and experience (which may be gleaned by reviewing a curriculum vitae), the added value of these letters would be substantially augmented (see example below).

The Inclusion of a Comparative Ranking of the Individual from Novice to Expert

Dreyfus and Dreyfus, as well as other researchers, have described 5 stages of skill acquisition: novice, advanced beginner, competent, proficient, and expert.^41,42 These authors believe that individuals progress through the stages regardless of the skill being acquired, from driving a car to playing chess to being an airplane pilot.

It will almost certainly not be intuitively obvious and will take some time before clinicians are adept and comfortable rating individuals on this scale. Nonetheless, it is our recommendation that reference letters end with a statement comparing the individual to others at the same level and assigning the person a specific rank from novice to expert, (see example below). In Table 2, an exemplary classification scheme using the novice to expert taxonomy is depicted for how one might rate a resident who is applying for a fellowship.

Table 2.

Classification Scheme Related to How One Might Rate a Resident Who Is Applying for a Fellowship Position at the End of a Reference Letter Based on the “Five Stages of Skill Acquisition”⁴¹,⁴²

Rating	Comparison to Peer Group	Descriptor
Novice*	•Not on track to graduate from program: remediation is unlikely to allow individual to achieve competence	→Below minimal acceptable level of competency
Advanced beginner*	•Not on track to graduate from program: remediation is likely to allow individual to achieve competence	→Below minimal acceptable level of competency
Competent	•Lowest 25% of resident class	→Resident has demonstrated that he has achieved the minimal level of competency to graduate from the residency; however, weaknesses in one or more of the core competencies are apparent
Proficient	•Middle two quartiles	→Resident is solid and reliable and performs at an admirable level
Expert	•Top 25% of resident class	→Resident is acknowledged by the faculty and his peers as an excellent role model. He knows his limitations (which derive primarily from not having experienced certain situations) and seeks guidance appropriately. Nonetheless, he has achieved a level of mastery with regard to the ACGME competencies compared to colleagues at his level.

^*Because the Outcomes Project of the ACGME⁴⁴ requires the attainment of competence as the minimum standard for graduation from a residency program, it would not be possible for a house officer to proceed to fellowship training if he were less than competent.

ACGME, Accreditation Council on Graduate Medical Education.

A model letter about “Letter of Reference” is included here to illustrate how these concepts can be employed:

Dear Sir or Madam:

I have been asked to write this brief note to you about Letter of Reference and he has waived his right to view this communication. I have had the opportunity to closely observe Letter of Reference for the past ten years and as such I feel as though I am able to accurately comment on his capabilities.

 Letter of Reference certainly has tremendous experience in describing people who are applying for jobs and his knowledge about what people want to hear is solid. While he does have a fairly good understanding of his role and how he can be most helpful to others, he frequently fails to live up his capacity and operate at a very high level. He is genuinely motivated in his work and his dedication can be attested by the sizeable number of communications that he performs each day. Due to the variability in the quality of his work, I have in the past questioned his integrity and the amount of pride that he places in his work.

 While Letter of Reference has the potential to recount accurate and valuable information, I have found him to frequently overstate the positives and either downplay or simply omit pertinent negatives about a given individual. This particular example became apparent once again last month when, after receiving exclusively written praise about a physician applying to work at our hospital, phone calls to colleagues at his institution easily and quickly uncovered concerns about that physician's professionalism and character.

 Compared to the other types of letters that I have read in my 30 years as a letter reader, I would rate Letter of Reference as an advanced beginner.^* Please don’t hesitate to call me if you desire additional information.

Sincerely,

Specific N. Precise, MD

Novice. These types of letters do not provide any useful information. Recipients often discard them without even reading them because of their inaccuracies or irrelevance. Lowest quintile (20%) of letters. Example: a conventional solicitation letter about a credit card.

Advanced beginner. These types of letters provide general information about a topic you are interested in, but their quality is highly variable and often unreliable. Second quintile of letters. Example: a typical letter of reference.

Competent. These types of letters provide accurate information that is frequently already known to the recipient and is usually unexciting. Third quintile of letters. Example: a standard letter of credit.

Proficient. These types of letters provide detailed, accurate information about a topic that is highly relevant to the reader. If the recipient is aware that such a letter is forthcoming, they usually eagerly look forward to receiving it. Fourth quintile of letters. Example: a typical letter from a teacher about your child.

Expert. These types of letters provide precise and truthful unambiguous information that is usually motivating or stimulating to the recipient. Because of their quality and rarity, they are highly valuable and cherished. Highest quintile of letters. Example: a classic love letter.

Acknowledge that Some Letters Will Be Exclusively Appreciative

Appreciative inquiry can be thought of as an approach or a philosophy that begins with and builds on discovering what is working well within a system, organization, or individual.⁴³ With this approach in mind, some reference letters may choose to focus exclusively on the positive aspects of an individual, provided that this is explicitly stated. This methodology may be appropriate and reasonable when one is asked to write a letter about an individual that one does not know well but about whom the letter writer is particularly impressed. Opening sentences of such a letter may begin as follows:

“I have been asked by Dr. Julia Smith to write this letter of reference. I have only had the opportunity to work with her for 3 weeks on the medical ward where she was the resident and I was the attending physician. Because of my limited exposure to her, I am choosing to write an appreciative letter wherein I will focus on the wonderful job that she has done while working with me and her positive characteristics.” Although this type of letter will undoubtedly be given less weight than one written by a program director who has had the opportunity to closely follow an individual over years, it allows the letter writer to succinctly share his point of view acknowledging the scope of the relationship with the candidate and his vantage point.

CONCLUSIONS

The writing of reference letters that are truly useful to the recipients and fair to the party described is difficult. We suggest that this challenge is best met by a letter that provides specific examples of how an individual's behavior or attitude compares to a reference group and that assesses aspects of an individual's character (especially integrity and motivation) that are more difficult for a reader to pick up from a curriculum vitae or from test scores.

Although each letter of reference deserves individualized consideration, the application of a sound framework as outlined in this paper should prove helpful to both the writer and the recipient of the communication.