CORR Insights®: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants

Where Are We Now?

Most specialties find it challenging to select medical students for residency programs; it is especially so in orthopaedic surgery, since programs must screen an average of nearly 700 applicants [1] just to decide whom to interview. After this screening process, most orthopaedic programs interview (either in person or virtually) 60 to 80 candidates to establish a rank order list [1, 3].

Medical students also rotate at both “home” and “away” programs to both “audition” for a residency and to learn more about the specialty as well as about the programs they visit [1, 3]. Orthopaedic rotations, both home and away, allow candidates to improve their chances of being selected through personal interactions with residents, faculty, and others who work with the students.

Programs use a mix of both objective and subjective information to rank their candidates including United States Medical Licensing Examination Step 1 (and 2) scores, grades in medical school, class standing, Alpha Omega Alpha status, research contributions, letters of recommendations, personal statements, and in-person rotation evaluations [1]. Letters of recommendation generally are used by residency selection committees in all specialties.

Traditionally, letters of recommendation are written by a faculty member or senior resident; these letters detail the qualities of the medical-student applicant. Until recently, these letters were unstructured narratives that often omitted key information for resident selection. In order to be more objective in presenting a student’s strengths and weaknesses, the American Orthopaedic Association developed the standardized letter of recommendation (SLOR). The SLOR includes nine scaled categories of assessment: The first five are “orthopaedic specific” (patient care, medical knowledge, interpersonal and communication skills, procedural skills, research) and the last four are deemed “global” (teamwork, professionalism, initiative and drive, and commitment to orthopaedic surgery). Evaluators are asked to rank the student in each of these categories.

In the current study, Pacana et al. [5] reviewed the initial results of one program with the SLOR. The authors evaluated 559 applicants who had 1376 of 1852 letters of recommendation eligible for the study. The authors found that 36% of the SLOR letters sent to their program ranked the applicant in the upper 10% of medical students. They did not find a difference in assessments based on whether they were written by program leadership (program directors and chairs) compared with other faculty members, nor based on whether the letters came from away versus home rotations [5].

It is important to put these results in context. First, medical students usually ask, “Can you write me a positive letter?” In other words, medical students can preselect the authors for positive letters. Second, there is also a drive for medical schools to match all of their students, and unselected students may reflect negatively on a department. Third, orthopaedic surgery applicants generally perform better than average compared with their entire medical student class [4].

Based on these findings, letter writers should offer medical students realistic advice about a career in orthopaedic surgery. Given the pressures, it is often difficult to have a candid conversation with a medical student. One technique I have used over the years, when asked for a letter by a medical student, is to state, “Certainly, and based on feedback from others I would have to rank you X out of the Y number of students I have worked with this year.” Of course, this technique does not work with small numbers of rotators.

Where Do We Need To Go?

I believe future studies should determine whether the SLOR provides a more objective analysis of medical student performance compared with traditional letters of recommendation. Such research might also provide a framework for making the SLOR more objective in assessing medical students. The ability to objectively measure and compare medical students is imperfect at best. Inflated assessments of those in medical training are common due to a lack of objective criterion-specific assessment tools for faculty, lack of time to write assessments and provide feedback, and the lack of assessment training for the assessors. The American Orthopaedic Association developed the SLOR to provide a more objective evaluation and to standardize assessments between raters and schools. But objective measures of medical student performance are limited for resident selection, and because of “grade inflation” as reported by the authors [5], we do not get a true comparison between medical students. While the SLOR is an attempt to mitigate this problem, it remains unclear whether the standardized letter actually accomplishes its goal. Adding clear criterion-specific objective measures that correlate to successful resident and staff surgeon performance would improve the SLOR and allow for more consistent ratings among assessors. This will require further research to better determine these qualities.

There are relatively few objective measures that selection committees can use to distinguish among candidates. The ideal assessment tool should provide an objective measure of medical student performance relative to the other candidates in areas such as communication, orthopaedic knowledge, and motor skills. The tool should level for such factors such as medical schools and evaluators [2].

How Do We Get There?

For this year’s medical student interviewing cycle, it is unclear how the SLOR has improved things. To determine this, a study comparing the SLOR to the previous narrative letters would be helpful. Thankfully, there are some students who receive both types of recommendations, making a direct comparison possible. Furthermore, there are still a large proportion of narrative letters in the pool, making more general comparisons possible. In the narrative letter, researchers would look for key phrases about knowledge, motor skills, patient care, communication, and professionalism. Additionally, we could review the narrative letter for a final summative statement about where the medical student ranks relative to those who are also applying to orthopaedic surgery.

Using these imperfect but available tools, we should consider “rating the raters” so that the SLOR recommendations become internally ranked. For example, a faculty member’s evaluation would be leveled toward ranking the assessed medical students. Then, all SLORs could be leveled as much as possible between faculty members, with a corrected value included with the SLOR. To work, each faculty member would need to have a sufficient number of evaluations. A faculty member who evaluates only one student, for example, would not be able to have his or her letters rated in this way. For departments, which have multiple medical students throughout the year, the addition of information about the rater should be included. In other words, the rating will include the phrase, “This evaluation is X out of Y students rated by this rater this year” and this student is “rated as number 3 out of 6 rated students this year.”

Additionally, faculty should be provided assessment education, both formal and informal, to better stratify medical students. Faculty may feel pressured by students, other faculty, and the medical schools to provide inflated rankings for the students whom they think should be picked up by orthopaedic programs. Because our specialty is competitive, some program directors view anything less than a positive assessment or recommendation as below average.

Furthermore, comparing the applicant to other orthopaedic candidates often gets confused with comparing the applicant to the entire medical student population. Most programs have a medical student curriculum, along with rotations, which should form the basis of medical student assessment. Just as the Accreditation Council for Graduate Medical Education (ACGME), American Academy of Orthopaedic Surgeons, and the American Board of Orthopaedic Surgery (ABOS) are seeking to identify improved performance standards, we should seek to do the same thing for medical students.

A more robust SLOR provides resident selection committees more objective information for medical student rankings. Improvements to the current mélange of data provided by applicants can only be made through educational research. For the SLOR, this would mean developing objectives aligned toward criteria that are important for successful residents [2]. Such research will be longitudinal, looking at outcomes of resident performance, to allow programs to have confidence in this measure. Sponsorship of these efforts would ideally be done on a national level and have support of the ABOS, ACGME, and Liaison Committee on Medical Education.