A Call to Action for Standardizing Letters of Recommendation

As more medical students are applying to residency and fellowship programs, the burden of reviewing applications and discriminating among individuals is increasing. In addition, studies highlight the need to intentionally target gender and other biases, which may affect underrepresented in medicine (UIM) applicants.^1-3 Integrating standardized letters into the application process may mitigate biases and improve the efficiency of the selection process.⁴ Standardized letters have been implemented in several medicine and surgery specialties, with a growing body of literature supporting their advantages over narrative letters of recommendation (LORs).^5,6 We pose a call to action to graduate medical education (GME) program leaders to collaborate in the process of developing and implementing residency- and fellowship-specific standardized letters across training programs.

What Is a Standardized Letter?

In 1997, the Council of Residency Directors in Emergency Medicine (CORD) reported on the development and implementation of a standardized letter of recommendation (SLOR) by emergency medicine (EM) residency programs.⁷ Subsequently, other specialties including orthopedics and otolaryngology developed and implemented their own SLORs.^8,9 In 2014, the EM SLOR was updated and renamed the CORD Standardized Letter of Evaluation (SLOE) to better reflect its purpose in providing a summative evaluation of a candidate. Now considered the gold standard by CORD for evaluating potential candidates, the EM SLOE has 4 sections: Background Information, Qualifications for EM, Global Assessment, and Written Comments. Background Information includes the length of the evaluator-student relationship, the nature of their contact, and the student's end-of-rotation grade on a 5-point scale: Honors, High Pass, Pass, Low Pass, and Fail. The Qualifications for EM section asks the evaluator to compare students to their peers on 7 criteria: commitment to EM, work ethic, differential development and treatment plan, teamwork, communication skills, guidance needed during residency, and likelihood of success. The Global Assessment section asks evaluators to rank students in comparison to others they recommended in the prior academic year: top 10%, top third, middle third, and lower third. Finally, the Written Comments section allows evaluators to expand on students' noncognitive attributes.¹⁰ Students must obtain a SLOE, completed by faculty trained in writing SLOEs, for each EM rotation they complete and submit it with their residency applications. Like narrative LORs, students may opt to waive their right to view the SLOE. In 2020, the Association of Professors of Gynecology and Obstetrics piloted the obstetrics and gynecology (OB/GYN) SLOE followed by full implementation in 2022; the sections of the OB/GYN SLOE include Competency Assessment Rubric, Most Outstanding Feature, Areas of Focus, and Narrative.¹¹

Evidence Supporting a Standardized Versus Narrative Approach to LORs

While SLOEs have been adopted by several specialties, most letters submitted for GME applicants use narrative approaches. Program directors identify narrative LORs as critical to the selection process; however, narrative LORs have not been found to reliably provide substantial information on applicants' competencies required by the Accreditation Council for Graduate Medical Education, discriminate between candidates, or predict future performance.^12-20 While there are guidelines on how to write narrative LORs, a paucity of literature exists to address the optimal content for narrative LORs to assist faculty who often have little or no training in writing narrative LORs.^21,22

The number of applicants to GME programs across the United States has been steadily rising.^23-34 Performing a thoughtful, holistic review of each candidate's application and selecting who to invite for interviews is a challenging task. Studies performed in various specialties have identified significant linguistic differences in narrative LORs, including infrequent use of language describing strong leadership traits for women and UIM candidates, suggesting that implicit biases related to both candidates and letter writers exist.^25-34 Several studies have demonstrated that SLOEs may be better than narrative LORs at mitigating implicit biases that adversely affect women and UIM candidates.^33,35-41 Additionally, the SLOE represents a potential tool to better differentiate applicants; Jackson et al demonstrated that the electronic EM SLOE provided more discrimination, even above the original SLOR, with improved spread of rank categorization.^42-44 While SLOE implementation may standardize applications and enable candidate comparison in a more objective and unbiased manner, the potential remains for inflation of applicants' qualifications by authors; medical schools and training programs have an interest in helping their graduates secure preferred positions.^38-40 It will be imperative to invest time and resources to provide faculty development related to the writing and interpretation of SLOEs for selection committees.

Developing the SLOE for Residency and Fellowship Programs

To develop and adopt SLOEs across more specialties, it is important to learn from the prior experience of GME programs where implementation has been successful.^41,45-47 The Table suggests an action plan with concrete steps that program leaders can take. The total number of applications that selection committees must review within a short time frame has risen sharply. For example, in 2022, 7 of the 17 pediatric subspecialties received, on average, over 100 applications per program, with pediatric emergency medicine (PEM) being one of the most competitive subspecialties. PEM program directors from across the country meet semiannually. During the most recent meeting, we polled attendees, and the overwhelming majority were interested in the development of a subspecialty-specific SLOE. Other discussion points included ensuring the SLOE encourages holistic review of candidates, is designed to meet the needs of institutions where PEM specialists are not present, and is prefaced by faculty development and training. The authors plan to continue this process and hope the effort will be expanded to engage members of the broader GME community.

Table.

Next Steps for the Development of Specialty-Specific Standardized Letters of Evaluation

Steps	Actions
Step 1: General needs assessment	Utilize annual meetings and/or surveys to obtain: • Broad input from key stakeholders: ○ Program directors, core faculty, and current trainees ○ Patient representatives and nurses ○ MDs, DOs, and IMGs • Assess interest, objectives, potential barriers
Step 2: Multidisciplinary working groups	Recruit and invite group members from key stakeholder groups Create and disseminate specific needs assessment Discuss content and design: • How to capture candidate qualities (eg, ACGME competencies) • Prompts for comparisons among applicants • Information from authors to support assessments • Template design (general/adapted as needed vs specific template)
Step 3: Pilot implementation	Determination of key outcomes • Correlation with ACGME competencies • Interrater reliability • Ranking and match distribution and correlation Design of training materials for SLOE completion Identification of faculty assigned to complete SLOEs PDSA cycles Identifying and addressing barriers
Step 4: Full implementation	Refinement of training materials and outcomes PDSA cycles Identifying and addressing barriers Integration with the Electronic Residency Application Service

Abbreviations: MD, Doctor of Medicine; DO, Doctor of Osteopathic Medicine; IMG, international medical graduate; ACGME, Accreditation Council for Graduate Medical Education; SLOE, Standardized Letter of Evaluation; PDSA, plan-do-study-act.