Table.

Next Steps for the Development of Specialty-Specific Standardized Letters of Evaluation

Steps	Actions
Step 1: General needs assessment	Utilize annual meetings and/or surveys to obtain: • Broad input from key stakeholders: ○ Program directors, core faculty, and current trainees ○ Patient representatives and nurses ○ MDs, DOs, and IMGs • Assess interest, objectives, potential barriers
Step 2: Multidisciplinary working groups	Recruit and invite group members from key stakeholder groups Create and disseminate specific needs assessment Discuss content and design: • How to capture candidate qualities (eg, ACGME competencies) • Prompts for comparisons among applicants • Information from authors to support assessments • Template design (general/adapted as needed vs specific template)
Step 3: Pilot implementation	Determination of key outcomes • Correlation with ACGME competencies • Interrater reliability • Ranking and match distribution and correlation Design of training materials for SLOE completion Identification of faculty assigned to complete SLOEs PDSA cycles Identifying and addressing barriers
Step 4: Full implementation	Refinement of training materials and outcomes PDSA cycles Identifying and addressing barriers Integration with the Electronic Residency Application Service

Abbreviations: MD, Doctor of Medicine; DO, Doctor of Osteopathic Medicine; IMG, international medical graduate; ACGME, Accreditation Council for Graduate Medical Education; SLOE, Standardized Letter of Evaluation; PDSA, plan-do-study-act.