Table.
Literature Summary
| Author, Year | Participants | Aims | Results | Evidence for Validity? |
| Content | ||||
| Keim et al, 1999 | SLOE Task Force | Describe the creation process of the EM SLOE | • Task force convened in 1995, consensus development process with EM education experts • Pilot first year, edits made after survey of program directors | Yes |
| Response Process | ||||
| Girzadas et al, 1998 | 20 SLORs and 20 NLORs submitted to one program | Compare SLOR to NLOR in EM applications | • Interrater reliability was 0.97 for the SLOR, compared to 0.78 for the NLOR • Average time to interpret a SLOR was 16 seconds vs 90 seconds for an NLOR | Yes |
| Harwood et al, 2000 | 432 SLORs submitted to one program | Assess grade and rank distribution on the SLOE | SLOR authors did not use the full scale • Grades: 55% honors, 36% pass, 9% pass • Global assessment: 37% outstanding, 49% excellent, 14% very good or good • Match: 23% guaranteed, 50% very likely, 27% likely and possible | No |
| Girzadas et al, 2004 | 835 SLORs submitted to one program | Assess for gender bias on rankings on the SLOE | • A female author writing a letter for a female applicant was highly associated with giving the highest Match rank on the SLOR • No other gender combination was significant | No |
| Love et al, 2013 | 602 SLORs submitted to 3 different programs | Assess grade and rank distribution on the SLOE | Showed ranking inflation • On global assessment, 40% of students were top 10% • 95% of students were in the top third compared to peers for the qualifications for EM section | No |
| Beskind et al, 2014 | 1253 SLORs submitted to 3 different programs | Determine whether characteristics of the letter writer affected rankings on the SLOE | • Less experienced writers were more likely to give a higher ranking • The length of time an author knew the applicant was associated with high rankings | No |
| Hegarty et al, 2014 | 320 of 695 (46%) CORD members | Survey SLOE authors on their practices regarding filling out SLOEs | • 67% of SLOE writers did not receive instruction in how to fill out a SLOE • 68% of SLOE writers state they do not follow the instructions on certain questions | No |
| Grall et al, 2014 | 1457 SLORs submitted to 3 different programs | Assess grade and rank distribution on the SLOE | Showed ranking inflation • For 4-point scale variables, 91% were ranked as the top 2 options • For 3-point scale ratings, 94.6% were ranked as the top 2 options • Less than 2% of SLOEs were ranked in the bottom third | No |
| Li et al, 2017 | 237 first-rotation SLOEs of applicants invited to interview at one program | Assess the narrative portion of the SLOE for gender bias | • Examined 237 SLOEs and found that the narrative portion was “relatively free of gender bias” | Yes |
| Hall et al, 2017 | 1075 applications to one program consisting of grades from 236 different clerkships | Assess grade variability between different schools | • The percentage of students that receive an honors grade at a school ranges from 1%–87% • Some schools are pass/fail • Some schools use 3-point grade scales, some use 5 • Some schools give grades, but not honors | No |
| Pelletier-Bui et al, 2018 | 99 respondents, survey sent to CORD and CDEM (clerkship directors in EM) listservs | Survey SLOE authors on their practices regarding filling out SLOEs | • 39% responded that they strictly adhere to the ranking guidelines | No |
| Jackson et al, 2019 | 6715 SLOEs for 3138 unique applicants accessed from the eSLOE database | Assess grade and rank distribution on the SLOE | Showed ranking inflation (although improved from the 2013 study) • Global assessment: 18% top 10%, 37% top third, 35% middle third, 10% lower third • Match rank list: 18% top 10%, 36% top third, 32% middle third, 12% lower third, 2% unlikely to rank | No |
| Boysen-Osborn et al, 2019 | 624 applicants to one program | Compare rankings on SLOEs written by a student's home institution to those written after a visiting rotation | • Authors created an overall composite score for a SLOE • The composite score was better on SLOEs written by a home school than those obtained on a visiting clerkship | No |
| Miller et al, 2019 | 822 first rotation SLOEs submitted to one program 64% male and 36% female | Assess differences in word type frequency by gender on the narrative portion of the SLOE | • No significant difference in word type frequency by gender in the narrative portion | Yes |
| Andrusaitis et al, 2019 | 2092 SLOEs submitted to one program | Assess for gender bias in overall scores on the SLOE | • Females have better overall scores on the SLOE than males | No |
| Internal Structure | ||||
| Girzadas et al, 2001 | 411 SLORs submitted to one program | Find associations between a ranking of “guaranteed match” (the highest rank at the time) and other rankings on the SLOE and author variables | A ranking of “guaranteed match” was highly correlated with both • An honors grade, an outstanding ranking on differential diagnosis, an outstanding ranking on work ethic, and an outstanding ranking on global assessment • The authors position and having clinical contact outside the ED | Mixed |
| Relation to Other Variables | ||||
| Hayden et al, 2005 | 54 graduating residents from one program | Compare SLOE rankings to residents' “final success” upon graduation | • Ranked graduating residents into percentiles (against all previous residents) at one institution • The SLOR was not strongly correlated with this measure of success | No |
| Oyama et al, 2010 | 102 SLORs from 5 programs | Compare predicted Match list position on the SLOE to the actual Match list position | • 26% of SLOEs had a predicted match rank that matched the actual match rank • 66% of the time the SLOE overestimated the rank position • 8% of the time it underestimated the rank position | No |
| Breyer et al, 2012 | 127 applications to one program | Compare predicted Match list position on the SLOE to the actual Match list position | • Global assessment on the SLOE was positively correlated with final rank list for Match • Spearman's correlation 0.332 | Yes |
| Bhat et al, 2015 | 277 residents consisting of 3 graduating classes from 9 programs | Compare SLOE rankings to residents' “final ability” upon graduation | Faculty ranked residents' “final ability” upon graduation, which • Correlated with the global assessment • Correlated with ranking of competitiveness on the SLOE | Yes |
| Consequences | ||||
| Love et al, 2014 | 150 of 159 (94.3%) EM program directors | Survey EM program directors about their perspectives regarding the SLOE | • SLOE was ranked as the number one data point when deciding who to interview | No |
| Negaard et al, 2018 | 120 members of the CORD listserv | Survey EM program directors to describe EM residency selection criteria | • The visiting rotation SLOE was ranked as the number one data point when creating the final Match list • The home rotation SLOE was third most important data point when creating the final Match list | No |
Abbreviations: SLOE, Standardized Letter of Evaluation; EM, emergency medicine; SLOR, Standardized Letter of Recommendation; NLOR, Narrative Letter of Recommendation; CORD, Council of Residency Directors in Emergency Medicine; CDEM, Clerkship Directors in Emergency Medicine; ED, emergency department.