Abstract
Background
Emergency medicine (EM) uses a standardized template for residency application letters of recommendations. In 1997, the Standardized Letter of Recommendation was developed with categories for applicant comparison. Now named the Standardized Letter of Evaluation (SLOE), it is a universal requirement in the EM residency application process. In 2016, a website called “eSLOE” was launched for credentialed academic EM faculty to enter applicant data, which generates a SLOE.
Objective
This article outlines website creation for the eSLOE and its successful national implementation in the 2016–2017 EM residency application cycle. We analyzed current trends in applicant assessments from the eSLOE data and compared them to prior data.
Methods
Data from 2016–2017 were sorted and analyzed for each question on the eSLOE. An analysis of Global Assessment and Qualifications for EM rankings, clerkship grade, and comparison with prior SLOE data was performed.
Results
Analysis of 6715 eSLOEs for 3138 unique applicants revealed the following Global Assessment rankings: top 10%, top one-third, middle one-third, and lower one-third. There was less spread with the distribution for clerkship grade and Qualifications for EM. The 2011–2012 standard letter of recommendation global assessment data, with top 10%, top one-third, middle one-third, and lower one-third, also revealed top-clustered results with less spread compared with the ranking usage in 2016–2017.
Conclusions
Results indicate an improved spread of all rank categories for Global Assessment, enhancing the eSLOE's applicant discrimination. There has been an overall improvement in rank designation when compared with previously published data.
What was known and gap
An electronic version of the Standardized Letter of Evaluation (eSLOE) used by emergency medicine program directors allows for the tracking and comparison of assessment data.
What is new
An analysis of current trends in applicant assessments from the eSLOE data, compared with prior data.
Limitations
Analysis of ranking distribution did not factor in single versus group faculty authorship and correlation with ranking distributions, and amount of SLOE authorship training was not performed.
Bottom line
There was an overall improvement in rank designation when compared with previously published data. The eSLOE is a reliable tool to distinguish among emergency medicine applicants.
Introduction
The Council of Emergency Medicine Residency Directors (CORD) Standardized Letter of Evaluation (SLOE) is the gold standard used by emergency medicine (EM) program directors during the residency application process. In 1995, a CORD Task Force developed the Standardized Letter of Recommendation (SLOR), creating a standardized, concise, and discriminating letter for students applying to EM residency programs.1,2
The SLOR was used broadly in the academic EM community for many years. In 2011, CORD reassessed the SLOR and, after survey research, made revisions.2,3 Based on its findings, some questions changed and letter writers received increased education and updated instructions on how to use it. In 2014, the name changed from SLOR to SLOE to more accurately reflect that students are being evaluated, but not necessarily recommended for EM. (The current version of the SLOE is available as online supplemental material.)
In 2016, CORD developed a website, called the electronic SLOE (eSLOE).4 Faculty from an EM residency program can enter content electronically and create a final SLOE that can be uploaded to the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS). The eSLOE includes text boxes, with word limits, to highlight student strengths and areas of focus. Individuals and institutions can create dashboards (provided as online supplemental material) to store data that can be imported into eSLOE forms and tracked over time. Assessments of all applicants by all users can be tracked, and data comparisons of eSLOE assessments can be made.
This article outlines the eSLOE website in the 2016–2017 EM residency application season and compares eSLOE scoring assessments to prior SLOE data analyses.
Methods
The eSLOE was created on the Amazon Web Services (Seattle, WA) platform using the free, open-source languages of PHP and MySQL. This platform was chosen as a low-cost, cloud-based solution to provide a fast, always available environment for the users to reduce server lags or downtime in the heart of eSLOE writing season. There is a nominal cost associated for use and storage with the platform. Building the eSLOE required the technical services of a computer programmer.
Before widespread launch, a beta version of the website was activated for a small number of national EM faculty. This group, comprising mostly interested CORD SLOE Task Force members (including all the authors), tested the website functions and provided feedback. The system was created to electronically capture the data from the original paper SLOE and then generate a flattened portable document format (PDF) file that could be uploaded in the AAMC ERAS letter of recommendation portal. The creation of the eSLOE website ensured that the PDFs were visible in ERAS and that they were truly standardized.
The eSLOE system has a limited number of administrators who can set up institutional accounts. Only the programmer and CORD staff who manage the eSLOE have administrator accounts. Each institution was provided a “Super User” account that can create additional user accounts at the institution and see all evaluations from the institution. Individual user accounts can complete eSLOEs, but only have access to their own eSLOEs. Institutional and individual user data autopopulates, saving time, and is easily transferred from year to year and readily available to conduct research.
This project was considered exempt for Institutional Review Board review based on the criteria of the University of Miami Human Subject Research Office.
The eSLOE allows en masse data analysis of how ranking boxes are used. All 2016–2017 data entered into the eSLOE were cataloged and exported into an Excel spreadsheet (Microsoft, Redmond, WA), which served as a database file. Data were analyzed and sorted by question and entry selection for easy tabulation and statistical analysis with SPSS Statistics 25 (IBM Corp, Armonk, NY). Using a significance level of P < .01, chi-square and Spearman's correlation testing were performed.
Results
Number of eSLOEs and 2017 EM Match Statistics
There were 6848 eSLOEs entered from 222 institutions during the 2016–2017 application cycle for the National Resident Matching Program (NRMP) 2017 Main Residency Match. Of those, 133 were excluded from further analysis because 104 were duplicate entries, 26 were “test” submissions, and 3 were incomplete. The total number of analyzed eSLOEs was 6715 (Table 1).
Table 1.
Total electronic Standardized Letter of Evaluation (eSLOE) Entries
| Parameter | Result |
| Total No. of eSLOEs | 6848 |
| Total No. of institutions | 222 |
| Excluded eSLOEs | 133 |
| Duplicate entries | 104 |
| Test entries | 26 |
| Incomplete entries | 3 |
| Total No. of eSLOEs included for data analysis | 6715 |
| Unique applicants | 3138 |
| Non-AAMC IDs | 74 |
Abbreviations: AAMC, Association of American Medical Colleges; IDs, identifications.
eSLOEs were submitted for 3138 unique applicants (Table 1). However, 74 applicants had non-ERAS identification numbers with eSLOEs authored by military-based institutions, presumably for applicants entering the military match. According to 2017 NRMP Match data, 2703 individual applicants applied for EM residency.5 Therefore, 435 medical students had eSLOEs written but ultimately did not apply for EM through the NRMP Match. Some of those students applied through the military, and some changed career plans.
Distribution of eSLOE Ranking Scale
Perhaps the most important section of the eSLOE is the Global Assessment in section C with question 1, “Compared to other EM residency candidates you have recommended in the last academic year, this candidate is in the ranking: top 10%, top one-third, middle one-third, or lower one-third?” and question 2b, “How highly would you estimate the candidate will reside on your rank list: top 10%, top one-third, middle one-third, lower one-third, or unlikely to be on our rank list?” The comparative ranking results for question 1 were top 10% (18%, 1227 of 6715), top one-third (37%, 2482 of 6715), middle one-third (35%, 2318 of 6715), and lower one-third (10%, 684 of 6715; Table 2).
Table 2.
Global Assessment and Clerkship Gradesa
| Question 1: Compared to other emergency medicine residency candidates you have recommended in the last academic year, this candidate is in the: | |
| Choices | Total No. (%) |
| Top 10% | 1227 (18) |
| Top one-third | 2482 (37) |
| Middle one-third | 2318 (35) |
| Lower one-third | 684 (10) |
| Other | 4 (0.1) |
| Question 2b: How highly would you estimate the candidate will reside on your rank list? | |
| Choices | Total No. (%) |
| Top 10% | 1184 (18) |
| Top one-third | 2433 (36) |
| Middle one-third | 2174 (32) |
| Lower one-third | 817 (12) |
| Unlikely to be on our rank list | 104 (2) |
| Other | 3 (0) |
| Emergency medicine clerkship grade | |
| Grades | Total No. (%) |
| Honors | 2644 (39) |
| High pass | 2457 (37) |
| Pass | 1552 (23) |
| Low pass | 27 (0.4) |
| Fail | 1 (0.01) |
| None selected | 34 (0.5) |
N = 6715.
When compared with the expected ranking designations using chi-square testing, the 2016–2017 eSLOE data distribution is statistically significant. The underrepresentation of the lower one-third category most negatively influenced the chi-square testing results (supplemental Table 1). When combining the section C final 2 eSLOE questions in the Global Assessment, most applicants received the same ranking for both questions, with a strong degree of correlation (supplemental Table 1). However, 1245 (19%) applicants had divergent rankings for the final 2 Global Assessment questions, 68% of which (840 of 1245) had a lower ranking for question 2b, to the applicant's disadvantage. This may be reflective of a student's poor fit at that individual residency program, rather than that student's overall potential as an EM resident. The EM faculty may provide more specific details to explain a ranking difference in the eSLOE's section D free-text narrative.
An analysis of EM clerkship grades revealed that 76% (5101 of 6715) of EM applicants received honors or high pass grades. Of those with honors, 38% (1012 of 2644) received top 10% Global Assessment ranking. Of those with high pass grades, 3% (83 of 2457) received top 10% and 53% (1293 of 2457) received middle one-third Global Assessment rankings (Table 2 and supplemental Table 2).
The overwhelming majority of applicants are ranked positively for the first 5 Qualifications for EM categories as either “above peers (top one-third)” or “at level of peers (middle one-third).” While the assessments for these categories may not be as discriminating as the overall Global Assessment, “below peers (lower one-third)” rankings are a notable negative outlier and may identify poor applicants (Table 3).
Table 3.
Qualifications for Emergency Medicinea
| Qualification | No. (%) |
| 1. Commitment to emergency medicine and carefully thought out career choice | |
| Above peers (top one-third) | 3560 (53) |
| At level of peers (middle one-third) | 3005 (45) |
| Below peers (lower one-third) | 150 (2) |
| 2. Work ethic, willingness to assume responsibility | |
| Above peers (top one-third) | 4441 (66) |
| At level of peers (middle one-third) | 2162 (32) |
| Below peers (lower one-third) | 112 (2) |
| 3. Ability to develop and justify an appropriate differential and a cohesive treatment plan | |
| Above peers (top one-third) | 2607 (39) |
| At level of peers (middle one-third) | 3619 (54) |
| Below peers (lower one-third) | 489 (7) |
| 4. Ability to work with a team | |
| Above peers (top one-third) | 3990 (59) |
| At level of peers (middle one-third) | 2562 (38) |
| Below peers (lower one-third) | 163 (2) |
| 5. Ability to communicate a caring nature to patients | |
| Above peers (top one-third) | 3773 (56) |
| At level of peers (middle one-third) | 2849 (42) |
| Below peers (lower one-third) | 93 (1) |
| 6. How much guidance do you predict this applicant will need during residency? | |
| Less than peers | 2339 (35) |
| Same as peers | 3797 (57) |
| More than peers | 578 (9) |
| 7. Given the necessary guidance, what is your prediction of success for the applicant? | |
| Outstanding | 3951 (59) |
| Excellent | 3278 (49) |
| Good | 486 (7) |
N = 6715.
2011–2012 SLOR Data Compared With 2016–2017 eSLOE Data
A partial representation of the 2011–2012 SLOR Global Assessment data, with top 10% (40%, 234 of 583), top one-third (43%, 251 of 583), middle one-third (15%, 89 of 583), and lower one-third (2%, 9 of 583), was compared with the ranking usage in 2016–2017 eSLOE.6 There has been a decrease in the higher ranking designations, from the top 10% (40%, 234 of 583) in 2011–2012 to the top 10% (18%, 1227 of 6715) in 2016–2017 (Table 4).
Table 4.
Comparison Breakdown From 2011–2012 and 2016–2017 SLOEsa
| Evaluation | 2011–2012, No. (%) | 2016–2017, No. (%) |
| Clerkship grade | ||
| Honors | 315 (57) | 2644 (39) |
| High pass | 189 (34) | 2457 (37) |
| Pass | 45 (8) | 1552 (23) |
| Global Assessment | ||
| Top 10% | 234 (40) | 1227 (18) |
| Top one-third | 251 (43) | 2482 (37) |
| Middle one-third | 89 (15) | 2318 (35) |
| Lower one-third | 9 (2) | 684 (10) |
2011–2012: N = 556 (clerkship grade) and N = 583 (Global Assessment); 2016–2017: N = 6715.
Discussion
The distribution spread of Global Assessment rankings and notable negative outlier Qualifications for EM rankings are significant findings from the eSLOE data. These data enhance the eSLOE's ability to discriminate among applicants during the residency application process and standardize identification of applicant strengths and weaknesses. There has been an improved spread of distribution of clerkship grades and applicant Global Assessment in the 2016–2017 eSLOE compared with a partial representative 2011–2012 SLOR data set.6 In contrast, the EM clerkship grade may be less discriminatory than the Global Assessment.
It is important to discriminate among applicants by utilizing the entire eSLOE ranking scale. Faculty members speculate that some medical schools routinely inflate student assessments and SLOE rankings, and thus do not use the full ranking scale as directed by the SLOE. However, the distribution of the 2016–2017 data for the Global Assessment rankings suggests that the eSLOE is more discriminatory for applicant comparison and assessment purposes, with less inflated assessments, compared with the past. Institutional eSLOE user dashboards with bar graphs help faculty visualize their overall Global Assessment ranking distribution, encouraging proper usage of all rank designations. Annual institutional statistics can identify programs that positively overly inflate assessments, instead of appropriately using the lower rankings.
The clerkship grade distribution is not as discerning as the faculty's overall Global Assessment on the eSLOE. Potential influences on the grade designation include schools with only pass/fail grading, schools with grade inflation, and schools with percentage grade quotas or criteria. Because of these confounding influences, there may not be congruence with the clerkship grade and the overall eSLOE Global Assessment. Since medical students waive their educational right to access their eSLOEs as part of the ERAS process, eSLOE data are particularly helpfully for guiding students when applying to EM residency programs. Since students know their final EM clerkship grades, they can use the data table to better understand and approximate their likely eSLOE Global Assessment rankings (supplemental Table 2).
Since the original implementation of the SLOR/SLOE for EM residency applicant assessments, there has been significant, annual effort in specific SLOE authorship training for academic EM faculty. The implementation of the website requires academic EM credentials for authorship log in. Use of a group eSLOE authorship, with synthesized and consensus assessments by clerkship or residency director leadership, is more widely and commonly used. The website eliminates eSLOE authorship by individual, nonacademic EM physicians and effectively creates a national cohort of experienced eSLOE authors to utilize the eSLOE in a standardized way. This will allow for enhanced future eSLOE data comparison over time.
This study has limitations. In analyzing the distribution of rankings, we did not analyze based on single versus group faculty authorship, which may affect overall rankings. Furthermore, correlation with ranking distributions and amount of SLOE authorship training was not performed. Although the distribution of assessments has improved significantly with the eSLOE, the accuracy of assessments and whether they predict future EM resident performance remain unknown.
With the electronic generation of the eSLOE and widespread adoption in the EM residency application process, the collection of inputted data can be sorted by different variables and used for future research. Future research projects related to the EM clerkship, institutional and geographic distribution of eSLOEs, and unsuccessful EM applicants would be of potential interest. Consideration and support for a formalized annual process to review and publish eSLOE data would be of great benefit to academic EM. Both the eSLOE website for academic faculty and its discerning utility in residency selection are potential successful models for other graduate medical education specialties.
Conclusion
This article emphasizes the utility of the eSLOE as a reliable tool to differentiate EM applicants, demonstrated by the improved distribution of the Global Assessment and significance of the negative rankings for Qualifications for EM, enhancing the EM residency application review process.
Supplementary Material
References
- 1.Keim SM, Rein JA, Chisholm C, Dyne PL, Hendey GW, Jouriles NJ, et al. A standardized letter of recommendation for residency application. Acad Emerg Med. 1999;6(11):1141–1146. doi: 10.1111/j.1553-2712.1999.tb00117.x. [DOI] [PubMed] [Google Scholar]
- 2.Love JN, Smith J, Weizberg M, Doty CI, Garra G, Avegno J, et al. Council of Emergency Medicine Residency Directors' standardized letter of recommendation: the program director's perspective. Acad Emerg Med. 2014;21(6):680–687. doi: 10.1111/acem.12384. [DOI] [PubMed] [Google Scholar]
- 3.Hegarty CB, Lane DR, Love JN, Doty CI, DeIorio NM, Ronan-Bentle S, et al. Council of Emergency Medicine Residency Directors standardized letter of recommendation writers' questionnaire. J Grad Med Educ. 2014;6(2):301–306. doi: 10.4300/JGME-D-13-00299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Council of Emergency Medicine Residency Directors. eSLOE electronic standardized letter of evaluation. 2018 https://www.cordem.org/esloe Accessed January 21.
- 5.National Resident Matching Program. Results and data: 2017 main residency Match. http://www.nrmp.org/wp-content/uploads/2017/06/Main-Match-Results-and-Data-2017.pdf Accessed February 6, 2019.
- 6.Love J, Deiorio NM, Ronan-Bentle S, Howell JM, Doty CI, Lane DR, et al. Characterization of the Council of Emergency Medicine residency director's standard letter of recommendation in 2011–2012. Acad Emerg Med. 2013;20(9):926–932. doi: 10.1111/acem.12214. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.