A Retrospective Cohort Study of the Effect of Home Institution on Emergency Medicine Standardized Letters of Evaluation

Abstract

Background

The standardized letter of evaluation (SLOE) was developed to make letters of recommendation in emergency medicine (EM) more objective and discerning. Typically, students obtain one SLOE from a home EM rotation and at least one more SLOE from an away clerkship. It is unclear if students perform better on their home or away EM rotations.

Objective

The purpose of this study was to determine if students perform better on the group SLOE at their home institution compared to an away institution.

Methods

The authors performed a retrospective application review of all allopathic applicants to an urban, academic EM residency program. The authors calculated a composite score (CS) for each group SLOE, using the global assessment scores for comparative rank and rank list position. A lower CS indicates better performance. The authors compared mean CS for students’ first home rotations with first away rotations. For students in the study who had a third (second away or second home site) SLOE available, the authors compared mean CS on the students’ first SLOEs with mean CS on the students’ third SLOEs.

Results

A total of 624 records were included in the primary analysis. There was a small, but significant difference between mean CS for students’ home rotations when compared to away rotations (4.67 vs. 4.85, p = 0.024). Students performed better on their home rotations. Students who had three SLOEs available performed worse on their third rotation (first = 4.40, second = 4.63, third = 4.77, p = 0.012 for first vs. third). For all available SLOEs, more than 50% of students fell into the top 10% or top one‐third categories.

Conclusion

Students perform slightly better on their home EM rotations. Students’ mean SLOE CS is slightly worse for a third rotation when compared to a first rotation.

The standardized letter of evaluation (SLOE) was created by the Council of Residency Directors in Emergency Medicine (CORD‐EM) to make letters in emergency medicine (EM) more standardized, concise, and discriminating.1, 2 Standardized letters are more efficient to review and offer better inter‐rater reliability than narrative letters of recommendation (NLOR).3 The SLOE stratifies students into quantiles based on their performance in various categories, including an overall assessment. The SLOE may provide a more objective assessment of student performance than a NLOR.2

Most advisors in EM recommend that students obtain at least one SLOE from an EM clerkship rotation at their home institution and at least one other SLOE from an “away” EM rotation.4 These SLOEs typically replace at least two NLORs in a student's electronic residency application service (ERAS) application to EM residency programs. Currently, the SLOE is the selection factor used most frequently by program directors (PDs) to make interview and ranking decisions.5, 6

Despite efforts to maintain objectivity, SLOE authors may inflate scores.1, 7, 8, 9, 10 Many SLOE authors do not adhere strictly to quantile categories because they are concerned about negatively impacting a student's ability to match.11 It is unclear whether students receive higher scores on the group SLOEs from home rotations versus away rotations. We hypothesized that faculty at a student's home institution would be more concerned for their own student's ability to match than for an external student. We sought to determine whether students perform better on the SLOE at their home institution compared to an away rotation.

Methods

Study Design

We performed a retrospective application review of all U.S. MD applications submitted to the University of California at Irvine EM residency program through ERAS in the 2015 and 2016 application cycles. We obtained institutional review board (IRB) approval from the University of California at Irvine prior to commencing this study. The IRB did not require subject consent.

Study Setting and Population

We included ERAS records for applicants from Liaison Committee for Medical Education accredited U.S. allopathic medical schools.

Study Protocol

Two trained, nonblinded data abstractors collected the following data onto a standardized data abstraction form: Association of American Medical Colleges identification number, year of application, sex of applicant, student's home institution, step 1 score, step 2 clinical knowledge (CK) score, step 2 CS score, and SLOE data. For each SLOE, we collected date and location of rotation, author type, number of letters authored by author in the previous year, comparative rank score (CRS), and rank list position score (RLPS). The CRS is the response to question C1: “Compared to other EM residency candidates you have recommended in the last academic year the candidate is: top 10%, top third, middle third, lower third.” The RLPS is the response to the question C2b: “How highly would you estimate the candidate will reside on your rank list: top 10%, top third, middle third, lower third, unlikely to be on our rank list.” The senior investigator held periodic meetings with the abstractors. We resolved any questions via consensus. These data were stored in an online secure database, REDCap. The senior investigator reviewed all final data to ensure its accuracy, including reviewing whether each home and away institution had been coded correctly. The senior investigator sampled 5% of all records for review and calculated Cohen's kappa for inter‐rater reliability.

Inclusion and Exclusion Criteria

We screened all SLOEs from traditional 4‐week or 1‐month EM rotations. We recorded data for all SLOEs from U.S. MD applicants. For the primary study analysis, we excluded SLOEs that were not authored by a faculty group, PD, clerkship director (CD), or any combination thereof. Thus, we did not include SLOEs authored by a single non‐PD or CD faculty member, chair, or nonfaculty member, since we felt that they may not have the most accurate information regarding the student's comparative clerkship performance and rank list position. We excluded SLOEs with incomplete data, such as a missing final global assessment or rank, and SLOEs from an author who had not written more than five SLOEs in the previous or current year. Finally, we excluded records from students whose first and second EM clerkships were not a home and away program (as determined by available SLOEs). Accordingly, we excluded applicants whose home SLOE occurred sequentially third, since not all applicants completed a third rotation. We considered a home program to be an EM training program affiliated with the student's medical school, with which the faculty have academic appointments, within 30 miles of the medical school campus. We considered any program to be an away if it was not considered a home program. When it was unclear whether a program should be considered a home or away, the primary and secondary authors e‐mailed the PD for clarification.

Key Outcome Measures

The primary outcome of this study was the home institution effect on SLOE outcome. Secondary outcomes included whether students did better on their first, second, or third SLOE rotation. We described the distribution of scores for all SLOEs. Although not in our original protocol, we compared mean SLOE scores for students who were excluded from the primary analysis for not having a home rotation with mean SLOE scores for students in primary study group.

Data Analysis

To determine home institution effect on SLOE outcome, we compared performance on students’ first home SLOE to performance on first away SLOE. We calculated a composite score (CS) for the SLOE by adding the students’ CRS and RLPS. Since the CRS has four options (top 10%, top third, middle third, lower third) and the RLPS has five options (top 10%, top third, middle third, lower third, unlikely to be on our rank list), the best possible CS attainable was a “2” (top 10% in both categories). The worst possible score was a “9” (lower third and unlikely to be ranked). We calculated the CS of each student's first home SLOE and compared it to his or her first away SLOE using the related‐samples Wilcoxon signed‐rank test. We analyzed data using IBM SPSS Statistics for Windows, version 25.0. We wanted to detect at least a 15% difference in the scores of home and away SLOEs, which corresponds to approximately a one tertile difference in one category for a middle third, middle third SLOE. We assumed the least amount of reliability in the outcome at 50%. To have a power of 80% and a significance of p = 0.05, the required sample size was 167 participants in each group.

We also describe the distribution of student performance on home versus away rotations. We calculated the difference between the first home CS and the first away CS for each student and reported the percentage of students performing better, worse, or no difference on home SLOEs. For example, a score difference of two means that the student did one quantile better (or worse) in both categories (e.g., middle third, middle third to top third, top third) or did two quantiles better in one category (e.g., lower third, middle third to top third, middle third).

For students who had a third SLOE available (either a second away or a second home site), we report the mean CS for the first, second, and third SLOEs for that cohort and compare performance between first and third SLOE, using paired‐sample t‐test.

Although not in our a priori protocol, we sought to describe performance for students without a home rotation. We calculated mean CS for students who were originally excluded from our primary analysis for not having a home rotation. We compared the CS from their second SLOE with the primary study mean away CS, using Mann‐Whitney U‐test.

Results

There were 2,078 US allopathic seniors applying to EM in 2015 and 1988 in 2016.12 For the 2015 application cycle, the University of California at Irvine EM Residency Program received 640 applications. For 2016, the program received 768 applications, for a total of 1,408 subjects. Of these, 1,053 were applicants from U.S. allopathic medical schools. We recorded data for these applicants.

For the primary data analysis, we excluded five records because the student did not have any SLOEs in their application. We excluded the records of 59 applicants because their home or away SLOE was not a group, CD, or PD SLOE or the author had not written more than five SLOEs in the previous or current year. We excluded 25 records for not having complete data (e.g., the student was not given a CRS). We excluded 156 records because the student's first or second SLOE was not a home SLOE; we excluded 184 records for not having an away SLOE. We analyzed records from 624 applicants in our primary analysis (Figure 1). The measured agreement between reviewers was very good (κ = 0.919–1.000).

Figure 1.

Inclusion and exclusion of study subjects. CD = clerkship director; DO = doctor of osteopathic medicine; IMG = international medical graduate; PD = program director; SLOE = standardized letter of evaluation.

Of the 624 applicants, 213 (34%) were female. The mean USMLE Step 1 was 232 (standard deviation [SD] = 16, range 167–271); the mean USMLE Step 2 score was 244 (SD ± 14, range 192–283). Applicants were from medical schools in the West (n = 205, 33%), Midwest (n = 122, 19%), South (n = 145, 23%), and Northeast (n = 152, 24%). Twenty‐five applicants were from our own medical school.

There was a small, but statistically significant difference between students’ home rotation CS and away rotation CS, with students performing better (lower CS) on their home rotations (4.67 vs. 4.85, p = 0.024). The majority of students completed their home rotation prior to their away rotation (n = 559, 90%). Students’ home CRSs were top 10% (n = 117, 19%), top third (n = 242, 39%), middle third (n = 206, 33%), and lower third (n = 59, 9%). Their away CRSs were top 10% (n = 101, 16%), top third (n = 250, 40%), middle third (n = 211, 34%), and lower third (n = 62, 10%). Students’ home RPLSs were top 10% (n = 121, 19%), top third (n = 249, 40%), middle third (n = 178, 29%), lower third (n = 73, 12%), and unlikely to be ranked (n = 3, <1%). Away RPLSs were top 10% (n = 95, 15%), top third (n = 216, 35%), middle third (n = 237, 38%), lower third (n = 71, 11%), and unlikely to be ranked (n = 5, 1%; Figure 2).

Figure 2.

Percentage of students (total n = 624) receiving a top 10% or top one‐third rating CRS and RLPS for home versus away rotations. CRS = comparative rank score; RLPS = rank list position score.

One‐quarter of students had the same home and away CS (n = 158, 25%). More students performed better on their home rotation (n = 250, 40%). Of these, 185 performed two or more points better (30%). Thirty‐five percent of students performed better on their away rotation (n = 216, 35%), with 142 (23%) performing two or more points better (Figure 3).

Figure 3.

Percentage of students whose CS is better, worse, or the same for home versus away rotations. CS = composite score.

A total of 175 (28%) had a third SLOE available. Of these, we excluded two because they did not have complete data. We excluded seven because they were not written by a group, PD, or CD or author that wrote more than five SLOEs in the previous year, leaving 166 for use in our study. A student's third SLOE was usually an away rotation (n = 162, 98%). For this cohort, the mean CSs for students’ first, second, and third rotations were 4.40, 4.63, and 4.77, respectively (p = 0.012 for first vs. third).

For all SLOEs, including the 624 home SLOEs, 624 away SLOEs, and 166 third SLOEs, 17% of CRSs were in the top 10% (n = 246), 40% (n = 562) were in the top third, 33% (n = 469) were in the middle third, and 10% (n = 137) were in the lower third. For the RLPS, 17% (n = 242) of students fell in the top 10%, 38% (n = 531) were in the top third, 33% (n = 465) were in the middle third, 12% (n = 168) were in the lower third, and less than 1% were “unlikely to be ranked” (n = 8).

We performed a secondary analysis on the cohort of applicants (n = 156) who were excluded from our primary analysis for not having a home first or second SLOE. Of these 156 applicants, 29 had SLOEs from only one clerkship. Another three applicants had SLOEs without complete data or they were written by authors who had not written more than five SLOEs in the previous year. Two students had third (but not first or second) SLOEs that were home SLOEs, leaving 122 applicants to analyze. The mean SLOE CS for these 122 students’ second SLOE was 5.43, when compared with our study mean for away SLOEs of 4.85 (p = 0.001). This cohort's first SLOE CS was 5.12. Third SLOE, when available, averaged 4.85.

Discussion

It is well documented that SLOEs are the most highly rated and used selection factor by program leadership when selecting applicants for their EM program.5, 6 Most advisors recommend that a student obtain at least two SLOEs to be a competitive applicant in EM.4 From our sample of U.S. allopathic medical students, most students (67%) did at least one home rotation and at least one away rotation. Most programs provided a group, PD, or CD SLOE, as very few applicants were excluded from the study for not having a group home or away SLOE.

Our study found a small, but statistically significant difference between students’ scores on their home rotation versus their away rotation. We suspect that students tend to do better on their home rotation because program and clerkship leadership may be hesitant to rank a student in a lower category for fear they will not match. Furthermore, at a student's home site, students may be more familiar with the hospital system, staff, and consultants and thus perform better. Additionally, students may have been involved in activities at their home institution that gave them more exposure to the SLOE writers. Familiarity with the applicant has been associated with higher SLOE scores in a previous study.13 We saw a trend of worsening SLOE scores with a student's first, second, and third rotations; it is possible our study difference could be explained by rotation sequence rather than home versus away.

The overall distribution of SLOE scores placed more applicants in the top 10% and top third categories than would be expected. In our study, 57% of candidates fell into either the top 10% or top one‐third categories for CRS and 55% of candidates fell into either category for RLPS, exceeding the expected 33%. These findings are consistent with previous reports.1, 10

The SLOE is intended to stratify applicants into quantile groups. Residency and clerkship leadership are in the roles of both advisor and evaluator. They may be faced with the struggle between objectively assessing students and wanting them to match. Program and clerkship directors likely acknowledge the impact that placing a candidate into the lower categories may have on their competitiveness for a residency position.11 With only 4 weeks to observe a student, it may also be difficult to neatly place them into a quantile. Since it is common practice to place more students into higher categories, perhaps the SLOE should adopt a system where the lower categories are intended for only a smaller percentage of students; however, if such a system were adopted, lower quantile students would likely have even more difficulty matching and even fewer students would be placed in the lower quantile. It would be interesting to study whether students who receive a lower third on their SLOE have decreased likelihood of matching into EM.

Advisors should acknowledge that—for students with a home EM rotation—students’ scores and rankings tend to decrease among their first, second, and third SLOEs. Students should be aware that they will likely not do better on their third rotation. They may still have other reasons to do a third rotation, such as finding whether they “fit” with a program or expanding the regional reach of their application.

While not in our a priori study design, we examined the average SLOE score for students who did not have a home rotation. To make the comparison fair, we compared the scores from their second SLOEs with the scores of the study subjects’ away SLOEs. Students who did not have a home rotation performed significantly worse on their SLOE than students who did have a home rotation. Most of these students were from medical schools without EM residency training programs. These students may have been less prepared than students from schools with an EM program, since they may not have an EM advisor. Interestingly, their scores improved on their third rotation. It may be wise to advise students without a home rotation to do a third rotation.

Limitations

There are several limitations to our study. First, we only reviewed SLOEs that were provided to our single program through ERAS. Applicants may not have provided SLOEs from all the rotations they completed, as there are only four LOR spots available in ERAS. There were more students than expected in the top quantiles of the SLOEs. It is possible that students withheld SLOEs from programs at which they feel they did not perform well. It is also possible a stronger cohort of applicants applied to our program; however, our study mean USMLE was similar/slightly lower than the national mean (2016 = 233, Step 1 = 245, Step 2 = CKCKCK)14 for matched U.S. senior EM applicants.

Universally, applicants who rotated at our program applied to our program through ERAS; thus, our program's ranking practices could have been disproportionately represented in the applicant pool. While it is possible that there was selection bias in the study, we believe that our study represents the general population of EM applicants nationwide. While the study USMLE mean was slightly lower than the average for matched EM seniors, our study population includes both matched and unmatched U.S. MD EM applicants. Applicants were geographically distributed throughout the United States, but there were more applicants from western medical schools. Applicants to our program represented approximately one‐quarter of total U.S. MD EM applicants.

For our first, second, and third SLOE data, we relied on programs reporting accurate dates for the student's rotation. While we did see a small advantage at applicants’ home institution, we did not adjust the study for quality of the home institution. It is possible that students rotated at more competitive away institutions, when compared to their home institution. It would be difficult to adjust for this variable, as there is not a reliable method for ranking residency training programs. Since most students did their away rotation second in the sequence of rotations, our study findings may be due to rotation sequence, rather than home versus away.

Conclusion

The standardized letter of evaluation is one of the most important parts of a student's application to emergency medicine residency. Students perform slightly better on their home rotation, when compared to an away rotation. Advisors should make sure that students are aware that standardized letter of evaluation performance may decline on a third rotation, unless the student is from a medical school without an emergency medicine residency.

AEM Education and Training 2019;3:340–346.