The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

ABSTRACT

Objectives

The standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE. However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.

Methods

A multi‐institutional cross‐sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019–2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.

Results

In total, 2,106 unique applicants (4,939 SLOEs) were included. Of the seven qualifications for EM questions, three were moderately to strongly correlated with global assessment and anticipated rank list position: “ability to develop and justify an appropriate differential and a cohesive treatment plan” (ρ = 0.65 and ρ = 0.63, respectively; p < 0.001), “how much guidance do you predict this applicant will need during residency?” (ρ = 0.68 and ρ = 0.68, respectively; p < 0.001), and “what is your prediction of success for the applicant?” (ρ = 0.69 and ρ = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.

Conclusions

There was a moderate to strong correlation between three of seven qualifications for EM questions (ability to develop and justify a differential and develop a cohesive plan, anticipated need for the amount of guidance, and prediction of success) with both global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data to residency selection while some of the other factors may not be needed.

INTRODUCTION

Letters of recommendation are an important component of residency applications. As the number of emergency medicine (EM) residency programs and student applicants to EM increases, it can be difficult for residency program directors (PDs) to select which applicants to invite for an interview and to rank. ¹ , ² While metrics such as U.S. Medical Licensing Examination (USMLE) Step scores and the Medical School Performance Evaluations (MSPE) provide applicant information, the EM standardized letter of evaluation (SLOE) has been identified by residency PDs as the leading metric for determining whether to interview an applicant and one of the leading metrics for ranking applicants. ² , ³ , ⁴

Previous research has focused on the grades and the global assessment portion of the SLOE, describing the importance of these sections but limitations of the processes by which grades and global assessments are determined. ⁵ , ⁶ However, there remains a paucity of literature on the SLOE’s qualifications for EM section. Within this section, applicants are compared to other applicants in (1) commitment to EM, (2) work ethic/willingness to assume responsibility, (3) ability to develop and justify an appropriate differential and a cohesive treatment plan, (4) ability to work with a team, (5) ability to communicate a caring nature to patients, (6) anticipated amount of guidance that the applicant will need during residency, and (7) prediction of success for the applicant. ⁷ In 2014, PDs categorized the qualifications questions as “good questions.” ³ Yet a recent survey raised concern on whether the qualifications section actually provides useful information. ⁶ Additionally, there is no literature evaluating the correlation of the qualifications questions with other aspects of the SLOE (e.g., grades, global assessment, and SLOE author's anticipated rank list position) as a means to provide a component of validity evidence for the seven questions, particularly for “internal structure” and “relationship to other variables” according to Messick's validity framework. ⁸ Because the SLOE is highly valued by PDs in selecting applicants to invite for an interview and ranking applicants, it is important to explore how the qualifications questions may correlate with overall performance and anticipated rank list position. The aim of this study was to determine the correlation between the seven qualifications questions and grades, global assessment, and anticipated rank list position for applicants applying into EM.

METHODS

Study design

This was a multi‐institutional cross‐sectional study of SLOEs from applicants to three U.S. EM residency programs (Rush University, Stanford University, and University of Florida–Jacksonville) in June 2020 after completion of the 2019–2020 application cycle. This study was deemed exempt by the institutional review board at all three institutions.

Study setting and population

All applicants from U.S. Liaison Committee for Medical Education (LCME)‐accredited allopathic medical schools and osteopathic medical schools who applied to at least one of the three institutions’ EM residency programs were included in the study. The participating institutions were deliberately selected to represent three geographically distinct locations (Midwest, South, and West) with 3‐ and 4‐year residency training programs and varying levels of academic and community training experiences. Exclusion criteria consisted of applicants from a non‐LCME institution and applicants with no provided SLOE. Subspecialty SLOEs, SLOEs with incomplete data including SLOEs from institutions without a residency programs to provide information on rank list position, SLOEs not written by program leadership (defined as a PD, assistant or associate PD, clerkship director, vice chair, or chair), and SLOEs written by a letter writer who wrote less than 10 SLOEs the previous year were also excluded. To limit data skew, SLOEs from programs who provide pass/fail grades to applicants were excluded for the analysis between grades and qualifications only.

Study protocol

We downloaded applicant data via the Electronic Residency Application Service (ERAS) including Association of American Medical Colleges (AAMC) identification number, self‐identified gender, self‐identified race, USMLE Steps 1 and 2 clinical knowledge (CK) scores, and SLOEs. SLOE data included EM rotation grade, ranking for all qualifications for EM questions, global assessment rating (“Compared to other EM residency candidates you have recommended in the last academic year, this candidate is in the” top 10%, top 1/3, middle 1/3, or bottom 1/3), and estimated rank list rating (“How highly would you estimate the candidate will reside on your rank list?” top 10%, top 1/3, middle 1/3, or bottom 1/3). Trained abstractors from each institution collected data using a prepiloted standardized data abstraction tool. A nondisclosure agreement between the three institutions allowed only AAMC numbers to be shared among the institutions. AAMC numbers were compared and duplicate data was removed prior to data analysis.

Measurements and data analysis

We used measures of central tendency to analyze demographic information and score distribution for the seven qualifications. Normality (e.g., parametricity) of the data was verified with the Shapiro–Wilk test; nonnormal data were described with median and interquartile range and was compared by nonparametric testing with the Wilcoxon–Mann–Whitney test. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grade and global assessment scores in a pairwise fashion. A moderate to strong correlation was defined as an r > 0.60. ⁹ Statistical significance was set at p < 0.05. We performed all analyses using Stata Statistical Software, Version 14 (StataCorp LP, College Station, TX, USA).

RESULTS

A total of 3,250 applicants met initial inclusion criteria (Rush University = 1,352; Stanford University = 813; University of Florida–Jacksonville = 1,085). A total of 1,064 applicants were excluded for overlap between the residency programs and 80 applicants were excluded for no provided SLOE, leaving 2,106 total applicants. This represented 61.9% (2,106/3,405) of all EM applicants for the 2019–2020 application cycle. ¹⁰ Demographic data of the applicants included 1,290 (61.3%) men, 813 (38.6%) women, three (0.1%) who declined to answer, with 382 (18.1%) underrepresented in medicine applicants (as defined by American Indian, Alaskan Native, Hispanic, Latino, Spanish origin, Mexican/Chicano, African American, African, Asian‐Filipino, Native Hawaiian/Pacific Islander). ¹¹ This was comparable to national demographic data of all EM applicants (men 62.7%, women 37.3%, declined to answer 0.04%, and underrepresented in medicine 17.7%). ¹⁰ The mean (±SD) USMLE Steps 1 and 2 CK scores were 230.0 (±16.18) and 247.7 (±20.7), respectively, compared to national means (±SDs) of 230.9 (±15.9) and 244.8 (±14.1), respectively. ¹²

There were 5,717 total SLOEs. After excluding subspecialty SLOEs (n = 60), SLOEs with incomplete data (n = 118), SLOEs improperly coded by abstractors (n = 19), SLOEs not written by program leadership (n = 157), and SLOEs written by a letter writer who wrote < 10 SLOEs (n = 424), we analyzed 4,939 (86.4%) SLOEs.

Distribution of scores for the seven qualifications and correlation coefficients appear in Table 1. Of the seven qualification questions, three were moderately to strongly correlated ⁹ with global assessment and anticipated rank list position, including “ability to develop and justify an appropriate differential and a cohesive treatment plan” (r = 0.65 and r = 0.63, respectively; p < 0.001), “how much guidance do you predict this applicant will need during residency?” (r = 0.68 and r = 0.68, respectively; p < 0.001), “what is your prediction of success for the applicant?” (r = 0.69 and r = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.

TABLE 1.

Distribution of scores and correlation coefficients of the seven qualifications for EM

Qualification for emergency medicine	Number of SLOEs with lower 1/3 (%)	Number of SLOEs with middle 1/3 (%)	Number of SLOEs with top 1/3 (%)	Correlation coefficient grade	Correlation coefficient global assessment	Correlation coefficient rank list position
Commitment to emergency medicine	83 (1.9%)	2,350 (47.6%)	2,506 (50.7%)	0.33*	0.45*	0.47*
Work ethic/willingness to assume responsibility	103 (2.1%)	1,516 (30.7%)	3,320 (67.2%)	0.42*	0.54*	0.55*
Ability to develop and justify an appropriate differential and a cohesive treatment plan	407 (8.2%)	2,679 (54.2%)	1,853 (37.5%)	0.48*	0.65*	0.63*
Ability to work with a team	125 (2.5%)	1,879 (38.0%)	2,935 (59.4%)	0.39*	0.55*	0.57*
Ability to communicate a caring nature to patients	83 (1.9%)	2,144 (43.4%)	2,712 (54.9%)	0.36*	0.48*	0.49*
Anticipated amount of guidance that the applicant will need during residency	448 (9.1%)	2,814 (57.0%)	1,677 (34.0%)	0.47*	0.68*	0.68*
Prediction of success for the applicant	362 (7.3%)	2,510 (50.8%)	2,067 (41.9%)	0.53*	0.69*	0.69*

Moderate to strong correlation consisted of an r = 0.6–1.0.

Abbreviation: SLOE, standardized letter of evaluation.

^* p < 0.001.

DISCUSSION

Our study found a moderate to strong correlation between three qualifications: ability to develop and justify a cohesive treatment plan, anticipated guidance for the applicant, and prediction of success for the applicant, with both global assessment and anticipated rank list position, suggesting that these qualifications could potentially provide the most useful data to residency selection. There has been debate in the literature about the value of the seven qualifications section and its use in interview selection and residency ranking. ³ , ¹³ Love et al. ³ surveyed PDs on which components of the qualifications of EM were “good” questions. All qualifications were deemed a “good” question by over 70% except “commitment to EM” (46.7%) and “ability to communicate a caring nature to patients” (67.3%). This is consistent with our findings that neither of these two qualifications were strongly correlated with grade, anticipated rank list position, or global assessment. Additionally, Hegarty et al. ¹³ conducted a survey of SLOE authors who recommended removal of “commitment to EM’’ (37.7%) and “given the necessary guidance, what is your prediction of success for the applicant?’’(30.2%). Again “commitment to EM” appeared to add little information. However, anticipated success of the residents was one of the few qualifications that correlated with other important metrics in our study. Our study adds to this by providing objective evidence on which qualifications are correlated with important metrics such as global assessment and anticipated rank list position and may offer guidance for residency interview and rank selection.

Consistent with other studies on grade inflation in letters of recommendation, our study also found that the qualifications section had disproportionately high rankings. ⁵ , ¹³ , ¹⁴ When examining the distribution of scores for the seven qualifications in our study, less than 10% of SLOEs in all categories fell in the lower one‐third category. This is in alignment with previous studies that show that only 39% of letter writers strictly adhere to the guidelines of ranking equal amounts of students in the top, middle, and lower third ⁶ and that 51% of letter writers were most likely to use the highest/most desirable category on qualifications variables. ⁵ , ¹⁴ Previous explanations suggest that SLOE writers are hesitant to use lower rating categories for fear of diminishing an applicant's invitation to interview. However, this inaccurate distribution of scores limits the ability of programs to adequately interpret sections of the SLOE, including the qualifications section. Thus, the qualifications section may benefit from revisions using validated workplace‐based assessment scales including behavioral anchoring descriptions to ensure more consistent rating by evaluators. ⁷ , ¹⁵ , ¹⁶ Finally, our study found no strong correlation between the qualifications and grades. Previous research has reported grade and rank inflation to occur in up to 60% of SLOEs. ¹³ The lack of strong correlation between the qualifications and grades may be due to grade inflation resulting in limited ability to discriminate between applicants.

LIMITATIONS

There are several limitations to this study. First, while this was a multi‐institutional study with three programs representing diverse geographic locations, several exclusion criteria may limit generalizability to all EM applicant SLOEs, particularly those not written by program leadership or those authored by a letter writer who wrote < 10 SLOEs in the prior year. Furthermore, this study provides information on relationship to other variables and internal structure evidence for the seven qualifications but does not offer content, response process, or consequence evidence. ⁸ Finally, it is possible that the noncorrelated qualifications may predict applicant abilities or provide other value to selection committees but lack internal consistency with other aspects of the SLOE. Further studies may be helpful to collect additional validity evidence for the SLOE and determine the value of the qualifications questions to programs beyond the correlation with grade, global assessment, and predicted rank list position.

CONCLUSION

There was a moderate to strong correlation between three qualifications with global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data for residency selection.

CONFLICT OF INTEREST

The authors have no potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

Danielle T. Miller: study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content. Sara Krzyzaniak: study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content. Alexandra Mannix: study concept and design, acquisition of the data, analysis and interpretation of the data, critical revision of the manuscript for important intellectual content. Al'ai Alvarez: study concept and design, acquisition of the data, critical revision of the manuscript for important intellectual content. Teresa Chan: study concept and design, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical expertise. Dayle Davenport: study concept and design, acquisition of the data, critical revision of the manuscript for important intellectual content. Daniel Eraso: study concept and design, acquisition of the data, critical revision of the manuscript for important intellectual content. C. J. Foote: Analysis and interpretation of the data, critical revision of the manuscript for important intellectual content, statistical expertise. Katarzyna Gore: Study concept and design, acquisition of the data, critical revision of the manuscript for important intellectual content. Melissa Parsons: study concept and design, acquisition of the data, critical revision of the manuscript for important intellectual content. Michael Gottlieb: study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, critical revision of the manuscript for important intellectual content.

Supporting information

Data Supplement S1. Supplemental material.