Factors Affecting Candidate Placement on an Emergency Medicine Residency Program’s Rank Order List

Abstract

Introduction:

Several factors influence the final placement of a medical student candidate on an emergency medicine (EM) residency program’s rank order list, including EM grade, standardized letter of recommendation, medical school class rank, and US Medical License Examination (USMLE) scores. We sought to determine the correlation of these parameters with a candidate’s final rank on a residency program’s rank order list.

Methods:

We used a retrospective cohort design to examine 129 candidate packets from an EM residency program. Class ranks were assessed according to the instructions provided by the students’ medical schools. EM grades were scored from 1 (honors) to 5 (fail). Global assessments noted on the standardized letter of recommendation (SLOR) were scored from 1 (outstanding) to 4 (good). USMLE scores were reported as the candidate’s 3-digit scores. Spearman’s rank correlation coefficient was used to analyze data.

Results:

Electronic Residency Application Service packets for 127/129 (98.4%) candidates were examined. The following parameters correlated positively with a candidate’s final placement on the rank order list: EM grade, ρ = 0.379, P < 0.001; global assessment, ρ = 0.332, P < 0.001; and class rank, ρ = 0.234, P = 0.035. We found a negative correlation between final placement on the rank order list with both USMLE step 1 scores, ρ = −0.253, P=0.006; and USMLE step 2 scores, ρ = −0.348, P = 0.004.

Conclusion:

Higher scores on EM rotations, medical school class ranks, and SLOR global assessments correlated with higher placements on a rank order list, whereas candidates with higher USMLE scores had lower placements on a rank order list. However, none of the parameters examined correlated strongly with ultimate position of a candidate on the rank list, which underscores that other factors may influence a candidate’s final ranking.

INTRODUCTION

There are several key components of Electronic Residency Application Service (ERAS) packets for medical students seeking to match to an emergency medicine slot, including preclinical and clinical medical school grades, emergency medicine (EM) clerkship elective grades, standardized letters of recommendation (SLOR), and scores on step 1 and step 2 of the US Medical License Examination (USMLE).¹ Another key portion of the application is the Medical Student Performance Examination (MSPE) or Dean’s Letter, which is compiled using these data.¹ The Association of American Medical Colleges advises that the MSPE contain “a summative assessment of the student’s comparative performance in medical school relative to his/her peers, including any information about school-specific categories used in differentiating among levels of student performance.”² The SLOR is a particularly important document for candidates hoping to match in EM. It is a letter of recommendation that is universally used by EM programs for medical students who request one. In addition to the student’s clerkship grade, proficiency ratings for certain attributes, and comments from faculty members regarding the student’s performance, the SLOR also contains a global assessment score (outstanding, excellent, very good or good).^3,4

Most medical student candidates for EM residency attempt to obtain high marks in all these categories, but does academic excellence as manifested by outstanding grades, competitive class rank (CR), high USMLE scores, or the highest global assessment on the SLOR really result in a top ranking on an EM residency program’s rank order list (ROL)? Several articles outline the EM application process and the value of each part of the application packet.^1,5,6 Further, a survey of EM program directors published in 1999 revealed that EM rotation grade, interview, clinical grades, and recommendations were most important in the selection of residents and that program directors placed moderate emphasis on USMLE steps 1 and 2.⁷ A more recent study which looked at the importance of criteria used for residency selection across 21 specialties found grades in required clerkships and USMLE step 1 scores were the most important factors for residency program directors.⁸ However, we were unable to identify studies that evaluated the actual value of the various pieces of the residency application packet.

We sought to determine the correlation between EM grades, SLOR global assessment rating, CR, and USMLE steps 1 and 2 scores with a candidate’s final placement on an EM residency program’s final submitted National Residency Match Program (NRMP) rank list. Because there are no validated scoring instruments to assess the interview, we did not examine whether there was a correlation between an interview rating and final rank position.

METHODS

Study Design

This was a retrospective cohort analysis of the applications of residency candidates interviewed during 2007–2008 for matriculation into a single EM residency program for the 2008–2009 academic year. This study was conducted after interviews had finished and after the final NRMP ROL had been submitted. This study was reviewed by our institutional review board and deemed exempt from informed consent.

Study Setting and Population

This study was conducted at a single, 3-year EM residency program that sees approximately 160,000 adult and pediatric patients per year. All candidates for that program’s 12 categorical EM positions were included in the analysis.

Study Protocol

All information contained in the candidates’ ERAS applications at the time of their interview was examined after interviews for that year had been concluded and the ROL had been submitted. Application information was available for 127/129 of the candidates and was subjected to analysis. These candidate application packets were examined by a single data abstractor and were extracted onto a Microsoft Excel Spreadsheet (Redmond Washington). The abstractor, who was a study author, was blinded to the results of the final rank order list.

A priori definitions were as follows: EM grade was noted from the SLOR or from the candidate’s transcript. All listed EM grades were included. If more than one EM grade was noted, then all EM grades were used to calculate a composite score. Grades were noted in descending order with the top grade as 1 and the lowest 5. Grades marked in between were adjusted accordingly (eg, a candidate assigned an honors/high pass was given a 1.5). A final composite score was given to each candidate that was then used for correlation purposes.

A candidate’s CR was determined by the MSPE. If the medical school did not provide a ranking system, no attempt was made to determine an overall CR for that candidate. Alpha Omega Alpha (AOA) was given a 1 ranking and the next subsequent category a 2. If AOA was not used or specified on a particular school’s MSPE, then the first category was the first breakdown specified by the school. Data were recorded according to the MSPE; therefore, a particular candidate may have an overall CR of 2/4 (second quartile) or a CR of 1/5 (AOA in a quintile system). Statistically, no distinction was made between a candidate who obtained first category of a 7-category system and first category of a 3-category system.

We used the global assessment scores on the SLOR to determine ratings on the LOR. Specifically, a global assessment score of 1 was given to outstanding, 2 for excellent, 3 for very good, and 4 for good. If more than 1 box was checked, a composite score was assigned (eg, 1.5 for outstanding and excellent). If the candidate had several SLOR global assessment scores, we averaged all of them to give the candidate a composite score. If no global assessment score was given, we did not assign a score and left it blank.

USMLE scores were used as determined from the ERAS packet. Three-digit scores for steps 1 and 2 were used. If no USMLE scores were available, this information was left blank.

Rank order list information was taken from the program’s final NRMP rank list. If a candidate was unranked, one number lower than the lowest-ranked candidate was assigned. All unranked candidates received the same number.

We assigned each candidate a randomized number, and information regarding each candidate was blinded. We retrospectively examined residency applications to a single EM training program. The study was conceived after interview season was completed and the program’s NRMP ROL was submitted. The retrospective design was selected to minimize the potential for data extractor bias to skew results.

Data Analysis

All statistical analyses were conducted using Statistical Analysis Software (SAS) Version 9.0 (SAS Institute, Inc, Cary, NC). Spearman’s rank correlation coefficient (ρ) was used to analyze and compare data. A P value of ≤0.05 was considered significant.

RESULTS

We were able to locate and examine ERAS applications for 127/129 (98.4%) of candidates. Of the 127 MSPE letters received, we were able to determine CR for 76 (59.8%) of the candidates. CRs were missing in 42/127 (33.1%) applicant packets, while for 9/127 (7.1%), the CR was reported as AOA only. CR systems utilized by medical schools included the following: thirds (21/76, 27.6%), quartiles (24/76, 31.6%), quintiles (16/76, 21.1%), 6 categories (6/76, 7.9%), 7 categories (3/76, 3.9%), and overall CR with the number of students in the class (6/76, 7.9%). Compared to CR, the following ERAS information was present more often in the applicant packets we examined: EM grades (117/127, 92.1%, P < 0.001), SLOR global assessment ratings (121/127, 95.3%, P < 0.001), and USMLE step 1 scores (125/127, 98.4%, P < 0.001). CR correlated positively with a candidate’s final placement on the ROL; ρ = 0.234, P = 0.035 (Table).

Table.

Spearman’s rank correlation (ρ) along with associated two-tailed P values for information about each candidate. Significant values denoted by asterisks.

	MSPE CR	EM grade	SLOR rating	USMLE step 1	USMLE step 2	Final rank
MSPE CR
Spearman’s (ρ)		0.126	0.189	−0.572*	−0.536*	0.234*
P		0.272	0.092	0.001	0.001	0.035
EM grade
Spearman’s (ρ)	0.126		0.636*	−0.178	−0.076	0.379*
P	0.272		0.001	0.055	0.545	0.001
SLOR rating
Spearman’s (ρ)	0.189	0.635*		−0.100	−0.083	0.332*
P	0.092	0.001		0.273	0.502	0.001
USMLE step 1
Spearman’s (ρ)	−0.572*	−0.178	−0.100		0.638*	−0.253*
P	0.001	0.055	0.273		0.001	0.006
USMLE step 2
Spearman’s (ρ)	−0.536	−0.076	−0.083	0.638*		−0.348*
P	0.001	0.545	0.502	0.001		0.004
Final rank
Spearman’s (ρ)	0.234*	0.379*	0.332*	−0.253*	−0.348*
P	0.035	0.001	0.001	0.006	0.004

MSPE CR, Medical Student Performance Examination class rank; EM, emergency medicine; SLOR, standardized letters of recommendation; USMLE, US Medical License Examination.

One EM grade was noted on 117/127 of candidate applications (92.1%) with 65/127 (52%) submitting 2 or more EM grades, 6/127 (4.7%) submitting 3 or more EM grades, and 1/127 (0.79%) submitting 6 EM grades. EM grades correlated positively with a candidate’s final placement on the ROL, ρ = 0.379, P < 0.001.

At least one SLOR was included in 121/127 (95.3%) of candidate applications. Two or more SLORs were submitted by 90/127 (70.9%) of candidates, while 37/127 (29.1%) candidates submitted at least 3 SLORs, 4/127 (3.1%) candidates submitted at least 4 SLORs, and 1/127 candidates submitted at least 6 SLORs. The SLOR global assessment rating correlated positively with a candidate’s final placement on the ROL, ρ = 0.332, P < 0.001.

USMLE step 1 scores were submitted by 125/127 (98.4%) of candidates. Step 2 scores were submitted by 72/127 (56.7%) of candidates. Both of these correlated negatively with final placement on the ROL: USMLE step 1 scores, ρ = −0.253, P = 0.006; and USMLE step 2 scores, ρ = −0.348, P = 0.004.

DISCUSSION

Our data show positive correlations between a candidate’s EM grade, SLOR global assessment, CR, and overall standing on the ROL. This supports the findings of a 1998 study by Crane and Ferraro where EM program directors indicated that EM grades, the interview, clinical grades, other, and recommendations were felt to be most important in the selection of residents.⁷ As these correlations were weakly positive, however, this may suggest other parts of the application or the application process itself may significantly affect the ultimate position of a candidate on the ROL.

The ERAS application has many components. We evaluated the relationship between EM grade, SLOR global assessment, CR, and the ultimate ROL ranking. We did not evaluate other aspects of the SLOR (ie, section on qualifications for EM), nor did we evaluate educational background, extracurricular activities, volunteerism, or research and publications. Other research in EM has shown these parts of the application to be comparatively undervalued by EM program directors and by program directors in other specialties.^7,9 That said, Hay den showed the medical school attended and the presence of distinctive factors in the application predict future success as an EM resident.⁹

The residency application process extends beyond the ERAS application. It can include informal gatherings at venues like residency fairs, visiting student audition electives, pre-interview gatherings with residents, the formal interview, return visits to programs of interest, and post-interview interactions. Survey-based research in EM and other specialties suggests that program directors of various specialties value the interview highly^7,10–12 In addition to the interview, EM program directors value expressions of interest on the part of the candidate.⁷ As none of our metrics scored a high correlation with a candidate’s final rank on the ROL, our findings suggest that other factors, either from the ERAS application or the residency application process itself, may significantly affect a candidate’s ultimate position on the ROL. This finding dovetails with other studies that found the selection process does not reliably predict those residents who would later succeed in residency.^13,14 Furthermore, other specialties have found nonobjective factors, such as the faculty interview, predict future residency performance.¹⁵ Finally, a questionnaire of pediatric emergency medicine fellowship directors found the most important factor in granting an interview was recommendation from colleagues,¹⁶ again underscoring the importance of other immeasurable factors in a candidate’s application.

In addition to the positive correlation with the above, our data show a negative correlation between USMLE steps 1 and 2 scores and NRMP rank position. This is in conflict with the data of the 1998 Crane survey where EM program directors indicated that USMLE steps 1 and 2 were moderately important in the selection of applicants.⁷ While perception may be that USMLE scores are important to match in an EM program, and indeed they may be used to screen applicants in many programs, in fact, these factors likely play a less important role than previously thought. This is helpful information for medical students and those who advise medical students as other factors, in particular performance on EM rotations, may have greater influence on the competitiveness of a candidate for EM residency.

Future directions for research include attempting to tease out various factors that were unmeasured in our study, such as institution-specific EM grades, SLOR author identity and role (eg, program director vs clerkship director), medical school reputation, the addition of post-interview data, and the impact of the candidate’s interview performance on ultimate rank position. Furthermore, expanding this study to other institutions would broaden its external generalizability.

LIMITATIONS

Generalizability is the primary limitation of the study. Our convenience sample of candidates at a single institution may not be representative of other programs’ candidates. Similarly, it may not be possible to generalize a single residency’s approach to ranking with others. We studied aspects of the ERAS application and their relevance to ranking. Factors separate from the ERAS application, however, may affect a candidate’s ROL placement, such as audition elective completion, personal correspondences from mentors, and interactions at social events with the residents. These were not studied. While every effort was made to extract all pertinent study data from ERAS packets, it is possible that some information may have been added after the interview date and thus excluded from analysis in this study Furthermore, a single data abstractor who coauthored this paper extracted the data used for this study. While this is a weakness of our methodology, the data extracted was objective and therefore should have limited the bias in this study. Also, as this study used a combination of interval and ordinal data, we chose to use a Spearman’s rank correlation to evaluation this data. Nonetheless, incorporating these 2 different types of datasets poses challenges statistically. Two candidate packets were missing from the study and therefore not included in data analysis. Extracting data from MSPE letters proved to be challenging, as has been noted in other publications.¹⁷ Along these lines, we decided a 1 ranking represented either AOA in schools with that designation or first category (quartile, quintile, etc) in schools that did not use AOA. It may be difficult to draw conclusions based on these categories, as there is a wide variability in how schools rank students.

CONCLUSIONS

Our evaluation of applications for an EM residency program shows a positive correlation between a candidate’s EM grade, SLOR global assessment, CR, and ROL position. Conversely, USMLE scores had a weakly negative correlation with candidates’ final ranks. The fact that none of these parameters scored a strong correlation with ultimate rank position suggests that other factors in the ERAS application or the application process may influence a candidate’s ultimate position on the ROL. Medical students applying for an EM residency position should be advised that other non-USMLE factors, in particular their performance on EM rotations, may carry greater weight in determining where they end up on an ROL than previously thought.