Distribution of Honors Grades Across Fourth‐year Emergency Medicine Clerkships

Abstract

Background

Medical student grades during emergency medicine (EM) rotations are a key factor in resident selection. The variability in grading among EM clerkships is not well understood.

Objective

The objective was to describe the current grade distribution of fourth‐year EM clerkships.

Methods

This was an observational study at an EM residency program. We identified grade distributions by reviewing the standard letter of evaluation from individuals applying to our residency program for the 2016 match. Descriptive statistics of proportions, standard deviations (SDs), and p‐values were calculated.

Results

A total of 1,075 applications from 236 individual clerkships were reviewed. Thirty‐four programs did not give an honors grade during the previous year. Four of these programs distributed a highest grade of “high pass” and 30 gave only “pass” and/or “fail.” Of the remaining 202 programs, the percentage of grades that were given as honors ranged from 1% to 87% with a mean (±SD) of 25% (±17.2%). Of the 202 programs that granted honors grades, 63 (31.2%) sites gave between 1 and 14.9% honors grades, 69 (34.2%) gave 15% to 29.9% honors grades, 27 (13.4%) gave 30% to 44.9% honors grades, and 24 (11.9%) programs granted honors to greater than 45% of their students. Medical schools required an EM rotation at 82 (40.6%) sites. Among these programs, honors grades were given to 24% (±16.7%) of students with a range of 4% to 85% while programs that did not require clerkships gave a mean (±SD) of 26% (±17.5%) with a range of 1% to 87% and a p‐value of 0.54.

Conclusions

Honor grade distribution varies markedly across U.S. fourth‐year EM clerkship sites. Requiring EM clerkships does not affect honor percentages. A minority of sites only give pass/fail grades. Program directors should consider this marked variation in grades when reviewing EM residency applications.

Emergency medicine (EM) program directors (PDs) are presented with a myriad of data when evaluating potential residents. This includes preclinical and clinical medical school grades, EM clerkship grades, United States Medical License Examination (USMLE) scores, standardized letters of evaluation (SLOEs), and the Medical Student Performance Examination (MSPE). PDs also evaluate interview performance, past research, community service, and other distinguishing characteristics of applicants when creating their rank order list.1, 2, 3, 4 Several studies have described which of these factors have the greatest influence in the creation of a rank order list1, 5, 6, 7, 8, 9 and have shown that EM PDs tend to value fourth‐year clerkship grades above all other metrics.5, 6, 8 One study found that the SLOE from clerkship directors was the most important factor, but even this investigation indicated that the grade from the rotation under review was the most valuable question on the evaluation form.9

While EM PDs rely on grades from EM clerkships, the objective accuracy of medical school grades in general is unclear. Previous research has shown that a plethora of medical school grading systems (such as pass/fail vs. honors/pass/fail vs. honors/high pass/pass/fail) exist,10, 11, 12, 13, 14 which makes the comparison of grades between different programs difficult. Other investigators have raised concerns about grade inflation9, 10, 14, 15, 16, 17 and the variability in the number of students being granted high grades,10, 11, 14, 18 which further obfuscates the meaning of medical student grades.

A search of the literature reveals several studies investigating medical student grade variability among different specialties (such as surgery,18 core third‐year clerkships,10, 11, 12, 14 internal medicine clerkships,15, 19 and obstetrics/gynecology13), but there are no such data for EM. Grading systems utilized by EM clerkship courses remain unknown. Specifically, it is unclear how many programs utilize the pass/fail system versus other grading scales. Some medical schools require an EM rotation while others do not. A search of the current literature was unable to elucidate how compulsory courses impact EM grade distribution.

Given the significant emphasis on fourth‐year clerkship grades, a more thorough understanding of EM clerkship grade distribution may be useful for EM PDs in the resident selection process. The objective of this study was to describe the distribution of honors grades in fourth‐year EM clerkship programs.

Methods

Study Design

This was a retrospective observational study of grade distribution of fourth‐year EM clerkships and was conducted at an academic EM residency program. It was reviewed by the institutional review board at our institution and determined to be exempt from further review.

Study Setting and Population

Grade distribution data were extracted by the primary investigator from the SLOEs of fourth‐year medical students who applied to the Beth Israel Deaconess Medical Center Emergency Medicine residency program for the 2016 match. Clerkship location, grade earned, grade distribution data, whether or not the course was required, and the total number of students in the clerkship from the prior year were recorded. Clerkship sites were excluded if they were not based in the United States or pediatric EM clerkships or did not have any students enrolled in the previous year.

Data Analysis

Data were recorded into a Microsoft Excel spreadsheet. Ranges, means, and standard deviations (SDs) were calculated using Microsoft Excel, while p‐values were calculated using Quick Calcs (http://www.quickcalcs.com, accessed Nov 11, 2015). p‐values less than 0.05 were considered significant.

Results

A total of 1,075 applications were reviewed yielding 245 individual clerkship locations. Two were excluded, as they were not within the United States (one in Lebanon and one in Australia), five were excluded as they were dedicated pediatric EM rotations, and two were excluded because they did not have any students in the preceding year (and thus no historical grading data were available). Thirty‐four clerkships did not grant honors grades and were further excluded from the primary analysis (Figure 1) leaving 202 programs for analysis. The clerkships were distributed across 44 states as well as Washington, DC, and Puerto Rico. Grade distribution was reported by percent in each category listed on the SLOE: honors, high pass, pass, low pass, and fail. At five clerkship sites, the sum of the grade percentages was equal to the total number of students enrolled in the course rather than 100. These numbers were converted to a percentage.

Figure 1.

Breakdown of SLOE selection. SLOE = Standardized Letters of Evaluation.

Grade Distribution

Of the 202 clerkship sites that granted honors, the percentage of grades that were given as honors ranged from 1% to 87% with a mean (±SD) of 25.2% (±17.2%) honors grades. The distribution is shown in Figure 2.

Figure 2.

Number of clerkship sites granting honors grades.

Figure 2 demonstrates honors grade distribution divided into cohorts of 5%. For example, 25 different programs granted between 15.0 and 19.9% of their students honors. Honors grades were fairly evenly spread across the range of 5% honors through 35% honors with most programs (35) granting between 10.0 and 14.9% honors grades.

Eighty‐two clerkship sites (40.6%) stated that the EM clerkships course was required at their institution. Among these, honors grades were given out to 24.3% of students with a SD of 16.7 and a range of 4% to 85%. Clerkship sites that did not require the course had a mean (±SD) of 25.8% (±17.5%) and range of 1% to 87% honors. This difference was not significant (p = 0.54). Figure 3 illustrates the distribution of honors grades at programs where the clerkship was marked as “required” versus clerkships that were not identified as such.

Figure 3.

Distribution of honors grades by requirement of EM clerkship.

Grades Offered

Of the 236 clerkships reviewed, 34 did not give any honors grades. Twenty‐nine of these programs granted “pass” to all students and one program gave one “fail” grade with the remainder of their students earning a “pass.” The remaining four programs gave a selection of “high pass,” “pass,” “low pass,” and “fail” grades. Table 1 demonstrates the number of grade designations offered. Most clerkship sites gave three different grades (133) followed by four grades (44), one grade (29), two grades (17), and five grades (13).

Table 1.

Number of Grades by Clerkship Site

Number of Clerkships	Number of Grading Options
29	1
17	2
133	3
44	4
13	5

Discussion

Our study demonstrates that there is marked variability in the distribution of honors grades among U.S. fourth‐year EM clerkships. Many residency programs rely heavily on the presence of “honors” grades when evaluating prospective residents.1, 5, 6, 7, 8 The results of this study indicate that the exceedingly important decision of residency program ranking is being made by EM residency PDs relying on data that are highly variable and subjective. This does not seem to be isolated to only EM clerkships, as previous studies have illustrated similar variability in clerkship grades during medical school.10, 11, 14, 18

The SLOE is the result of an iterative process to clarify the evaluation of medical students. Prior to this tool, PDs voiced concern regarding the lack of uniform information, excessive superlatives, and ambiguity rampant in traditional narrative letters of recommendation. The Council of Emergency Medicine Residency Directors (CORD) developed the standardized letter of recommendation (SLOR, later amended to the SLOE), to create a standardized rubric by which to evaluate potential residents.9, 20 PDs have the benefit of extracting grade distribution data when reviewing this form. Given the variability in student grades, perhaps PDs would be better served by putting greater emphasis on the “global assessment” and “qualifications for EM” sections of the evaluation than on the grade itself. However, further research into the student distribution is needed in this area of the SLOE before this could be recommended. Efforts to better standardize grade distribution across all clerkships would certainly help better evaluate fourth‐year medical students applying in EM. Efforts by clerkship directors to standardize evaluations and grades are currently being evaluated and will hopefully help future students when applying to residency.21

We initially hypothesized that there would be fewer honors grades granted at programs where EM was a required rotation. As is evident in Figure 3 and the p‐value of 0.54, there is no significant difference between the percentage of students earning honors in programs that require an EM rotation and those that do not. Required programs were thought to grant fewer honors grades since fewer students would be invested in the course and specialty whereas programs where EM was not required were hypothesized to grant more honors grades as the students enrolled were interested in matching into EM. The reason for this lack of difference in honors distribution is unclear. It is possible that evaluators are indeed objective in grading and do not “reward” students for going into EM or it is possible that those students not pursing EM are equal or better performers and do just as well without the added motivation. Further investigation into this topic is needed.

Our study also highlights the lack of a standardized grading system utilized across EM clerkship rotations. Many programs granted all “pass” grades. Using the available data it is impossible to determine how many grades were possible at these programs; however, it seems likely that all these programs were pass/fail and did not fail any students. Other programs grant a “high pass” score and others include “honors.” This wide variance in grading systems makes analysis of student performance difficult. Comparing two students with “high pass,” for example, is difficult when one student has the chance to earn one of five grade designations and the other student one of three grade designations. Previous research has demonstrated that an increasing number of grading tiers is correlated with a greater percentage of higher grades.14 As several studies across many different medical specialties have raised concerns regarding grade inflation9, 10, 14, 15, 16, 17 it would seem prudent to better standardize the grading system to accurately evaluate prospective residents. A recent survey of clerkship directors in EM revealed that 90% favored development of a national tool for evaluating students during a shift but the preference for a standardized grading system is unknown.21 Unfortunately, grading systems are often established by the medical school and not EM clerkship directors.

Both medical schools and residency programs have recently begun to implement competency‐based grading systems that seek to identify discrete entrustable professional activities that can be easily evaluated.22, 23, 24, 25 If EM clerkships were to rely on competency‐based assessment, some of the subjectivity that contributes to grade inflation may be eliminated and perhaps less variability would be present in clerkship grades. Further research will need to be conducted looking into grade distribution at programs implementing this strategy.

Limitations

There were several limitations to this study. Data were collected using SLOEs submitted to a single academic institution. This could lead to bias given that certain students may not have been inclined to apply to this residency program. Additionally, because data were extracted from a single residency application database, the overall total number of clerkships was not represented in our sample. The total number of existing U.S. clerkships, however, is unknown so determining the percentage represented in our study is not possible. Clerkships from 44 states were included, making geographic bias unlikely.

Conclusions

Honors grade distribution varies markedly across U.S. fourth‐year EM clerkship sites and this may obscure the meaning of these grades. An EM clerkship requirement does not correlate with a significant difference in the percentage of students receiving an honors grade. A minority of sites appear to only grant pass/fail grades and there is a great variety in the number of possible grades across the country. Given concern for grade inflation, EM educators should work to standardize their approach to grade distribution to make clerkship sites more comparable.

AEM Education and Training 2017;1:81–86