Impact of USMLE Pass/Fail Step 1 Scoring on Current Medical Students

Abstract

OBJECTIVES

The aim of this study was to examine students’ perceptions of pass/fail scoring for Step 1, particularly when applying to competitive specialties. This study also investigated whether this transition increased anxiety among medical students and elicited student perspectives on the most critical components of residency applications.

INTRODUCTION

The United States Medical Licensing Exam (USMLE) Step 1 Board Exam transitioned from a traditional numeric score to pass/fail for exams taken on or after January 26, 2022. One justification for this change, according to a statement from USMLE, was to ease the transition between undergraduate and graduate medical education. Although many factors are considered when selecting candidates to interview for residency, Step 1 scores were historically used as an important metric. Few studies specifically evaluated medical student's attitudes toward the change in Step 1 grading.

METHODS

An online survey was distributed to Penn State College of Medicine medical students in the 2024 to 2027 classes. Statistical analysis was conducted on responses to research plans, attitudes regarding Step 1 pass/fail, match rates, and importance of application components.

RESULTS

There was a 21.3% response rate (127/596) to the survey with 30.7% pursuing a competitive specialty. Students applying into competitive specialties felt their chances of matching changed significantly compared to students applying to other specialties (either increased (25.6% vs 10.2%, P < .001), or decreased (28.2% vs10.3%, P < .001). Majority of students (73.2%) agreed with the decision to make Step 1 pass/fail, however, students reported increased anxiety surrounding Step 2 scores. There was a significant association between which aspects students believed were most important for matching (letters of recommendation and performance on Acting Internships) and those they believed program directors would weigh heaviest (P < .001).

CONCLUSION

Medical students report increased anxiety with Step 2 after Step 1 became pass/fail, although students overwhelmingly agreed with this change. To address this, medical schools should offer further guidance and resources to students, focusing on mentorship and methods to optimize residency applications for their chosen specialties.

Introduction

The United States Medical Licensing Exam (USMLE) Step 1 Board Exam transitioned from a traditional numeric score to pass/fail for exams taken on or after January 26, 2022. ¹ One justification for this change, according to a statement from USMLE, was to ease the transition between undergraduate and graduate medical education. ¹ Although many factors are considered when selecting candidates to interview for residency, Step 1 scores were historically used as an important metric. According to one study, up to 87% of medical students strategically prioritized their efforts to achieve a high Step 1 score to secure a competitive edge in the residency selection process. ² With the change of Step 1 to pass/fail, there is no specific data on how this will impact medical students applying into competitive residencies. Furthermore, this change will influence how program directors (PDs) select candidates to interview, such as shifting weight toward Step 2 scores, research experience, applicant familiarity, and letters of recommendation. ³ One survey found that 89.1% of otolaryngology residency PDs reported Step 2 scores were the new “most important” metric to assess an applicant, ⁴ while another survey sent to Plastic Surgery PDs found that “applicant familiarity” was most important. ⁵ Another survey of otolaryngology PDs suggested that other metrics including class rank would be emphasized when reviewing applicants. ⁶

Few studies specifically evaluated medical student's attitudes toward the change in Step 1 grading. A cross-sectional study of U.S. medical students investigated the attitudes and implications of this change on medical students, reporting 54% of medical students preferred numerical grading scales. ⁷ Another study found vastly differing opinions regarding the transition in score reporting, noting that students applying into competitive specialties and those who scored high on Step 1 were more likely to oppose the new policy. ⁸

Another justification for pass/fail scores set forth by USMLE was to reduce anxiety among medical learners. ¹ According to the American Medical Association (AMA), the “numeric score was detrimental to medical student well-being” because it stole the focus of students, hindering the development of communication and teamwork skills. ⁹ A single-institution study found that students who took Step 1 on a pass/fail basis experienced reduced anxiety prior to entering dedicated study periods. However, when surveyed again during their dedicated study period, this disparity in anxiety levels between the pass/fail and three-digit score cohorts disappeared, indicating no significant changes in anxiety levels regardless of the scoring method used for Step 1. ¹⁰ However, another study surveying students across three institutions found that Step 1 pass/fail scores failed to decrease student anxiety surrounding exam performance.^1,2 Similar conclusions were drawn from a survey of osteopathic medical students regarding both Step 1 and Comprehensive Osteopathic Medical Licensing Examination (COMLEX). ¹¹

The purpose of this study was to examine student attitudes toward the Step 1 pass/fail grading system and how these attitudes influence their approach to preparing residency applications in terms of their research plans, perceptions of matching into their desired specialty, and perceptions of the relative significance of different components within their applications. Additionally, we sought to compare these findings among students who were planning to apply to competitive residency programs to those planning to apply to other specialties.

Methods

A single institution cross-sectional survey study was reviewed by the Pennsylvania State College of Medicine International Review Board (IRB) and deemed to be exempt (Study ID# 23013). Written informed consent was not required to participate in the study as completing the survey implied participant consent. An online survey (via REDCap) was distributed to Penn State College of Medicine first through fourth year medical students via class email listservs. The survey was open for a six-week period between the months of August and September 2023, and students received two reminder emails to submit responses before the survey closed. The contents of the survey were included as Supplement 1.

Data was collected from current medical students from all four class years to elicit their perceptions on the change of the Step 1 grading scale and how students believed their futures would be impacted. All responses were deidentified during analysis to preserve anonymity. Inclusion criteria included current students enrolled in the MD or the MD, PhD programs at Penn State College of Medicine. Exclusion criteria consisted of those who did not fall into the above categories.

Competitive specialties were defined by the recorded unmatched percentages and median USMLE scores for U.S. seniors, as per the Elite Medical Prep website. ¹² These specialties were found to “consistently attract a significantly higher number of applicants than available slots.” ¹² These included Dermatology, Interventional Radiology, Neurosurgery, Ophthalmology, Orthopaedic Surgery, Otolaryngology, Plastic Surgery, Surgery, Urology, and Vascular Surgery. The remaining specialties were categorized into the “other” category.

Participants who completed the survey were invited to enter their email into a drawing for a $15 gift card. Funding for this study was provided by a gift fund from the Department of Otolaryngology – Head and Neck Surgery at the Penn State Milton S. Hershey Medical Center.

The reporting of the data adhered to the Consensus-Based Checklist for Reporting of Survey Studies (CROSS). ¹³ The completed checklist was included as Supplement 2.

Statistical Analysis

Statistical analyses were conducted on responses to research plans, attitudes regarding Step 1 pass/fail, match rates, and importance of application components. Frequencies and percentages of respondent preferences were reported. Chi-square test and Fisher's Exact test were used to evaluate the associations between variables. Significance was defined as P < .05. Statistical analyses were performed using SAS statistical software version 9.4 (SAS Institute Inc., Cary NC).

Results

The survey was sent to the 596 medical (MD) and medical scientist (MD/PhD) students enrolled at Penn State College of Medicine as of the summer of 2023. 148 students completed the demographics portion of the survey, and 127 students completed the entire survey, yielding a total response rate of 21.3% (127/596).

Participants’ demographics are summarized in Table 1. The majority of the 148 respondents who completed the demographics portion of the survey were fourth-year students (n = 39, 26.5%).

Table 1.

Respondent demographics.

Demographic	Category	Number of respondents	Percentage of respondents (%)
Age	18-24	50	34
	25-30	90	61.2
	30+	7	4.8
Gender Identity	Male	46	31.3
	Female	97	66
	Non-binary	2	1.4
	Prefer not to say	2	1.4
Ethnicity	Hispanic or Latino or Spanish origin	2	1.4
	Non-Hispanic or Latino or Spanish origin	138	93.9
	Ethnicity unknown	1	0.7
	Prefer not to answer	4	2.7
	Prefer to self-describe	2	1.4
Race	Asian	35	23.8
	Black or African American	6	4.1
	White or Caucasian	95	64.6
	Prefer not to answer	4	2.7
	Prefer to self-describe	7	4.8
Graduation Year a	2024	39	26.5
	2025	35	23.8
	2026	35	23.8
	2027	29	19.7
	2028	2	1.4
	2029	2	1.4
	2030	1	0.7
	2031	4	2.7

^a MD/PhD students were included in the survey, resulting in graduation years past 2027.

Of the 127 students who completed the entire survey, Internal Medicine was the most frequently chosen specialty students were interested in pursuing (n = 22, 17.3%) (Figure 1). About a third of students (n = 39, 30.7%) indicated intent to pursue a competitive specialty. The most frequently chosen competitive specialty students were interested in pursuing was Ophthalmology (n = 6, 15.4%). The percentage of students planning to complete a research year differed between residency specialty types (P = .004), with 10.3% (n = 4/39) of students vying for competitive residencies planning on taking a research year, compared to only 2.3% (n = 2/88) of students applying into one of the other specialties. In both categories, there were several students who were undecided at the time of the survey regarding whether they intended to pursue a research year, 28.2% (n = 11/39) of students in competitive specialties and 11.4% (n = 10/88) of those in other specialties (Table 2).

Figure 1.

Student-reported specialty of interest—students reported the specialty they are most interested in pursuing.

Table 2.

Comparison of students planning to take a research year based on intended specialty.

Planned research year	Competitive specialty (N = 39) [N (%)]	Other specialty (N = 88) [N (%)]	P-value
Yes	4 (10.3%)	2 (2.3%)	0.0037 a
No	24 (61.5%)	76 (86.4%)
Not sure	11 (28.2%)	10 (11.4%)

^a Fisher's exact test.

Students reported Step 2 scores (n = 106, 71.6%) to be the most important aspect of their residency applications, followed by letters of recommendation (n = 78, 52.7%) and performance in acting internships (n = 28, 18.9%). Acting Internship rotations were defined as rotations that fourth-year medical students complete in which they assume the responsibilities of an intern, which may also be known as a “Sub-internship” or “Advanced Clinical Elective.” When asked what they believed PDs would weigh the most, students still believed Step 2 scores (47.2%) were the most important aspect of their residency applications followed by letters of recommendations (20.5%). There was a significant association between which aspects of their applications a student believed were most important for matching (Step 2 scores, letters of recommendation, and performance on Acting Internship rotations) and those the same student believed program directors would weigh heaviest (P < .001). There was no significant difference in perceptions of important factors between students who were interested in pursuing a competitive specialty compared to another specialty (Figure 2).

Figure 2.

Student opinions on important application factors from the perspective of students and PDs—student-reported factors of residency applications they felt would be most important and those they thought PDs would find most important. Abbreviations: LOR, letters of recommendation; AI, acting internships; AOA, Alpha Omega Alpha Honor Society.

There was a significant difference in how students felt their chances of matching into their specialty of choice were between those applying to competitive specialties and those applying to other specialties (P < .001). 79.5% of students applying to other specialties felt there was no change in their chances of matching into their specialty of choice as a result of the pass/fail scoring compared to 46.2% of students applying to competitive specialties. Students applying to competitive specialties were more likely to feel they had an increased chance of matching (25.6% vs 10.2%) or a decreased chance of matching (28.2% vs 10.2%) compared to students applying to other specialties (Figure 3).

Figure 3.

Student perceptions on likelihood of matching into desired specialty following the transition to Step 1 pass/fail scoring—student responses separated by specialty of interest, comparing those interested in “competitive” versus “other” specialties (P = .0008).

Overall, 73.2% (n = 93) of all students, agreed with the decision to make Step 1 pass/fail. Of the students applying to competitive specialties, only 67% (n = 26) of students agreed with making Step 1 pass/fail while 76% (n = 67) of students applying to other specialties agreed (Figure 4). There was no statistically significant difference in opinion on pass/fail scores for Step 1 based on the type of specialty students were applying to (P = .44).

Figure 4.

Student level of agreement with USMLE Step 1 pass/fail scoring—student responses were separated by specialty of interest to compare those interested in “competitive” versus “other” specialties (P = .4417).

The vast majority of students (n = 93, 73.2%), regardless of specialty they were pursuing, reported increased anxiety surrounding Step 2 scores as a result of the change in Step 1 scoring. Although not significant, this was more prevalent in students applying to competitive specialties, with 82% (n = 32) reporting increased anxiety compared to 69% (n = 61) of students applying to other specialties reporting increased anxiety (Figure 5) (P = .26). There was also no significant relationship between a student's rating of the decision to make Step 1 pass/fail and level of anxiety toward Step 2 (P = .56). When queried about additional support that medical schools should provide to medical students, 40.9% of respondents (n = 52) cited improved opportunities for networking/advising throughout their undergraduate medical education.

Figure 5.

Student-reported anxiety surrounding USMLE Step 2 scores following transition to pass/fail Step 1 scores—student responses were separated by specialty of interest to compare those interested in “competitive” versus “other” specialties (P = .2634).

Discussion

The transition to pass/fail grading for preclinical courses in many medical schools and the recent change in the grading system of Step 1 will undoubtably impact the residency match process. The impetus for this change was to standardize and improve the transition from undergraduate to graduate medical education. While board exams were initially designed as a method to evaluate learners and their undergraduate medical programs, over time, Step 1 scores became increasingly utilized as a “screening tool” for residency programs. This reliance on Step 1 scores has been a result of multiple factors including a larger shift in pass/fail grading in undergraduate medical programs as well as increased number of applications per slot. Subsequently, students focused their efforts on achieving a “competitive” Step 1 score, at times at the expense of other extracurricular activities that contribute to developing well-rounded physicians. Furthermore, there are concerns with the overall well-being of students given the immense pressure and stress placed on an increasingly high-stakes exam. ¹⁴

Though there is no consensus within the medical community, the decision to make Step 1 pass/fail was a very intricate process involving medical learners at every level. ¹⁵ Following a multistakeholder Invitational Conference on USMLE Scoring (InCUS), representatives from the Association of American Medical Colleges (AAMC), AMA, the Educational Commission for Foreign Medical Graduates (ECFMG), the Federation of State Medical Boards (FSMB), and the National Board of Medical Examiners (NBME) put forth a general consensus supporting the pass/fail Step 1 grading system, stating “The current undergraduate medical education-graduate medical education transition system is flawed and not meeting the needs of various stakeholders; unilateral changes to USMLE alone will not ‘fix’ the overall system; and changes, both systemic and specific to USMLE, must be identified and implemented on a reasonable timeline.” ¹⁴ One important criticism to numeric Step 1 scores is that, although it is widely assumed to directly correlate with medical competency, it falls short of being a valid holistic assessment of a physician's ability to practice medicine. Applicants who perform medical humanities research at the undergraduate medical level were viewed as more holistic, compassionate, empathetic, and effective communicators. ¹⁶ Though the basic science concepts tested on Step 1 are assuredly necessary for a physician trainee, there is an art to medicine that is not as readily evaluated by standardized tests and can be further highlighted now that applicants are being reviewed by metrics other than Step 1 scores by PDs.

As expected, a survey distributed to department chairs, PDs, associate PDs, and department faculty in academic otolaryngology found that Step 2 scores would play a more significant role in an applicant's profile, followed by core clerkship grades, elective rotation evaluations, Alpha Omega Alpha status, and letters of recommendation. ⁴ This same study showed that academic otolaryngologists believed that the reputation of a medical school would weigh more heavily on the chances of a student matching due to the pass/fail grading of Step 1. ⁴ Another study emphasized the importance of applicant familiarity. ⁵ In summary, these studies collectively suggest that transitioning Step 1 to pass/fail has introduced new challenges for medical students applying into competitive specialties in the absence of the previously utilized cutoff metric.

We found that students reported uncertainty with what metrics are most important in residency applications now that Step 1 is pass/fail. Specifically, 71.6% of students felt Step 2 was the most important factor, but only 47.2% of students felt that PDs would prioritize Step 2 scores when selecting residency candidates. This can explain the increased anxiety students report around Step 2 scores; students are now faced with the decision of whether to take Step 2 and include this metric on residency applications or trust the process that promoted the switch to pass/fail scoring for Step 1. Some institutions are requiring students to take Step 2 before the application deadline, which may contribute to anxiety around these scores. One intention behind the change to Step 1 pass/fail scores is to allow for more holistic review of candidates and encourage medical students to be well rounded rather than focused on a few metrics. ¹⁷ Given the increasing awareness of the importance of developing well-rounded physicians, students require direction and mentorship to help prepare for their futures beyond test-taking skills. A previous study showed that otolaryngology applicants desired more mentorship opportunities. ¹⁸ Our results also show that students are seeking additional mentorship opportunities throughout their undergraduate medical education. Therefore, medical schools should provide adequate mentorship and networking opportunities within the home institution representing as many specialties as possible. This will allow for increased exposure to different specialties and help students discover, and ultimately match into, the specialty of their choice.

We also found that while most students supported the transition to pass/fail grading for Step 1, many reported experiencing heightened anxiety concerning Step 2 scores, regardless of pursued specialty. Of note, at the time of survey administration, the only class to have taken Step 1 was the graduating class of 2024 which encompassed 26.5% of survey respondents. Step 2 is now required by the institution to be completed prior to the residency application deadline, specifically August before the intended application cycle. Medical school is inherently challenging and stressful, and the importance of prioritizing mental health within medical education has recently become increasingly prominent. One study demonstrated that, in the wake of the COVID-19 pandemic, medical learners are now placing higher emphasis on their mental well-being than before. ¹⁹ The observed increase in anxiety surrounding Step 2 scores underscores the need for medical schools to proactively offer support for students, particularly in terms of examination preparation and managing stress. ¹⁰

Furthermore, with the increasing awareness of the importance of wellness in the medical community, prioritizing the health and well-being of medical students is paramount. A systematic review from 2011 discovered that pass/fail curricula enhanced medical student well-being without reducing objective academic performance. ²⁰ Given that the AMA believed numeric Step 1 scores were detrimental to medical student well-being, transitioning to pass/fail grading systems can reduce anxiety and improve the well-being of students. One single-institution study comparing anxiety levels between students taking a numerical score versus a pass/fail Step 1 found that students had reduced anxiety in the pass/fail cohort prior to entering their dedicated study period. ¹⁰ Another study reported that supporting and cultivating medical student well-being helps sustain this well-being into their careers, which improves patient care. ²¹ One paper reported that the only cause of mortality higher among physicians compared to the general public was suicide. ²² Therefore, measures must be taken to support physicians and trainees, enabling them to learn coping skills while simultaneously learning medicine. Transitioning Step 1 to pass/fail might be a key change that can help further contribute to improved wellness within the culture of medicine.

As a result of the pass/fail grading for Step 1, PDs are tasked with adapting to different and, perhaps, unconventional ways to evaluate residency applications, placing more emphasis on other components of a student's portfolio. How this approach will vary from specialty to specialty and from program to program remains yet to be seen and is outside the scope of this study. Nonetheless, new pressures placed on other metrics will undoubtably arise. Consequently, students are reporting uncertainty over which metrics those will be. This study sought to expand on our understanding of medical student perspectives on the pass/fail change. To our knowledge, is the first study that compares the attitudes of students applying to competitive residencies with those applying into other specialties.

Our study demonstrated that most respondents (73.2%), regardless of specialty aspiration, agreed with pass/fail Step 1 scores. This contrasts a previous study that showed a slight majority of medical students (53.5%) favored numerical scores. ⁷ However, that study did not specifically assess intended specialties of students favoring the numerical score. The consensus favoring pass/fail scores may stem from its potential to alleviate stress associated with an examination that previously determined potential medical specialties for residency applications. The decreased stress associated with pass/fail grading in the preclinical curricula has been illustrated in the literature, with one study reporting a reduction in preclinical stress levels when Step 1 were graded as pass/fail. ¹⁰

Interestingly, we found that students applying into competitive specialties felt their chances of matching were impacted both positively and negatively by pass/fail Step 1 scores. One hypothesis for this finding is an inverse relationship between matching and test scores: those who typically score highly on standardized tests may feel their chances of matching are decreased as they no longer have this metric to stand out from fellow applicants. In contrast, students who do not typically perform strongly on standardized tests may be encouraged by holistic application review and thus feel their chances of matching are increased.

Our results also show students believed the most important factors on residency applications would be Step 2 scores and performances on Acting Internships. There was also a significant association between what metrics they believed that PDs would emphasize when reviewing a residency applicant. One possible explanation is that students and PDs agree on the most important metrics of a residency application. Another possibility is that students ranked items highly based on a perceived projection of what PDs would rank highly.

One major limitation in this study—intrinsic to quantitative surveys—is sample bias as survey respondents were self-selecting. Additionally, this was a single, allopathic medical school, so results may not be representative of other undergraduate medical programs. Given that 26.5% of students took Step 1 at the time of survey administration, those students’ perceptions could be affected based on their score results, which is another potential limitation of this study. Another limitation of the study is sample size/power analysis was not performed for the data. Lastly, the data was not analyzed between class years because the curriculum for all the students at our institution is purely Pass/Fail with tiered grading being reintroduced to clerkships. Future studies should investigate how pass/fail Step 1 scores impacted the overall landscape of the residency application process. After a few application cycles, PDs can be queried about the new metrics they utilized in screening applicants. Students and faculty can also be surveyed on mentorship opportunities and mental health services. Additional studies can survey medical students from multiple institutions, encompassing osteopathic and allopathic programs, to determine if differences exist between these cohorts.

Conclusion

Although students overwhelmingly agree with the change in Step 1 scores to pass/fail, we found students now report increased anxiety surrounding Step 2. This contradicts the goals USMLE set by making this change. Further exploration can be performed to discover ways to avoid turning Step 2 into the new Step 1. To address this, medical schools should offer further guidance and resources to students, focusing on mentorship and methods to optimize residency applications for their chosen specialties.