The Optimal Timing of Step 1 in Medical Education Following the Transition to Pass/Fail: A Unique Perspective from Post-clerkship Step 1 Schools

Abstract

The National Board of Medical Examiners’ decision to change Step 1 of the United States Medical Licensing Examination (USMLE) from a three-digit score to Pass/Fail (P/F) represents a disruptive change for students, faculty, and leaders in the academic community. In the context of this change, some schools may re-consider the optimal timing of Step 1 as they strive to align their assessment practices with sound educational principles. Currently, over 20 schools administer USMLE Step 1 after the core clerkships. In this commentary, we review the educational rationale for a post-clerkship Step 1, highlighting how adult learning theories support this placement. We discuss some short-term challenges post-clerkship Step 1 schools may encounter due to the proposed timing of the change in scoring, which creates three unique scenarios for learners that can introduce inequity in the system and provoke anxiety. We review outcomes of potentially heightened importance when Step 1 is P/F, including lower clinical subject exam scores in some clerkships, lower failure rates on Step 1 and stable Step 2 Clinical Knowledge scores with implications for the residency match. We outline the future potential for performance-based time-variable Step 1 study periods that are facilitated by post-clerkship placement of the exam. Finally, we discuss opportunities to achieve the goal of enhancing student well-being, which was a major rationale for eliminating the three-digit score.

Background

In February of 2020, the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB) announced that Step 1 of the USA Medical Licensing Examination (USMLE) would change from a three-digit score to Pass/Fail (P/F), noting the new policy would be effective “no earlier than January 1, 2022”[1]. The decision was met with a mixture of joy, fear, and consternation by students and faculty in the academic community, and several commentaries have been written about this disruptive change [2–6].

We anticipate that many schools will seek to adjust their curricula and assessment strategies as emphasis on achieving a high Step 1 score is eliminated. Currently, over 20 schools administer USMLE Step 1 after the core clerkships [7]. Some schools may re-consider the optimal timing of Step 1 as they strive to align their assessment practices with sound educational principles.

As representatives of schools who administer Step 1 after the core clerkships, we have a unique perspective to add to the national dialogue concerning the change to P/F scoring. In this paper, we review the educational rationale for a post-clerkship Step 1, highlight short-term challenges created by the timing of the change to pass/fail scoring, describe outcomes of potentially heightened importance in a P/F model, outline the potential for performance-based time-variable Step 1 study periods, and discuss opportunities to enhance student well-being.

Rationale for a Post-clerkship Step 1

Our schools have described a clear pedagogical rationale for a post-clerkship Step 1. The key point is that contemporary curricular reform and associated assessment strategies should be guided by principles of adult learning theories [8–10]. Particularly relevant are those that emphasize (1) integrating basic and clinical science, (2) reinforcing intrinsic motivation by highlighting clinical applications, and (3) ensuring retention and transfer of concepts into practice by emphasizing the application of basic science in clinical medicine [11–13]. Integrated curricula capitalize on the principles of distributed learning, interleaving, transfer, and elaboration [14]. Empiric evidence suggests that students engaged in integrated curricula demonstrate higher order reasoning and achieve more enduring learning [15, 16].

Many schools have successfully implemented reforms to their pre-clerkship curricula that integrate basic and clinical science, emphasizing pedagogies such as case- or problem-based learning, and early clinical immersion [17]. Integrating foundational science into clinical training has proved more challenging. While the USMLE has increased basic science content on Step 2, we believe the placement of Step 1 prior to clerkships accentuates the basic/clinical science divide. This timing inadvertently signals to students that foundational science only needs to be memorized for the Step 1 exam, and once the exam is successfully passed, the material is no longer relevant, so it can be quickly forgotten [18]. Daniel et al. described several strategies to enhance basic science education during the core clerkships, one of the most significant of which is the post-clerkship placement of Step 1 [19]. Delaying the timing of Step 1 taps into students’ motivation for learning and encourages students to both apply and deepen their knowledge of relevant basic science while engaging in direct patient care. With the shift to P/F scoring, students can be encouraged to engage in basic science learning with an aim of achieving mastery for the benefit of patients, rather than focusing on learning only what will be on the test [6, 20].

The Timing of the Change to P/F Scoring

For schools that already administer Step 1 after the clerkships, the timing of the proposed scoring change to P/F (January 1, 2022) creates three unique scenarios that will impact learners on the cusp of the change differently (see Fig. 1). Schools considering moving Step 1 to after clerkships in the near term, as part of a previously planned change or as an urgent response to COVID disruptions, will need to carefully consider which of these 3 scenarios might apply to them.

Fig. 1.

Three scenarios for post-clerkship Step 1 schools concerning the timing of the P/F scoring change

Scenario 1: Students at schools with very short (~ 1-year) pre-clerkship curricula are scheduled to take the exam in November/December, just before Step 1 changes to P/F. These students will have a three-digit score similar to other graduates applying for residency in their cohort.

Scenario 2: Students at schools with longer pre-clerkship curricula are scheduled to take Step 1 from January–May after the change to P/F. They will apply to residency with a cohort of learners from schools who administer an earlier Step 1, the majority of whom will still have three-digit scores.

Scenario 3: Students at schools that offer flexible testing options must decide if they want to take the exam before or after the P/F scoring change and must decide if a three-digit or P/F score is more desirable based on their unique circumstances and preferences.

During the scoring transition, program directors are likely to treat students in Scenario 1 similar to other learners in the match, as they are familiar with how to process learners with three-digit scores. What is less certain (and purely speculative) is how program directors will manage applicants in Scenarios 2 and 3, who either by necessity or choice, will have a P/F score in an application cycle comprised largely of learners with a legacy three-digit-score. Schools and learners alike are worried that those applications will receive lower priority.

Uncertainty as to whether the scoring change will occur on or after January 1, 2022, is adding additional complexity to these scenarios. This is because a small difference in the timing of implementation can “flip” a school from Scenario 2 to Scenario 1. If not provided with a choice of timing, some students may try to “game” the system, choosing a path that allows them to obtain either a P/F or three-digit score. For example, students planning to obtain additional degrees (PhD, Masters) or needing to take time off for personal or academic reasons may make strategic decisions about when to take their leaves of absence to obtain desired scoring [21]. If given a choice, students will need to develop a solid understanding of the advantages and disadvantages of each scenario, and be fully aware that any advice provided by faculty or institutions will be purely speculative.

Students who anticipate achieving a high score (i.e., at the mean or above), may be better served by preserving a three-digit score, particularly if they are planning to pursue a competitive specialty. A high score will provide a “competitive edge,” whereas an average score will still ensure their application is considered in a cycle where the majority of students will have three-digit scores. Students who anticipate attaining a lower exam score may be better served by taking Step 1 as a P/F assessment. A passing score may be more favorably received than a very low three-digit score. If a lower performing student fails, they will lose the advantage of being able to demonstrate a higher score on retake—they will only be able to convert their score to a pass. Of course, this is also true for students who obtain a three-digit score, who cannot retake the exam before the scoring transition.

Some students may want to delay the exam and take it in the P/F model purely to reduce the stress and time associated with preparing to earn a high score. Other students may need to delay the exam for other reasons (e.g., due to a health concern or academic difficulty). In all cases, students will benefit from individualized support and guidance in order to manage the anxiety that these choices may evoke. Learners with P/F scores who graduate with a cohort of students with three-digit scores will clearly need proactive advising concerning other important metrics in the residency match to ensure they remain competitive. Schools with Step 1 deadlines or policies concerning advancement may need to adapt their policies to respond to diverse student needs, as unique circumstances create widely disparate views about scoring. Decisions must be managed carefully, respecting student autonomy, while providing the best possible guidance.

Outcomes to Consider for Post-clerkship Step 1 Schools

In prior studies, Jurich et al. showed that moving the Step 1 exam increases mean three-digit scores and reduces failure rates on Step 1 [9]. Step 2 Clinical Knowledge (CK) scores and failure rates remain stable [10]. Clinical subject exam scores decrease slightly in some clerkships, particularly early in the clerkship year.[22]

As Step 1 changes to P/F scoring, institutions that currently administer the exam before clerkships may consider shifting to a post-clerkship Step 1. One of the most compelling reasons to consider moving Step 1 is the potential reduction in failure rates, particularly given the likely increased emphasis on a first-time pass with the P/F scoring change. In a study of four schools that recently moved Step 1 to after the clerkships, failure rates decreased from 2.87% (n = 48) in the 3-year pre-change to 0.39% (n = 6) in the 3-year post-change (P < 0.001) [9]. While this sample may not be representative of all schools, even a small reduction in failures has the potential to have a profound impact on students’ residency match options. For example, in 2018, more than 21,000 students took Step 1 and the first-time pass rate was 96% [23]. If all schools were to shift Step 1 post-clerkship and a similar reduction in failure rates was observed, this would equate to ~ 520 fewer students per year who would fail the exam. This effect is likely to benefit students most who historically perform the lowest on standardized tests. As Step 1 scores have been previously shown to systematically disadvantage those who are underrepresented in medicine (e.g., women and minority applicants), minimizing the number of Step 1 failures may help advance diversity [24, 25].

When Step 1 changes to P/F scoring, other metrics such as clinical subject exams (CSEs), clerkship grades, and performance on Step 2 CK will likely assume greater weight in the undergraduate to graduate medical education (UME-GME) transition [3]. Jurich et al. found slight (~ 1–2 point) degradations on Medicine, Neurology, Pediatrics, and Surgery CSEs when Step 1 was moved after the clerkships [22]. The effect was most marked in the earliest clerkships and gradually disappeared with subsequent examinations. The study also found that a higher number of students scored below the fifth percentile nationally on some CSEs [22]. Few schools directly report CSE scores to residency programs; however, CSE scores significantly contribute to clerkship grades at most schools, and methods of determining grades likely disproportionately reflect exam scores [26, 27]. Many schools utilize national cut points for assessing performance on CSEs and use this as a basis for assigning Honors/High Pass/Pass/Fail. If clerkship grades receive more emphasis in the match when Step 1 changes to P/F, lower CSE scores will be a cause for concern and schools will need to proactively mitigate this effect. Pock et al. described strategies to address lower CSE scores in post-clerkship Step 1 schools, including ensuring adequate practice with NBME-style questions prior to taking CSEs, adjusting minimum passing standards, lowering the contribution of CSE scores to clerkship grades, or eliminating tiered grading in clerkships altogether [28].

Of note, the outcomes of a post-clerkship Step 1 (i.e., fewer Step 1 failures, lower CSE performance, and unchanged Step 2 CK scores) were studied when Step 1 was reported as a three-digit score and the results may be different when Step 1 is graded P/F. Learner motivation, changes in Step 1 content (e.g., the removal of management questions), alterations to school curricula, or shortening of Step 1 study periods all have the potential to influence results.

Flexibility and Variable Length Study Periods at Post-clerkship Step 1 Schools

When Step 1 is administered after the clerkships, learners may have enhanced flexibility concerning the duration of the Step 1 study period, as the study period is followed by electives, rather than core clerkships. This flexibility can allow learners to take study periods ranging from 0 to 12 weeks, allowing study time to be individualized to meet the needs of a diverse student body. In contrast, when Step 1 is administered before the clerkships, it is difficult to have flexibility in timing since students typically enter the clerkships at a set time.

A Step 1 study period of variable duration will arguably be more important to learners when the exam changes to P/F, as increasing emphasis will be placed by program directors (PDs) on first-time pass success, as well as other metrics in the residency match. Students who are weaker in the medical knowledge competency, who have longstanding learning disabilities, who struggle on standardized multiple-choice tests, or who have unique situations (e.g., a major life event) may appreciate and need a longer Step 1 study period than their peers to ensure they pass. Students who are stronger in the medical knowledge competency may wish to take shorter study periods to allow for more time to engage in other activities, such as sub-internships, away electives, research, scholarship, leadership, or other impactful activities that can make them stand out in the residency application process.

For students taking Step 1 after clerkships, guidance concerning the duration of the study period could be based on performance metrics, rather than being standardized for all. Several performance metrics have been previously shown to have predictive value concerning Step 1. The comprehensive basic science self-assessment (CBSSA) has a strong correlation with Step 1 performance when taken immediately before the dedicated study period (r = 0.7211, p ≤ 0.001) [29]. The CBSE has been shown to have a similar correlation (r = 0.73, p ≤ 0.001) [30]. One institution found performance on fourteen pre-clerkship NBME basic science subject exams accounted for 57% of the variance in Step 1 scores [31]. Another school developed a regression model based on NBME clinical subject exams that accurately predicted post-clerkship Step 1 scores for 85% of learners within 15 points [unpublished data]. Of course, all predictors have margins of error that must be accounted for to ensure passing success. Furthermore, new predictive models may need to be generated once Step 1 changes to P/F scoring, as historical models were developed when learners had 6–8 weeks to study for the exam and their study efforts were differently motivated.

The Association of Program Directors of Internal Medicine has already determined that they will now require Step 2 CK results for their residency applications. Other specialties are likely to follow suit as PDs seek alternative national standardized assessments to help them separate low and high performers. Depending on a school’s curricular structure, students may struggle to find time to complete Step 2 CK before interview season. Post-clerkship Step 1 schools typically shorten their pre-clerkship curricula as part of larger curricular reforms, creating longer post-clerkship phases and increased time for electives. When this is combined with a flexible Step 1 study period, post-clerkship Step 1 students will have marked advantages as it concerns scheduling Step 2 CK, additional time to study for the exam and added time to engage in other activities viewed favorably by PDs.

Achieving the Aim of Enhanced Well-Being as Step 1 Scoring Changes to P/F

One of the major rationales for transitioning Step 1 to a P/F exam was to enhance student wellbeing [1], but it is unlikely that the transition to P/F will achieve this goal without other shifts in the system for the UME to GME transition. Indeed, there is evidence that the competitive focus will simply shift to other metrics, negating potential positive effects. This risk is already recognized by students and residents: In the 2019 public response to the Invitational Conference on USMLE Scoring (InCUS), only 44% of medical students and 39% of residents supported a change to three-digit scoring (including the possibility of P/F) compared with 75% of deans and 67% of clerkship directors [32]. We believe post-clerkship Step 1 schools may be uniquely positioned to optimize certain drivers of well-being outlined by Shanafelt et al.[33] First, curricular reforms associated with a post-clerkship Step 1 typically emphasize early clinical immersion, optimizing meaning and purpose in one’s work through experiential learning. Second, students could have more control and flexibility concerning the duration and timing of their Step 1 and 2 study periods, and in turn more time for choices in the elective phase of the curriculum. Students could even be allowed to take Step 2 CK before Step 1. This option may be desirable for students considering a competitive specialty who want a three-digit score earlier in their elective phase to guide future planning. Third, the workload and studying demands associated with Step 1 preparation may be diminished with Step 1 after clerkships, because of learners’ greater experience (i.e., more total time on task) learning medicine after the core clerkships. Studying for a post-clerkship Step 1 also better integrates clinical information into the dedicated basic sciences review because students’ prior clinical experience means they have a more developed mental scaffold for Step 1 review.

Conclusions

The majority of post-clerkship Step 1 schools plan to continue delayed administration of the exam, because they believe that the pedagogical reasons for the change continue to justify this timing. Management of the scoring transition will be challenging, as some learners, by necessity or choice, will have a P/F score in an application cycle where the majority of students will have a three-digit score, and program directors may be ill-prepared to navigate this in the short term. Proactive, individualized counseling and mentoring will be paramount to support student success. Ultimately, the scoring change heralds a unique opportunity for medical schools to critically re-evaluate the topic of high stake summative assessment, its role in the UME-GME transition, and its impact on learner well-being. New opportunities may emerge for flexible study periods and basic and clinical science integration as Step 1 transitions. The ramifications of converting the exam to P/F will likely manifest for years to come and will deserve ongoing monitoring and rigorous study.

Declarations

Disclaimer

The views expressed are those of the authors and do not reflect the official policy or position of their universities, the Department of Defense, the US Air Force, or the US Government.

Conflicts of Interest

The University of California San Francisco School of Medicine, University of Michigan Medical School and Virginia Commonwealth University School of Medicine all have Accelerating Change in Medical Education grants from the American Medical Association.