Many Paths to the Summit: Survey of Step 1 Study Methods with Pass/Fail Scoring

Abstract

Introduction

After the United States Medical Licensing Examination (USMLE) shifted the Step 1 licensure exam to pass/fail, there have been limited studies to analyze changes in student study strategies.

Material and Methods

Surveys were distributed to third- and fourth-year medical students at the University of Michigan Medical School (UMMS). The response rate was 66%.

Results

The largest proportion (18.81%) of students chose 8 weeks of study time. 40.59% of students increased the length of their study period. To determine time allocated for dedicated study, 37.26% of respondents consulted near-peers who had already taken Step 1. Students also considered prior experiences with standardized tests (15.57%), personal reasons (14.62%), and conversations with school advisors (13.21%). 44.55% of students studied for 9–11 h a day, and 42.57% studied for 5–8 h a day. 52.69% of students scored between 70 and 80% on their final practice NBME test before their Step 1 exam. One hundred percent of respondents passed the exam. Exam non-extenders achieved higher end average practice test scores with shorter study periods. No differences in Step 1 study time or intensity were found when comparing students by intended specialty competitiveness.

Conclusion

Our results demonstrated patterns in study strategies for the new pass/fail Step 1 exam that may prove useful for curriculum design and schedule plan for future cohorts.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-024-02072-2.

Introduction

On January 26, 2022, the United States Medical Licensing Examination (USMLE) Step 1 exam grading shifted from a 3-digit numerical score to pass/fail. Since Step 1’s inception in 1992, the National Board of Medical Examiners has cautioned that the 3-digit scoring scale should not be used for purposes beyond determination of competence for licensure [1]. However, since the scale’s introduction, scores have been extrapolated from their original purpose of certification to guide the selection of residents, with program directors using scores to screen and stratify applicants [2]. The USMLE’s decision to transition to a pass/fail model in 2022 was also driven by a multitude of contributing factors. Negative performance impact on diverse student populations, detrimental alteration of the preclinical curriculum, and concerns about student wellness were noted as decision elements [1].

The pass/fail transition has been met with a range of responses from program directors to students. Girard et al. surveyed 852 medical students and found 39.0% in favor of the new scoring model, with students interested in highly competitive specialties as more likely to oppose pass/fail [3]. Given the hypothetical chance to take Step 1 pass/fail, a majority of students said they would dedicate less time to Step 1 (72.7%) and spend more time on Step 2 (70.5%) and research (59.6%) [3]. Time is a critical and limited asset for medical students. While part of the USMLE’s motivation in changing the scoring model was to reduce medical student stress and burnout, some studies suggested the change has contributed to an opposite effect, especially for students applying into competitive specialties [4]. Clinical proficiency, leadership experiences, and a heavy emphasis on research accomplishments have been suggested as the new metrics for residency applicant selection, especially in terms of highly selective residency programs [4]. Overall, students appear to be confused regarding the decision to transition to a pass/fail Step 1, expressing anxiety and uncertainty of how the exam would now play into residency applications [5].

Little data exists regarding how medical students now approach Step 1 exam preparation in light of the transition from 3-digit scoring to pass/fail. An analysis of the current needs and study preferences of students in the face of a new scoring system is critical to support the educational goals of future medical students. As students shift their focus from a pass/fail Step 1 exam to a scored Step 2 exam, our analysis aims to help students make informed decisions about how to both (1) approach their study strategy for Step 1 and (2) schedule their dedicated study period in relation to Step 2 preparation. This study described experiences of a cohort of medical students from a Midwestern medical school who took Step 1 pass/fail. We analyzed student study methods, duration of studying, resources used, assessment of preparation, and correlation to specialty choices. Further, we described student decision-making and sources of information in crafting study plans.

Materials and Methods

Institutional Context

For context, at the University of Michigan, students spend their first year in the preclinical “Scientific Trunk,” learning foundational medical knowledge predominantly via traditional classroom–based learning. The second year (“Clinical Trunk”) consists of the core clinical clerkships (psychiatry, neurology, family medicine, internal medicine, pediatrics, obstetrics and gynecology, and surgery). Following this period, students enter the post clerkship “Branches,” which includes self-directed scheduling of Step 1 and Step 2 exams, sub-internships, and electives.

Prior to Step 1 pass/fail scoring, students were allocated a total of 8 weeks dedicated study time immediately following the conclusion of core clerkship rotations (October and November of the third year). Students had to take and pass Step 1 before continuing into the branches. In 2022 with the transition to Step 1 pass/fail scoring, post clerkship students were given flexible scheduling of their study periods, with the option of self-scheduling two separate 4-week study periods, and the ability to reschedule their Step exam if needed.

Survey Creation and Study Respondents

We developed a survey that included questions regarding (1) study schedule, (2) resources utilized, (3) performance on practice tests, and (4) trusted sources of advice (see Supplemental Digital Appendix). The survey questions were developed by Student Council members at the University of Michigan in collaboration with faculty and staff within the Office of Medical Student Education. The survey was constructed in Qualtrics with final review by four third-year medical students and two medical school administrators. In February 2023, third- and fourth-year medical students at the University of Michigan Medical School were recruited via email and class GroupMe (an online messaging platform that students have created to communicate with each other in an informal setting). The survey was administered between 01/26/2023 and 05/08/2023. A total of 113 medical students completed the survey. Eleven students who graduated in 2023 were excluded from the dataset as we were only surveying students graduating from the class of 2024 and beyond since they expected to take Step 1 pass/fail and therefore planned accordingly. One response that failed to indicate graduating class year was also excluded from the final dataset. The final dataset contained a total of 101 respondents, all graduating in 2024 or beyond. One hundred fifty-three students had taken Step 1 P/F during the time of survey administration between 01/26/2023 and 05/08/2023 who graduated in the class of 2024 or beyond for a total response rate of 66%. Students included in final data analysis began taking the pass/fail Step 1 exam as early as 10/2022. Participation was voluntary and respondents did not receive payment or other incentives for their time. Respondents were informed their responses would remain anonymous and personal identifying information was not collected. The study was deemed exempt by the University of Michigan Institutional Review Board (HUM00230385). Statistical analysis was conducted in Excel and R Studio using the ggplot2 statistical package. We used an alpha level of 0.05 for all statistical tests.

Exam Extension vs. Non-extension

Two sub-groups examined in this study were labeled the exam non-extenders (NE, respondents who took their exam as scheduled) and the exam extenders (EE, respondents who rescheduled their exam to a later date). For this sub-group analysis, we focused on the impact of exam extension on outcomes; therefore, a total of five respondents who decreased their study period or had unique responses were removed from the analysis. A total of four blank and outlier responses were excluded from resource utilization analysis. All analyses were completed in Excel.

Comparison Between Specialties

Analysis of variance (ANOVA) was used to determine an association between specialty competitiveness and length of dedicated study. Five blank response values relating to the number of total practice questions completed were present, 2 blank response values relating to UWorld completion percentage, and 2 blank response values relating to NBME completion percentage. These were imputed using multiple imputation, with five imputation iterations performed. Ordinal logistic regression was then used as the most appropriate test to estimate a relationship between specialty competitiveness and other aspects of test preparation, using datasets with imputed values.

Results

Determining Dedicated Study Time

In this study, respondents studied anywhere between 2 weeks or less and 10 weeks or more, with the largest proportion (18.8%) of students choosing 8 weeks of study time (Fig. 1a). Notably, none of the respondents started studying during their second-year clinical clerkship rotations. The majority of students took their exam as initially scheduled; however, 40.6% of students increased the length of their study period beyond their initial scheduled date (Fig. 1c). To determine the amount of time to allocate for dedicated study, 37.3% respondents talked to students who had already taken Step 1. Students also took into account personal prior experiences with standardized tests (15.6%), life events/personal reasons (14.6%), and conversations with medical school advisors (13.2%). None of the students surveyed considered prior experiences with preclinical materials when determining dedicated study time (Table 1).

Fig. 1.

Overview of dedicated study time. a Dedicated study time in weeks. b Dedicated study time in hours per day. c Changes in Step 1 study period duration

Table 1.

Strategies for determining amount of dedicated study time

Response	Count of responses	Percentage of total (%) Rounded to nearest tenth
Conversations with students who had taken Step 1	79	37.3
Prior experiences with standardized tests (e.g., MCAT, shelf exams)	33	15.6
Life events/personal reasons	31	14.6
Conversation with medical school advisors (e.g., house counselor, learning specialist)	28	13.2
Other	17	8.0
Specialty competitiveness	13	6.1
Conversation with clinical advisors (e.g., residents, attendings)	11	5.2
Prior experiences with scientific trunk assessments*	0	0

Summary of themes from “other” free-text responses
Response	Count of responses	Percentage of total (%) Rounded to nearest hundredth
Performance on practice exams	6	54.6
Online forums	1	9.1
Random	1	9.1
Information from peers	3	27.3

*Scientific trunk is the preclinical stage in the UMMS curriculum

Dedicated Study Time and Resources

During dedicated study time, 44.6% of students studied for 9–11 h a day and 42.6% studied 5–8 h a day. Of the remaining respondents, 10.9% students studied for 12 h or more and 2.0% studied for 4 h or less (Fig. 1b). On average, students completed a total of 2982 (SD 1253) practice questions (including test bank questions from UWorld, Kaplan, AMBOSS, and National Board of Medical Examiners (NBME) practice tests). The total number of practice questions completed did not correlate with score range prior to sitting for STEP 1 (r = 0.01), based on 89% of respondent results due to data availability.

Out of the most commonly used Step 1 resources traditionally used by students, the most completed resource (expressed as % of the resource completed based on student subjective report) was Pathoma, followed closely by Sketchy Micro, NBME practice tests, and UWorld (Fig. 2, Table 2). These average percentages may be an overestimate of the actual dataset since many students left areas blank, which we analyzed as “N/A” rather than assuming it meant 0%.

Fig. 2.

Amount of resources completed (%)

Table 2.

Amount of resources completed (%)

Resource	Average % completed	Standard deviation
Pathoma	75.61	34.9
Sketchy Micro	69.61	35.5
NBME* Practice Tests	67.51	29.7
UWorld	63.65	25.9
Sketchy Pharm	56.28	37.9
First Aid	50.66	38.2
Online Videos	49.83	35.00
Anki	37.32	28.2
Other	33.75	32.5
Boards and Beyond	13.00	13.2

Online video preferences
Response	Count of responses	Percentage of total (%) Rounded to nearest tenth
Dirty Medicine	24	64.9
Online MedEd	5	13.5
Biostatistics Youtube Videos	3	8.1
Osmosis	1	2.7
HyGuru	1	2.7
Medical Moose	1	2.7
Divine Intervention	1	2.7
Bootcamp	1	2.7

“Other” free-text responses
Response	Count of responses	Percentage of total (%) Rounded to nearest tenth
Amboss	2	40.0
Personal Anki cards from practice questions	1	20.0
Duke Pathoma Anki Deck	1	20.0
Pepper Micro and Pharm Anki Deck	1	20.0

*National Board Medical Examiners

Determining Test Readiness

Since transitioning to pass/fail scoring on January 26, 2022, NBME self-assessments no longer release a 3-digit score and instead show the percent correct as well as an estimated likelihood (in percentile) of passing the exam within the next week. In our study, 52.7% of students had a percent correct score between 70 and 80% on their most recent NBMEs before sitting for their actual Step 1 exam (Fig. 3). One hundred percent of students who completed the survey passed the exam. Students who responded to the survey took Step 1 in both 2022 and 2023. The UMMS Step 1 first time passing rate for both years was 97%, whereas the national first time passing rates were 93% and 92% respectively [6].

Fig. 3.

Range of percent correct for final NBME practice test prior to Step 1 exam

Specialty and Degree Choice

17.8% of students who participated in the survey stated specialty interest in internal medicine, followed by OB/GYN (11.9%) (Fig. 4a). The majority of students (76.2%) sampled were obtaining an M.D. only as opposed to pursuing a dual degree or higher education (Fig. 4b).

Fig. 4.

a Distribution of respondent specialty interest. b Distribution of respondent degree path

Exam Extenders vs. Non-extenders

Fifty-five respondents (54.5%) took their exam as scheduled, 41 (40.6%) extended their study period, two (2.0%) shortened their study period, and three (3.0%) reported other unique responses (Fig. 1c). 40.6% of respondents rescheduled their exams to allow for additional study time. Given this significant proportion, respondents were then broken up into two primary groups, NE and EE. Comparing these sub-groups provided an opportunity to assess how students planned and benchmarked their progress before sitting for their Step 1 exam. In terms of the average weeks spent studying, a majority (61.0%) of EE spent 8 to 10 weeks or more, with 8 weeks being the most common length of time (31.7%). In contrast, a majority (61.8%) of NE had a study period of 4–6 weeks. Hours studied per week were similar for the NE and EE groups. For NE, 41.8% studied for 5–8 h while 49.1% studied for 9–11 h. Within the EE sub-group, 41.5% of students studied for 5–8 h and 41.5% studied for 9–11 h. The survey also examined how students planned their study approach. Students in both groups sought study planning information from various sources. Still, ultimately both groups predominantly consulted medical school peers who had previously taken Step 1 (NE 36 students, EE 37 students) and fellow cohort peers (NE 23 students, EE 31 students). Across both groups, only 16 students consulted medical school advisors, and five consulted clinical advisors. The survey provided an option for additional responses. Several students listed Reddit, a social news and discussion platform, and prior Medical College Admissions Test (MCAT) performance as study period planning influences.

Popular test preparation resources include UWorld, a question bank widely utilized by medical students, NBME practice exams, and the Free120, a USMLE developed practice test that utilizes test questions retired from prior Step 1 exams. In terms of the number of practice questions completed in total between UWorld practice sets and exams, NBME practice exams, and the Free 120, the two groups completed a similar average number of total exam questions before taking the STEP 1 exam: NEs completed an average of 2988 questions (SD 1177) and EEs completed an average of 3065 (SD 1322) questions. While the standard deviations are large, the measure still provides a rough estimate of how many questions students completed prior to testing. Regarding resource utilization, NEs used an average of 5.7 unique resources, and EEs used an average of 6.7. First Aid, a popular preparation resource, was utilized by both sub-groups, with 60.0% of NE and 63.4% of EE utilizing the resource and completion rates of 55.2% and 46.4%, respectively. Regarding individuals’ percent correct scores before sitting for the exam, we compared the two groups’ reported score ranges. EEs scored almost equally in the 61–70% (46.0% of respondents) and 71–80% (46.0% of respondents) average practice test score bands. NEs scored better on average prior to testing, with 32.7% of students in the 61–70% band, 55.8% in the 71–80% band, and 9.6% in the 81–90% band. Given that every survey respondent passed Step 1, the NE and EE groups reached similar endpoints of 3000 practice questions and an average score of around 70% before sitting for the Step 1 exam. However, the exam non-extenders achieved these endpoints, on average, in less time and received the same passing score.

Comparison Between Specialties

In this study, 101 respondents stated plans to apply into a total of 20 different specialties. Specialty competitiveness was determined by the percentage of MD applicants to the field that went unmatched during the Match 2022 cycle [7–9]. Specialties were then categorized into three categories: less competitive (MD Match rate > 90%), moderately competitive (MD match rate of 70.01–90%), and highly competitive (MD Match rate ≤ 70%) [7–9]. Thirty-five respondents (34.7%) applied into less competitive fields, 33 respondents (32.7%) into moderately competitive fields, and 30 respondents (29.7%) into highly competitive fields. These tests broadly showed that specialty competitiveness did not significantly affect study time or assessment of readiness for Step 1 among students (Table 3).

Table 3.

Differences in preparation intensity between specialty competitiveness

Study variable	Less competitive specialties	Moderately competitive specialties	Highly competitive specialties	p-value Rounded to nearest tenth
Total questions completed (mean (SD))	2873 (1259)	2846 (1110)	3178 (1161)	0.3*
NBME % completed (mean (SD))	68.5 (29.9)	70.6 (29.2)	65.2 (28.2)	0.8
UWorld % completed (mean (SD))	64.6 (25.6)	67.0 (25.4)	60.5 (24.7)	0.6
# Dedicated study weeks (mean (SD))	6.20 (2.01)	6.81 (1.89)	6.10 (2.19)	0.3*

*p-value from ANOVA. All other p-values were determined with ordinal logistic regression, using multiple imputation with five sets of imputations

Discussion

Survey data from the analysis of the University of Michigan’s medical students who took the exam in between 10/2022 and 05/08/2023 show many different approaches to the new pass/fail Step 1 exam. A similar survey in 2014 and 2015 of student study strategies at the University of Michigan Medical School was conducted by Burk-Rafel et al. when Step 1 was taken for a score. They found that 77% of students surveyed began studying for Step 1 during the preclinical curriculum. For their dedicated study period, students only studied for 5 weeks but averaged 11 h per day. Students also used numerous third-party resources and completed an average of 3597 practice multiple-choice questions [10]. A major decision for students to make when commencing their Step 1 study period is the duration of their “dedicated” study period, with the norm being 6–8 weeks under 3-digit numerical scoring. The aforementioned results indicate that a majority of students took 6 or more weeks to study for the exam (63.4%). In our cross-sectional survey, a large variation was seen in the amount of time taken across all survey respondents (range 2 weeks or less to 10 weeks or more). 11.9% of students were at the extremes of this range, with 2.0% of students taking 2 weeks or less and 9.9% of students taking 10 weeks or more.

In terms of exam scheduling, 37.3% of respondents noted having conversations with students who had previously taken Step 1 to determine an optimal studying timeline. The utilization rate of peer conversations with prior Step 1 testers was more than double that of utilization of other planning resources (medical school counselors, clinical advisors, prior experiences, etc.). This finding highlights the importance of peer-to-peer engagement for medical school success.

Examining the outcomes and study patterns of students who took their Step 1 exam as scheduled versus a delayed exam also provided interesting insights into Step 1 study planning. Although all students passed the examination, this sub-group comparison of EE vs. NE raises the question of whether achieving a certain threshold of practice questions and practice exam score may be useful information for future medical student cohorts as they prepare for USMLE Step 1. While the standard deviations are large for practice question analysis, the data still provides a rough estimate of how many questions students completed prior to testing. These benchmarks are not guidelines to follow, but they might provide direction for study planning and content completion. Ultimately, most students are asking the question “What do I need to pass Step 1?” and this work helps address part of that question. Exam non-extenders achieved the same end goal as extenders of a pass on Step 1 and a significantly higher end average practice test score with a shorter study period. Non-extenders studied similarly in terms of hours, but were able to accomplish approximately the same amount of content and question review in less weeks.

It is interesting that there were no differences in study time or intensity based on choice of specialty during Step 1 study. Prior to the Step 1 being made pass/fail, achieving a certain score on Step 1 was correlated with matching into a particular specialty [3]. Specifically, average scores for more competitive specialties were generally higher than those that were less competitive, likely necessitating a more intense, or longer study period [3]. Transitioning to a pass/fail Step 1 likely eliminated the specialty-based variation in pressure to do well on the exam, perhaps explaining the lack of any significant difference in study time, number of questions completed, or percentage correct on NBME and UWorld resources seen in our analysis. Our analysis attempted to neutralize outliers by categorizing all those who took 10 weeks or more to complete their Step 1 study period, and all those who took 2 weeks or less to complete their dedicated study, as singular categories respectively.

Given our study design, we were unable to assess the specific effects of potential outliers. However, there were only 14 students who fell in either category, constituting only a small portion of our total respondents. Furthermore, when categorizing specialties based on competitiveness, we assumed that every specialty included gave equal importance to the Step 1 exam in their resident selection process, using competitiveness statistics from a Match class that took a scored Step 1 exam. Finally, although the survey was administered within a single institution, students are preparing for a national standardized examination using outside resources not specific to the curriculum at this institution. The strategies and approaches used by students to gauge their readiness to take the exam may be of interest to other students, especially given the lower national passing rates since the transition to P/F scoring. These findings may also be of interest to medical school educational leaders and administrators who are considering the optimal timing of Step 1 at their institutions as part of curriculum transformation or to optimize student success.

Future Research

Those students preferring to take less time than the 6–8-week norm likely felt that they required less time to study, given that they would need to achieve a “pass” rather than a specific numerical score to match into a particular specialty. Students taking longer may have done so due to “underestimating” the exam and requiring more time to achieve a passing score or score of their satisfaction. Further qualitative analysis is required to elucidate these complexities. It is important to note that many people pushed back the exam (40.6%), while only 2.0% of survey respondents “preponed” their exam. This supports the aforementioned suggestion that students may be entering their dedicated study period underestimating exam difficulty or breadth of subject material, and postponing their exam to allow for extra study time after initial diagnostic practice testing.

Conclusions

Overall, our research aims to address questions about medical students’ current study approach given the shift in Step 1 scoring to a pass/fail model in January 2022. Given our sub-group analysis, we wonder if practice question totals and practice exam scores could be better indicators for Step 1 readiness compared to study time alone. Our results suggest that there were many paths to achieving the same result of a passing score. Further research based on comparative models to previous strategies when the examination was taken for a score may be useful for additional guidance.

Supplementary Information

Below is the link to the electronic supplementary material.

Data Availability

The data that support the findings of this study are not openly available due to reasons of sensitivity and are available from the first author upon reasonable request.

Declarations

Competing Interests

The authors declare no competing interests.