The Impact of Changing Step 1 to Pass/Fail Reporting on Anxiety, Learning Approaches, and Curiosity

Abstract

Purpose

Given the significance of the US Medical Licensing Exam (USMLE) Step 1 score moving from a 3-digit value to pass/fail, the authors investigated the impact of the change on students’ anxiety, approach to learning, and curiosity.

Method

Two cohorts of pre-clerkship medical students at three medical schools completed a composite of four instruments: the State-Trait Anxiety Inventory, the revised two-factor Study Process Questionnaire, the Interest/Deprivation Type Epistemic Curiosity Scale, and the Short Grit Scale prior to taking the last 3-digit scored Step 1 in 2021 or taking the first pass/fail scored Step 1 in 2022. Responses of 3-digit and pass/fail exam takers were compared (Mann–Whitney U) and multiple regression path analysis was performed to determine the factors that significantly impacted learning strategies.

Results

There was no difference between 3-digit (n = 86) and pass/fail exam takers (n = 154) in anxiety (STA-I scores, 50 vs. 49, p = 0.85), shallow learning strategies (22 vs. 23, p = 0.84), or interest curiosity scores (median scores 15 vs. 15, p = 0.07). However, pass/fail exam takers had lower deprivation curiosity scores (median 12 vs. 11, p = 0.03) and showed a decline in deep learning strategies (30 vs. 27, p = 0.0012). Path analysis indicated the decline in deep learning strategies was due to the change in exam scoring (β = − 2.0428, p < 0.05).

Conclusions

Counter to the stated hypothesis and intentions, the initial impact of the change to pass/fail grading for USMLE Step 1 failed to reduce learner anxiety, and reduced curiosity and deep learning strategies.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01878-w.

Introduction

In February 2020, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME) announced the US Medical Licensing Exam (USMLE) Step 1 would change score reporting from a 3-digit value to pass/fail in 2022. This decision was based on findings from the Invitational Conference on USMLE Scoring (InCUS) [1] and was a response to the overemphasis on Step 1 scores in residency screening and selection, the racial demographic disparity in USMLE performance, inflation of the “parallel curriculum,” and negative impact on students’ well-being [2–4] and learning [5]. With approximately half the student population reportedly experiencing burn-out [4] and a third of students experiencing signs of depression or anxiety [6], the shift to pass/fail appeared to have merit. While the long-term and ensuing consequences of this change will become apparent in the future, the immediate impact on learner well-being and learning approaches can be explored now.

Learner well-being is comprised of many elements, both positive and negative, that ultimately define the ability of an individual to cope (or thrive) in their environment [7]. One major negative contributor to learner well-being is anxiety, and although this can originate from a number of sources in the pre-clerkship years, Step 1 is reportedly a major anxiety-provoking experience [2]. High levels of anxiety diminish goal-directed attention and concentration, memory, and perceptual-motor function [8] — all of which are critical for effective learning, preparation for high-stakes evaluations, and clinical practice [8]. In the context of undergraduate medical education, the impact of anxiety and heightened stress on learning strategies may manifest as superficial or shallow learning [9] or potentially decreased curiosity. When engaging in shallow learning, many students employ strategies like mnemonics, highlighting, and note-taking to learn vocabulary and essential content. By contrast, when students commit to deep learning, they employ self-regulation strategies such as content synthesis, elaborative interrogation, and metacognition [10]. The transition to shallow learning strategies is driven by fear of failure and is a coping strategy with the short-term goal of passing assessments [11]. However, shallow learning hampers development of a cognitive schema of sustained connections essential for lifelong learning [12].

Conversely, a deep learning approach is associated with genuine curiosity and better academic outcomes [13, 14]. Curiosity is at the heart of medical education, but unfortunately, so is learner anxiety [6, 9]. With anxiety’s detrimental impact on learner curiosity, the Step 1 transition to pass/fail might afford the opportunity to re-engage curiosity about medical knowledge. Epistemic curiosity, defined as “the desire to know” [15], impacts student behavior when setting goals, and helps determine the level of effort learners are willing to expend in attaining those goals. Reducing anxiety’s deleterious effect on curiosity (and well-being) may promote a deeper, more curious learner and have a positive impact on the profession [16].

The antithesis of anxiety is “grit,” defined as “the perseverance to achieve long-term goals” [17, 18]. High levels of grit are associated with lower anxiety and burn-out [18, 19] (Fig. 1) as well as higher levels of well-being [19], self-regulated learning, and academic performance [20, 21]. While higher levels of grit correlate with increased performance [22], it also fosters mastery-oriented learning. Although an indicator of inherent resilience [19, 23], accounting for grit is important when determining the impact of anxiety.

Fig. 1.

Summary of the study design and rationale. The change to a pass/fail would decrease student anxiety, increasing curiosity, and result in deeper learning strategies. Grit was measured as a covariant given its influence on several of these factors. STA-I, State-Trait Anxiety Inventory; R-SPQ-2F, revised two-factor Study Process Questionnaire

The positive impact of the transition to pass/fail in pre-clerkship grading (e.g., learner well-being, reduced anxiety) indicated that the change to pass/fail in STEP1 would also have positive implications [24]. This, and the intentions of INCUS, prompted us to consider the following hypothesis supported by a framework adapted from LePine et al. [25] (Fig. 1): Students who will take USMLE Step 1 reported as pass/fail will be less anxious, thereby more curious and engage in deeper learning processes. To test this, we assessed medical students’ anxiety, learning approaches, and curiosity, prior to taking a 3-digit scored Step 1 and prior to taking a pass/fail Step 1.

Methods

Participants and Learning Environments

We invited participation from two medical school classes: 2023 (the last to take the 3-digit scored Step 1) and 2024 (the first to take a pass/fail graded Step 1). Invitees studied at one of three schools (sample of convenience): Zucker School of Medicine (ZSOM; 100 students/class), Virginia Tech Carilion School of Medicine (VTCSOM; 42 students/class), and Tulane University School of Medicine (TUSOM; 190 students/class). The study was approved with waived written consent by the Virginia Tech IRB (Protocol #21-241) and VT approval was accepted by the IRBs at ZSOM and TUSOM.

Both VTCSOM and ZSOM deliver content in a problem-based curricular format while TUSOM uses a systems-based active lecture approach. At all programs, both cohorts of students had a minimum of 6-weeks dedicated study time prior to taking their Step 1 exam. Similar resources for exam preparation across programs included vouchers for Comprehensive Basic Science Self Assessments, access to learning support specialists, faculty, and peer tutors. Students at VTCSOM and ZSOM also had university-provided access to UWorld Question banks; TUSOM provided voluntary, faculty led preparation sessions.

Assessment of Learning Approaches, Anxiety, Curiosity, and Grit

Participants anonymously completed the following instruments: (1) the State-Trait Anxiety Inventory (STA-I) [26] to assess the level of anxiety, (2) the revised two-factor Study Process Questionnaire (R-SPQ-2F) [27] to determine approach and depth of learning, (3) the Interest(I)/Deprivation(D) Type Epistemic Curiosity Scale [13], (4) the Short Grit Scale [17]. Grit was measured as a covariable for inclusion in the multiple regression path analysis as it impacts anxiety and burn-out [18, 19] well-being [19], self-regulated learning, and academic performance [20, 21]. Epistemic curiosity was divided into I and D components. I-type curiosity is driven by the personal satisfaction of gaining new knowledge and is associated with mastery-oriented (deep) learning [13, 14]. Alternatively, D-type curiosity manifests as a judicious and immediate approach to address the discomfort associated with a gap in knowledge [13, 28]. Both types of curiosity are associated with deep learning approaches [29].

The class of 2023 was invited to participate in the study during the spring of 2021, prior to taking the last 3-digit scored Step 1. The class of 2024 was invited to participate in the study during the spring of 2022, prior to taking the first pass/fail graded Step 1. Invitations were sent via email to all members of each class with a reminder sent a week later. Survey instructions were intentionally general so as not to bias participant response, stating only “Thank you for choosing to participate in this research study to gain a better understanding of the impact of Pass/Fail grading on the STEP 1 exam. This study will extend the current body of knowledge of information that influenced the change to this exam.” Survey responses were stored on a secure drive at VTCSOM.

Analysis

The final scores for each of the four survey responses were calculated as previously described [13, 17, 26, 27] for each participant. The Short Grit and STA-I surveys generated single values for grit and anxiety, respectively, whereas the R-SPQ-2F survey generated measures of both deep and shallow learning strategies and the Interest/Deprivation Type Epistemic Curiosity Scale generated values for interest curiosity (I-type) and deprivation curiosity (D-type).

Prior to pooling data from the participating schools, any effect of the school the participants attended on each collected survey response was assessed using a Kruskal–Wallis test (critical value < 0.05, corrected for multiple comparisons). Collated scores from each class, for each instrument, were compared using Mann–Whitney U tests.

Pooled data from both classes was assessed to determine whether Grit scores correlated with perceived anxiety, curiosity, or learning strategies (Spearman’s ranked correlation coefficient). For outcome factors (deep and shallow learning strategies) that demonstrated inter-class differences, a path (multiple regression) analysis [30] was performed to determine the impact (magnitude and direction) of explanatory variables (grit, anxiety, I- and D-curiosity). Because an association between grit and anxiety has been documented [31], an a priori interaction between these two variables was included in the regression model.

Results

The aggregate survey response rate was 37% (n = 86 3-digit scored; n = 154 pass/fail scored exam). Individual program survey responses were Tulane 3-digit scored = 24, Tulane pass/fail scored = 61, VTCSOM 3-digit scored = 21, VTCSOM pass/fail scored = 26, Zucker 3-digit scored = 49, and Zucker pass/fail scored = 67. There was no effect (Kruskal–Wallis) of school on grit (p = 0.630), anxiety (p = 0.898), I-type curiosity (p = 0.789), D-type curiosity (p = 0.975), deep learning strategies (p = 0.589), or shallow learning strategies (p = 0.077). Median instrument scores (range and IQR) for each instrument are shown for both classes in Fig. 2. There was no statistical difference in grit (p = 0.22), anxiety (p = 0.85), or I-type curiosity (p = 0.07) between 3-digit and pass/fail exam takers. However, compared to students taking a 3-digit scored Step 1 exam, students taking a pass/fail scored Step 1 showed significantly lower levels of deep learning (p = 0.0012) and D-type curiosity (p = 0.03). Correlation analysis between grit and the responses to the other instruments confirmed previous findings, summarized in Table 1, showing grit was inversely correlated with anxiety and shallow learning (p < 0.0001) [21, 31] and directly correlated with both types of curiosity and deep learning approaches [29, 32] (p < 0.0001). Anxiety was inversely correlated with use of deep learning strategies (Spearman r = − 0.257, p < 0.001) but was not correlated with the adoption of shallow learning strategies (Spearman r = 0.079, p = 0.119).

Fig. 2.

Grit, anxiety, learning approaches, and curiosity prior to taking a 3-digit or pass/fail Step 1. Survey responses to the Short Grit, STA-I, R-SPQ-2F, and I-D Curiosity scales by medical students about to take numeric scored — or pass/fail Step 1 exam. Median values are shown and labelled with interquartile range (boxes) and ranges (whiskers). *Significant difference (one-tailed, Mann–Whitney U)

Table 1.

Cross-correlates for grit, anxiety, learning approaches, and curiosity

Cross correlates	r value	p value
GRIT vs. STA-I	−0.243	0.000104
GRIT vs. R-SPQ-2F DEEP	0.348	0.00000003
GRIT vs. R-SPQ-2F Shallow	−0.262	0.00002892
GRIT vs. I-Curiosity	0.3274	0.00000019
GRIT vs. D-Curiosity	0.214	0.00057275

The decline in deep learning approaches observed with pass/fail exam takers compared to 3-digit exam takers was analysed further using path analysis (Fig. 3). The model developed used exam format, anxiety, grit, and both forms of curiosity as explanatory variables and accounted for the known interaction of grit and anxiety, adapted from Richards et al. [14]. The adjusted r² of the model was 0.5. Exam format had a relatively large (β = − 2.0428) negative impact on deep learning that was statistically significant (p < 0.05). Both I- and D-type curiosity had direct effects on deep learning approaches (β = 1.0815 and 0.4746 respectively) that were significant (p < 0.05). There was no significant impact of either grit (p = 0.333) or anxiety (p = 0.915). When the same model was used to describe shallow learning approaches, the adjusted r² was 0.145 (see Supplemental Fig. 1).

Fig. 3.

Path diagram of the relationships between deep learning approaches and exam format (3-digit score or pass/fall), anxiety (STA-I), grit, deprivation curiosity, and interest curiosity of second year medical students prior to taking either a 3-digit scored or pass/fail Step 1 exam (n = 240). Significant paths and their b coefficients are in bold; nonsignificant paths are dashed. The model has an adjusted r.² of 0.5 and includes a constant (shown in the circle)

Discussion

In changing Step 1 score reporting to pass/fail, the USMLE responded to recommendations in an effort to curtail the “Step 1 climate” [1, 33]. While many of the speculated impacts will take years to evaluate, this study demonstrates that the initial impact on anxiety, learning strategies, and curiosity with this cohort is contrary to our original hypothesis. Instead, our data better reflect the pre-change perspective of medical students that anxiety would not be mitigated [33–36]. More importantly, these data suggest the shift to pass/fail was associated with a decreased deprivation-driven (D-type) curiosity and negatively impacted deep learning approaches in our learner population. These outcomes are also counter to the reduced anxiety and increased intrinsic learning approaches observed after intramural implementation of pass/fail grading in undergraduate medical education [24, 37].

Continued Anxiety

Why would the changes to a national standardized exam fail to ease anxiety in this set of learners? While the degree of anxiety in the two classes was the same, we did not ask about the source of anxiety, and we speculate that it is likely different for each cohort [38]. Indirectly related to the change to pass/fail is an immediate period of uncertainty regarding what metric(s) will replace the numeric score in residency directors’ selection criteria. Although learners taking a pass/fail exam were relieved of the pressure of attaining a high numeric Step 1 score, they may exhibit a “fear of the unknown” [39] surrounding the speculation of how the change in scoring may impact the residency selection process [34, 40–43]. The predicted change in emphasis by residency directors to clerkship grades and/or the still numerically scored Step 2CK [44] may be the next distraction. If so, the intended mitigation of anxiety by changing to pass/fail may only emerge as the priorities of residency directors becomes clearer in future years calling for the need for continued monitoring.

Change in Learning Approaches

The intricate connections between learning approaches and curiosity in medical education are largely unexplored, but all are inversely influenced by learner anxiety [31, 45, 46]. While levels of anxiety could still be reduced as the consequences of the change to pass/fail on residency selection unfold, we postulate that our findings regarding learning approaches may be more persistent and potentially impactful. Our path analysis suggests that although I- and D-curiosity still promoted deep learning strategies as previously shown [29, 32], the negative impact of the change to pass/fail in our cohorts was larger and led to a net decrease in this approach. Our data are consistent with previous studies with grit being directly correlated with deep learning and inversely correlated with anxiety [31, 47] and shallow learning [9]. However, while the path analysis shows a large regression coefficient for grit and deep learning approaches (Fig. 3), it was not statistically significant, suggesting other factors need to be considered. We suggest that this can also, in part, be attributed to the learners’ re-prioritizing their efforts toward successful residency placement.

Deemphasizing Step 1 could contribute to diminished focus on medical knowledge [24] as the change to pass/fail affords students the choice to focus on external achievements (e.g., research projects, community service) to strengthen their residency applications. Consequently, the diminished deep learning approaches observed in this study could be attributed to distraction and re-prioritization, rather than anxiety (consistent with the path analysis model). Although we have treated D-type curiosity as an inherent characteristic, curiosity has previously been shown to be impacted by pre-clinical education [46]. The small but significant decline in D-type curiosity might be also attributed to re-prioritization as students are more tolerant of not knowing basic science content. It has already been suggested that current constructs of medical education are not optimized to support curiosity [46] and this change may contribute to this phenomenon. As curiosity has been touted as essential to the practice of medicine [48], this finding is of concern and continued monitoring will be critical.

Unlike our suggestion that the mitigation of anxiety may come with more confidence in knowing the expectations of residency directors, the same may not be true for deep learning of medical knowledge. If the criteria for a successful residency application no longer include evidence of sustained medical knowledge, the motivation to adopt deep learning strategies may continue to decline. Although only addressing the immediate impact of the transition to pass/fail, our data could therefore signal the onset of a trend, rather than an aberration at the point of transition.

There was no difference in the degree of shallow learning approaches between 3-digit and pass/fall exam takers in the studied populations, and our path analysis model for deep learning strategies did not hold true for shallow learning (explaining only 14.5% of variance, Supplemental Fig. 1). The lack of impact on shallow learning seems somewhat at odds with the decline in deep learning, but this may be due to shallow learning strategies having already been adopted prior to the transition of exam grading format (i.e., by 3-digit and pass/fail exam takers alike). The phenomenon of the parallel curriculum and use of external, third-party resources specific to Step 1 exam success (rather than long-term, deep learning) had been established prior to changes in Step 1 [49, 50]. The multiple-choice format encourages memorization of medical knowledge rather than the curious exploration of relevant topics [29, 33, 51], and the transition to pass/fail grading may perpetuate this use of shallow learning approaches.

Strengths and Limitations

The current study presented results consistent with previous findings [38] and begin to describe the impact of the Step 1 scoring change, but limitations should be considered.

The study cohort is a relatively small sample of undergraduate medical students and there is a noted difference in response rates between classes (for which we have been unable to formulate a viable explanation). Responses to all four surveys were independent of which school participants attended (despite the schools’ varying curricula and matriculants), suggesting that these findings may represent a more generalizable change. Our data only represents the immediate response and our continued data collection and inclusion of more schools (Phase 2) will be important to see if this trend persists. Phase 2 of this study has begun, includes more participating schools, and will allow us to glean insights into how the change in Step 1 scoring influenced student perceptions of the residency application process. Like the change in Step 1 grading, there has been much speculation about how this will change program directors’ approach to the residency applicant selection [52–54], consequently the impact on the learner should continue to be investigated.

The path analysis model explained 50% of the variance in deep learning approaches (adjusted r² = 0.5). However, our model includes complex behaviors and characteristics. For example, although we measured anxiety, we did not ask participants to attribute their anxiety to any particular source (i.e., taking a 3-digit graded exam, or applying for residency without having taken one). It is very possible that other explanatory variables missing from our model, such as participant demographics and the changing influence of the COVID-19 pandemic (e.g., student well-being and changes in teaching modalities), may contribute to the unexplained variance. Given the complexity of intrinsic and extrinsic variables involved, our model is robust and sufficient to describe a moderate effect but continued and more focused data collection will improve our understanding of the impact from the change to pass/fail.

Conclusions

Although the impetus behind the policy change was multifaceted, reducing student anxiety was a major goal. Our early data suggests this attempt did not result in reduced anxiety; rather, it may have encouraged reliance on less effective learning strategies and reduced curiosity. Taking these early data into consideration, the medical education community should remain vigilant in monitoring the impact on learner well-being in considering some of the potential consequences of this programmatic change. In the long term, as more certainty about the residency application criteria is established and there is less “unknown” to fear, it will be important to track student learning approaches. The commonalities between a psychologically healthy physician and medical student are many [55], and negative impacts to early medical education will likely be perpetuated in the professional, clinical environment. Finally, the potential, long-term de-prioritizing of medical knowledge by undergraduate learners should be carefully monitored and the need to re-imagine both the delivery and assessment of basic sciences should be considered.

Supplementary Information

Below is the link to the electronic supplementary material.

Funding

This project was funded in part by the Virginia Tech Center for Teaching and Learning.

Declarations

Ethical Approval

The study was approved on March 9, 2021, by the Virginia Tech Internal Review Board (IRB), Protocol #21-241. It was deemed exempt under 45 CFR 46.104(d) category 2(i). VT approval was accepted by the IRBs at ZSOM and TUSOM.

Conflict of Interest

The authors declare no competing interests.