The Change to Pass/Fail Scoring for Step 1 in the Context of COVID-19: Implications for the Transition to Residency Process

Abstract

In this Invited Commentary, the author considers the February 2020 announcement that scoring on the United States Medical Licensing Examination (USMLE) Step 1 will change to pass/fail no sooner than January 2022 and its effects on the transition to residency process in the context of both the recommendations of the Invitational Conference on USMLE Scoring (InCUS) held in March 2019 and the disruptions caused by the COVID-19 pandemic in the spring of 2020. The author suggests that the medical education community must embrace any positive changes that come about as a result of the pandemic while continuing to systematically review the strengths and areas for improvement in the current transition to residency process.

In its recommendations, InCUS provided a thoughtful set of action priorities and an effective process to work together, which can inform and guide the work ahead. The COVID-19 pandemic is dominating the educational and clinical environments and is now the biggest disruptor in all aspects of life, not just medical education. It is the responsibility of leaders in medical education to have a vision for and then implement an improved continuum of education that maintains the core values of the field and fits the health care delivery needs of today and the future.

On February 12, 2020, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME) announced that students taking the United States Medical Licensing Examination (USMLE) Step 1 will receive a pass/fail grade instead of a numeric 3-digit score.¹ This change will take effect no earlier than January 1, 2022. The announcement was met with both relief and consternation in the academic medicine community, as medical students, educators, and residency program directors weighed the implications of significantly altering the scoring on a key assessment of students’ basic science knowledge.^2–4

At the time of the announcement, many in academic medicine felt that this change was one of the most abrupt and significant disruptions in medical education they had ever experienced. As the chief medical education officer for the Association of American Medical Colleges (AAMC), I had been deeply involved with the Invitational Conference on USMLE Scoring (InCUS), which preceded the decision to change Step 1 scoring, and I anticipated that my leadership role in the InCUS follow-up work would be among the most complex and challenging of my career.

That changed with the onset of the COVID-19 pandemic in March 2020. Now, we are all challenged in ways we never imagined. Already the pandemic is necessitating changes in medical education and the transition to residency, with much more disruption sure to come.

I suggest that we recognize the disruptive force of the COVID-19 pandemic as an opportunity for change and that we continue to develop a framework for improving the transition to residency process to move the educational continuum to an integrated solution, rather than a series of well-intentioned, unilateral “fixes” that create unintended and perhaps negative consequences. While InCUS and the USMLE Step 1 scoring announcement feel long ago compared with the pace of work related to the COVID-19 pandemic, the conference remains an important milestone.

Held in March 2019, InCUS was cosponsored by the AAMC, American Medical Association, NBME, FSMB, and Educational Commission for Foreign Medical Graduates, with a goal to “collaboratively review the USMLE program’s practice of numeric score reporting within the context of its primary use of licensure, and to discuss any secondary uses and the broader regulatory and educational environments in which USMLE exists.”⁵ The conference was unique in that representatives of all stakeholder groups came together; put aside their advocacy for a single group; and dug in to listen, learn, and look beyond their own perspectives to detail the problem and begin thinking about solutions.

The conclusions of InCUS must inform our consideration of “what’s next.”⁵ While substantial conversation was devoted to USMLE scoring, the key conclusions from this first-of-its-kind conference went far beyond that, just as the work ahead must go far beyond the change in Step 1 scoring.

First, InCUS concluded that the current undergraduate medical education (UME) to graduate medical education (GME) transition process is flawed and not meeting the needs of its stakeholders. Over time, various stakeholder groups have tried to optimize this system for their own purposes, which at a minimum has left some groups, including applicants, with an undue burden and at worst has negatively influenced workforce diversity.

Second, unilateral changes to the USMLE alone will not “fix” the UME to GME transition, absent changes to other parts of the system. Changes, both systemic and specific to the USMLE, must be identified, explored, and implemented. When FSMB and NBME made the announcement to change Step 1 scoring to pass/fail, they created a unilateral change to the USMLE, at least for the short term. That change resulted in student anxiety about Step 1 shifting to bigger questions: What will program directors use instead of Step 1 scores? How will students distinguish themselves if they are interested in a competitive specialty, if they are from a newer or less well-known medical school, or if they are a graduate of a non-U.S. medical school?

Before the announcement, program directors widely acknowledged that they used Step 1 scores as a key differentiator among applicants because they lacked other objective data by which to assess students.⁶ Therefore, the change to Step 1 scoring has understandably also increased program director anxiety. How will they wade through the vast numbers of applications they receive for each residency slot? Should Step 2 Clinical Knowledge scores become the new filter? Do specialties need to develop specialty-specific assessments? How would that even work?

The chatter since the Step 1 scoring change announcement has provided further support for the InCUS conclusion that thoughtful systemic changes need to be identified, explored, and implemented.^2–4 Changing Step 1 scoring alone did not solve the underlying problem, which is the complexity in the GME application process for all stakeholders. I believe that these stakeholders—medical students, educators, and residency program directors—have an obligation to explore solutions within their respective domains. For example:

• Medical schools need to commit to more clearly and transparently communicating to residency programs their students’ relative competencies and professionalism. This will afford program directors a clearer understanding of applicants’ knowledge and skills.

• Program directors must commit to devising a way to more clearly communicate to students and their advisors what distinguishes their program from others, what they are looking for in applicants, and who they are likely to accept. This will allow students to better select programs that are the right fit for them and will, ideally with good advising, decrease the number of applications that students submit.

• Medical schools and program directors need to have an honest, serious conversation about how to develop either a better screening tool or potentially another set of assessments to give program directors more tools to appropriately screen applicants.

• The medical education community needs to consider changes to the application process; the interview process; and perhaps the Match process that will reduce the burden, anxiety, and cost for applicants and the burden for program directors.

• Finally, we all must recognize that international medical graduates are an important part of our resident and physician workforces and that our actions must consider their unique challenges.

These changes can and should be considered and acted on in the shorter term, and the AAMC is committed to helping lead this work. Equally important, this is an opportunity to consider not only the transactional part of the UME to GME transition but also the true continuum of medical education and the health care ecosystem of which education is a part. How can medical education move toward competency-based education? How can we move toward a true continuum of professional development? And how can we transform the medical education system to be certain that we are preparing our current and future health professions workforces to serve society’s greatest needs?

The change to Step 1 scoring has not altered the need to reform the UME to GME transition, but it has accelerated the process. Fortunately, InCUS provided a framework to tackle this reform by identifying both a set of action priorities and a way of working together effectively. Even before the Step 1 scoring announcement, the major associations that represent medical education, under the auspices of the Coalition for Physician Accountability, had convened a cross-organizational committee to create solutions to the challenges inherent in the UME-to-GME transition.⁷ This group should move forward in systematically reviewing the entire transition to residency process and make recommendations for change.

The COVID-19 pandemic is dominating our educational and clinical environments and is now the biggest disruptor in all aspects of life, not just medical education. Undoubtedly, there will be changes to health care and medical education as a result. Some will be temporary; others will be sustained. Some will have deeply negative consequences that must be managed. Some will be positive changes that should be embraced. As leaders in medical education, it is our responsibility to have a vision for and then implement an improved continuum of education that maintains our core values and fits the health care delivery needs of today and the future.

At this moment, we cannot predict with any certainty what changes will come. If the medical education community continues to work on the frontlines to adapt to the challenges of the COVID-19 pandemic while executing a review of the transition to residency process and commits to thoughtfully integrating these efforts, we can emerge from this pandemic in an improved system for learners, medical schools, and residency programs.