Demise of the USMLE Step-2 CS exam: Rationalizing a way forward

Abstract

The COVID-19 pandemic has compelled rethinking and changes in medical education, the most controversial perhaps being the cancelation of USMLE Step-2 Clinical Skills exam (Step-2 CS). What started in March of 2020 as suspension of this professional licensure exam, because of concerns about infection risk for examinees, standardized patients (SPs), and administrators, soon became permanent cancelation in January 2021. Expectedly, it triggered debate in medical education circles. Positively, however, the USMLE regulatory agencies (NBME and FSMB) saw an opportunity to innovate an exam tainted with perceptions of validity deficits, cost, examinee inconvenience, and worries about future pandemics; they therefore called for a public debate to fashion a way forward. We have approached the issue by defining Clinical Skills (CS), exploring its epistemology and historic evolution, including assessment modalities from Hippocratic times to the modern era. We defined CS as the art of medicine manifest in the physician-patient encounter as history taking (driven by communication skills and cultural competence) and physical examination. We classified CS components into knowledge and psychomotor skill domains, established their relative importance in the physician process (clinical reasoning) of diagnosis, thus establishing a theoretical framework for developing valid, reliable, feasible, fair, and verifiable CS assessment. Given the concerns for COVID-19 and future pandemics, we established that CS can largely be assessed remotely, and what could not, can be assessed locally (school/regional consortia level) as part of a USMLE-regulated/supervised assessment regimen with established national standards, thus maintaining USMLE's fiduciary responsibilities. We have suggested a national/regional program for faculty development in CS curriculum development, and assessment, including standard setting skills. This pool of expert faculty will form the nucleus of our proposed USMLE-regulated External Peer Review Initiative (EPRI). Finally, we suggest that CS evolves into an academic discipline/department of its own, rooted in scholarship.

1. Introduction

1.1. Defining clinical skills and its epistemology

DDoctoring is as much a science as an art of medicine, both being two sides of the same coin actively interacting throughout the physician-patient encounter. The art which we hereby define as clinical skills (CS) manifests in what the physician says and does- the seen behaviors /actions of the physician. These behaviors are driven by the unseen science via a hypothetico-deductive reasoning process (clinical reasoning) which we define as the bridge between the art and science of medicine. What, when, where, and how the physician says things is defined as communication skills which drives the patient interview (history taking) and is itself optimally rooted in cultural competence. Lastly, physical examination (PE), with or without instrumentation (procedural skills) is what is seen when the physician displays psychomotor skills.

The essence of the physician-patient encounter is to arrive at a diagnosis as the basis for moving forward with a shared management plan that includes treatment, health education, and an action plan with the goal to cure when possible, prevent secondary complications, rehabilitate, and palliate when everything else fails. In this encounter the physician arrives at a diagnosis through the professional instruments of history taking, physical examination, and laboratory tests. The information gathered at each step is subjected to clinical reasoning to arrive at the diagnosis and a management plan. The clinical reasoning process is informed by the basic and clinical sciences of medicine, including the social sciences. Of the aforementioned instruments, the information/data gathered from history taking contributes approximately 80% of the physician's ability to arrive at a diagnosis, the physical examination, 10%, and laboratory tests, 8–9%. In the remainder 1–2% of encounters, the physician may never determine a diagnosis.^{1 , 2} The importance of history taking as borne out by these data underlies the famous ever-relevant historic quote by Sir William Osler (1849- 1919) who said, “Listen to your patient, he is telling you the diagnosis.”³ In effect, if we must choose one dimension of the physician-patient encounter (CS) for maximum attention to ensure competence, it is history taking with its underlying communication skills (verbal, non-verbal, and written) and cultural competency. Understanding these definitional principles is crucial to delving into the debate on the demise of the Step-2 CS and how to respond to the USMLE call for an authentic reform of CS assessment.

1.2. Background to Step-2 CS demise

In March of 2020, cognizant of the deadly nature of the COVID-19 contagion and its public health impacts, the USMLE-sponsoring organizations- the National Board of Medical Educators (NBME) and the Federation of State Medical Boards (FSMB)- announced a temporary cessation of the Step-2 CS exams which was followed by a May 2020 suspension of the exams for a period of 12–18 months. The goal, among others, was to relaunch the exam with changes that “harnessed technology, reduced or eliminated exam-associated COVID-19 risk, reduced or eliminated the need for examinee travel, and reflected changes in medical education arising since the exam was first launched in 2004.”⁴ With the benefit of hindsight, this was great thinking but also a huge task for such a short time frame as acknowledged by the NBME and FSMB.⁴ Not surprising, therefore, on January 26, 2021 these agencies announced the discontinuation of the plan to relaunch the Step-2 CS. This development raised serious concerns within the medical education community with some (including authors FIA and SO) challenging the wisdom behind the decision, especially when no alternative assessment plan was in place. With hindsight and the increasing literature on the subject, it has become easier to understand the compelling circumstances of the decision and therefore the ability for a more balanced evaluation. In this vein, a must-read article for anyone delving into the pros and cons of the cancelation of the Step 2 CS exams is that co-authored by the leaders of the two agencies (NBME and FSMB) that sponsor the USMLE. The reasons advanced included the need to safeguard the physical and mental health of student examinees, SPs and staff, and the untenable timeframe for achieving the desired goal of a new assessment approach.⁴ The high passing rates of about 95% in the Step-2 CS exam was another reason reportedly advanced for the decision.⁵ We find this rationale unacceptable for reasons shared below. In the circumstance of the pandemic, one cycle of the national Step-2 CS exam suspended, and the second cycle fast approaching with no clear pathway to developing and deploying a safe and valid nationally applicable assessment, the USMLE likely had no choice than to cancel the exams outright. Given the certifying nature of the exam and the circumstances of COVID-19, continuing the suspension could have led to a crisis scenario with public perception of “letting loose USMLE-uncertified medical graduates on the public.” On the positive side, the NBME and FSMB also explained that the plan was to evolve Step-2 CS assessment by advancing to a phase of mobilizing the medical education community to develop novel approaches for achieving the same goal of a national level assessment of CS. This plan may yield what becomes a positive and significant COVID-19- induced change. In the rest of this article, we briefly explore the changing historic phases of CS in the context of curriculum and assessment, establish a theoretical framework for, and offer our suggestions for going forward.

2. Historic phases of CS development

In initially suspending the Step-2 CS, the USMLE announced that the intention was to relaunch an iteration of the exam to reflect, among others, changes in medical education which arose since the exam was first launched in 2004.⁴ This was an acknowledgment that the nature of CS and therefore its teaching and assessment change with time as dictated by developments in medical science and technology. Understanding the impact on CS of historic developments in medical science and education could hold the key to understanding the way forward. In a recent commentary, Howley and Engle traced the history of the Step-2 CS exam from its early origins through to its current demise.⁶ In this section we expand on their perspective by briefly reviewing not just CS assessment but also the historic milestones/changes in medicine and medical education that have impacted CS content, delivery, and assessment, and we relate these to the current climate as a basis for defining a way forward; in effect visualizing the future through the lens of history. We divide the historic developments into two distinct periods- the pre-modern and the modern, the dividing line being in the 1960s with the origins of Problem-Based Learning (PBL) with its watershed and defining clinical problem-driven and student-centered learning approaches.

2.1. The pre-modern era

Historically and as can be gleaned from the Hippocratic oath, medicine was largely an art with little documented science behind the clinical skills of the ancient physician. Medical education of that era, not surprising therefore, was purely by observation (apprenticeship). A valid assessment of CS in that era would therefore have constituted essentially of observing the student apprentice in a real-life practice environment, akin to our modern Workplace-based Assessment (WBA).⁷ The apprenticeship model with little science generally persisted until the Flexner intervention of 1908.⁸ The major contribution of Flexner to medical education was the call to firm medical training (apprenticeship) with science. It resulted in the two-plus-two curriculum (two years of basic sciences followed by two years of clinical training). Flexner also called for national standards in medical training. Following the Flexner report CS was taught and assessed in the last two (clerkship) years of training and the typical exam involved the long and short cases (spot diagnosis). With the long case, the examinee was allotted a real patient and time to take a full history, perform a physical examination, arrive at a diagnosis, and a treatment/ management plan. The examinees presented their findings to a team of faculty examiners who grilled them on any aspects of the CS encounter, including the history, physical examination, laboratory tests, and clinical reasoning. The examinee would often be required to demonstrate some of the physical examination skills relevant to the patient. Communication skills were minimally observed or assessed. Another downside of this CS assessment was in its validity and fairness. From a validity perspective, each student was assessed based only on one or two randomly assigned patients purportedly representing the whole spectrum of clinical conditions in that clerkship. In addition, the randomness of assigned patients and raters meant that within a class of students, the testing experiences varied, and thus passing or failing could depend significantly on chance.

2.2. The modern era

The introduction of Problem-Based Learning in the late 1960s revolutionized medical education by heralding the shift from teacher- to a student- centered learning paradigm driven by clinical problems. In effect, the Flexner exhortation to firm practice with science took a giant leap with the learning of basic sciences of the pre-clerkship curriculum in clinical context, albeit largely through paper-based clinical scenarios. In this vertical integration model, real patient encounters occurred mainly in the clerkship years, so while CS could be fully and authentically assessed in the clerkship years, some degree of CS assessment could now be authentically performed in the pre-clerkship years by virtue of the paper-based clinical scenarios. This era also saw the emergence of other significant developments in medical education, including Barrow's programmed patient (today's Standardized Patient) in an era (1960s- 1970s) described by Howley and Engle as the “Era of Origination.”⁶ In the same time frame, the Objective Structured Clinical Exam (OSCE) was introduced in 1978.⁹ This opened the door to the use of SPs with the capacity to emulate multiple clinical scenarios within a single exam session, thus enhancing validity of the exam through a broad coverage of the curriculum. It also promotes fairness as all students are tested on the same clinical scenarios. By the turn of the 21st century the concept of Early Clinical Immersion (ECI) had gained ground, providing opportunities for medical students to encounter real patients in real-life clinic in the early pre-clerkship phase of the curriculum and thus the prospect of putting into practice the knowledge and skills gained in the simulation labs. The Service-Learning preceptorship curriculum is a good example of the ECI curriculum.¹⁰ In these scenarios, students’ CS can be assessed in multiple ways, including (1) simulation lab OSCE approaches, (2) real-life Workplace-based Assessment (WBA) by trained faculty /resident preceptors and /or peers using assessment modalities such as the portfolio assessment, and (3) the 360-degrees peer evaluation. This era continues to evolve and has reached the current stage where in addition to ECI, emphasis is now placed on Competency-Based Medical Education (CBME), Telehealth, and Interprofessional Education (IPE). Each of these dictates specific curricular requirements with appropriate delivery and assessment modalities that are authentic for measuring the relevant CS. It is pertinent to state that the reason for the increasing call for CBME is in part the observation that medical school graduates entering various residency programs tend to be deficient in the clinical skills required for that level of training,^{11 , 12} thus calling to question the validity of the Step-2 CS residency- qualifying exams. The use of tele-simulation (i.e. remote SP encounters) and telehealth for low-stakes formative assessment are reported to have increased since COVID-19.^{13 , 14} These experiences could form the nucleus/trigger for a national level assessment plan for these skills.

3. Rationalizing a way forward

A major purpose of the S2CS is to enable regulatory agencies (e.g., the NBME) to serve their fiduciary duty by providing the public an assurance that our medical graduates have attained a level of competence to become practicing physicians in whose hands they can in all reasonableness entrust their lives. It is both a moral and professional obligation, and achieving this entails a national level valid, reliable, fair, feasible, verifiable, appropriately delivered, and managed competency-based assessment by whatever name.

It can be argued that without the external power of regulatory agencies, such as the USMLE/ LCME, schools could deliver a substandard CS curriculum and/or substandard assessment and thus graduate “substandard” physicians.^{15 , 16} This was essentially the case pre-Flexner where schools proliferated and institutions were doing their own thing, turning out graduates with no minimum national standard.⁸ In the modern era schools have been reported to contribute in lowering standards through reluctance to fail their own students in a phenomenon described as “failure-to-fail.”¹⁶ The introduction of the Step-2 CS in 2004 as a licensure exam arose in part because of then-prevalent concerns about the clinical skills competency of medical graduates entering various residencies.¹⁷ That these concerns persist^{11 , 12} (and if valid) indicates that as a licensure exam the Step-2CS was technically not fit for purpose. In effect our challenge going forward is in developing valid assessments instruments and protocols for CS to ensure the attainment of well-defined and measurable national standards.

A major reason for cancelling the Step-2 CS was the concern for COVID-19 or any future contagion increasing the morbidity/mortality risk for examinees, SPs and exam administrators who come in physical contact. The USMLE explored the possibility of an online CS exam but was legitimately worried about its limitations/validity for assessing psychomotor skills.⁴ The literature on establishing novel assessment approaches for the Step 2 CS is replete with similar concerns. While these are apt, perhaps an evaluation that focuses on the percentage importance/weightage of the psychomotor skills (PE) component of CS will begin to lead to more theoretically and practically sound conclusions. In effect, an assessment modality that is able to validly, reliably, fairly, feasibly, and verifiably assess more than 90% of the essence of the CS curriculum (i.e. history taking with communication skills (oral, non-verbal, and written), cultural competence, and clinical reasoning) should not be considered deficient, even as we must find ways to make up for the assessment of the remaining 10% (PE) as discussed below. In focusing on the remote assessment of PE which falls essentially in the Bloom's taxonomy psychomotor skills learning domain, we suggest applying the skills-learning/assessment philosophy depicted in the Millers pyramid of assessment¹⁸ (Fig. 1 ) wherein the foundational step is to “Know” about the skill, followed by “Knows How,” then “Shows How,” and finally at the apex is “Does.” While we are unable to assess the psychomotor component of this triangle (Shows How), the rest, which are essentially cognitive, can be assessed remotely in a COVID-19 safe environment, thus taking care of about 5% of the 10% allotted to PE. The remaining 5% or more could be reliably and validly assessed at multiple levels in the medical school, including using WBA in clerkship rotations. Other reasons advanced for cancelling the Step-2 CS, such as the high (95%) pass rates⁵ should not on their own constitute an argument for cancelling, but rather one for recalibrating the validity factors of the assessment, including content, coverage, assessment types, standard setting, and administration. As an example, using the 95% pass rate as basis for cancelation is untenable as it is based on the traditional norm-referenced assessment mindset which runs counter to the criterion-referenced approach that is required for the current push towards a competency-based medical education under which a 100% pass rate is the goal and is achievable with ipsative assessment curricula.¹⁹ The rest of this article offers suggestions for a way forward based on these arguments.

Fig. 1.

Miller's pyramid of clinical competence matched with appropriate level of assessment.

miller's pyramid of assessment - Google Search (Accessed 02/08/2023)

4. A way forward

Based on our definition of CS, its epistemology and the rationalizations derived therefrom, including the historic evolution of CS, we offer two approaches, one involving the establishment of a hugely revamped Step-2 CS and the other which accepts the demise of that exam. A condition necessary for either approach is that there should be a national level USMLE-regulated assessment regimen for reasons discussed above, including the all-important fiduciary responsibility. Other commonalities for the two approaches are:

• 1.The assessment modalities must meet strict criteria for validity, reliability, fairness, feasibility, and verifiability. Standard setting should be a requirement for all USMLE-approved licensure tests and should be conducted by well-trained faculty (see 3 below).
• 2.COVID-19/pandemic compliant. This is a unique feasibility condition requiring feasibility of exams without the risk of exposure to infection.
• 3.Formal training for all faculty (and non-faculty, such as SPs) who teach CS with certification requirement for all who shall be involved in USMLE (licensure)-recognized/regulated exams.

4.1. Proposal with Step-2 CS alive

Consistent with the conditions stated above, this proposal would involve:

• 1.Re-establishing all aspects of the Step-2 CS as we knew it, except for the significant difference that the exam will now focus solely on history taking, communication skills, cultural competence, and clinical reasoning. In effect all aspects of the original exam without the psychomotor components observed mainly in the PE component.
• 2.PE to be conducted at school level regulated by USMLE via the External Peer Review Initiative (EPRI) described below. Assessment of the psychomotor components of communication skills, represented by the acronym SOFTEN (Salutation, Open body language, Forward leaning, Touch, Eye contact, and Nod) to be done at this level.
• 3.All exams to be conducted online for COVID-19 compliance and to remove the cost of travel to national exam centers, one of the reasons given for cancelling the Step-2 CS.
• 4.Standard setting by teams of USMLE-certified faculty implemented as a strict validity requirement.

4.2. Proposal with Step-2 CS demise

Under this proposal, national CS assessment is not centrally organized and administered by the USMLE as with the Step-2 CS. Rather, it is all performed at the local individual (possibly regional) school level(s). However, the rules and regulations guiding all aspects of this assessment are set by the USMLE which must certify each exam, thus attesting to a national standard. This proposal depends entirely on our novel USMLE-regulated External Peer Review Initiative (EPRI).

4.3. External peer review initiative (EPRI)

Our EPRI idea is borrowed from medical education practices in the British Commonwealth. In the US, national standards are set, assured, and regulated centrally via the USMLE step exams. In the British Commonwealth there are no national level professional exams in Undergraduate Medical Education (UME) as against Graduate Medical Education (GME); UME standards are maintained at the school level by internal regulatory efforts coupled with a system of external peer review. The external examiner (peer reviewer) is always a senior academic (often associate or full professor level) in the particular discipline who is empowered to preside over all aspects of the professional exam, including its development (setting the exam questions, delivery, supervision, grading, standard setting) and judgment (determining who passes or fails). The external examiners ensure that the assessment meets high professional and therefore, national standards. This proposal would require the following steps:

• 1.The development of a national pool of well-trained CS faculty with some level of (USMLE-approved/recognized/managed) certification in the principles of medical education, CS curriculum development, delivery, and assessment.
• 2.Definition of the number (one or two) of USMLE- certified CS exams each medical student must take to qualify for graduation. We suggest two exams, one being fully and internally conducted by the individual school USMLE-certified faculty. This exam would serve as the mock exam, thus meeting the assessment principle of not delivering a high stakes big-bang (summative) exam without the fairness criterion of prior experience of the test environment.
• 3.The substantive /final USMLE CS licensure exam would be run locally at the school level, using each school's CS lab facilities, similar to the mock exams described above, except that the examiners shall all be external, chosen by the USMLE from its national pool of certified CS faculty, and reporting to the USMLE. Administrative support would be provided by the school and led by its internal USMLE-certified CS faculty. The exam questions (clinical scenarios utilizing standardized patients) shall be drawn from the USMLE question bank by the team of external examiners. The bank should contain questions that have already undergone USMLE-approved standard setting. This approach would benefit residency program directors if standards were assessed and reported similarly across schools.²⁰

4.4. Standard setting initiative

No assessment (exam) is meaningful without standard setting. This principle must strictly guide the way forward. Without proper standard setting, we cannot validly and reliably attest to the competency of our graduates. This paper is not focused on discussing standard setting and does not delve into the details of that subject. We, however, recognize the need to train CS faculty in the skills of standard setting as part of the USMLE- required CS-faculty certification proposal made above. The national Directors of Clinical Skills Education (DOCS) organization has recognized this need and has assembled a group of faculty with interest and expertise in assessment, including standard setting, to conduct regular seminars/workshops on this subject. We suggest that the USMLE reaches out to activate this group as a nucleus for growing a national CS assessment faculty development program. Authors SO and FIA are members of that DOCS group.

5. Conclusion

Whether or not one agrees with the reasons for the Step-2 CS cancelation, it has created the benefit of opening a conversation on the future of the national CS curriculum, especially its assessment. Our proposals have been based on two guiding principles. One is the need to keep a national level licensure exam as a basis for defining and certifying national CS standards for medical graduates, including foreign graduates. The other is the requirement that the exam/assessment regimen conforms to the best of medical education assessment principles around validity, reliability, fairness, feasibility, and verifiability. Given that assessment drives learning, such an authentic assessment regimen would set the standard for medical schools’ CS curricula and would drive student learning towards nationally defined competency levels. In addition, sustaining the national level exam by whatever name (we prefer Step-2 CS) would not perturb and could potentially enhance the positive impact on growth and development of CS infrastructure and human capital as reportedly induced by the advent of the Step-2CS in 2004.²¹ Our recommendations are also consistent with promoting any curriculum designed to handle the pandemic-induced resurgence of focus on telehealth/ telemedicine. They are also consistent with CBME, especially when combined with an ipsative assessment regimen that would align UME CS with GME and Continuing Professional Development (CPD). Our suggestions could also be explored for any future USMLE-level assessment of learning in Interprofessional Education (IPE). With the expanse and depth of technical knowledge required for the CS curriculum and assessment modalities prescribed herein, we propose that CS be recognized as a formal academic discipline for which scholarship is a requirement for its faculty. Our recommendation and those of others¹⁴ for a structured national faculty development program will help achieve and sustain this goal. Institutional and regulatory agency levels of leadership is critical for developing and sustaining the physical and administrative infrastructures coupled with the faculty training that any of our suggested paradigms would require. Finally, it is in the art of medicine (CS) that the physician is seen and appreciated. It defines the physician persona and therefore must be authentically inculcated in all who bear that name. The demise of the Step-2 CS and the debate it has sparked provides the medical education community a great opportunity to define that physician of the future.

Funding/Support

This work did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.