Sufficient Competence to Enter the Unsupervised Practice of Orthopaedics: What Is It, When Does It Occur, and Do We Know It When We See It?

Abstract

The goal of residency programs is to provide an educational venue with graduated responsibility and increasing levels of independence as preparation for entering the unsupervised practice of medicine. Surgical programs are required to both cultivate and convey skills pursuant to three fundamental domains: a sufficient fund of knowledge, technical competence in surgical procedures, and a degree of professionalism to enable ethical independent practice. Never before has the expectation that residency programs provide graduated responsibility in preparation for entering the unsupervised practice of medicine been so clearly articulated as it has by Nasca in the recent Accreditation Council for Graduate Medical Education (ACGME) work-hour guideline revisions. The Royal College of Physicians and Surgeons has provided similar guidance in Canada. Yet, as we progress further into the second decade of work-hour restrictions, it is unclear that we have adequately defined or can recognize the critical end points essential to trainee competency. What is clear is that we must achieve these end points in a manner different from that prior to the introduction of work-hour restrictions. We present the current state of thinking from North America and contrast this with the evolving medical educational process in the United Kingdom.

Never before has the expectation that residency programs provide graduated responsibility in preparation for entering the unsupervised practice of medicine been so clearly articulated as it has by Nasca in the recent Accreditation Council for Graduate Medical Education (ACGME) work-hour guideline revisions¹. The Royal College of Physicians and Surgeons has provided similar guidance in Canada².

Competence in Cognitive Knowledge

The importance of the physician as a communicator, professional, and scholar cannot be overstated. However, it is the command of a relevant fund of knowledge that defines the physician as competent and expert in medical subspecialty practice.

One can define various levels of competence as milestones in the performance of specific tasks. Clearly the most critical milestone is that which defines when an individual is capable of functioning as an independent practitioner without supervision, which is the threshold at which certification takes place. This performance threshold has historically been subjectively determined by the training program and then confirmed by a certifying body by a summative examination of knowledge.

Recent initiatives in medical education aim to precisely define levels of achievement on the basis of predefined objective measures and predicate progression on the attainment of these levels. Ongoing formative assessment and feedback facilitates the identification of gaps in knowledge requiring further study. These principles form the basis of variable time points of advancement for individuals, whose aptitudes and rates of learning may differ. Knowledge-acquisition theory describes different models of progression over time. Continuous improvement is characterized by a gradual linear increase in knowledge over a prolonged period. The breakthrough model describes dramatic and sudden increases from a steady state. A more ideal method would combine the two, with a series of linear increases and interspersed breakthroughs with dramatic progression.

Competency-based medical education is founded on the concept of progression based on the demonstration of competence rather than the completion of a predetermined time period. This concept has been successfully demonstrated in a modular pilot orthopaedic training program in which residents can progress through training at a variable rate³. In this new paradigm, some residents have completed training and passed credentialing examinations in less than four years, while others have required a longer time. The key to this model is in the assessment and evaluation of acquired knowledge. The creation of a rigorous evaluation scheme, whereby learners know exactly the objectives and required material, is essential to determining the competence of the learner. Several methods exist to assess an individual’s cognitive knowledge. Assessments that use multiple-choice questions are easily administered and objective, but they evaluate applied knowledge less well than they do factual knowledge. Assessments that use short-answer questions are more subjective and difficult to administer, but they evaluate applied knowledge more effectively. Oral examinations require a substantial instructor time commitment but effectively assess applied knowledge, including patient-management scenarios. Ideally, a combination of these tools should be utilized to broadly evaluate cognitive knowledge.

The Orthopaedic In-Training Examination (OITE) is a validated multiple-choice examination that traditionally has been used to evaluate individual cognitive knowledge as well as to provide programs with feedback on areas of consistent resident underperformance. Despite its limitations as a multiple-choice examination, the OITE has been shown to correlate with performance on the two-part ABOS (American Board of Orthopaedic Surgery) certification examination. Swanson et al.⁴ found that OITE scores increased throughout training and that clinical performance in later years correlated well with performance on part-I of the ABOS examination; residents who were below the 10th percentile were more likely to fail part-I of the ABOS examination. Similarly, Herndon et al.⁵ found that OITE performance, particularly in years two through four, was predictive of success on both parts of the ABOS examination. Importantly, the OITE correlation with success on ABOS part II suggests that a multiple-choice examination of largely factual knowledge can be predictive of an individual’s ability to assimilate that knowledge into effective clinical practice.

It is unlikely that final summative credentialing examinations will be abandoned in the near future as the standard by which individuals are judged to have mastered sufficient cognitive knowledge to practice medicine independently. However, it will be essential for residency programs to develop evaluation plans that continuously and effectively assess performance and readiness to take the summative examination. Although performance on a cognitive knowledge-based examination may correlate with performance on the certifying examination and may be predictive of the mastery of sufficient knowledge to function competently in independent, unsupervised practice, what has yet to be determined is, when are individuals capable of taking the test, and when should they be allowed to do so? In focusing on preparedness to “take the test,” it is important to acknowledge the unstated confidence placed in the validity of the test. We must trust that a “sampling error” will not inappropriately deem an unfit individual to be competent on the basis of a narrow surrogate that misses testing an area of knowledge that then remains critically deficient.

Competence in Technical Skills

I hear and I forget. I see and I remember. I do and I understand.

—Confucius (551-479 b.c.)

The complete education of a surgeon requires not only mastery of a body of medical knowledge but also technical proficiency in performing tasks—more specifically, surgical procedures—that are requisite to the successful practice of surgery. Additionally, in orthopaedics, there are skills related to fracture care that require manual dexterity and three-dimensional spatial orientation. The orthopaedic surgeon must master a requisite fund of knowledge in addition to demonstrating technical competency necessary to perform a “craft” involving fracture care and surgery.

Conventional surgical education has been largely time-defined, typically by the number of years in training. Each surgical specialty has rather arbitrarily determined the length of its training program, and these preordained periods have changed very little during the history of modern surgical education. The acquisition of technical skills has been predominantly through the repetition of low-density learning interactions while logging countless hours in service in the hospital. Indeed, to many learners, doing is learning. With duty-hour restrictions, however, the “luxury” of poorly focused surgical redundancy as a learning vehicle has been lost, and the surgical apprenticeship model will end. Conservative calculations suggest that “time in training” in an orthopaedic surgical residency since the implementation of duty-hour regulations has decreased by more than ninety-five work weeks, or nearly two years’ worth of eighty-hour work weeks⁶. Moreover, with increased public awareness of medical error, heralded by an Institute of Medicine report⁷, and the implied role of trainees in the propagation of error, “on-the-job” surgical training enjoys little social acceptance today. Finally, the expansion of medical knowledge and the proliferation of technical advances have increased the knowledge and skill base expected of a newly minted surgical practitioner.

These issues have collectively contributed to an unsettling deficit in confidence, if not perceived competence, of recent graduates of surgical residency programs. Over the past decade, failure rates for the American Board of Surgery oral certifying examination⁸ and the ABOS written examination⁹ have increased. In a 2012 survey conducted by the Southeastern Surgical Congress, nearly 25% of senior surgical residents thought that they were unprepared for independent practice. Surgical program directors similarly opined that 37% of current graduates were not prepared to enter unsupervised practice¹⁰. Accordingly, 80% of graduating surgical residents were pursuing additional fellowship training, which closely corresponds with the more than 95% of orthopaedic residents who currently pursue additional fellowship education. Likewise, in a 2013 survey of subspecialty surgical fellowship directors, an alarming 66% of residency graduates were thought to be unable to operate unsupervised for thirty minutes during a major surgical procedure¹¹. In the cognitive and professionalism domains, one-quarter were thought to not recognize early signs of surgical complications, and more than one-third exhibited a lack of patient “ownership.” Moreover, the visually graded technical skill of bariatric surgeons has been associated with three-fold greater complication rates and five-fold increased mortality rates when comparing bottom and top quartiles¹². Collectively, these factors provide a compelling reason for changing the way that surgical education is conducted in an era in which compliance with limited work hours is required.

Ericsson introduced the concept of deliberate practice as a means to improve technical performance^13,14, Colvin popularized and embellished the idea¹⁵, and Gladwell most recently referenced it¹⁶. Deliberate practice is defined as purposeful activity designed to optimize improvement and ultimate performance; Ericsson et al. estimated that the attainment of expert performance requires intense practice for a period of no less than ten years or 10,000 hours¹³. Indeed, purposeful training and coaching has been shown to be far more essential to superlative performance than any individual innate talent or inborn skill^13-15. While sophisticated simulation laboratories have been promoted as integral to modern surgical education, their cost has precluded widespread adoption, and they do not, in fact, provide the essential time and commitment necessary for effective practice. Indeed, the tenets of deliberate practice are straightforward and require neither high technology nor great expense for successful implementation. Colvin suggests that deliberate practice is characterized by five essential elements¹⁵; it is focused on tasks that are specifically designed to improve performance, can be repeated frequently, and provide continuous feedback to the student. The final two elements are critical: deliberate practice is mentally demanding, and it is not much fun! The learner is persistently challenged by deliberate practice to master critical but difficult tasks that are outside of the zone of comfort, somewhere between those that are easy to perform and those that are not yet mastered but are attainable.

Historically, the attainment of technical proficiency during surgical residency has been subjectively determined by qualified experts. However, contemporary educational frameworks demand more precise assessment of competency, whether or not that assessment can be quantified. Such assessment would ideally take on a formative, as well as a summative, function in the evaluation of surgical trainees. In some programs, training within a surgical-skill laboratory has been designed as a prerequisite to resident participation in the operating room. Surgical tasks can be taught, practiced, and measured in a skills laboratory, and the time required to complete less easily measured tasks can be used both for teaching and as an indicator of technical proficiency. While general surgical and orthopaedic residency review committees (RRCs) now mandate the existence of such facilities for residency education, their use as a prerequisite in determining readiness for operating-room participation is not currently specified but should be not far into the future. Both our social contract with our patients and our obligation to resident education demand nothing less. Moreover, and perhaps more importantly, the foundational habits of the learner needed for the successful implementation of deliberate practice will concurrently foster the tenet of “life-long learning” that is essential for the continued success of a practitioner throughout a professional career.

The Challenge of Assessing Medical Professionalism

Professionalism lies at the core of what it means to be a physician and is classified as an essential competency by both American and Canadian certifying bodies. Merton stated that the function of medical education is to “transmit the culture of medicine and…shape the novice into an effective practitioner of medicine…to provide him with a professional identity so that he comes to think, act and feel like a physician.”¹⁷ We now understand that “thinking, acting, and feeling like a physician” is embodied in the word professionalism. In assessing the professionalism of students, residents, or practitioners, the term professionalism is a surrogate for the presence of a professional identity¹⁸. While a competency-based resident education framework has many theoretical advantages, it is difficult to determine whether an individual demonstrates sufficient competence and professionalism to enter the unsupervised practice of medicine. We do know that the acquisition of a professional identity best ensures the presence of professional behaviors and the progressive emergence of a competent orthopaedic surgeon throughout residency.

An emphasis on the explicit learning of professionalism emerged during the past two decades and now exists in most programs. While physicians have always referred to themselves as professionals, the subject was never taught and its related assessment was largely subjective. We now know that professionalism can, and must, be taught and learned throughout the continuum of medical education and requires specific longitudinal programs¹⁹. Explicit teaching of the definition of professionalism and the reason for the existence of the privileged position of professions in society is a necessary starting point. The values and attitudes of the profession must be incorporated into the “self” of each learner by the development of specific activities designed to promote reflection on the various aspects of professionalism.

Less progress has been made in the assessment of the professionalism of learners, primarily because attitudes and values are largely subjective and defy reliable assessment in the absence of a conscious mindfulness of the intrinsic subjectivity of the task. However, behaviors that reflect an individual’s values and attitudes can be more reliably assessed. The past decade has witnessed the development of such methods^20,21; while some general principles have emerged, much remains to be done.

General Principles for the Assessment of Professionalism

The assessment of professionalism should be guided by the following general principles²²: (1) It must be longitudinal and stage-appropriate throughout the continuum of medical education. Professional identity is acquired gradually, with much more expected in the final than in the first year of residency. (2) It must be aligned with the learning objectives and based on the definition of professionalism used in the program, which is the common starting point of understanding, or the cognitive foundation. (3) It must be both formative and summative. Formative assessment assists in the development of a professional identity through constructive feedback. Summative assessment ensures that those entering practice are fit to do so. (4) The nature of professionalism—the cognitive base—must be both taught and assessed to emphasize its importance. The assessment can be accomplished in a variety of ways. (5) A validated list of observable behaviors that reflect the attributes of the professional should be developed and assessed on a regular basis; such a list is used for the P-MEX (Professionalism Mini-Evaluation Exercise), a core evaluative tool at McGill University²³. (6) The reliability and validity of this subjective assessment can only be ensured if multiple observations are made by multiple observers; ten to twelve observations are necessary to be defensible. (7) It can be strengthened by inclusion of diverse groups, such as patients, peers, other health-care professionals, and support staff in a structured, rigorous, and fair process. (8) It is essential to assess perceived unprofessional behavior in the appropriate context and determine whether it constitutes a single lapse or a pattern of chronic transgressions²⁴. (9) Consequences and rewards must be attached to the assessment of professionalism. Unprofessional behavior during residency is predictive of similar behavior in practice; remediation is necessary. If the behavior persists, residents must not be allowed to enter practice²⁵.

The emphasis on observable behaviors limits the assessment of professionalism to qualities that are frequently demonstrated, leaving many aspects, such as self-regulation, difficult to assess. However, professionalism is much more than a systematic sum of a few individual behaviors and requires a holistic approach best reflected in a narrative summary by one or more observers consistently engaged over a prolonged period of time²⁶.

Orthopaedic Training: The U.K. Perspective

Historically, the U.K. training program was an apprenticeship, with trainees working eighty to 100 hours per week, no curriculum or ongoing formalized assessment, and no exit examination. The average time to become a consultant after qualifying from medical school was between ten and twenty years. Modernizing Medical Careers was introduced between 2005 and 2007 as a program of radical change designed to be transparent and efficient and to improve the quality of care by reforming postgraduate medical education according to structured national standards. The first competency-based curriculum for trauma and orthopaedics was launched in 2007 and revised in August 2013.

The U.K. system considers three inter-related facets of training, loosely known as head, hands, and heart: (1) clinical knowledge, (2) clinical skills, and (3) professional behavior and leadership. After they qualify from medical school, the training pathway for orthopaedic and trauma surgeons includes foundation years one and two, which cover all relevant aspects of general surgery, medicine, primary care, and psychiatry. Trainees then select a specialty; if they choose surgery, they will undertake two core training years to develop their surgical competencies. If successful, they apply to join the higher surgical training program, which progresses through six years from ST (surgical training)3 to ST8.

The Intercollegiate Surgical Curriculum Program (ISCP) utilizes the e-logbook, which includes all surgical specialties, can be queried at will, and currently contains over twenty-three million logged procedures. Trainees submit their procedures according to their role as primary surgeon or assistant and whether the trainer had scrubbed in or not; under a system of work-based assessments, trainees must log forty to eighty events yearly. In order to advance to the next training year, each trainee in the U.K. must pass an annual review of competency progression, which includes both written and oral components assessed by a committee inclusive of the training program director. On completion of six years with satisfactory reviews and meeting minimum logbook numbers and procedure-based assessments demonstrating skills progression indicative of competence, the trainee joins the Specialist Register. Currently, there are twelve index orthopaedic procedures with minimum case requirements that must be logged by the trainee as primary surgeon; seven are trauma-based and include hemiarthroplasty, hip fracture repair with a compression screw, ankle open reduction and internal fixation, intramedullary nailing of the femur and tibia, tension-band wiring of the patella or olecranon, tendon repair, and external fixation. Elective practice minimums are expected for total hip replacement, total knee replacement, arthroscopy, first ray surgery of the foot, and carpal tunnel decompression. A natural progression is expected, with concentration on trauma cases in the early years and the development of expertise in elective surgery during the later years.

When trainees start a new post, they meet with their trainer and formulate a learning agreement. Two further assessments occur during their six-month rotation to ensure progression. Trainees are expected to do at least two work-based assessments each week, which may include case-based discussions or procedure-based assessments; all learning events and assessments are entered into the e-logbook. The trainer validates and rates each training experience, including obtaining consent, preoperative planning, performing the operative procedure, and postoperative care, to provide an assessment of the trainee’s ability to formulate a plan, carry out surgical procedures, and deal with complications. It is imperative that progression is demonstrated throughout training from level one to four, which confirms the trainee’s competence to perform the procedure unsupervised and deal with complications. Following trainer validation, all events are entered into the e-logbook on the web site. Each trainee is encouraged to obtain multisource feedback from a minimum of eight diverse individuals providing broad input on trainee performance within the team. These assessments are considered important to ensuring the quality of training, motivating learners for self-assessment, and providing direct learning and critique for unsatisfactory trainees.

This curriculum is thought to provide effective assessment of trainees and to be valid, reliable, and feasible. It provides different views from numerous people and a continuous cycle of assessment, and is conducted by trainers specifically schooled in assessment. Orthopaedic simulation is becoming an important element of training surgeons and is being added to the curriculum. The advantages of the current training process are the curriculum, excellent assessment tools, and a high level of trainer engagement. In addition, the collected information provides insight into where trainees receive the best education. Training has been affected by the European Working Time Directive (EWTD), which was introduced in 1998. Medical staff were exempt until August 2009, when, despite concerns expressed by surgical training organizations, all trainees were directed to work a maximum of forty-eight hours weekly²⁷. Before the EWTD, annual available training hours were about 21,000, but with full current implementation, this has been reduced to 7640 hours. Consequently, the mean number of cases logged by trainees has substantially dropped, by nearly 20% for trauma and 30% for elective procedures²⁷. As a result, graduating trainees are inexperienced; most undertake one or two fellowships and will require mentoring from senior colleagues as young consultants.

The Way Forward

What is becoming increasingly clear is that time-defined medical education has run its course and has outlived its practical usefulness. Challenged by constrained work hours, increasing student debt, and an ever-enlarging body of knowledge necessary for competent practice, “time in the saddle” can no longer be accepted as the yardstick by which we determine readiness for the independent practice of medicine. Not only is the dawn of competency-based education upon us, but it seemingly cannot appear soon enough. To support its widespread adoption, we will need to refine methods of assessment that are essential to the recognition and appraisal of the core professional activities on which milestones and competency are predicated. Moreover, while “hours on the job” are constrained by regulatory bodies, we cannot allow such a “clock-punching” mentality to supersede the ethos of professionalism and altruistic commitment to patient care that we must inculcate in our trainees. Finally, we will need to more creatively optimize active learning within the imposed constraints of duty-hour limits and provide the opportunity, as well as the inspiration, for trainees to commit to “deliberate practice” needed to achieve expert performance in their chosen field. Yet, even with alignment and effective execution of all of these educational refinements, the data from the U.K. suggest that our newly minted surgeons lack the needed experience for effective clinical practice. Better use of the time we have across the continuum of medical education will be our mandate. In North America, this may translate into a three-year medical curriculum, a morphing of the current fourth year of medical school into an apprenticeship that resembles a rotating internship, or earlier focusing of medical education to fit the available time constraints. While we strive to preserve the core principles of medical education, it is both evident and unsettling that the education of tomorrow’s orthopaedic surgeon will not, and cannot, resemble anything that is familiar to the current generation of clinical and educational leaders.