A Review of THE ROLE OF SIMULATION IN DEVELOPING AND ASSESSING ORTHOPAEDIC SURGICAL SKILLS

Abstract

Orthopaedic surgical skill is traditionally acquired during training in an apprenticeship model that has been largely unchanged for nearly 100 years. However, increased pressure for operating room efficiency, a focus on patient safety, work hour restrictions, and a movement towards competency-based education are changing the traditional paradigm. Surgical simulation has the potential to help address these changes. This manuscript reviews the scientific background on skill acquisition and surgical simulation as it applies to orthopaedic surgery. It argues that simulation in orthopaedics lags behind other disciplines and focuses too little on simulator validation. The case is made that orthopaedic training is more efficient with simulators that facilitate deliberate practice throughout resident training and more research should be focused on simulator validation and the refinement of skill definition.

Introduction

Orthopaedic surgical skill is traditionally acquired during training in an apprenticeship model that has been largely unchanged for nearly 100 years. However, increased pressure for operating room efficiency, a focus on patient safety, work hour restrictions, and a movement towards competency-based education are changing the traditional paradigm. Surgical simulation has the potential to help address these changes. This manuscript reviews the scientific background on skill acquisition and surgical simulation as it applies to orthopaedic surgery. It argues that simulation in orthopaedics lags behind other disciplines and focuses too little on simulator validation. The case is made that orthopaedic training is more efficient with simulators that facilitate deliberate practice throughout resident training and more research should be focused on simulator validation and the refinement of skill definition.

The Need for Deliberate Practice in Surgical Skill Acquisition

It is widely accepted that surgeons benefit from practice. The more often a surgical procedure is performed, the lower its morbidity^1-6 and the better the outcome⁷. However, practice should not be confused with repetition; performance does not improve simply because a task is repeated. The key to consistent improvement is “deliberate practice” combined with structured training. Deliberate practice involves engaging learners in focused, effortful skill repetition in progressive exercises that provide informative feedback. Trainees must receive immediate, informative feedback while trying to improve, with the particular feedback matching the characteristics of the task. Trainees should also perform the same or a very similar task repeatedly to increase performance⁸.

Motor skills are essential to surgical precision. Deliberate practice improves motor skills^9-10. Learners advance quickly by repeating well-defined, level-appropriate tasks and receiving immediate feedback that allows for error correction. For example, the steep learning curve for a radical prostatectomy, a mildly complex surgery, does not plateau until 250 performed operations^11-12. For a hip fracture surgery, residents require an average of 20-30 trials before attaining expert speed, but learning curves vary widely¹³.

Ericsson, Krampe, and Tesch-Römer proposed a theoretical framework for the acquisition of expert performance through deliberate practice¹⁰. Using the framework of deliberate practice, Ericsson et al. found the time to achieve expertise depends on cumulative time spent deliberately practicing, rather than the time since the activity was initiated. The famous standard of 10,0 hours of deliberate practice to become an expert is based on a study of the number of hours of deliberate practice by violinists (Fig. 1). By the age of 20, the most advanced group of expert violinists accumulated 10,0 hours, while the next most accomplished group had 2,500 fewer hours. The least accomplished group of experts accumulated 5,000 hours of deliberate practice¹⁰.

Figure 1. The relationship between level of expertise in violinists and the accumulated hours of deliberate practice. From Ericsson, et al.¹⁰, used with permission.

The concept of deliberate practice has also been explored in the medical domain. Residents who distributed practice on a drilling task over four weeks outperformed residents who logged all of their practice in one day¹⁴. The benefit of periodic practice over lumped practice is consistent with research in many other domains¹⁵. This advocates for repetitious skills practice throughout a curriculum, rather than an all-day session focusing on a specific skill.

Of fourteen studies reviewed in a meta-analysis, deliberate practice using simulation-based techniques was found overall to be a superior method of training compared to the apprenticeship model in the medical field¹⁶. The current apprenticeship model in orthopaedics does not facilitate deliberate practice, because it is organized around ad hoc experiential opportunities driven by patient needs rather than providing repetition of key skills based on learner needs. The key to facilitating deliberate practice is effective simulation.

The Role of Dedicated Simulation in Surgical Skills Training

Over the past decade, the role of surgical simulation in the acquisition of cognitive and technical skills has grown, particularly for minimally invasive and limited- exposure surgery. Many disciplines have shifted away from the apprenticeship model to teach surgical skills. The field of orthopaedics has not kept pace with these developments. Medical students and residents trained on simulators demonstrate improved performance in actual surgeries¹⁷. For example, experts performed reliably better on a bronchoscopy simulator than doctors with less or no experience¹⁸. Residents who had been trained to a certified level of competency on a laparoscopic simulator performed their first actual surgery with fewer errors, and caused fewer injuries, than a control group of residents without simulator training¹⁹.

A primary factor motivating programs to search for new, more efficient surgical training methods is a 2003 ACGME-mandated reduction in residents' work hours to a maximum of 80 hours per week. Although these changes threaten the amount of time residents have to practice surgical skills, it does not necessarily follow that their competency will decrease. Some practice is more effective at increasing skills than others, so the number of hours of practice may not be the most reliable measure of skill acquisition. To borrow a sports analogy, a baseball player comes up to bat three to four times a game. During each at-bat, the player sees an average of four pitches. This means in a game that averages nearly three hours, a batter sees roughly 12-16 pitches. There are simply not enough in-game hours to reach the expert batting level. In a batting cage a player can hit 50 balls in five minutes, the equivalent of playing four consecutive games. Simulation could supplement apprenticeship training by enabling residents to effectively practice more in less time.

The High Cost of Resident Training in an Apprenticeship Model

The apprenticeship model is the epitome of “hands-on, real world” training, but it comes at a cost. Any increased time in the operating room is expensive, a cost which is passed to the patient and the health care system. Bridges and Diamond compared operating times in general, pediatric, vascular, plastic, urologic, and trauma surgery procedures where residents were present to those with no residents in attendance²⁰. In their four-year study, operations with residents took a net duration of 12.64 minutes longer on average, from the first incision to leaving the room. Over a total of four years, the lost time was approximately 11,184 minutes per resident. In orthopaedics, a study of ACL reconstruction noted an average increased cost of $661.85 due to longer operative time when a resident completed the surgery compared to a faculty surgeon²¹.

Thus, the extra cost of resident training in the operating room further justifies the expense of simulation training. Simulation training before entering the operating room significantly decreases operation time^{19, 22-25}. Thus, simulation has the potential to decrease the cost of training. In order to realize these potential cost savings, the simulation must represent the skills to be trained well enough that training on simulators will reliably improve performance.

Developing and Validating Surgical Simulators

Building an effective simulator requires the designer to understand both the simulated task and the underlying skills required for the effective task execution. This requires an assessment of the procedure and its constituent tasks. Some surgical tasks are common among different procedures. Other tasks are unique to a particular surgery or situation. Some skills are applicable to many tasks. Other skills are task-specific. There may always be skills that need to be learned in the operating room, but many skills can and should be acquired elsewhere. Often, the initial acquisition of a surgical skill depends on developing the necessary psychomotor skills. Psychomotor skill is the coordination between the physical movement of the trainee and their cognitive processes. Psychomotor skills effectively develop through deliberate practice and structured training.

Kneebone²⁶ proposed the following criteria for critically evaluating a new or existing simulation:

• Simulations should allow for sustained, deliberate practice within a safe environment, ensuring that newly acquired skills are consolidated within a defined curriculum that assures regular reinforcement.

• Simulations should provide access to expert tutors when appropriate, ensuring that such support fades when it is no longer needed.

• Simulations should map onto real clinical experience, so that learning supports the experience gained within communities of practice.

• Simulation-based learning environments should provide a supportive, motivational, and learner-centered milieu that is conducive to learning.

Physical and virtual reality surgical simulators, alone or in tandem, have proven themselves as viable training platforms. Physical simulators range in sophistication from low-fidelity representations of a prescribed task, such as laparoscopic suturing, to very realistic, instrumented mannequins. Some simulators are task trainers. These are designed to teach a solitary task in a procedure through repetition. Other simulators are full procedural trainers. These are designed to replicate the entire scope of a complex scenario, such as a surgical procedure.

Some virtual reality surgical simulations involve haptic force-feedback (Fig. 2)^27-29. This is useful when the targeted technical skill includes both tactile force-feedback and eye-hand coordination, such as fluoroscopic wire navigation. When forces do not play an important role, basic eye-hand coordination may be trained without force feedback.

Figure 2. Haptic force-feedback shown in orthopaedic drilling simulation. (Left) Tsai and Tsai²⁷, (middle) Froelich, et al.²⁸, and (right) Vankipuram, et al.²⁹, each used with permission.

The determination of a simulator's psychometric properties (i.e., its reliability and validity) is one of the most important facets in the development of a simulator. Reliability refers to the degree of consistency or reproducibility with which an instrument measures what it is intended to measure. If something cannot be measured reliably, then the question of validity is rendered largely moot. The concept of validity addresses the question of whether the measurements obtained from the simulator vary with the educational construct the simulator is intended to measure. The most common categories of validity found in the medical simulation realm include face, content, construct, concurrent, and predictive validity. Each type of validity should ideally be defined within the context of the particular assessment³⁰.

Ultimately, a new simulator should pass multiple validity tests in order to be considered for skills training and competency assessment³¹. Face validity addresses the question, “To what extent does the platform simulate what it is supposed to simulate, e.g., the surgical procedure?” It refers to the subjective opinion about a test - its appropriateness for the intended use within the target population. Face validity is important for a test's practical utility and success of implementation³², particularly with respect to whether or not trainees will accept the simulation as a valid educational tool. Face validity is usually assessed with expert responses to questionnaires or surveys.

Content validity addresses the question, “Does the simulation measure the relevant dimensions of the task under study?” Content validity is often assessed with a thorough search of the literature and by interviewing expert surgeons. Face and content validity are subjective and arguably not as rigorous as some other assessments of a simulator's validity.

Construct validity defines the extent to which the simulator measures the specific trait or traits that it was designed to measure. Many construct validity studies demonstrate that different learner skill levels are associated with variations in the measurements made by the simulator. Demonstrating a significant difference in expert and novice scores demonstrates that the simulator correctly identifies quantifiable aspects of surgical skill.

Concurrent validity measures the extent to which the simulator agrees with an existing performance measure of a surgical task or procedure. The concurrent validity of surgical simulators is often assessed by comparing simulator scores with the Objective Structured Assessment of Technical Skill (OSATS) or against another, previously validated, simulator. It is frequently difficult and often impossible to assess transfer of learning when attempting to validate educational simulation technology¹⁶. The most convincing evidence is provided when researchers can correlate performance on the simulator with real world performance. With appropriate estimates of reliability of both the simulation measure and the measure obtained from actual practice, a correct correlation can provide a ‘true score’ correlation between the constructs measured by the two assessments.

Validity is not a binary determination, but reflects a gradual judgment, depending on the purpose of the measurement and the proper interpretation of the results. A single instrument may be used for many different purposes, and resulting scores may be more valid for one purpose than for another³³. This means that validity statements based on the evaluation of one task can be, and probably will be, different than those based on another task. Indeed, it is fundamental for creating a useful simulator-based teaching environment to recognize that a surgical procedure has to be divided into a series of steps that can be trained and measured separately³⁴.

A validated simulator can address the lack of opportunities for deliberate practice in surgical training. It will also provide residents with immediate skill feedback, an indispensable component of deliberate practice and the development of expertise. Residents need carefully devised educational variations providing incremental challenges to improve their surgical skills³⁵. The apprenticeship model struggles to control the real world challenges faced by the residents, and feedback requires time and attention from busy staff surgeons. Simulators can provide real-time feedback and instruction outside the operating room³⁶.

Simulator Fidelity, Complexity, and Transfer of Skill to the Operating Room

Training simulators recreate aspects of reality. The degree of verisimilitude affects the cost, time, complexity, and technology required in development. The current trend in simulation is to strive towards building the highest fidelity simulator possible. Unfortunately, in trying to mimic minute details, these ultra-realistic simulators occasionally include irrelevant tasks.

The complement to physical fidelity is psychological fidelity. Caird defines psychological fidelity as “the degree that a simulation produces the sensory and cognitive processes within the trainee as they might occur in operational theaters³⁷.” It is important to note that physical fidelity and psychological fidelity are not mutually exclusive. In some cases, they have synergistic effects. In many other cases, higher physical fidelity simulators produce little to no quantifiable benefit in training over lower physical fidelity simulators^38-39.

Skill transfer from the simulator to real scenarios should be the pinnacle of all training simulator development goals. Skill transfer is much more closely tied to psychological fidelity than to physical fidelity. A simulator's psychological fidelity with a task is more difficult to assess than its physical fidelity. The designer must distill the essential task elements that must be supported in the simulated environment. If the correct elements are distilled, training on a simulator will transfer to the real task environment.

Each simulator presumes that some perceptual cues are important and others may be ignored. The designer's insightful selection of the correct cues is tightly coupled to the success of the training transfer to the real environment and to the cost and complexity of the simulator. A simulator in orthopaedics must make simple, explicit assumptions about what is important in the task it represents: the relative position of the hand, tool and bone; the obfuscation of the direct view of the bone; the accurate haptic sensation of the task; and the presentation of the basic bone and tool geometry in the simulated environment.

Simulation in the Training and Assessment of Orthopaedic Surgeons

Orthopaedic surgery has lagged behind other surgical disciplines in developing and incorporating simulation of surgical skills into education and assessment paradigms, particularly in comparison with laparoscopic surgery, a leader in the simulation field^40-47. Previous simulation in orthopaedics has mostly emphasized learning anatomy and surgical approaches on cadavers, and placing products supplied by the medical device industry on naked surrogate bone specimens. Models for surgical skills training outside the operating room have been implemented in a few orthopaedic sub-specialties, primarily arthroscopy⁴⁸ and hand surgery⁴⁹. Thus, it is difficult to build on previously established simulators that demonstrate which perceptual cues are essential to orthopaedic surgery.

A 2010 review of virtual reality simulators for orthopaedic surgery⁵⁰ found only 23 articles that dealt with specific simulators, compared with 246 citations for laparoscopic simulators. Most of the other recent contributions to orthopaedic simulation technology have emphasized haptic feedback. Several such systems have been built to simulate drilling^{27, 29}, and to simulate the forces in amputation surgery⁵¹. Other simulators emphasizing 3D graphics over haptics have enjoyed some success. Blyth and co-workers have developed a highly idealized virtual reality training system (Fig. 3) for basic hip fracture fixation, including a surgical simulator and an assessment component^52-53.

Figure 3. (top) Schematic of images seen by trainee during procedure showing intercept with center of femoral head. (bottom) The implant as viewed after the procedure highlighting 3D placement of screw. From Blyth, Stott and Anderson⁵², used with permission.

Unfortunately, few of these orthopaedic simulators have yet been rigorously evaluated. In a recent literature review⁵⁴, Schaefer et al. identified three critical evaluative aspects of simulation in healthcare education: (1) the validity and reliability of the simulator; (2) the quality of the performance measures used to assess learning outcomes; and (3) the level of inquiry or strength of educational study design to support the evolution of theoretical perspectives. Of 4189 simulation articles reviewed, only 221 qualified as having adequate study design, only 51 meaningfully evaluated their simulators, and only 39 documented significant translational outcomes. Less than 30% examined their simulator's validity, typically assessing only face and content validity, the two most subjective validation methods.

Because simulation is increasingly being contemplated for use in high-stakes evaluation-related licensure and certification, and for identifying mechanisms to mitigate error and decrease mortality and morbidity, it is essential that data supporting the validity and reliability of simulators be vigorously pursued. When a performance assessment is conducted with a simulator, the validity and reliability of the assessment can be evaluated independently from that of the training. This allows the scoring procedure to be separated from the simulator designs, and therefore increases the generality of the work. Performance assessment validity evaluations are made with respect to the performance characteristics of the trainee, rather than with respect to the simulator. For example, simulator researchers typically assess the construct validity of a performance assessment by determining whether or not the assessment measures a performance difference between experts and novices.

Program directors and orthopaedic faculty assess orthopaedic residents using methods mandated by the Accreditation Council for Graduate Medical Education (ACGME). For procedural skills the ACGME requires residents to document surgical experience using a web- based case log system. The Residency Review Committee in orthopaedic surgery uses these case logs to assess the overall experience of residents in terms of the numbers of procedures performed. However, case logs consist only of resident self-reported participation in cases and do not specifically assess the demonstration of actual skill or competency.

Orthopaedic residents are but one group whose competency in this area needs to be assessed. The most important and widely accepted method to assess competence of orthopaedic surgeons in practice is through Board Certification. The American Board of Orthopaedic Surgery (ABOS) mandates recertification every ten years, and it has recently adopted an evolving program of more continuous maintenance of certification. This ongoing process of board certification assesses orthopaedists' competency in medical knowledge, professionalism, communication, and judgment. However, the assessment of a candidate's surgical skill is indirect, relying on radiographic review and case discussion rather than a direct measure of procedural skill.

High-fidelity simulations may be too expensive for implementation within orthopaedic surgery certification, but many of the foundational skills could be economically implemented as part of an objective structured clinical examination⁵⁵. For example, once the relationship between foundational skills and performance in high- fidelity environments is established, it becomes possible to test such skills as part of competency-based assessment. Moreover, to the degree that there is an aptitude aspect to skills, skills-testing may also be an important tool for resident applicant selection.

Education in general surgery has adopted surgical simulation into a robust curriculum that has been widely adopted into general surgery Graduate Medical Education programs^55-64. Residents progress through three phases that are divided into a series of modules. The Residency Review Committee in general surgery highly recommends that this curriculum be adopted by general surgery training programs. The American Board of Surgery requires passing the Fundamentals of Laparoscopic Surgery program^65-66 (Fig. 4) as a basic skills competency for their certification and recertification programs in general surgery. There is no fundamental barrier preventing orthopaedics from adopting similar practices, except for the lack of validated surgical simulation training and assessment approaches.

Figure 4. An example of a commercial product featuring a Fundamentals of Laparoscopic training simulator. Figure courtesy of VTI Medical™.

Recommendations for Future Research

Simulation has a great potential to facilitate faster and more effective learning in orthopaedic surgery. In order to capture the benefits of this training, it is important that researchers and practitioners organize their efforts to gain the maximum advantage in the shortest time with the least cost. This summary of research reveals a number of opportunities for growth in the future of orthopaedic surgery simulation.

Residents need opportunities for regular, deliberate practice. Simulators should be designed so that residents can exercise their surgical skills at regular intervals, with clear feedback. Some skills may require dozens of repetitions while others require hundreds, but one or two repetitions with cadaver parts or plastic models is likely insufficient for complex skills requiring the development of specific psycho-motor behaviors.

Training opportunities for many surgical skills should be repeated in simulation until many of the underlying skills are at a level of automaticity for the residents. In fact, if an effective and valid simulator can be developed for a surgical skill at a reasonable cost, that skill should be simulated.

Orthopaedic surgery simulation should extend beyond the needs of first-year residents to later years of training. The performance of general practitioners in the area of orthopaedic surgery is likely to plateau or even diminish over time without opportunities for deliberate practice^67-68. Simulators should be developed to facilitate the assessment and improvement of their skills.

Research should provide a greater emphasis on validity studies that go beyond establishing simple face validity. They should specify and, ideally, benchmark the performance of simulator elements designed to replicate critical tasks and they should design performance measures that can be tied back to the task model.

Research is also needed to further understand what specific skills are needed for orthopaedic surgery, how they are currently trained, and how effective the current approaches are. Work on new simulators should specifically describe the training and assessment procedure that is used with the simulator.

Finally, we would caution researchers to focus their simulator development on key task elements and psychological fidelity rather than focusing on the realism of the simulation.