A Vision for Using Simulation & Virtual Coaching to Improve the Community Practice of Orthopedic Trauma Surgery

Abstract

Background:

Many orthopedic surgeries involve the challenging integration of fluoroscopic image interpretation with skillful tool manipulation to enable procedures to be performed through less invasive approaches. Simulation has proved beneficial for teaching and improving these skills for residents, but similar benefits have not yet been realized for practicing orthopedic surgeons. A vision is presented to elevate community orthopedic practice and improve patient safety by advancing the use of simulators for training and assessing surgical skills.

Methods:

Key elements of this vision that are established include 1) methods for the objective and rigorous assessment of the performance of practicing surgeons now exist, 2) simulators are sufficiently mature and sophisticated that practicing surgeons will use them, and 3) practicing surgeons can improve their performance with appropriate feedback and coaching.

Results:

Data presented indicate that surgical performance can be adequately and comparably measured using structured observations made by experts or non-expert crowds, with the crowdsourcing approach being more expedient and less expensive. Rigorous measures of the surgical result and intermediate objectives obtained semi-automatically from intra-operative fluoroscopic image sequences can distinguish performances of experts from novices. Experience suggests that practicing orthopedic surgeons are open to and can be constructively engaged by a family of mature simulators as a means to evaluate and improve their surgical skills.

Conclusions:

The results presented support our contention that new objective assessment measures are sufficient for evaluating the performance of working surgeons. The novel class of orthopedic surgical simulators available were tested and approved by practicing physicians. There exists a clear opportunity to combine purpose-designed simulator exercises with virtual coaching to help practicing physicians retain, retrain, and improve their technical skills. This will ultimately reduce cost, increase the quality of care, and decrease complication rates.

Clinical Relevance:

This vision articulates a means to boost the confidence of practitioners and ease their anxiety so that they perform impactful procedures more often in community hospitals, which promises to improve treatment and reduce the cost of care while keeping patients closer to their homes and families.

Introduction

The manner in which orthopedic surgical skills are trained and assessed is rapidly changing. New training methods, higher quality simulators, public health concerns, and a mandate from certifying bodies have together catalyzed the structured teaching of surgical skills outside of the operating room (OR) across surgical specialties.^1-3 Unfortunately, the ubiquitous Halstedian model of training is stymied by limited resources for the evaluation of technical competency. Worse, existing evaluation methods may be flawed; suffering from bias, impractical for high-volume use, and of unverified accuracy. Programs routinely credential surgeons for procedures they have never done.⁴ Self-reported caseloads are inconsistent across institutions, and caseload or academic rank provide imperfect measures of true technical skill.^5,6

Compared to residency training, the adoption of simulation in the larger community of practicing surgeons has lagged behind. This is an important missed opportunity. Surgeons with lower technical skills demonstrate a 5x higher mortality rate, 3x higher complication rate, and 29% longer operative time compared to top-quartile technical skills performers.⁷ Indeed, with over 135,000 surgeons performing 51,000,000 procedures in the U.S. each year,⁸ identifying surgeons at high risk for poor patient outcomes and providing them with targeted coaching is an acute, high-priority need.

Community practitioners could benefit from targeted training. In orthopedics, common fracture surgeries, such as the operative reduction of articular fractures and pinning of pediatric elbow fractures, are too often referred from community practitioners to tertiary care centers or treated using out-of-date techniques.^9,10 In fact, the skill required to pin a pediatric elbow fracture is among those with the lowest self-reported competence among orthopedic surgeons, despite elbow fractures accounting for approximately 8% of all pediatric fractures.¹¹ This presents an opportunity for simulation and coaching to improve the community practice of orthopedic trauma surgery. At the nexus of this opportunity is the skillful use of fluoroscopy, which enables surgical procedures to be performed through less invasive approaches. Less invasive approaches reduce the risk of wound complications, lead to quicker recovery times, and improve patient comfort, all while ensuring comparable results, at least for experts.¹²

Unfortunately, not all orthopedic surgeons are adept with the skills that enable these less invasive approaches, and others lack the confidence to utilize them. Two target surgeries are emblematic: articular fractures of the distal tibia and pediatric supracondylar humerus fractures. Complex articular fractures require precise reduction to achieve optimal outcomes. These surgeries have historically been done through extensile open approaches that risk further trauma to compromised soft tissues. More limited approaches are associated with fewer complications, but they rely heavily on fluoroscopic guidance and skillful reduction maneuvers. Similarly, when treating a pediatric supracondylar humerus fracture, a surgeon aims to restore pre-injury position and alignment. Then K-wires are placed across the fracture site to help maintain alignment until the fracture heals, while protecting the growth plate in these young children. Supracondylar humerus fractures are the most common pediatric fracture treated surgically worldwide.¹³ Their treatment requires skill that takes time and concerted effort to master.¹⁴ Many community surgeons do not feel competent to perform this surgery.¹⁵ This leads to referrals that delay care, and it also costs more to be treated at a tertiary care facility than at a community hospital.¹⁶

We propose that the coupling of simulation with virtual coaching provides a unique opportunity to facilitate retaining, retraining, and improving the technical skills of practicing surgeons, which can reduce the cost of care, increase its quality, and decrease complications. In the following we present the rationale and arguments for the potential of this approach and a path forward. This vision to improve the community practice of orthopedic trauma surgery builds upon three recent fundamental advancements. First, unique and rigorous analysis approaches for modeling and assessing surgical skill based upon fluoroscopic sequences and/or video collected in the OR have been developed.^17,18 Second, simulators that incorporate technologies mixing physical realism with synthetic fluoroscopy images^19-21 have been shown to be effective,²² and they capture exciting and attractive technical targets for practicing surgeons. Finally, the simulators and assessment procedures facilitate virtual coaching that leverages scientifically-supported formative and summative feedback approaches imbued with a deep understanding of successful surgical behaviors.

Methods

Assessment Techniques are Ready to Measure the Skill Level of Practicing Surgeons

Most technical skills assessment methods employ video-based evaluation by expert surgeons utilizing survey instruments based on the Objective Structured Assessment of Technical Skills (OSATS).²³ In bariatric surgery, OSATS scores have been shown to correlate with patient outcomes,⁷ but this correlation has not been well studied in orthopedics. Structured survey instruments such as OSATS²³ are subject to confirmation bias in expert ratings, and OSATS scores fail to correlate with surgical results measured in a laboratory setting.^24,25

A key challenge to establishing simulator skill transfer to the OR is finding a way to objectively measure OR performance. We have pioneered a technique analyzing surgical behavior from fluroscopic image sequences. Our experience has shown that important insights can be extracted from a surgical fluoroscopic image sequence,^18,20,26 including details of surgical strategy, surgical skill, and the quantitative assessment of the final implant placement. One challenge in obtaining these insights involves collecting detailed measurements from as many as 150 images in a sequence for a long or complex surgery.

To address this challenge, we developed a semi-autonomous process involving artificial intelligence and customized image analysis software. The software speeds the sequence analysis so that dozens of surgical images can be analyzed by a non-surgeon in minutes, rather than days. The original implementation was tailored toward analyzing fluoroscopic sequences of a guide wire being placed during intertrochanteric hip fracture reduction and fixation. The analysis provides the wire entry-point, the actual wire trajectory versus that desired, the duration of the surgical step, the number of images collected, and the tip-apex distance (TAD). A large TAD is a strong predictor of later implant failure.^27,28 For this reason, a central surgical objective is to minimize the TAD, but it can come at the cost of substantial fluoroscopic radiation and additional operative time when skills are poorly developed. The details extracted from these OR images has proven useful in distinguishing expert from novice performance.

A more recent adaptation of the software analyzes the pinning of pediatric supracondylar humerus fractures (Figure 1).²⁹ We are extending the software to analyze the reduction and fixation of distal tibia fractures, as well. This semi-automated analysis approach has yielded powerful quantitative, repeatable insight into surgeon behavior that allows us to assess skill in new ways. The technique is also uniquely suited to compare performance in simulation and in the OR. Finally, the technique provides insights that lead directly to timely formative feedback, because analysis at the decision-making level captures each step of a surgical procedure.

Figure 1.

Automatically detected K-wires during the pinning of a pediatric supracondylar humerus fracture. Calculated performance metrics include the angles of the wires relative to the fracture plane, the spacing of the wires at the fracture plane, and the wire spread ratio.

Another problem with OSATS video evaluation methods is that they are impractical for widespread use, largely due to the excessive time burden demanded of experts or faculty raters. This motivated the pioneering use of “crowds” of non-expert raters to provide inexpensive evaluations that proved statistically concordant with expert ratings,³⁰ obtaining thousands of ratings at unprecedented speeds.^31,32 This method has come to be called crowd-sourced assessment of technical skill (CSATS). Unlike expert raters, ratings aggregated from non-expert crowds are fast (available within hours of video upload), scale to practically unlimited numbers of ratings, and are available for repeated ratings around the clock at near-minimum wage. Further, CSATS ratings prove statistically favorable compared to typical panels of experts or faculty and correlate well to patient outcomes.³³

Simulator Technologies Are Ready for Practicing Surgeons

The key to developing an effective simulator is understanding and defining specific educational objectives and the manner in which performance is measured. Properly used, simulators can be more effective than traditional training approaches.³⁴ They are also useful for identifying skill performance issues.³⁵ Thus, simulation makes an attractive target for assessing, training, and retraining the skills of practicing surgeons.

Our group has developed and tested simulators for navigating surgical wires using fluoroscopic guidance and for reducing fractures through limited open incisions, two common and difficult skills in orthopedic surgery. These simulators feature replaceable, plastic foam bones and synthetic fluoroscopic imagery to help surgeons recognize and interpret key image features, connecting their hand movements to surgical goals. Our wire navigation orthopedic surgical simulation platform replicates the look and feel of navigating a wire through bone using fluoroscopic guidance (Figure 2).¹⁹ Two differentiating features of the platform are: (1) camera-based tracking of the wire replaces fluoroscopic radiation exposure, and (2) the foam bone surrogate replicates the feel of drilling through actual bone.

Figure 2.

The orthopedic surgical simulation platform relies upon camera-based tracking of a laser-etched stainless steel K-wire, otherwise identical to that used in surgery. In the case of the hip wire navigation simulation shown, a foam bone surrogate is housed within a soft tissue mimicking sleeve to obscure the trainee’s view of the object.

We initially developed this platform to simulate the navigation of a wire in the treatment of intertrochanteric hip fractures. After several years of development and testing, this hip fracture wire navigation simulator^19,25 is currently under study by the American Board of Orthopaedic Surgery (ABOS) and is used twice annually as part of resident fracture courses offered by the Orthopaedic Trauma Association (OTA). Seeking to get more data from advanced users, we tested at the OTA Trauma Fellows course in 2019, observing the performance of 30 fellows near the pinnacle of their surgical training in orthopedic trauma. We have demonstrated that the platform is extensible to other relevant orthopedic applications,²⁰ including the pinning of a pediatric supracondylar humerus fracture (Figure 3).³⁶

Figure 3.

The pediatric supracondylar humerus fracture simulator provides residents with the opportunity to place three wires across a fracture. The position of the wires on the plastic bone (left) are represented in synthetic fluoroscopic images rendered by the simulator (right). Performance is measured by the spread of the wires across the fracture line, the time taken in placing the wires, and the number of fluoroscopic images obtained along the way.

Our team has also recently developed an articular fracture reduction simulator to help train more limited surgical approaches that are associated with faster recovery and fewer complications (Figure 4).^21,37 This is accomplished through an electromagnetic system that provides real-time tracking of surgical tools and artificial fracture fragments buried in a soft tissue model. The system provides quantitative feedback on 17 different performance metrics including novel features, such as a summative snapshot of fracture displacement over time.

Figure 4.

(a) Articular fracture reduction simulator. (b) The articular fragment is tethered to resist reduction to mimic the clinical setting. (c) Representation of tenaculum holding reduction as a K-wire is inserted as fixation.

Although each of these simulators was initially developed for training residents, we have begun to use them to assess experienced surgeons, typically to provide a comparative performance assessment or training goal. These experiences with practicing surgeons suggest that the simulators are capable of distinguishing skill levels among practicing surgeons and that surgeons find the experience challenging and enjoyable, as described in the results section.

Simulator Training is Likely to Improve Performance

Constructive feedback strategies are essential for effective skill training and performance improvement,^38,39 but different types of feedback are more or less effective at different stages in skill mastery. Formative feedback is information provided to the learner during an educational exercise to modify behavior and improve learning.⁴⁰ In general, timely detailed formative feedback is most effective for early training stages, and delayed, summative feedback is more effective for later retaining, retraining, and improving performance. Experienced surgeons may find immediate, detailed, formative feedback frustrating. For instance, the formative feedback currently provided with the wire navigation simulator includes recommendations such as “drop your hand” or “shift the wire angle 5° distally.” This level of detail, effective for novice residents, projects a presumed procedural approach for which an experienced surgeon may have developed their own effective and appropriate alternative strategies.

Summative feedback, often more effective for advanced learners, provides information about the results of an entire procedure that is generally intended to provide the basis of comparison with peers or an external standard.⁴¹ Our articular fracture reduction simulator currently emphasizes summative feedback on the duration and the precision of the fracture reduction, for example. Summative feedback is typically provided upon completion of the procedure, but intermediate extrapolation can be used to project some elements of performance, like reduction quality. This type of feedback can be useful for someone who is fine-tuning details of their approach as they integrate the tradeoffs between different steps of a procedure. However, if the learner is unclear on appropriate strategies to achieve particular intermediate goals, summative feedback can be less effective than formative feedback.

There are many ways to provide feedback and not all are equally helpful. For example, providing an explanation for why an answer was wrong is 50% more effective than providing the right answer and 10 times more effective than saying that a given answer was incorrect.⁴² The Interactive Tutor Model is one approach to understanding the diversity of factors that affect feedback design in order to produce effective feedback.⁴³ This model depends on a close match between the learner and the instructors’ understanding of the objectives and intermediate goals. With a loosely defined surgical task, different surgeons may approach the task with different strategies, all equally acceptable. If the feedback presumes one goal when the surgeon is appropriately choosing another, it may decrease the surgeon’s trust in the automated feedback, weakening its effectiveness.⁴⁴

Results

The following sections summarize data from various experiments that we have conducted over the past decade supporting the three foundational arguments that underlie the vision of expanding simulation training to practicing surgeons.

Assessment Techniques Can Measure Transfer of Training

The predominant assessment techniques have important flaws. Surgical assessment approaches have assumed that faculty ratings are the most reliable measure of performance and identity-blind video ratings are the “gold-standard” for assessment. But mounting evidence suggests that both approaches are flawed. Experts appear to reward speed over technical accuracy.⁶ Preliminary results suggest that expert raters’ expectations of a surgeon’s technical skill based on expertise/reputation do not agree with their own scores assigned based upon anonymized video.^45,46 When two faculty coders reviewed 56 anonymized videos of benchtop laparoscopic exam tasks, they identified 30 as below average competency or non-competent. They were not told that these were videos of surgeons whom they had just nominated as “experts” by reputation or academic rank (Figure 5a).^46,47

Figure 5.

(a) A panel of two faculty surgeons disqualified over 50% of videos from surgeons reputed to be experts when blinded to their identity. [Figure reprinted with permission from.⁴⁵] (b) Concordance between experts and crowds for three different studies. [Figure reprinted with permission from.⁴⁸] (c) On average, expert faculty spent 6 minutes reviewing a 13-minute video, while non-expert crowd workers from Amazon Mechanical Turk CSATS spent 16 minutes reviewing the video³³.

One lesson we learned from this experience is how onerous it can be to have faculty surgeons grade 56 videos. This motivated seminal work in 2011 to test whether ratings from non-expert crowds are concordant with expert ratings. It was found that when sufficient non-expert crowd workers (~35) from Amazon Mechanical Turk used a survey to assess video, their aggregate scores agreed remarkably well with expert ratings (Figure 5b). These results were confirmed for laparoscopy,³² robotic surgery,^30,49 animate training,⁵⁰ and real patients.³³ The crowd-sourced ratings even correlated well with patient outcomes.^33,51 The crowd-sourcing approach provided 16,418 ratings for 430 videos at 1.3 ratings/min for under $5000.⁵² Finally, deeper inspection revealed that expert raters do not watch full videos (Figure 5c), and in some cases, non-expert ratings appear to predict surgical outcomes better than experts. For these reasons, the CSATS method enjoys increasing adoption in medical skills assessment research.^{30,31,49,50,53-56} The crowd-sourcing approach provides an inexpensive and expeditious method for analyzing surgical performance both with simulators and in the OR. Another objective assessment method links performance to specific surgical behaviors.

We evaluated the transfer of training on the wire navigation simulator in both a mock OR²² and in the actual OR (Figure 6).^17,18 This allowed us to begin to differentiate performance improvements caused by simulator training from those that accrue with experience. We define consistent measures that are common to both the simulator and the actual OR. For example, residents were assessed in the mock OR based on their use of fluoroscopy, total time, and TAD. Residents that trained on the simulator had a lower TAD than those who did not (p = 0.001), and their performance on the simulator (i.e., TAD, image use, and overall time) was correlated with performance in the mock OR.²² This assessment approach was clearly sufficient to measure the performance gain achieved by residents with some focused training, but as the next section will demonstrate, it was also sufficient to distinguish different cohorts of experts.

Figure 6.

The current orthopedic wire navigation surgical simulator was validated by measuring later performance in a mock OR (left). Performance measures were defined from information that can also be collected in the actual OR (right) to test skill transfer.

Simulators are Ready for Practicing Surgeons

In addition to the numerous tests with first- and second-year residents, in November 2019 we attended the Hennepin Orthopaedic and Trauma Seminar in Minneapolis to gather data on the performance of community orthopedic surgeons. After orientation to the simulator, 17 community surgeons completed three assessments with the hip fracture wire navigation. We compared the practicing surgeons’ performance with that of a cohort of residents we previously studied.²² Figure 7 illustrates normalized composite performance scores on the wire navigation. In this admittedly limited sample of community surgeons, we found their performance to be on average better than the residents, but there was a broad distribution of performance amongst both residents and community surgeons. Subjective rating questionnaires and informal interviews with the practicing surgeons confirmed that they felt the simulation was interesting, sufficiently realistic and challenging for practicing surgeons.

Figure 7.

A plot of the normalized composite scores of two cohorts of surgeons placing a hip wire with the wire navigation simulator. The data suggest that most community surgeons perform better than most residents, but that there are outliers in both groups that might benefit from additional training.

Results with the fracture reduction simulation promise even greater interest among practicing surgeons. Figure 8 presents performance results comparing an expert and a novice reducing a distal tibia fracture on the fracture reduction simulation. Although both surgeons achieved an acceptable final reduction (<2mm),⁵⁷ the unique in-the-moment summative feedback provided by the simulator (Figure 8a) clearly demonstrates how the novice struggled in achieving their result, unnecessarily manipulating the fragment by moving it cumulatively 158 mm more than the expert. This line of inquiry also indicates that experienced surgeons utilize tools more efficiently during fracture reduction (Figure 8b).³⁷ The novice surgeon lacked an understanding of proper tool placement to assist in the reduction, instead relying on an iterative method where they continuously attempted to replace their tool in equally ineffective locations, in all contacting the bone over a 517% larger area than the expert. As experts have attested when using the simulator, the unique, performance-specific feedback from the simulator is particularly helpful, because it provides viewpoints and measures that cannot be replicated in the OR environment. This provides practicing surgeons with the opportunity to reflect on their performance with a new perspective, offering new insights.

Figure 8.

(a) Fracture reduction accuracy (step-off) over time (seconds). Vertical red lines indicate the reduction period of a mock surgery. Compared to an expert (left), the novice (right) showed a large amount of fragment manipulation. Note the time scaling is compressed in the novice performance to fit. (b) Heat maps of tool and fragment interaction points. Blue = no interaction, yellow = an interaction lasting longer than 1.5 seconds. The tool-fragment interaction of the expert (left) differed notably from that of a novice (right).

Training is Likely to Improve the Skills of Practicing Surgeons

In the course of our work defining performance assessment metrics, we found limitations in the use of simple “final result” measures (e.g., the TAD) that do not account for in-the-moment decision-making that is a hallmark of performance. We came to recognize that it is precisely during these moments of what we have come to call micro-decisions, that there is a unique opportunity to teach. In response, we surveyed performance as reflected by the series of fluoroscopic images, and defined a short list of five errors we routinely observed, such as when a novice adjusts the wire in the wrong direction, or over compensates for an alignment error, or switches from the antero-posterior to the lateral view inappropriately. These clearly indicate errors in judgment that can be readily detected, and each error has an unambiguous corrective behavior that can be taught.

For the application in the hip, we have defined a composite score that includes the TAD metric, the number of decision errors made, and the average angle of the wire movement errors. Each metric was first normalized based on the overall population. A higher composite score indicates better performance. A score that is equal to zero would indicate that it matches perfectly with the average performance across all subjects. Our studies with residents have supported the idea that relevant surgical experience correlates with performance on the simulator. This shows the value of considering the behavior at the micro-decision level and also strengthens our ability to distinguish expert from novice performance.

The advantage of this micro-decision analysis framework, as well as the other objective performance measures, is that they provide benchmarks against which a specific performance can be assessed. This benchmarking strategy allows us to provide specific, timely feedback to surgeons as they work or immediately afterwards. Subjective ratings by experts and crowd-sourced assessments do not always provide such direct feedback. When timely, specific feedback is provided, it naturally results in improved performance.

Discussion

The results presented support our contention that the current assessment methods are sufficient for measuring the performance of working surgeons. We demonstrated the advantages of two approaches: wire position analysis and crowd sourcing. Figure 7 demonstrates that the simulator assessment approach can distinguish skill levels between a cohort of residents and practicing surgeons. It is important to remember that reliably measuring the performance of a single individual is more difficult, however, than measuring group characteristics. We did not yet demonstrate how much testing is required to quantify an individual as having sufficient skill. Measuring performance with repeated exercises will enable such an assessment. Providing a variety of exercises will make the activity more enjoyable and the measurement more robust. Overcoming the vagaries of bias and the challenge of noisy measurements was an essential obstacle for objectively measuring performance level and performance gains, which are both essential for training practicing surgeons.

The novel class of orthopedic surgical simulators we have developed were tested and approved by practicing physicians. The successful adaptation of the hip wire navigation simulator to pediatric elbow and illiosacral screws indicates that the simulators can be adapted as needed to create training environments that are clinically relevant and intellectually demanding for practicing physicians. Novel summative performance assessment provided by the fracture reduction simulator promises to open new opportunities for training and assessment that is not possible in the OR nor with typical cadaver exercises. The informal feedback and enthusiasm of practicing physicians who work with the simulators suggests that the simulator technology is ready for the next step.

Finally, the micro-decision analysis and quantitative analysis approaches to feedback generation make excellent input for a virtual coach. The micro-decision analysis can provide important observations of specific moments that a practicing surgeon could use to further refine his or her performance. The analytic analysis of the whole performance provides an opportunity for practicing surgeons to gain insights into their performance that go beyond the more rigidly defined decision analysis. The feedback is ready for practicing surgeons.

Our research approach leverages existing relationships with a regional consortium of residency programs, the ABOS, and the OTA. We have begun to expand our reach to include statewide Orthopedic Societies in Iowa and Minnesota, as well as an annual orthopedic and trauma seminar run in Minneapolis. This will allow us to extend our experience with recruiting and training hundreds of residents to the relatively new challenge of working with community practitioners. The work promises to extend our insight into how expertise is manifested and can be measured in the OR, how that can be compared with behaviors observed during simulated surgeries, what type of feedback is effective in simulation, and how this knowledge can be integrated into a virtual coach to make surgery more effective.

Efforts are already underway to refine our existing simulators so that they effectively allow working surgeons to practice their skills in a manner that will enable them to retain, retrain, or improve their performance. More generally, the simulators will improve skill in the broader area of navigating surgical wires with fluoroscopic guidance. The work will permit surgeons to rehearse these technical skills before further exercising them on live patients in the high-stakes atmosphere of the OR. The methods derived will boost the confidence of practitioners and ease their anxiety with performing these procedures in community hospitals, which will improve treatment and reduce the cost of care while keeping patients closer to their homes and families. We expect to demonstrate the clinical benefits of the surgical simulator and virtual coaching by increasing the number of procedures done by practitioners in their community hospital settings.