A Simulation Trainer for Complex Articular Fracture Surgery

Abstract

Background:

The purposes of this study were (1) to develop a physical model to improve articular fracture reduction skills, (2) to develop objective assessment methods to evaluate these skills, and (3) to assess the construct validity of the simulation.

Methods:

A surgical simulation was staged utilizing surrogate tibial plafond fractures. Multiple three-segment radio-opacified polyurethane foam fracture models were produced from the same mold, ensuring uniform surgical complexity between trials. Using fluoroscopic guidance, five senior and seven junior orthopaedic residents reduced the fracture through a limited anterior window. The residents were assessed on the basis of time to completion, hand movements (tracked with use of a motion capture system), and quality of the obtained reduction.

Results:

All but three of the residents successfully reduced and fixed the fracture fragments (one senior resident and two junior residents completed the reduction but were unsuccessful in fixating all fragments). Senior residents had an average time to completion of 13.43 minutes, an average gross articular step-off of 3.00 mm, discrete hand motions of 540 actions, and a cumulative hand motion distance of 79 m. Junior residents had an average time to completion of 14.75 minutes, an average gross articular step-off of 3.09 mm, discrete hand motions of 511 actions, and a cumulative hand motion distance of 390 m.

Conclusions:

The large difference in cumulative hand motion distance, despite comparable numbers of discrete hand motion events, indicates that senior residents were more precise in their hand motions. The present experiment establishes the basic construct validity of the simulation trainer. Further studies are required to demonstrate that this laboratory-based model for articular fracture reduction training, along with an objective assessment of performance, can be used to improve resident surgical skills.

Reduction and internal fixation of articular fractures is a basic skills competency for orthopaedic surgeons. Most orthopaedic residents acquire these skills through an apprenticeship model, training on patients in the operating room under close supervision. With current Accreditation Council for Graduate Medical Education (ACGME) work hour restrictions, the quantity of surgical case exposures to acquire these skills may become more limited in the future for residents. Surgical skills training models outside of the operating room have been implemented to a limited extent in other areas of orthopaedics, such as hand surgery¹ and arthroscopy². Methods to acquire and practice orthopaedic surgical skills outside of the operating room, combined with validated objective assessment techniques for formative and summative resident feedback, are needed³.

This need is especially strong in areas such as articular fracture reduction. Residents must understand the three-dimensional aspects of a fracture pattern and correlate this understanding with two-dimensional fluoroscopy views, a skill that is integral to fracture reduction techniques. This understanding of fracture patterns must be integrated with a surgeon’s hand-eye coordination to achieve reduction and fixation, all while minimizing soft-tissue trauma. Therefore, creating a simulation in which this difficult surgical skill can be acquired and objectively assessed in a safe, controlled, and reproducible environment would be an important advance in orthopaedic resident training.

There were three purposes of this study: (1) to develop a physical model to improve resident skills in articular fracture reduction, (2) to objectively assess residents while they were performing this specific surgical skill, and (3) to assess the construct validity of the model and the assessment techniques by comparing the results of junior residents with those of senior residents.

Materials and Methods

The Model

To create the fracture pattern that was used in the simulation, a human distal tibia replica was machined from high-density (640 kg/m³) polyurethane foam (Grade FR7140 LAST-A-FOAM; General Plastics, Tacoma, Washington). This material was selected because it exhibits structural and fragmentation behavior comparable with that of human bone, and when suitably doped with barium sulfate, it displays similar radiographic and computed tomographic (CT) appearance⁴. An innovative rapid machining technology⁵ was used to recreate the distal tibia geometry, working off models generated from CT segmentations of intact anatomy. A 7.5-kg mass was dropped from a height of 0.5 m in a drop tower to fracture the tibia replica using a talus surrogate impactor (cast from polymethylmethacrylate [PMMA]) to deliver the impact⁶. A silicone rubber mold (TinSil 70 Series RTV; U.S. Composites, West Palm Beach, Florida) of the fracture fragments was negative-cast and was then used to positive-cast identical replica fragments. The fragments were molded with use of 256.3 kg/m³ (16-lb/ft³) density polyurethane foam that was doped with barium sulfate (15% by weight) to mimic the radiographic appearance of bone. To imitate the denser subchondral plate’s radiographic appearance, the articular surface facets of the fragments were coated with a mixture of glue and barium sulfate (40% by weight).

To verify the accuracy of the casting process, the original and replicated fragments were laser-scanned (NextEngine Model 2020i; NextEngine, Santa Monica, California); the average error between castings was <0.2 mm. Original fracture fragments were positioned in a moderately displaced configuration by an experienced orthopaedic surgeon (J.L.M.) (Fig. 1), and were lightly held together with very low-strength glue that was applied just at interfragmentary point contacts. A PMMA impression made of the displaced configuration’s articular surface served as a template for positioning identical sets of casted fragments in identical configurations. Laser scans of the original and casted constructs verified that configurations were replicated with submillimeter accuracy. With use of this novel approach, identical sets of fragments could be held in identical positions of displacement, yet were immediately separable upon manipulation. The repeatability of this process allowed for providing a large number of trainees with identical fracture reduction challenges.

Fig. 1.

Photographs and illustrations showing the creation of the tibial plafond fracture surrogate model. The surrogate fracture fragments were placed in a moderately displaced configuration by an experienced surgeon, and a PMMA template was created to enable identical replication of the configuration. A three-dimensional (3D) laser scan was used to establish submillimeter accuracy in placement, prior to placement within a soft tissue-mimicking sleeve.

To simulate the presence of the soft-tissue and surrounding bone structures, an anatomic replica ankle and foot model (Model 1518; Sawbones, Vashon, Washington) that conforms to the normal anatomy of the lower leg and that mimics the bulk soft-tissue properties was used (Fig. 2). (This model did not include individual anatomic structures such as the tendons, ligaments, muscles, or neurovascular structures.) The soft-tissue features of the as-received Model 1518 lower-leg surrogate were preserved, but a fractured surrogate tibia was substituted for the intact tibia. This substitution was done posteriorly, so as to not influence the anterior surgical access to the fracture site. The surgical access site for each pattern was restricted to 5 cm in length, located anteriorly, adhering to a limited approach philosophy. The opening was created prior to the simulation, with its ends marked by colored tape to prevent study participants from lengthening the incision.

Fig. 2.

Photographs showing the fracture surrogates placed within a soft tissue-mimicking housing, shown on the left from an external view and on the right from a posterior view through the opening used for placement of the surrogate fracture model.

Subjects and the Simulation

Five senior (fourth or fifth year) and seven junior (first or second year) orthopaedic surgery residents participated in the study. The group of study participants was a convenience sample of residents readily available and willing to participate at a single institution. The experience level of the junior residents was very limited, with an average of less than thirty trauma cases for each resident. Prior to beginning the exercise, the residents were given verbal instructions describing the simulation and the ultimate goal of the exercise. No pretest or other skills assessment was performed prior to the simulation. The residents were provided radiographs of the fracture, as well as three-dimensional CT reconstructions to assess prior to the simulation. These imaging data provided the residents familiarity with the fracture pattern, but a detailed surgical plan was not formulated prior to the simulation, and the exercise was begun within ten minutes of the instructions and image review. The reduction simulation was performed through a limited anterior approach (Fig. 3) with use of standard surgical tools including tenaculua, hemostats, and self-retaining retractors. The reduction achieved was fixed with Kirschner wires. To closely simulate the clinical practice of articular fracture reduction, fluoroscopy was provided and was utilized by the residents during the reduction. The time at which the participant declared the reduction and fixation complete was recorded, with a time limit of thirty minutes for this exercise. The hand motions of the participant were tracked by means of a four-camera Qualisys motion capture system (Oqus 300; Qualisys AB, Gothenburg, Sweden), using four passive fiducial markers attached to the back of each hand (Fig. 4), the motions of which were captured at 100 Hz. The simulations were also recorded with use of video. Lead protective gowns were utilized for all trainees involved in the study to minimize the radiation risk from fluoroscopy. This project was reviewed by the institutional review board and determined to be exempt. Resident participation was voluntary.

Fig. 3.

Photographs and radiographs showing fractured surrogate tibiae reduced percutaneously under fluoroscopy while surgeon hand motions were tracked (see the markers on the back of the trainee’s hands). The motions were synchronized to the video of the simulation task. Kirschner-wire fixation was used to hold the fracture reduction that was obtained.

Fig. 4.

Photograph showing the articular fracture reduction task executed in a surgical skills laboratory with use of intraoperative fluoroscopy. Hand motion of the trainees was tracked with use of fiducial markers placed on the back of the hands. A visual representation of the image capture space shows motion data that were acquired at 100 Hz with use of a Qualisys motion capture system. (Reproduced, with modification, from: Karam MD, Kho JY, Yehyawi TM, Ohrt GT, Thomas GW, Jonard B, Anderson DD, Marsh JL. Application of surgical skill simulation training and assessment in orthopaedic trauma. Iowa Orthopaedic Journal. 2012;32:76-82. Reproduced with permission.)

The analysis of the simulation included first synchronizing the video with the Qualisys motion capture data, which enabled disregarding any motions that were not directly related to the reduction. (Examples included reaching for tools and gestures to direct the fluoroscopy technician.) The total distance of hand motion and the number of discrete hand motions were calculated⁷ and were recorded. The fracture reductions that were obtained were laser-scanned with use of the NextEngine system. The laser scans were then compared with an ideal reduction with use of Geomagic Qualify software (Geomagic, Research Triangle Park, North Carolina), and the total average error and average articular surface error were recorded. The ideal reduction was created by the alignment of the pre-fracture laser scan and the post-fracture fragment laser scans, with use of an iterative closest point algorithm.

Videos of the simulations were then delivered to an experienced orthopaedic trauma surgeon and educator (J.L.M.) for review. A checklist of key steps in the surgical procedure (Table I) was used to document whether or not each step was performed. To prevent observer bias, the videos were cropped so that the face of the participant was not visible, all sound was removed, and the videos were labeled with a coded number. This checklist of key steps was developed by our faculty trauma surgeons including two authors of this study (M.D.K. and J.L.M.). The global rating scale that was used (Table II) is a modified version of a scale previously validated by Reznick et al.⁸, and it consists of multiple dimensions, each related to some aspect of surgical reduction of articular fractures. A modified global rating score included procedure-specific tasks, including use of fluoroscopy and use of Kirschner wires. In addition, overall performance was also rated, giving a total of nine dimensions. Two orthopaedic trauma surgeons (M.D.K. and J.L.M.) verified the content of both the procedural checklist and the global rating score. A pass or fail determination was also made for each trial. The pass or fail determination was subjective, and outcome scores were not compared between the two evaluating surgeons (M.D.K. and J.L.M.).

TABLE I.

Expert Assessment: Procedural Checklist^*

Procedural Step	Performed	Not Performed
Appropriate identification of landmarks and incision
Appropriate selection of surgical instruments
Appropriate handling of surgical instruments
Appropriate preparation and manipulation of fracture fragments
Appropriate placement of reduction devices and aids
Appropriate reduction achieved
Appropriate use of fluoroscopy
Appropriate placement of fixation devices

^*The boxes are marked corresponding to the candidate’s performance during the procedure.

TABLE II.

Expert Assessment: Global Rating Scale^*†

	1	2	3	4	5
Preparation for procedure	Did not organize equipment well. Has to stop procedure frequently to prepare equipment.		Equipment generally organized. Occasionally has to stop and prepare items.		All equipment neatly prepared and ready for use.
	1	2	3	4	5
Respect for tissue	Frequently used unnecessary force on tissue or caused damage.		Careful handling of tissue but occasionally caused inadvertent damage.		Consistently handled tissue appropriately with minimal damage.
	1	2	3	4	5
Time and motion	Many unnecessary moves.		Efficient time and motion, but some unnecessary moves.		Clear economy of movement and maximum efficiency.
	1	2	3	4	5
Instrument handling	Repeatedly makes tentative or awkward moves with instruments.		Competent use of instruments, but occasionally appeared stiff or awkward.		Fluid moves with instruments and no awkwardness.
	1	2	3	4	5
Use of fluoroscopy	Frequently requested excess images. Requested images with repeated improper orientation.		Used images effectively, but occasionally had to ask for extra images because of error.		Successfully completed procedure with minimal images. Used proper orientations during image capture.
	1	2	3	4	5
Use of Kirschner wires	Consistently placed wires in poorly placed positions or used in excess.		Good use of wires with occasional repositioning.		Efficient use of wires with appropriate placement.
	1	2	3	4	5
Flow of procedure	Frequently stopped procedure and seemed unsure of next moves.		Demonstrated some forward planning with reasonable progression of procedure.		Obviously planned course of procedure with effortless flow from one move to the next.
	1	2	3	4	5
Knowledge of procedure	Deficient knowledge.		Knew all important steps.		Demonstrated familiarity with all aspects of the procedure.
	1	2	3	4	5
Overall performance	Very poor		Competent		Clearly superior

^*Reproduced, with modification, from: Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg. 1997 Mar;173(3):226-30. Reproduced with permission.

^†The reviewer name and candidate video number were recorded and a score from 1 to 5 was circled corresponding to the candidate’s performance in each category. The candidate was determined to have passed or failed on the basis of the overall scores.

Data Analysis

The outcomes defined were time to completion, gross articular step-off, cumulative hand travel distance, and the number of discrete hand motions (defined as involving a change of direction). Student t test statistical analysis was used for comparison between senior and junior residents regarding these quantitative outcomes. Significance was set at p < 0.05.

Costs

Each Sawbones lower-leg model cost approximately $170, and each fractured surrogate tibia insert cost an additional $30. Fluoroscopy utilization was $12 per hour at our institution, and each simulation was approximately twenty minutes long (approximately $4) for a given subject. The NextEngine laser scanner and the Qualisys motion analysis system were only used for the purposes of measuring performance to establish construct validity in the present study and would not need to be used in actual future training.

Source of Funding

This research was funded in part by external research grants: the OMeGA Medical Grants Association, the Orthopaedic Trauma Association, and NIH/National Institute of Arthritis and Musculoskeletal Skin Diseases (AR055533 and AR054015). The funds were used to support personnel and to purchase equipment and supplies in support of this study.

Results

There was a wide variation in resident performance (Table III, Fig. 5), generally more so among junior residents than senior residents. All but three of the residents successfully reduced and fixed the fracture fragments (one senior resident and two junior residents completed the reduction task but were unable to achieve fixation of all fragments). For the most part, junior and senior residents performed comparably on this difficult task. However, there was a dramatic difference in the cumulative hand distance category (79 m for senior residents and 390 m for junior residents), which even with the small sample size was significant (p < 0.01). There were no significant differences in articular reduction error, time to completion, or number of discrete hand motions.

TABLE III.

Comparison of Performance Data Between Junior and Senior Residents on the Simulated Articular Fracture Reduction Task

Performance Data	Senior Residents*	Junior Residents*	P Value
Articular error (mm)	3.00 ± 0.43	3.09 ± 1.25	0.86
Time to completion (min)	13.43 ± 4.68	14.75 ± 7.78	0.73
Cumulative hand distance† (m)	79 ± 48	390 ± 176	<0.01
Discrete hand motions (no.)	540 ± 303	511 ± 227	0.88
Global rating score (points)	3.20	2.57	0.27

^*The values are given as the mean and the standard deviation.

^†Among the variables that were physically measured, only this variable, as measured by optoelectronic motion capture, differed between the two groups.

Fig. 5.

Column plot showing little difference between junior and senior resident performance in the time to complete the fracture reduction task or in the number of discrete hand motions tracked. However, there was a nearly fivefold higher cumulative hand distance traversed by junior residents when compared with their senior counterparts.

No significant difference was observed in the global rating score between senior and junior residents (p = 0.27). A small negative correlation (r = −0.31) between global rating score and cumulative motion was observed. There was little correlation between global rating score and duration (r = 0.18), articular reduction error (r = −0.04), or discrete actions (r = −0.01). No significant differences between junior and senior residents were observed with respect to the procedural checklist.

Discussion

This educational exercise was well received by the participants. A goal of this study was to develop objective methods to assess resident performance and to use these assessments to demonstrate construct validity of the simulation. The data showed that senior residents performed slightly better than junior residents in most categories, and that there was less variability in their performance. The differences in time to completion, discrete hand motions, articular step-off, and global rating of performance were not significant. The cumulative hand motion distance was significantly less for senior compared with junior residents, suggesting more deliberate task performance in this fracture reduction simulation.

Construct validity has been demonstrated in other orthopaedic surgical simulation models. For example, Van Heest et al.¹ evaluated the technical skills of twenty-eight orthopaedic surgery residents performing open carpal tunnel release on cadaver specimens. The investigators evaluated the residents’ cognitive knowledge and then assessed their technical skills, using a checklist score, a global rating score, and a pass or fail assessment. When the residents were compared on the basis of years of training, the data showed significant differences in the technical assessment scores and the cognitive knowledge test scores that were obtained before the surgical simulation was performed. Although a poor knowledge test was associated with poor technical skill scores, a high knowledge test score did not ensure a successful surgical performance or high technical skill scores. The results confirmed the logical deduction that assessment of resident knowledge is insufficient to accurately determine competency to perform a specific surgical procedure. For this reason, validated methods to objectively assess surgical skill need to be developed to ensure that orthopaedic surgery residents obtain minimum levels of skills competency.

Surgical and procedural skill assessment tools have become more technologically advanced over the past several years. Motion capture and similar technologies have been utilized in the areas of anesthesia and general surgery to assess resident skills. For example, Chin et al.⁹ studied ultrasound-guided supraclavicular blocks performed by junior residents, fellows, and experienced consultant anesthesiologists. A validated assessment tool known as the Imperial College Surgical Assessment Device (ICSAD), which includes an electromagnetic hand-tracking system, was used to measure three dexterity parameters. The data showed that experts performed significantly better than residents in all ICSAD parameters, and that fellows demonstrated significant improvement between their early fellowship and late fellowship performances. That study supports using motion capture technologies to assess the procedural skills of trainees.

Yamaguchi et al.¹⁰ studied experienced and novice surgeons performing an intracorporeal suturing task in an inanimate laparoscopic trainer box while hand movements were tracked with an electromagnetic motion-tracking system. The outcomes assessed (time to completion, path length, and average speed of the forceps in each hand) were able to clearly differentiate between novice and experienced surgeons’ laparoscopic suturing skills.

One of the few examples of motion capture technology in orthopaedics involved simulated arthroscopic procedural tasks on a shoulder arthroscopy skills trainer¹¹. That study found significant differences in performance between senior and junior surgeons, with less time taken and improved economy of movement by the more experienced surgeons.

In the current study, analysis of the hand motion data showed interesting and perhaps important differences between the two groups of residents. The junior residents had far more total hand motion, despite roughly the same number of discrete direction changes. Excess cumulative hand motion by the junior residents may be interpreted as non-purposeful movements, often characterized by surgical educators as flailing. This disparity in hand motion is visualized most clearly in Figure 6, which shows comparable arcs of hand motion in one junior resident and one senior resident, each chosen as the most representative of his or her respective group-wide average for cumulative hand motion distance. Senior residents clearly demonstrated a smaller range of hand motion, which could be interpreted as more precise and purposeful movements. The data could also suggest that less soft-tissue trauma would be likely to occur were this an actual surgery, rather than a simulation.

Fig. 6.

Cumulative recorded hand motion tracings (colored dots) of one junior resident (Junior1) and one senior resident (Senior4) during the simulation, with each subject chosen as the most representative of his or her respective group-wide average for cumulative hand motion distance. The dots are colored differently for each hand. The plots show comparable arcs of hand motion but demonstrate much more precise and purposeful movements for the senior resident.

The differences between senior and junior residents in time to completion and articular step-off were not significant. Although the reasons for this negative finding are uncertain, it may be explained simply by the small sample size of residents involved in the study or by the exceptional challenges in the task making it too difficult to differentiate between trainee performances at this experience level. An anatomic reduction remains the ultimate goal of articular fracture surgery, but, to our knowledge, an acceptable degree of step-off has not been clearly defined in the literature. Because of the small number of study participants, the present investigation serves as a pilot study. However, although these initial small-group findings hold interest in their own right, arguably the study’s greater importance is to document the feasibility of a new testing protocol that can be used for training and/or for quantitative performance comparisons in much larger groups of orthopaedic surgeons.

The cost of the model is a concern related to wider adoption of this model. The faculty time commitment necessary to provide the assessments was a total of approximately twenty minutes per subject and could be done at their leisure, as the performances were available on video for viewing at any time. Faculty involvement was primarily in approving the model setup and then viewing the video of the residents in the exercise to complete the global rating score and key steps checklist. Therefore, the total time to assess and to score the residents’ performance was approximately two hours. The other outcomes (such as time to completion, articular step-off, and hand motion data) were assessed objectively, were measured on the basis of raw data, and did not require staff evaluation.

This articular fracture reduction model is more realistic than the usual Sawbones fracture exercises, because there was an attempt to more closely imitate the operating room environment. Although the synthetic materials used in the tibial plafond surrogates are not fully representative of the detailed soft-tissue anatomy surrounding the fracture site, the synthetic soft-tissue envelope provided a realistically limited window of access to the fracture site. This degree of access challenged residents to achieve fracture reduction and articular surface congruity through restricted visualization, similar to the situation in the operating room when limited approaches are used. In turn, this limited access required effective use both of fluoroscopy and of fracture reduction techniques and instrumentation, and contributed to the difficulty of the task, with the achieved articular reductions having on average 3 mm of residual step-off. An attraction of this simulation approach is that a limitless range of specific fractures can be used for various resident skill levels and task challenge, for each of which an arbitrarily large number of identical test specimens can be produced.

This study has demonstrated that a relatively complex task (reduction of a three-segment articular fracture) can be simulated in a laboratory environment, and that task performance can be objectively measured. The present study establishes the basic construct validity of the simulation trainer. At its present stage, the simulation is primarily a training tool. It illustrates the potential value of surgical simulation in orthopaedic trauma surgery education. Our focus on improving resident education by utilizing only residents in the exercise was an important first step. We feel that more significant differences in performance within some of the outcome measures were not seen primarily because of the power of the study and the limited numbers of subjects involved. It is likely that larger numbers of subjects and comparisons of junior residents with practicing orthopaedic traumatologists would have shown more significant differences in the performance measures.

In ongoing work, we are looking at the use of these validated techniques for training to determine if performance improvement can be achieved. We plan to increase the numbers of surgeons tested and to use before-and-after comparisons to measure the degree to which training improves performance. Future studies will focus on using the model to train residents with a goal of demonstrating improvement in resident surgical skills in articular fracture reduction. Although further work is necessary, laboratory-based models of orthopaedic tasks and operations, combined with objective assessments of performance such as those presented here, hold great promise to improve resident training in surgical skills and to make valuable operating room experiences as safe and effective as possible.

Supplementary Material

Supporting Data

Disclosure of Potential Conflicts of Interest