Construct validation of a cost-effective vessel ligation bench-top simulator

Abstract

Objective

Many bench-top surgical simulators assess laparoscopic proficiency, yet few address core open surgical skills. The purpose of this study is to describe a cost-effective bench-top vessel-ligation simulator and provide construct validation.

Design

Prospective comparison of blinded proficiency assessments among participants performing a bench-top vessel-ligation simulation task. Evaluations were performed using Objective Structured Assessments of Technical Skills.

Setting

This study took place at the University of Virginia School of Medicine, a large academic medical institution.

Participants

Participants included fourth-year medical students participating in a focused surgical elective course (N = 16), post-graduate year (PGY) 2–3 surgery residents (N = 6), and surgical faculty (N = 5).

Results

Fixed costs of the vessel-ligation simulator totaled $30. Flexible costs of operation were less than $0.20 per attempt. Median task-specific checklist scores among medical students, residents, and faculty were 4.83, 7.33, and 7.67, respectively. Median global rating scores across the three groups were 2.29, 4.43, and 4.76, respectively. Significant proficiency differences were noted between students and residents/faculty for both metrics (p < 0.001).

Conclusions

A cost-effective bench-top simulator can effectively measure proficiency with basic open surgical techniques such as vessel-ligation. Among junior surgical trainees, this tool can identify learning gaps and improve operative skills in a pre-clinical setting.

INTRODUCTION

The role of simulation training has expanded over the last decade, driven by scrutiny of surgical outcomes and reductions in operative experience due to work hour restrictions.¹ In addition to increasing exposure to advanced techniques, simulation can equip junior-level residents with fundamental skills in open surgery.² While most program directors see the value of a simulation laboratory, formal evaluations of basic skills are rare.³ Furthermore, among medical students, simulation affords a low-stress environment in which fundamental proficiencies can be developed.⁴ Acquiring these skills in a pre-clinical setting can enhance participation in the operating room, improve learning experiences, enhance mentoring, and nurture interests in surgical careers.^{5, 6} This last aim is becoming increasingly important, as the number of medical students interested in pursuing a surgical career has decreased.⁷

In response to these pressing needs, the American College of Surgeons and the Association of Program Directors in Surgery developed the Surgical Skills Curriculum for Residents. The first phase of this program addresses basic open techniques such as knot-tying, suturing, and dissection.⁸ However, without pragmatic and cost-effective simulators for these modules, the demand for practice and evaluation in a pre-clinical setting is unmet. Numerous studies have shown that low- and mid-fidelity bench-top simulators are as effective as expensive biological and virtual-reality models.^9–12 Low fixed costs permit broad implementation, while low flexible costs facilitate repetitive self-practice.

Despite clear clinical relevance, an inexpensive bench-top simulator for open vessel-ligation does not currently exist. Such a device would fill an important learning gap: most graduating medical students have minimal exposure to vessel ligation, yet junior surgery residents are expected to perform this task safely and efficiently in the operating room. The purpose of this study was to create a cost-effective vessel-ligation simulator and demonstrate its construct validity as an evaluative tool. The study was designed to test the hypothesis that this tool could stratify between proficiency levels of graduating medical students, mid-level surgery residents, and experienced surgical faculty.

METHODS

Study Participants

Invitations for voluntary participation were extended to graduating fourth-year medical students enrolled in a focused, two-week surgical elective near the end of the academic year. All students had matched into residencies in surgical specialties. To better capture variations in skill level, no exclusion criteria relating to operative experience were instituted. Because open vessel-ligation is a basic technique typically acquired in the first 1–2 years of residency, voluntary participants were also recruited among post-graduate year one (PGY 1) general surgery interns during the first month of residency, PGY 2–3 general surgery residents, and experienced surgical faculty.

Before evaluation, each participant received verbal instruction regarding the steps involved in the stimulation task. Medical student participants were also provided with a video demonstration of the task, which they were expected to have reviewed prior to assessment. Review of this demonstration was not required for resident and faculty participants given pre-existing clinical experience. In order to capture initial technical proficiency, participants were not allowed to practice on the simulator prior to assessment.

Model Design

The vessel-ligation simulator was designed with several objectives in mind. The aim was to replicate the process of controlling a section of vascularized tissue in a deep abdominal space. The model would ideally recreate the roles of both the primary surgeon and the surgical assistant. Budgetary parameters for construction were set to less than 100 USD per model. Finally, because the expected settings of implementation were pre-clinical evaluation and self-directed practice, the flexible costs of repetitive use were minimized.

With these goals in mind, a four-sided box was created from pine wood to simulate a confined abdominal compartment, and a proprietary vessel mounting device was secured to the base of the replicated surgical field. The total cost of simulator construction was 30 USD per model (Figure 1). Several synthetic materials were tested as the simulated vessel, including pen-rose drains, flexible intravenous tubing, and silicone tubing. Ultimately, disposable latex surgical gloves proved to be the most realistic substitute—glove fingers were deformable enough to clamp and tie, yet elastic enough to require steady tension for successful ligation. Mounting a glove in the simulator required less than 30 seconds, and allowed for two ligation attempts. With expired gloves in constant abundance, flexible costs were reduced to solely those of 3-0 silk suture ties, which were obtained at 0.10 USD apiece (Hefei Fast Nonwoven Products Co, Anhui, China).

Figure 1.

The VesselBox training model. Participants obtain proximal and distal control of simulated vascularized tissue and perform suture tie ligation with the help of an assistant.

Implementation

The vessel ligation task included roles for a primary surgeon (the study participant) and a surgical assistant (research assistant). The primary surgeon begins by requesting a curved Kelly forceps and gains proximal control of the synthetic vessel. The assistant then gains distal control with an opposing forceps. The primary surgeon then divides the vessel between the two forceps with Mayo scissors and performs a suture tie ligation of both ends using 3-0 silk ties with a one- or two-handed tying technique. During this process, the participant must prompt the assistant to release each forceps with appropriate timing.

Participants were video recorded as they enacted the role of the primary surgeon and ligated two vessels using the simulator. Camera field was limited to include only the simulator and the participants’ forearms. For each evaluation, the role of the surgical assistant was fulfilled by a PGY 3 surgery resident experienced in using the simulator; the assistant was the same for all participants (YH). Following participation, residents and faculty were verbally interviewed for subjective reviews of the simulation model prior to receiving their evaluation results. Specifically, participants were asked whether they felt the simulator would be useful to prepare new interns for open surgeries, and whether the model accurately captured their own experiences in vessel ligation.

Recordings for all participants were uploaded for blinded evaluation by three surgical faculty members who routinely perform open operations. Evaluations were completed using a task-specific checklist of eight binary items (one point per item) and the Objective Structured Assessments of Technical Skills (OSATS) global rating scale (GRS) for hemostasis (Figure 2). The task-specific checklist was designed to capture objectively the components critical to every vessel ligation such that individual feedback could be provided in a specific manner. Comprised of a series of scoring items each rated on a 1 to 5 scale, the more subjective OSATS GRS has high inter-observer reliability and proven utility in assessing vessel ligation.^{13, 14} Scores on the GRS items were averaged to produce a representative composite score.

Figure 2.

Task-specific checklist and Objective Structured Assessment of Technical Skills Global Rating Scale for hemostasis.

Statistical Analysis

For each participant, checklist and composite GRS scores were averaged across evaluators. Due to non-normal data distribution, the Kruskall-Wallis test was used to assess variations across all three participant groups for checklist and GRS scores, while the Wilcoxon rank-sum test was used to compare performance between paired groups. Cronbach’s coefficient α was calculated for both metrics to measure internal consistency. Results were considered significant at 2-tailed p < 0.05. All data were analyzed using SAS statistical software (version 9.3; SAS Institute, Inc). This study was approved by the University of Virginia Institutional Review Board (IRB-SBS protocol # 2013-0457-00).

RESULTS

Sixteen medical students, 6 PGY 1 surgery interns, 8 PGY 2–3 surgery residents, and 5 surgical faculty completed the vessel-ligation simulator task. All interview respondents (N = 15) believed that the model would be useful to prepare junior trainees for open surgery. Most (93%, 14/15) felt that the model closely emulated intraoperative processes; one participant felt that proximal tie ligation should precede vessel division. Additional critiques pointed out that the latex material imitates larger vessels and vascularized tissue (such as bowel mesentery) relatively well, but inadequately represented smaller, more friable vessels.

Three sets of blinded evaluations (checklist and composite GRS) were accrued, and scores from the three faculty evaluators were averaged for each participant. Proficiency comparisons across groups are summarized in Figure 3. Median task-specific checklist scores for the student, PGY 1, PGY 2-3, and faculty groups were 4.83, 5.83, 7.33, and 7.67, respectively (Max = 8). In pairwise comparisons of checklist scores, students and PGY 1 interns scored significantly lower than PGY 2–3 residents and faculty (p < 0.001). No significant difference was noted between students and PGY 1 interns (p = 0.797), nor between PGY 2–3 residents and faculty (p = 0.530).

Figure 3.

Comparison of task-specific checklist (a) and Global Rating Scale (b) between participants at the fourth-year medical student (MS4), post-graduate year 1 (PGY 1), PGY 2-3, and faculty levels.

Median composite GRS scores for the student, PGY 1, PGY 2-3, and faculty groups were 2.29, 2.38, 4.43, and 4.76, respectively (Max = 5). Students and PGY 1 interns scored lower on the GRS than PGY 2–3 residents and faculty (p < 0.001). There was again no difference in performance between students and PGY 1 interns (p = 0.890). Faculty trended toward higher GRS scores than PGY 2–3 residents, however the difference was not significant (p = 0.057). The internal consistency of the checklist and GRS metrics, as measured by Cronbach’s α, were 0.71 and 0.96, respectively.

DISCUSSION

This study describes the innovation and implementation of a cost-effective vessel-ligation simulator, and demonstrates its utility in a pre-clinical evaluative setting. Although only one surgical technique was studied, the same model can be used to evaluate and practice transfixation ligation, hernia sac ligation, excisional biopsies, and even vascular anastomosis. The total cost of building the model and evaluating 35 participants was less than 50 USD, and each additional evaluation cost less than 0.20 USD. With this minor investment, broad screening of all new PGY 1 surgery residents can be performed at the beginning of each academic year, and proficiency can be verified prior to entering the operating room.

This study’s results demonstrate that, for basic surgical skills such as vessel ligation, incoming PGY 1 residents have comparable technical proficiency as fourth-year medical students. By showing a significant difference in proficiency between PGY 1’s and PGY 2–3’s, the model provides evidence that these fundamental skills are typically acquired through practice in the operating room on live patients. As educators increasingly recognize simulation’s role in surgical training, more and more models are being adopted.^{2, 15} However, few programs are willing to invest in simulation for fundamental open surgical skills, as it is expected that these skills are eventually mastered in the operating room. Nevertheless, new surgical trainees’ initial operative encounters are almost always in the setting of open surgeries such as hernia repairs, excisional biopsies, and small mass resections. Poor technical fundamentals in these settings increase operative duration and decrease the effectiveness of intraoperative teaching. Furthermore, among these operations, bleeding is consistently among the most common post-operative complications.^{16, 17} By screening trainees for skill deficiencies prior to operative participation, educators can continue to offer appropriate teaching without jeopardizing patient safety.¹⁸ Surgical interns who score poorly on this model may be encouraged to practice in the simulated setting before applying this fundamental technique on live patients.

Thus far, an open vessel-ligation simulator cannot be readily purchased in the United States. Kits are available from overseas; however, these cost between 200 and 500 USD each, with variable pricing for synthetic vessel replacements. Although knot-tying boards often include a section that simulates tying in deep spaces, these components offer minimal relevance to vessel-ligation. Innovative, low-fidelity micro-anastomosis models have previously been described.^{9, 19} By comparison, the present model focuses on ligation of larger vessels and vascularized tissue in a confined compartment, thereby developing a more fundamental and widely-applicable skillset at reduced flexible cost. While this model does not accurately simulate the more advanced technique of ligating delicate vessels, it does cultivate an important fundamental skill among inexperienced trainees. As with any simulation model, clinical fidelity is juxtaposed against operational cost.²⁰ Because vessel-ligation is an elementary surgery technique, an inexpensive model is more likely to hold broad appeal. However, when considering low-fidelity simulators, face and construct validity should be verified before implementation.²¹ By demonstrating construct validity for a simple simulation model, this study lays compelling groundwork for further investment and application. Grober and colleagues previously demonstrated equivalent effectiveness between low- and high-fidelity models for teaching vascular anastomosis among junior and mid-level surgery residents.⁹ A similar experiment measuring the effectiveness of the vessel-ligation simulator at improving technical skills is ongoing.

This study has several limitations. First, only 6 PGY 1 interns, 8 PGY 2–3 residents, and 5 faculty participants were enrolled. Nevertheless, group sizes were adequate to stratify participants by skill level. Although no statistical differences were noted between PGY 2–3 resident and faculty outcomes, this was at least in part due to the rudimentary nature of the tested technique. Because the target training populations are graduating medical students and incoming PGY 1 residents, the model objectives were fulfilled despite the small sample size. Second, although the OSATS GRS has previously been validated,¹⁴ the vessel-ligation checklist has not. Therefore, despite its binary nature, this checklist may be scored inconsistently across evaluators. However, it was felt that the checklist offered a necessary objective complement to the more subjective GRS, and internal consistency of the checklist was high.²² The argument could even be made that, with a Cronbach’s α of 0.71 compared to GRS’s α of 0.96, the task-specific checklist may actually be a more efficient and informative proficiency metric.²³ Nevertheless, the fact that a more notable difference between PGY 2–3 residents and faculty was detected on the GRS indicates that the GRS does provide some intangible insight not available through the task-specific checklist. Similarly, the GRS showed more variability in the performance of PGY 2–3 residents than the checklist metric. Hence, even among participants who are familiar with the steps of vessel ligation, disparities exist in efficiency, fluidity, and tissue handling.

In the interest of patient safety, even the most rudimentary techniques performed on low-morbidity operations should be practiced and assessed in the pre-clinical setting. Moreover, operative mentoring is enhanced in the presence of technical proficiency. These factors are frequently overlooked due to surgical simulation’s stigma as an expensive teaching medium. By providing the means to master basic techniques outside of the operating room, inexpensive open surgical simulators like the vessel-ligation model can accelerate the acquisition of operative autonomy across a broad population of surgical trainees.

ABBREVIATIONS

PGY

Post-graduate year

OSATS

Objective Structured Assessments of Technical Skills

GRS

Global rating scale