A Laparoscopic Simulator Tool for Objective Measurement of Residents' Laparoscopic Ability

Abstract

Objective:

We sought to develop an objective measurement of residents' laparoscopic ability by using a laparoscopic simulator assessment tool.

Methods:

An inexpensive laparoscopic simulator was developed. Three laparoscopic assessment procedures were created: 1) bead/pom-pom drop, 2) checkerboard drill, and 3) bead manipulation. Two minimally invasive surgeons and 8 PGY 3/4 and 15 PGY 1 residents were timed performing the 3 procedures. Ten of the PGY 1 residents were retested at the end of their PGY 1 year.

Results:

The minimally invasive surgeons completed the laparoscopic drills in approximately half the time of the PGY 3/4 (P=0.02), and PGY 3/4 were 60% faster than PGY 1 (P=0.01). PGY 1 completed the drills in half the time at the end of the PGY 1 year (P=0.005). As an objective measurement of residents' laparoscopic surgery competency, by the completion of the academic year, all PGY 1 residents must be able to complete the drills as fast as or faster than the original PGY 3/4 times.

Conclusion:

We developed an inexpensive, objective, simple laparoscopic simulator assessment tool for measurement of residents' laparoscopic ability.

INTRODUCTION

In 2001, the Accreditation Council for Graduate Medical Education (ACGME) set forth 6 core competencies (patient care, medical knowledge, practice-based learning and improvement, communication skills, professionalism, and system-based practice) in which residents must demonstrate proficiency. June 2006 marks the end of Phase 2 of 4 of the ACGME guidelines when programs are expected to have evaluation processes of resident performance in all 6 competencies. Patient care competency includes the demonstration of surgical skills including laparoscopy.

Assessment of residents' surgical skills is subjective, depending on faculty evaluations of their operative performance. Flaws typically associated with this type of evaluation are lack of standardization, poor attending objectivity based on prior history with the resident, and overall noncompliance of faculty evaluation of resident performance. Vogt et al1 found that by revealing a resident's identity to evaluators, the faculty scoring of the same videotaped performance was altered 50% of the time. Gosman et al² found a mere 36% compliance rate in faculty completion of residents' surgical evaluations. In response to the ACGME guidelines and in recognition of the need for standardized evaluation, we developed an inexpensive laparoscopic simulator assessment tool to provide an objective measurement of residents' laparoscopic skills.

METHODS

Laparoscopic Simulator

A laparoscopic simulator was created in the Falor Division of Surgical Research at Summa Health Systems3 and was utilized in our research (Figure 1). The laparoscopic simulator was constructed with a heavy-duty plastic storage container (231/2″x13″x193/4″) with a 5″ high particle-board table the inside of which serves as the “operative field.” The cost for construction of the simulator was approximately $300. Three laparoscopic assessment drills were developed, which were performed using a laparoscopic camera and video monitor, which is similar to what takes place in the operating room

Figure 1.

Laparoscopic trainer.

Assessment Drills

Bean and Pom-pom Drop

Three beans, 3 large pom-poms, and 3 small pom-poms were grasped and deposited into a small flask (Figure 2).

Figure 2.

Bean/Pom-pom Drop.

Checkerboard Drill

Five beads were placed onto their corresponding letter or symbol on a 3×3 grid (Figure 3).

Figure 3.

Checkerboard drill.

Figure 4.

Figure 4. Bead manipulation.

Bead Manipulation

Three beads were placed onto 1 of 2 hooks (Figure 1C).

All drills began with the bean/pom-poms in the same starting position. One of the authors timed every drill with a stopwatch. Pretesting instructions were given on how to perform each task. No warm-up was allowed, and each physician was tested during his or her initial performance of each assessment drill.

Evaluation

At the inception of the study, all 8 PGY III/IV residents in Obstetrics and Gynecology at Summa were timed with a stopwatch during their initial performance of the 3 laparoscopic assessment drills. All 8 PGY III/IV residents were considered competent in basic laparoscopy on their surgical evaluations. The 8 PGY III/IV residents had no prior exposure to the laparoscopic assessment drills.

Two minimally invasive laparoscopic surgeons (pelvic pain specialist and gynecologic oncologist) from the Minimally Invasive Surgery Institute at Summa were also evaluated.

Statistical Analysis

The Wilcoxon rank-sum test (Mann-Whitney U) was used to compare the times of drill completions between the groups (minimally invasive surgeons, PGY 3/4, PGY 1), and Wilcoxon's signed rank test was used to compare the PGY 1 retakes.

RESULTS

The median times in seconds to complete each assessment drill on the laparoscopic simulator are presented in Table 1. The minimally invasive surgeons completed the laparoscopic assessment drills in approximately half the time of the PGY 3/4 (statistically significant, P=0.02). The PGY 3/4 completed the assessment drills approximately 60% faster than the PGY 1 did (statistically significant, P=0.01). The PGY 1 completed the assessment drills in half the time at the end of their PGY 1 year compared with the beginning of their PGY 1 year (statistically significant, P=0.005). The PGY 1 retakes at the end of the PGY 1 year were equivalent to the PGY 3/4 times (not statistically significant, P=0.25).

Table 1.

Time to Completion (Median and range in seconds)

Gynecologist (#)	Bead/Pom-pom	Checkerboard	Bead Manipulation
Minimally invasive Surgeon (2)	50 (48-51)	18 (14-22)	33 (30-36)
PGY ¾ (8)	86 (70-155)	50 (27-65)	95 (42-257)
PGY 1 (15)	145 (58-243)	70 (28-160)	224 (57-739)
PGY 1 retake (10)	76 (53-89)	41 (36-49)	62 (35-91)

*Minimally invasive surgeon vs PGY ¾, PGY ¾ vs PGY 1, PGY 1 vs PGY 1 retake, all statistically significant, P < 0.05.

As an objective measurement of residents' laparoscopic surgical ability, by the completion of the academic year, all PGY 1 residents must be able to complete the drills as fast as or faster than the original PGY 3/4 times.

CONCLUSION

Laparoscopic simulators have become increasingly popular over the last 5 to 10 years. Laparoscopic simulators enhance surgical education by providing a safe, costefficient means for practicing surgical technique outside of the operating room without medicolegal or ethical constraints. Compared with laparotomy, laparoscopy requires greater technical training because of the loss of depth perception, loss of haptic feedback, fulcrum effect, and the use of instruments with a limited range of motion. Two main categories of laparoscopic simulators exist: virtual reality simulators and video-trainer simulators. Although virtual reality simulators are technically sophisticated and simulate actual surgical procedures (ie, surgical treatment of ectopic pregnancies) the environment is artificial and relatively game-like. Virtual reality simulators lack the tactile feedback of manipulating actual laparoscopic instruments and are frequently costly. Advantages of video-trainer laparoscopic simulators are the tactile feedback of the instruments and decreased cost. A disadvantage is that the tasks tend to be artificial, such as bead manipulation. Residents prefer video-trainers to virtual reality laparoscopic simulators,⁴ and acquired skills are retained longer with video-trainer laparoscopic simulators.⁵ With the development of core competencies as set forth by the ACGME in 2001, residency programs are required to develop objective, reliable, inexpensive means of evaluating the 6 core competencies. In this study, we present one such method to objectively evaluate residents' laparoscopic ability with a video-trainer laparoscopic simulator.

Although our assessment drills were not exact replications of surgical tasks, we feel that our laparoscopic simulator assessment tool is reliable because of the progression of times to completion of the tasks: minimally invasive surgeons were faster than the PGY 3/4 who were faster than the PGY 1. Also, our PGY 1 scores at the end of the PGY 1 year were faster than at the beginning of the PGY 1 year. Our bean/pom-pom drop and checkerboard drills are 2 of the more common drills used in laparoscopic video-trainer simulators.^6–7 Unfortunately, we did not record the amount of practice the PGY 1 residents performed before their retake at the end of the PGY 1 year.

The trainer utilized in our study can be manufactured for approximately $300 in conjunction with existing laparoscopic equipment, compared with commercially available laparoscopic trainers that range from $1500 to $15 000.³ Another major advantage of our laparoscopic simulator assessment tool is that it is completely objective, based on time of completion of the drills. Most observed structured assessment of technical skills (OSATS) examinations are video taped and then subjectively reviewed.⁸ A third advantage is its simplicity. Our examination is directly timed rather than video taped and subjectively assessed in the future. Because we wanted our laparoscopic surgical assessment tool to be objective and simplistic, we only evaluated speed.

We feel that the main advantage of our laparoscopic simulator assessment tool is that it provides an objective measurement of residents' laparoscopic surgical ability. Our residency program is presently using the laparoscopic simulator assessment tool to establish objective measurements of laparoscopic surgical ability. By the completion of the academic year, all PGY 1 residents must be able to complete the drills as fast as or faster than the original PGY 3/4 times. We chose the original PGY 3/4 times because all 8 PGY 3/4 were considered competent in basic laparoscopy by subjective faculty evaluation. Other authors have chosen median faculty times for competency.⁶ Because of the objectivity and simplicity of our laparoscopic simulator assessment tool, other residency programs could use our drills for assessing residents' surgical ability either using our established times or establishing their own criteria. We will compare our objective measurements of residents' laparoscopic surgical ability to subjective faculty evaluations. We are developing more advanced laparoscopic simulator assessment tools including camera manipulation, suturing, and knot tying for evaluating PGY 3/4 laparoscopic competencies.

Of interest, the times at the end of the PGY 1 year were equivalent to the PGY 3/4 times. It is our opinion that this is secondary to 2 factors. First, the laparoscopic simulator is located in one of our 2 lecture rooms, which is directly attached to the residents' study hall. It is available at all times for the residents to practice, and therefore repetition of the drills may have improved their laparoscopic skills. Second, the Minimally Invasive Institute was started after the PGY 3/4 were tested, and therefore they had minimal interaction with the minimally invasive surgeons before performing the drills. We are planning on testing our present PGY 3/4 residents who have extensive surgical experience in the Minimally Invasive Institute and compare these times with the original PGY 3/4 times.

In conclusion, we have developed a laparoscopic simulator assessment tool for evaluation of residents' laparoscopic ability. Advantages of our laparoscopic simulator assessment tool are that it is inexpensive ($300), objective (timed, not subjectively evaluated), simple (not video taped), may improve laparoscopic dexterity by repetition, and importantly can be used as an objective measurement of residents' laparoscopic ability.