Simulation Improves Resident Performance in Catheter-Based Intervention

Abstract

Objectives:

Surgical simulation has been shown to enhance the training of general surgery residents. Since catheter-based techniques have become an important part of the vascular surgeon's armamentarium, we explored whether simulation might impact the acquisition of catheter skills by surgical residents.

Methods:

Twenty general surgery residents received didactic training in the techniques of catheter intervention. Residents were then randomized with 10 receiving additional training with the Procedicus, computer-based, haptic simulator. All 20 residents then participated in 2 consecutive mentored catheter-based interventions for lower extremity occlusive disease in an OR/angiography suite. Resident performance was graded by attending surgeons blinded to the resident's training status, using 18 procedural steps as well as a global rating scale.

Results:

There were no differences between the 2 resident groups with regard to demographics or scores on a visuospatial test administered at study outset. Overall, residents exposed to simulation scored higher than controls during the first angio/OR intervention: procedural steps (simulation/control) (50 ± 6 vs. 33 ± 9, P = 0.0015); global rating scale (30 ± 7 vs. 19 ± 5, P = 0.0052). The advantage provided by simulator training persisted with the second intervention (53 ± 6 vs. 36 ± 7, P = 0.0006); global rating scale (33 ± 6 vs. 21 ± 6, P = 0.0015). Moreover, simulation training, particularly for the second intervention, led to enhancement in almost all of the individual measures of performance.

Conclusion:

Simulation is a valid tool for instructing surgical residents and fellows in basic endovascular techniques and should be incorporated into surgical training programs. Moreover, simulators may also benefit the large number of vascular surgeons who seek retraining in catheter-based intervention.

We explored whether simulation might impact the acquisition of catheter skills by surgical residents. In this randomized controlled study, simulation was found to be a valid tool for instructing surgical residents and fellows in basic endovascular techniques.

Catheter-based minimally invasive interventions are rapidly becoming the preferred initial treatment in patients with peripheral vascular disease. In progressive centers, up to 70% of all procedures performed by vascular surgeons are at least partially catheter-based. This changing venue requires a shift in our approach to training both fellows as well as general surgical residents during their exposure to vascular surgery. To this end in 2002, the resident review committee in general surgery created requirements for graduating vascular fellows that include serving as primary operator on 50 angiograms and well as 25 catheter-based interventions.¹ It is likely over time that these training requirements will eventually expand and include specialty training in more advanced catheter techniques, including thrombolysis and carotid angioplasty and stenting. This change in venue requires that general surgical residents, when rotating on the vascular surgery service, be exposed to catheter-based techniques. Familiarity with these techniques will allow general surgical residents to make an informed decision about vascular surgery as a specialty. Moreover, exposure to catheter-based intervention will prepare surgical residents for similar minimally invasive approaches that are likely to be the future of the majority of surgical subspecialties.

The last several years have also brought changes in the techniques used to train surgeons. As the technology for treating patients has evolved from the scalpel to laparoscopes, lasers, and catheters, so has the technology for teaching. The traditional approach of “see one, do one, teach one” is rapidly being replaced with the more progressive concept of “learn the operation before the operating room.” The internet, didactic educational tools with feedback, web casts, 3-dimensional imaging techniques, and simulation have allowed surgical trainees as well as already trained surgeons to become familiar with new interventional techniques before entering the operating suite.

Simulators were initially used extensively in the airline industry, where it seemed imprudent to train pilots to fly using planes filled with actual passengers. The segue to surgery seemed obvious and for the past several years simulators have been used in the training of general surgery residents, primarily in laparoscopic techniques. Although simulators initially consisted of cardboard boxes that mimicked the abdominal cavity, their level of sophistication over the past several years has increased substantially. Now, in a randomized controlled fashion, simulation has been proven to enhance the operative skills of general surgery residents learning laparoscopy.²

Simulators for catheter-based intervention have been available for approximately 5 years, and the technology is evolving rapidly. Currently, simulators are expensive and access has been limited. Despite the early popularity of these devices, little has been accomplished in validating their efficacy. Our group has previously demonstrated that novices not previously exposed to catheter-based intervention were able to perform simulator-based interventions with an increased level of competence following didactic training on the simulator. However, the ultimate test of simulation is to demonstrate that performance improves in the operating room. To this end, we designed a randomized prospective study to measure the effectiveness of simulator training on the performance of catheter-based intervention by surgical residents.

METHODS

Simulation Device

The Procedicus VIST system (Mentice Inc., Evanston, IL) is a multimedia device designed to simulate endovascular procedures for the treatment of a variety of clinical scenarios including carotid, renal and iliofemoral occlusive disease (Fig. 1). The system consists of a standard desktop PC (Intel Xeon 2.66 GHz, 1 GB RAM, nVIDIA GeForce4 Ti 4200 with AGP 8X) with software that contains a 3-dimensional representation of the human arterial system. This is coupled to a haptic module utilizing a force feedback system containing external input devices that allow the user to use standard angiographic catheters and guidewires, inject contrast dye, perform angioplasty, deploy stents, and perform fluoroscopy with digital subtraction angiography. The instructional system is displayed on a touch screen monitor that also allows for the selection of devices and catheters for simulation. A simulated fluoroscopic image is displayed on a second monitor.

FIGURE 1. The Procedicus VIST endovascular simulation system.

Study Design

This study was designed to validate the use of simulators in improving skills in the angiography suite in individuals without prior exposure to endovascular techniques. Following approval by the institutional review board, twenty general surgery residents rotating on the vascular surgery service were enrolled. Didactic teaching was provided to all residents in the form of reading material and a lecture on basic catheter-based intervention. Specific instruction was provided regarding arteriography, guidewire manipulation, correct vessel selection, catheter exchange, and angioplasty and stenting.

Only residents with no prior endovascular experience were enrolled. An entrance survey was performed to determine the demographics and as well as previous experiences that might be relevant to a residents’ ability to assimilate catheter techniques (Table 1). To evaluate prestudy differences between the capability of residents to perceive 3-dimensional structures, an ability that is important when performing catheter-based interventions, a visuospatial evaluation consisting of a card rotation and a cube comparison test was administered to all participants (Education Testing Service, Princeton, NJ).

TABLE 1. Background Information and Demographics Collected at Study Outset

Following the didactic sessions, participants were randomized using sealed envelopes to receive either mentored simulation training on a standardized iliofemoral angioplasty/stenting model available on the Procedicus simulator (n = 10) (Fig. 2), or no simulation training (n = 10) (control group). The endpoint of simulation training was the independent completion of the procedure by the operator with proficiency in all basic endovascular techniques. Training was not allowed to exceed 2 hours. Within 2 weeks of randomization, each participant completed 2 consecutive catheter-based interventions for lower extremity occlusive disease supervised by an attending vascular surgeon who was blinded to the training status of the resident. Lesions treated included a variety of iliac and/or femoral/popliteal stenoses or occlusions. Immediately preceding each case, the patient presentation, examination, and workup were reviewed with the resident, and the operative plan was discussed in detail. The participant's performance was then evaluated by the attending surgeon utilizing a checklist of the required steps for a standard catheter-based intervention (Table 2), as well as a global rating scale of the participant's endovascular technique and ability to complete the case (Table 3) adapted from a previously validated scoring system.³ A Likert rating scale (0–4; 4 = best) was used for checklist grading. All participants were verbally guided through the steps of the procedure by the attending surgeon while their technical skills were evaluated.

FIGURE 2. Iliac angioplasty simulation model: an external iliac stenosis is visualized with a selective contrast injection delivered from a contralateral sheath pre and post angioplasty and stenting.

TABLE 2. Examiner Checklist for Diagnostic Angiography, Angioplasty, and Stenting

TABLE 3. Global Rating Scale of Endovascular Performance

Subjects were informed of the manner by which they were being scored. Specific steps evaluated are summarized in Tables 2 and 3 and include guidewire and catheter manipulation, catheter exchanges, and over the wire balloon technique. If a resident performed a particular technique incorrectly, he or she was provided with constructive feedback. Performance of the 2 groups of residents was compared following each of the 2 endovascular cases to determine if there was an initial and a subsequent sustainable advantage of simulator training over traditional mentored instruction.

Statistical Methods

The results of these assessments as well as the questionnaire data were entered into a database and subsequently analyzed using the SAS system. Wilcoxon 2-sample test and Fisher exact test were used to evaluate for statistically significant changes in performance pre and post instruction. All values were represented as mean ± SD and mean differences were considered significant for a P value less than 0.05.

RESULTS

We first attempted to demonstrate that the 2 resident cohorts were identical in terms of background and technical intellect. We were not able to identify any prestudy differences between control and simulator trained residents. Both groups were comparable in terms of their age and gender, as well as previous experiences that might be relevant to a residents’ ability to assimilate catheter techniques (Table 1). Moreover, performance on the visuospatial test was not different between the 2 groups (P = not significant). There were no intraoperative complications that developed as a result of resident participation in this study. There were also no differences in perioperative morbidity or mortality between patients treated by the 2 groups of residents.

All residents randomized to simulation training were able to complete the mentored simulator training session in less than 2 hours (mean of 90 ± 21 minutes).

Endovascular Performance Checklist

Resident performance was graded by attending surgeons blinded to the resident's training status, using 18 procedural steps (Table 4; perfect score = 72). Overall, residents exposed to simulation scored higher than controls during the first endovascular intervention (simulation/control) (50 ± 6 vs. 33 ± 9, P = 0.0015). The advantage of simulator training persisted with the second intervention (53 ± 6 vs. 36 ± 7, P = 0.0006). For the first intervention, simulation training led to a numerical enhancement of each of the individual measures of performance, although several of these differences were not statistically significant. This may have been related to the small sample size. For the second interventions, the advantage of simulation over control was statistically significant for all variables with only 3 exceptions. There were 2 procedural steps where a significant difference was not found with either the first or the second intervention (advance femoral wire and image after percutaneous angioplasty). The first technique, interestingly, is not well taught by the simulator. The second measures the ability of a resident to remember to perform a completion angiogram after angioplasty. We were somewhat surprised to find that overall resident performance did not improve from the first to the second intervention. This lack of improvement was observed in both the nonsimulator and simulator groups. One might conclude from this finding that a mentored experience in the OR is less impactful that simulator training in improving resident skills.

TABLE 4. Mean Checklist Scores on Individual Measures of Performance for Simulator- and Non–Simulator-Trained Residents

Global Rating Scale of Endovascular Performance

A more subjective evaluation of resident performance was also performed by attending surgeons (Table 5; perfect score = 48). This evaluation included outcome measures such as the ability of the resident to complete the case, attending take over, and an overall assessment of wire and catheter skills. Residents exposed to simulation scored higher overall on this evaluation than control residents during the first intervention (30 ± 7 vs. 19 ± 5, P = 0.0052). Moreover, the advantage of simulator training persisted for the second intervention (33 ± 6 vs. 21 ± 6, P = 0.0015). For both procedures, simulation training led to numerical enhancement in all of the individual measures of performance. However, for the first intervention, 4 of these differences were not statistically significant, whereas only 1 variable was not significant for the second intervention. We found no significant improvement in resident performance from the first to the second intervention for either the nonsimulator or simulator cohorts. Thus, once again, an initial exposure in the angiography suite appeared to have little mentoring value.

TABLE 5. Mean Endovascular Global Rating Scale Scores on Individual Measures of Performance for Simulator- and Non–Simulator-Trained Residents

DISCUSSION

Invasive image-guided techniques, such as ureteroscopy, arthroscopy, laparoscopy, and endoscopy have become important components of surgical patient care. Minimally invasive procedures designed to treat a variety of diseases have become commonplace and are now often replacing traditional therapy. This same evolution has occurred in vascular surgery with the advent of catheter-based interventions. Although this new technology is of considerable advantage to patients, it poses certain challenges to surgeons as well as those responsible for mentoring new trainees. These remote procedures differ from standard surgical operations as a consequence of the loss of direct tactile and visual feedback, and the requirement for an increased need for hand-eye coordination.⁴

Despite these new challenges, current surgical training for the most part continues with the traditional mentored approach, where trainees are first exposed to procedures performed on actual patients under the guidance of an experienced teacher.⁵ This exposure is unstructured and dependent on the random admission of patients rather than a broad and methodical exposure of the residents to fundamental clinical scenarios. This method of training also frequently does not provide for objective feedback on the trainee's performance. The assumption is that these shortcomings will be overcome with multiple cases and the long duration of surgical training.

In addition, the initial introduction of new technology may potentially increase the number of adverse events that occur, such as the rate of common bile duct injuries during laparoscopic cholecystectomy.^6,7 Therefore, the need for adequate training at the time of introduction of a new technique is paramount. Recent changes in the treatment of carotid artery occlusive disease illustrate the challenges created by new technology. Greater than 100,000 carotid endarterectomies are performed each year for both symptomatic and asymptomatic extracranial carotid stenosis.⁸ Recent data, however, have demonstrated that carotid angioplasty and stenting is an equivalent form of therapy in high-risk populations.⁹ Therefore, the need exists to train a large number of practitioners, including vascular surgeons, in the area carotid stenting, while simultaneously maintaining low rates of morbidity and mortality. The traditional apprentice model, which has been used successfully in training residents in open surgery, will likely not be applicable for procedures that are extremely technical, single operator dependent, and associated with a small margin of error.

Fortunately, advancements in therapy are occurring in parallel with advancements in computerized simulation technology designed to train practitioners in the use of these new techniques. The Procedicus VIST system (Mentice Inc.) is a multimedia device designed to simulate endovascular techniques in a variety of clinical scenarios. Our group has previously established construct validity of this system.¹⁰ In comparing novice and expert groups, the simulator consistently reported numbers that were statistically significantly better for the experts than for the novices. This indicates that the simulator is able to accurately reflect the skill of an individual participant. Moreover, we found that instruction with the Procedicus VIST endovascular simulator, utilizing a carotid angioplasty and stenting module, resulted in significant subjective and objective improvement in the use of catheter-based techniques. Specifically, novice interventionalists benefited from a mentored simulation training program and were able to learn and complete a complex percutaneous intervention. Other randomized trials have shown that virtual reality surgical simulation is a valid tool for training of laparoscopic psychomotor skills.^2,11 However, to our knowledge, the effect of endovascular simulator training on subsequent performance in the operating room has not been established.

This randomized prospective study found that simulator training enhanced resident performance in the angio-operating room suite. This enhancement of performance was broad but not uniform for the individual techniques that comprise catheter intervention. This is a preliminary study and the lack of significance achieved with some of the techniques may well be related to the small number of residents studied. Moreover, exposure to trainers was longer in the virtual reality group because of the additional time spent in simulation training. Is it unlikely, however, that this is the single cause for the improvement in their operative performance since mentoring was performed by a surgeon fellow and was limited to simulation training. The subsequent instruction on catheter-based intervention was equally provided for both groups in the operating room angiography suite by the attending surgeon. Our findings therefore suggest that simulation may and should play a substantial role in the training of surgical residents and fellows in catheter-based intervention.

There are, however, several issues that deserve consideration. First, although immeasurable, it was clear to those of us who conducted this study that surgical residents are not a homogeneous group. Some residents had a much greater ability to learn and master catheter techniques than others, irrespective of whether or not they were randomized to simulator training. This is not a new finding for surgical mentors and native talent has long been recognized to have a profound effect on the acquisition of surgical skills. What we were not able to understand from this study is whether simulation had a beneficial effect on both “gifted” surgical residents as well as those less capable. There was a sense, although unsubstantiated by data, that simulation might play a more important role in the training of the latter group. A second relevant issue that deserves exploration is the durability of the beneficial effect of simulation. To begin to address this issue, we designed this study so that the performance of surgical residents could be evaluated after 2 successive procedures rather than one, the hypothesis being that it would be relevant to determine if there was a sustained advantage of simulation over traditional training. We were encouraged to find that the advantage of simulation training did persist after the second intervention. The duration of this benefit, however, remains unclear. A third issue worthy of addressing is the cost versus benefit of simulation. The average cost of currently available simulators is in the range of $200,000 to $400,000. This includes a variety of software material for carotid, renal, and iliofemoral interventions. The health system costs related to the use of the operating room for resident teaching have been found to be significant. This number is estimated to be approximately $50,000 per surgical resident over a training period of 4 years (due to increased operative time and decreased efficiency that occur when operating with a trainee).¹² As might be anticipated, it is difficult to calculate the specific benefit in cost provided simulation of catheter-based techniques. However, a durable benefit to simulation (even beyond 2 cases) would likely need to be established before any cost advantage could be recognized. Even if costs are reduced by providing procedural training in a teaching laboratory, it may be unreasonable to expect every surgical training program to have an endovascular simulator. The cost of installing and supporting high-fidelity endovascular simulators is substantial, and the cost-effectiveness of a simulation training facility requires a high utilization rate. Therefore, the shared use of regional simulation training centers would be the practical alternative.

It appears that simulation training is more valuable to novice participants than to physicians with extensive experience.¹⁰ For novices, simulation presents a risk-free environment for the acquisition of new skills. On the other hand, experienced interventionalists tend to learn most from real-life situations. Tactile feedback is also important in developing the skill set of the advanced interventionalist, and this is not available with current endovascular simulators. Nevertheless, advanced interventionalists may still benefit from simulator training when learning newer techniques. An experienced surgeon wishing to acquire a new skill set can use simulation to learn the steps of a new procedure. This is particularly applicable to carotid angioplasty and stenting where not only the order of the operation can be learned, but also techniques to treat complex anatomic variations and complications can be rehearsed. Another emerging potential benefit of virtual reality simulation is the ability to process real patient data, which enables the interventionalist to practice the anticipated procedure on the simulator, prior to performing the actual case. Any difficult parts of a procedure can be rehearsed, reducing the likelihood of adverse events occurring due to technical difficulties. Virtual reality simulation may also be used to provide realistic testing of new devices prior to their widespread commercial use.

CONCLUSION

High-fidelity simulation provides a promising opportunity for risk-free training in procedures and management of potential complications. While it does not replace clinical training, it does offer a means for mentored instruction in a realistic way, allows the interventionalist to make procedural errors and then experience the consequences, and completely avoids the risks of patient injury and medico-legal liability associated with “hands on” training in a patient care setting. Our data support the contention that simulation is a valid tool for instructing surgical residents and fellows in basic endovascular techniques and should be incorporated into surgical training programs.

ACKNOWLEDGMENTS

The authors thank Dr. Richard Reznick, MD, MEd, FRCSC, FACS, and Dr. Dennis Fowler, MD, FACS, for their valuable assistance in the design of this study.

Discussions

Dr. Carlos A. Pellegrini (Seattle, Washington): This study was designed to test the role of simulators in the teaching of catheter-based interventions in the angiography suite among individuals without prior exposure to endovascular techniques. The results clearly show that performance in the operating room was significantly better among those taught using the simulator than among those taught using traditional techniques. Interestingly, the difference in favor of those trained in the simulator persisted when the measurements were made at a second intervention, suggesting that the simulator provided a superior learning platform than at least the performance of the procedure itself in a human.

While I believe this unique observation is in and of itself worth the effort of this study, it gives rise to my first question, which has to do with the design of the study. Both groups had teaching materials and a lecture regarding the techniques to be used. The simulation group then had mentored teaching in the simulator, whereas the control group had no further learning opportunities. Do you think it is possible that some of the differences observed are simply due to the additional teaching that occurred in the test group, independent of the fact that you used a simulator? After all, these individuals were able to spend additional time with a faculty member versed in these procedures. One may have corrected for this potential bias by having 3 groups: 1 control as yours, 1 that continued having the same amount of total time with faculty but without the use of the simulator, and 1 exposed to the simulator training during that time. I would be interested in your thoughts in this regard.

You also mentioned in your paper that faculty noted that some residents had a much greater ability, you called it “a native talent,” to learn and master catheter techniques than others, irrespective of whether they were randomized to simulator training. You hypothesized that the use of the simulator may have a greater role for those who are less talented.

We have also observed significant variations on the native ability to perform a task and a substantial difference among residents’ learning curve. We also observed that if, instead of assigning a predetermined number of hours in the simulator as you did in your study, we allowed the learners to perform the task as many times as they needed to reach a plateau, defined as an inability to improve their scores more than 10% in 2 studies, that all individuals tested reached a similar plateau albeit at different times. Our findings go along with the concept advanced by Ericsson first and Reznick most recently that repetition rather than exclusively native talent is most important in the acquisition and maintenance of surgical skills.

Thus, my second question to you: did you consider training in the simulator to a certain criterion rather than over a predefined period of time? Have you had any experience with your previous studies testing the hypothesis that one can train most if not all individuals to a certain level of performance?

My comments notwithstanding, I believe this is an excellent study, one in a series that has now shown the power of simulation in the teaching of residents. I congratulate you and your group in carrying this out.

Dr. K. Kent (New York, New York): We struggled with the issue that you first mentioned, and that is the effect of mentoring itself, the didactic interactions of an attending, what the resident might have versus the effect of a simulator. And the reason that we had our residents spend time with reading materials and also in a didactic session before they went to simulation was to somehow equate the 2 groups so that in fact both groups would spend time with faculty members and they both would have some didactic teaching. Admittedly, the group that then went on to simulation had more didactic teaching and more time with attending faculty, but still both groups we felt had significant exposure ahead of time. So we hope that we have corrected for that somewhat. But there is no doubt that adding a third control that had an additional didactic interaction without simulation would be a way of sorting that out.

The issue of native talent is one that is fascinating. And of course I realize that because I have been practicing surgery for years. And some residents are better than others. But it became extraordinarily obvious as we were mentoring these residents through catheter-based intervention. I think the more technology there is, the more differential you may see in talent in individuals.

Because it was a study, confined people spend only 2 hours on the simulator. But I think for a practical side it probably is better over the long run to let people spend as much time on the simulator as they want. And I am sure the more talented ones will spend less time and the ones with less talent will spend more and eventually achieve the desired goal. I think your approach is in reality probably the right way to approach simulator training in practice. We confined it to a specific period of time just for purposes of the study.

Dr. Richard J. Shemin (Boston, Massachusetts): I think this is a very important study. And in the world of cardiac surgery, we, too, are challenged by learning endovascular techniques, and particularly the new frontiers of percutaneous valve interventions. My real question looks at the costs of these simulators and the regionalization. And they seem to be quite limited.

Do you envision being able to use a simulator not necessarily for training, but for better planning of a particular procedure where you can insert real-time information on a patient such as a 3-dimensional CT scan reconstruction of the aorta and its branches and actually practice an operation and plan better before you actually perform it? If so, that would greatly increase the value of these machines to not only train individuals or retrain experienced surgeons, but at the same time allow surgeons to better plan their endovascular procedures.

Dr. K. Kent (New York, New York): Excellent point. There is no question that that is already beginning to happen at least in an experimental phase and will happen in the future. And if we get to a point that it is an obligatory necessity to practice an endovascular intervention before the operating room, then it will be the responsibility of the hospital to provide these devices for us. So once that happens, you are absolutely right, I think cost issues will fade very rapidly.

Dr. John J. Ricotta (Stony Brook, New York): Dr. Kent, that was a great presentation. I have 3 questions.

The first is: how many times did your simulator group actually repeat this procedure? Do you have any idea of whether the number of repetitions correlated with their skill level?

The second question is: what parts of the procedure do you think were most likely to require repetition? Where did they score the worst in terms of the various steps in the endovascular procedure?

The third question is just your thoughts on how durable this might be, how often might you have to repeat this kind of thing in someone who may do a stent and then may go 2 or 3 weeks or 4 weeks without doing another procedure? How often do you think that they need to get back in the lab?

Dr. K. Kent (New York, New York): In answer to your first question, we didn't report the number of times that residents performed the intervention and the simulator lap. That is actually an excellent point. Maybe it would be of value to know that. It would give us an idea of how quickly residents are able to assimilate the technology. In terms of was there a part of the assimilation procedure that was more challenging than another, the answer is no. Actually, the differential in scores was fairly equivalent for all of the different procedural steps. So it didn't appear that there was one part that was more challenging than the other.

Dr. John J. Ricotta (Stony Brook, New York): The frequency of retraining. The third question is: they had training and then they did the procedure, did you take them back in a week, or do you have plans to go back and see how well they retained?

Dr. K. Kent (New York, New York): In the study design, the residents were required to perform the 2 major interventions quite rapidly after this initial experience. So we really don't have any information. But it does bring up an important point. My own experience with catheter training, and having mentored quite a number of people through it, it is important, I think, that the experience be quite confined, that you don't try to mentor somebody where they do a couple of procedures and then wait a few months and then a couple more procedures. To really learn catheter intervention, I think it is critical that there be a set 2- or 3-month period of time where you have concentration of the procedure so that you can really well learn the techniques.

Dr. Gerald M. Fried (Montreal, Quebec, Canada): Dr. Kent, congratulations on an excellent paper that adds evidence to the body of literature supporting the value of simulation in surgical education. I have a couple of brief questions.

The first relates to your outcome measure of assessment intraoperatively. Have you evaluated this assessment for reliability and validity? Is your assessment sensitive enough to differentiate levels of performance that we would like to be able to differentiate clinically? A problem common to many of the simulation papers in the literature is that this validation has not been done beforehand and therefore the instrument that is being used as the endpoint for the study may not be appropriate.

The next question is how to maximize the value of the curriculum. The curriculum is comprised of 2 components. One is the didactic material, which may clearly impact performance. We can't just put students in a simulator lab, let them go, and expect that they will develop the good habits and skills that we want them to. So I think that the formal didactic educational component needs to be done well. The second component is the conduct of the simulation training. This needs to be based on some evidence that the duration of training, the number of iterations, or the proficiency level to which we train our students in the simulator is based on some pilot data, and is not just chosen arbitrarily. I wonder if you could comment on these issues.

Dr. K. Kent (New York, New York): The first point that you make in terms of validating the outcome measures I think is an important one. The answer is no, we haven't validated the instruments that we use for measuring outcome. Actually, the process of that validation is a very complex one, as you know. One of the reasons that we had 2 different forms and so many different questions asked was so that we made sure that we were assessing outcomes in a very broad fashion. And I guess it is encouraging to us that the advantage of simulation was uniform for both surveys over the broad range of questions that we asked. So even without validation I think we have a sense that there must have been some gain for these people with simulation.

The whole question of how one should spend your time with a simulator is one that we haven't answered. Should it be for a period of time, is 2 hours the right period of time, 4 hours the right period of time? Should it be to a certain outcome in terms of your performance on the simulator? I don't know the answer to that. I think that is something that still needs to be evaluated. My prejudice is, as you and Dr. Pellegrini have suggested, that it probably should be to a certain outcome, that if it takes somebody a half hour to learn how to do the procedure on a simulator, that is fine, if it takes somebody else 4 hours. But it surely should be to an outcome that we can use as a measurement.

Dr. Haile T. Debas (San Francisco, California): My question is very brief. This was done in the Department of Surgery. What interactions did you have with interventional radiology, if any? And if so, how did you manage that?

Dr. K. Kent (New York, New York): That is a dangerous question that I will slide right over the top of. I think that this was a study that was conducted just within the Department of Surgery, so actually there wasn't any involvement in interventional radiology protocol.

Dr. Richard P. Cambria (Boston, Massachusetts): Dr. Kent, I enjoyed your study and thank you for the opportunity to read the manuscript. A technical question. Your outcome measure assessed lower extremity interventions, nonselective catheterizations, if you will. So it is a technical question about the flexibility of the system. Is it capable of, number one, providing tactile feedback to the operator? And number 2, has there been any experience with more advanced selective catheterization procedures such as the carotid that you showed in the video?

Dr. K. Kent (New York, New York): Great question. There are some programs now that are available for carotid intervention, and in fact simulators are being used quite widely for training more experienced interventionalists in the area of carotid stenting. That said, there is no tactile feedback. And for experienced interventionalists that is really key, it is very important to be able to feel the wire. And I think until simulators progress to the point that they allow you to have tactile feedback, they are going to be of limited value in training experienced interventionalists.

Dr. Richard H. Bell, Jr. (Chicago, Illinois): I enjoyed the paper very much, and I agree with your assertion that simulation is going to be very important in the future. I have a methodologic question. I assume that the vascular surgeon who was taking the resident through the procedure was blinded as to whether the resident had had the simulation training. But does it become obvious to the supervising surgeon within a few minutes who has had the simulation training and who hasn't? Could that bias the scoring?

Dr. K. Kent (New York, New York): In the operating room, in the angio suite, an attending took the residents through the procedure. And the attending in fact was blinded. And being one of the attendings, I have struggled to, of course I always wanted to try to understand, but struggled to figure out who in fact had been simulator-trained persons. And as pointed out by Dr. Pellegrini, there were some non–simulator-trained residents that actually performed quite well, again because perhaps of native talent. So we were very effective I think in blinding the attendings and feel very comfortable with our results.