Effects of Visual Force Feedback on Robot-Assisted Surgical Task Performance

Abstract

Background

Direct haptic (force or tactile) feedback is negligible in current surgical robotic systems. The relevance of haptic feedback in robot-assisted performances of surgical tasks is controversial. We studied the effects of visual force feedback (VFF), a haptic feedback surrogate, on tying surgical knots with fine sutures similar to those used in cardiovascular surgery.

Methods

Using a modified da Vinci robotic system (Intuitive Surgical, Inc.) equipped with force-sensing instrument tips and real-time VFF overlays in the console image, ten surgeons each tied 10 knots with and 10 knots without VFF. Four surgeons had significant prior da Vinci experience while the remaining six surgeons did not. Performance parameters, including suture breakage and secure knots, peak and standard deviation of applied forces, and completion times using 5-0 silk sutures were recorded. Chi-square and Student’s t-test analyses determined differences between groups.

Results

Among surgeon subjects with robotic experience, no differences in measured performance parameters were found between robot-assisted knot ties executed with and without VFF. Among surgeons without robotic experience, however, VFF was associated with lower suture breakage rates, peak applied forces, and standard deviations of applied forces. VFF did not impart differences in knot completion times or loose knots for either surgeon group.

Conclusions

VFF resulted in reduced suture breakage, lower forces, and decreased force inconsistencies among novice robotic surgeons, although elapsed time and knot quality were unaffected. In contrast, VFF did not affect these metrics among experienced da Vinci surgeons. These results suggest that VFF primarily benefits novice robot-assisted surgeons, with diminishing benefits among experienced surgeons.

Introduction

Robot-assisted surgical systems, such as the teleoperated da Vinci surgical system¹ (Intuitive Surgical Inc., Sunnyvale, CA), have facilitated minimally-invasive approaches to selected cardiothoracic operations including mitral valve repair² coronary artery bypass grafting³, and thoracic tumor resections⁴. The enhanced degrees of freedom afforded by such computer-augmented systems over conventional manually-actuated thoracoscopic instruments permit the dextrous manipulations required to perform these operations in the relatively confined and rigid thoracic space. However, currently available robotic surgical systems do not yet provide noticeable force or tactile feedback, otherwise known as haptics, to the operating surgeon. There exist several practical and technical challenges that need to be overcome in order to implement direct haptic capabilities to the hands of the surgeon in complex surgical systems.

Although it would seem intuitive that haptic feedback would be critically important in the manipulation of the delicate tissues and suture materials inherent in cardiothoracic operations, its clinical relevance in the performance of robot-assisted surgical tasks is controversial^{5, 6}. The effects of haptic feedback on the performance of robot-assisted cardiothoracic surgery have not been well characterized. Many successful robotic surgeons suggest that the lack of sensory force feedback can be adequately compensated for with the use of visual queues, such as local tissue deformation incurred with tension, retraction, or needle insertion. Conversely, we previously reported that sensory substitution, a provisional construct to approximate true haptic feedback, resulted in objective advantages during the performance of robot-assisted surgical tasks, including improved applied force accuracy and consistency⁵.

We hypothesize that haptic feedback confers quantifiable improvements in applied force and technical error metrics during robot-assisted surgical task performances. Here we use a modified da Vinci robotic system equipped with force-sensing instrument tips and real-time graphic overlays to study the effects of visual force feedback (VFF) on tying surgical knots with fine suture materials. We also sought to determine whether the effects of VFF differed between surgeons with and without prior experience operating with the da Vinci system.

Methods

da Vinci Surgical Robotic System

We used the da Vinci surgical system as our teleoperated robotic platform. The da Vinci surgical system consists of three major components: an input device (surgeon’s console), a computer interface, and an output device (patient-side manipulator).

da Vinci Instrument Modifications

Force sensors were added to detect the instruments’ bending forces. Two sets of strain gauges in full Wheatstone bridge arrangements were orthogonally placed along each of the instrument shafts to measure the forces being applied at each instrument’s tip (Figure 1). Customized “clamshell” cannulae were used to accommodate the modified instruments and minimize damage to the strain gauges. Simulations were performed to determine ideal strain gauge positions in relation to the instrument tip. The position and orientation of the da Vinci instrument tips and stereoendoscope were traced using robot kinematics. Due to the low-resolution encoders on the robot “setup” joints (i.e., the joints closest to the robot base that remain stationary during instrument manipulations) and flexure in the kinematic chain, we noted a discrepancy of up to three centimeters between the true position of the instrument tips and where the da Vinci Application Program Interface (API) software reports them to be. To correct this offset, a calibration was performed between each instrument and the stereoendoscope.

Fig 1.

Real-Time Visual Force Feedback Graphic Overlays

We developed a VFF system schematically represented in Figure 2. Frame grabbers are image acquisition computer boards that capture and store image data. In this system, frame grabbers capture real-time stereoscopic images from the da Vinci InSite video system and convert the video signals from analog to digital to be processed by the computer. A semi-transparent graphic overlay is superimposed on the instrument image displayed on the surgeon’s master console. This overlay is derived from force magnitude data obtained from the corresponding strain gauges and computed kinematics data from the da Vinci operational software (i.e., Application Program Interface).

Fig 2.

The applied forces at each instrument tip were calculated as the vector sum for F₁ and F₂, where F₁ and F₂ are the forces detected by the strain gauge bridges (Figure 1). A weighted average over a time interval of these measurements was used to smooth out high-frequency readings. The force magnitudes for each of the two force-sensing da Vinci instruments were graphically overlaid on the surgeon’s stereoscopic display console at 30 frames per second. The force display consists of two semi-transparent circles superimposed over the corresponding instrument tips as they move freely in space (Figure 3). These VFF overlays changed color across three discrete force magnitude intervals dictated by bending forces measured by the strain gauges on the instrument shafts. The force thresholds were set by an experienced da Vinci surgeon based on an average of the forces he exerted while performing this task:

• Minimal Force Range (green indicator): Minimal or no force was applied by the surgeon.

• Ideal Force Range (yellow indicator): Ideal applied force range for tying a secure knot.

• Excessive Force Range (red indicator): Excessive force with significant potential to break suture or damage tissue.

Fig 3.

Human Subjects

Ten surgeon subjects, comprised of three attending cardiac surgeons and seven postgraduate year VIII cardiothoracic surgical residents (Division of Cardiac Surgery, The Johns Hopkins Hospital, Baltimore, MD), participated in this study. Four of the surgeon subjects had completed a formal da Vinci training course and had substantial prior experience using the da Vinci system. These studies were performed under a protocol titled “Augmented Reality Displays for Robot-Assisted Surgical Systems” which was approved by the Johns Hopkins University Institutional Review Board on January 27, 2005.

Knot-Tying Task Trials

The surgical task consisted of securing the second throw of a 5-0 silk surgical knot on a standard knot-tying training board (SmithKline Beecham, SKB-10) with the modified da Vinci surgical system. To standardize the task, all surgeons were instructed to tighten the knot by (1) grasping the suture with the two forceps close to the knot and (2) pulling the suture laterally in the plane of the task board.

Each surgeon performed 20 knot-tying trials, ten with VFF and ten without VFF. The first trials with and without VFF were used as practice trials and were not included in the analysis. Hence, each surgeon subject performed a total of 18 trials. The order of trials was randomized. For each trial, the recorded parameters included instrument/environment bending forces, instrument tip position, suture breakage, knot security (i.e., secure versus loose knots), and task completion time.

Survey

A written qualitative survey was completed by each subject at the conclusion of the trials to assess whether or not VFF was preferred and the reason(s) for the preference.

Statistical Analysis

Using a significance level of p < 0.05, we conducted Chi-square analyses to determine whether significant differences in the number of broken sutures or loose knots existed between trials performed with and without VFF. Student’s t test analyses were used for peak force, force standard deviation, and task completion time comparisons. We also compared these performance variables between surgeons with and without prior operative experience with da Vinci system. Statistical computations were performed with Stata 9.2 statistical software (StataCorp LP, College Station, TX).

Results

Effect of VFF on Task Performance

In the knot-tying trials, there was only one suture breakage incident in 36 trials among da Vinci trained surgeons compared to 11 incidents in 54 trials among surgeons without prior da Vinci experience (2.8% versus 20.4%, p < 0.01 Chi-square) (Figure 4). Notably, VFF completely eliminated suture breakage among surgeons without prior da Vinci training.

Fig 4.

Among surgeons without da Vinci experience, significantly reduced peak applied forces (2.42 versus 3.12 Newtons, p < 0.01 T-test) and lower applied force standard deviations (0.64 versus 1.20 Newtons, p < 0.03 T-test) in knot-tying trials were associated with VFF. VFF did not affect any of these performance measures among surgeons with prior da Vinci experience.

Finally, VFF did not significantly reduce task completion times or the number of loose knots regardless of prior da Vinci experience. Task completion times among da Vinci trained surgeons with and without VFF were 6.04 versus 6.31 seconds, respectively. Among surgeons without robotic experience, completion times were 6.46 versus 6.62 seconds (p > 0.05 T-test), respectively. Among da Vinci trained surgeons, no loose knots were noted in 36 trials with VFF versus two out of 36 trials without VFF (0% versus 5.6%, p > 0.05 Chi-square). Among surgeons without prior da Vinci experience, three loose knots were noted among 54 trials (5.6%) performed with and without VFF.

Qualitative Survey Results

VFF was preferred by all subjects during the knot-tying trials largely due to the confidence it conferred in avoiding excessive applied forces while ensuring a secure knot.

Discussion

The computerized interface between the surgeon and robotic surgical instrumentation provides several advantages over manually-actuated thoracoscopic approaches in performing minimally-invasive operations. Increased dexterity, tremor filtering, and motion scaling afforded by these systems have greatly facilitated the minimally invasive performance of highly dextrous surgical tasks inherent in many cardiothoracic operations. Nevertheless, the physical separation between the operating surgeon and instrumentation in current teleoperated robot-assisted surgical systems, including the da Vinci, presently deprive the surgeon of noticeable force and tactile feedback during performance of conventional suture and tissue manipulation. The commercially available da Vinci system does have direct haptic feedback capabilities, but the magnitude of force fed back to the surgeon during typical surgical maneuvers is so low as to have negligible effect on perception and performance. The low quality feedback is due to inaccuracies in the estimation of tool/tissue interaction forces.

The clinical importance of haptic feedback in the conduct of robot-assisted operations is controversial and has prompted several recent studies⁷. During a blunt dissection task, Wagner et al. found that the presence of force feedback in a teleoperated system reduced the peak force magnitude by a factor of two and the number of errors by a factor of three versus no force feedback⁹. Wagner also concluded that these gains resulted in better user performance. Other studies have suggested that inadvertent tissue trauma is, at least in part, related to the lack of haptic feedback^{6, 10, 11}. Nevertheless, most published series of robot-assisted cardiac operations do not report heightened complications related to the lack of haptic feedback^{2, 12–14}.

Since direct force feedback has heretofore not been integrated into current commercial surgical robotic systems, recent experimental work has relied on a provisional construct to approximate true haptic feedback. Sensory substitution is the replacement of one sense, in this case direct tactile or force feedback, with another sensory modality that, here, relays to the surgeon a representation of the forces applied by the robotic telemanipulators. Our group previously compared applied force parameters during a standardized surgical knot-tying task under four different sensory-substitution scenarios: no feedback, auditory feedback, visual feedback, and combined auditory-visual feedback⁵. We found that either sensory substitution mode, alone or in combination, resulted in quantifiable improvements of applied force accuracy and consistency during traditional or robot-assisted task executions. Similarly, Prasad¹⁵, Sabatini¹⁶, and Tavakoli¹⁷ reported functional advantages in performing telemanipulations with sensory substitution. Unlike these early experimental sensory substitution systems that used static VFF overlay information, we developed a more sophisticated three-dimensional stereo VFF overlay that tracks and moves with the robotic instrumentation in real time. We believe that this is the first report of a VFF augmented reality system that is fully integrated into the da Vinci system. The tool tracking overlay allows surgeons to more accurately estimate the force magnitudes they are applying without having to divert their eyes away from the instruments. Based on our qualitative survey, all surgeons derived confidence or reassurance of appropriate force application from our VFF system and preferred performing the task with it on because it helped reduce the mental workload required to perform the task.

No significant differences in the measured performance parameters were noted between task execution with and without VFF among surgeon subjects with prior da Vinci experience. However, significantly fewer broken sutures and reduced peak and standard deviations of applied forces were observed among surgeons with little to no da Vinci experience. This observation suggests that VFF provides novice da Vinci surgeons the means to avoid excessive forces and to apply more consistent forces in this particular task performance. It is probable that experienced da Vinci surgeons were more adept at using natural visual cues to gauge suture tension than their less experienced colleagues.

Despite the differential results between da Vinci-trained and untrained surgeons, our qualitative subjective survey indicates that all surgeons preferred having VFF because it conferred confidence as an objective indicator of applied force magnitudes. The quantal green-yellow-red color scheme used to segment the different force ranges quickly and effectively conveyed force information to the surgeon. Although considered in the design process, we felt that more graded VFF schemes would be too distracting to the surgeon and too subtle to quickly alert the surgeon of excessive applied forces.

Interestingly, VFF did not significantly reduce the number of loose knots or task completion times. The lack of effect of VFF on knot security might be due to the fact that knot security can be readily determined visually without haptic feedback. Similar task completion times between trials with and without VFF might be explained by our observation that there was substantial variation between how each subject used the overlay. For example, in trials with VFF, some subjects released the suture exactly when the overlay indicated excessive tension while others varied tension between the ideal and excessive ranges to assure a secure knot.

In summary, we describe an instrument-tracking VFF system which is fully integrated into the da Vinci robotic system. Our results suggest that haptic feedback specifically moderates excessive force application and consistency during the performance of delicate repetitious robotic manipulation of fine sutures among surgeons who are new to the da Vinci system. Measurable benefits, however, were lacking among experienced users of the da Vinci system.