Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2005 Dec 20.
Published in final edited form as: Ind Rob. 2004 Dec;31(6):499–508. doi: 10.1108/01439910410566362

Methods for haptic feedback in teleoperated robot-assisted surgery

AM Okamura 1
PMCID: PMC1317565  NIHMSID: NIHMS2116  PMID: 16429611

Abstract

Teleoperated minimally invasive surgical robots can significantly enhance a surgeon’s accuracy, dexterity and visualization. However, current commercially available systems do not include significant haptic (force and tactile) feedback to the operator. This paper describes experiments to characterize this problem, as well as several methods to provide haptic feedback in order to improve surgeon’s performance. There exist a variety of sensing and control methods that enable haptic feedback, although a number of practical considerations, e.g. cost, complexity and biocompatibility, present significant challenges. The ability of teleoperated robot-assisted surgical systems to measure and display haptic information leads to a number of additional exciting clinical and scientific opportunities, such as active operator assistance through “virtual fixtures” and the automatic acquisition of tissue properties.

Keywords: Medical equipment, Robotics, Surgery

Introduction

Robot-assisted surgery is enhancing the ability of surgeons to perform minimally invasive procedures by scaling down motions and adding additional degrees of freedom to instrument tips. Thousands of general, urologic, and cardiac surgical procedures were performed worldwide in the last year with robotic surgical systems. Despite these successes, progress in this field is limited by an unresolved problem: the lack of haptic (force and tactile) feedback to the user. Current research at the Johns Hopkins University Haptics Laboratory and elsewhere seeks to correct this problem by providing the physician with feedback indicating the amount of force applied by the robot. This requires the integration of haptic sensors into the instruments used by surgical robots, as well as methods for displaying haptic information to the human operator. Once this is accomplished, a number of exciting clinical and scientific opportunities arise. Clinically, such feedback can improve a surgeon’s sense of telepresence, hopefully leading to better performance. The master telemanipulator can also use haptic feedback to provide intelligent assistants, generating “virtual fixtures” that support various manipulation tasks performed by the surgeon. There are many ways in which haptic data can be displayed (i.e. visually, aurally, or haptically), and many options for the type of assistance provided. Scientifically, a robot with haptic sensing and/or adaptive control can provide in vivo data sets for the modeling of tissue properties. There is burgeoning interest in tissue analysis for surgical planning, diagnosis and training, yet until now it has been very difficult to acquire realistic data.

It is important to distinguish between haptic, tactile, and force feedback. Haptics is a broad term used to describe both cutaneous (tactile) and kinesthetic (force) information. Both are necessary to form the typical sensations felt with the human hand. In this paper, we consider force feedback, where forces are resolved to a single point, and are displayed to the user through a tool. A haptic device such as the PHANTOM from SensAble Technologies (Woburn, MA) can provide this type of feedback. Tactile display devices are not yet commercially available, and are not likely to meet the size and weight constraints for multi-degree-of-freedom systems in the near future.

Our work involves several approaches to haptic feedback, representing operative technology that can provide significant improvements in patient outcomes, as well as lay the groundwork necessary to address several other exciting research issues. For example, as the field moves toward minimally invasive beating-heart (off-pump) procedures, sophisticated task and tissue models will assist in the creation of augmented reality systems that make a dynamic environment appear to be visually and haptically motionless. While we focus on operative technology, estimates of tissue properties generated by adaptive control laws could also apply to the design of more realistic surgical simulators. Such systems are already replacing unrealistic phantom and cadaver tissues, and eliminate the ethical challenges of using animal models. In such environments, skill can be objectively evaluated through calculation of applied forces and “virtual” functional outcomes.

This paper provides an overview of our work to date in the area of haptics for teleoperated surgery. It is organized as follows. First, we describe related work in the field of haptics for teleoperated surgical robotics, including existing clinical systems and results, teleoperation control, and virtual fixtures. Second, we demonstrate experimental results that underscore the importance of haptic feedback during the execution of surgical tasks. Third, we describe several methods for bilateral telemanipulation control and assistance modes, and examine the effects of different control limitations through theory and experiments. Fourth, we present some practical methods for providing haptic information to the surgeon when true bilateral control is not feasible. We conclude with a summary of our work and describe critical topics for future research.

Related work

Robot-assisted surgical systems

The clinical benefits of computer-augmented, robot-assisted surgery are derived from a minimally invasive approach with smaller incisions and reduced tissue trauma translating into reduced wound complications, patient discomfort, and hospital stays. Another putative benefit of robot-assisted surgery includes the potential for telesurgery, which would enable consultant expert surgeons to train or assist other surgeons without having to travel long distances or even to perform operations from afar in a battlefield scenario.

Robot-assisted surgery has enhanced the ability of surgeons to perform minimally invasive procedures in several ways. First, by filtering high-frequency signals, surgical tremor is eliminated. Second, the interface allows for motion scaling in which easy-to-perform macroscopic movements at the surgeon console are scaled down inside the patient, enhancing accuracy. Third, the robot control system permits the accurate translation of the surgeon’s hand motions to an endoscopic “wrist” placed within the chest cavity, conferring much higher degrees of freedom (dexterity) than traditional manually actuated endoscopic instruments. Finally, the system provides magnified, three-dimensional visual feedback. The da Vinci System (Figure 1, Intuitive Surgical, Inc., Sunnyvale, CA) is FDA approved for some cardiac procedures, such as mitral valve and atrial septal defect repair, and is undergoing clinical trials for others, such as totally endoscopic coronary artery bypass surgery (TECAB). Over 5,000 cardiac and general surgeries were performed with the da Vinci Surgical system in 2003. The master of the da Vinci is equipped to provide force feedback, although currently little to no feedback is provided.

Figure 1.

Figure 1

Open in a new tab

The da Vinci Surgical system

Notes: (a) Operating room setup with surgeon seated at the control console. The patient side manipulator consists of two manipulations arms and one camera arm. (b) The surgeon’s console with three-dimensional endoscopic viewer and hand controls. Images used with permission from Intuitive Surgical, Inc.

The literature describes the initial application of such systems to many abdominopelvic operations. An overview of the use of robotics in surgery is provided by Howe and Matsuoka (1999). Mohr et al. 2001 described their initial clinical experience with the da Vinci Surgical system, performing 35 single-vessel (left internal mammary-to-left anterior descending coronary artery) coronary artery bypass operations and 17 mitral repair operations using an endoscopic robot-assisted approach. Damiano (2000) conducted the first prospective clinical trial of robot-assisted endoscopic coronary artery bypass grafting using the da Vinci system, confirming the safety and efficacy of this approach. These early operations have been performed in carefully selected patients by a few surgeons who were intimately involved with the development of these systems in the laboratory and the operating room. Early follow-up suggests that cardiac operations performed with these robotic systems have achieved clinical success in terms of safety and efficacy. These successes must be qualified, however, in that significantly longer operative times are generally required.

The need for haptics in robot-assisted cardiac surgery

Much of our work focuses on suture manipulation in cardiac surgery because it presents a challenging environment with measurable functional outcomes. Forces applied to sutures are critical in creating knots that are firm enough to hold, but do not break fine sutures or damage tissue.

Early work with computer-enhanced robotic-assisted cardiac surgery represents a field in its infancy. Evidence strongly suggests that the ability to confer haptic feedback to present surgical robotic systems would contribute significantly to the safe performance of cardiac surgical procedures with these complex systems. The deficiency of haptic feedback in current robotic systems is a significant handicap in performing the technically more intricate and delicate surgical tasks inherent in specializations such as cardiac surgery. For example, suturing a coronary arterial anastomosis (surgical connection of severed arteries to form a continuous channel) with fine polypropylene suture is a dexterous task, where the surgeon typically uses his or her sense of touch to puncture tissue with a fine needle, pull the suture through, and tie and tighten knots. From our own observations of experienced and talented cardiac surgeons training with the da Vinci Surgical system, fine polypropylene sutures are often broken and delicate tissues torn due to the application of excessive forces conventionally attenuated with haptic feedback. The consequences of surgical errors or excessive delay with cardiac surgery (e.g. coronary microvascular or great vessel trauma, prolonged cardiopulmonary bypass) present much greater potential for irreversible injury, excessive hemorrhage, or even death of the patient. A common refrain in the current surgical literature with respect to the clinical applicability of these robotic systems include a rather steep learning curve and the lack of tactile sensation as major restrictions:

…This is a very demanding technology. At the beginning there is a learning curve, and one has to proceed very wisely, but the technology can be learned. The only limitation for the moment is loss of tactile feedback … (Mohr et al., 2001)

These limitations translate into generally longer, more technically challenging operations.

Telerobotics research

The robotics literature provides an abundance of theoretical and experimental work on telerobotic manipulation modeling and stability (Hannaford, 1989; Colgate, 1993), robust control (Kazerooni et al., 1993), time delays and telepresence (Hashtrudi-Zaad and Salcudean, 2002), scales (Yan and Salcudean, 1996), and hardware types. There have been only preliminary studies dedicated to the characterization of force feedback in teleoperated medical robotics. One study of palpation (Sherman et al., 2000) recommends a hybrid control law (position and force feedback) to provide the highest sensitivity to changes in environmental compliance. In a blunt dissection task (Wagner et al., 2002), it was found that force feedback reduces the number of errors that damage tissue by a factor of three. These results indicate that haptic information will be useful in some surgical tasks, although they do not analyze procedures involving dexterous manipulation such as suturing. A number of other researchers have addressed force feedback during teleoperation in both medical (Madhani et al., 1998; Hill and Jensen, 1998; Zemiti et al., 2004) and non-medical (Niemeyer and Slotine, 1998) domains, without methodically testing the effects of feedback on complex task performance. There have recently been some studies of complex manipulation task performance (O’Malley and Ambrose, 2003; Griffin et al., 2003) with and without haptic feedback.

Virtual fixtures

Recent work has also begun to examine the extent to which a robot can actively assist a surgeon. At one end of the spectrum, a robot can autonomously carry out some procedures (e.g. hip joint replacement and radiosurgery) (Kazanzides et al., 1995; Schweikard et al., 1993). At the other end of the spectrum, where most current systems lie (including the da Vinci), the systems only allow the robot to be explicitly controlled by the surgeon (Sheridan, 1992). Relevant to the proposed work are systems falling in the middle, where the robot is allowed some freedom to assist the surgeon. “Virtual fixtures” (Rosenberg, 1993), “active constraints” and “surgical macros” are all terms that have been used to describe interaction modes where the motions and/or forces of the robot are limited versions of those requested by the operator. These assistance modes allow the surgeon to remain in control of the procedure, while the robot adds safety and accuracy. Recent work by Howe and colleagues used virtual fixtures for robotic surgery to create a virtual wall to assist in blunt dissections with the ZEUS system (Park et al., 2001). Their virtual fixture reduced completion time by 27 percent and eliminated penetration into sensitive areas, but did not provide haptic feedback to the operator.

The role of haptic feedback in robot-assisted surgery

A major problem encountered by otherwise experienced and skilled cardiac surgeons in performing fine suture manipulations with robot-assisted surgical systems is excessive suture breakage, particularly during knot tying. Our first goal was to determine whether the lack of haptic feedback played a role in this problem. Therefore, we began characterizing the forces that should be applied to various grades of suture materials. We analyzed the suture forces applied during the first throw of a suture knot. A tension measurement device was constructed to sense left- and right-hand suture tensions (Figure 2).

Figure 2.

Figure 2

Open in a new tab

Suture tension was measured to determine the forces applied to various sutures: (a) by hand; (b) by instrument; and (c) using the da Vinci robot

The (non-medical) human subjects in this study included surgical residents and attending surgeons within the Division of Cardiac Surgery at the Johns Hopkins Hospital. In addition to providing data to determine the general level and resolution (temporal and magnitude) of force sensing needed for knot tying, the experiments served to compare the effects of different levels of haptic feedback. Subjects first performed hand ties, which provided the “standard” data for each subject. Hand ties provide complete haptic feedback to the subject, including both tactile and force feedback. Second, instrument ties mimicked the ideal force feedback (with no tactile feedback) that could be obtained through telemanipulation. Third, a da Vinci robot with no force feedback was used. For the hand and instrument ties, the subjects directly viewed the task, whereas the da Vinci stereo display was used for the robotically executed task. A paired student’s t-test was used to analyze the knot-tying data (Figure 3) with two hypotheses:

Figure 3.

Figure 3

Open in a new tab

Data summary for a single subject (attending surgeon, left hand). The forces applied to various sutures change with suture strength

  • H1. We hypothesized that the force magnitudes applied with the needle driver are indistinguishable from those applied by hand, while the forces applied with the robot are different from those applied by hand. This experiment sought to show that forces could be applied more accurately with resolved force feedback than without. Of the trials 63.3 percent showed that there is a difference between the instrument tie and the hand tie, and 73.3 percent of the trials showed that there is a difference between the robot tie and the hand tie. These results indicate that forces used for instrument ties are slightly better than robot ties, when the goal is to apply the same force as for hand ties. However, this difference is not large enough to conclude that accuracy would be improved to the level of hand ties with the inclusion of resolved-force feedback in a robot-assisted surgical system. Thus, tactile information or amplified force feedback is needed to maintain accuracy in the application of suture forces.

  • H2. We hypothesized that the coefficient of variation (CV) of force (standard deviation as a percentage of the average force level) for instrument ties is indistinguishable from hand ties. However, the CV for robot ties is different from that of hand ties. This experiment intended to demonstrate that forces could be applied with better repeatability with the instrument than with the robot. None of the subjects demonstrated a difference between instrument ties and hand ties, and 60.0 percent of the subjects demonstrated a difference between robot ties and hand ties. These results indicate that instrument ties provide a CV more similar to the hand ties than do the robot ties, and the hand tie had the lowest CV of all methods. Thus, repeatability would be improved with the inclusion of resolved force feedback in a robot-assisted surgical system.

Thus, the acquired data indicate that the force levels applied to the sutures are more consistently reduced with correspondingly finer sutures using a hand tie than using the robot, for both experienced surgeons and surgical residents/fellows. In addition, the standard deviation is much higher with the robot, indicating a lack of consistency compared to hand ties. Using the robot, both the attendings and the residents broke the finer polypropylene sutures on several occasions. Although we attribute the differences between the hand, instrument and robot ties entirely to the level of haptic feedback, it is possible that the indirect view during the robotically executed task could change task performance. However, the stereo display system is of sufficient speed and resolution that this effect is minor. We also note that, without haptics, surgeons are able to obtain a rough estimate of the force applied by the robot by visually observing the deformation of tissue when the knot was pulled away from the tissue. Further details are available in the work of Kitagawa et al. 2002. In summary, these experiments indicate that the haptic feedback is desired in robot-assisted surgical systems.

Bilateral telemanipulation: control laws and practical considerations

There exists a large set of previous work on telemanipulation with force feedback, but little on designing control for medical applications, especially in environments with changing contact conditions and impedances such as occur during suture manipulation. We have implemented several different types of telemanipulation control laws, which provide different capabilities for position, force, and environment impedance “transparency”. This section provides a brief review of control laws and their applicability to robot-assisted surgery.

The most common method for bilateral control is impedance control, whereby virtual impedance forces are used to connect the master and remote robot and cause them to track one another. The stiffness and damping of this impedance force, as well as system dynamics, time delays, sampling intervals, and position measurement resolution, determine the stability of such a system (Abbott and Okamura, 2003). If impedance control is used to cause the remote robot to track the master and vice versa, this is “position exchange control” and the operator will receive haptic feedback. Force sensors are not required, but the mechanical impedance of the remote robot will be transmitted to the operator and may mask any delicate forces arising from interaction with the environment. For the surgical robotics, the body wall and friction forces at the trocar (Figure 4) will be displayed to the surgeon, although these are not the forces we desire to display. If the remote robot tracks the master, but force sensors on the remote robot are used to determine the forces displayed to the operator, this is “position forward/force feedback control”. This approach is also problematic because of the large number of degrees of freedom (DOF) for force sensing required for manipulation (at least six DOF for position and orientation, plus an additional DOF for gripping). The da Vinci tools are disposable, making the cost of force sensing a significant drawback. In addition, since the tool may contact the environment at a number of locations, sensor placement must be chosen carefully. Biocompatibility and sterilization are also considerations.

Figure 4.

Figure 4

Open in a new tab

The body cavity and possible forces applied to the surgical tool

When the DOF of force feedback differ from the DOF of position control/sensing, the system is termed to have sensor/actuator asymmetry (Barbagli and Salisbury, 2003). At best, this limits the information provided and will thus feel strange to the operator. At worst, it can inject energy into the system, making it non-passive and potentially unstable. We have performed two experiments to determine the effect of limiting the DOF of force sensing for practical implementation. In the first experiment, we studied the effect of sensing-only (two-DOF) bending forces on a blunt dissection tool. Thus, the operator misses forces that sensed along the tool axis. In the second experiment, we looked at a phantom artery extraction task that uses gripping. We examined the role of grip force feedback in task performance. Both of the experiments were carried out with a pair of modified PHANTOM haptic interfaces configured for teleoperation.

In the blunt dissection experiment (Figure 5(a)), three different force feedback conditions were applied to a 3D teleoperation system: 3D force feedback, force feedback without the axial forces measured on the slave tool, and no force feedback. The task was also performed manually using a hand-held stylus. Results show that the absence of measured axial forces does not create a statistically significant difference in the level of applied forces, in comparison with complete 3D force feedback. In addition, this partial force feedback is a significant improvement over teleoperation with no force feedback. Surprisingly, this indicates that missing tool axis forces for such a task as blunt dissection is not significantly detrimental. Detailed analysis is provided by Semere et al. 2004. In the gripper experiment (Figure 5(b)), four different forcing conditions were presented: no force feedback, gripper force feedback only, translation force feedback only, and full force feedback. Initial experimental results show that gripping forces do not greatly affect user performance, but may influence the user’s confidence of their perception of the environment as shown by an increased number of errors when only gripper force feedback is used. Detailed analysis is provided by Verner et al. 2004. Both of these experiments will be performed with a higher-degree-of-freedom system such as the da Vinci in order to validate their applicability to robot-assisted surgery.

Figure 5.

Figure 5

Open in a new tab

(a) A blunt dissection task performed with a missing degree of freedom of force feedback along the tool axis. (b) An artery extraction task performed with a missing gripper force feedback

Because of the limitations of these basic types of impedance control, it may prove difficult to provide completely realistic haptic feedback. We can consider many methods to improve performance, given these constraints. One is adaptive control, both with and without force sensing. An adaptive controller can estimate mechanical properties of the environment (mass, stiffness, and damping) during telemanipulation (Hashtrudi-Zaad and Salcudean, 1996; Roy et al., 2002). One caveat of this method is that, because this type of adaptive model assumes a linear environment, the adapted environment properties will be constantly changing to reflect the new “local” linear environment properties. It is well known that tissue properties are nonlinear beyond very small deformations (Fung, 1993), which is likely to be the case during surgical procedures. Another method is to apply a pseudo-admittance control to the underlying impedance control system, which forces the operator to move slowly and accurately (Abbott et al., 2003). A very different approach is to use “virtual fixtures” to assist in surgical tasks.

The goals of traditional telemanipulator design all revolve around giving the user the highest possible control over the slave. In contrast, the goal of a virtual fixture is to remove some control from the user. Because these goals generally conflict with one another, it is not obvious how to best implement virtual fixtures on a telemanipulation system. We have considered two types of virtual fixtures: guidance and forbidden-region virtual fixtures. Guidance virtual fixtures are designed to work with admittance control systems, where the velocity of the manipulator is proportional to the force applied by the human operator. A guidance virtual fixture can be implemented by simply rejecting the force inputs in certain directions (Abbott et al., 2003). Forbidden-region virtual fixtures can be implemented with virtual springs, or through motion scaling. Predicting the passivity of the virtual spring method is related to the problem of stable simulation of haptic virtual environments (Abbott and Okamura, 2003). The motion scaling type attenuates or rejects inputs from the haptic interface at the remote robot. We have found experimentally that such operator assistance is most efficient when the operator receives some haptic feedback regarding the location and strength of the virtual fixture.

There are several important considerations for practical application of virtual fixtures in robot-assisted minimally invasive surgery, which are subjects of ongoing work. The first is appropriate virtual fixture geometry, which can be selected by the surgeon, through computer vision recognition, and/or modeling of tissue deformation. Second, virtual fixture assistance should be tuned based on the task at hand and the surgeon’s actions within that context. We are exploring methods for automatic tuning of virtual fixture strength based on Hidden Markov model recognition of operator motions. Finally, there is significant uncertainty in robot position relative to anatomical structures due to unmodeled dynamics. Robot design, modeling and control approaches are required to place accurate virtual fixtures.

Methods for sensory substitution

As described in the previous section, there are a number of practical challenges to true bilateral telemanipulation. Thus, we now explore the idea of substituting direct haptic feedback with visual and auditory cues. This eliminates stability issues and can be done with very limited force sensing. It has been shown in the teleoperation literature that sensory feedback substitution does enhance the ability of an operator to sense the environment and control the robot in non-medical applications (Massimino, 1995; Richard and Coiffet, 1995; Debus et al., 2001). Of note, most previous experiments on sensory substitution have not focused upon the magnitude of force generated by the user, but rather on performance in terms of time to completion.

In our experiments, we measured the tension applied to sutures during the first throw of a surgical suture knot by the left and right da Vinci instruments using a tension measurement device, the same as in Figure 2(c). Four different sets of conditions were studied; they differed only by the feedback methods implemented on the da Vinci and an additional data processing computer. The first scenario involved no feedback. The second scenario included an auditory feedback method (AF), which provided a single tone when the magnitude of the applied tension reached the “ideal” tension. The ideal tension is defined to be the average of the tensions incurred during hand ties performed by a series of cardiac surgical residents and attending staff. This value was obtained from our previous experiment (Figure 3). We note that in other work (Prasad et al., 2003), we used continuous frequency modulated audio feedback, but the surgeons preferred discrete audio signals in the already noisy operating room. The third scenario was a visual feedback method (VF), which provided a graphical display of the force levels. Two colored bars resized their height and shade according to the measured tension at the corresponding hand (Figure 6(a)). The fourth condition combined both auditory and visual feedback (AVF). In this final condition, auditory and visual feedbacks were provided simultaneously to the user. Five surgeons (three attendings and two residents) performed the hand and robotic ties. The attendings each had over 9 h of experience with the robot (tying sutures on phantoms). A total of 30 throws were recorded for each surgeon under each feedback scenario (hand, no feedback, auditory feedback, visual feedback, and combined feedback). Experiments to quantify the effect of the various sensory substitution scenarios addressed the following hypotheses.

Figure 6.

Figure 6

Open in a new tab

(a) An image of visual feedback observed by the surgeon through the da Vinci console, as used in the experiments. (b) Strain gages applied directly to da Vinci surgical instruments. (c) In ongoing work, we dynamically overlay the stereoimage with force information over the instruments for a more intuitive visualization

  • H1 (Accuracy).

    The first hypothesis proposes that the force magnitudes applied using any force feedback method more closely approximate the ideal suture tension than forces applied without feedback. First, we found a p-value of 0.021 for comparison between observations of hand ties and robotic ties without feedback, suggesting that the robot tie without feedback did not approximate the ideal tension as closely as hand ties. Second, we compared the means of the forces applied during ties executed with AF, VF, and AVF to the ideal tension within each suture type (n = 6) using Dunnett’s test. Our results indicate that the only forces achieved during robotic ties with some form of sensory substitution are comparable to the ideal forces for each suture type, thereby confirming this hypothesis.

  • H2 (Precision).

    The second hypothesis proposes that the coefficients of variance (CV) of forces for ties executed with sensory feedback substitution are indistinguishable from those of the hand ties. The CV for each suture type was analyzed (n = 6) using Dunnett’s test. We compared the average CV of the hand ties to the average CV of the robot ties with three sensory substitutions and no feedback for each suture type. Statistical analysis led us to conclude that this hypothesis is not completely satisfied; precision would be improved with the inclusion of VF and AVF (which provide continuous force information), but not AVF, in knot-tying with a robot-assisted surgical system. Figure 7 shows the coefficient of variance for the various feedback methods and suture types.

  • H3 (Improvement in Precision).

    The third hypothesis proposes that feedback improves the performance of subjects in comparison to the robotic ties with no feedback. The effect of AF and VF on the improvement in precision was examined. The correlation between suture type or subject and average CV value was not found to be statistically significant. An improvement of 50.2 percent was recorded when AF was provided; there was an 84.1 percent improvement for VF. Using Duncan’s multiple range test, we also found that when VF is present, additional AF did little to improve precision. Thus, visual and auditory feedback appeared to improve performance precision, although visual feedback was more effective.

Figure 7.

Figure 7

Open in a new tab

Comparing the coefficient of variance of various feedback methods to that of the hand data. The error bar corresponds to the critical difference for Dunnett’s multiple range test

All of the hypotheses were partially satisfied in that user performance (both accuracy and precision) of robotic ties with sensory substitution of forces was comparable to performances during hand ties, if not better. The coefficient of variance for the robotic ties under visual feedback was found to be lower than that of the hand ties. The dominant effect of visual feedback is evident from the relative effect of visual and auditory feedback. This is likely due to the fact that our subjects were provided with continuous information with our visual feedback mechanism, whereas our auditory feedback method signaled the user only when the ideal tension was reached. Further details are available in the work of Bethea et al. 2004.

We are currently extending this work to use limited (two-DOF) force sensing on teleoperated surgical instruments and a dynamic augmented reality system. This system displays a visual representation of forces to the surgeon in real time. Our methodology includes three engineering components and a validation study:

  1. Strain gauges are applied to the lower shaft of a needle driver tool of the da Vinci Surgical System (Figure 6(b)). Two pairs of gauges are configured in a full bridge arrangement to maximize sensitivity with a linear response and provide measurement of bending forces.

  2. The orientation and position of the tool tips and endoscope for the da Vinci are tracked using robot kinematics. The tool data is then registered to the endoscope data via computer vision. This registration will allow us to superimpose force feedback signals from the strain gauges over the corresponding instrument tip as it moves freely in space.

  3. Our current approach to visual feedback is a bar graph overlaid on each tool (Figure 6(c)), whose relative dimensions and color change in response to the forces measured at the tool tip.

  4. Validation with a number of tasks and phantom and animal models can now be carried out, since the sensing is on the tools rather than a specialized tension measurement device.

Summary and future work

In summary, our work has focused on the following topics: characterization of the degradation in performance without haptic feedback, methods for telemanipulation control, and experimental analysis of the effectiveness of sensory substitution as a practical method for haptic feedback.

There are a number of critical topics for future work in this field. First and most significant is force sensing. For true bilateral telemanipulation, forces must be measured accurately and in many DOF. Yet they must be inexpensive, biocompatible and sterilizable. Second, when perfect force sensing is not available, we must consider sensor/actuator asymmetries. Although we have examined this problem from an experimental approach, there is no framework for theoretical analysis. The lack of passivity in systems with sensor/actuator asymmetry requires further study to guarantee safety for clinical application. Third, complete haptic feedback requires both force and tactile information. Even perfect bilateral telemanipulation will not approximate the skill with which some tasks can be performed by hand, because of the distributed and frequency-dependent nature of human tactile sensors (mechanoreceptors). The combination of force and tactile feedback, as well as design of new tactile sensors, is thus an important area for future work. Fourth, virtual fixtures, as described earlier, have great potential to provide low-level assistance to the surgeon. While many of the control issues have been addressed, a priori knowledge, medical images and other sensing methods are required to determine appropriate virtual fixture geometry. Finally, tissue modeling based on sensed forces during surgery can be used to detect anomalies, provide diagnosis, and create realistic surgical simulations. This requires the development of sophisticated tissue models that can be populated with force, position and image data, possibly in real time.

Footnotes

This paper describes work resulting from a number of collaborations. Past and current engineering students Jake J. Abbott, Takintope Akinbiyi, Daniell Dokko, Kristin Jeung, Masaya Kitagawa, Wagahta Semere, and Lawton N. Verner contributed significantly to this work. Brian T. Bethea, Vincent L. Gott, Mark A. Talamini, and David D. Yuh at the Johns Hopkins Medical Institutions, provided insight regarding clinical applications and facilitated experiments. This work was supported by NSF grants EEC-9731478 and ITR-0205318, NIH grant R01 EB002004, and Whitaker Foundation grant RG-02-911.

References

  1. Abbott, J.J. and Okamura, A.M. (2003), “Virtual fixture architectures for telemanipulation”, Proceedings of the IEEE International Conference on Robotics and Automation, Taipei, September 2003, pp. 2798–805.
  2. Abbott, J.J., Hager, G.D. and Okamura, A.M. (2003), “Steady-hand teleoperation with virtual fixtures”, paper presented at the 12th IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN), San Francisco, CA, November 2003, pp. 145–51.
  3. Barbagli, F. and Salisbury, K. (2003), “The effect of sensor/actuator asymmetries in haptic interfaces”, Proceedings of the 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Los Angeles, CA, March 2003, pp. 140–7.
  4. Bethea BT, et al. “Application of haptic feedback to robotic surgery”. Journal of Laparoendoscopic and Advanced Surgical Techniques. 2004;14(3):191–5. doi: 10.1089/1092642041255441. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Colgate JE. “Robust impedance shaping telemanipulation”. IEEE Transactions on Robotics and Automation. 1993;9:374–84. [Google Scholar]
  6. Damiano RJ. “Editorial: endoscopic coronary artery bypass grafting – the first steps on a long journey”. The Journal of Thoracic and Cardiovascular Surgery. 2000;120:806–7. doi: 10.1067/mtc.2000.109542. [DOI] [PubMed] [Google Scholar]
  7. Debus T, et al. “Multichannel vibrotactile display for sensory substitution during teleoperation”. Proceedings of SPIE – The International Society for Optical Engineering. 2001;4570:42–9. [Google Scholar]
  8. Fung, Y.C. (1993), Biomechanics: Mechanical Properties of Living Tissues, Springer Verlag, New York, NY.
  9. Griffin, W.B., Provancher, W.R. and Cutkosky, M.R. (2003), “Feedback strategies for shared control in dexterous telemanipulation”, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, NV, October 2003, Vol. 3, pp. 2791–6.
  10. Hannaford B. “A design framework for teleoperators with kinesthetic feedback”. IEEE Transactions on Robotics and Automation. 1989;5(4):426–34. [Google Scholar]
  11. Hashtrudi-Zaad, K. and Salcudean, S.E. (1996), “Adaptive transparent impedance reflecting teleoperation”, Proceedings of the IEEE International Conference on Robotics and Automation, Minneapolis, MN, April 1996, pp. 1369–74.
  12. Hashtrudi-Zaad K, Salcudean SE. “Transparency in time-delayed systems and the effect of local force feedback for transparent teleoperation”. IEEE Transactions on Robotics and Automation. 2002;18(1):108–14. [Google Scholar]
  13. Hill JW, Jensen JF. “Telepresence technology in medicine: principles and applications”. Proceedings of the IEEE. 1998;86(3):569–80. [Google Scholar]
  14. Howe RD, Matsuoka Y. “Robotics for surgery”. Annual Review of Biomedical Engineering. 1999;1:211–40. doi: 10.1146/annurev.bioeng.1.1.211. [DOI] [PubMed] [Google Scholar]
  15. Kazanzides P, et al. “An integrated system for cementless hip replacement”. IEEE Engineering in Medicine and Biology Magazine. 1995;14(3):307–13. [Google Scholar]
  16. Kazerooni H, Tsay TI, Hollerbach K. “A controller design framework for telerobotic systems”. IEEE Transactions on Control Systems Technology. 1993;1(1):50–62. [Google Scholar]
  17. Kitagawa, M. et al. (2002), “Analysis of suture manipulation forces for teleoperation with force feedback”, Proceedings of the Fifth International Conference on Medical Image Computing and Computer Assisted Intervention – MICCAI, Tokyo, October 2002, Springer Lecture Notes in Computer Science, Vol. 2488, pp. 155–62.
  18. Madhani, A.J., Niemeyer, G. and Salisbury, J.K. (1998), “The black falcon: a teleoperated surgical instrument for minimally invasive surgery”, Proceedings of the IEEE/RSJ International Conference on Intelligent Robotic Systems, Victoria, October 1998, Vol. 2, pp. 936–44.
  19. Massimino MJ. “Improved force perception through sensory substitution”. Control Engineering Practice. 1995;3(2):215–22. [Google Scholar]
  20. Mohr FW, et al. “Computer-enhanced ‘robotic’ cardiac surgery: experience in 148 patients”. The Journal of Thoracic and Cardiovascular Surgery. 2001;121:842–53. doi: 10.1067/mtc.2001.112625. [DOI] [PubMed] [Google Scholar]
  21. Niemeyer, G. and Slotine, J-J. (1998), “Towards force-reflecting teleoperation over the internet”, paper presented at the IEEE International Conference on Robotics and Automation, Leuven, May 1998, Vol. 3, pp. 1909–15.
  22. O’Malley MK, Ambrose RO. “Haptic feedback applications for robonaut”. Industrial Robot. 2003;30(6):531–42. [Google Scholar]
  23. Park, S., Howe, R.D. and Torchiana, D.F. (2001), “Virtual fixtures for robotic cardiac surgery”, paper presented at Fourth International Conference on Medical Image Computing and Computer-Assisted Intervention, Utrecht, October 2001, pp. 1419–20.
  24. Prasad, S. et al. (2003), “A modular 2-DOF force-sensing instrument for laparoscopic surgery”, Proceedings of the Sixth International Conference on Medical Image Computing and Computer Assisted Intervention – MICCAI, Montreal, November 2003, Springer Lecture Notes in Computer Science, Vol. 2878, pp. 279–86.
  25. Richard, P. and Coiffet, P. (1995), “Human perceptual issues in virtual environments: sensory substitution and information redundancy”, Proceedings of the IEEE International Workshop on Robot and Human Communication, Tokyo, July 1995, pp. 301–6.
  26. Rosenberg, L.B. (1993), “Virtual fixtures: perceptual tools for telerobotic manipulation”, Proceedings of IEEE Virtual Reality International Symposium, Seattle, September 1993, pp. 76–82.
  27. Roy, J., Rothbaum, D.L. and Whitcomb, L.L. (2002), “Haptic feedback augmentation through position based adaptive force scaling: theory and experiment”, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Lausanne, September 2002, Vol. 3, pp. 2911–19.
  28. Schweikard, A., Adler, J.R. and Latombe, J-C. (1993), “Motion planning in stereotaxic radiosurgery”, Proceedings of the IEEE International Conference on Robotics and Automation, Atlanta, GA, May 1993, Vol. 1, pp. 909–16.
  29. Semere, W., Kitagawa, M. and Okamura, A.M. (2004), “Teleoperation with sensor/actuator asymmetry: task performance with partial force feedback”, Proceedings of the 12th Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, Chicago, IL, March 2004, pp. 121–7.
  30. Sheridan, T.B. (1992), Telerobotics, Automation, and Human Supervisory Control, MIT Press, Cambridge, MA.
  31. Sherman, A., Cavusoglu, M.C. and Tendick, F. (2000), “Comparison of teleoperator control architectures for palpation task”, Proceedings of the ASME Dynamic Systems and Control Division (Haptics Symposium), Orlando, FL, November 2000, Vol. 69-2, pp. 1261–8.
  32. Verner, L.N., Jeung, K.A. and Okamura, A.M. (2004), “The effects of gripping and translational forces on teleoperation”, Proceedings of the Workshop on Multi-point Interaction in Robotics and Virtual Reality, IEEE International Conference on Robotics and Automation, New Orleans, LA, April 2004 (in press as a chapter in the upcoming Springer book Multi-point Interaction in Robotics and Virtual Reality.).
  33. Wagner, C.R., Stylopoulos, N. and Howe, R.D. (2002), “The role of force feedback in surgery: analysis of blunt dissection”, Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environments and Teleoperator Systems, Orlando, FL, March 2002, pp. 68–74.
  34. Yan J, Salcudean SE. “Teleoperation controller design using H∞-optimization with application to motion-scaling”. IEEE Transactions on Control Systems Technology. 1996;4(3):244–58. [Google Scholar]
  35. Zemiti, N. et al. (2004), “A force controlled laparoscopic surgical robot without distal force sensing”, Proceedings of the International Symposium on Experimental Robotics, Singapore, June 2004.

RESOURCES