Initial Experiments with the Leap Motion as a User Interface in Robotic Endonasal Surgery

Abstract

The Leap Motion controller is a low-cost, optically-based hand tracking system that has recently been introduced on the consumer market. Prior studies have investigated its precision and accuracy, toward evaluating its usefulness as a surgical robot master interface. Yet due to the diversity of potential slave robots and surgical procedures, as well as the dynamic nature of surgery, it is challenging to make general conclusions from published accuracy and precision data. Thus, our goal in this paper is to explore the use of the Leap in the specific scenario of endonasal pituitary surgery. We use it to control a concentric tube continuum robot in a phantom study, and compare user performance using the Leap to previously published results using the Phantom Omni. We find that the users were able to achieve nearly identical average resection percentage and overall surgical duration with the Leap.

1 Introduction

Many surgical robots are operated using a teleoperation framework in which the user manipulates a master device that records his or her motions and relays them to a patient side (sometimes called a “slave”) robot that interacts with the patient. This teleoperation framework is the basis for the widely used da Vinci surgical system (Intuitive Surgical, Inc., USA) and is also found in numerous research systems [8, 25].

Current master devices for surgical robotic systems typically consist of mechanical linkages that include motors and encoders. Toward creating less expensive master interfaces that are not subject to mechanical wear, there has been increasing recent interest in hand tracking as an alternative approach [15, 16, 30, 26, 7]. This trend has been spurred by the recent introduction of low-cost, consumer-based tracking devices such as the Kinect (Microsoft, Inc., USA) and the Leap Motion (Leap Motion, Inc., USA). The Kinect has been used in a variety of hand tracking and teleoperation experiments (see e.g. [9, 10, 20]). It has also been productively used in rehabilitation [27]. But the suitability of optical hand tracking systems like the Kinect or Leap in surgical teleoperation remains an open question.

In this paper we explore the use of the Leap Motion controller in the specific surgical task of pituitary tumor resection. The Leap is an infrared, camera-based system designed to track hands, fingers, and tools. The controller is low cost ($79 as of this writing), has a stated accuracy of 1/100th of a millimeter [17], and is readily available. The research community has found numerous uses for the device in the two and a half years it has been available. For example, Potter et al. used the Leap in a sign language translation application [21]. Igor et al. used the built-in gesture recognition software to control a 3-finger gripper [31]. Bassily et al. used the Leap to control a 6-DOF Jaco robotic arm [3]. Sabir et al. used it to create an interface for visualization of three-dimensional molecular models [23]. There is some evidence that the Leap’s accuracy in dynamic settings (e.g. when the user’s hand is moving) is substantially less than the manufacturer’s 1/100 mm accuracy specification [15, 14, 28]. There is also some indication that touchless interfaces may have a shorter learning curve than touch-based or haptic interfaces [30]. Prior studies have typically focused on evaluating the accuracy and repeatability of Leap measurements in static and dynamic settings [15, 14, 28]. However, it is unclear how these accuracy and precision numbers translate to the feasibility of accomplishing a surgical task, due to the presence of the human surgeon in the loop. Thus, in our current paper we aim to experimentally test the Leap in a specific surgical task, with a specific slave robot. We compare the Leap against the Phantom Omni (formerly Sensable, Inc., USA, now the Geomagic Touch, Geomagic, Inc. USA), which was previously suggested to be a good example haptic device for comparison to the Leap [7]. It is worth noting that the Leap is at least an order of magnitude less expensive than the Phantom Omni at the time of this writing and that the Omni is one of the lower-cost examples of commercial haptic devices, overall.

Our surgical task of interest in this paper is pituitary tumor resections using a concentric tube robot delivered through the nose. This task is useful because we have prior data using the same robot and phantom model, but in which the Phantom Omni was used as the master [24]. There is ample clincal motivation for accomplishing transnasal pituitary resections. Pituitary adenomas account for 15–20% of primary brain tumors [2]. Endoscopic transnasal techniques are challenging for the surgeon, because of the constrained space inside the nostril, and the challenge of manually manipulating multiple tools in it to accomplish a delicate surgical procedure, motivating robotic assistance. Prior research on robotic systems for transnasal surgery has focused on image-guided drilling [18] and control of surgical endoscopes [19]. We have previously developed a complementary multi-arm robotic system specifically for the tumor resection portion of the surgery, which features needle-sized manipulators that can bend and elongate in a manner conceptually similar to tentacles (see Figure 1) [22, 13, 24, 5].

Fig. 1.

A concentric tube continuum robot made from three precurved, superelastic nitinol tubes. The robot can bend and elongate by rotating and translating the tubes inside one another. The inset line drawing shows the degrees of freedom. An angled ring curette typically used in hand tools for endonasal tumor removal is attached to the tip of the robot and can be axially rotated.

2 Experimental Setup

The robot used in our experiments is shown in Figure 2 and it delivers a concentric tube robot consisting of three telescoping, precurved tubes made from superelastic nitinol (see Figure 1 and [22, 11, 13] for further information on robots of this type). The robot used in this paper is identical to the one used in [24], and is described in more detail in [5]. It features three tubes with diameters of 2.4, 1.7, and 1.2 mm, each of which can be telescopically extended and axially rotated at its base. The robot also features an axial wrist, which enables the Hardy transsphenoidal curette (P/N SP0007011, Codman) mounted at its tip to rotate as shown in Figure 3.

Fig. 2.

The actuation unit used to translate and rotate the tubes. Each tube is grasped at its base and may be translated and rotated independently of the others, enabling the concentric tube manipulator to bend and elongate.

Fig. 3.

An illustration of the axial rotation ability of the tip mounted curette. Shown above are two superimposed images of it after a 180° rotation.

We implemented a damped least squares teleoperation approach as described previously [5, 24]. The Leap Software Development Kit (SDK) was used to measure the position of the center of the hand and the roll, pitch, and yaw angles about this point, and these were mapped directly to the tip of the concentric tube robot. When the user wished to move the robot without spinning the curette, the user simply adjusted the pose of his or her hand with fingers spread. When the user wished to re-orient the curette, the user adducted his or her fingers, in which case the roll angle (i.e. the angle about the vector aligned with the fingers) was mapped to the axial angle of the curette, causing it to rotate as shown in Figure 3. The hand movements were scaled down to reduce tremor and to enable precise movements in the small workspace of the pituitary gland. A standard 4 mm, 30°, endoscope provided visualization of the surgical site (i.e. the same endoscope used in [24]). The experimental setup was as shown in Figure 4.

Fig. 4.

Experimental setup for phantom tumor resection experiments.

The phantom tumor model was also the same as used in [24]. It was made from a 5:1 ratio of SIM-TEST (Corbin, Inc., USA) to water. This produces a phantom tumor that is similar to the consistency of a typical pituitary tumor [4]. This phantom tumor was placed inside an anatomical skull model (# A20, 3B Scientific, Germany), which had previously been prepared by the surgeon to closely replicate the enlarged sella as commonly found in pituitary tumor patients. The volume of the sella cavity in this model was approximately 6.92 cm³. During resections, both the endoscope and the skull were fixed in place, and manual suction was used to clean the curette from time to time, but not to directly remove phantom tissue from within the sella.

3 Experimental Results

Using the setup shown in Figure 4, co-author Philip Swaney and co-author and neurosurgeon Kyle Weaver performed a total of 16 phantom pituitary tumor resections. It is worth noting that while Philip Swaney was familiar with the robot in general, these were the first resections he had attempted with the robot. We measured the time required to complete the procedure and the quantity of tumor removed. The latter was measured by weighing the skull before introduction of the phantom tumor, weighing it again after introduction of the tumor, and weighing it a final time after resection. Figure 5 presents the results in terms of both time and resection percentage. The average percentage removal was 78.5 ± 10.9% and the average time to complete the removal was 12.4 ± 2.0 minutes. Figure 6 presents the same metrics, but as previously achieved in [24]. In that work, experienced surgeons performed resections on the same phantom, with the same slave robot, but using the Phantom Omni as the master interface. There, the average percentage removal was 79.8 ± 5.9% and the average time to complete the procedure was 12.5 ± 4.1 minutes.

Fig. 5.

Percentage removal and time to complete removal are shown for all 16 resections performed with the Leap. The average percentage removal and average time for removal are overlaid on the data. The trials performed by co-author Philip Swaney are shown in blue, and the trials performed by neurosurgeon Kyle Weaver are shown in green. The trials are listed in order of completion -i.e. the first experiment by Weaver had a lower resection percentage, indicating that some learning curve - but apparently not a long one - may be present.

Fig. 6.

Percentage removal and time to complete removal are shown for the twenty resections done with the Phantom Omni, and previously presented in [24]. The average percent removal and average time for removal are overlaid on the data.

4 Discussion and Conclusions

It remains somewhat controversial whether touchless master interfaces will ultimately be suitable for use with surgical robots. At least one group has concluded that they are not yet up to the challenge [15], and at least one other group has concluded that they are promising [7]. In this paper we have studied one particular surgical application in which an existing data set was available for comparison, and found that similar metrics were achieved with both a mechanical linkage and the Leap. This provides an indication that there are likely to be at least some surgeries where a touchless surgical robot interface may be suitable. It also remains to be seen if the loss of haptic feedback with a touchless interface is important. The da Vinci surgical robot has found commercial success without the use of haptic feedback, providing some indication that surgeons can adapt to visual feedback only. The ergonomics of using a touchless interface for surgical procedures is also an open ended research question, as fatigue is an important factor for any lengthy surgical procedure.

As was noted in [24], the 78% average tumor removal should be considered a successful outcome. This is because pituitary tumors are typically benign and slow growing, and the goal of the surgery is to decompress nearby structures like the optic nerve and corotid artery. Thus, it is not necessary to remove all of the tumor, and 100% removal is not achieved in many surgeries today. One study identified “definite tumor remnants or at least suspicious findings” in post-operative MRI scans in 42% of patients [12]. Furthermore, experienced surgeons believe that the results of this experiment are conservative, since the hydrostatic pressure in the brain was not modeled in our phantom. This pressure tends to push tumor material forward where it is easier to resect during a pituitary tumor surgery.

In the future we plan to improve both hardware and software of the system described in this paper. First, we are actively pursuing design of the actuation unit for sterilizability and biocompatibility. One example of a sterilizable actuation unit is given in [6], and we are developing similar concepts for application to a multi-arm system. Furthermore, we are developing miniature wrists to make the manipulators even more dexterous [29]. We are also considering the implementation of virtual fixtures to enforce “no fly zones” around sensitive anatomical structures [1].

In terms of Leap-based control, we are interested in studying learning curves for both novice and expert surgeons using our system. It will also be useful in the future to explore a wider range of surgical procedures and types of slave robot, so that general trends may be observed.

The results of our current study, while by no means a definitive statement on the value of hand tracking as a master interface, do illustrate at least one specific scenario where it appears to work sufficiently well for a human user to accomplish a surgical task. If this holds true in other contexts, in the future one will be able to confidently say that hand tracking is a viable solution for creating effective, lower cost, master interfaces for surgical robots.