Position-Based Virtual Fixtures for Membrane Peeling with a Handheld Micromanipulator

Abstract

Peeling delicate retinal membranes, which are often less than 5 µm thick, is one of the most challenging retinal surgeries. Preventing rips and tears caused by tremor and excessive force can decrease injury and reduce the need for follow up surgeries. We propose the use of a fully handheld microsurgical robot to suppress tremor while enforcing helpful constraints on the motion of the tool. Using stereo vision and tracking algorithms, the robot activates motion-scaled behavior as the tip reaches the surface, providing finer control during the critical step of engaging the membrane edge. A hard virtual fixture just below the surface limits the total downward force that can be applied. Furthermore, velocity limiting during the peeling helps the surgeon maintain a smooth, constant force while lifting and delaminating the membrane. On a phantom consisting of plastic wrap stretched across a rubber slide, we demonstrate our approach reduces maximum force by 40–70%.

I. Introduction

Membrane peeling is a common retinal surgery to remove epiretinal or internal limiting membranes (ILM) [1]. Epiretinal membranes are thin fibrous layers that grow on the retina over time or in conjunction with other retinal diseases. Because they distort the retinal surface, they are often referred to as “macular puckers” and cause straight lines to appear wavy. The ILM is a very thin layer that separates the vitreous from the retina and is often removed during the treatment of macular holes, a common condition where the vitreous pulls and tears the retina. To facilitate the closure of the macular hole, removal of the ILM around the hole is advocated to alleviate tangential pressure from the surrounding retina [2]. Removing these membranes involves a technique known as membrane peeling, a technically challenging procedure that sometimes requires minutes’ worth of attempts [1]. Unsuccessful peeling can result in poor visual outcome [3] and one study showed up to 50% of patients exhibited inadvertent injury and defects in the nerve fiber layer after performing ILM peel [4].

Using Micron [5], a fully handheld micromanipulator that is more fully described in Section II, we develop microsurgical aids for the surgeon during membrane peeling operations. For instance, tremor suppression filters out unintentional and involuntary hand motion, thereby avoiding fast, jerky movements that might tear the membrane or retina. In Section III, we also propose position-based virtual fixtures to provide finer control near the surface using motion scaling. A hard stop virtual fixture just under the surface prevents the application of excessive force. Likewise, velocity limiting during the membrane delamination helps maintain a smooth, even force. Section IV reports force results in a phantom involving peeling plastic wrap from a rubber slide. Conclusions and future work are discussed in Section V.

II. Background

In vitreoretinal microsurgery, membrane peeling involves three steps: engaging, lifting, and delaminating. During the engaging stage, the surgeon slips a bent blade or hook just under the membrane. The membrane is then lifted very slowly to prevent tears. Often this involves slight up and down movements while advancing the tool to tunnel under the membrane. Once an edge of the membrane has been partially lifted, the surgeon carefully delaminates, or peels, the membrane from the surface. See Figure 5 for an example of peeling on a phantom. High forces and thus high velocities should be avoided to avoid damage. Gupta et al. measured forces during vitreoretinal procedures to be under 7.5 mN [6] while Balicki et al. report typical tip velocities during membrane peels around 100–500 µm/s [7].

Fig. 5.

The membrane peeling procedure with plastic wrap on a rubber slide. First, the operator approaches with the tool (a) and engages the membrane by sliding the tip underneath the membrane (b–c). Lifting (d–e) involves a slight up and down movement as the edge of the membrane as it is separated from the surface. Finally, the operator delaminates (f–g) the membrane by pulling it away from the surface, resulting in a peeled membrane (h).

A. Existing Approaches

Various micromanipulators and robotic aids exist to help surgeons during vitreoretinal microsurgery, such as the Japan Eye Robot [8], CMU Micron [5], and JHU SteadyHand [9]. Designed specifically for retinal surgery, SteadyHand is a cooperatively controlled robot that shares control between the surgeon and a stiff, non-backdrivable robot arm. Using force feedback and scaling techniques, Balicki et al. demonstrated membrane peeling in a phantom, successfully limiting forces applied to the membrane [7]. However, the focus was on the delaminating step and did not include engaging the membrane; furthermore, large forces needed were needed to control the SteadyHand arm (~2.5 N), which caused fatigue and decreased precision.

B. Micron

To address issues in the delicate micromanipulation of retinal membranes, we propose to use our fully handheld micromanipulator, Micron [5]. Pictured in Figure 1, Micron is a unique 3 DOF micromanipulator with three actuators located between the handle and instrument end-effector. Arranged in a star-pattern, the piezoelectric actuators drive the tip, allowing for movement independent of the handle motion. A variety of behaviors have been demonstrated with Micron, including tremor suppression, motion scaling, and virtual fixtures [10]. The range of motion of Micron is about 2×2×1 mm centered on the handle (or null) position.

Fig. 1.

Fully handheld micromanipulator Micron, without housing to show the three pizeoelectric motors and LEDs for optical tracking.

Positioning information is obtained from ASAP, a custom optical tracking hardware system [11]. By triangulating three frequency-multiplexed LEDs on the shaft of the instrument and one LED on the handle, full 6 DOF pose information of both the handle and tip information can be calculated. Two Position Sensitive Detectors (PSDs) triangulate position at 2 kHz within a workspace of approximately 4 cm³. Measurement errors are less than 10 µm RMS, allowing for fast-rate, closed-loop position control of the manipulator.

C. System Setup

Micron is designed to aid microsurgical tasks under high magnification. In this paper, a Zeiss OPMI 1 surgical microscope with 29X magnification is used. A 27 guage hypodermic needle is attached to Micron and equipped with a bent 5 mil Nitinol wire to simulate a microvitreoretinal pick. The Nitinol wire is tapered and painted white to facilitate tracking by cameras. Two PointGrey Flea2 cameras attached to the microscope beamsplitters record the same view the surgeon sees. Capturing 1024×768 resolution images at 30 Hz with a pixel size of 3.5 µm, the cameras are used to reconstruct the surface and instrument tip locations. To measure force, all tests are performed on top of a load cell, which measures Z-force at 2 kHz and has a resolution of 0.3 mN RMS. Figure 2 shows the complete system setup.

Fig. 2.

Micron setup for membrane peeling. Micron (a) uses the ASAP optical sensors (b) for operation under a surgical microscope (c). A red rubber slide mounted on the load cell (d) is observed by the surgeon and the stereo cameras (e). An image injection system (f) overlays visual cues in the microscope eyepieces.

III. METHODS

We propose to aid micromanipulation of retinal membranes and facilitate peeling procedures by enforcing tremor suppression, multi-part vision-based virtual fixtures, and velocity limiting with Micron.

A. Tremor Suppression

Because tremor, the involuntary physiological motion of the human hand, is on the order of 100 µm [12], it is a chief source of difficulty when manipulating at sub-mm scales. Tremor suppression aims at reducing tremor components in hand motion, while allowing voluntary motions. We use two forms of tremor suppression, depending on the stage of the procedure. During the engaging and lifting steps, a first order low-pass Butterworth filter with a corner frequency of 1.5 Hz improves manipulation precision while leaving most of the manipulator range available for the more important depth-limiting virtual fixtures. During the delaminating phase, a more advanced low-pass shelving filter is used [5]. The shelving filter acts as a hybrid algorithm that also includes a relative motion scaling effect. It provides an extra 30–50% improvement in pointing accuracy over the basic low-pass filter [5].

B. Multi-Part Virtual Fixtures

While tremor suppression aims to improve micromanipulation accuracy by reducing tremor, this will not aid cases where the surgeon presses too deeply into the retina while trying to engage or lift the membrane. Assuming displacement to force proportionality (i.e. F = −kx), regulating the Z-depth excursions into the retina is a more viable solution for limiting retinal forces. We propose a multi-part virtual fixture control system: motion scaling near the retina and a hard stop at a maximum allowable depth.

1) Motion Scaling Virtual Fixture

During membrane engagement, it is desirable to scale all motion down by a fixed factor to increase manipulation precision. For instance, with a scaling factor of two, a 50 µm hand motion will yield a 25 µm motion at the tip of the instrument. Higher scaling factors afford more precision, but use more of the manipulator range; thus for a good balance, we choose a scaling factor of three. Likewise, to conserve manipulator range and ensure unimpeded motion parallel to the membrane, only motions normal to the surface are scaled. Ideally, motion scaling is activated just above the membrane to benefit the engaging and lifting steps.

2) Hard Stop Virtual Fixture

As a last-ditch effort to prevent excessive downward force, the tip can be stopped altogether from moving too deeply into the surface. This “hard stop” virtual fixture imposes a limit at a pre-defined particular Z-depth. The depth limit can be chosen based on the F = −kx relationship for a force that is deemed excessive. The hard stop can be maintained as long as the manipulator range is not exceeded. Because the apparent stop to downward motion breaks the eye-hand coordination loop, it is important to notify the surgeon that the hard stop has been reached. This can be accomplished audibly as in [7] or via some other cue; we discuss this issue in more depth in Section III.E.

3) Virtual Fixture Control

Since the measurements available to Micron are the 6 DOF poses of the tip and handle, all control of Micron is position-based. For instance, tremor suppression works by calculating a filtered goal position from the null (handle) position and performing position control to bring the instrument tip coincident with the calculated goal. Likewise, our formulation of virtual fixtures uses the input null position from the handle motion to drive the tip position in a way that is compatible with the intentions of the user, yet still complies with the virtual fixture. For instance, a hard stop virtual fixture defined by a surface can be enforced by projecting the null position onto the surface, thus finding a goal position that satisfies the virtual fixture while simultaneously being as close to the desired, human-indicated position as possible. Motion scaling around a surface is accomplished by projecting the null position to the surface, then linearly interpolating between the virtual fixture position on the surface and the null position indicated by the handle. Figure 3 depicts the relationship between various virtual fixtures. For details on the implementation of position-based virtual fixtures with Micron, see [10].

Fig. 3.

Illustration of varying levels of position-based virtual fixtures. As the surgeon moves the null position (c) below the virtual fixture boundary (a), the tip position (b) is influenced by the activated behavior. Without Micron, the tip position follows the null position. With a hard stop, the tip position follows the virtual fixture boundary regardless of the input null position. Motion scaling compromises between the fixture and human.

C. Displacement Measurements

Central to the virtual fixture control of limiting depth is the distance measurement from the tip of the instrument to the surface. Without good estimation of this displacement, the multi-part virtual fixture control will be less effective at providing helpful aids and thus limiting forces. To sense this displacement, we use dense stereo reconstruction and locally planar surface estimation.

1) Dense Stereo Reconstruction

Although a variety of sensing modalities exist to measure displacement from a surface, we find an inexpensive, yet effective method is dense stereo reconstruction using the stereo images from the cameras attached to the surgical microscope. While any sufficient discussion of dense stereo is too long for this paper, a review of stereo vision techniques may be found in [13]. For efficiency, we resize the camera images from 1024×768 to 640×480. We then rectify the images with a pre-computed homography obtained from automatically matching SIFT key points between the left and right images.

The Open Computer Vision (OpenCV) library is used for processing images and performing block stereo matching. Because the surface is known to be relatively smooth and continuous with limited local variations, a large block-size of 151 pixels is used. This also has the advantage of better rejecting camera noise/blur, the tip of the instrument, and local deflections of the surface caused by contact with the tip. Full dense reconstruction with 16 levels of disparity (and 4 fractional bits of precision) runs at 13 Hz. Figure 4 shows an example of the disparity depth map.

Fig. 4.

Left image of the stereo pair (a) and the corresponding disparity map (b). Note the disparity map has been exaggerated to illustrate the overall reconstruction; the maximum height difference in the surface is only a couple hundred microns. Figure best viewed in color.

2) Surface Estimation

Although displacement measurements could be made directly from the calculated disparity at the tip location, stereo matching noise and low frame rates are problematic for accurate and fast reactions. To remedy these problems, we first transform the calculated disparity into 3D world coordinates and then locally approximate the surface. To transform the disparity into the 3D world space used by the optical tracking, the stereo cameras are registered to the optical trackers with a 30 s calibration procedure. This initial registration is re-used for each run; nonlinearities are corrected with an adaptive least-squares calibration [14].

Locally approximating the surface immediately below the tip of the instrument has two advantages: less noise and less lag. We assume a locally planar surface representation, although higher fidelity models such as quadratic or nonparametric (mesh/b-spline) fits could also be employed. For additional robustness to outliers and noise in the disparity, outlier rejection is performed using 20 iterations of RANSAC [15]; the least-squares surface fit is then calculated from the resulting inliers. With a robust local surface representation transformed into 3D world space, displacements can be measured at 2 kHz with each new tip position update from the high-bandwidth optical trackers, resulting in high bandwidth control.

The precision of the surface estimation is 2.0 µm RMS. Any high frequency noise in plane estimation is smoothed by the tremor suppression filters. The combined dense stereo reconstruction and surface estimation code runs at 10 Hz.

D. Velocity Limiting

While the motion scaling and hard stop virtual fixtures aid during the engagement and lifting steps, a different mechanism is needed for the delaminating step when the tip is relatively far away from the surface. During delamination, very low tip speeds are desirable to avoid ripping the membrane. Assuming a velocity to force proportionality, we place limits on the velocity of the tip to avoid excessive upwards force. During delamination, velocities are capped at 100 µm/s. Combined with the shelving filter [5] that acts as relative motion scaling, very smooth, low velocity movements are possible.

E. Visual Cues

Because Micron has a limited range of motion, providing feedback to the user during operation about how much of the range of motion is being used is beneficial. In fact, when the eye-hand coordination feedback loop is broken, e.g. during velocity limiting or the operation of virtual fixtures, it is necessary to provide some indication of the error so the operator does not drift away from the goal unbounded. For example, when the hard stop is active, the tip is prevented from dipping below a preset Z-depth. However, from the perspective of the user, the tip stops for no apparent reason, leading the user to assume that more downward motion is needed. To prevent this, cueing is required to appropriately signal the operator. Our system displays two circular visual cues injected into the optics of the microscope. One circle displays the goal position and the other displays the null (or hand) position. Relative Z-positions are depicted by the radius of the circle representing the null position. We vary the circle color from green to red as an indication of how close the manipulator is to saturation. This approach keeps the surgeon informed of drift or when manipulator saturation is imminent.

IV. Experiment And Results

All experiments are performed on a phantom for repeatability. The phantom is a red rubber slide mounted to a load cell that measures vertical force during the entire operation. To simulate the membrane, Glad® ClingWrap (10–12 µm thick) is laid on top of the rubber and smoothed down using a Q-Tip to form a tight seal. The peeling procedure is then to engage the edge of the plastic wrap, lift, and pull it away from the rubber slide. Primary evaluation and criteria for success are force measurements.

A. Clamped Experiments

Several experiments were conducted with Micron clamped in a vise and attached to a manual micropositioner to verify the efficacy of various subsystems of the proposed system. First, to test the quality of force measurements, we applied a constant force to the load cell and calculated a precision of 0.3 mN RMS. To measure the combined precision of the dense stereo reconstruction, surface estimation, actuation, and force sensing, the tip of Micron was deflected 50 µm below the estimated surface. The RMS precision of the force under this scenario was measured to be 0.5 mN, indicating good system performance in a static environment.

To demonstrate the behavior of the depth virtual fixtures during the engaging and lifting step, we simulate a down and up maneuver by lowering the micropositioner holding Micron 600 µm. The multi-stage virtual fixtures consist of motion scaling that is activated automatically when the tip drops below 50 µm and a hard stop virtual fixture that prevents the tip from moving below −50 µm (which corresponds to about 4–5 mN of force with our rubber phantom). The tip behavior and force were evaluated under three conditions: tremor suppression only, tremor suppression with motion scaling, and finally including a hard stop. From Figure 6, we observe the desired behavior for both the motion scaling and hard stop virtual fixtures.

Fig. 6.

Displacement above the surface and force measurements during clamped experiments with three different behaviors: (a) tremor compensation, (b) tremor compensation + motion scaling below 50 µm, and (c) tremor compensation + motion scaling below 50 µm + a hard stop at −50 µm. In all cases, Micron was manually lowered and raised 600 µm using a micropositioner over a 10–15 s period.

B. Handheld Peeling Experiments

Finally to validate the entire system during handheld operation, 8 trials of membrane peelings were performed in the phantom under the surgical microscope by a novice. Two scenarios were tested: unaided without Micron and aided with the described system. As before, motion scaling is activated 50 µm above the surface and a virtual fixture hard stop 50 µm below the surface is enforced. The transition between depth-limiting virtual fixtures to the shelving filter with velocity limiting occurs when the system detects the membrane has been lifted. Currently, the system monitors force for a change from downwards to upwards force of a certain magnitude (2 mN) as the transition cue. However, computer vision could be used to detect the membrane being lifted in the disparity map. Throughout the experiment, force data was gathered at 2 kHz and passed through a bidirectional 50 Hz lowpass filter to remove noise. Figure 7 shows a typical unaided (without Micron) and aided (with Micron) trial. Notice the hard stop limiting the downward force and the velocity limits regulating the upwards force while the tremor suppression greatly reduces jerkiness. The maximum upwards and downwards forces averaged over the 8 runs are listed in Table 1.

Fig. 7.

Displacement and force results from a typical unaided trial (a) and aided trial (b) of membrane peeling. Shaded areas indicate which behaviors are active during the experiment. Note that since Micron is off in (a), the tip position is the null position. Figure best viewed in color.

TABLE I.

Maximum Upward/Downward Force During Peeling

Scenario	Trials	Downward Force (mN)	Upward Force (mN)
Unaided	8	11.4±1.5	8.4±1.3
Aided	8	3.0±1.1	5.8±0.9
% Reduction		73.6%	42.9%

Max forces on the phantom during unaided and aided peeling trials.

V. Discussion and Future Work

We have described a robotic system for retinal membrane peeling and demonstrated its effectiveness in regulating force in a realistic phantom by reducing forces by 40–70%. The novelty of our solution in aiding membrane peeling is three-fold. First, it utilizes a light-weight, fully handheld micromanipulator, avoiding the bulkiness of master/slave or cooperative robotic arms and capitalizing on surgeon’s familiarity with handheld tools. Second, we encompasses the entire membrane peeling procedure (engage, lift, and delaminate) on a phantom with similar force characteristics to the retina under a surgical microscope. Third, our approach uses no direct force feedback. Tremor suppression, position-based virtual fixtures derived from visual information, and velocity limiting modes are shown to provide not only smooth micromanipulation, but also sufficient information to regulate forces on the membrane, even in the absence of direct force feedback in the control loop. This is particularly useful because no modifications to existing tool tips or instrument attachments are required.

In the future, we would like to include more robust mechanisms for transitioning between behavioral aids, such as using the disparity for swapping to velocity limiting mode. Future work will also focus on extending the approach here into more realistic environments such as chicken egg membranes or ex vivo porcine retinal membranes.