Real-Time Sclera Force Feedback for Enabling Safe Robot-Assisted Vitreoretinal Surgery

Abstract

One of the major yet little recognized challenges in robotic vitreoretinal surgery is the matter of tool forces applied to the sclera. Tissue safety, coordinated tool use and interactions between tool tip and shaft forces are little studied. The introduction of robotic assist has further diminished the surgeon’s ability to perceive scleral forces. Microsurgical tools capable of measuring such small forces integrated with robotmanipulators may therefore improve functionality and safety by providing sclera force feedback to the surgeon. In this paper, using a force-sensing tool, we have conducted robotassisted eye manipulation experiments to evaluate the utility of providing scleral force feedback. The work assesses 1) passive audio feedback and 2) active haptic feedback and evaluates the impact of these feedbacks on scleral forces in excess of a boundary. The results show that in presence of passive or active feedback, the duration of experiment increases, while the duration for which scleral forces exceed a safe threshold decreases.

I. INTRODUCTION

Vitreoretinal surgery continues to be one of the most challenging surgical procedures as it requires micron scale tool manipulations and delicate tissue interactions in which physiological hand tremor may degrade the performance during the surgery. Over the last two decades, several kinds of robotic systems have been deployed to either eliminate or diminish the hand-tremor to provide more accurate surgical tool positioning. Wei et al. proposed a robotic assistant for microvascular stenting in ophthalmic retinal surgery [1]. Tanaka et al. proposed that robotic assistance could provide motion stability [2]. Towards intraocular automated surgery, Wilson et al. designed and evaluated a novel master-slave robotic surgical system (IRISS) [3]. Using a telemanipulation system, the worlds first robot-assisted vitreoretinal membrane peeling surgery was performed by surgeons at Oxfords John Radcliffe Hospital [4]. Though telemanipulated systems provide better ergonomics, it takes the surgeon away from the surgical site. A cooperatively controlled robotic system in which operator and robot share control of the surgery tool attached to the robot could potentially aim for this problem; Taylor et al. developed a cooperatively controlled SteadyHand Eye Robot (SHER) [5] (Fig. 1). A cooperative robotassisted retinal vein-cannulation (one of the most demanding tasks in vitreoretinal surgery) has recently been performed in in vivo porcine eyes for the first time [6]. Also, differentiated from table-mounted robotic systems, Riviere et al. have developed a light handheld piezo-actuated eye surgery device, called Micron, which is able to compensate for surgeon hand tremor [7].

Fig. 1.

Experimental setup showing: SHER, FBG interrogator for the force sensing tool, microscope for looking in to the eye phantom and speakers to provide the audio feedback.

Master-slave and cooperatively controlled robotic systems could provide precise tool motion, however the dominant stiffness and inertia of the robot prevents the surgeon from perceiving very small scleral forces (resulting from contact between the surgeical tool shaft and the sclera). This diminishes the surgeons ability to bimanually manipulate the eye and puts the sclera at risk for injury. To address these shortcomings, integrating force sensing capability in the tool shaft of microsurgical tools could provide necessary force feedback to the surgeon. Smits et al. developed a force and distance sensing needle utilizing Fiber Bragg grating (FBG) strain sensors and Optical Coherence Tomography (OCT) [8]. Also, our team developed a family of force-sensing tools based on FGB strain sensors [9], [10]. A novel eye surgery tool was developed by He et al., which is able to simultaneously measure the sclera force, tool tip force and the length of the tool inside the eye (insertion depth, Fig. 2) [11]. A similar tool has been developed for the current study.

Fig. 2.

Left figure is the dual tool with indicated FBG sensing zones and the right figure is a schematic of dual tool inside the eye

Sclera force information can be communicated passively to the surgeon e.g. via auditory feedback or alternatively can trigger active tool control mechanisms as in haptic force feedback. In a comparison-based study, Gonenc et al. reported performance improvement when audio feedback from tip force measuring tools was integrated with SHER and Micron [12]. In another study, Cutler et al. used auditory force feedback via FBG-enhanced tools [13]. Gijbels et al. have utilized their novel force sensing needle and robotic platform to investigate event-based force feedback for puncture detection during retinal vein cannulation [14].

In each of the above mentioned studies, the audio feedback was derived from the tool tip force only. However, the sclera force becomes more impactful as the surgical tool is always in contact with the sclera, while tool tip has intermittent contact with the retina. Thus, providing real-time sclera force feedback could potentially improve the scleral tissue safety. The focus of this study is therefore to provide subjects with various types of sclera force feedback in order to assess the influence on sclera safety. We hypothesize that providing either passive or active (audio or haptic) feedback may help limit forces that exceed a predefined safety threshold. To validate this hypothesis, four subjects (two clinicians and two non-clinicians) were asked to follow prescribed colored vessels on the retina inside an eye phantom with a forcesensing tool both with and without assistance from SHER for which the sclera force feedback (audio or haptic feedback) may be provided or withheld. .To the best of our knowledge, it is the first time that haptic feedback for sclera force is being provided during eye manipulation which is accomplished by modifying the control algorithm of the SHER. Finally, the results are compared to find out the efficiency of providing feedback.

II. MATERIALS AND METHODS

During robot-assisted eye surgery the surgeon’s perception of the sclera forces diminishes since tool-to-tissue contact forces are small compared to the robot inertia and stiffness. To restore the perception which is the focus of this study, we provide sclera force feedback, for which we need a surgical tool capable of quantitatively measuring the sclera force between the tool and the eyeball in a real-time manner. The other necessary equipments for this study include SHER and the eye phantom which will be elaborated in the following sections.

A. Dual Force-Sensing Tool

The sclera force sensor has to be able to pass through a 23 Ga or 25 Ga opening and to measure forces in the order of mN which are typical forces during eye manipulation. FBG optical strain sensors can address all of these constraints and were used in building these tools. As presented in Fig. 2, 3 optical FBG strain sensors are attached around the perimeter of a 25-gauge nitinol needle at 120° separation from each other. As explained in [11], by finding the related calibration matrices for this configuration of FBG sensors, we are able to relate the raw optical wavelength data of FBGs to the sclera force (F_s), tool tip force (F_t) and insertion depth (Fig. 2). The FBG fibers are connected to an optical sensing interrogator (sm130–700 from Micron Optics Inc., Atlanta, GA) which sends the FBG raw data with maximum frequency of 1 KH to the computer to calculate real-time force data (Fig. 1).

Fig. 3.

Validation results for sclera force and insertion depth

After calibrating the tool, we performed validation experiments to evaluate the accuracy of the calculated sclera force and insertion depth. By using the tool as a cantilever beam, 50 known sclera forces at known insertion depths were applied to the tool shaft and the force values were recorded by a very precise scale (Sartorius ED224S Extend Analytical Balance, Goettingen Germany). Then, the sclera force and insertion depth for those points were calculated using the calibration matrices. Fig. 3 depicts the calculated and real values for sclera force and insertion depth. The RSME for sclera force and insertion depth validation experiments are 1.2 mN and 0.5 mm, respectively (calculated with MATLAB).

B. Steady-Hand Eye Robot

The Steady-Hand Eye Robot (SHER) shown in Fig. 1 is a cooperatively controlled robot that is intended to reduce hand tremor. Both the surgeon and the robot simultaneously hold the tool, and by an impedance control enforced by the robot the operator performs a steady and tremor-free manipulation. The robot impedance control scheme sets the end-effector velocity to be proportional to operators contact force F_h (the interaction force between robot end-effector and operators hand which is measured by a 6 DOF force sensor attached to the end-effector).

C. Eye Phantom

The artificial eye phantom is made from Silicon. It is placed into a 3D-printed socket lubricated with mineral oil to produce realistic friction coefficient between the eye phantom and the socket. Also, to create an environment similar to a real eye surgery, we have utilized a ZEISS microscope to observe inside the eyeball during the experiments. A Point Grey camera (FLIR Systems, Inc.) is attached to the microscope for recording the user interaction with the eyeball. In addition, as shown in Fig. 4 colored vessels printed on a paper are attached into the eyeball.

Fig. 4.

Eye phantom with OD of 30 mm with colored vessels

D. Experimental Setup and Method

The focus of the study is to increase sclera force safety by providing feedbacks during eyeball manipulation. Therefore, as the first step we should define the safe range for sclera force. This is done in section III by looking at an expert’s behavior while manipulating the eyeball without a robot.

The experimental setup for conducting the experiments is depicted in Fig. 1. The dual force-sensing tool is attached to the robot with a tool holder and is cooperated by the subject and the robot. The interrogator receives the raw wavelength change due to the FBG fibers strain and transmit the data through TCP/IP communication to the computer. The computer calculates the real-time sclera force and sends back the relevant feedback (audio feedback through speakers or haptic feedback). The closed-loop system runs with a frequency of 200 Hz.

Four subjects including two engineers (subjects 1 and 2) and two clinicians, one beginner surgeon (subject 3) and one retinal surgeon with more than 20 years of experience (subject 4) participated in this study, and they were asked to look into the microscope and follow the colored vessels depicted in Fig. 4 while maintaining the tool tip as close as possible to the vessels without touching them. During the experiments all the sclera force, insertion depth and robot movement information were recorded. After inserting the tool into the eyeball through the sclerotomy hole, each subject went through the following steps:

• 1)Subject rotates the eyeball and brings the tool tip on top of the home position (Fig. 4) in the vertical view.
• 2)An assistant reads one random sequence of four vessel colors for the subject. (e.g. yellow, red, blue, green)
• 3)The subject follows the vessels with the tool tip with the order specified in step 2 without touching the vessels.
• 4)After following all vessels, the subject brings the eye to the home position similar to step one.

Each subject is supposed to perform the abovementioned procedure in five sets of experiment including:

• 1)Non-robot-assisted (Freehand) without feedback (No fb)
• 2)Robot-assisted without feedback
• 3)Freehand with audio feedback (Audio fb)
• 4)Robot-assisted with audio feedback
• 5)Robot-assisted with haptic feedback (Haptic fb)

Each subject is supposed to repeat each of these five sets of experiments for 10 trials with different sequence of colors.

E. Auditory Force Feedback

Audio feedback was provided to the participants for realtime force monitoring. Two speakers which are connected to the computer sound beeps with varying frequency depending on the measured sclera force level. As it is explained in the section III, the level of 120 mN of sclera force is determined to be detrimental. To have a gradual warning about the increasing sclera force, the subjects are provided with three levels of audio feedback. The speakers sound a low-pitch low-volume noise when the sclera force passes 80 mN. This level of noise is sustained until the force reaches 100 mN; thereafter, a noise with higher frequency and medium volume is played until the sclera force approaches 120 mN. After this force level, a very high frequency tone would be emitted continuously with a high-volume.

F. Haptic Force Feedback

To provide the haptic feedback for users, first we defined the general governing behavior of the robot. The admittance control of the robot is stated in eq.(1) in which the velocity for the SHER end-effector $\left({\dot{x}}_{des}\right)$ is proportional (by a constant factor of Λ) to the force being applied to the robot end-effector by the operator (F_h).

$${\dot{x}}_{des}=\Lambda F_h$$ (1)

In order to gradually increase the robot resistance as the sclera force approaches the upper safe value of 120 mN (the reason for choosing this number is elaborated in the next section), another variable exponential coefficient is applied to the right-hand-side of Eq. 1. This coefficient would be equal to one if the sclera force (F_s) is below a threshold F_start and will exponentially decrease from one and converge to zero as the sclera force increases. Thus, the new admittance control for F_s > F_start mN is indicated in Eq. 2.

$${\dot{x}}_{des}|=e^c\left(-F_s+F_{start}\right)\Lambda F_h$$ (2)

We have tuned the exponential power coefficient (constant c in Eq. 2) to be 0.1 by experimenting with various values for it and observing the robot behavior. After fixing this constant to 0.1, we have plotted the exponential gain for different values of starting force (F_start) : 60, 80 and 100 mN in Fig. 5. It is obvious that for the case of 60 mN starting point, the gain reaches zero when the sclera force reaches 120 mN (the upper safe limit for sclera force) and the robot will have a high resistance to movement for the sclera forces more than 120 mN. Thus, we chose F_start to be 60 mN to have a very high resistance to movement when the force finally reaches the upper safe limit. Aside from that, by choosing F_start to be 60 mN we would have a more gradual decrease in the exponential gain before the sclra force reaches the upper safe limit of 120 mN. It should be noted that the constant values in the exponential gain can be tuned for a more optimized and desirable robot manipulation.

Fig. 5.

Exponential gain for various starting points (F_start)

III. RESULTS

Before analyzing the results with MATLAB, we established a quantitative measure of safety for the scleral force by conducting some preliminary freehand experiments executed by the expert retinal surgeon (subject 4). That is because this set of experiment should be similar to what a surgeon usually does in a real vitreoretinal surgery and would be a reliable basis for safe manipulation. In Fig. 6 all the ten trials done in the preliminary experiments by the expert surgeon are plotted together. It is observed that the sclera forces are well below the 120 mN line (red dashed line in Fig. 6) for most trials. Although we do not know that this limit is really what will maintain the human sclera tissue in a safe zone, we have chosen this limit as an upper bound for safe sclera manipulation based on expert user data.

Fig. 6.

Sclera force distribution vs time for 10 trials of the preliminary experiments of the expert surgeon (the dark blue curve shows the average value of all experiments and the red dots show the chosen upper limit for safe sclera force)

As an example, in Fig. 7 the variations of sclera force for a single trial of each set of freehand experiments for one of the subjects are shown. In 8 the same plots for the robot-assisted experiments are presented. In contrary to Fig. 7 and 8 which are just one trial of each set of experiments, the sclera force averaged over all trials done in each set of experiments again for the same user is plotted versus the insertion depth in Fig. 9.

Fig. 7.

A trial for one set of all experiments done by one of the subjects in freehand experiments

Fig. 8.

A trial for one set of all experiments done by one of the subjects in robot-assisted experiments

Fig. 9.

Sclera force averaged over all 10 trials executed in each of 5 sets of experiments versus insertion depth for one of the subjects.

In Table 1 and Table 2, we can see the average results obtained from all of the robot-assisted and freehand experiments, respectively. Each table has 4 main columns. In the first main column, the total time (T) elapsed for that particular experiment is indicated. In the second column, the total time spent with forces exceeding the 120 mN (t₀) safe boundary is shown. The ratio of the quantities in the second main column to the corresponding value in the first main column is written in the third main column. This ratio shows the time percent of each experiment which is spent using detrimental forces. In the last column in each table, the median of sclera force (m) which is the most frequent sclera force in each experiment set is indicated. The standard deviation for time data for the average rows in Tables 1 and 2 are written in the parenthesis in each cell of those rows. In Table 2 there is no haptic feedback results since the haptic feedback can only be applied during the robot-assisted experiments.

Table 1.

RESULTS FROM ROBOT-ASSISTED EXPERIMENTS

	T = Total time elapsed average (s)			t0 = Time elapsed on force more than 120 mN (s)			100 × Ratio of $\frac{t_0}{T}$			m=Median of sclera force(mN)
	No fb	Audio fb	Haptic fb	No fb	Audio fb	Haptic fb	No fb	Audio fb	Haptic fb	No fb	Audio fb	Haptic fb
Subject 1	46.8	49.4	52.3	24.8	10.2	6.5	53.0	20.7	12.5	131	50	71
Subject 2	50.4	91.7	72.3	26.9	18.2	10.7	53.5	19.9	14.8	112	81	98
Subject 3	45.57	86.9	85.2	23.9	18.9	12.2	52.6	21.0	14.4	71	102	107
Subject 4	24.4	24.5	-	16.5	10.4	-	67.6	42.4	-	155	97	-
Average (std)	41.8(11.6)	63.2(29.6)	69.9(19.8)	23.1(7.6)	14.5(7.7)	9.8(7.6)	56.6(13)	26.2(11)	13.9(7.5)	117	82	92

Table 2.

RESULTS FROM FREEHAND EXPERIMENTS

	T= Total time elapsed average(s)		t0 = Time elapsed on force more than 120 mN (s)		100 × Ratio of$\frac{t_0}{T}$		m=Median of sclera force (mN)
	No fb	Audio fb	No fb	Audio fb	No fb	Audio fb	No fb	Audio fb
Subject 1	21.8	29.9	3.4	5.4	15.4	17.9	102	80
Subject 2	33.4	37.1	10.7	4.4	31.9	11.8	102	74
Subject 3	21.4	27.0	8.2	6.2	38.2	22.9	136	78
Subject 4	7.6	7.2	0.8	1.5	10.0	20.0	69	64
Average (std)	21.0(10.7)	25.3(13.1)	5.7(5.1)	4.4(3.7)	23.9 (15.7)	18.2(12.2)	102	74

The forth subject did not attend the haptic feedback experiments to completion because the process of training for the haptic feedback experiments was cumbersome and time-consuming.

IV. DISCUSSION AND CONCLUSION

As an overall conclusion from Figs. 7 and 8 which include a single trial of each experiment set, it is apparent that providing feedback has decreased the sclera force level, but the total time for those experiments have increased.

Based on the results in the last row of Table 1, the total time averaged over all users for the experiment with audio feedback (63.2s) and haptic feedback (69.9s) were increased relative to the no feedback case (41.8s). The p-values for both of these conclusions that 41.8 is statistically less than 63.2 and 69.9 are less than 0.001 (calculated using the IBM SPSS Statistics software). This should presumably be anticipated when executing an experiment with haptic or audio feedback because the user is continuously paying attention to the feedback and reacts to them. Therefore, instead of merely following the vessels and terminating the task, the subject has to reorient the tool to bring that to a straighter position and to decrease the sclera force when sensing an alert which results in spending more time in the feedback experiments. On the other hand, the time t₀ is increasing in the haptic feedback or audio feedback compared to the no feedback experiment (from 9.8s to 14.5s to 23.1s). The same decreasing trend is seen for the ratio of t₀ to T in the last row of Table 1. The p-value for all of these conclusions are less than 0.001. That could again be a result of the provided feedbacks which in turn the subject will try to bring down the force and thus spending less percent of time on damaging sclera forces during each experiment. Also, the average median of sclera force for the no feedback case is 117 mN which is close to the upper safety limit. It can be observed in Table 1 that this average value has decreased to below 100 mN after providing feedback.

On the other hand, by looking into the last row of Table 2 for the freehand experiments the total experiment time has not changed much (from 21.0s to 25.3s) when the audio feedback is provided. Our p-value analysis also shows that these two numbers are statistically the same. That is because in freehand experiments, the user has much more flexibility and degrees of freedom for manipulating the eyeball (relative to the robot-assisted case) and can rapidly handle the feedback. For the freehand experiments, we can also see that the audio feedback has not made a significant improvement in time t₀ (from 5.7s to 4.4s which are statistically the same based on our p-value analysis) because the upper limit 120 mN is high enough for freehand experiments and even without providing audio feedback, the subject generally does not reach the 120 mN limit. Moreover, for the expert surgeon the ratio of t₀ to T has even increased in the freehand experiments after providing the audio feedback (from 10% to 20%). As evidenced by this increased ratio, it seems auditory feedback in freehand experiments provides less guidance to the experienced subject who rely primarily on learned visual cues and force perception. Thus, providing feedback should be more impactful in robot-assisted experiments.

In Fig. 9, it is observed that higher sclera forces are applied in greater insertion depths. That is because the subjects first insert the tool and then bring the tool tip as close as to the vessels as required, and then move the eyeball to follow the vessels. When moving the eye, more sclera forces are applied. In addition, this figure also indicates that by applying the feedback in robot-assisted manipulations (red and blue curves), the average sclera forces are reduced to the same safe level as freehand experiments (black curve) where the user can perceive the sclera force. In other words, active and passive feedback provided in robot-assisted experiments can influence users at all skill levels to attain the same boundary safety levels as freehand experiments. This is further supported by comparing the average ratio of t₀ to T in Table 1 for feedback experiments (26.2% and 13.9%) and the same ratio for freehand experiments without feedback (23.9%). These numbers state that, it is possible to perform the robot-assisted eye manipulation as safe as or even safer than freehand experiments by providing audio or haptic sclera force feedback.

We can also make some subject-oriented conclusions by looking directly into the results of specific subjects. As mentioned before, subjects 3 and 4 are beginner and expert retinal surgeons, respectively. Another difference between these two clinicians is that subject 3 has been trained more with the robot before conducting the experiments, but subject 4 did not have enough robot trainings beforehand. As a result, it is apparent from the 3rd column in Table 1 that by providing feedback user 3 has made more improvements relative to subject 4. Also, in freehand experiments the total amount of time for the expert surgeon is remarkably less than other users. This means even by providing the feedback, the expert surgeon is performing the task fast and dexterously and at the same time maintaining the force in safe levels.

This preliminary study has created sentinel data that warrants further study with additional clinicians. The effect of higher levels of robot training will also be examined. Moreover, more efficient control algorithms and optimizing the gains in the current controller may be a future direction of further study in order to enhance robot utility for practical use while implementing safety parameters. Based on these experiments, we can conclude that providing feedback is influential on users of all levels of training and that in its most optimized form may diminish damaging forces during robotic and freehand eye manipulations. However, it was observed that it may not have enough promising improvement for even an expert surgeon with little trainings with the robot, and thus for more efficacy more trainings with the robot is required.