Stochastic Force-based Insertion Depth and Tip Position Estimations of Flexible FBG-Equipped Instruments in Robotic Retinal Surgery

Abstract

Vitreoretinal surgery is among the most delicate surgical tasks during which surgeon hand tremor may severely attenuate surgeon performance. Robotic assistance has been demonstrated to be beneficial in diminishing hand tremor. Among the requirements for reliable assistance from the robot is to provide precise measurements of system states e.g. sclera forces, tool tip position and tool insertion depth. Providing this and other sensing information using existing technology would contribute towards development and implementation of autonomous robot-assisted tasks in retinal surgery such as laser ablation, guided suture placement/assisted needle vessel cannulation, among other applications. In the present work, we use a state-estimating Kalman filtering (KF) to improve the tool tip position and insertion depth estimates, which used to be purely obtained by robot forward kinematics (FWK) and direct sensor measurements, respectively. To improve tool tip localization, in addition to robot FWK, we also use sclera force measurements along with beam theory to account for tool deflection. For insertion depth, the robot FWK is combined with sensor measurements for the cases where sensor measurements are not reliable enough. The improved tool tip position and insertion depth measurements are validated using a stereo camera system through preliminary experiments and a case study. The results indicate that the tool tip position and insertion depth measurements are significantly improved by 77% and 94% after applying KF, respectively.

I. Introduction

THE prevalence of Retinal Vein Occlusion (RVO) has been reported to be the second most common retinal vascular occlusive disease after diabetic retinopathy in the elderly [1]. Branch RVO (BRVO) or central RVO (CRVO) can lead to horrendous visual outcomes and are often accompanied by blurred and distorted vision [2]. On the other hand, retinal microsurgery remains one of the most demanding surgical procedures, involving delicate tasks and ultra-fine tissue manipulations. Among the previously proposed but not yet practical clinical procedures to treat RVO is retinal vein cannulation (RVC) followed by local infusion of a clot-dissolving tissue plasminogen activator (t-PA) directly into the occluded vein [3]. Available therapeutic targets include occluded veins with a diameter of approximately 80 μm or greater. The skills required to perform the relevant micro surgical tasks in RVC are at the limit of human motor capabilities. For an ophthalmic surgeon, a root mean squared (RMS) of 182 μm was measured for tremor amplitude [4], which is significantly larger than the retinal vessels diameters (i.e., between 50 − 150 μm [5]). Furthermore, surgeons are not able to perceive the interaction forces at the tool tip (below 10 mN [6]), which forces them to rely primarily on visual input (microscope view) to closely monitor the surgery and its safety. However, the microscope view provides a limited depth perception and does not cover the entire field of view. Therefore, a surgeon’s physiological hand tremor, limited motor control, and degraded manual dexterity which all might be intensified by visual, physical, and mental fatigue may put the eye at high risk of sight-damaging iatrogenic trauma.

A technically feasible remedy to these challenges is robot-assisted retinal surgery. The benefits of robotic intervention include but are not limited to enhancing tool tip position accuracy, suppressing involuntary hand tremor, and provision of various sensing capabilities during the surgery. Robots developed for the purpose of microsurgery can be classified into two main groups of collaborative and tele-operated systems. In the collaborative category, surgeon and robot synergistically share control of the surgical tool. The Steady-Hand Eye Robot (SHER), developed at the Johns Hopkins University, is an example of a collaborative robot [6], [7] (Fig. 1). Other examples of collaborative robots are referenced in [8], [9]. Furthermore, literature documents other studies (e.g., [10]–[16]) representing recently developed tele-manipulated robots for eye surgery. The most clinically advanced use of robots in eye surgery to date is attributed to the first in-human robot-assisted eye surgeries by Edwards et al. [17] and Gijbels et al. [18].

Fig. 1.

SHER and its five degrees of freedom. The first three axis indicate the translation motion and the last two axis represent the rotational motions around the associated axis. The handle frame is shown in the close-up view of the robot end-effector.

A review of the literature indicates that the focus of the above-mentioned robots have been mostly on hardware development (mainly for providing a tremor-free manipulation) while less attention has been paid to sensing and control, which are required for a reliable and safe robotic assist. In other words, the tool-to-eye interaction parameters including sclera forces (F_sx and F_sy), tool insertion depth (d), and the tool tip position (P), which are depicted in Fig. 2, need to be autonomously and safely handled by the robot. For instance, in a robot assisted RVC procedure, the sclera forces must always be maintained in safe ranges by the robot [19] and/or the tool tip should be restrained from inadvertent tip-to-retina contact. To autonomously keep the sclera forces and insertion depth in safe ranges, Ebrahimi et al. [19], [20] implemented an adaptive control strategy on the SHER using a Fiber Bragg Grating (FBG)-equipped surgical instrument. In this work, however, the inconsistent FBG-based estimations for insertion depth, which occurs in the low-magnitude sclera forces [21], weakens the control performance. Of note, this may result in a poor insertion depth control and subsequently endangers patient safety. In another study performed by He et al. [22], [23] on the SHER, a deep learning approach was utilized to predict unsafe sclera forces in order to act proactively and ensure safety. The limitations of this method, however, is the large data required for training and also unreliable predictions due to network errors for untrained tool-retina interactions.

Fig. 2.

Schematic model for a deflected surgical tool inside the eyeball during a robot-assisted retinal surgery showing the sclera forces (F_sx and F_sy both perpendicular to the tool shaft), tool insertion depth (d), the tool tip (P), tool tip frame and the base frame. The dashed box shows the tool projection on the x-z plane of the tool frame.

Besides using FBG-equipped instruments, computer vision algorithms have been used for robotic eye surgery [24], [25]. As a recent example, Becker et al. [26] used stereo vision to create virtual fixtures to improve needle tip positioning and control of a piezo-actuated handheld surgical device called Micron [27]. However, the used vision-based algorithm has the following limitations. First, it is difficult to always have the vision system see the tool tip since the only part that vision system can look through is the cornea and the tool tip might not be always visible through the small cornea circle. Secondly, due to the motion of instrument in various depths inside the eye, keeping the vision system to stay focused on the instrument tip during the surgery is difficult and demands specific microscopes. Optical Coherence Tomography (OCT)-based insertion depth and 3D position estimations of a needle tip under retina, have also been recently proposed by Cheon et al. [28] and Zhou et al. [29], respectively. In [29], authors assumed that the used needle does not experience large deformations during the procedure and relied on the tool 3D rigid model for their calculations. However, surgical instruments utilized in the eye surgery have typically large length to diameter ratio (e.g., less than 25 Gauge, ϕ = 0.5 mm) and, therefore, are prone to bending with excess sclera forces specially during a robot-assisted manipulation [30].

In contrary to the vision and OCT-based methods, and to address their mentioned limitations, in this paper a novel framework for simultaneously improving the estimations of the 3D tip position and insertion depth of a generic deformable surgical instrument during robot-assisted eye surgery has been formulated, implemented and validated. Using an FBG-equipped surgical instrument, the vision-independent force-based framework boosts robot sensing capabilities which are crucial for safety enhancement during robot-assisted ophthalmic surgery. Hence, our contributions in this study are as follows:

1. Develop a state space framework for tool-tip position and insertion depth estimations in robot-assisted eye surgery for a generic FBG-equipped deformable instrument. This innovative formulation is broadly suitable for various filtering and estimation algorithm implementations.

2. Use the developed state space models to obtain a vision-independent force-based stochastic Kalman Filter (KF) approach to improve the estimations.

3. Implement the developed method on the SHER to obtain simultaneous estimations of depth and deflection of a deformable instrument, potentially in the absence of visualization techniques.

4. Evaluate the proposed estimations through different phantom experiments inspired by real surgical scenarios and compare the estimations with the results obtained by rigid-body kinematics and stereo-vision algorithms.

The proposed estimation algorithm could be further utilized in the development of autonomous robot capabilities such as following an anatomical curve for laser ablation, guiding a suture/needle along a blood vessel and other potentially relevant applications. This framework can be extended to a broader range of applications in which an FBG-equipped instrument is utilized. The improved sensing capabilities enable safety-enhancing control design for various eye surgical tasks and autonomous or semi-autonomous robot-assisted surgery inside of the eye.

II. Dual Force Sensing Tool

In order to measure sclera force components (F_sx and F_sy) and insertion depth (d) as shown in Fig. 2, a dual force-sensing instrument has been used [19]. The tool consists of three parts including the tool shaft (a stainless steel wire with diameter of 0.63 mm), the tool adapter, and the tool handle, which are shown in Fig. 3. In order to enable the tool to measure sclera forces and insertion depth, three optical fibers equipped with FBG sensors (Technica S.A, Beijing, China) with diameter of 80 μm are placed in the v-shaped grooves along the tool shaft with a radial separation angle of 120° (Fig. 3). FBGs are very sensitive strain sensors capable of detecting strains less than 1 μϵ, based on the wavelength shifts of optical beams sent through the fibers. Furthermore, due to their light weight and bio-compatibility, FBGs are suitable for opthalmic surgery applications. As it can be seen in Fig. 3, each optical fiber contains three FBG active areas represented by the short red segments, summing up to 9 FBG sensors in total along the tool shaft. A calibration procedure is then conducted to obtain the calibration matrices $K_I\in {ℝ}^{2\times 3}$ and $K_{II}\in {ℝ}^{2\times 3}$, as instructed by [21]. These calibration matrices relate the moments $M_I\in {ℝ}^2$ and $M_{II}\in {ℝ}^2$—induced by the sclera F_s = [F_sx F_sy]^T on the FBG sections I and II the FBG readings $\Delta S_I\in {ℝ}^3$ and $\Delta S_{II}\in {ℝ}^3$ (because there are FBG sensors at each cross section i, where i = I and II), respectively. The sclera force can then be obtained as follows:

$$F_s=\frac{M_{II}-M_I}{\Delta l}=\frac{K_{II}\Delta S_{II}-K_I\Delta S_I}{\Delta l}$$ (1)

where Δl is the distance between the centers of F BG_I and F BG_II. After finding the sclera force components, the insertion depth can be found using either of equations in (2):

$$\mathbf{\text{d}}=l_{II}-\frac{\Vert M_{II}\Vert }{\Vert F_s\Vert }=l_I-\frac{\Vert M_I\Vert }{\Vert F_s\Vert }$$ (2)

where l_II and l_I are the distance between the tool tip and the F BG_II and F BG_I, respectively. It is noteworthy to mention that although FBG fibers are sensitive to temperature, the calibration procedure delineated in [21] makes the measurements robust against temperature variations.

Fig. 3.

Force-sensing tool. The three FBG zones are marked with the short red lines. The lengths l_I and l_II indicate the distance from FBG_I and FBG_II to the tool tip. A close-up view of how the fibers are placed along the tool shaft and the related dimensions are also provided. The vector F_s shows the place where the sclera force is applied to the tool shaft.

As it can be observed in (2), the sclera force appears in the denominator. One drawback of this approach, therefore, is the insertion depth estimation instability when the sclera force is small and noisy. Additionally, when the tool shaft is not in contact with the sclera, no insertion depth measurement will be available. These are the present limitations associated with this method of measuring the insertion depth. Of note, using (1) and (2), the presented dual force sensing tool in this study is able to measure sclera force components and insertion depth with an accuracy less than 1 mN and 0.5 mm, respectively [23].

III. Steady-Hand Eye Robot

The SHER, as depicted in Fig. 1, is a 5 degrees of freedom (DoF) collaborative robot designed to provide steadier manipulation with less tremor in eye surgical interventions. Additionally, using a fast release mechanism for the robot end-effector, various surgical tools can be integrated with the SHER [7]. Surgeons can grasp the tool handle, which is also held by the robot, to move it to a desired location and orientation in a collaborative way with the robotic assistance.

A schematic plot for the tool integrated to the robot end-effector is represented in Fig. 2. As it can be seen, when a tool is inserted into the eye through the sclera, the tool shaft may deflect due to the relative motion between the eyeball and the robot, resulting in sclera forces F_sx and F_sy. Both of these sclera force components are perpendicular to the original (unbent) tool shaft direction as shown by the dashed lines in Fig. 2.

As shown in Fig 1, three coordinate frames are incorporated in the robot modelling. The robot base frame is a fixed frame placed next to the robot base while the handle frame and the tip frame are rigidly attached to the robot end-effector. The tip frame is always located at point o (Fig. 2) with z direction along the original (unbent) tool shaft direction. Of note, the tip frame will be used throughout the rest of the paper, particularly in deriving the equations for the KF.

The robot control works based on a proportional admittance control which is looped around a low-level embedded joint velocity controller. The low-level joint velocity controller consists of PID controllers for the five joints of the robot. The admittance controller provides the desired velocity, $V_d\in {ℝ}^6$, for the handle frame as written in (3). The first and last three elements of V_d correspond to the linear velocity of the origin of the handle frame and the angular velocity of the handle frame, respectively. This desired velocity is generated based on $F_h\in {ℝ}^6$ representing the forces and torques applied by the surgeon to the tool handle (Fig. 1), such that the surgeon will have an intuitive sense of movement when manipulating the robot. It is noted that the handle frame and the tip frame have a fixed offset along the z direction of the tip frame and they are always aligned.

$$\begin{gathered} V_d^{\mathit{\text{handle}}}=K{F}_h \\ {\dot{\theta }}_d=J_{\mathit{\text{handle}}}^{†}(\theta )V_d^{\mathit{\text{base}}} \end{gathered}$$ (3)

where the matrix K is a diagonal 6 × 6 matrix including the admittance control gains. The variable $\theta \in {ℝ}^5$ represents the joint angles. The desired joint angles velocity θ is then produced using the pseudo inverse of the handle frame Jacobian, $J_{\mathit{\text{handle}}}\in {ℝ}^{6\times 5}$, which gives the least squares solution for θ_d. The superscripts handle and base for V_d in (3) indicate the frame in which the vector V_d has been expressed. In other words, after obtaining V_d in the handle frame using the first equation in (3), this vector is then transformed to the robot base frame using the robot FWK and is then fed into the second equation in (3). Finally the obtained θ_d will be given to the embedded joint velocity controller of the robot.

IV. Kalman Filtering

KF is one the most popular and fundamental stochastic-based parameter estimation tools for analyzing and solving a wide class of optimal estimation problems [31]. Loosely speaking, KF provides a recursive method of estimating the state of a dynamical system in the presence of noise by simultaneously maintaining estimates of both the state vector $\left(X_k\in {ℝ}^n\right)$ and the error covariance matrix (Σ_k) at time step k [32]. One of the primary applications of KF is when a known but imprecise state-space model representing an evolving dynamics for state variables X_k and noisy sensor measurements $\left(Y_k\in {ℝ}^m\right)$ are available. KF is advantageous when one wants to combine these two sources of information to improve the accuracy of the system state variables. A further requirement for KF is having estimations for the system noise covariance matrices. Our problem matches well with the assumptions of KF as will be shown in the following sections. For a discrete linear dynamical system, the following state equation can be considered:

$$\begin{gathered} X_k={\mathbb{A}}_{k-1}{X}_{k-1}+{\mathbb{B}}_{k-1}{U}_{k-1}+w_{k-1} \\ {Y}_k={ℂ}_k{X}_k+v_k \end{gathered}$$ (4)

where the first line demonstrates the state evolution model and the second line represents how the sensor measurements are related to the state variables. The noise processes w_k and v_k are white, zero-mean and uncorrelated with known covariance matrices Q_k and R_k. Assuming Gaussian noise, we will represent the marginal posterior probability distribution at step time k as $p\left(X_k\mid Y_{1:k}\right)=\mathcal{N}\left(X_k\mid {\widehat{\mu }}_k,{\widehat{\Sigma }}_k\right)$, where $\mathcal{N}$ denotes a multi variable Gaussian distribution with mean vector ${\widehat{\mu }}_k$ and covariance matrix ${\widehat{\sum }}_k$. The KF procedure for estimating X_k consists of two steps of prediction and refinement. The prediction is to find the Bayes Least Squares (BLS) estimate of X_k given the first k − 1 sensor measurements, Y_1:k−1, which is defined as $\mathbb{E}\left[X_k\mid Y_{1:k-1}\right]$ and $\mathbb{E}[\cdot \mid \cdot ]$ is the conditional expected value. To find this value, the following probability should be computed [33]:

$$\begin{gathered} p\left(X_k\mid Y_{1:k-1}\right)=\int \mathcal{N}\left(X_k\mid {\mathbb{A}}_{k-1}{X}_{k-1}+{\mathbb{B}}_{k-1}{U}_{k-1},Q_k\right) \\ \mathcal{N}\left(X_{k-1}\mid {\widehat{\mu }}_{k-1},{\widehat{\Sigma }}_{k-1}\right)d{X}_{k-1}=\mathcal{N}\left(X_k\mid {\mu }_k^{\prime },{\Sigma }_k^{\prime }\right) \end{gathered}$$ (5)

where ${\mu }_k^{\prime }$ and ${\Sigma }_k^{\prime }$ are a priori estimations for X_k and its covariance which can be found using (6):

$$\begin{gathered} {\mu }_k^{\prime }={\mathbb{A}}_{k-1}{\widehat{\mu }}_{k-1}+{\mathbb{B}}_{k-1}{U}_{k-1} \\ {\Sigma }_k^{\prime }={\mathbb{A}}_{k-1}{\widehat{\Sigma }}_{k-1}{\mathbb{A}}_{k-1}^T+Q_{k-1} \end{gathered}$$ (6)

Using (6), the BLS estimate of $\mathbb{E}\left[X_k\mid Y_{1:k-1}\right]$ would be ${\mathbb{A}}_{k-1}{\widehat{\mu }}_{k-1}+{\mathbb{B}}_{k-1}{U}_{k-1}$.

In the refinement step, this estimation gets updated by incorporating the new sensor information Y_k to the estimations (${\mu }_k^{\prime }$ and ${\Sigma }_k^{\prime }$) obtained in the prediction step for X_k. This is accomplished by recalculation of the BLS estimate of X_k given Y_1:k (i.e., $\mathbb{E}\left[X_k\mid Y_{1:k}\right]$). Using a similar procedure performed in the prediction step and by adding the Bayes rule, the probability mass function p(X_k|Y_1:k) can be found and then the BLS estimate of X_k given Y_1:k and its covariance denoted by ${\widehat{\mu }}_k$ and ${\widehat{\Sigma }}_k$, respectively, can be computed [33]. These estimations after the refinement step are called the posterior estimations. The final equations for finding ${\widehat{\mu }}_k$ and ${\widehat{\Sigma }}_k$ are given in (7):

$$\begin{gathered} {\widehat{\mu }}_k={\mu }_k^{\prime }+{\mathbb{K}}_k\left(Y_k-{\widehat{Y}}_k\right) \\ {\mathbb{K}}_k={\Sigma }_k^{\prime }{ℂ}^T{\left({ℂ}_k{\Sigma }_k^{\prime }{ℂ}^T+R_k\right)}^{-1} \\ {\widehat{Y}}_k={ℂ}_k{\mu }_k^{\prime } \\ {\widehat{\Sigma }}_k={\Sigma }_k^{\prime }-{\mathbb{K}}_kℂ{\Sigma }_k^{\prime } \end{gathered}$$ (7)

where ${\mathbb{K}}_k$ is called is Kalman gain matrix at time step k.

The prediction and refinement procedures should be performed at each time step k. In the following sections we will explain how KF is used to improve the estimations for simultaneous insertion depth and tool tip position in a robot-assisted retinal surgery scenario as depicted in Fig. 2.

A. Insertion depth estimation

As explained in section II, obtaining reliable FBG-based measurements for insertion depth (d) is highly dependent on the presence of sufficient contact between the tool shaft and the sclera (large enough sclera forces). Thus, it is not possible to perform a safety control for the tool insertion depth via the SHER, due to the above-mentioned uncertainties associated with the FBG measurements for d. Of note, the evolution of insertion depth can also be calculated using the FWK. The linear velocity of point o (the imaginary tool tip point on the tool center line as shown in Fig. 2) when projected on the $\overrightarrow{z}$ axis of the tip frame provides the time derivative of the tool insertion depth. However, this modeling also encompasses some errors since it only provides the time derivative of d and not the absolute value of the insertion depth. In other words, this kinematic model does not recognize whether the tool is inside the eye or not and it only returns the rate of the robot motion along the $\overrightarrow{z}$ direction of the body frame. An elegant approach for improving the insertion depth measurements is to combine the available sources of information (i.e., FWK model and FBG sensor data) using the explained KF approach in section IV, which to our knowledge has not been studied in the past. This paper, therefore, proposes the following linear model for the insertion depth estimation, as a basis for the KF:

$$\begin{gathered} d_k=d_{k-1}+\stackrel{{\text{U}}_{k-1}^d}{\overbrace{{\left[J_{\mathit{\text{tip}}}\dot{\theta }\right]}_{k-1}\cdot {\overrightarrow{z}}_{k-1}\Delta t}}+w_{k-1}^d \\ {Y}_k^d=d_k+v_k^d \end{gathered}$$ (8)

where d_k denotes the insertion depth at the k^th time step. $w_k^d$ and $V_k^d$ denote the Gaussian noises for insertion depth model and sensor measurements at time step k, respectively. The FBG measurement for insertion depth is denoted by $Y_h^d$. The superscript d in (8) indicates that the variable is related to the insertion depth modeling and measurements. Of note, since the insertion depth is a scalar, all the variables $w_k^d$, $V_k^d$ and $Y_h^d$ are also scalars. J_tip is a 3 × 5 Jacobian matrix that relates the joint velocities $\theta \in {ℝ}^5$ to the linear velocities of the point o, which is the origin of the tool tip coordinate frame. Δt indicates the time difference between time steps k − 1 and k. The entire scalar expression ${\left[J_{\mathit{\text{tip}}}\dot{\theta }\right]}_{k-1}\cdot {\overrightarrow{z}}_{k-1}\Delta t$ can be considered as the input ${\text{U}}_{k-1}^d$ in (4).

By comparing (4) and (8) and using equations 6 and 7 the Kalman gain can be obtained and the necessary calculations can be performed.

B. Tool tip estimation

In previous tele-manipulation studies including two robots (e.g., [15], [16]), the position of a surgical tool attached to one of the robots is sent to the other robot as a setpoint (e.g. when one of the robots holds a light pipe and autonomously illuminates the working area of the tool attached to the other robot). In the studies mentioned above, only the robot FWK was used to obtain the tool tip position (P in Fig. 3), assuming the tool is a rigid body without any deflection. This assumption, however, is not realistic in a practical use-case of the robot where contacts between the tool and the sclera are present. Moreover, the tool tip of the surgical instrument in the confined area of the eye should be controlled precisely (especially when the tool is being controlled autonomously by the robot), necessitating an exact estimation of the tool tip location. In our previous studies, we have been able to measure the sclera forces using FBG sensors and used them in different control strategies [19], [22]. In this study, we aim at improving the tool tip localization, using the existing FBG sensors, without any additional equipment.

Considering the tool shaft deflection in the x-z plane of the tip frame (Fig. 2), we can find the deflection of the tool tip in the same plane using the beam theory equations as the following [34]:

$${\delta }_x=F_{sx}\left(\frac{{(h-d)}^3}{3EI}+\frac{{(h-d)}^2}{2EI}\mathbf{\text{d}}\right)$$ (9)

where h and d are the length of the cantilever beam and the insertion depth of the tool shaft, respectively, as shown in Fig. 2. E and I denote the tool modulus of elasticity and the tool second moment of area, respectively. The inverse of the expression relating δ_x to F_sx is denoted by α_x, resulting in ${\delta }_x=F_{sx}{\alpha }_x^{-1}$. A similar projection and calculation can be considered for the tool tip deflection in the y-z plane of the tip frame, δ_y. It is noted that δ_x and δ_y represent the tool tip position in the robot tip frame.

So far, two sources of information for the tool tip position are present: 1) the robot FWK, and 2) the tool tip position estimation based on the sclera forces obtained from the FBG in (9). The former is always available but does not take into account the tool deflection. The latter does consider the tool deflection, however, it requires a consistent contact between the tool shaft and the eyeball, which is not always the case in a typical retinal surgery practice. Therefore, combining these two sources of information using KF will be a viable strategy to obtain a more accurate tool tip localization.

If we denote the tool tip position in the robot base frame at step time k by $P_k\in {ℝ}^3$, then P_k−1 and P_k can be related using the following equation:

$$P_k=P_{k-1}+{\left[J_{\mathit{\text{tip}}}\dot{\theta }\right]}_{k-1}\Delta t+w_{k-1}^P$$ (10)

where Δt is the time between step k − 1 and k. The vector $w_{k-1}^P\in {ℝ}^3$ is the Gaussian noise vector for the tool tip position model. Furthermore, we can assume that the z coordinate variation of the tip position in the tip frame is negligible, therefore, we can define the tool tip deflection vector in the tip frame as δ = [δ_x , δ_y, 0]^T. The tool tip position in the base coordinate frame (P) can be related to the vector δ using (11):

$$\delta =R^{T}P-R^{T}S$$ (11)

where the matrix R ∈ SO3 and vector $S\in {ℝ}^3$ are the rotational and translational parts of the homogeneous transformation between the base frame and the tip frame. Now we can write the following matrix equation between the sensor readings and the tool tip position:

$$\underset{F}{\underbrace{\left(\begin{array}{c}{F}_{sx}\\ F_{sy}\\ 0\end{array}\right)}}=\underset{\Gamma }{\underbrace{\left(\begin{array}{ccc}{\alpha }_x& 0& 0\\ 0& {\alpha }_y& 0\\ 0& 0& {\alpha }_z\end{array}\right)}}\left(R^{T}P-R^{T}S\right)$$ (12)

α_z can be any non-zero value. The reason is that by having a non-zero value for α_z we can enforce the realistic assumption that the third element of R^T P − R^T S, which would be the third element of δ is zero. Additionally, enforcing this assumption in the sensor measurements equation makes the system observable, which is a necessary requirement for KF to work properly. Using the terms F and Γ, which are defined in (12), we can rewrite (12) in the following form:

$$F+\Gamma R^{T}S=\Gamma R^{T}P$$ (13)

Now if we denote the left hand side of (13) with Y^P, using (10) and (13) the final state space equations for tool tip position written in step time format will be as following:

$$\begin{gathered} P_k=P_{k-1}+{\left[J_{tip}\dot{\theta }\right]}_{k-1}\Delta t+w_{k-1}^P \\ {Y}_k^P={\left[\Gamma R^T\right]}_k{P}_k+v_k^P \end{gathered}$$ (14)

where $v_k^P$ denotes the 3 × 1 Gaussian measurement noise for tool tip position. The superscripts P in (14) mean that the corresponding variable belongs to the state space equations for the tool tip estimation. Equation (14) indicates the state space model as a basis for the KF for tool tip position estimation. As noted, there is not any direct FBG measurement available for tip position, however, the FBG-based measurements for the sclera force components were transformed into sensor information for tip position using (9)-(13). Now using (6) and (7) the Kalman gain can be computed for this case and the necessary computations can be performed. The block diagram for the closed loop system is shown in Fig. 5.

Fig. 5.

Block diagram for the closed-loop system showing the SHER normal impedance control and the KF estimations for tool tip position and insertion depth

V. Validation

In order to perform the validation experiments for the KF estimations for the tool tip position and the insertion depth, a stereo camera system and the robot FWK have been used, respectively. The validation experiments are explained in the following sections.

A. Validation for insertion depth

In order to validate the KF output for the insertion depth, a phantom was made out of rubber with a small hole in it, mimicking a tool insertion spot on sclera during a real surgery (Fig. 4-c). The phantom was fixed in space and the robot was then moved to a configuration such that the tool tip was touching the small hole on the rubber. Using the robot FWK, the position of this space-fixed point was obtained and recorded in the robot base frame, denoted by $Ti{p}_E$ (Fig. 4-c). The tool was then inserted into the rubber phantom and at each sample time the distance between the location of the point o in Fig. 2 (which can be obtained via FWK) and the recorded point $Ti{p}_E$ was calculated and considered as the ground truth for the insertion depth. The KF output for the insertion depth estimation and the FBG-based insertion depth measurements were then compared to the ground truth for validation. It is noted that the insertion depth (d) is defined as the length of the tool inside the eyeball along the imaginary straight (not deflected) tool as shown in Fig 2.

Fig. 4.

Experimental setup. (a) Hardware components including the force-sensing tool, SHER, FBG interrogator, Stereo camera system, Eye socket. (b) A close-up view of the tool tip and the vessels on the eye socket for the case study. (c) The rubber with a hole used for the validation experiments for insertion depth.

B. Validation for tool tip position

To validate the KF estimations for the tip position of the needle, a stereo camera setup with the resolution of 1024 × 768 pixel was used to track a red marker attached to the tip of the needle (Fig. 4). The stereo camera pair was calibrated using the Stereo Camera Calibration Toolbox in MATLAB (MathWorks, Natick, MA). The 3D location of the marker was found in space using a color segmentation algorithm followed by triangulation [35]. The overall position accuracy error for the stereo camera system was measured 0.2 mm using the distance between two markers with known pre-determined spacial locations on a custom-designed validation jig.

Because the outputs of the KF for the tool tip position are reported in the robot base frame, in order to compare the ground truth (camera measurements for the red marker position) with the KF outputs, a frame registration step should be performed to obtain the homogeneous transformation that relates these two frames, which is defined in (15).

$$F_b^c=\left(\begin{array}{cc}{R}_b^c& S_b^c\\ 0_{1\times 3}& 1\end{array}\right)$$ (15)

where the subscript and superscript b and c correspond to the base frame of the robot and the camera frame, respectively. To find the matrices $R_b^c$ and $S_b^c$, the robot is moved (without deflecting the tool) by a user to different locations around the desired workspace and the tool tip position is recorded both in the camera frame and the robot base frame. Since the robot and the stereo camera system have different data collection frequencies (200 Hz for the robot and 15 Hz for the camera system), the same-time data are extracted by synchronising the time stamps of the two systems.

After stacking the 3D points for the tool tip measurements in the robot base frame and the camera frame in two N × 3 matrices B and C, where N is the number of synchronized data samples, the following mean values are calculated:

$$\overline{B}=\frac{1}{N}\sum _{i=1}^N{B}_i\text{\hspace{1em} and \hspace{1em}}\overline{C}=\frac{1}{N}\sum _{i=1}^N{C}_i$$ (16)

where B_i and C_i are the ith rows of matrices B and C, respectively. Then, the cross-covariance matrix Σ_BC is calculated in (17).

$${\Sigma }_{BC}=\frac{1}{N}\sum _{i=1}^N{\left(B_i-\overline{B}\right)}^T\left(C_i-\overline{C}\right)$$ (17)

Next, using (17), we form the the following 4 × 4 matrix Q(Σ_BC):

$$Q\left({\Sigma }_{BC}\right)=\left(\begin{array}{cc}\mathit{\text{trace}}\left({\Sigma }_{BC}\right)& {\Delta }^T\\ \Delta & {\Sigma }_{BC}+{\Sigma }_{BC}^T-tr\left({\Sigma }_{BC}\right){\mathbb{I}}_3\end{array}\right)$$ (18)

where ${\mathbb{I}}_3$ is the 3×3 identity matrix and the column vector $\Delta ={\left[H_{23}{H}_{31}{H}_{12}\right]}^T$ is obtained from $H_{ij}={\left({\Sigma }_{BC}-{\Sigma }_{BC}^T\right)}_{ij}$.

As shown in [36], the unit eigenvector corresponding to the maximum eigenvalue of Q(Σ_BC) returns an optimal quaternion vector $\overrightarrow{q}=\left[q_0{q}_1{q}_2{q}_3\right]$, where q₀ ≥ 0 and $q_0^2+q_1^2+q_2^2+q_3^2=1$. Using this quaternion vector, the corresponding optimal value for 3×3 rotation matrix $R_b^c$ can be calculated using quaternion to rotation matrix transformations. The vector $S_b^c$, therefore, can be obtained using (19).

$$S_b^c={\overline{B}}^T-R_b^c{\overline{C}}^T$$ (19)

Eventually, using (15) the homogeneous transformation F ^c_b can be obtained. This matrix will be used to bring the 3D coordinates of the red marker described in the camera frame to the robot base frame.

VI. Experimental Setup

The experimental setup is depicted in Fig. 4. A dual force sensing tool is attached to the SHER to measure the sclera force components F_sx and F_sy, as well as the insertion depth. The three fibers on the dual force sensing tool are attached to an optical sensing FBG interrogator (sm130–700 from Micron Optics Inc., Atlanta, GA) to measure the changes in the reflected optical wavelengths as shown in Fig. 4. An embedded low-level velocity controller (Galil 4088, Galil, Rocklin, CA, USA) controls the velocity of the motor associated with each joint in SHER. The robot is equipped with an ATI force sensor (Nano17, ATI Industrial Automation Inc., Apex, NC, USA) placed at the robot end-effector to measure the forces and toques applied to the robot handle by the operator (F_h in (3)) as shown in Fig. 4-a. This force sensor provides the components of F_h in the handle coordinate frame which is used in (3).

The measurements for sclera force components, insertion depth, handle force and torque components, robot tool tip coordinate and the time information for all experiments were recorded using the software package for controlling the SHER, developed using the C++ CISST-SAW libraries [37]. The KF algorithms were also implemented and recorded within the same software package.

Two Point Grey cameras were used for the stereo camera system, which were attached to an arch-shaped laser-cut acrylic sheet, clamped to the experiment table (Fig. 4). The cameras were places at approximately 30 cm above the working area of the robot tool tip. Sufficient illumination was provided to make the red marker clear enough for both cameras. During the experiments the robot was moved by the operator in a way such that the tool tip was always visible by both cameras. The images of the stereo camera system were recorded by custom ROS packages (Robot Operating System, a collection of software frameworks for robot software development packages [38]). The images were then fed to the Python code mentioned in section V-B for triangulation and extraction of the tool tip position in the camera frame.

VII. Results

As it is observed in (9), to calculate δ_x (or δ_y) in each sample time, the parameters h, d and EI are required to be determined for the tool shaft. Using the CAD model of the tool, the parameter h was measured to be 60 mm. The parameter d (insertion depth) is updated in each loop using the KF output. The parameter EI is found using the same calibration data that was used to identify the matrices K_I and K_II in (1) without performing any additional experiments. As instructed by [21], the calibration procedure consists of applying various sclera forces at different points along the tool shaft. In this process, three variables are recorded: 1) the tool shaft deflection at the point where the sclera force is applied, δ, 2) the distance between the tool tip and the the point of application of sclera force, (d), and 3) the sclera force (which are obtained from another sensor during the calibration experiments). We know that the deflection of the tool shaft at the point of slcera force exertion is $\delta =\frac{{‖F‖}_s{(h-\text{d})}^3}{3EI}$. If we plot the value of ||F_s||(h−d)³ on the y axis and the value of δ on the x axis for all data samples recorded during the calibration process, the slope of the line that fits through this point cloud will provide the coefficient 3EI. In our experiments, this coefficient was obtained to be 3.39 × 10⁶ mN.mm².

After performing the registration as described in section V-B and finding the matrix F ^c_b in (15), a validation experiment similar to what is delineated during the registration process (in section V-B) was performed. This validation data set containing the tool tip position in the camera frame $\left(C_i^{\prime }\in {ℝ}^3\right)$ and the robot base frame $\left(B_i^{\prime }\in {ℝ}^3\right)$, where i = 1, …,n and n is the number of data points collected during the validation experiment, were then used to calculate the registration error. The ith element of the error vector E is defined as follows:

$$E_i=\Vert B_i^{\prime }-\left(R_b^c{C}_i^{\prime }+S_b^c\right)\Vert$$ (20)

where $R_b^c$ and $S_b^c$ can be found using (15). The registration error (the mean value of the vector E) was calculated to be 0.67 mm.

The covariance matrices for the model and the measurement noises were chosen with trial and error. We assumed the covariance for the insertion depth measurement noise is a function of the sclera force (F_s), since for low sclera forces we cannot rely on the the FBG measurements for computing the insertion depth, as discussed in section II. However, for larger sclera forces, we can obtain reliable FBG measurements for estimating the insertion depth. For this reason, the covariance matrix (which is a scalar here) for v^d is chosen the large number of 10⁶ when ||F_s|| < 50 mN, while for ||F_s|| ≥ 50 mN it is set to 0.005. The covariance matrix for w^d is set to the fixed value of 0.0025. The 50 mN cutoff was chosen experimentally for best performance. Similarly, the covariance matrix for the measurement noise, v^P in (14), is assigned to diag(10, 10, 0) when ||F_s|| < 50 mN, since the F_s measurements are not reliable enough to be used for tip position estimation. The element (3, 3) of the v^P covariance matrix is set to zero, since in IV-B we assumed that the tool deflection does not change the tip position along the tool shaft. Thus, whether or not there exists small or large ||F_s||, the measurement equations should always give zero value for the tip displacement along the tool shaft. For ||F_s|| ≥ 50 mN, this matrix is set to diag(0.002, 0.002, 0). The covariance matrix for w^P is set to the fixed value of diag(0.01, 0.01, 0.01), as well. It is noted that no correlation between different directions of tip estimation is assumed.

A. Insertion depth and tool tip position estimation

Validation experiments were conducted to evaluate the accuracy of insertion depth and tool tip position estimations. The validation experiment for the insertion depth was performed as explained in section V-A and the results are plotted in Fig. 6. As mentioned in section V-A, the ground truth for this case are obtained from robot FWK. The errors between the ground truth and both KF outputs (d) and FBG measurements for insertion depth Y^d are plotted in Fig. 7. In the same figure, the sclera force norm $\left({‖F‖}_s=\sqrt{F_{sx}^2+F_{sy}^2}\right)$ is also plotted for the same experiment.

Fig. 6.

Results for the validation experiments for KF for insertion depth. The red dashed curve indicates the gound truth which was measured using the FWK.

Fig. 7.

Error variations for insertion depth validation experiment. Left y axis: Sclera force norm, ||F_s||. Right y axis: The error between the KF output and the ground truth (robot) is plotted with blue. The error between the FBG measurements for insertion depth and the ground truth (robot) is plotted with black.

To validate KF output for the tool tip position, the robot was moved around (and stopped) in free space by a user, and sometimes the motion was stopped and the tool shaft was bent from random locations along the shaft using another needle. Notably, the instances when the tool shaft is bent are when the robot FWK does not provide accurate location of the tool tip. In this experiment, the location of the red marker was recorded with the stereo camera system and transformed to the robot base frame using (15). Meanwhile, the tool tip position values obtained from the KF algorithm and also from the FWK were recorded. The results for this experiment are plotted in Fig. 8, showing the tool tip position coordinates in the robot base frame.

Fig. 8.

Results for the tool tip position validation experiment. The X, Y, Z positions if the tool tip are plotted in the robot base frame.

Similar results for the tool tip estimations are represented in Fig. 9, where the errors between the KF output and the robot FWK compared to the ground truth from the stereo camera system are shown. The sclera force norm ||F_s|| for the tip position experiment is also plotted in Fig. 9.

Fig. 9.

Error variations for tool tip position validation experiment. Left y axis: Sclera force norm, ||F_s||. Right y axis: The error between the KF output and the ground truth (camera) is plotted with blue. The error between the robot FWK and the ground truth (camera) is plotted with black.

B. Case study

In order to better evaluate the performance of the developed method in a realistic situation, a retinal vein following task as part of an RVC was mimicked through a case study. An eye socket with diameter of 36 mm was 3D-printed and a hole with diameter of 5 mm was created at one side of the eye socket simulating the insertion point for the tool (Fig. 4-b). The dual-force sensing tool was then attached to the SHER and moved by a user to the eye socket through the 5 mm hole. The hole was intentionally made larger than the tool diameter to create a range of motion for the tool similar to an actual surgery. In addition, a larger hole enables possibility of performing larger maneuvers by the user, as well as the ability to study various types of contact with the hole edges from different angles and orientations. Although the eye phantom is stationary, the large 5 – mm hole allows a simulation of eye motion as the tool can be inserted at various angles and orientations as if the insertion point is moving in space.

As it can be seen from Fig. 4, retinal vessels were printed and attached to the posterior of the eye socket phantom. The phantom vessels exhibit similar distribution and arrangement compared to real retinal vessels since they were reconstructed from a real retina image. After inserting the tool through the hole into the eye socket, the user followed the black vessels on the phantom. In this case study, the tool was bent from different locations on the tool shaft in different planes. In addition, in this experiment the tool deflection and robot motion occurred simultaneously. The results for the tool insertion depth are plotted in Fig. 10. Tool tip position estimations obtained from the KF, the stereo camera system, and the robot FWK when the user was following the vessels are also plotted in Fig. 11. The error plot and the sclera forces are shown in Fig. 12. The average errors for all three experiments conducted in sections VII-A and VII-B are represented in Table I. The errors for the insertion depth are scalar values and thus are simply averaged. However, because the errors for tool tip position are 3×1 vectors, the average of the vectors 2 – norm are reported in Table I. In this Table, T₀ corresponds to time intervals when there is a significant mismatch (larger than 0.1 mm) between the KF output and the associated sensor measurements (FBG for insertion depth and robot FWK for tool tip position). The time interval T₁ includes any time other than T₀ intervals during each experiment. Clearly, T₀ intervals are of higher importance for analysis. While T₀ and T₁ intervals span the entire time axis, only a single sample of them is represented in Figs. 10–12.

Fig. 10.

Insertion depth variations in the vessel following task.

Fig. 11.

Results for the tool tip position during the vessel following task. The X, Y, Z positions if the tool tip are plotted in the robot base frame.

Fig. 12.

Error variations for the vessel following task. Sclera forces (left y axis) and the KF and FWK errors for tool tip position (right y axis).

TABLE I.

Errors for the Validation Experiments and the Case Study.

Experiment	Signal	Total Error (mm)	Error over T0 (mm)	Error over T1 (mm)
Insertion depth experiments	Kalman output	0.20	0.21	0.17
	FBG measurement	3.14	4.26	0.18
Tip position experiments	Kalman output	0.81	1.53	0.57
	Forward Kinematics	2.30	6.42	0.59
Case study	Kalman output	0.78	1.01	0.44
	Forward Kinematics	2.76	4.38	0.43

VIII. Discussion and Conclusion

To address the limitations that we encountered in [19] for FBG-based insertion depth measurements and to further extend robot sensing capabilities of the system states, in this study we have presented a novel method to simultaneously improve the tool tip position and insertion depth measurements during robot-assisted retinal surgery. The measurements were based solely on the existing FBG sensors and the robot kinematics which were combined using KF. As it can be seen from Fig. 6, the FBG sensors are not able to provide reliable insertion depth measurements and as soon as sclera forces approach zero the FBG measurement for insertion depth does not change and becomes a fixed line. In addition, by investigating Fig. 7 and also comparing it to Fig. 6, it can be observed that the error for FBG-based insertion depth measurements becomes larger and noisier when then sclera force is diminishing (for instance when sclera force is less than 40 mN around t = 2 s in Fig. 7). This justifies why the covariance value for w^d was chosen as a large number when the sclera force is less than 50 mN. However, the KF is able to precisely keep track of the insertion depth. In Table I, the mean value of insertion depth estimation error over the T₀ intervals of Fig. 6 are reported as 0.21 mm and 4.26 mm for the KF and FBG-only approaches, respectively, thus, indicating a significant improvement in insertion depth measurements using KF. In the case study, Table I shows that the tool tip position estimation error has been improved to 1.01 mm. Although this accuracy is less than tens of microns and hundreds of microns precision using OCT-based and vision-based methods as reported in [29] and [26], our approach lacks the limitations of [29] and [26] as described in detail in section I.

There are six bumps visible in Figs. 8 and 9 indicating the tool deflections that were manually imposed on the tool shaft. Fig. 9 shows large errors (up to 13 mm around t = 42 s) in the FWK estimations when tool deflection occurs. Fig. 8 indicates that the KF estimations are able to follow the camera output when deflections (bumps) occur. Table I indicates that during T₀ time intervals in Fig. 9, the average error for the KF and FWK approaches are 1.53 mm and 6.42 mm, respectively, demonstrating a 76% improvement in tool tip position estimation using KF.

In the case study experiment, the user followed the phantom vessels with the tool tip. For better clarification, a sample of T₀ and T₁ intervals are shaded in Figs. 10, 11 and 12. Comparing the shaded T₀ and T₁ intervals in Figs. 10, we can realize that in T₀ intervals the FBG measurements are not reliable, and they return constant unchanging values similar to what was explained in Fig. 6, while the KF estimations are reflecting the changes correctly. In Figs. 11 and 12 a similar behavior occurs during the T₀ interval for the tool tip positioning estimation. In the shaded T₀ interval, the FWK is not able to provide a good estimation of the tool tip position because the tool is deflected. However, the KF results in correct values even in areas where tool deflection is present. This observation is further supported by comparing the average tool tip position estimation errors for the KF and FWK during the case study experiment. The average error is reduced from 4.38 mm to 1.01 mm, which again indicated a 77% improvement for the tool tip position estimation.

In conclusion, this study introduced a force-based estimation method that can improve simultaneous insertion depth and tool tip position estimations in robot-assisted retinal surgery. Of note, such information is valuable and critical for safety enhancement and more accurate feedback in semi-autonomous robot-assisted tasks in eye surgery. The introduced method showed significant improvements in the associated estimations for insertion depth and tool tip position. Future work includes, but not limited to, the development of robot control schemes for autonomous surgical tasks utilizing the improved estimations. Although we predict that the developed method should work in a moving eye environment (because the algorithm is based on the relative motion of the instrument and the eye, rather than absolute motion and because this relative motion was simulated in the case study presented), assessing the algorithm in the setting of a moving eye is important to validate the prediction. Furthermore, evaluating the algorithms in vivo is a necessary direction for the future work.

Biographies

Ali Ebrahimi received his M.S.E in Robotics from the Johns Hopkins University (JHU) in 2019, where he has been working toward the Ph.D. degree in Mechanical Engineering since 2017. He received his B.Sc. and M.Sc. in Mechanical Engineering from Amirkabir University of Technology (Tehran Polytechnique) and Sharif University of Technology, Tehran, Iran, in 2014 and 2016, respectively. His research interests include control theory, parameter estimation and artificial intelligence with special applications in surgical robotics and autonomous systems.

Farshid Alambeigi (Member, IEEE) received his M.S.E. in Robotics and Ph.D. in Mechanical Engineering from the Johns Hopkins University in 2017 and 2019, respectively. Prior to joining Johns Hopkins, he received his B.Sc. and M.Sc. degrees in Mechanical Engineering from the K.N. Toosi University of Technology and Sharif University of Technology, Tehran, Iran, in 2009 and 2012, respectively. He is currently an Assistant Professor in the Department of Mechanical Engineering at the University of Texas at Austin. His research interests include development, sensing, and control of high dexterity continuum manipulators, soft robots, and flexible instruments especially designed for less-invasive treatment of various medical applications.

Shahriar Sefati (Member, IEEE) received his M.S.E. in Computer Science and Ph.D. in Mechanical Engineering from the Johns Hopkins University in 2017 and 2020, respectively. He has been a member of the Biomechanical- and Image-Guided Surgical Systems (BIGSS) Laboratory, as part of the Laboratory for Computational Sensing and Robotics (LCSR), where he is currently serving as a postdoctoral researcher at the Autonomous Systems, Control, and Optimization (ASCO) laboratory. Prior to joining Johns Hopkins, he received his B.Sc. in Mechanical Engineering from Sharif University of Technology, Tehran, Iran, in 2014. His research interests include robotics and artificial intelligence for development of autonomous systems

Niravkumar Patel (Member, IEEE) is currently an assistant professor with the Engineering Design department at Indian Institute of Technology Madras. Prior to joining IIT Madras, he was a postdoctoral fellow with the Laboratory for Computational Sensing and Robotics at Johns Hopkins University. He received his Ph.D. in Robotics Engineering from Worcester Polytechnic Institute in 2017, M.Tech. in Computer Science and Engineering in 2007 from Nirma University, India and B.E in computer engineering in 2005 from North Gujarat University, India. His current research interests include medical robotics, image guided interventions, robot assisted retinal surgeries and path planning.

Changyan He received the B.E degree in Mechanical Engineering and Automation from Beijing Jiaotong University, Beijing, China, in 2015. He is currently working toward the Ph.D. degree in Mechanical Design and Theory at Beihang University, Beijing, China. His research interests include robot-assisted retinal surgery, FBG-based force-sensing, and admittance control.

Peter Gehlbach is a Professor of Ophthalmology at the Wilmer Ophthalmological Institute at Johns Hopkins University (JHU) and the director of the Wilmer Eye Institute Echography Center. He received the M.D. degree from the University of Vermont in 1986, and the Ph.D. degree in Cell Physiology from the University of Minnesota in 1994. He did his Ophthalmology Residency at Washington University in St. Louis, the Retina Fellowship at Oregon Health Sciences University. He has been continuously grant funded and has been an author of numerous peer-reviewed publications, patents, and talks. His current research interests pertain to advancing and translating novel microsurgical technology into vitreoretinal applications.

Iulian Iordachita (Senior Member, IEEE) is a faculty member of the Laboratory for Computational Sensing and Robotics, Johns Hopkins University, and the director of the Advanced Medical Instrumentation and Robotics Research Laboratory. He received the M.Eng. degree in industrial robotics and the Ph.D. degree in mechanical engineering in 1989 and 1996, respectively, from the University of Craiova. His current research interests include medical robotics, image guided surgery, robotics, smart surgical tools, and medical instrumentation.