Automatic Light Pipe Actuating System for Bimanual Robot-Assisted Retinal Surgery

Abstract

Retinal surgery is a bimanual operation in which surgeons operate with an instrument in their dominant hand (more capable hand) and simultaneously hold a light pipe (illuminating pipe) with their non-dominant hand (less capable hand) to provide illumination inside the eye. Manually holding and adjusting the light pipe places an additional burden on the surgeon and increases the overall complexity of the procedure. To overcome these challenges, a robot-assisted automatic light pipe actuating system is proposed. A customized light pipe with force-sensing capability is mounted at the end effector of a follower robot and is actuated through a hybrid force-velocity controller to automatically illuminate the target area on the retinal surface by pivoting about the scleral port (incision on the sclera). Static following-accuracy evaluation and dynamic light tracking experiments are carried out. The results show that the proposed system can successfully illuminate the desired area with negligible offset (the average offset is 2.45 mm with standard deviation of 1.33 mm). The average scleral forces are also below a specified threshold (50 mN). The proposed system not only can allow for increased focus on dominant hand instrument control, but also could be extended to three-arm procedures (two surgical instruments held by surgeon plus a robot-holding light pipe) in retinal surgery, potentially improving surgical efficiency and outcome.

I. Introduction

RETINAL surgery requires precise and bimanual manipulation of surgical instruments inside the confined space of an eyeball. In a typical retinal surgery, surgeons use their dominant hand to operate a functional instrument (e.g. cannula) to perform a surgical task, while using their non-dominant hand to hold a light pipe that illuminates the retina. Both tools are inserted into the eye through trocars installed at scleral ports (the diameters of which are smaller than 1mm). The surgeon looks into the interior of the eye through a surgical microscope that is placed above the patient’s head. The procedures in retinal surgery involve manipulation of tissues at micron scales, during which exertion of submillinewton forces (well below human tactile sensing ability) [1] can cause irreversible structural and functional damage to the eye. Also, given that the surgical instruments are physically constrained by the trocars in the scleral ports, their manipulation generates continuous forces on the sclera, and excessive forces may cause sclera injury. The outcomes of retinal surgery are impacted by many factors, including, but not limited to, physiological hand tremors, fatigue, poor kinesthetic feedback, patient movement, and the absence of force sensing. Some of these challenges have been evaluated and addressed in early work done by different teams including our own. However, the bimanual operation required in retinal surgery makes the procedures even more challenging and demands both strong surgical skills and excellent coordination. To address the distraction and additional fatigue caused by having to manually hold a light pipe throughout a lengthy surgical procedure, this paper introduces an automatic light pipe actuating system that leverages robotic technology to free up the surgeon’s non-dominant hand.

Robotic technology development for applications in retinal surgery has been ongoing for over 20 years. Various robotic devices and systems have been proposed to enhance and expand surgeons’ surgical skills. These can broadly be categorized into the following: robotic manipulators, force/distance sensors, and robotic controllers, which are described in more detail below.

(a). Robotic manipulators:

With the inherent challenges of retinal surgery, robotic manipulators can optimize surgical procedures by acting as intelligent mediators between surgeons and direct manipulation of surgical instruments. Robotic manipulators can filter out hand tremors, impose virtual fixtures and establish safe operating limits, and hold instruments in place without active input from the surgeon. They can be further divided into two subcategories: teleoperated and hands-on operated.

For teleoperated manipulators, the surgeon typically controls a master console, which remotely commands a slave system to perform the surgical task. An example of such a system was created by Edwards et al. [2], who developed the “Preceye” robotic system and demonstrated its operation in human clinical trials. Other groups have also developed different types of robotic arms and master consoles [3]–[6], which were either evaluated in in-vivo animal models or exvivo eyeball models.

Unlike a teleoperated system, a hands-on device does not utilize a separate master console, but instead keeps the user in the loop by having them directly manipulate tools mounted on the system’s end effector. An example of such a system is the Steady Hand Eye Robot (SHER) developed at Johns Hopkins University (JHU) [7]. Rather than scaling movements like a teleoperated system, SHER enables admittance-based force scaling, in which forces exerted by the user on the surgical instrument are interpreted and scaled through an admittance control strategy before appropriate velocities are transmitted to the robotic manipulator.

(b). Force/Distance sensors:

Optical sensors with high resolution, enhanced sensitivity, and reduced size are potentially useful in retinal environments. Yu et al. [8] designed optical coherence tomography (OCT)-guided forceps, which can provide real-time intraocular visualization of retinal tissues and also determine the distance between the tool tip and the tissue. Song et al. [9] developed OCT-based forceps to detect the contact between their tip and the retinal surface. A JHU research group developed microsurgical instruments based on F7iber Bragg Grating (FBG) sensors to measure not only forces at the tool tip, but also the location of force application and amount of force applied on the tool shaft [10]. Apart from optical sensors, impedance sensors have also been employed to detect puncture signals between the cannula tip and retinal vessel in retinal vein cannulation [11]. The above-mentioned sensors can be incorporated into robotic platforms to improve tactile and force perception of surgeons via auditory feedback [12] and/or haptic feedback [13].

(c). Robotic controllers:

To improve the safety of robot-assisted retinal surgery, on top of existing low-level robot controllers (e.g., velocity controller), Ebrahimi et al. [14], [15] proposed a 3 degree of freedom (DoFs) adaptive controller to restrict instrument movement in response to exceeding a prescribed force limit. He et al. leveraged deep learning to predict the manipulation force, in which the prediction result is then either fed back to surgeon via auditory substitution [16] or integrated into an admittance controller [17] to keep the force level below a desired threshold. In the realm of automatic surgery, Yang et al. [18] explored techniques for robot-aided intraocular surgery using monocular vision and proposed a new retinal surface estimation method. Braun et al. [19] proposed the “EyeSLAM” algorithm that can deliver 30 Hz real-time simultaneous localization and mapping of the human retina and vasculature during intraocular surgery.

The previously developed robotic technologies described above provide solutions for various operational challenges in retinal surgery. However, the primary research focus has been on the active surgical instruments held in the surgeon’s dominant hand. The topic of bimanual control and automatic light pipe actuation has not been explored, but it is a promising area for improving surgical efficiency and decreasing complexity. In this paper, an automatic light pipe actuating system is proposed based on a hybrid velocity-force control scheme.

The system consists of a customized light pipe with force-sensing capability and a force-sensing cannulation tool, which are installed at the end effectors of two SHERs. The SHER with the cannulation tool (designated as the leader robot) is manipulated by the user to perform the surgical task and establishes the target position on the retinal surface. The other SHER with the light pipe (designated as the follower robot) is commanded automatically without any active user input through a hybrid velocity-force controller to track and illuminate the target position with the light pipe. Apart from the tracking task, the light pipe is controlled to pivot about the scleral port to keep the scleral force within a prescribed limit. The leader robot uses an admittance control scheme that takes the manipulation force applied by the user and the scleral force as inputs. The admittance controller provides a force-based virtual fixture at the scleral port for the cannulation tool to minimize the scleral force.

The proposed system is evaluated on an rubber eyeball phantom. The accuracy of target tracking of the light pipe as well as the ability to maintain acceptable scleral force values applied by both tools are analyzed. The results demonstrate that the light pipe can successfully track the target position on the retinal surface with an insignificant offset, and both tools can maintain scleral forces within a preset threshold. Therefore, the proposed system is capable of freeing up the surgeon’s non-dominant hand, potentially allowing the surgeon to concentrate more closely on their dominant hand and the surgical task. Furthermore, the automated system can enable three-arm procedures (two surgical instruments held by the surgeon plus a light pipe held by the follower robot) in retinal surgery, making it possible to develop new procedures that may improve surgical outcome and efficiency.

II. Hardware system

The proposed automatic light pipe actuating system mainly consists of two components: the robotic system and the force sensing system, shown in Fig. 1 and described in detail as follows. For this study, it is assumed that the leader robot holds a functional surgical tool (e.g., cannulation tool) that the surgeon would use to actively perform a task and that the follower robot holds a light pipe.

Fig. 1.

(a) Experimental setup consisting of the SHER 2.0 and the SHER 2.1, two force sensing tools that are each connected to an FBG interrogator, an eyeball phantom, and a microscope to view inside the eyeball. (b) The force sensing tools, which include a light pipe and a cannulation tool. The scleral force is defined as the interactive force between the tool shaft and the scleral port on the eyeball. (c) Orientation of FBG fibers around the circumference of the tool shaft. Both the cannulation tool and light pipe outer diameters are 23Ga, and the FBG fibers are secured along the tool shafts using medical-grade adhesive.

A. Robotic system

The robotic system is comprised of the previously developed SHER 2.0 [20] and SHER 2.1 [7]. Both SHERs have five DoFs and similar mechanical structure. Three linear motion stages, one rotational stage, and a parallel link mechanism enable the achievement of high translation accuracy (less than 3μm) and rotational accuracy (0.005°). Surgical tools are mounted on the SHER’s end effectors, and users are able to grasp and manipulate the tools to perform tasks. Instead of the interaction forces applied by the user being translated directly to the tool, these forces are captured in real time by force/torque sensors and used to compute SHER velocities. This enables the robot to filter out hand tremors and/or excessive tool-to-tissue forces. In this study, SHER 2.1 acts as the leader robot that is controlled by the user’s dominant hand via a surgical tool. SHER 2.0 acts as the follower robot that actuating the light pipe automatically.

B. Force sensing system

The force sensing system consists of two sensorized instruments, a cannulation tool and a light pipe, each connected to a separate FBG interrogator. The tools are designed and fabricated based on the method proposed in our previous work [10] to measure the scleral force as shown in Fig. 1 (b). Each tool integrates three FBG fibers (Technica S.A, Atlanta, GA, USA), with three active areas per fiber, along the tool shaft as shown in Fig. 1 (c). Using the calibration algorithm presented in [10], changes in FBG wavelength data collected by the interrogators can be correlated to applied force, resulting in sensorized tools that are able to sense scleral force. Root mean square (RMS) errors of the scleral force measurement for the light pipe and cannulation tool are 2.6 mN and 1.4 mN, respectively.

III. Control system

The leader robot is controlled by the user manipulating the functional surgical tool. The follower robot (which holds the light pipe) is programmed to move automatically. The goal of the follower robot is to align the light pipe with the calculated intersection point between the leader tool axis and the retinal surface, thus illuminating this desired point on the retina. Given that the location of the scleral ports may change during the procedure due to eyeball rotation within the socket, both robots use force-based virtual fixtures at their respective scleral ports to minimize damage to the sclera. The control system can be divided into three main components that are detailed in subsequent sections: preoperative registration, light pipe actuation control for the follower robot, and user-involved admittance control for the leader robot.

A. Preoperative Registration

1). Robot registration:

The first step in the control scheme is to establish the coordinate frames of the leader and follower robots with respect to one another. The robot registration scenario is shown in Fig. 2. A calibration cube with the dimension of 40 × 40 × 40 mm is placed in the shared workspace of the two robots. Papers drawn with target points are pasted on the four side surfaces and the top surface of the cube. Each surface has 5 target points, that is to say, a total of 25 target points is well-spread in the robots’ common workspace. When the registration starts, one of the robots is manipulated manually to touch the 25 target points with its tool tip in a prescribed sequence, the tip coordinates of the tool in its robot base frame are recorded when the tool tip hits the target points. Then, in the same sequence, the other robot is controlled manually to touch the target points with the tool tip and, the tip coordinates are recorded the same way as the former robot. The microscope is used to observe the contact between the tool tips and the markers to minimize data collection error. In total 25 sets of datapoints are collected. For each pair of recorded points, the registration of two robots can subsequently be formulated as follows:

$$\{\begin{array}{l}{P}_1^{r_2}=T_{r_1}^{r_2}{P}_1^{r_1},\hfill \\ P_2^{r_2}=T_{r_1}^{r_2}{P}_2^{r_1},\hfill \\ \dots \hfill \\ P_n^{r_2}=T_{r_1}^{r_2}{P}_n^{r_1}\hfill \end{array}$$ (1)

where $P_i^{r_1}$ and $P_i^{r_2}$ denote the tool tip coordinates of the leader robot and follower robot¹, respectively, n is the number of collected points. $T_{r_1}^{r_2}$ represents the transformation from the leader robot base frame {r₁} to the follower robot base frame {r₂}. $T_{r_1}^{r_2}$ can be resolved using the least squares method².

Fig. 2.

Robot registration scenario. A cube with markers is placed in the workspace of both robots. The two robots are manipulated in turn to touch the same markers in a prescribed sequence with the tool tips.

To evaluate the registration accuracy, 20 points were randomly selected in the shared workspace of the two robots. The robots were controlled to have their tool tips touch these points in the same order. The tip coordinates were recorded in their respective base frames, and the coordinates in the leader robot base frame were transformed using $T_{r_1}^{r_2}$ so that all of the coordinates could be directly compared. The average error of the registration was calculated according to Eq. 2 and found to be 0.4 mm.

$${\mathit{\text{error}}}_{\mathit{\text{robot}}}=\frac{1}{n}\sum _i^n{\Vert P_i^{r_2}-T_{r_1}^{r_2}{P}_i^{r_1}\Vert }_2,$$ (2)

where ∥·∥₂ is the Euclidean distance, i = 1,2,…,n, and n = 20.

2). Eyeball registration:

Following the registration of the two robots with respect to one another, the eyeball also needs to be registered so that the target point for the follower robot, i.e., the intersection point between the axial line of the instrument on the leader robot and the retinal surface, can be calculated. Since the two robots are registered in section III-A1, the eyeball only needs to be registered in the base frame of either the leader or the follower robot. The chosen robot is manipulated under the microscope view to randomly touch different points on the eyeball surface with the tool tip, and the tip coordinates are recorded. Then the eyeball center location, denoted as (e_x,e_y,e_z), and its radius, denoted as r, can be formulated as Eq. 3.

$$\{\begin{array}{l}{\left(x_1-e_x\right)}^2+{\left(y_1-e_y\right)}^2+{\left(z_1-e_z\right)}^2=r^2,\hfill \\ {\left(x_2-e_x\right)}^2+{\left(y_2-e_y\right)}^2+{\left(z_2-e_z\right)}^2=r^2,\hfill \\ \cdots \hfill \\ {\left(x_n-e_x\right)}^2+{\left(y_n-e_y\right)}^2+{\left(z_n-e_z\right)}^2=r^2\hfill \end{array}$$ (3)

where (x_i,y_i,z_i) (i = 1,2,…,n) are the coordinates of the tool tip when in contact with the eyeball surface and n is the number of collected data points.

By subtracting the first row from the remaining rows one by one, Eq. 3 can be resolved as Eq.:

$$\left[\begin{array}{ccc}{a}_1& b_1& c_1\\ a_2& b_2& c_2\\ \dots & & \\ a_{n-1}& b_{n-1}& c_{n-1}\end{array}\right]\cdot \left[\begin{array}{c}{e}_x\\ e_y\\ e_z\end{array}\right]=\left[\begin{array}{c}{d}_1\\ d_2\\ \dots \\ d_{n-1}\end{array}\right]$$ (4)

where a_i = 2(x₁ −x_i+1),b_i = 2(y₁ −y_i+1),c_i = 2(z₁ −z_i+1), $d_i=x_1^2-x_{i+1}^2+y_1^2-y_{i+1}^2+z_1^2-z_{i+1}^2$, and 1 < i < n − 1.

Then, the eyeball center (e_x,e_y,e_z) can be obtained from Eq. 4 using least squares method. Lastly the radius of the eyeball, i.e., r, can be calculated using one of the rows in Eq. 3.

Similar to the robot registration evaluation procedure described in the previous section, an evaluation experiment for the eyeball registration is performed by collecting tip coordinates as the robot is manipulated to touch the eyeball surface at random locations. The average error was obtained using Eq. 5 and found to be 0.1 mm.

$${\mathit{\text{error}}}_{\mathit{\text{eye}}}=\frac{1}{n}\sum _i^n{\Vert {\left(x_i-e_x\right)}^2+{\left(y_i-e_y\right)}^2+{\left(z_i-e_z\right)}^2-r^2\Vert }_2,$$ (5)

where (x_i,y_i,z_i) are the coordinates of the tool when touching the eyeball surface, i = 1,2,…,n is the number of collected data points, and n = 20.

B. Automatic light pipe actuating control algorithm

The goal of this study is to automatically control the light pipe mounted on the follower robot based on the position and orientation of the surgeon-controlled tool mounted on the leader robot. To achieve this, control of orientation and insertion depth of the light pipe are decoupled and considered separately. Additionally, a force-based virtual fixture is applied to keep the light pipe moving about a pivot point, which is located at the scleral port.

1). Orientation alignment control:

Orientation control of the light pipe is designed to align the axis of the light pipe with a desired point on the retina, denoted as P_d. This desired point (as shown in Fig. 3) is the intersection between the axis of the leader instrument, which can be calculated with Eq. 6, and the retinal surface, which can be formulated using the eyeball center (e_x,e_y,e_z) and the radius r calculated with Eq. 3. Given these constraints, $P_d^{r_1}$ can be obtained by solving the set of equations consisting of Eq. 6 and Eq. 3, where the eyeball center location, (e_x,e_y,e_z), and radius, r, are known from the registration procedures.

$$\{\begin{array}{l}x=x_a+\left(x_a-x_b\right)t\hfill \\ y=y_a+\left(y_a-y_b\right)t\hfill \\ z=z_a+\left(z_a-z_b\right)t,\hfill \end{array}$$ (6)

where (x_a,y_a,z_a) and (x_b,y_b,z_b) are the coordinates of two arbitrary points on the tool axis in the base frame of the leader robot, e.g., tool tip and end.

Fig. 3.

Illustration of the light pipe orientation control. The light pipe is controlled to point to the desired point P_d, which could be achieved by pivoting the light pipe around the scleral port and aligning the scleral frame {s₂} to the desired frame $\left\{s_2^{*}\right\}$. The Z-axis of the desired frame $\left\{s_2^{*}\right\}$ aligns with the vector $P_d{P}_{s_2}$. $P_d^{r_1}$ and $P_d^{r_2}$ are the coordinates of the desired point in the base frame of leader robot and that of the follower robot, respectively. ω_x and ω_y are the andangular velocities around X and Y-axis of the scleral frame {s₂}, respectively.

With this desired location on the retina calculated in the base frame of the leader robot, the point can then be transformed into the base frame of the follower robot, denoted as $P_d^{r_2}$, with Eq. 7:

$$P_d^{r_2}=T_{r_1}^{r_2}{P}_d^{r_1}$$ (7)

This transformation allows for subsequent calculations for the light pipe orientation control to be performed in the follower robot base frame.

The light pipe is constrained to pivot about the scleral port, designated as $P_{s_2}$, so that it does not apply excessive force to the sclera while making orientation adjustments³. $P_{s_2}^{r_2}$ can be obtained by solving for the intersection point between the original light pipe axis, which can be described with Eq. 6 with two arbitrary points on the light pipe axis, and the eyeball sphere from Eq. 3. As shown in Fig. 3, a scleral frame, designated as {s₂}, can be established with its origin at the scleral port and its Z-axis aligned with the axis of the light pipe. Given the robot joint angles used to achieve the current light pipe orientation, the frame {s₂} can be resolved using kinematics of SHER [20].

A vector $P_d{P}_{s_2}$ is defined to represent the desired light pipe orientation, as shown in Fig. 3. This desired orientation can be extended to establish a desired frame $\left\{s_2^{*}\right\}$, whose origin coincides with the light pipe scleral port, $P_{s_2}$, and whose Z-axis aligns with $P_d{P}_{s_2}$. The X- and Y-axes can be resolved by using SHER’s kinematics [20] and the known Z-axis direction, $P_d{P}_{s_2}$.

With the {s₂} and $\left\{s_2^{*}\right\}$ frames established, the next step is to determine the relative transformation between them so that the current scleral frame, {s₂}, can be aligned with the desired frame, $\left\{s_2^{*}\right\}$. Since the origins of both frames coincide at the scleral port, the transformation consists only of rotation without any translation. The rotation matrix of the frame {s₂}, denoted as $R_{s_2}$, represents the current orientation of the light pipe relative to the base frame r₂. Likewise, the rotation matrix of frame $\left\{s_2^{*}\right\}$, denoted as $R_{s_2^{*}}$, represents the desired orientation of the light pipe relative to the base frame r₂. The transitional rotation matrix, ΔR, which represents the transformation from the current scleral frame to the target desired frame by pivoting about the scleral port, can be calculated using Eq. 8:

$$\Delta R={\left(R_{s_2}\right)}^{-1}{R}_{s_2^{*}}.$$ (8)

This transitional rotation ΔR can be decomposed into incremental rotations about the scleral frame {s₂} X- and Y-axes, Rot(Δx) and Rot(Δy), respectively, as shown in Eq. 9:

$$\mathit{\text{Rot}}(\Delta x)\mathit{\text{Rot}}(\Delta y)=\Delta R$$ (9)

These two incremental rotations, Δx and Δy, can be used to calculate angular velocities ω_x and ω_y as shown in Eq. 10 according to a desired velocity profile (Fig. 4). The profile has two phases: a uniform phase and a deceleration phase. At the start of the control scheme, when the initial rotational offset between the current and desired orientations is large, the uniform phase sets a maximum angular velocity threshold for safety purposes. A sinusoidal trajectory is used for the deceleration phase to achieve a smooth and gradual velocity decrease as the current and desired orientations align.

$${\omega }_i=\{\begin{array}{l}A\text{sin}\left(B{\Delta }_i\right),\left|{\Delta }_i\right|\le \mathit{\text{max}}\Delta \hfill \\ \text{sign}\left({\Delta }_i\right)\mathit{\text{max}}{\omega }_{xy},\left|{\Delta }_i\right|>\mathit{\text{max}}\Delta ,\hfill \end{array}$$ (10)

where ∣·∣ represents absolute value, ω_i is the angular velocity and Δ_i is the incremental rotations in Eq. 9, i = x,y; maxΔ defines the range of deceleration phase, and maxω_xy is the maximum value for the angular velocity. A and B are the parameters tuning the desired velocity trajectory, which could be set as: A = maxω_xy,B = π/(2maxΔ). Based on experimental evaluations, maxΔ and maxω_xy are set as 0.2 rad and 0.4 rad/s, respectively.

Fig. 4.

Desired angular velocity profile that consists of two phases: a uniform phase and a sinusoidal deceleration phase. Here, only the positive part of the profile is illustrated.

2). Insertion/Retraction control:

In addition to aligning the light pipe with the desired orientation, the insertion depth of the light pipe also needs to be controlled to provide desired illumination for the retinal surface. In this control scheme, the distance between the tip of the light pipe and the retinal surface, L₂, matches the distance between the leader instrument tip and the retinal surface, L₁, as shown in Fig. 5. L₂ and L₁ can be obtained using Eq. 11. The aim of this approach is to achieve an adjustable illumination range, i.e., illuminating a larger retinal area when the leader instrument is further away from the retina and providing more focused illumination when the leader instrument is closer to the retina.

$$L_i={\Vert P_{ti}^{r_i}-P_d^{r_i}\Vert }_2$$ (11)

where $P_{ti}^{r_i}$ is the tool tip point in the robot base frame and $P_d^{r_i}$ is the intersection point between the tool axis and the retinal surface, i = 1,2.

Fig. 5.

Illustration of the insertion/retraction control of the light pipe. The distance between the light pipe tip and the retinal surface, L₂, is limited to stay within the range specified by maxL and minL. If the former condition is true, L₂ is controlled to match the distance from the leader tool tip, L₁.

The distance tracking control scheme described above for the light pipe is calculated in the scleral frame {s₂} as shown in Eq. 12. A similar desired velocity profile combining a sinusoidal phase and a uniform phase shown in Fig. 4 is used to generate the insertion/retraction velocity v_z.

$$v_z=\{\begin{array}{l}A\text{sin}(B\Delta L),\mathit{\text{ min}}L\le L_2\le \mathit{\text{max}}L\text{ and }|\Delta L|\le \mathit{\text{max}}\Delta L\hfill \\ \text{sign}(\Delta L)\mathit{\text{max}}{v}_z,\mathit{\text{ min}}L\le L_2\le \mathit{\text{max}}L\text{ and }|\Delta L|>\mathit{\text{max}}\Delta L\hfill \\ \mathit{\text{max}}{v}_z,L_2<\mathit{\text{min}}L\hfill \\ 0,\text{ other}\hfill \end{array}$$ (12)

where ΔL = L₁ − L₂, minL and maxL are the motion limits to avoid the light pipe touching the retinal surface and retracting out of the eyeball, respectively, maxΔL defines the range of the deceleration phase of the desired velocity, and maxv_z is the maximum value of the insertion/retraction velocity. A,B are parameters that are used to control the desired velocity trajectory that could be set as A = maxv_z,B = π/(2maxΔL). Based on our experimental evaluations, minL, maxL, maxΔL, and maxv_z are set as 5 mm, 20 mm, 5 mm, and 12 mm/s, respectively, for the eye phantom in this study.

3). Force-based virtual fixture adjustment:

During retinal surgery, it is possible for the eyeball to rotate within the eye socket due to involuntary movement from the patient. Translational motion of the eyeball is limited by fixing the patient’s head prior to starting the procedure. However, since the orientation and insertion/retraction controls described above pivot the tool about a virtual fixture at the scleral port, any eyeball rotation could cause excessive scleral force that may damage the eyeball tissue. To address this, a force-based virtual fixture adjustment scheme is proposed. Using the sensorized capabilities of the light pipe, if the scleral forces exceed a certain threshold, these forces are fed into an admittance controller that applies velocity in the X and/or Y direction to adjust the X and/or Y coordinates of the virtual fixture at the scleral port and accommodate for any eyeball rotation as shown in Fig. 6. The velocity trajectory shown in Fig. 4 can be modified to calculate the X and Y velocities, v_x and v_y, of the light pipe relative to the follower robot scleral frame {s₂} as shown in Eq. 13.

$$v_i=\{\begin{array}{l}A\text{sin}\left(B\Delta f_{si}^{s_2}\right),0<\Delta f_{si}^{s_2}\le \mathit{\text{max}}\Delta f_s^{s_2}\hfill \\ \text{sign}\left(f_{si}^{s_2}\right)\mathit{\text{max}}{v}_{xy},\Delta f_{si}^{s_2}>\mathit{\text{max}}\Delta f_s^{s_2}\hfill \\ 0,\text{\hspace{1em}other}\hfill \end{array}$$ (13)

where $\Delta f_{si}^{s_2}=\left|f_{si}^{s_2}\right|-f_{\mathit{\text{gate}}}$, $f_{si}^{s_2}$ is the scleral force, i = x,y; f_gate defines the threshold value of the signalling force to activate the virtual fixture adjustment controller. maxv_xy denotes the maximum value of the linear velocities, and $\mathit{\text{max}}\Delta f_S^{S_2}$ determines the deceleration range of the desired velocity trajectory. A,B are the parameters controlling the sinusoidal shape in the velocity trajectory that could be set as A = maxv_xy,B = π/(2maxΔf). Based on our experimental evaluations, f_gate, maxv_xy, and $\mathit{\text{max}}\Delta f_S^{S_2}$ are set as 50 mN, 5 mm/s, and 20 mN, respectively.

Fig. 6.

Illustration of the force-based virtual fixture. For the follower robot, the tool is controlled based on the scleral forces $f_s^{s_1}$ to pivot about the corresponding scleral ports, the locations of which are subject to changes due to eyeball rotation. For the leader robot, the user applies manipulation force at the tool handle to fully control the tool, where the handle frame is denoted as {h₁}; when the scleral force is beyond the threshold, the linear motions at X and Y directions of the tool are constrained based on the scleral force.

The above-calculated velocity components form the light pipe velocity in the scleral frame {s₂}, as shown in Eq. 14,

$$v_{s_2}^{s_2}=\left[v_x,v_y,v_z,{\omega }_x,{\omega }_y,0\right]$$ (14)

where the last component is 0 because the robot has 5 DoFs.

Then, $v_{s_2}^{s_2}$ is transformed into the follower robot base frame {r₂} as shown in Eq. 15.

$$v_{s_2}^{r_2}=A{d}_{g_{s_2}^{r_2}}{v}_{s_2}^{s_2},$$ (15)

where $A{d}_{g_{s_2}^{r_2}}=\left[\begin{array}{cc}{R}_{s_2}^{r_2}& {\widehat{p}}_{s_2}^{r_2}{R}_{s_2}^{r_2}\\ 0& R_{s_2}^{r_2}\end{array}\right]$, with $R_{s_2}^{r_2}$ as the rotation matrix and $p_{s_2}^{r_2}$ as the translation vector of the rigid transformation from the sclera frame to the robot base frame; ${\widehat{p}}_{s_2}^{r_2}$ is the skew symmetric matrix⁴ that is associated with the vector $p_{s_2}^{r_2}$.

Lastly, the robot joint velocities can be calculated using standard Jacobian pseudo-inverse solution as shown in Eq 16. and transmitted to the low-level velocity controller to actuate the robot.

$${\dot{q}}_{r_2}=J_{r_2}^{-1}{v}_{s_2}^{r_2}$$ (16)

where $J_{r_2}^{-1}$ is the pseudo-inverse of the follower robot’s velocity Jacobian matrix.

C. Virtual fixture adjustment for leader robot instrument

A variable admittance control scheme is adopted, based on our previous velocity-level admittance control method [22], to implement a virtual fixture at the scleral port for the leader robot instrument. During operation, both the user and the robot hold the tool, and interaction forces from the user’s hand applied to the instrument handle are measured and fed into the admittance controller. Tool motion is constrained at the scleral port, i.e., the tool can only pivot about the scleral port and insert/retract along the tool axis. At the same time, the scleral forces fed into the admittance controller are used to calculate the linear velocities whose directions are perpendicular to the tool axis. This controls the instrument to accommodate the eyeball rotational movements, as shown below.

Firstly, the handle force is resolved in the scleral frame (Fig. 6) as shown in Eq. 17:

$$f_h^{s_1}=A{d}_{g_{h_1}^{s_1}}{f}_h^{h_1}$$ (17)

where $f_h^{h_1}\in {ℝ}^{6\times 1}$ and $f_h^{s_1}\in {ℝ}^{6\times 1}$ are the user’s hand force in the handle frame {h₁} and scleral frame {s₁}, respectively. $A{d}_{g_{h_1}^{s_1}}$ is the adjoint transformation from the handle frame to the sclera frame, which can be written as $A{d}_{g_{h_1}^{s_1}}\left[\begin{array}{cc}{R}_{h_1}^{s_1}& {\widehat{p}}_{h_1}^{s_1}{R}_{h_1}^{s_1}\\ 0& R_{h_1}^{s_1}\end{array}\right]$, with $R_{h_1}^{s_1}$ as the rotation matrix and $p_{h_1}^{s_1}$ as the translation vector of the rigid transformation from the handle frame to the scleral frame; ${\widehat{p}}_{h_1}^{s_1}$ is the skew symmetric matrix associated with the vector $p_{h_1}^{s_1}$.

Then, the velocity of the instrument at the scleral frame $v_{s_1}^{s_1}$ can be obtained using Eq. 18:

$$v_{s_1}^{s_1}=W_1{v}_d+\gamma W_2{f}_{h_1}^{s_1}$$ (18)

where W₁ and W₂ are diagonal admittance matrices and can be set as W₁ = diag([1,1,0,0,0,0]^T),W₂ = diag([0,0,1,1,1,1]^T), γ is the admittance gain adjusted by the user in real-time via a foot pedal. v_d is the desired compensational velocity, determined by the scleral forces ($f_{sx}^{r_1}$ and $f_{sy}^{r_1}$), to control the instrument to follow the eyeball motion, given as Eq. 19.

$$v_d={\left[v_{dx},v_{dy},0,0,0,0\right]}^T$$ (19)

where v_dx and v_dy are given by Eq. 20. The trajectory described in Fig. 4 is modified for these desired velocity phases.

$$v_{di}=\{\begin{array}{l}A\text{sin}\left(B\Delta f_{si}^{s_1}\right),0<\Delta f_{si}^{s_1}\le \mathit{\text{max}}\Delta f_s^{s_1}\hfill \\ \text{sign}\left(f_{si}^{s_1}\right)\mathit{\text{max}}{v}_{xy},\Delta f_{si}^{s_1}>\mathit{\text{max}}\Delta f_s^{s_1}\hfill \\ 0,\mathit{\text{\hspace{1em}other}}\hfill \end{array}$$ (20)

where $\Delta f_{si}^{s_1}=\left|f_{si}^{s_1}\right|-f_{\mathit{\text{gate}}}$, $f_{si}^{s_1}$ are scleral force in the X or Y direction in the scleral frame {s₁}, respectively, i = x,y, f_gate, maxv_xy, $\mathit{\text{max}}\Delta f_s^{s_1}$ A, B are the parameters that can be obtained by referring Eq. 13.

Lastly, the tool velocity is resolved in the leader robot base frame using Eq. 21, which is can be further used to obtain robot joint velocity using Jacobian pseudo-inverse solution similar with Eq. 16.

$$v_{s_1}^{r_1}=A{d}_{g_{s_1}^{r_1}}{v}_{s_1}^{s_1}$$ (21)

where $A{d}_{g_{s_1}^{r_1}}\left[\begin{array}{cc}{R}_{s_1}^{r_1}& {\widehat{p}}_{s_1}^{r_1}{R}_{s_1}^{r_1}\\ 0& R_{s_1}^{r_1}\end{array}\right]$, with $R_{s_1}^{r_1}$ as the rotation matrix and $p_{s_1}^{r_1}$ as the translation vector of the rigid transformation from the sclera frame to the robot Cartesian frame; ${\widehat{p}}_{s_1}^{r_1}$ is the skew symmetric matrix that is associated with the vector $p_{s_1}^{r_1}$.

IV. Experiments and Results

A. Experimental setup

The experimental setup is shown as Fig 1. The two robot systems are connected by a local area network to minimize the delay of signal transmission between the two robot controllers. The robot control systems are programmed based on the CISST library [23]. Data transmission between the two robots is accomplished by Robot Operating System (ROS) [24]. The controllers run at a rate of 2 kHz. The force sensing tools are mounted at the end effectors of the SHERs, and FBG interrogators (SI 115 and SM 130, Micron Optics Inc., GA USA) collect FBG sensor readings at a 2 kHz refresh rate. An eye phantom created using silicon rubber is placed into a 3D-printed socket. The eye phantom can freely rotate, mimicking the human eyeball. A binocular microscope (ZEISS, Germany) is utilized to provide a magnified view of the target and inner part of the eyeball. A Point Grey camera (FLIR Systems Inc., BC, Canada) is attached on the microscope for recording purpose, and a monitor is used to display the camera view for easy visualization of the eyeball interior.

B. Static evaluation

To statically evaluate the tracking accuracy of the proposed system, two non-force-sensing tools are mounted on the end effectors of the leader and follower SHERs. Long, thin needles, with a diameter of 0.3 mm, are inserted into each tool. The needles can manually telescope within the tool shafts to provide physical extensions of the needle trajectories along the tool axes. 20 paper targets, each with 3 printed concentric circles, are glued uniformly on the bottom surface of the eyeball phantom as shown in Fig. 7. The diameters of circles are 1 mm, 2mm, and 3mm, respectively. For the experiment, the two tools are passed through separate scleral ports into the eyeball. The user controls the leader robot and directs the leader tool tip towards the center of a target, and the follower robot simultaneously and automatically adjusts accordingly. The experiments are carried out according to the following steps:

• Insert the two tools into the eyeball through the scleral ports, then switch the follower robot control to automatic mode;

• Manipulate the leader robot to align the tool with one of the targets. The follower robot moves automatically to the same target;

• With both robots stationary, insert the telescoping needles into both leader and follower tools, and extend the needles so they are touching the eyeball surface;

• Record the position tracking error e_p based on the location of two extended tool tips:

  • e_p = 1, if two tips are located inside of the inner circle (Φ = 1mm);
  • e_p = 2, if two tips are located inside of the middle circle (Φ = 2mm) but one of the tips or both tips is/are out of the inner circle;
  • e_p = 3, if two tips are located inside of the outer circle (Φ = 3mm) but one of the tips or both tips is/are out of the middle circle;
• Measure and calculate the distance tracking error e_d:

  • Secure the telescope needles within the tools;
  • Switch the robot controls to manual control mode and retract the tool outside of the eyeball;
  • Measure the length of the needles that extend beyond the tool tips using calipers with resolution of 0.01 mm to obtain distances: L₁ and L₂.
  • Calculate the distance tracking error e_d = ∣L₁ − L₂∣.

• Repeat the above-mentioned procedures until all the targets have been used.

Fig. 7.

The microscope view of the tracking accuracy evaluation experiment. Paper targets are attached on the bottom surface of the eyeball. Telescoping needles are inserted inside the tools to serve as physical extensions and touch the target.

To evaluate system repeatability, the experiment is performed a total of 10 times. The results show that the average position-tracking error is 2.0 mm with standard deviation of 0.6 mm, and the average distance-tracking error is 0.37 mm with standard deviation of 0.22 mm. The distributions of the position-tracking error and distance-tracking error are shown in Fig. 8.

Fig. 8.

Box-plot for position-tracking offset and depth-tracking error.

C. Dynamic evaluation

Following the static evaluation, a dynamic evaluation experiment is performed to verify the tracking accuracy and assess the ability of the force-based virtual fixtures to maintain scleral forces within a given threshold while the tools are moving. The force-sensing light pipe and the force-sensing cannula are used to collect real-time scleral forces. A paper cutout printed with several curved lines representing retinal vessels is glued on the eyeball inner surface to serve as a target. The different colored curved lines all intersect at the central point, which represents the optic disc. At the start of the dynamic experiment, the two tools are inserted into the eyeball through the scleral ports. The user then manipulates the leader robot to follow the vessels with the cannulation tool tip, while the light pipe is automatically controlled by the follower robot to provide illumination to the desired point on the retina. During the vessel following, the user keeps a close distance between the cannula tool tip and the eyeball surface, whereas the light pipe tip is maintained at a safe separation distance according to Eq. 12. The vessel following procedure is performed in a random order on the four vessels, and each are traced 10 times to assess repeatability. For this experiment, the force thresholds in Eq. 13 and Eq. 20 are both set to 50 mN.

The experiments are recorded using the camera installed on the microscope. To quantitatively evaluate the tracking accuracy of the light pipe, each frame of the video is processed by the algorithm 1.

The cannulation tool tip, P_t, is extracted, then the illumination range is estimated by a circle with the radius r. The center of the light, P_l, is calculated as shown in Fig. 9. Two metrics for assessing how well the light pipe follows the leader tool are proposed, including tracking offset d, which can be obtained using Eq. 22 and tracking offset ratio d/r.

$$d=\alpha \Vert P_l-P_t{\Vert }_2,$$ (22)

where α is the scale from single pixel of the image to mm. After processing and aggregating the results from all of the video, the average tracking offset, d was 2.45 mm with standard deviation of 1.33 mm, and the average offset ratio, d/r, was calculated to be 29% with standard deviation of 16%. The distributions of the tracking offset and offset ratio are shown shown in Fig. 10 (a) and (b).

Fig. 9.

Examples of detected illumination range (blue line) and the tool tip (green dot). The images are captured using the microscope and camera.

Fig. 10.

Box-plot for the position tracking offset and scleral forces in the dynamic experiments.

In addition, the scleral forces of two tools during the experiment were also collected and evaluated. The average force is calculated to be 29.2 mN with a standard deviation of 15.6 mN and 42.8 mN with standard deviation of 13.1 mN for the cannulation tool and light pipe, respectively. The force distributions for the cannulation tool and light pipe are shown in Fig. 10 (c) and (d), respectively.

V. Discussion

The experimental results show that the tracking offset is less than 2.5 mm. As confirmed by our clinical lead, such a small offset between the center of illumination on the retinal surface and the intersection point between the leader tool tip axis and the retinal surface would not significantly impact the surgeon, given the relatively wide illumination cone of the light pipe. Furthermore, the offset ratio, d/r is reported to be 29%, which indicates that the cannulation tool tip could be always covered by the central light cone with sufficient brightness. The distance tracking error of 0.37 mm indicates that the distance between the light pipe tip and the retinal surface relative to the same measure for the leader tool can be successfully controlled.

The average manipulation force on the sclera is approximately 50 mN for an expert clinician operating freehand on a rubber eyeball [25]. Herein, the same value was adopted as the scleral force threshold to restrict movement of the tools at the virtual fixtures. The results of the dynamic experiments show that the average scleral forces from both the leader cannulation tool and the light pipe were within the prescribed threshold shown in Fig. 10 (c) and (d), which suggests that the proposed control system can prevent injury to the sclera. It should be noted that the scleral force outliers are likely generated when the robots transition from stationary to moving. Nevertheless, the outliers fall below 120 mN, which is less than the maximum force exerted by the expert clinician in [25], so even the maximum forces exerted by either tool in this study are acceptable to avoid scleral damage.

In the static evaluation experiment, three circles with diameter differences of 1 mm were utilized to measure the tracking offset, and the measurements were observed manually, which is subject to human error. In order to minimize the inherent error, the measurements were repeated multiple times for each target, and the average value was calculated for the final reading. Additionally, the microscope was used to provide a magnified view of the targets and needle tip locations. In the dynamic evaluation experiment, since there has not been literature characterizing endo-illumination inside the eyeball during retinal surgery, a threshold-based image binarization method was used to determine the light projection area and quantify the illumination range from the video. The proposed method worked well for this study, as the light source emits light with constant intensity and the distance between the light pipe tip to the retinal surface was also constant during the dynamic experiment. Therefore, the illumination brightness also stayed fairly consistent throughout the experiment, allowing the results to be easily compared and aggregated. The dynamic evaluation experiment showed the tracking performance of SHER with user’s subjective manipulation, SHER can also follow injected movements, for e.g., the reciprocating motion with the amplitude of 240 μm at frequence of 1 Hz [26].

For the orientation alignment control, the solutions for Rot(Δx)Rot(Δy) = ΔR in Eq. 9 theoretically require sequential movements, i.e., first rotate about the X-axis by Δx then rotate about the Y-axis by Δy. However, in this case, they were considered as simultaneous rotations about the two axes. This is a valid approximation for a robot controller running at sufficiently high frequency, since Δx and Δy are continuously updated. In this case, the SHER controller runs at 2 kHz, a frequency high enough to diminish the user-perceived differences between the sequential and simultaneous controls. Also, as a qualitative assessment observed during the experiments, the follower robot was able to smoothly track the leader robot, which illustrates the effectiveness and validity of the proposed control solution.

In the robot registration process, the SHERs were manually manipulated to touch the targets with its tool tip, the tip positions were then recorded as datasets for registration. Although the data collection was performed under the vision of microscope, the process would inevitably introduce data collection error due to the user’s subjective error. Therefore, to eliminate the data noise, a bunch of datapoints were collected and the registration were treated as a linear regression problem. The least squares method was finally utilized to obtain the transformation matrix. Such solution would result in that the obtained transformation matrix would not comply with the property of a homogeneous transformation matrix. However, this negative side effect only has an insignificant impact on the results, which can be seen by the small registration error of 0.4 mm. In fact, the above compromise is caused by the inaccurate data collection procedure, which could be relieved by using non-touch data collection approaches, e.g., mounting markers at the robots’ tool tips and obtaining the tip position using a laser tracker system, which is also one of the focuses in our future work.

One limitation of the current work is that, in order to obtain the desired location (the intersection point between the leader instrument axial line and the retinal surface) to direct the light pipe, the eyeball must be preoperatively registered in the robot frame. Furthermore, the eyeball is assumed to be a standard sphere. In a clinical scenario, other available technologies could be leveraged to handle the eyeball registration step, such as image based technology, e.g. OCT [27]. However, even without registration of eyeball, the proposed automatic light pipe actuating system could still work by instead directing the light pipe to follow the leader tool tip or another desired retina location that was obtained by real-time imaging [19]. An the same time, the registration accuracy of two SHERs is obtained as 0.4 mm using the presented least square method, which could be further improved by using an optical tracking system.

In this work, two SHERs are utilized to implement the automatic light pipe holding system. As an area of future investigation, the SHER used as the leader robot could be replaced by another robotic arm, a hand-held device, or even a regular surgical instrument. Future work will also consider human-robot interactions in surgery scenarios and will employ robotic manipulators with extra DoFs to explore redundancy resolution in robot control [28].

VI. Conclusion

In this study, an automatic light pipe actuating system for bimanual retinal surgery was presented to provide robot-controlled, targeted illumination inside the eye. The system consists of a customized force-sensing light pipe and a cannulation instrument that are mounted on the end effectors of two SHERs. The leader SHER with cannulation tool is manipulated by the user to perform surgical tasks, and the follower SHER with the light pipe is controlled automatically to track the desired location on the retina. Two sets of experiments were carried out, including a static evaluation of the tracking and distance following accuracy and also a dynamic evaluation to verify that the light pipe is able to successfully illuminate the desired location on the retina without applying excessive forces to the sclera. The results show that, (a) in the static experiment, the average tracking offset was 2.02 mm with standard derivation of 0.62 mm for position and 0.37 mm with standard derivation of 0.22 mm for distance, (b) in the dynamic experiment, the average light following offset was 2.45 mm with standard derivation of 1.33 mm and the average scleral forces were 29.2 mN with standard derivation of 15.6 mN and 42.8 mN with standard derivation of 13.1 mN for the cannulation tool and light pipe, respectively. The results demonstrate practicability of the proposed system to provide automatic illumination in retinal surgery.

Biographies

Changyan He received the B.E degree in school of mechanical engineering and automation at Beijing Jiaotong University, China, in 2015 and currently is a Ph.D. candidate in Beihang Univeristy. He was a visiting Ph.D. student at the Laboratory for Computational Sensing and Robotics (LCSR), Johns Hopkins University in 2017–2019. His research interests include medical robotics and instrumentation, force control, and deep learning.

Emily Yang received her M.S.E in Robotics from the Johns Hopkins University (JHU) in 2020 and her B.S. in Mechanical Engineering from the Massachusetts Institute of Technology (MIT) in 2014. Her research interests include MRI-compatible medical devices as well as surgical robots and instrumentation.

Niravkumar Patel is a postdoctoral fellow at Laboratory for Computational Sensing and Robotics, Johns Hopkins University. He received his Bachelors degree in Computer Engineering in 2005 from North Gujarat University, India and M.Tech. degree in Computer Science and Engineering in 2007 from Nirma University, India. He received the PhD degree in Robotics Engineering from Worcester Polytechnic Institute in 2017. His current research interests include medical robotics, image guided interventions, robot assisted retinal surgeries and path planning.

Ali Ebrahimi received his M.S.E in Robotics from the Johns Hopkins University (JHU) in 2019, where he has been working toward the Ph.D. degree in Mechanical Engineering since 2017. He received his B.Sc. and M.Sc. in Mechanical Engineering from Amirkabir University of Technology (Tehran Polytechnique) and Sharif University of Technology, Tehran, Iran, in 2014 and 2016, respectively. His research interests include control theory, parameter estimation and computer vision with special applications in surgical robotics and instrumentation.

Mahya Shahbazi is currently an R&D Engineer at Google LLC, Sunnyvale, CA, USA. She was a Postdoctoral Fellow at the Laboratory for Computational Sensing and Robotics, Johns Hopkins University in 2018–2019. Mahya received her Ph.D. degree in Electrical and Computer Engineering from the University of Western Ontario in 2017. She was a visiting research scholar at the University of Alberta in 2014, and a Postdoctoral Associate at Canadian Surgical Technologies and Advanced Robotics Centre in 2017–2018. Mahya has been recipient of several prestigious awards including the 2018–2019 NSERC Postdoctoral Fellowship. Her main research interests include Medical Robotics, Haptics and Teleoperation, Human-Robot Interaction, Advanced Control Systems, and Artificial Intelligence.

Peter Gehlbach M.D., Ph.D. is the J.W. Marriott Professor of Ophthalmology at the Johns Hopkins University School of Medicine’s Wilmer Eye Institute with secondary appointment in the Whiting School of Engineering. He has built an internationally recognized retinal surgery practice and maintains a long standing interest in robotic applications to microsurgery.

Iulian Iordachita (IEEE M’08, S’14) is a faculty member of the Laboratory for Computational Sensing and Robotics, Johns Hopkins University, and the director of the Advanced Medical Instrumentation and Robotics Research Laboratory. He received the M.Eng. degree in industrial robotics and the Ph.D. degree in mechanical engineering in 1989 and 1996, respectively, from the University of Craiova. His current research interests include medical robotics, image guided surgery, robotics, smart surgical tools, and medical instrumentation.