Optimization-based Concurrent Control of a High Dexterity Robot for Vitreoretinal Surgery

Abstract

Vitreoretinal surgery requires dexterity and force sensitivity from the clinician. A system to cooperatively control an integrated surgical robot for high dexterity manipulation within the eye’s vitreous space was developed and validated in simulation. The system is composed of a 2 degrees of freedom (DoF) snake-like continuum manipulator that is attached to the end-effector of a 5-DoF rigid robot arm. It is capable of receiving position and orientation commands from a 5-DoF input device in real-time, as well as following pre-planned trajectories. The manipulator is moved to each target pose in real-time, using an optimization method to calculate the inverse kinematics solution. Constraints on the position and orientation ensure the target pose does not harm the patient within the vitreous space, enabling the robot to safely assist the clinician with vitreoretinal surgery when operating in real-time. The simulation demonstrates the system’s feasibility and benefits over the existing non-dexterous system.

I. Introduction

Vitreoretinal surgery requires advanced skills at or over the limit of clinicians’ physiological capabilities. This type of surgery is performed in a confined space with restricted free motion of surgical tools. Insertion via a sclerotomy (an incision in the sclera of the eye) limits how the clinician can hold the tool. The forces exerted between ophthalmic tools and eye tissue are often well below human sensory thresholds [1]. For Epiretinal Membrane (ERM) peeling surgery [2], where a micron-scale internal limiting membrane on the retinal surface is removed (Fig. 1, A), the forces exerted by the surgeon are required to be less than 7.5 millinewtons to avoid irreversible damage to the retina [1]. As such the benefits of surgical robots can be exploited to improve patient care during vitreoretinal surgery.

Fig. 1.

Envisioned high dexterity intraocular manipulator: (A) Epiretinal membrane peeling [2]; (B) Steady Hand Eye Robot; (C) Improved integrated robotic intraocular snake robot; (D) Phantom Omni; (E) Distal snake-like tool-end inside eye phantom.

Snake robots have the benefit of higher adaptivity and flexibility in constrained environments, which increases performance and safety. Various snake robots with different mechanisms and shapes and different degrees of freedom have been developed for surgeries performed in constrained environments to maximize dexterity [3]–[7], [19]. These snake robots can be categorized into continuum with continuous elastic backbone [8], discrete with linkages and pivot joints [9], or a hybrid of the two [10]. These robots can be driven by wires and push-pull actuators. Surgical applications of snake robots include lesion treatment [11], middle ear endoscopy [5], and vitreoretinal surgery [12].

Specific to robot assisted vitreoretinal surgery, the Steady Hand Eye Robot (SHER) and the Improved Integrated Robotic Intraocular Snake (I²RIS) have been previously developed to provide increased dexterity within the vitreous space [14], [15]. The I²RIS robot has a variable neutral line mechanism that results in a dexterous snake-like manipulator enabling more reach and manipulability compared to a rigid tool. A combined robot, consisting of the I²RIS, a robot with an 2-DoF distal-end “snake” manipulator (Fig. 1, C & E), attached to the SHER, a 5-DoF positioning robot (Fig. 1, B), was previously created [12]. The system was operated with the SHER and I²RIS controlled separately [13]. In this study we developed a system to improve upon this by adding concurrent control of the SHER and I²RIS for the purpose of performing robot assisted vitreoretinal surgery in real-time. The integrated system allows the clinician to directly control the position and orientation of the tool tip via a 6-DoF Phantom Omni device (Fig. 1, D), while keeping within constraints such that the system does not contribute to risk for the patient. Direct control of the tool tip position and orientation reduces the cognitive load on the clinicians, as they would not have to consider the control of the I²RIS and SHER robots separately. The integrated system was implemented in simulation to evaluate the proposed control scheme and to facilitate future implementation of the physical system. The simulation can also potentially function as a clinical training tool for vitreoretinal surgical procedures like ERM peeling. The major contributions of this study, done in simulation, are:

• Cooperative control of SHER robot and I²RIS robot to achieve desired tool tip poses;

• Implement constraints to the system for safe real-time operation within the workspace;

• Control a high dexterity combined robot in real-time with a Phantom Omni device.

II. Method

Fig. 2 provides an overview of the system. The Robot Operating System (ROS) framework and packages (ROS Melodic, Open Robotics) are used to facilitate communication between different components of the system. The control processing runs on MATLAB (2021a, MathWorks). The robot is simulated in Gazebo (Open Source Robotics Foundation) using a Unified Robot Description Format (URDF) representation. The Phantom Omni communicates with MATLAB by publishing to the ROS /tf topic via a previously developed ROS package. Communication between the control loop and the Gazebo simulation is done over exposed ROS_Control motor controller command topics, as well as the Gazebo /link_states topic for validation.

Fig. 2.

System overview: (L) ROS; (R) Components.

A. SHER and I²RIS

The SHER has five actuated degrees of freedom [14]. It is comprised of three translational DoFs (‘X’, ‘Y’, ‘Z’) and two rotational DoFs (‘Roll’, ‘Pitch’), shown in Fig. 3. The robot is also designed with a mechanical Remote Center of Motion (RCM) mechanism, so clinicians can insert surgical tools into the vitreous space through a sclerotomy and perform procedures without large translational movement by the SHER.

Fig. 3.

Joints of combined SHER and I²RIS robot (7-DoFs).

Several design iterations of the I²RIS have been developed and tested. The most compact is the latest iteration I²RIS design. Its snake-like end has a diameter of 0.9 mm with a designed ±45° bending range [15]. Its 2-DoF (Fig. 3) actuation is made possible by four wires on a drive pulley perpendicular to the actuation direction. The relationship between the control input, the rotation of drive pulley, the control output, and the bending angle of the snake-like distal end has been determined by geometric model and confirmed by experiments [15].

B. Assumptions

To quickly achieve a proof of concept, we set up several assumptions. We assume that the robot is rigid and the dimensions from CAD models are perfect to eliminate the necessity for robot calibration. We approximate the vitreous space in the ellipsoid eyeball phantom as a perfect sphere. During an ophthalmic procedure, there is usually a microscope available to show the intraocular space. We assume the eyeball to be registered to the robot, and completely stationary during the procedure. The sclerotomy incision point is also assumed to be perfectly registered. This allowed us to quickly develop the system by only having to address issues systemic to the control system. Future work can remove these assumptions to prepare for real world application.

C. Phantom Omni

When the user moves the Phantom Omni (3D Systems Inc., Rockhill, SC), the transformation from base to tip of the device is published at 10 Hz and then extracted from the /tf ROS topic. This forms the target input, which is scaled in terms of absolute position to simplify proof of concept implementation. A relative position with clutch system would be more suitable for clinical purposes. The scaling is from the device motion space with approximate diameter of 130 mm, to the vitreous space, which has an approximate diameter of 21 mm. The scaling increases the fidelity of the user’s control of the tool tip by a factor of approximately 6.

D. Simulation

A combined CAD model (Fig. 3) was generated from files created from prior work. The model is used as a reference for the kinematics and simulation representation. Additionally, a 3D model of an eyeball was created to represent an 3D phantom for the simulation. It uses a spherical cavity to approximate the vitreous space. An opening was added from the side for visualization purposes. The combined robot’s 7 degrees of freedom are each controlled via velocity_controllers/JointPositionController from the ROS Control package. The command values for the joints are exposed as ROS topics. The SHER is controlled by joints q₁ - q₅, representing each of the physical degrees of freedom. For the I²RIS snake manipulator, two orthogonal degrees of freedom are abstracted as the joints q₆ and q₇ (Fig. 3). This is further elaborated in the Forward Kinematics section. The URDF description accurately represents the entire system in simulation.

III. Forward Kinematics

The forward kinematics equations of the system describe the position and orientation of the tool tip, given the joint states vector $\overrightarrow{q}$, as a homogeneous 4 × 4 transformation F. From the transformation we can extract the rotation R and translation $\overrightarrow{p}$.

$$F=\left[\begin{array}{cc}R(\overrightarrow{q})& \overrightarrow{p}(\overrightarrow{q})\\ \overrightarrow{0}& 1\end{array}\right]=f(\overrightarrow{q})$$ (1)

Where F is the transformation between the global reference frame and the tool tip. We can calculate this by multiplying the forward kinematics functions of the SHER and I²RIS robots, F_SHER and $F_{I^{2}RIS}$ respectively, which individually consist of variable and rigid transformations:

$$F=F_{SHER}\left(q_{1-5}\right)*F_{I^{2}RIS}\left(q_{6-7}\right)$$ (2)

A. SHER Forward Kinematics

The kinematics of the SHER consists of 3 linear transformations and 2 rotational transformations representing the joint motion between the positions of each joint. The kinematic chain is described identically to the joint chain in the robot URDF file.

B. I²RIS Forward Kinematics

I²RIS has two angles as parameters to determine the transformation of the tool tip (see Fig. 3, bottom right) [15]. These two inputs control the amount and direction of rotation between alternating sets of pairs of segments of the snake (Fig. 4, B). The direction of rotation alternates orthogonally for each line of contact between segments. q₆ is defined as bending about the I²RIS Y-axis, and q₇ is defined as bending about the I²RIS X-axis (Fig. 3). The contact between two segments of the snake can be visualized as two cylindrical surfaces rolling together without slippage (Fig. 4, C). Two virtual rotational joints are constructed in the center of virtual circles that are concentric with the cylindrical surfaces, describing rotation between two adjacent segments (Fig. 4, A). Since the virtual circles for the two surfaces on each segment do not share the same center, effectively there are two virtual joints for each segment of the manipulator. The kinematics chain of the I²RIS also matches the joint chain described in the URDF file.

Fig. 4.

I²RIS kinematics: (A) Virtual joint representation; (B) Segment bend side-view; (C) Contact region.

C. Frames of Reference

For ease of computation, multiple frames of reference were used to make it easy to perform relative additions and subtractions, as well as calculating things such as insertion distance relative to the sclerotomy (Fig. 5).

Fig. 5.

Frames of reference.

The ’Eye’ frame uses the sclerotomy as the origin and uses a different set of variables to describe the forward kinematics. The origin of the ’Robot’ frame lies on the mechanical RCM point on the tool rod. The two frames (’Eye’ and ’Robot’) use the following variables to describe the configuration:

• ${\overrightarrow{q}}_{Eye}=\left[roll\text{,}pitch,dist,q_6,q_7\right]$

• ${\overrightarrow{q}}_{Robot}=\left[x,y,z,roll,pitch,q_6,q_7\right]$

Where roll and pitch are equivalent to q₄ and q₅ respectively, and x, y, z are q₁, q₂, q₃ offset to the RCM point, and dist is the distance from the RCM point to the Sclerotomy. Calculating the Jacobian for ${\overrightarrow{q}}_{eye}$ is less computationally intensive, as it has only 5 variables instead of 7 for ${\overrightarrow{q}}_{robot}$. The offset between the ’Robot’ frame and Global frame, which is determined by the distance from initial origin of the ’Robot’ frame relative to the sclerotomy, is calculated at runtime once the sclerotomy point is registered.

IV. Control

A. Constraints

Fig. 6 shows constraints that are applied on the target pose input by the clinician or command trajectory to produce the target pose. F_rod and F_snake are the transformations in the Eye frame that result in the tool tip pose, while F_goal is the transformation between the Eye frame and the target pose. The constraints from Fig. 6 are explained:

1. The target pose is constrained to be on the nearest point on the inner vitreous space surface of the eyeball if the input pose is outside the vitreous space. This ensures the tool tip stays within the vitreous space regardless of input (Fig. 6, Left, A and supplementary video).

2. Between each movement, the next pose goal is generated with consideration of defined max_speed_pos and max_speed_rot limits, for positional and rotational velocity, respectively. This limits how fast the tool tip can be moved which is naturally important for patient safety (Fig. 6, Left, B).

3. The orientation is constrained towards the surface normal as a function of the distance from the surface (Fig. 6, Left, C). At the surface, it is restricted to be less than approximately 23 degrees from the normal direction (Fig. 6, Right, C). This ensures the bending body of the I²RIS cannot geometrically collide with the surface, while still giving the clinician enough flexibility to orient it such that the line of sight through the eye lens is not obstructed regardless of the positioning (see supplementary video).

Fig. 6.

Constraints on input pose: (A) Position restriction; (B) Velocity limit; (C) Orientation restriction.

When solving the optimized inverse kinematics, the following joint limits are applied:

1. Each joint is limited to a predefined range [13].

   $$q_{lower}<q<q_{upper}$$
2. The change in joints is proportionally scaled down to the maximum angular/linear velocities of the physical motors. This supersedes the tool tip max_speed limits if the velocities required to are too high.

   $$d{q}_{lower}<dq<d{q}_{upper}$$

A sclerotomy RCM constraint is implicity applied by the conversion from ’Eye’ to ’Robot’ frame; the inverse kinematics is solved in the ’Eye’ frame, and then published to the joints in the ’Robot’ frame.

B. Inverse Kinematics

To solve for the desired joint values at the next step, an optimization algorithm is performed iteratively within the ‘eye’ frame of reference, minimizing the objective function at each iteration. The objective function (3) is the absolute difference between the Jacobian (4) multiplied by the change in joint values, and the change from Pose_i to Pose_i+1. Minimizing the objective function for Δq gives the change in joint values that minimize the joint movement given the iteration’s Jacobian matrix. Multiple iterations move the pose closer until the error is below the threshold. If it cannot find a solution below the threshold within an set time limit the system will maintain its current state.

$$\begin{array}{l}Objective=\Vert Jacobian*\mathrm{\Delta }q-\mathrm{\Delta }X\Vert \text{,}\\ \text{subject}\text{to}\text{joint}\text{constraints}\end{array}$$ (3)

Where Δq is the change in joint vector, ΔX is the difference between the current and next pose as a 6 × 1 vector.

$$Jacobian=\frac{\delta \left[\begin{array}{l}x\left(q_{eye}\right)\hfill \\ y\left(q_{eye}\right)\hfill \\ z\left(q_{eye}\right)\hfill \\ \overrightarrow{e}\left(q_{eye}\right)\hfill \end{array}\right]}{\delta q_{eye}}=6\times 5\text{matrix}$$ (4)

Where $\overrightarrow{e}$ is a unit vector pointing in the direction of the orientation of the tool tip, in the Eye frame.

Algorithm 1.

InvKinSolver

Input: Pose Xi+1 in terms of x,y,z,$\overrightarrow{e}$, and $q_{ey{e}_i}$

Output: $q_{ey{e}_{i+1}}$

while errorpos > 0.05 mm and errorrot > 1.5 deg do

Jacobian ← [Jacobianpos(qeye); Jacobianrot(qeye)]

Xloop ← FwdKineyeorigin_tip(qeye)

ΔX ← Xi+1 – Xloop

errorpos ← norm(ΔX(1 : 3))

errorrot ← norm(ΔX(4 : 6))

Δqeye ← argminΔq||JacobianΔqeye – ΔX||

qeye ← qeye + Δqeye

end while

$X_i=\left[\begin{array}{llll}x\hfill & y\hfill & z\hfill & \overrightarrow{e}\hfill \end{array}\right]$ is the position and orientation of the tool tip currently, in the Eye frame. X_i+1 is the next position and orientation to move to.

As we cannot rotate the tool tip about its own axis because it is not physically implemented, we generate the Jacobian with the orientation represented as an unit vector $\overrightarrow{e}=\left[e_1{e}_2{e}_3\right]$ pointing in the direction of the tool tip, which can be generated with only two rotations.

V. Evaluation and Results

A. Pre-planned Path Traversal

The system was evaluated on two pre-planned paths shown in Fig. 7. The first path (A) consists of a linear traversal around the middle of the space in the x-y plane. The second path (B) is an arc along the bottom of the spherical surface. They are designed to mimic movements that are somewhat similar to the movements when doing the ERM peeling procedure. Paths are generated in the form of an array of [x y z $\overrightarrow{e}$] values, representing the points along the path. The points are loaded into a queue data structure. During each time step of the control loop, the next goal in the queue is popped and set as the new input goal.

Fig. 7.

Planned path traversal: (A) Linear path; (B) Arc on surface.

While traversing the paths, the position of the tool tip was recorded and compared with the recorded target path. Fig. 8 demonstrates that the error of the tool tip when moving along path B from Fig. 7 does not exceed the tolerance for solving the inverse kinematics. This error of less than 1 micrometer is relatively small compared to potential error from registration and the physical SHER and I²RIS systems.

Fig. 8.

Positional error between target and actual positions for arc path B.

B. Phantom Omni Real-time Control

The system was evaluated with real-time input from the Phantom Omni. The tester moves the handle to the desired position and orientation and the tool tip moves toward that input pose. The real-time control of the integrated robot system is shown in the supplementary video. As mentioned in Section II-C, the input position from the Phantom Omni is scaled down and translated to increase fidelity for the clinician.

Fig. 9 shows a test movement conducted along the bottom of the vitreous space surface, similar to the pre-planned path B from Fig. 7. The input from the Phantom Omni is followed up to the point where constraints are enforced. The position of the tool tip follows the input until it exits the spherical space, and becomes the nearest surface point to the input position. The orientation is also in alignment starting from the left, until the input exceeds the maximum deviation from the nearest surface normal towards the right, at which point the system constrains the orientation for patient safety. Fig. 10 shows a similar constrained motion of the tool tip following the user input, with a more complex test movement conducted along the bottom of the vitreous space surface in different directions.

Fig. 9.

Real-time input constrained into the intraocular space.

Fig. 10.

Movement task in the intraocular space: (A) Top view; (B) Side view.

C. Comparison between Dexterous Tool and Rigid Tool

Performance of the concurrent control of the high dexterity robot was quantitively compared between the cases where the I²RIS manipulator is dexterous and rigid like a conventional tool, respectively. Using the same control algorithm and constraints that enable direct manipulation of the tool tip, the reachable positions on the surface of the space is plotted using a dense spherical ’grid’ search. Fig. 11 shows the combined robot system offers 180% more reachable workspace compared with the non-dexterous configuration when subject to the same constraints. Additionally, in areas where both workspaces overlap, the dexterous tool enables the tool tip allows the clinician to approach from multiple angles.

Fig. 11.

Comparison of rigid vs dexterous I²RIS reachablity.

Manipulability was calculated for each accessible sample point on the surface using equation (5).

$$\begin{gathered} A=Jacobian*Jacobia{n}^T \\ ConditionNumber=\sqrt{\frac{{\lambda }_{max}(A)}{{\lambda }_{min}(A)}} \end{gathered}$$ (5)

λ(A) is the eigenvalues of matrix A and Condition number is a measure of manipulability, with lower being more manipulable.

Fig. 12 is a histogram of the condition number of the two systems at all the points sampled on the surface. The histogram shows that the I²RIS’ snake contributes significantly to the manipulability of the system.

Fig. 12.

Histogram of manipulability in region of surgical application.

D. Real-time Execution

Fig. 13 shows the time to solve for the inverse kinematics plotted against each incremental movement step for a traversal across the planned path in Fig. 7 B. Taking less time to solve than the time between each step means the system is capable of operating in real-time (at 100 Hz).

Fig. 13.

Time to solve inverse kinematics.

Fig. 14 plots the total time to traverse planned path B from Fig. 7 against the maximum allowed positional speed for the tool tip. It is evident there is a non-linear relationship between the maximum speed and the actual time it takes to traverse the path. The implication is that the time to solve for inverse kinematics does not scale proportionally with the maximum speed.

Fig. 14.

Positional speed limit vs. total traversal time.

VI. Discussion and Conclusion

The results show that the high dexterity system is accurately following the constrained input goal in simulation, which validates the forward and inverse kinematics implementation. The evaluation for manipulability and positional reachability show the benefits of the I²RIS compared to a conventional system. Of note is how the higher manipulability allows the clinician to approach the position from different angles within the cone of constraint (Fig. 6, Right), compared to the single possible configuration for any position with a traditional rigid system. The improvement to clinician dexterity can be beneficial for vitreoretinal surgical procedures such as ERM peeling. It is important to consider the compounding error of the different components of the physical robot, and how that can impact the precision of a physical system. The simulation and kinematics do not take into account any error, and currently does not have any sort of calibration built in. The assumption that the surface of the vitreous space is perfectly registered and is spherical would not hold true in reality. Additionally, in reality the surface would deform slightly with the movement of the robot, through forces applied at the sclerotomy and tool tip. Implementation on the physical hardware would be a logical next step for this work. Some other work needs to be done to ensure the control system works on a physical robot, such as calibrating the values for the I²RIS joints. Hand-eye calibration can be implemented to reduce the error of the tool tip. Fiber Bragg Grating sensors can be implemented in the I²RIS to enable force sensing on the tool tip and sclerotomy. In conclusion, this simulated system shows that it is possible to concurrently control the SHER and I²RIS robots and move the tool tip in a constrained manner. The system is able to provide real-time control of the tool tip and can do so using clinician input from a 5-DoF input device, making it a step towards sophisticated robot-assisted vitreoretinal surgery.