Spotlight-based 3D Instrument Guidance for Retinal Surgery

Abstract

Retinal surgery is a complex activity that can be challenging for a surgeon to perform effectively and safely. Image guided robot-assisted surgery is one of the promising solutions that bring significant surgical enhancement in treatment outcome and reduce the physical limitations of human surgeons. In this paper, we demonstrate a novel method for 3D guidance of the instrument based on the projection of spotlight in the single microscope images. The spotlight projection mechanism is firstly analyzed and modeled with a projection on both a plane and a sphere surface. To test the feasibility of the proposed method, a light fiber is integrated into the instrument which is driven by the Steady-Hand Eye Robot (SHER). The spot of light is segmented and tracked on a phantom retina using the proposed algorithm. The static calibration and dynamic test results both show that the proposed method can easily archive 0.5 mm of tip-to-surface distance which is within the clinically acceptable accuracy for intraocular visual guidance.

I. Introduction

Robot-assisted surgery (RAS) setups are known as the solution for reducing the work intensity, increasing the surgical outcomes and prolonging the viable service time of experienced surgeons in ophthalmic surgery. The introduction of autonomy into RAS can potentially help the surgeon perform the surgery with better outcomes and higher efficiency [1]. Retinal surgery contains a complex workflow and delicate tissue manipulation (see Fig. 1), which needs critical surgical skills and considerations. In 2019, there have been more than 342 million patients having retinal disease, mainly with age-related macular degeneration (196 million) and diabetic retinopathy (146 million) [2]. Many of these patients lack access to proper and timely treatment and increase their chances of blindness [3].

Fig. 1.

(a) The robot-assisted retinal surgery setup on the ex-vivo pig eye using iRAM!S robot [4]. The robot hold the instrument instead of the surgeon’s hand. (b) The cross-section view of surgical setup. The needle movement is restricted by RCM control where two translation motions are constrained.

In September 2016, surgeons at Oxford’s John Radcliffe Hospital performed the world’s first robot-assisted eye surgery. The eye surgical robot named Robotic Retinal Dissection Device (R2D2) with 10 μm position resolution was used in the clinical trials for subretinal injection [5]. They used the microscope-integrated Optical Coherence Tomography (MI-OCT) (RESCAN 700, Carl Zeiss Meditec AG., Germany) to enhance the visual feedback during needle insertion beneath the retina. The OCT imaging modality provides a very suitable resolution for retinal surgery with around 2 μm in depth but with image range limited to around 2 mm [6]. Zhou et al. [6]-[8] performed the preliminary research to show the feasibility of needle manipulation using an eye surgical robot with OCT images for navigation. However, the contradictory resolution of the image and image range results in less suitable for large range guidance of instrument movement. The instrument movement range depends on different application and cloud be treated as a volume of 10 mm×10 mm×5 mm based on the microscope view for some typical retinal surgeries, e.g. vessel tracking, subretinal injection, membrane peeling.

To navigate the instrument intraocular in 3D with a large range, Probst et al. proposed [9] the stereo-microscope vision system which uses deep learning to reconstruct the retina surface and localize the needle tip. The benefit of this method is that it will not introduce any other additional instrument inside the eye, while the drawback is that the deep learning method requires a big amount of data with annotation with different surgical tools. Different from this purely passive stereo-microscope vision system which can be influenced by the illumination, in this paper we proposed a proactive method using a spotlight source to navigate instrument 3D in retinal surgery. The contributions of this paper are listed as follows,

• The theoretical model of the spotlight projection is analyzed for planar and spherical surfaces;

• An algorithm is developed for 3D navigating the instrument with spotlight projection;

• The calibration and dynamic tests are performed with Steady-Hand Eye Robot (SHER) to verify the feasibility and performance of the proposed method.

The remainder of the paper is organized as follows: in the next section, we briefly present related work. The proposed method is described in Section III. In Section IV, the performance of the proposed method is evaluated and discussed. Finally, Section V concludes this paper and presents future work.

II. Related work

There are currently some possible solutions to navigate the instrument 3D intraocular. With the benefit of suitable resolution and radiationless imaging mechanism, the OCT imaging modality now is popular not only in the retina diagnostics but also in the intraoperative operation to enhance the visual feedback for the surgeon [10]-[12]. The research on instrument localization in OCT images has attracted the attention of researchers. Zhou et al. [13] introduced a fully conventional neural network to segment the needle in volumetric OCT images when the needle is above the tissue. Weiss et al. [14] introduced a method to estimate the 5 degree of freedom (DoF) needle pose for tool navigation during subretinal injection. Gessert et al. [15] introduced a 3D convolutional neural network to directly estimate a marker’s 6D pose from the OCT volume. They used a marker with obvious geometrical features instead of a surgical instrument to simplify the problem. However, the OCT has limited image range in depth direction which makes it not suitable for large range guidance of instrument movement.

Many studies have been carried out with significant progress in the instrument tracking and localization using microscopic image with a large movement range [16]-[18]. These works achieved satisfactory results using either color-based or geometry-based features. However, due to the limitation of using purely 2D microscopic image, these methods can not provide additional depth information to navigate the instrument in 3D. To localize the instrument [19], Yang et al. used a cone beam with structured-light reconstruction to estimate surface in the coordinate system of custom-built optical tracking system (ASAP) with additional stage, which is very related to our research. After surface reconstruction, the needle tip to surface distance is estimated in the coordinate system of ASAP [20]. Differently, our proposed method integrated the beam into the instrument, thus directly estimate the tool tip to surface distance in real-time without complications of surface reconstruction.

Probst et al. proposed [9] the stereo-microscope vision system which uses deep learning to reconstruct the retina surface and localize the needle tip. This method can obtain accuracy with around 0.1 mm in 3D over a large range. The drawback is that the deep learning method requires a large amount of data for annotation with different surgical tools and while other passive vision systems are influenced by variations in illumination. To overcome this problem, we propose a novel method in which a proactive spotlight integrated into the instrument is used to navigate the instrument in 3D via a single microscope image. The projection of spotlight from a single image can be used to locate the surgical target on the retinal surface and then infer the distance to the retinal surface. An advantage of the proactive light is that projection detection is less influenced by variations in illumination.

III. Method

A. Distance estimation using spotlight projection

The spotlight contains the pattern itself with a very simple structure. The lens at the end of spotlight beam is used to concentrate the light. The light after being concentrated is a cone which intersects the surface as a circle on the surface when the light beam is targeted perpendicular to the flat surface, as shown in Fig. 2. The distance d between the spotlight end-tip and surface is calculated as follows,

$$d=kR+b$$ (1)

where b is the diameter of the circular shape when spotlight has zero distance to the plane, k is related to the property of the lens, which equals $1∕tan(\frac{\theta }{2})$, θ denotes the angle of the cone.

Fig. 2.

The basic mechanism of spotlight. The distance d can be inferred from the projected pattern.

When the spotlight is placed with a non-zero angle to plane surface, as shown in Fig. 3(a), the projection of the pattern will be an ellipse. Based on the cross-section A-A shown as in Fig. 3(c), we can obtain the following equation,

$$d_p=k{e}_1+b$$ (2)

where d_p is the distance between the spotlight end-tip and plane surface. e₁ is the short axis of the ellipse.

Fig. 3.

(a) The spotlight has a non-perpendicular angle with the plane surface. SS′ denotes the generatrix of cone where S is the spotlight end-tip and S′ is the intersection point of the generatrix and plane. e₁ and e₂ denote the short axis and long axis of the ellipse, respectively. (b) The spotlight inside a eyeball sphere. (c) The cross-section of A-A plane in (a). (d) The cross-section of B-B plane in (b). O is the the intersection point of the generatrix and sphere.

During the retinal surgery, the spotlight is projected on the intraocular surface which can be considered as a sphere with diameter of 22-23 mm [21]. Due to the projected pattern is controlled within around 2 mm light spot, which is significantly smaller than the eyeball diameter, shown in Fig. 3 (b) and (d), the projected area on the sphere can be treated as a small plane to simplify the calculation. The distance can be calculated as,

$$d_s=SO=S{S}^{\prime }+S^{\prime }O=k{e}_1+b+S^{\prime }O$$ (3)

where d_p is the distance between the spotlight end-tip and plane surface. SO can be considered as the radius r, when the distance is small. S′O is related to the value of e₁ which can be calculated as,

$$S^{\prime }O=r-\sqrt{r^2-e_1^2}$$ (4)

where r is the radius of the eyeball in the cross section of B-B. Combining Eq. 3 and Eq. 4,

$$d_s=k{e}_1+b+r-\sqrt{r^2-e_1^2}$$ (5)

The microscope for retinal surgery is considered as a pinhole camera model. If the microscope and plane are parallel to each other, the detected ellipse short axis $e_1^{\prime }$ in the microscope image has the proportional relation with the $e_1^{\prime }$,

$$d_p=\delta k{e}_1^{\prime }+b$$ (6)

where δ is the ratio factor.

However, when the spot light pattern is projected on the sphere, as shown in Fig. 4, then e₁ will be influenced by other factor which can be calculated as follows,

$$\begin{gathered} e_1=S^{\prime }Ecos\angle E{S}^{\prime }D+DEcos\angle S^{\prime }DE \\ =\delta e_1^{\prime }cos\angle E{S}^{\prime }D+DEcos\angle S^{\prime }DE \end{gathered}$$ (7)

where ∠ES′D is the angle between the tangent line of the circle and the line of S′E. Due to the fact that the view size of microscope v is within 10 mm typically, it can be inferred that DE < S′E. The minimum of ∠S′DE appears when the O locates at the edge of the view area. ∠S′DE can be calculated,

$$\begin{gathered} \angle S^{\prime }DE\approx atan(\frac{S^{\prime }F}{S^{\prime }D})\approx atan(\frac{DF}{S^{\prime }D}) \\ =atan(\frac{FPcos(asin(\frac{c}{r})∕2)}{S^{\prime }D}) \\ =atan(\frac{(r+\sqrt{r^2-c^2})cos(asin(\frac{c}{r})∕2)}{S^{\prime }D}) \end{gathered}$$ (8)

Fig. 4.

The spotlight projected on the sphere. FC denotes the center of microscope. S′P is perpendicular with FC. S′D denotes the length of e₁. E is the intersection point between S′P and FD. EG is perpendicular with OD. c denotes the distance between point O and line FC. v denotes the view size of microscope intraocular. F lies on the eyeball circle center.

Based on the above equations and v which is within 10 mm, the minimum value of ∠S′DE is more than 87.5°. Therefore, value of DEcos∠S′DE in Eq. 7 is much smaller than δS′E∠ES′D, which Eq. 7 can be rewritten as,

$$e_1\approx \delta e_1^{\prime }cos\angle E{S}^{\prime }D=\delta e_1^{\prime }cos(atan\frac{c}{\sqrt{r^2-c^2}})$$ (9)

Combining Eq. 5, we can obtain,

$$d=k\delta cos(atan\frac{c}{\sqrt{r^2-c^2}})e_1^{\prime }+b+r-\sqrt{r^2-e_1^2}$$ (10)

B. Image algorithm design

A framework is proposed to track the projection pattern on the surface from the spotlight, shown as in Fig. 5. The input image is transferred into the gray image. In order to select the small projection pattern out of the whole image for reducing the unnecessary noise from background, the square area with a width size of s is used to crop the gray image into a patch image. Since the spotlight projection pattern usually has brighter intensity compared to the rest of the background in the patch image, a fixed threshold (threshold intensity is 200 for 8 bits image) is used to segment the image into the binary image. We eliminate the noise inside the binary image, by applying a median filter followed by a Gaussian filter. To further assure the projection pattern in the patch image, the connected component labeling method is used to output the maximum number of the connected components, meanwhile, this maximum number also needs to be more than T_s. Based on the projection pattern binary image, an ellipse fitting is applied to the boundary of the projection pattern in the patch image. we constrain the ellipse’s minor axis to a value lower than m_e, where m_e is defined based on the pre-defined range value of the ellipse’s minor axis. The detailed algorithm description is shown in Algorithm 1. Algorithm 2 explains the PatchCheck(M) function in Algorithm 1, which is to make sure the boundary of patch image is inside the original image.

Fig. 5.

The framework of projection pattern tracking. Spotlight projection is fitted with ellipse and tracked in every frame.

Algorithm 1 Spotlight projection pattern tracking

$$\begin{array}{cc}\hfill & \mathrm{INPUT}:I_{RGB}-\text{Input RGB image from microscope image}\hfill \\ \hfill & \mathrm{OUTPUT}:E\{a_{min},e_x,e_y\}-\text{Output of parameters for the}\hfill \\ \hfill & \text{ellipse}\hfill \\ \hfill & RGB2GREY()\leftarrow \text{Transfer RGB image to grey image}\hfill \\ \hfill & PathcCheck()\leftarrow \text{Check whether path image boundary}\hfill \\ \hfill & \text{within the original image or not}\hfill \\ \hfill & Crop()\leftarrow \text{Crop the image}\hfill \\ \hfill & Filter()\leftarrow \text{Filter the imge with a median filter and a}\hfill \\ \hfill & \text{Gaussian filter}\hfill \\ \hfill & GREY2BW()\leftarrow \text{Transfer grey image to binary image}\hfill \\ \hfill & BwLabel()\leftarrow \text{Label the imgae with connected component}\hfill \\ \hfill & \text{method}\hfill \\ \hfill & EllipseFit()\leftarrow \text{Find the ellipse boundary in the image}\hfill \\ \hfill & 1:\mathbf{procedure}\text{TRACKINGPATTERN}(I_{RGB})\hfill \\ \hfill & 2:E\{a_{min},e_x,e_y\}=\{\},maxIndex=0,vote=0\hfill \\ \hfill & 3:\mathbf{while}{I}_{RGB}!=\varnothing \mathbf{do}\hfill \\ \hfill & 4:\mathbf{if}Mouse\_left\_click==true\mathbf{then}\hfill \\ \hfill & 5:M=Mouse\_cursor\_location\hfill \\ \hfill & 6:\mathbf{end}\mathbf{if}\hfill \\ \hfill & 7:\mathbf{if}M\{m_x,m_y\}!=\varnothing \mathbf{then}\hfill \\ \hfill & 8:I_{GREY}=RGB2GREY(I_{RGB})\hfill \\ \hfill & 9:\mathbf{if}imcount==0\mathbf{then}\hfill \\ \hfill & 10:M_c\{m_{cx},m_{cy}\}=PatchCheck(M)\hfill \\ \hfill & 11:\mathbf{end}\mathbf{if}\hfill \\ \hfill & 12:M_c\{m_{cx},m_{cy}\}=PatchCheck(M)\hfill \\ \hfill & 13:P=Crop(I_{GREY},[m_{cx}-p,m_{cy}-p,2p,2p])\hfill \\ \hfill & 14:P=Filter(P)\hfill \\ \hfill & 15:B=GREY2BW(P)\hfill \\ \hfill & 16:L=BwLabel(B)\hfill \\ \hfill & 17:E=EllipseFit(L)\hfill \\ \hfill & 18:\mathbf{if}{a}_{min}<m_e\mathbf{then}\hfill \\ \hfill & 19:\text{Return}E\{a_{min},e_x,e_y\}\hfill \\ \hfill & 20:M\{m_x=e_x,m_y=e_y\}\hfill \\ \hfill & 21:\mathbf{else}\hfill \\ \hfill & 22:E\{a_{min},e_x,e_y\}=\varnothing \hfill \\ \hfill & 23:\mathbf{end}\mathbf{if}\hfill \\ \hfill & 24:\mathbf{else}\hfill \\ \hfill & 25:E\{a_{min},e_x,e_y\}=\varnothing \hfill \\ \hfill & 26:\mathbf{end}\mathbf{if}\hfill \\ \hfill & 27:\text{Return}E\{a_{min},e_x,e_y\}\hfill \\ \hfill & 28:\mathbf{end}\mathbf{while}\hfill \\ \hfill & 29:\mathbf{end}\mathbf{procedure}\hfill \end{array}$$

Algorithm

Algorithm 2 Check patch boundary of the image

$$\begin{array}{cc}\hfill & \mathrm{INPUT}:M\{m_x,m_y\},I_{GREY}-\text{Input}\text{RGB}\text{image}\text{from}\hfill \\ \hfill & \text{microscope image}\hfill \\ \hfill & \mathrm{OUTPUT}:M_c\{m_{cx},m_{cy}\}-\text{Output of parameters for the}\hfill \\ \hfill & \text{ellipse}\hfill \\ \hfill & SizeOf()\leftarrow \text{Return the size of the image}\hfill \\ \hfill & \mathbf{function}\text{PATCHCHECK}(M\{m_x,m_y\},I_{GREY})\hfill \\ \hfill & M_c\leftarrow \{m_{cx}=0,m_{cy}=0\}\hfill \\ \hfill & [r_I,c_I]=SizeOf(I_{GREY})\hfill \\ \hfill & \mathbf{if}{m}_x<p\mathbf{then}\hfill \\ \hfill & m_{cx}=p\hfill \\ \hfill & \mathbf{else}\mathbf{if}{m}_x>c_I-p-1\mathbf{then}\hfill \\ \hfill & m_{cx}=c_I-p-1\hfill \\ \hfill & \mathbf{else}\hfill \\ \hfill & m_{cx}=m_x\hfill \\ \hfill & \mathbf{end}\mathbf{if}\hfill \\ \hfill & \mathbf{if}{m}_y<p\mathbf{then}\hfill \\ \hfill & m_{cy}=p\hfill \\ \hfill & m_{cy}=r_I-p-1\hfill \\ \hfill & \mathbf{else}\mathbf{if}{m}_y>r_I-p-1\mathbf{then}\hfill \\ \hfill & m_{cy}=r_I-p-1\hfill \\ \hfill & \mathbf{else}\hfill \\ \hfill & m_{cy}=m_y\hfill \\ \hfill & \mathbf{end}\mathbf{if}\hfill \\ \hfill & \text{Return}{M}_c\leftarrow \{m_{cx},m_{cy}\}\hfill \\ \hfill & \mathbf{end}\mathbf{function}\hfill \end{array}$$

IV. Experiment and result

To verify the proposed method, we set up the experimental platform shown in Fig. 6(a). The phantom plane and phantom eye can be selected and fixed with phantom fixation. The human retina image is printed and attached on the phantom to make the background realistic. The surgical instrument with spotlight fiber is mounted on the SHER and directed at the phantom. The spotlight fiber is created by a normal 0.5 mm diameter light fiber with super glue to create a glue lens in the end-tip shown in Fig. 6(c). This simple but efficient method creates a suitable spotlight pattern size easily. SHER is a cooperatively controlled 5-DoF robot for microsurgical manipulation in eye surgery that helps surgeons reduce hand tremor and have more stable, smooth and precise manipulations, as shown in Fig. 6(b) [22], [23]. SHER is a velocity-controlled robot and the velocity of the motors of each joint is controlled by an embedded velocity controller (Galil 4088, Galil, 270 Technology Way, Rocklin, CA 95765). The cooperative control is realized by the 6-DoF ATI Nano17 force/torque sensor (ATI Industrial Automation, Apex, NC) mounted on the end-effector mechanism shown as Fig. 6(d), the force sensor can measure the 6 DoF force and toque applied on the end-effector.

Fig. 6.

(a) The overall experimental platform with SHER. (b) The detail structure of SHER. (c) The spotlight fiber with glue made lens in the lab. The fiber is placed on the surgical tool. (d) The end effector of SHER.

We first perform the calibration experiment on the plane platform. The light fiber attached to the instrument is relocated to slightly touch the plane to sense the initial point. Afterward, the robot is constrained and controlled to move at a speed of 0.1 mm/s in the Z direction of the end effector frame shown in Fig. 6 (d). Thus the movement in Z direction is recorded as the distance change of the spotlight end-tip to surface, shown as Fig. 7, which can be denoted as the ground truth value of the distance d_r. Based on Eq. 6, the value of δk can be estimated based on the $e_1^{\prime }$ calculated from the image and d_r. Three trials with different location on the phantom plane are tested. The fitting result is shown in Fig. 8, and Table I. The average model is listed in Table I which represents the model for all data of three points. The linear fitting coefficient of determination R² is very close to 1 which means the data is fitted with the proposed model very well. The variation of the fitted model may come from the imperfect plane and the inaccuracy of the touch method for defining of initial point. The average model for the root mean square error (RSME) is 0.163 mm which demonstrates the performance of proposed method at a very slow speed 0.01 mm/s. Based on the known parameters of δk and b with the previous deduction, in the eyeball phantom, we only need to know c then we could estimate the distance from the end-tip to sphere surface. To test the performance of the estimation error on the eyeball phantom, three points location inside the phantom eyeball is selected and test with the same procedure on the phantom plane. The adjusted model with the ground truth distance are shown in Fig. 9 and Table II, which we can find that $e_1^{\prime }$ vs d_r shows linear relationship very well. The error performance shown in Fig. 10 indicates that the maximum error is within 0.372 mm. The error may cause by the simplified mathematical model and the imperfect phantom sphere surface for the eyeball model.

Fig. 7.

The projection pattern changes with distance of the spotlight end-tip to surface. The distance change at a speed of 0.1 mm/s. The initial point for 0 mm distance is recorded when the fiber slightly touch the surface.

Fig. 8.

Three trials with different location on the phantom plane. $e_1^{\prime }$ denotes the the short axis of the ellipse in microscope image. d_r denotes the ground truth distance.

TABLE I.

The three trials on the different location of phantom plane (in mm).

	δk	b	R 2	RSME
point 1	0.1496	−6.791	0.9959	0.0548
point 2	0.1343	−5.069	0.9909	0.0974
point 3	0.1347	−5.226	0.9961	0.0663
average	0.1321	−5.117	0.9703	0.1630

Fig. 9.

Three trials with different location on the phantom eyeball.

TABLE II.

The three trials on the different location of phantom sphere (in mm).

	c	standard deviation	mean absolute error	max error
point 1	5.531	0.064	0.146	0.328
point 2	6.982	0.097	0.140	0.372
point 3	4.584	0.088	0.160	0.396

Fig. 10.

The error performance for the three trials on eyeball phantom with adjusted model and ground truth distance. Index is the data point index.

In order to test the dynamic performance for the estimation of distance, we move the SHER with cooperative control mode, during which the robot is controlled by the operator’s hand. The instrument’s trajectory is calculated by the forward kinematics of the robot with updating at 200 Hz of position information from joint sensors, which is shown as Fig. 11(a). The projection of spotlight light on the phantom is recorded and calculated by the USB microscope camera with 10 Hz. The distance between spotlight end-tip and the plane phantom is calculated by the position and pose of the spotlight mounted on the instrument with the known plane formulation in space. The plane formulation is known by using the robot to touch the three points on the phantom plane. The relationship between the estimated distance error and speed of the spotlight end-tip could be found in Fig. 11(b). It shows that the estimated error has a positive correlation with the speed, with higher end-tip speeds causing larger error. This is due to the faster speed that will cause the image tail of the spotlight projection, which makes the inaccuracy of the estimation of $e_1^{\prime }$. This could be potentially improved by using a faster and more sensitive camera. In applications with a combination of OCT imaging or keep the safe distance between the instrument tip to retinal surface, 0.5 mm would be an acceptable error. Based on the observation of Fig. 11(b), to restrict the error into 0.5 mm the speed of the end-tip of spotlight should be controlled within 1.5 mm/s in the test scenario.

Fig. 11.

(a) The trajectory of spotlight pose of position in 3D space. The arrow denotes the pose of spotlight. The dot denotes the end-tip of spotlight. The blue line denotes the end-tip trajectory for the spotlight. (b) The speed of the spotlight end-tip and the estimated error for the distance between end-tip and phantom plane during motion.

V. Conclusion and future work

In this paper, we demonstrate a novel method for 3D guidance of an instrument using the projection of a spotlight in a single microscope image. The spotlight projection mechanism is analyzed and modeled on both a plane and a spherical surface. We then design and apply an algorithm to track and segment the spotlight projection pattern. Here the SHER with the spotlight setup, is used to verify the proposed idea. The results demonstrates that the static performance of the proposed method has a RSME of 0.163 mm on the plane phantom and maximum error of 0.372 mm on the spherical eyeball phantom. The dynamic error performance is dependent on the speed of the spotlight end-tip motion. To restrict the error within 0.5 mm, the speed of the end-tip of spotlight should be limited to 1.5 mm/s as evidenced in the test scenario. In next steps, we plan to validate the proposed method by performing visual navigation tasks relevant to retinal surgery and to explore potential benefits and opportunities for further refinement.