A Novel Semi-Autonomous Control Framework for Retina Confocal Endomicroscopy Scanning

Abstract

In this paper, a novel semi-autonomous control framework is presented for enabling probe-based confocal laser endomicroscopy (pCLE) scan of the retinal tissue. With pCLE, retinal layers such as nerve fiber layer (NFL) and retinal ganglion cell (RGC) can be scanned and characterized in real-time for an improved diagnosis and surgical outcome prediction. However, the limited field of view of the pCLE system and the micron-scale optimal focus distance of the probe, which are in the order of physiological hand tremor, act as barriers to successful manual scan of retinal tissue.

Therefore, a novel sensorless framework is proposed for real-time semi-autonomous endomicroscopy scanning during retinal surgery. The framework consists of the Steady-Hand Eye Robot (SHER) integrated with a pCLE system, where the motion of the probe is controlled semi-autonomously. Through a hybrid motion control strategy, the system autonomously controls the confocal probe to optimize the sharpness and quality of the pCLE images, while providing the surgeon with the ability to scan the tissue in a tremor-free manner. Effectiveness of the proposed architecture is validated through experimental evaluations as well as a user study involving 9 participants. It is shown through statistical analyses that the proposed framework can reduce the work load experienced by the users in a statistically-significant manner, while also enhancing their performance in retaining pCLE images with optimized quality.

I. Introduction

Retinal detachment is a vision threatening condition in which the retina separates from the Retinal Pigment Epithelium (RPE) and the choroidal blood vessels that underlie the RPE. These supportive structures provide nourishment and oxygen to the attached retina. While success following surgical repair depends on a myriad of factors, one important factor correlating with functional recovery of the reattached retina is the duration of detachment [1]. Therefore, it is reasonable to speculate that visualization of changes occurring during detachment of the retina at cellular level can predict functional outcomes.

One promising approach for real-time imaging and in-vivo characterization of tissues at the cellular level is probe-based confocal laser endomicroscopy [2], [3]. pCLE is a recent optical visualization technique that translates conventional microscopy to a clinical setting. pCLE can be used to facilitate cellular level imaging of biological tissue at confined sites within the body. Effectiveness of using pCLE in a robot-assisted setting for optical biopsy has been investigated in [4], showing promising results in transvaginal peritoneoscopy.

pCLE is, however, limited by its Field-of-View (FOV). The FOV in pCLE is constrained by the small size of the fiber bundles and is typically less than half a millimeter [5]. Consequently, the number of morphological features that can be viewed at a given instant may limit accurate characterization of the tissue. To address the limited FOV in pCLE, mosaicking algorithms can be used to synthesize a larger view of the tissue [6]. While mosaicking algorithms are able to provide macro coverage of the tissue surface for accurate histopathological analysis, challenges with manual acquisition of high-quality contiguous image streams from the confocal laser endomicroscopy probe largely prohibit successful mosaic synthesis due to [6]:

• physiological hand tremor, with magnitudes up to a couple of hundreds micron [7], which is in the order of magnitude of the probe image; and

• fragility of the detached retina, which extremely limits the tolerable range of forces that can be applied to the tissue; and

• patient movement and also motion of the detached retina, which add to complexities of manual image acquisition.

Moreover, for high-quality image acquisition using pCLE, the probe should be at an optimal micron-scale distance to the tissue surface depending on the distal optics design of the probe. Consistently achieving and maintaining these levels of accuracy is extremely challenging with manual manipulation of the pCLE probe. For example, our pCLE imaging system with non-contact lens [8] (manufactured at Hamlyn Centre, Imperial College, London, England), has an optimal focus distance of around 600 μm and a focus range of 200 μm, and the image sharpness drops considerably beyond this focus range. The drop in sharpness occurs in both directions of deviation from the optimal tissue-probe distance [9]. Fig. 1 shows a comparison of the probe’s view for four cases of out-of-range, in-focus, back-focus, and front-focus, measured at a tissue distance of 2.34 mm, 1.16 mm, 0.69 mm, and fully in contact with the tissue, respectively.

Fig. 1.

Sample probe views, (a) Out-of-range, (b) Back-focus, (c) In-focus, and (d) Front-focus view.

Attempts have been made in the literature at automation of probe handling using both active and passive force-control methodologies. In [10], a hand-held device was presented to enable steady probe-tissue contact using a Cellvizio pCLE system. To maintain a desired contact between the target tissue and the pCLE probe, the device relies on contact force measurements acquired using a force sensor as an additional sensing modality. Work in this area also includes a versatile pick-up probe [5], which uses a low-air friction bearing with adaptive force control strategy to enable force-based scanning. Robotic integration of pCLE probe has also been achieved in [11], [12], by presenting a framework that combined pCLE with Optical Coherence Tomography (OCT) for large-area scanning.

While the above techniques offer promising results, they rely on external sensing modalities to provide force or distance information for axial control of the probe motion (and therefore probe-tissue distance) during scanning procedures. Precise acquisition of such measurements for retinal scanning purposes, however, is very challenging due to extremely constrained environment of a patient’s eyeball, and significantly small FOV. Further, the delicacy of the retinal tissue prohibits use of contact-based sensing modalities.

Therefore, in this paper, a novel image-based control framework is presented for semi-autonomous mosaic-based image acquisition and scan of the retina at cellular level in a sensor-less fashion. The proposed approach enables pCLE scan of the tissue, without the need for additional contact/distance-sensing modalities, such as a force sensor or an auxiliary vision system.

The setup consists of the Steady-Hand Eye Robot (SHER) [13] (developed at Johns Hopkins University), integrated with a high-speed Line-Scan Confocal Laser Endomicroscopy (LS-CLE) system with a custom non-contact fiber-bundle imaging probe (developed at the Hamlyn Centre). The proposed shared control framework enables a surgeon to use the confocal microscopy probe to scan over various areas of the tissue surface, while the robot adjusts autonomously the probe-to-tissue distance such that the confocal image quality is optimized. The autonomous adjustment of the probe-to-tissue distance is performed solely based on the feedback from the confocal image itself, eliminating the need for any external modality of distance/force measure. The proposed shared control strategy has been validated through experimental results. A set of user studies involving 9 participants is also conducted to evaluate effectiveness of the framework. It is shown that the proposed semi-autonomous platform results in statistically-significant reduction in the work load level of the users and increase in their ability of maintaining an optimized view of the pCLE images.

The rest of this paper is organized as follows: Section II presents the proposed methodology. Experimental validations are given in Section III. Section IV discusses the user study, and Section V concludes this paper.

II. The Proposed Methodology

The setup includes the SHER integrated with an LS-CLE system. SHER is a cooperatively controlled, 5-Degree-of-Freedom (DoF) robot for performing vitreoretinal surgeries. The SHER has a bidirectional repeatability of 3 μm and positioning resolution of 1 μm.

The pCLE system used for image acquisition is a high-speed line-scanning fibre bundle endomicroscope. The endomicroscope is coupled to a customized probe (Fujikura FGH-30–800G) of 30,000-core fibre bundle with distal micro-lens (SELFOC ILW-1.30). The distance between the probe and the lens is optimized so that it has an optimal focus distance of around 600 μm and a focus range of 200 μm.

The proposed control framework consists of: 1) a hybrid semi-autonomous motion controller, 2) a mid-level optimizer, and 3) a low-level controller. The high-level semi-autonomous hybrid controller includes two components:

• cooperative control [14] of the confocal endomicroscopy probe (shared control between the robot and the surgeon) in directions lateral to the scanning surface to cancel out hand tremors of the surgeon as s/he moves the probe to scan the region of interest,

• autonomous control of the probe in the axial direction (perpendicular to the surface of the tissue) to control the probe-tissue distance for an optimized image quality.

The control law for the high-level hybrid controller is given as follows:

$${\dot{x}}_{des}=K_c{\dot{x}}_{des,c}+K_a{\dot{x}}_{des,a}$$ (1)

where subscripts c and a refer to cooperative and autonomous components of the desired motion, respectively; ${\dot{x}}_{des,a}$ and ${\dot{x}}_{des,c}$, indicate the desired components of the motion derived based on the cooperative mode in the lateral direction and the autonomous mode in the axial direction, respectively. Also, K_c and K_a are projection matrices that extract the lateral and axial motions of the robot along the tool shaft, respectively. The following subsections describe each component in further details.

A. High-Level Hybrid Controller- Lateral Direction

The desired motion along the lateral direction of the tissue surface, ${\dot{x}}_{des,c}$ is specified based on an admittance cooperative scheme to provide the surgeon with control over the scanning region. SHER has a 6-DOF force/torque sensor (ATI Nano 17, ATI Industrial Automation, Apex, NC, USA) measuring user interaction forces at the tool handle, which are fed as an input to the admittance control law [13]:

$${\dot{x}}_{des,t}=\alpha F_{h,t}$$ (2)

$${\dot{x}}_{des,c}=A{d}_{rt}{\dot{x}}_{des,t}$$ (3)

where ${\dot{x}}_{des,t}$ and ${\dot{x}}_{des,c}$ are the desired robot handle velocities in the tool frame and in the robot frame, respectively. F_h,t is the interaction force applied by the user hand in the tool frame; α denotes the admittance gain, which can be controlled in real-time by the user through a foot pedal; Ad_rt is the adjoint transformation associated with the coordinate frame transformation from tool to robot coordinate systems, as follows:

$$A{d}_{rt}=\left[\begin{array}{cc}{R}_{rt}& {\widehat{p}}_{rt}{R}_{rt}\\ 0& R_{rt}\end{array}\right]$$ (4)

where ${\widehat{p}}_{rt}$ is the skew symmetric matrix that is associated with the vector p_rt, the translation component of the tool frame in the robot frame. R_rt is the rotation component of the tool frame in the robot frame.

Also, K_c in Equation 1 stands for the projection matrix applied to extract the lateral motions of the robot, given by:

$$K_c=\left[\begin{array}{cc}{R}^T{\Sigma }_{c}R& 0_{3\times 3}\\ 0_{3\times 3}& 1_{3\times 3}\end{array}\right],{\Sigma }_c=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 0\end{array}\right]$$ (5)

where R is the orientation of the tissue normal expressed in the base frame of the robot. Due to the small workspace inside the eyeball and, therefore, small variation of the retinal surface normal, it is assumed that the tissue surface normal is always aligned with the axis of the robot’s tool (confocal probe).

B. High-Level Hybrid Controller- Axial Direction

While the lateral motion of the robot is specified based on the cooperative control strategy, the desired motion along the axial direction of the tissue surface, ${\dot{x}}_{des,a}$, is adjusted autonomously by the robot in such a way that the confocal image quality is optimized. Due to the constrained space inside a patient’s eyeball, including a probe-to-tissue distance (depth) sensing apparatus is extremely challenging, if not impossible. Therefore, the image blur quality is used in this paper as an indirect and sensor-less measure of probe-to-tissue distance. Effectiveness of using image blur metrics as a depth sensing modality has been previously validated by our collaborators at Imperial College London [9]. The control strategy presented therein, however, is only effective for contact-based confocal probes, which is not a suitable type of probe for retinal scanning due to fragility of the retina tissue. Moreover, the algorithm presented in [9] is dependent on characteristics of the tissue, necessitating a pre-operative calibration phase. The calibration process requires pressing the contact-based probe onto the tissue, while collecting a series of images from the pCLE system as well as the corresponding force values applied to the tissue. Performing this calibration process along with exertion and measurement of force values, however, is not feasible for retinal scanning due to constrained working environment and fragility of the retinal tissue. Therefore, in this paper, we have developed a novel image-based approach for optimizing the image quality and sharpness without the need for any extra sensing modality, while also accommodating for the use of the non-contact LS-CLE imaging probe.

For this purpose, we investigated effectiveness of several blur metrics to use in real-time control of the robot: Crété-Roffet (CR) [15], Marziliano Blurring Metric (MBM) [16], Cumulative Probability of Blur Detection (CPBD) [17], and image intensity. Fig. 2 shows the four metrics calculated for the confocal images during an experiment. In this experiment, the robot was commanded to move from a far distance to almost touching the surface, also shown in Fig. 2. In this experiment, the optimal view was achieved around t = 160 s. As can be seen in Fig. 2, all the four metrics indicated a consistent pattern around the optimal view (maximized value for CR and MBM and minimized value for CPBD and intensity). Among these four metrics, however, the CR has the lowest level of noise, and highest signal-to-noise ratio. CR score is a no-reference blur metric with a low implementation cost and high robustness to the presence of noise in the image. CR is also capable of evaluating motion blur and focal blur, all of which make it an effective and efficient score for real-time control of robotic systems.

Fig. 2.

Evaluation of four image metrics with respect to probe-to-tissue distance. The optimal view is achieved around probe-to-tissue distance of 0.73mm.

Therefore, this metric was chosen to incorporate into our auto-focus control strategy. The working principle of the CR metric is that, compared to blurring a sharp image, re-blurring an already-blurred image will not result in significant changes in intensities of neighbouring pixels. Given an input image I of pixel size m × n, the image is first convolved with a low-pass filter, obtaining the blurred images B. The absolute differences between the before-blur and after-blur images in each pixel neighbourhood are then found as follow along the horizontal and vertical axes:

$$\begin{gathered} d_{IB,v}=\sum _{i,j=1}^{m,n}max\left(0,\left|I_{i,j}-I_{i-1,j}\right|-\left|B_{i,j}-B_{i-1,j}\right|\right) \\ {d}_{IB,h}=\sum _{i,j=1}^{m,n}max\left(0,\left|I_{i,j}-I_{i,j-1}\right|-\left|B_{i,j}-B_{i,j-1}\right|\right) \end{gathered}$$ (6)

where subscript v and h refer to vertical and horizontal axes, respectively. The differences (d_IB,v and d_IB,h) are then normalized between 0 and 1 using the following equations:

$$\begin{gathered} blu{r}_v=\frac{d_{I,v}-d_{IB,v}}{d_{I,v}} \\ blu{r}_h=\frac{d_{I,h}-d_{IB,h}}{d_{I,h}} \end{gathered}$$ (7)

where d_I,v and d_I,h are the differences in pixel neighbourhood in the original image, given as follows:

$$\begin{gathered} d_{I,v}=\sum _{i,j=1}^{m,n}\left|I_{i,j}-I_{i-1,j}\right| \\ {d}_{I,h}=\sum _{i,j=1}^{m,n}\left|I_{i,j}-I_{i,j-1}\right| \end{gathered}$$

The sharpness (quality), s, of the given image, I, is then calculated as

$$s_I=1-max\left(blu{r}_v,blu{r}_h\right)$$ (8)

Fig. 3 shows an illustration of the CR score with respect to the probe-to-tissue distance. An interesting observation is that while the metric is almost symmetric around the optimal probe-to-tissue distance, it has an asymmetric pattern at farther distances. This asymmetric pattern is used in our framework to distinguish too-far distance of the probe to the tissue. Two thresholds are defined to categorize the probe-to-tissue distance into two cases. Details of the logic is depicted in Algorithm 1. First, the current image-quality score smoothed by a moving average filter, s_I, is obtained, and the change in the score with respect to the previous frame, δs, is then calculated. The movement of the robot, δx, along the normal direction of the tissue, U, is also obtained based on the current and previous values of the robot position (x_robot,curr and x_robot,prev, respectively). When the image quality score is below T₁ (example shown in Fig. 1a), the probe is specified as too far away from the scanning surface by the system, transferring the full control of the robot along both lateral and axial directions to the surgeon. When the score is higher than T₁, the hybrid motion control is activated to autonomously control the probe along the axial direction and towards the optimized image view.

Fig. 3.

CR score with respect to the probe-to-tissue distance.

Due to the relatively symmetric pattern of the CR score when greater than T₁ (shown in Figs. 1b and 1d, and also in Fig. 3), the absolute image-quality is insufficient for determining the axial direction of the movement to increase the image quality. Therefore, a stochastic gradient-ascent approach is used by taking into account the past states of the image scores and the robot motion directions. When the robot motion between the current and previous frames has the same sign as the variation of the image score from the previous to the current frames, the algorithm commands the robot to continue moving in the same direction, and otherwise, to reverse the motion direction.

Any score larger than T₂ (shown in Fig. 1c) is considered by the algorithm as an appropriate stopping condition, for which axial motion of δx = 0 is sent to the robot. Therefore, within the optimal range, i.e. s_I > T₂, the robot axial motion does not change unless the score drops beyond the T₂, possibly due to the eyeball motion or a sudden curvature change on the surface being scanned. In this case, since the robot has not had a previous state motion to use for identifying the right direction to move, the robot will perform an exploration of one sampling step to specify the appropriate motion direction. As a safety measure, this exploration step is performed by moving away from the tissue to ensure that the probe does not go into a contact with the tissue as a result of the exploration step.

Here, the threshold value of T₁ is chosen as 0.1. T₂ is chosen as 0.47, where the image view looks sharp visually. Fig. 4 shows a comparison of 3 images with CR scores of 0.45, 0.47 and 0.61 respectively. It should be mentioned that, since the optimization algorithm is based on a gradient-ascent approach, the algorithm performance is not dependent on the exact values of the threshold, as far as the chosen value corresponds to a visually clear image.

Fig. 4.

Sample probe views of different CR score, (a) CR=0.45 at 0.51 mm, (b) CR=0.47 at 0.73 mm, and (c) CR=0.61 at 0.62 mm.

The desired δx is calculated based on 1 − s_I from the auto-focus algorithm to ensure a smaller step sizes near the optimal-view region. δx is then being sent to the hybrid control law in Equation 1 based on ${\dot{x}}_{des,a}=\frac{\delta x}{\delta t}$, where δt stand for the control loop sampling time.

Using the projection matrix K_a, given below, the axial component of the motion is extracted to be input into the hybrid control law in Equation 1.

$$K_a=\left[\begin{array}{cc}{R}^T{\Sigma }_{a}R& 0_{3\times 3}\\ 0_{3\times 3}& 1_{3\times 3}\end{array}\right],{\Sigma }_a=\left[\begin{array}{ccc}0& 0& 0\\ 0& 0& 0\\ 0& 0& 1\end{array}\right]$$ (9)

Due to the different sampling frequency of pCLE images and robot control loop, the same value is used until the next smapling of image and the image-feedback gain is tuned such that the performance is optimal.

C. Mid-Level Optimizer and Low-Level Controller

The output of the high-level hybrid controller, ${\dot{x}}_{des}$, is then sent to a mid-level optimizer to calculate the corresponding desired joint values, while satisfying the joint limits of the robot:

$$\underset{{\dot{q}}_{des}}{min}\left|J\left(q\right)q-{\dot{x}}_{des}\right|$$ (10)

subject to

$${\dot{q}}_l\le \dot{q}\le {\dot{q}}_u\&q_l\le q\le q_u$$

where J(q) stands for the Jacobian of the SHER. In addition, q_l, q_u, ${\dot{q}}_u$, and ${\dot{q}}_u$ refer to the lower and upper limits over the joint values and joint velocities, respectively.

Finally, the desired joint motions are sent to the low-level PID controller of the SHER to satisfy the desired objectives generated based on the high-level hybrid semi-autonomous controller. Fig. 5 shows the block diagram of the overall architecture.

Fig. 5.

General schematic of the proposed hybrid control strategy.

III. Experimental Results

The experimental setup is shown in Fig. 6. The confocal imaging probe is attached to the SHER through a tool holder. The surgical microscope is placed directly above the eyeball phantom to visualize the tool movement. The sampling frequency of the pCLE is 60 Hz, and the control loop of the robot runs at a frequency of 200 Hz. The software framework is built upon the open source cisst library developed at Johns Hopkins University [18].

Fig. 6.

Experimental setup showing the confocal endomicroscopy system, steady-hand eye robot, surgical microscope, and an artificial eye phantom.

To validate effectiveness of the proposed framework, two experiments were conducted: 1) using the SHER with its traditional co-operative framework, and 2) using the SHER augmented with the proposed semi-autonomous framework. The task was defined to scan a circular path with a radius of around 1.5cm, and in both cases, the user was instructed to try to maintain the maximum image quality. In this set of experiments, the user was highly familiar with the SHER and manipulating the tool using the SHER in micron scales. Elbow support was provided to improve ergonomics. Fig. 7a shows a 3-D comparison of the path between the two cases. As can be seen, the semi-autonomous mode has resulted in a much smoother path, where the deviations along Z direction (probe shaft) has been considerably minimized. In addition, Fig. 7b shows a comparison of the CR scores between the two experiments. As can be seen, the semi-autonomous mode has resulted in a more consistent and higher image quality, compared to the traditional cooperative framework.

Fig. 7.

Experimental results for the two cases of with and without image-feedback, (a) Scanned path in the robot frame, (b) Comparison of CR scores, where the dashed line indicates the threshold T₂ used in the semi-autonomous controller.

To quantitatively evaluate and compare performance between the two experimental cases, the following metrics were extracted from the images collected during the two cases:

1. Mean CR score: The CR score throughout the experiment is collected and averaged, where a higher CR score indicates a sharper image.

2. Duration of In-focus View: The duration percentage of instances with in-focus view of the confocal image was also calculated. This was done by measuring the duration in which the CR score was higher than T₂, and then, normalizing with respect to total duration of the task.

Calculating the above metrics for the two experimental cases, it was observed that the average CR score increased from 0.4201 (with the traditional cooperative approach) to 0.5622 (with the semi-autonomous approach). The duration percentage of the in-focus view also increased from 43.85% to 85.12%, indicating a substantial improvement using the proposed framework. To further evaluate performance of the system, a user study was also conducted as described below.

IV. User Study

A. Study Protocol

The experiments were performed at the Labratory for Computational Sensing and Robotics (LCSR), JHU, Baltimore, MD. 9 participants, including 7 males and 2 females, were recruited for the study, with 8 participants being right-handed. The participants were enrolled after they read a letter of information, any question they had was answered, and they signed a consent form.

The task was defined to follow a path while maintaining the view of the pCLE image as clear as possible. The path was specified using a marked triangular region inside an eyeball phantom, where the retina tissue lies. Fig. 8 shows the view provided to the users during the experiment.

Fig. 8.

Participant’s view during the experiment, where the confocal image is overlayed on the top left of the surgical microscopic view looking through the eyeball opening.

At the beginning of the trial, the participants were given around 10 minutes to familiarize themselves with the system before they proceed with the two experiments. The order of the experiments was randomized to eliminate the learning curve effect. After the trial, the participants were asked to fill out a post-study questionnaire. The questionnaire included a form of the NASA Task Load Index (NASA TLX) survey, which is widely used to evaluate operator workload [19].

B. Metrics Extraction and Evaluation

In addition to the CR and duration of in-focus view, described in the previous section, the Task Completion Time (TCT) and the following metrics were also extracted for further investigation.

1. Cumulative Probability for Blur Detection (CPBD): CPBD is a no-reference image blur metric developed based on the study of human blur perception for varying contrast values. The metric uses probabilistic model to estimate the probability of blur detection at each edge in the image, and then pools the information by computing the cumulative probability [20]. A lower CPBD score corresponds to a sharper image.

2. Marziliano Blurring Metric (MBM): MBM is a no-reference perceptual blur metric based on the analysis of the spread of the edges in an image. It has been shown that the MBM metric has a high correlation with the subjective ratings [16]. A higher MBM score indicates a sharper image.

3. Motion Smoothness (MS): MS can be calculated based on the time-integrated squared jerk, where jerk refers to the third derivative of the end-effector position of the robot. Maximally smooth movements have minimal time-integrated jerks, which makes the MS an appropriate metric for quantitative comparison between the two modes. Discrete version of Motion Smoothness [21], MS, is defined as follows:

   $$MS=\sum _{i=0}^N\sqrt{J_x{\left(i\right)}^2+J_y{\left(i\right)}^2+J_z{\left(i\right)}^2}$$ (11)

   where

   $$J_u\left(i\right)=\frac{u_{i+2}-2u_{i+1}+2u_{i-1}-u_{i-2}}{2\Delta t^3}$$

   is an approximation of the third-order derivative of the signal in discrete domain.

Statistical analysis was also performed to compare performance between the two cases: human-robot cooperative control of the confocal microscopy probe using the SHER with and without the autonomous image-based compensation. The Mann-Whitney U test was performed to statistically evaluate the results.

C. Results and Discussion

Table I shows the participants’ responses to the NASA TLX questionnaire. Among the 6 factors and based on the participants responses, statistically significant reduction was observed in mental demand (p − value = 0.010), physical demand (p − value = 0.014), effort (p − value = 0.006), and frustration (p − value = 0.030) levels of the participants during the semi-autonomous approach compared to the traditional cooperative method. No statistically significant difference was observed in their perceived levels of temporal demand and performance between the two approaches.

TABLE I.

Results of the NASA TLX questionnaire.

	SHER; (Mean±STD)	SHER with Semi-Autonomous Control; (Mean±STD)
Mental Demand*	5.55±1.87	3.44 ± 1.23
Physical Demand*	5.88±1.53	4.22±1.56
Temporal Demand	4.00±2.23	3.11±1.90
Performance	3.22±1.78	4.22±1.78
Effort*	5.66±0.50	4.22±1.30
Frustration*	5.22±1.48	3.66±1.87

Asterisks indicate the factors with statistically significant differences.

Moreover, the boxplot of the six quantitative metrics (in-focus duration, CR, MBM, CPBD, MS, TCT) are shown in Fig. 9. Using the semi-autonomous approach, compared to the traditional cooperative method, the CR score increased from 0.35 to 0.44 (p − value = 0.0385). While the average CR for the cooperative approach was well below the threshold T₂, the semi-autonomous approach resulted in statistically significant improve in average image quality. Statistically significant improvements were also observed in CPBD and MBM scores using the semi-autonomous approach (both p − values = 0.025). The in-focus duration also increased from 30% to 50% (p − value = 0.0252) indicating a statistically significant improvement. No statistically significant differences were observed with respect to the task completion time and motion smoothness.

Fig. 9.

Results of the user study: (a) Duration of in-focus view, (b) Task completion time, (c) CR score, (d) CPBD score, (e) MBM score, and (f) MS score.

In general, the results of the user study indicated statistically-significant performance improvement and reduced demand on the users with the proposed semi-autonomous approach. We, however, identified some factors that limited further performance improvement. First of all, except one user, all of the participants, were using the SHER for the first time and had no prior experience with the robot. Due to the confined space in the eyeball phantom, the novice users were overwhelmed by the complexity of the task and experience high levels of work load, as also reflected in their NASA TLx questionnaire. This limited their ability in properly controlling the robot to efficiently perform the task. Therefore, a part of our future work will focus on user studies involving participants with expertise in using the SHER and with micro-surgical procedures.

Another source of limitation in further performance enhancement through the semi-autonomous framework was the assumption of the retina surface having similar normal direction as the axis of the probe shaft. After careful examination of the videos collected during the user studies from of the confocal probe moving inside the eyeball phantom, it was concluded that the assumption should be relaxed for an enhanced performance. This will be another part of our future work by taking into account an approximate model of the eyeball curvature based on pre-operative information.

Another source of limitation was the slow image transmission rate of our pCLE system. In the current setting, the Transmission Control Protocol/Internet Protocol (TCP/IP) is being used to transmit the images captured by the pCLE to the robot computer. TCP/IP is rather unfavorable for real-time control, as it can introduce considerable delays into the control loop [22]. Therefore, another aspect of our future work will focus on replacing the TCP/IP with User Datagram Protocol (UDP), which is a more efficient transmission protocol for real-time control purposes.

V. Conclusions and Future Work

A novel hybrid semi-autonomous control framework was presented for enabling pCLE scan of the retinal tissue. The platform consists of the SHER integrated with a pCLE system, along with a novel hybrid semi-autonomous control framework. Using the proposed architecture, the surgeon is able to control the motion of the confocal probe to scan the retinal tissue in a tremor-free manner, while the system optimizes the pCLE image quality in an autonomous manner without the need for any extra sensing modality for the probe-to-tissue distance. The proposed architecture was evaluated through experiments and a set of user study involving 9 participants. Statistical analyses were performed and it was shown that the proposed architecture results in decreased work load experienced by the users in a statistically-significant manner, while enhancing their performance in maintaining optimized quality/sharpness of the pCLE images. Limitations and various aspects of our future work were also discussed.