Investigation of Micro-motion Kinematics of Continuum Robots for Volumetric OCT and OCT-guided Visual Servoing

Abstract

Continuum robots (CR) have been recently shown capable of micron-scale motion resolutions. Such motions are achieved through equilibrium modulation using indirect actuation for altering either internal preload forces or changing the cross-sectional stiffness along the length of a continuum robot. Previously reported, but unexplained, turning point behavior is modeled using two approaches. An energy minimization approach is first used to explain the source of this behavior. Subsequently, a kinematic model using internal constraints in multi-backbone CRs is used to replicate this turning point behavior. An approach for modeling the micro-motion differential kinematics is presented using experimental data based on the solution of a system of linear matrix equations. This approach provides a closed-form approximation of the empirical micro-motion kinematics and could be easily used for real-time control. A motivating application of image-based biopsy using 3D optical coherence tomography (OCT) is envisioned and demonstrated in this paper. A system integration for generating OCT volumes by sweeping a custom B-mode OCT probe is presented. Results showing high accuracy in obtaining 3D OCT measurements are shown using a commercial OCT probe. Qualitative results using a miniature probe integrated within the robot are also shown. Finally, closed-loop visual servoing using OCT data is demonstrated for guiding a needle into an agar channel. Results of this paper present what we believe is the first embodiment of a continuum robot capable of micro and macro motion control for 3D OCT imaging. This approach can support the development of new technologies for CRs capable of surgical intervention and micro-motion for ultra-precision tasks.

I. Introduction

Continuum robots can navigate deep into human anatomy for surgical intervention. Such robots have shown their dexterous motion capabilities in confined spaces for applications involving natural orifice surgery [1–5] or single port access surgery [6], [7]. These applications typically require a large range of motion and millimeter-level precision. There are, however, a host of potential applications that require a very small micro-motion workspace with micrometer-level precision. Such applications have been first mentioned in [8] and they include micro-surgery, cellular-level surgery, and image-based biopsy which was motivated by the early works of [9] of optical coherence tomography (OCT). These potential applications motivate this investigation into the design, modeling and control of CRs with multi-scale motion capabilities (defined as macro-motion and micro-motion capabilities).

Most works focusing on multi-scale motion use serial stacking of a micro-motion and a macro-motion robot. Such solutions may be difficult to miniaturize or to design in a manner that respects size, cost, and sterilizability constraints imposed by the surgical scene. For example, miniature parallel robots have been designed for surgical applications [10], [11]. Other systems incorporated parallel mechanisms on top of serial mechanisms for surgery, e.g., [12]. To date, there are no robotic systems capable of offering macro-scale and microscale motion capabilities in a form factor that can be easily miniaturized for dexterous surgical intervention.

Works on CRs have focused mostly on several architectures reviewed in [13], [14]. The multi-backbone continuum robot (MBCR) architecture is a parallel robot architecture with constrained flexible legs. By extending the length of these legs, these robots can achieve controllable bending. Such robots can obtain a large workspace and high precision approaching the limits of visible motion when operated under normal endoscope visualization. For example, the insertable robotic effectors platform (IREP) has been shown to achieve ≈ 300μm precision under telemanipulation and a much higher motion resolution [6]. Other continuum robot architectures such as concentric tube robots [15], [16] can offer large motion. Such robots achieve their motion through changes in the static equilibrium of antagonistic pre-bent tube pairs. Althought these robots could theoretically be designed to achieve micro-motion, they are incapable of providing multi-scale motion because macro-scale motion requires a large difference in curvature between their antagonistic tube pairs. This requirement is contrary to achieving high-motion resolution for micro-scale motion.

OCT-guided robotics is still also in its infancy. In [17] we introduced the concept of OCT-guided intervention using a parallel robot and a custom B-mode OCT probe. Other works in this domain include [18–20] where a hand-held robot was introduced for stabilizing the depth of a needle based on an integrated A-mode OCT probe. Balicki et al. [21] also used depth virtual fixtures with A-mode OCT. OCT-guided needle placement using an external OCT was demonstrated in [22] for deep anterior lamellar keratoplasty. Despite this progress, to date, there are no works that present a deployable B-mode OCT probe that can be integrated within a continuum robot for closed-loop micro-motion control and 3D OCT imaging.

The goal of this work is to address the need for robots capable of multi-scale motion at a form-factor compatible with deployability constraints in surgery. Specifically, we main to establish the feasibility of continuum robot motion with a resolution of less than 5μm because we our long term goal is to provide 3D OCT and motions suitable for retinal surgery. We achieve this goal by offering a new design of MBCRs capable of multi-scale motion. These robots achieve macro-motion in the same way MBCRs do through direct actuation of their backbones. They achieve micro-motion through modulation of their static equilibrium as explained in Section II.

Our group has published two papers on the validation of these robots. In [8] we focused on offering a solution for visual tracking with sub-micron precision, validating the micro-motion capabilities of these robots, and demonstrating an early feasibility of integrating OCT into these robots. In [23] we presented a moment coupling model that does not explain why an experimentally observed micro-motion profile occurs, but rather introduces a calibration approach to capture this unexplained phenomenon. In [24] we presented a first validation of feasibility of 3D OCT imaging using these robots.

The contribution of this work relative to prior studies is in offering a solution for a continuum robot that can provide multi-scale motion. Relative to our own prior works, this paper offers new modeling insights that explain the micro-motion of these robots. Specifically, section III presents a first explanation to the turning point phenomenon, originally reported in [23] without an explanation. Section III explains this phenomenon using energy considerations and presents a detailed kinematic model explaining the source of this phenomenon based on internal kinematic constraints in multi-backbone CRs. Section IV presents the micro-motion kinematics first in a way that captures the turning point phenomenon using the internal kinematic constraints, but without accounting for mechanical losses. To overcome this shortcoming, this section also presents a second kinematics modeling approach based on experimental data. Section V presents our system integration to enable 3D OCT and OCT-guided visual servoing. Compared to our validation in [24] where the feasibility of 3D OCT using an external commercial OCT probe was demonstrated, in section VI we use a custom B-mode probe that can be easily integrated into the robot. We also present new validation of 3D OCT volume reconstruction using micro-motion. Section VII presents our feasibility demonstration of OCT-guided micro-macro motion control for targeted injection in a micro channel.

II. Continuum Robots with Equilibrium Modulation

Figure 1 and Multimedia Extension 1 depict the concept of CRs for equilibrium modulation (CREM). This concept was introduced in [8] as a modification o f t he d esign proposed in [25]. A multi-backbone continuum segment includes four tubular backbones made of superelastic NiTi. These backbones include a central backbone ② surrounded by 3 secondary backbones ⑥ which are responsible for the macro-motion actuation. All the backbones slide through a base disk ⑤ and spacer disks ③ that maintain a constant radial distance between them. All the backbones rigidly connect to the end disk ①.

Fig. 1:

(A) Continuum robot with equilibrium modulation (B) Micro-motion trajectory of end-effector.

Download video file ^{(2.6MB, m4v)}

The (green) double line arrows in Fig. 1A indicate the direct actuation of the secondary backbones in order to bend the continuum segment to achieve a desired end-disk orientation in two degrees of freedom. Micro-motion capability is obtained by controlling the insertion of Niti wires ④ inside the secondary tubular backbones. This motion is designated by (red) single line arrows in Fig. 1A. By controlling the portion of the wires inserted into the secondary backbones, the flexural rigidity along the length of the continuum robot is changed. The equilibrium modulation actuation therefore causes an indirect actuation of the end effector (EE) that changes its static equilibrium pose to a new pose. In [23] we presented experimental results that show that the micro-motion path follows the general shape depicted in Fig. 1B, but offered no explanation to the source of this phenomenon.

III. Modeling Micro Motion Kinematics

It was observed in [8], [23] through EE tracking during experiments that a counterintuitive turning point behavior occurs during insertion of the insertion wires. An example of this turning point behavior is shown in Fig. 1B. In this section, we offer what we believe is the first explanation to the source of this turning point behavior. This explanation is presented via an energy-minimization approach and a purely kinematics-based approach.

The energy minimization approach we present provides insight into why the turning point behavior occurs and predicts an ideal turning point behavior, but in experimental data (as shown in Fig. 1), we observe that the tip position does not completely return to the starting point after the insertion wires are fully inserted. We call this the observed turning point behavior. We believe this is caused by unmodeled energy dissipation (e.g. friction, hysteresis), which are nontrivial to incorporate into a mechanics model. For this reason, we present a purely kinematics-based model that modifies one of the kinematic model parameters (the bending angle β) to replicate the observed turning point behavior.

The analysis in subsections III-B and III-C provides important insight into the counterintuitive micro-motion behavior. Section IV-A provides an empirical derivation of the micro-motion Jacobian, which accounts for unmodeled effects that may shift the turning point from its theoretical value.

A. Kinematic Assumptions and Notation

Although our formulation in Section IV-A allows spatial macro-motion, in this paper we utilize micro-motion within a single bending plane only. The reader is referred to [26] for a spatial model of continuum robot kinematics. In this section, we introduce the planar kinematic modeling assumptions and notation for the micro-motion, i.e. when the insertion wires are inserted through the secondary backbones.

Figure 2 shows the continuum segment before and after the insertion of the micro-motion wires at s insertion depth. The insertion wires are synchronously slid through the secondary backbones maintaining the insertion plane, defined by the tip of the 3 wires and shown in Fig. 2, always parallel to the cross section of the continuum segment. For simplicity of description, we treat the continuum robot as two different consecutive constant-curvature subsegments. The first (proximal) subsegment is defined from the base disk of the continuum segment to the insertion plane generated by the tips of the insertion wires. The second (distal) subsegment is defined from the insertion plane to the end disk. The solid (black) outline refers to the continuum segment before the insertion of the wires. The dashed (blue) outline identifies the inserted proximal subsegment while the dotted (red) outline identifies the empty distal subsegment after insertion.

Fig. 2:

Modeling micro motion using two circular-bending subsegments. ① is the proximal subsegment terminating with the insertion plane passing through the tips of the three insertion wires. ② is the distal subsegment. Segment in solid lines is before insertion of modulation wires. Segment in dashed lines is after insertion of said wires.

The nomenclature henceforth used is listed herein. Subscripts p and d will designate properties of the proximal and distal subsegments. Also, subscripts e and f will denote the states where the proximal subsegment is empty or filled with insertion wires, respectively:

• s: insertion depth arc length along the central backbone (measured from the base disk).

• ${\Delta }_i$: the radial offset of the i-th secondary backbone from the primary backbone.

• L: continuum segment central backbone length.

• L_i: length of the i^th secondary backbone of the continuum segment.

• ${\rho }_0$: radius of curvature of the continuum segment prior to insertion of modulation wires.

• ${\rho }_p$ and ρ_d: radii of curvature of the proximal and distal segments, respectively.

• ${\theta }_0$: the angle of the upward-pointing normal to the base disk (θ₀ = π/2).

• ${\theta }_s$: the angle of the upward-pointing normal to the insertion plane (shown in Fig. 2).

• ${\theta }_L$: the continuum segment end disk angle.

• $\beta$: the change in the angle of the insertion plane when the modulation wires are inserted.

• We also make the following kinematic assumptions:

• The radius of curvature of each continuum segment are constant at each insertion step.

• The bending angle ${\theta }_L$ is constant through the entire insertion process, as shown in [27].

• The circular distal and proximal subsegments are tangent.

To identify the shape of the continuum segment during the insertion process, the following equations are derived in order to obtain the radius of curvature of the proximal and distal subsegments and to predict the EE micro-motion. We first present an energy-minimization approach that predicts a turning-point behavior similar to what was experimentally observed in [23] and shown in Multimedia Extension 1.

B. Energy Minimization Approach

We now provide a simple mechanics model that predicts the turning point via energy minimization. We make a simplifying assumption that the tubes and intermediate disks making up the MBCR structure (without any insertion wires) can be modeled as a single tube with bending stiffness k_t and precurvature u₀. These two parameters are a direct outcome of the macro-motion equilibrium configuration at which micro-motion is generated via equilibrium modulation. This precurvature is related to the bending angle:

$$u_0=\frac{1}{{\rho }_0}=\frac{{\theta }_L-{\theta }_0}{L}$$ (1)

We next consider the change in elastic energy relative to macro-motion equilibrium configuration at which there are no wires inserted. We model the insertion wires as a single beam with bending stiffness k_w and we assume straight wires. Using these assumptions, the curvature of the proximal subsegment with insertion wires is $u_p=\frac{1}{{\rho }_p}$ and the curvature of the distal subsegment without insertion wires is $u_d=\frac{1}{{\rho }_d}$. With these simplifying assumptions, the change in bending energy relative to the macro-motion equilibrium configuration is given by:

$$\begin{gathered} \Delta E=\frac{1}{2}\underset{0}{\overset{s}{\int }}\left(\underset{\text{proximal}}{\underbrace{k_t{\left(u_p-u_0\right)}^2}}+\underset{\text{wires}}{\underbrace{k_w{\left(u_p\right)}^2}}\right)ds+ \\ \frac{1}{2}\underset{s}{\overset{L}{\int }}\underset{\text{distal}}{\underbrace{k_t{\left(u_d-u_0\right)}^2}}ds \end{gathered}$$ (2)

Integration gives:

$$\Delta E=\frac{1}{2}s{k}_t{\left(u_p-u_0\right)}^2+\frac{1}{2}s{k}_w{u}_p^2+\frac{1}{2}\left(L-s\right)k_t{\left(u_d-u_0\right)}^2$$ (3)

We seek the unknown curvatures u_p and u_d that minimize the energy for a fixed end-disk angle. The rationale for the fixed end-disk angle stems from the parallel routing of all the backbones as was shown in [25]. The corresponding constrained optimization problem is therefore stated as:

$$\underset{u_p,u_d}{\min }\Delta E\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}s.t.\hspace{0.17em}}{\theta }_L=L{u}_0+{\theta }_0$$ (4)

where the constraint is given by solving (1) for θ_L.

We solve (4) with a Lagrange multiplier , V, using the following Lagrangian and conditions of optimality:

$$V=\Delta E+\lambda \left({\theta }_L-L{u}_0-{\theta }_0\right)$$ (5)

$$\begin{array}{ccc}\frac{\partial V}{\partial u_p}=0,& \frac{\partial V}{\partial u_d}=0,& \frac{\partial V}{\partial \lambda }=0\end{array}$$ (6)

Solving these equations leads to the following result:

$$\begin{gathered} u_p=\frac{k_t{u}_0-\lambda }{k_t+k_w},u_d=\frac{k_t{u}_0-\lambda }{k_t} \\ \lambda =\frac{-s{k}_t{k}_w{u}_0}{L\left(k_t+k_w\right)-s{k}_t} \end{gathered}$$ (7)

Figure 3B shows the results for the tip motion as the insertion wires are inserted from $s=0\dots L$. The plot shows multiple leafs for different ${\theta }_L$ and different ratios of $\frac{k_t}{k_w}$. We note that each leaves shows a symmetric behavior where the tip of the continuum segment reverses direction of motion at a turning point $s=\frac{L}{2}$. As the wire stiffness is increased, the amount of tip motion is increased as expected.

Fig. 3:

(A) β(s) for θ_L = 45° and for different stiffness ratio $\frac{k_t}{k_w}$, (B) Change in tip position predicted by the energy model as the insertion wires are inserted. Plots are shown for different θ_L angles and stiffness ratios $\frac{k_t}{k_w}$, (C) β(s) designed as two quintic polynomials (D) experimental (asterisks) and theoretical tip motion (solid line) using β(s) shown in C for θ_L = 45°.

In Section III-C we explain this turning point phenomenon due to a change $\beta \left(s\right)$ in the equilibrium angle ${\theta }_s$ of the proximal subsegment. These two angles are shown in Fig. 2. The angle $\beta \left(s\right)$ may be obtained if either $u_p$ or $u_d$ are given. Referencing Fig. 2, we have:

$${\theta }_s+\beta =s{u}_p+{\theta }_0$$ (8)

We then substitute ${\theta }_s={\theta }_0+s{u}_0$ (the angle ${\theta }_s$ before deformation due the inserted wires has occurred) to get:

$$\beta \left(s\right)=s\left(u_p-u_0\right)$$ (9)

Similarly, we can have from Fig. 2:

$${\theta }_L={\theta }_s+\beta +\left(L-s\right)u_d$$ (10)

Substituting ${\theta }_L={\theta }_s+\left(L-s\right)u_0$, (the angle ${\theta }_L$ prior to insertion of the wires), we have:

$$\beta \left(s\right)=\left(L-s\right)\left(u_0-u_{\mathrm{d}}\right)$$ (11)

Figure 3A shows the profiles of $\beta \left(s\right)$ for the tip motion leafs shown in Fig. 3A. This figure assumes no frictional losses.

The following subsection uses this $\beta \left(s\right)$, along with the kinematic constraints of the continuum robot to explain how the turning point behavior occurs.

C. Explaining the Source of the Turning Point Behavior Through Kinematic Constraints

Subsection III-B explained the source of the turning point behavior through an energy minimization argument. It resulted in a predicted change $\beta \left(s\right)$ of the tangent angle at the end of the proximal segment, but it did not explain internally what happens from a point of view of kinematic constraints. This section offers this explanation and also shows how the observed turning point behavior from experimental results may be explained by adjusting $\beta \left(s\right)$.

When the continuum segment is empty, it assumes a circular shape as was shown in [28]. The arc length along the central backbone corresponding with the insertion plane at insertion depth s is related to the radius of curvature as the following:

$$s={\rho }_0\left(\frac{\pi }{2}-{\theta }_s\right)$$ (12)

For a given s position of the insertion plane, we can calculate the length of the i^th secondary backbone of the proximal segment $L_{i_{p,e}}$ when empty as:

$$L_{i_{p,e}}=\left({\rho }_0-{\Delta }_i\right)\left(\frac{\pi }{2}-{\theta }_s\right)$$ (13)

where the radial offset is given by ${\Delta }_i=r\cos \left(\left(i-1\right)\pi \right)$, i = 1, 2, and r is the pitch circle radius of the spacer disks.

When the micro-motion wires are inserted through the secondary backbones, the proximal subsegment straightens and results in a change in the bending angle of the proximal segment ${\theta }_s$. Its radius of curvature changes from the initial ${\rho }_0$ to ${\rho }_{p,f}$. This change is shown in Fig. 2 by $\beta$, ${\rho }_0$ and ${\rho }_{p,f}$. Therefore, equation (12) becomes:

$$s={\rho }_{p,f}\left(\frac{\pi }{2}-{\theta }_s-\beta \right)$$ (14)

The new radius of curvature of the proximal segment is:

$${\rho }_{p,f}=\frac{s}{\frac{\pi }{2}-{\theta }_s-\beta }$$ (15)

Recalling (12), ρ_p,f may also be expressed as:

$${\rho }_{p,f}={\rho }_0+\frac{s\beta }{\left(\frac{\pi }{2}-{\theta }_s-\beta \right)\left(\frac{\pi }{2}-{\theta }_s\right)}$$ (16)

which shows that the proximal segment straightens.

The lengths of the secondary backbones in the proximal continuum segment, designated by $L_{i_{p,f}}$, are given by:

$$L_{i_{p,f}}=\left({\rho }_{p,f}-{\Delta }_i\right)\left(\frac{\pi }{2}-{\theta }_s-\beta \right)$$ (17)

Using ${\rho }_{p,f}={\rho }_0+\Delta \rho$, eq. (17) may be rewritten as:

$$\begin{gathered} L_{i_{p,f}}=\underset{L_i}{\underbrace{\left({\rho }_o-{\Delta }_i\right)\left(\frac{\pi }{2}-{\theta }_s\right)}}+ \\ \Delta \rho \left(\frac{\pi }{2}-{\theta }_s-\beta \right)-\left({\rho }_0-{\Delta }_i\right)\beta = \\ {L}_i+\Delta \rho \left(\frac{\pi }{2}-{\theta }_s-\beta \right)-\left({\rho }_0-{\Delta }_i\right)\beta \end{gathered}$$ (18)

Initially, when the insertion depth of the modulation wires is s = 0, the proximal and distal subsegments have a common radius of curvature. The length of the i^th secondary backbone of the distal subsegment is:

$$L_{i_{d,e}}=\left({\rho }_0-{\Delta }_i\right)\left({\theta }_s-{\theta }_L\right)$$ (19)

The length of the continuum segment backbones is fixed when only micro-motion control is used. This length is given by:

$$L_{i_{{\theta }_L}}=\left({\rho }_0-{\Delta }_i\right)\left({\theta }_0-{\theta }_L\right)$$ (20)

Therefore, with increased insertion depth of the modulation wires, the distal segment must curl as the proximal segment straightens. Next, we analyze the effect of the increased depth of the modulation wires on the curvature of the distal segment.

When the proximal segment is filled with the insertion wires, the length of the secondary backbones of the distal segment can be written as the length difference between the CR segment backbone and the proximal subsegment:

$$L_{i_{d,f}}=L_{i_{{\theta }_L}}-L_{i_{p,f}}$$ (21)

Equation (21) can be also written using the radius of curvature of the distal subsegment and its bending angle:

$$L_{i_{d,f}}=\left({\rho }_{d,f}-{\Delta }_i\right)\left({\theta }_s+\beta -{\theta }_L\right)$$ (22)

Comparing (21) and (22) while using (18) and (20) results in:

$${\rho }_{d,f}={\rho }_0-\frac{\left({\rho }_s-{\rho }_0\right)\left(\frac{\pi }{2}-{\theta }_s-\beta \right)}{\left({\theta }_s+\beta -{\theta }_L\right)}$$ (23)

This equation shows that distal segment curls as a result of the inserted wires in the proximal segment.

The above energy minimization and kinematic equations provides insight into the reason for the turning point. However, due to unmodeled energy dissipation, it does not exactly replicate the observed turning point of a physical system with friction. The motion leaves of a system without friction are shown in Fig. 3B to be symmetric. Also the turning point occurs at insertion depth 0.5L as can also be seen from the symmetric β(s) in Fig. 3A. In a non-ideal system this symmetry is broken and the turning point occurs with a lag (insertion depth > 0.5L) and the paths are no longer closed due to stiction-induced hysteresis. Nevertheless, the observed turning point behavior can be recovered with an experimental-based modification to β(s).

To demonstrate that our approach can replicate the turning point behavior, β(s) was modified and designed based on results presented in [8] using EE tracking. From this experimental data, it was observed that the turning point happened consistently for our prototype close to 75% of the insertion length. Using two quintic polynomials for modeling β(s) as shown in Fig. 3C, we simulated the tip motion as shown by the solid line in Fig. 3D, which exhibits a similar behavior to our experimental data shown by asterisks. An accurate representation of kinematics using experimental data is presented in section IV-A.

IV. Experimental Approach to Differential Micro-Motion Kinematics and Control

Although the kinematic model above provides insight into the turning point behavior, for purposes of control it still requires derivation of the micro-motion Jacobian and calibration of β(s) across the entire workspace. In the next sections, we present an experimental approach that is more convenient for control since it can directly determine the micro-motion Jacobian from experimental data without going through a kinematics/mechanics model.

A. Experimental-based kinematic modeling

During micro-motion control, the modulation wire depth s is changed to affect a minute change in the EE position $\text{p}={\left[x,z\right]}^T$ where x and z are the horizontal vertical coordinates of the EE in frame $\left\{R\right\}\triangleq \left\{{\widehat{\text{x}}}_r,{\widehat{\text{y}}}_r,{\widehat{\text{z}}}_r\right\}$ as depicted in Fig. 1. For the purpose of the following discussion, this frame has its ${\widehat{\text{x}}}_r$ axis as the projection of the bent backbone on plane of the base disk and its ${\widehat{\text{z}}}_r$ axis perpendicular and at the center of the base disk. These coordinates are obtained through a microscope tracking setup as described in [8], which results in the coordinates in camera frame $\left\{C\right\}\triangleq \left\{{\widehat{\text{x}}}_c,{\widehat{\text{y}}}_c,{\widehat{\text{z}}}_c\right\}$, as shown in Fig. 1. It is assumed that the transformation from the camera frame {C} to {R} is known. In the following, we explain our process for empirical modeling the micro-motion kinematics. This process will be repeated for the x and z coordinates in frame {R}, but since the process is identical, we will illustrate the process only for the x coordinate.

In the following, we adapt a modal approach first presented in [29] within the context of modeling the kinematics of soft robots. Let x(s) be the x-coordinate of the EE micro-motion path, $s\in \left[0\dots L\right]$. For a given macro motion configuration characterized by θ_L in the bending plane, the local tangent to the curve $c=\left(s,x\left(s\right)\right)$ is designated by x′:

$$x\text{'}=\frac{dx}{ds}\text{\hspace{0.17em}where\hspace{0.17em}\hspace{0.17em}}x\text{'}=x\text{'}\left({\theta }_L,s\right)$$ (24)

For a fixed ${\theta }_L$, a modal representation for $x\prime \left(s\right)$ may be used

$$x\text{'}\left({\theta }_L,s\right)=\psi {\left(s\right)}^{\text{T}}\text{a, a,}\psi \in {\mathrm{IR}}^n$$ (25)

where $\psi$ and a are the modal basis and coefficients. Since the EE micromotion path is smooth and requires a low order polynomial to approximate it, we use a monomial basis:

$$\psi \left(s\right)={\left[1,s,s^2,\dots ,s^{n-1}\right]}^T$$ (26)

To find the modal coefficients a, a data matrix $\Phi$ containing $x\text{'}\left(s\right)$ for z different CR configurations with r insertion depths $s=s_1,s_2,\dots s_r$ was constructed. The entries of $\Phi$ were filled based on the finite difference approximation of the local tangent to the experimental data x{s, θ_L).

$$\Phi =\left[\begin{array}{ccc}x\prime \left(s_1,{\theta }_{L_1}\right)& \cdots & x\prime \left(s_1,{\theta }_{L_z}\right)\\ ⋮& \ddots & ⋮\\ x\prime \left(s_r,{\theta }_{L_1}\right)& \cdots & x\prime \left(s_r,{\theta }_{L_z}\right)\end{array}\right]\in {\mathrm{IR}}^{r\times z}$$ (27)

Since the elements of a depend on ${\theta }_L$, we aggregate the modal approximations of a_i such that:

$$\begin{array}{ccc}\text{a}\left({\theta }_L\right)=\text{A}\eta \left({\theta }_L\right),& \text{A}\in {\mathrm{IR}}^{n\times m},& \eta \in {\mathrm{IR}}^m\end{array}$$ (28)

where η (θ_L) = [1, θ_L, θ_L², … , θ_L^m−1]^T.

The tangent curve $c=\left(s,x\left(s\right)\right)$ can therefore be obtained by the following modal approximation:

$$x^{\prime }\left({\theta }_L,s\right)=\psi {\left(s\right)}^{\text{T}}\text{A}\eta \left({\theta }_L\right)$$ (29)

Assuming z experiments corresponding macro configuration angles ${\theta }_1\dots {\theta }_z$ and where the insertion depth is changed with r values, the i^th column of Φ is given by:

$${\Phi }^i=\left[\begin{array}{c}x\prime \left({\theta }_{L_i},s_1\right)\\ ⋮\\ x\prime \left({\theta }_{L_i},s_r\right)\end{array}\right]=\left[\begin{array}{c}{\psi }^{\text{T}}\left(s_1\right)\\ ⋮\\ {\psi }^{\text{T}}\left(s_r\right)\end{array}\right]\text{\hspace{0.17em}A}\eta \left({\theta }_{L_i}\right)$$ (30)

and the full Φ matrix is given by:

$$\Phi =\underset{{\Omega }_{\left[r\times n\right]}}{\underbrace{\left[\begin{array}{c}{\psi }^{\text{T}}\left(s_1\right)\\ ⋮\\ {\psi }^{\text{T}}\left(s_r\right)\end{array}\right]}}{\text{A}}_{\left[n\times m\right]}\underset{{\Gamma }_{\left[m\times z\right]}}{\underbrace{\left[\begin{array}{ccc}⋮& ⋮& ⋮\\ \eta \left({\theta }_{L_1}\right)& \cdots & \eta \left({\theta }_{L_z}\right)\\ ⋮& ⋮& ⋮\end{array}\right]}}$$ (31)

This is a matrix equation with A as the unknown. It can be rewritten using Kröneker product as:

$$\left[{\Gamma }^{\text{T}}\otimes \Omega \right]\text{vec}\left(\text{A}\right)=vec\left(\Phi \right)$$ (32)

where the symbol $\otimes$ indicates the Krönecker’s matrix product [30] and vec(A) is a vector sequence of all the columns of A. Equation (32) is a linear equation with a vector unknown, which can be solved using the pseudoinverse of $\left[{\Gamma }^{\text{T}}\otimes \Omega \right]$. Finally, given the solution to A, one can calculate $c=\left(s,x\left(s\right)\right)$ by substituting ${\theta }_L$ and s in (29). Finally, the EE coordinate $x\left(s\right)$ may be obtained as:

$$x\left({\theta }_L,s\right)={\int }_0^{s}x\text{'}\left({\theta }_L,s\right)={\int }_0^s\psi {\left(s\right)}^{\text{T}}\text{A}\eta \left({\theta }_L\right)ds$$ (33)

Taking the time derivative of (33) using the chain rule:

$$\dot{x}\left({\theta }_L,s\right)=\frac{\partial }{\partial s}\left(x\left({\theta }_L,s\right)\right)\dot{s}+\frac{\partial }{\partial {\theta }_L}\left(x\left({\theta }_L,s\right)\right){\dot{\theta }}_L$$ (34)

The first term in the equation above includes micro motion speed due to the insertion wires and the second term includes the effect of a change in the macro-motion configuration on the micro-motion speed. We will next consider the scenario where the macro-motion configuration is held fixed when one invokes the micro-motion control. Therefore (34) can be rewritten as:

$$\dot{x}\left({\theta }_L,s\right)=\psi {\left(s\right)}^{\text{T}}\text{A}\eta \left({\theta }_L\right)\dot{s}=J_{{\mu }_x}\dot{s}$$ (35)

where $J_{{\mu }_x}$ is the micro-motion Jacobian matrix for the x coordinate. Repeating this process for the z coordinate will also result in similar equations and a Jacobian $J_{{\mu }_z}$.

This method was experimentally validated with our system described in section V. For each θ_L ∈ [15°, 35°, 45°, 60°, 75°] we moved our robot through 45 EE poses and the EE position was segmented using our method in [8]. Figure 4 presents the segmented EE coordinates using asterisk markers. The numerically integrated curve using (29) is shown with dashed lines. Table I presents the root mean squared error between the observed EE micro-motion paths, obtained in [8] and shown as solid lines. These results suggest that the modeling approach we presented can represent the robot kinematics well.

Fig. 4:

x and z end effector coordinates over the full insertion length at 5 different CR configurations

TABLE I:

Root mean squared error (maximum error) [μm] between measured and modeled EE micro-motion paths.

θL	15°	35°	45°	60°	75°
x	0.8 (3.3)	0.5 (2.6)	1.1 (3)	1 (2)	0.2 (0.3)
y	0.6 (4.4)	0.3 (3.5)	3.7 (3.9)	1.9 (1.2)	0.5 (0.3)

V. Experimental Setup and System Integration

Figure 5 and Multimedia Extension 2 show the experimental setup used to validate our modeling approach, to demonstrate the feasibility of 3D OCT via CREM micro-motion, and to validate our approach for OCT-guided micro-motion control. For our experiments, the continuum robot actuation unit ① and the continuum robot segment ② were used with a commercial external OCT probe ③ and a custom miniature B-mode OCT probe ④ that was embedded within the robot. The custom B-mode OCT probe is a modification of the design presented in [31] and it uses custom Arduino-based closed-loop control electronics ⑤. The probe was actuated by a voice coil actuator © (H2W NCM02-05-005-4JBMC). The external OCT probe (Thorlabs Telesto-II-1325LR-5P6) has 3.5 to 7mm imaging depth with 5.5 to 12.0μm axial resolution in air.

Fig. 5:

Experimental setup: ①-CREM system ②-Continuum Segment ③-OCT Commercial probe ④-Custom made OCT probe deployed within the CR segment ⑤-OCT/linear-slides Arduino Based control box ⑥-Voice coil OCT actuation unit ⑦-Planar Robot ⑧-High Level Control computer ⑨-Matlab RealTime computer ⑩-Mid Level Control computer ⑪-OCT COmputer ⑫-Micro-motion Camera.

Download video file ^{(10.5MB, mp4)}

The commercial probe was used for proof of concept for 3D volumetric OCT with a calibrated system that can provide quantifiable data about the accuracy of our 3D OCT and micro-motion resolution and for demonstrating closed-loop visual servoing in OCT image space. The custom OCT probe was used for testing the feasibility of embedding a custom B-mode OCT probe within the robot for generating 3D OCT scans.

In addition to the continuum segment, the setup used a planar robot ⑦ comprised of two orthogonally-aligned linear stages (VELMEX A1506B-S1.5) driven by two DC gearmotors (Maxon combination #351325) and controlled by our Arduino-based control box ⑤ . The need for adding this planar actuation unit is explained in section Section VII.

The control system used four computers depicted in Fig. 5. The bottom portion of this figure shows how the control was delineated among these machines. Although this system may be implemented on a single high-powered computer, we chose to preserve the modularity of our software environment to enable rapid-prototyping of our proof-of-concept control system. The four computers included: a high-level controller (HLC) ⑧ for communication and synchronized control, a low-level controller (LLC) for continuum robot control ⑨ using Matlab RealTime, a mid-level controller (MLC) ⑩ for acquiring EE position data from LLC and for sending the control-reference signal to the Arduino-based controller of the OCT and the planar robot ⑦, and a computer for OCT image acquisition and online image segmentation ⑪.

Data communication between these computers were carried out as the following. The HLC computer obtained the EE position via a microscope camera using MATLAB image acquisition toolbox. This information is used only for 3D OCT volume generation. The acquired scans are synchronized with robot position using time-stamped user datagram protocol (UDP) messages relayed to the MLC and the MATLAB Realtime target machine. For visual servoing, the OCT image segmentation data is relayed to the MLC via UDP communication. The low-level control frequency on the MATLAB Realtime target machine for controlling the continuum robot was 1KhZ. The MLC obtained image segmentation data and relayed the reference control signal to the Arduino at 30Hz. In turn, the Arduino-controller using HiLetgo ESP32 microcontroller implemented its own low-level PID control for the voice-coil actuator and the planar robot.

VI. Micro-Motion for Volumetric OCT Imaging

This section details the validation of our approach for achieving volumetric OCT reconstruction via micro-motion. While the final goal is to obtain a volumetric scan of a vessel using a custom-made OCT probe integrated inside the CR, we first validated feasibility using an external commercial OCT probe. These experiments were first reported in [24] and they provide an ideal baseline for expected performance where OCT image aberrations are minimized due to the use of a commercial OCT probe. This section also details validation using our own CR with a custom B-mode OCT probe that is a modification of the design first presented in [32].

A. 3D image reconstruction using commercial OCT probe

The setup used is shown in Fig. 5. We used the external OCT probe for these experiments. The CR segment was bent to θ_L = 60° using macro-motion control and the macro motion joints were held fixed. The CR segment was used to carry an OCT scanning sample on its tip while the OCT probe was fixed perpendicularly to the samples. By inserting three equilibrium modulation wires, micro-motion was used to move the sample perpendicular to the scanning plane of the probe (we will refer to the motion as normal motion direction (NMD)). During micro-motion, every Δs = 0.5mm of insertion depth the HLC machine (⑧ in Fig. 5) recorded a B-mode OCT scan image together with the robot configuration joint values. At the same micro-motion tip tracking was used based on the method presented in [8] to record the motion profile of the EE and the CR joint values. At the end of each experiment, MATLAB image segmentation using Canny filter and/or Hough circles was used to construct the 3D images.

Three samples were used in three experiments to test feasibility of our method on different materials and shapes. First, a metric brass screw (0.8mm diameter, 0.2mm pitch) was scanned, Fig. 6.A. A profile image of the screw was taken to compare to the known screw pitch geometry (Fig. 6A ①-②). Micro-motion was initiated to generate a 3D scan which was reconstructed in Fig. 6A ③-⑤. The pitch measurement, based on the profile image (Fig 6A ①-②) was 209.85μm while based on the 3D scan reconstruction it was 210.75μm based on analysis of Figure 6A ④. This designates an average error of 0.5%. The micro motion displacement shown in Fig. 6A ③-⑤ corresponded with 92μm motion in NMD and 89 scans. Therefore, the NMD resolution was 1.03μm.

Fig. 6:

(A)Screw pitch reconstruction: ①-OCT Image ②-Screw sample image (OCT camera view) ③-⑤ Different views of the reconstructed profile (B) Multilayer cellophane tape reconstruction ①-OCT Image ②-Side view of the multilayer cellophane tape with the double-sided tape added on top ③-Multilayer tape Sample image (OCT camera view) ④-⑥-Different views of the reconstructed profile (C) Micro Channel reconstruction ①-OCT Image ②-NiTi wire used to create the channel still in the agar sample ③-Channel sample image (OCT camera view) ④-⑥-different views of the reconstructed profile.

In the second experiment, we used a multilayer cellophane tape with an additional layer of double-sided tape on top, Fig. 6.B2–3. The double-sided tape was added to simulate a thicker layer at the top of the sample. This experiment was motivated by a future application of 3D retinal layer reconstruction. Results of the 3D reconstruction are shown in Figure 6B ④-⑥. The thickness of the layers measured based on a single OCT image (Fig. 6B ①) had an average of 71.7μm while the thickness measured from the reconstructed model had an average of 67.4μm. The average error is circa 6%. According to the tape supplier, the tape layer has an average thickness of 70μm, including the glue layer.

The last experiment aimed to demonstrate reconstruction of a capillary blood vessel, Fig. 6C. The channel was created by placing a NiTi wire, 220μm in radius, in Agar. After solidification, the wire was removed, and the sample scanned in three experiment sets for (θ_L = 30°, 45°, 60°). The results over the 3 experiment sets are shown in Tab II. The average error between the areas measured using the B-mode OCT images, as in Fig. 6C ①, and the cross-sectional area measured using the 3D reconstructed model is 5%-10%.

TABLE II:

Agar channel 3D OCT reconstruction results.

θ°	Scan Length μm	Max/Min d μm	STD(d) μm	Segmented / Reconstructed Area mm2	% Area error
30	99.6	3.8/0.6	1.0	0.13/0.12	9
45	73.8	2.9/0.3	0.7	0.13/0.12	10
60	55.8	1.8/0.3	0.4	0.15/0.14	5

B. 3D reconstruction of an optic nerve blood vessel

This section shows the experimental result for a 3D OCT reconstruction of an organic tissue. A freshly harvested retinal optic nerve in a cadaveric swine eye was used as a scanning sample as shown in Fig. 7A.3. Figure 7A.1 shows the first OCT scanned image before that the micro-motion is enabled while Fig. 7A.2 shows the last scanned image after the micro-motion scanning is completed. It is possible to notice how a vessel branch in Fig. 7A.1 bifurcates by the end of the scanning. The 3D reconstruction, Fig. 7A.4–5, shows the corresponding anatomy in Fig. 7A.1 and Fig. 7A.2 while the branching of the scanned vessel is clearly visible.

Fig. 7:

(A) 3D reconstruction of a branching in an optic nerve of a swine cadaver retina. ①-②-First and last OCT scanned images. ③-Microscope view of the sample showing the B-mode scan direction (red) and micro-motion scan direction (blue). ④-⑤-Two views of the 3D OCT reconstruction model showing the corresponding anatomy in ① and ② (B) 3D OCT reconstruction using a custom made OCT B-mode probe integrated within the CR. ①-Mock up vessel generated in Agar Sample ②-Vessel OCT image ③-④-Volumetric reconstruction of the scanned sample

C. Volumetric OCT using a miniature OCT probe

After testing the robustness of the 3D OCT reconstruction using a commercial OCT probe, we established the feasibility of using the custom-made miniature B-mode OCT probe. As shown in Fig. 5, the OCT probe ④ was embedded within the CR segment. An agar sample (Fig. 7B ①) was prepared with a 0.66mm tube embedded in it. Once the agar was solidified, the tube was removed, a channel remained and was used as an imaging target for the probe of a mock up vessel. Using equilibrium modulation, the robot moved the probe perpendicular to the scan direction of the custom miniature OCT probe. Results of the 3D reconstruction are shown in Fig. 7B ③-④. The average area measured using three B-mode OCT images as in Fig. 7B ② is 0.32892mm² while the area calculated using the 3D reconstruction is 0.31341mm² which represents an average deviation of 10%.

VII. OCT-GUIDED VISUAL SERVOING

This section presents the experimental validation of our proposed mixed-feedback control that uses OCT image feedback and the CR joint-space. The same setup in Fig. 5 and in Multimedia Extension 3 was used for these experiments. Figure 8.E depicts the control loop block diagram and the data workflow between the subsystems involved in theses experiments. We used the commercial OCT probe for these experiments. The aim of the experiment was to mimic an automated blood vessel injection by driving a NiTi needle (0.4mm in diameter), which was attached to the CR segment, into a mockup vessel (0.16mm² cross sectional area) within an agar sample carried by the planar robot as shown in Fig. 8A. OCT-guided visual control was achieved by using a custom computer code for OCT image segmentation. An online (interactive rate ≈ 30Hz) image segmentation was achieved using a cross-correlation template matching with manual initialization. At the beginning of the experiment, the user defined the two templates for the two tracking targets (vessel and needle tip) by mouse selection as shown in Fig. 8.B by two red rectangles. The template matching uses online segmentation and tracking of the mockup vessel and the NiTi needle as viewed in the OCT system.The frames definition used for the experiment, shown in Fig. 8B, follows the standard image frame definition in the MATLAB environment which place the origin of the picture frame in the top-left comer of the image with the positive Y-axis pointing downward and the positive X-axis pointing to the right (frame $\left\{{\widehat{X}}_c,{\widehat{Y}}_c\right\}$) shown by yellow dashed-lines in Fig. 8B. The robot frames follows the definition used for the continuum segment EE which place the origin of the frame origin at the center of the CS end disk with the Y-axis normal to it. Moreover, as shown in Fig. 8B, the red frame $\left\{{\widehat{X}}_n,{\widehat{Y}}_n\right\}$ aligns with the longitudinal axis of the needle that is concentrically mounted in the primary CS backbone. The relative angle between those two frames is measured beforehand using Matlab.

Download video file ^{(8.7MB, mp4)}

Fig. 8:

A) Microscope view of the needle tip and the agar vessel mockup. B) OCT scan image and visual servoing frames definition, C) Gamma angle, D) Components of e_p in {${\widehat{\text{x}}}_n$, ${\widehat{\text{y}}}_n$} frame as shown in (B). Solid line referring to the left y-axis is e_px in [mm] and dashed line referring to right y-axis is e_py in [mm], E) Visual servoing control block diagram.

A. Experimental Validation of OCT-Guided Visual Servoing

The experiment procedure requires user input for choosing two tracking target templates via mouse selection on a Matlab2016a window that acquires the OCT image from a commercial OCT control software. The OCT image in Fig. 8B shows the result of the tracking method (two red squares) and the frame definitions. The picture frame is identified by $\left\{{\widehat{\text{x}}}_c,{\widehat{\text{y}}}_c\right\}$ while $\left\{{\widehat{\text{x}}}_n,{\widehat{\text{y}}}_n\right\}$ identifies the needle tip frame. Since the OCT image was starting to distort when the needle is piercing the agar sample, the origin of the needle frame was chosen with a specified offset along the needle axis and is accounted for in the tracking code. Once the two target points were chosen, the control code used micro-motion to close the error in position along the ${\widehat{\text{x}}}_n$ direction and consequently along ${\widehat{\text{y}}}_n$ direction using macro-motion. To avoid laceration of the mockup tissue caused by the sideways motion of the needle, the HLC first used the micro-motion to converge the needle laterally and then used macro-motion for needle insertion.

The tracking results of the two red square centers are shown in Fig. 8C–D. The vector e_p indicates the the position of the target vessel relative to the current needle tip position. Figure 8C presents the tracking result of the relative angle α between e_p and ${\widehat{\text{x}}}_n$. Figure 8D presents the tracking results of the components of e_p along ${\widehat{\text{x}}}_n$ and ${\widehat{\text{y}}}_n$. The blue shaded area indicates the portion of the process where only the micro-motion is enabled. We used two thresholds for position convergence. When using macro-motion we used ϵ_py = 300μm and when using micro-motion we used ϵ_px = 5μm as the corresponding convergence threshold radii in the depth and lateral directions. The final position error between the two centers calculated in the needle frame $\left\{{\widehat{\text{x}}}_n,{\widehat{\text{y}}}_n\right\}$ is 0.0028mm along ${\widehat{\text{x}}}_n$ and 0.2683mm along ${\widehat{\text{y}}}_n$. The considerable difference of the magnitude in the error directions came from the different motion profile associated with each direction. The first one is completely controlled by micro-motion using the equilibrium modulation while the piercing direction is controlled by macro-motion actuation of the linear stage of the planar robot that has a manufacturer specification of backlash of 200μm. The need for the linear stage comes from the missing third degrees of freedom of the single segment CR which would prevent the straight motion for the injection process. The data flow and control scheme is summarized in Fig. 8E.

VIII. Conclusion

This work presented a novel approach to solving the problem of multi-scale manipulation. A new approach for micro-motion using equilibrium modulation of CRs was presented. The work was motivated by the potential benefits of using CR for in-situ inspection and image-based biopsy. The source of the turning point phenomenon was explained from an energy and internal kinematic constraint standpoints, respectively. Empirical micro-motion kinematics modeling via modal interpolation maps was presented. The feasibility of 3D OCT has been verified experimentally using both commercial (external) and a miniature (internal) OCT probes.Results showed that 3D OCT volumes with slice separation of 1μm or 5μm may be obtained using accurate EE tracking or the closed-form micro motion kinematics, respectively. Also, closed-loop OCT-guided servoing for delivering a needle into channel has been successfully demonstrated. The RMS error using our modeling approach was less than 1.2 microns, which could be used for generating a lower-resolution 3D OCT scan that would still be useful since each image slice still retains its high resolution. We believe this work is a first key step for future development of new robotic assistants that can offer multiscale manipulation for surgery and for image-based biopsy. Hysteretic effects due to motions with direction are difficult to model and solutions using micron-level EE tracking are needed to overcome this challenge. Nevertheless, applications such as 3D OCT scanning can be planned and achieved while avoiding direction reversal.

Biographies

Giuseppe Del Giudice received his B.S degree in mechanical engineering from University of Rome ”La Sapienza”, Rome, Italy, in 2012. He is currently working toward his Ph.D. degree in mechanical engineering at Vanderbilt University, Nashville, TN. Since 2015 he has been working as a research assistant. His research interests include modeling and calibration of continuum robots, surgical robotics and robot control exploration.

Andrew L. Orekhov received the B.S. degree in mechanical engineering from the University of Tennessee, Knoxville, TN, USA, in 2016, and is currently working toward the Ph.D. in mechanical engineering at Vanderbilt University, Nashville, TN, USA. His current research interests include design, modeling, and control of continuum manipulators. He received the NSF Graduate Research Fellowship in 2016.

Jin-Hui Shen received his B.S., and M.S. degrees in Laser Technology and Optical En- gineering from Tianjin University, PRC, in 1984 and 1987, respectively, and the Ph.D. degree in Ultrafast Optical Phenomena from Shanghai Institute of Optics and Fine Mechanics, PRC, in 1991. Following a Postdoctoral Biomedical Engineering Fellowship performing ophthalmic laser research at the University of Miami, he joined Novatec Laser System, Inc. as an opto-electronic engineer. He then joined Vanderbilt University in 1997 as a Research Assistant Professor.

Karen M. Joos received the B.S., M.D. and Anatomy Ph.D. degrees from the University of Iowa, Iowa City, IA, in 1982, 1987, and 1990, respectively. After an ophthalmology residency at the University of Iowa, she completed a glaucoma fellowship at the Bascom Palmer Eye Institute, Miami, FL where she engaged in ophthalmic laser research. She then joined Vanderbilt University in 1994 as an Assistant Professor of Ophthalmology. She is currently Joseph and Barbara Ellis Professor of Ophthalmology, Professor of Biomedical Engineering, Chief of the Glaucoma Service, a Fellow of the Association for Research in Vision and Ophthalmology (FARVO) and American Society for Lasers in Surgery and Medicine (ASLMS), and a SPIE Senior Member.

Nabil Simaan (M’04) received his Ph.D. degree in mechanical engineering from the Technion—Israel Institute of Technology, Haifa, Israel, in 2002. During 2003, he was a Postdoctoral Research Scientist at Johns Hopkins University National Science Foundation (NSF) ERC-CISST. In 2005, he joined Columbia University, New York, NY. In 2009 he received the NSF Career award for young investigators to design new algorithms and robots for safe interaction with the anatomy. In Fall 2010 he joined Vanderbilt University. In 2020 he was named an IEEE Fellow for contributions to dexterous continuum robotics. His research interests include medical robotics, kinematics, robot modeling and control and human-robot interaction.