FBG-based Shape-Sensing to Enable Lateral Deflection Methods of Autonomous Needle Insertion

Abstract

In diagnosing and treating prostate cancer the flexible bevel tip needle insertion surgical technique is commonly used. Bevel tip needles experience asymmetric loading on the needle’s tip, inducing natural bending of the needle and enabling control mechanisms for precise placement of the needle during surgery. Several methods leverage the needles natural bending to provide autonomous control of needle insertion for accurate needle placement in an effort to reduce excess tissue damage and improve patient outcomes from needle insertion intraventions. Moreover, control methods using lateral deflection of the needle intra-operatively to steer the needle during insertion have been studied and have shown promising results. Thus, to enable these autonomous control methods, real-time, intra-operative shape-sensing feedback is pivotal for optimal performance of the needle insertion control. This work presents an extension of our proven Lie-group theoretic shape-sensing model to handle lateral deflection of the needle during needle insertion and validate this extension with robotic needle insertions in phantom tissue using stereo vision as a ground truth. Furthermore, the system configuration for real-time shape-sensing is implemented using ROS 2, demonstrating average feedback frequency of 15 ± 8 Hz. Average needle shape errors realized from this extension under 1 mm, validating the shape-sensing models’ extension.

I. INTRODUCTION

In the treatment and diagnosis of prostate cancer, flexible bevel tip needle insertion is a common minimally-invasive surgical technique for procedures like prostate biopsy, focal laser ablation and cryoablation [1]–[4]. Due to the asymmetric geometry of bevel tip needles, they provide needle steering capabilities enabling methods for accurate placement of the needle into the patient. However, due to natural random occurrences, deviations of the flexible needle’s intended trajectory occur during insertion resulting in difficulties while placing the needle. To correct insertion trajectories, physicians typically retract and reinsert the needle, causing excess tissue damage [5]. Furthermore, as the needle is inserted into the patient, loss of visualization of the needle occurs, increasing the uncertainty of the needle placement requiring for guidance solutions to provide feedback to the physician of needle placement intra-operatively [6]. These methods can include magnetic resonance imaging (MRI) to track the needle’s trajectory through identifying the needle’s location relative to the patient’s anatomy [7]–[9]. MRI produces a high resolution of the image, but is slow to produce enough feedback and results in challenging identification of the needle in the MRI image. To mitigate the shortcomings using MRI needle placement feedback, researchers have studied methods using shape-sensing devices for real-time needle shape feedback. One of these approaches embeds fiber optic sensors inscribed with fiber-Bragg grating (FBG) sensors in conjunction with mathematical models to reconstruct the 3D needle shape intra-operatively [10]–[16]. Since these FBG sensors are optical sensors, their medium is constructed of glass, allowing for these sensors to be used in conjunction with MRI scanning, providing richer needle placement information through balancing the rich information provided by MRI and the consistent feedback FBG-based shape-sensing. A proven method includes using discrete FBGs embedded inside a flexible bevel tip needle and a Lie-group theoretic model to enable real-time FBG-based shape-sensing [17]–[19]. This method balances the mechanics-based rich understanding of the needle-tissue interaction with the flexibility of usage with sensor-based abstraction, providing accurate shape-sensing in varying insertion situations. This shape-sensing information not only provides intra-operative feedback for the physician, but also better supports control mechanisms for autonomous needle insertion.

Autonomous needle insertion attempts to mitigate patient discomfort by assisting the physician controling the placement of the needle more precisely through a modelled understanding of the needle insertion dynamics. These methods typically require feedback of the current placement of the needle in order to update their control scheme and steer the needle towards a desired location. Control schemes include active controlling, where the needle is actively deformed in the patient during insertion, or passive controlling where the needle is oriented in the body for needle steering [20]. While active control schemes are proven to be effective, they require specialized needles and devices to operate [14], [15], [21]–[27]. Passive control methods for needle insertion range from axial rotation-based control of an asymmetrical tip needle to using water jets to control the insertion trajectory [13], [28]–[35]. In [36], their control mechanism includes a data-adaptive method to estimate a Jacobian relating lateral needle base movement to the tip deflection induced upon insertion for the application of transperineal prostate needle insertion surgical interventions. Afterwards, the control of the needle utilizes the estimated Jacobian for precise placement of the needle, validated in ex vivo tissue. For needle shape feedback, the needle tip is tracked with an electromagnetic (EM) sensor. The limitation of using an EM sensor is that the sensor is not MRI-compatible and does not provide entire shape feedback. Thus, this work is motivated to fill the need of an MRI-compatible, real-time shape feedback to autonomous needle insertion with the method referenced in [36].

The contributions of this work include the presentation and validation of an extension of the Lie-group theoretic model for FBG-based needle shape-sensing presented in [18] to accommodate lateral deflection of the needle’s base during needle insertion as well as the real-time system configuration to modularly use the FBG shape-sensing needles to provide shape feedback. Experimental validation of this model’s extension was performed with robotic insertion of a flexible bevel tip needle embedded with a multicore fiber FBG (MCF) sensor into phantom tissue under stereo visualization. The structure of this paper is as follows: Sec. II discusses the shape-sensing model’s extension validated in this work and the shape-sensing system overview written in ROS 2, Sec. III presents the experimental methods used to validate the presented model extension, Sec. IV provides the results of the experiments performed and discussion of errors, and Sec. V summarizes this work and discusses future directions of study.

II. METHODS

A. FBG Sensor Model

FBGs are optical sensors with the ability to measure strain induced in the fiber optic cable through a shift in the Bragg wavelength, $\lambda$. These sensors are sensitive to strain and temperature changes and the change in the Bragg wavelength can be approximated by

$$\Delta \lambda =k_{ϵ}ϵ+k_T\Delta T,$$ (1)

where $k_{ϵ}$ and $k_T$ are the coefficients of the FBG associated with strain and temperature sensitivity, respectively; $ϵ$ is the strain induced in the fiber at the sensor; and $\Delta T$ is the change in temperature in the fiber at the sensor. Temperature compensation methods for the FBG sensor are presented in [37] in order to remove the effect of change in temperature and isolate strain measured in the fibers. Using a combination of these fibers and rod theory, a linear correlation is observed between the 2D curvature cross-sectional curvature induced in the fiber, $\mathit{\omega }$ with the temperature-compensated changes in Bragg wavelength at an FBG sensor array denoted by,

$$\mathit{\omega }={\left({\kappa }_{xz}{\kappa }_{yz}\right)}^T=C{\left(\overline{\Delta {\lambda }_1}\overline{\Delta {\lambda }_2}\overline{\Delta {\lambda }_3}\right)}^T,$$ (2)

where ${\kappa }_{xz}$ and ${\kappa }_{yz}$ are the curvatures measured in the local $xz$- and $yz$-planes, respectively, along the needle’s central axis, as illustrated in Fig. 1; $\overline{\Delta {\lambda }_i}$ for $i=1,2,3$ are the temperature compensated Bragg wavelength shifts; and $C\in {\mathbb{R}}^{2\times 3}$ is the calibration matrix, determined by our FBG-sensorized needle calibration process referenced in [37].

Fig. 1:

Frame configuration of the insertion point modeling with an air gap between needle base and tissue. Here, ${\mathbf{p}}_{\mathit{\text{ins}}}$ is the location of the insertion point relative to the needle’s base (origin) and ${\mathbf{p}}_{\mathit{\text{base}}}$ is the relative translation of the needle’s base after a lateral deflection.

B. Shape-Sensing Model

The shape-sensing model used in this work is derived from [18] using a sensor-based, Lie-group theoretic model to determine the needle’s 3D shape. Using local deflection modeling, the local curvature, $\mathit{\omega }\in {\mathbb{R}}^3$ induced at an arclength $s$ along the needle is modelled by

$$\mathit{\omega }\left(s\right)={\left(R^T(s)\frac{\text{d}R(s)}{\text{d}s}\right)}^{\vee }$$ (3)

where $R\left(s\right):{\mathbb{R}}_{\ge 0}\to SO\left(3\right)$ is the body-fixed frame orientation along the needle, and $\vee$ defines the mapping from the associated Lie algebra of $SO\left(3\right)$, $so\left(3\right)$, to ${\mathbb{R}}^3$ [38]. The local deformation along the needle can be characterized by the Euler-Poincaré equation

$$\frac{d}{ds}\left[B\left(\mathit{\omega }-{\mathit{\omega }}_0\right)\right]+\mathit{\omega }\times B\left(\mathit{\omega }-{\mathit{\omega }}_0\right)=\mathbf{0}$$ (4)

where $B$ is the stiffness matrix of the needle and ${\mathit{\omega }}_0$ is the intrinsic curvature of the needle to model the natural bending effect from a bevel tip needle, solved in conjunction with eq. (3). Using a sensor-based optimization, referenced in [18], the parameters of the intrinsic curvature used in eq. (4) are optimized using the curvature measurements of the FBGs from eq. (2). Finally, the needle shape $\mathbf{r}\left(s\right):\mathbb{R}\to {\mathbb{R}}^3$, can be determined by integrating eq. (2) along the needle length as such,

$$\mathbf{r}\left(s\right)={\int }_0^s R\left(\sigma \right){\mathbf{e}}_{3}d\sigma$$ (5)

where ${\mathbf{e}}_3={\left[001\right]}^T$.

C. Lateral Deflection Model

To model the needle’s natural bending experienced from lateral deflection of the needle base outside of the tissue, labeled as the air gap shown in Fig. 1l the needle is treated as a beam constrained at the both end points. Given that the insertion point of the needle is a known location, the insertion point of the needle is defined, relative to the needle’s origin, the needle’s constrained base, as ${\mathbf{p}}_{\mathit{\text{ins}}}={\left(p_{\mathit{\text{ins}},x},p_{\mathit{\text{ins}},y},p_{\mathit{\text{ins}},z}\right)}^T$, illustrated in Fig. 1. Given that the needle’s base will move in the $xy$-plane, ${\mathbf{p}}_{\mathit{\text{base}}}$, this shift will create an inverted shift of the insertion point relative to the needle’s base described by $p_{\mathit{\text{ins}},x}=-p_{\mathit{\text{base}},x}$ and $p_{\mathit{\text{ins}},y}=-p_{\mathit{\text{base}},y}.p_{\mathit{\text{ins}},z}$ will be determined from the measured distance from the needle’s constrained base to the tissue’s closest boundary. Using the assumption of a constant curvature beam model with end points, fixed in position and orientation, experiencing relatively small local deformation, the needle shape can be described by a quadratic equation. Thus, the shape of the needle is defined in the air gap parameterized by the $z$-coordinate as,

$$\mathbf{p}\left(z\right)=\left(\begin{array}{c}{a}_x{z}^2\\ a_y{z}^2\\ z\end{array}\right),$$ (6)

where $a_x$ and $a_y$ are the quadratic coefficients to be determined by a polynomial fit with the insertion point $\mathbf{p}\left(z\right)={\mathbf{p}}_{\mathit{\text{ins}}}$. To determine the body frame of the needle at the insertion point, the Frenet-Serret frame formulation is leveraged and differentiate eq. (6) to calculate the orientation matrix from the tangent, normal and binormal vectors [39]. This pose of the needle is used as the initial condition when integrating eqs. (3) and (5) to approximate the effect of the needle’s lateral deflection upon the inserted portion of the needle and maintain differentiability of the entire needle shape.

D. ROS 2 System Overview

There are four main components that comprise of the shape-sensing system: the robot interface, the interrogator interface, the curvature model and the shape-sensing model. The robot interface controls the robot’s movements during needle insertion and publishes the current pose of the needle base. The interrogator publishes the FBG wavelength peaks of the FBG-sensorized needle over all of the MCF channels used. Using the calibration matrices for the FBG-sensorized needle, the curvature estimator node samples 200 wavelengths from the published topic and computes the curvature over the average signal response over the 200 sample set size. The shape-sensing node subscribes to the needle base pose to determine the needle’s insertion depth and base location relative to the needle’s insertion point and the curvature for shape-sensing computation. The insertion point is provided to the needle and the needle shape-sensing node computes the current needle shape from the provided information to the module using the shape-sensing model and publishes the current shape. The publisher runs in real-time with an average publishing frequency of 15 ± 8 Hz, with a minimum publishing frequency of 2.8 Hz on an AMD Ryzen 9 16-core CPU. Finally, there is a stereo camera system for the needle insertion to be visualized with the stereo camera system. ROS bags were recorded for offline ground truth generation from the recorded needle shapes during insertion using stereo visualization methods referenced in Sec. III-B.

III. EXPERIMENT

A. Needle Configuration

An MCF-sensorized needle was built using an MCF sensor (125-micron cladding diameter, Fujikura America, Sunnyvale, CA) coaxially-embedded into an 20G nitinol tube placed into an 18G bevel tip needle. The MCF sensor was embedded into this needle by gluing the base of the sensor to the base of the needle. The additional inner stylet was included in to increase the strain transfer from the needle’s outer stylet and coaxially-mounted MCF sensor. The MCF sensor was comprised of seven fiber optic cores embedded into a single fiber optic cable in a hexagonal pattern and a central core, as seen in Fig. 3. Based on our previous work referenced in [37], the even-labeled channels were used and the central core for FBG-based shape-sensing as these channels were the optimal configuration for shape-sensing among the tested configurations.

Fig. 3:

Sensor configuration of the 18G FBG-sensorized needle used for experimental needle insertion. (a) The placement of FBGs distributed along the needle. (b) The cross-section of the needle containing an outer stylet, an additional inner stylet and the MCF sensor. (c) The cross-section of the seven channel MCF sensor with a central core and even- and odd-labeled cores.

B. Experimental Setup

Needle insertions were performed in single-layer homogeneous phantom tissue in a similar platform to [18], as seen in Fig. 4. FBG wavelengths were interrogated with an optical interrogator (HYPERION si155, Luna Inc., Virginia, United States) to collect 200 FBG wavelength samples per insertion depth per trial. An isotropic gel phantom was constructed from plastic of various stiffness (Plastic from M-F Manufacturing Company, TX, US) to simulate hard tissue, using a 4:1 liquid plastic to plastic hardener ratio. The first 45 mm of the tissue were removed in order to create an air gap between the needle guide and the needle’s insertion point, illustrated in Fig. 4b. The air gap length was used to evaluate the model’s performance under a magnified effect of lateral deflection on the needle shape with a sizable air gap and maintain the small beam deformation assumption. Needle insertion was performed on a ROS 2 platform with the robotic interface in [18], as shown in Fig. 2. Two insertion experiments were performed quasi-statically with an initial straight 20 mm insertion depth, followed by a 5 mm deflection in the direction of the bevel or opposite the direction of the needle’s bevel direction as shown in Fig. 4b, followed by two 30 mm incremental insertions, finishing at 80 mm deep into the tissue. The 5 mm deflection is to keep within the safety range listed in [36] and to maintain the small beam deformation assumption. In the direction and opposite the direction trials were split evenly among total 10 insertion trials, 5 insertion trials each. The insertion point in the needle’s coordinate system was $(\mathrm{0,0},45{)}^T$ with the needle’s exit point from the needle guide to be the origin, illustrated in Fig. 1, published in the ROS interface to the needle shape-sensing node. Stereo vision was used for 3D needle shape ground truth as listed in [18] imaged with two identical cameras (Flea2 FL2–0852C, Point Grey Research Inc., Richmond, BC, Canada) found to have a 0.160 ± 0.055 mm shape reconstruction error.

Fig. 4:

Needle insertion experimental setup for robotic insertions into phantom tissue. (a) Experimental setup including the needle insertion robot, stereo cameras, needle, tissue and ROS system interface. (b) The phantom tissue with a 45 mm air gap between the end of the needle guide to the entry point into phantom tissue with annotations of “in bevel” and “out of bevel” directions for lateral deflection.

Fig. 2:

ROS 2 high-level diagram of the needle shape-sensing system. ROS nodes are indicated in black boxes, ROS topics are shown in blue ellipses, the connection between topic and node publisher is indicated with a blue arrow from node to topic, and topic subscriptions are indicated with red arrows from topic to node.

IV. RESULTS

Four metrics were used to evaluate the accuracy of our sensed needle shape, $\mathbf{r}$, to the ground truth needle shape, ${\mathbf{r}}_{gt}$, discretized by their arclength $s_i,i=1,\dots ,N$. These metrics are listed below.

Root-Mean Square Error (RMSE):

the overall RMS error of the needle shape,

$$\mathit{\text{RMSE}}=\sqrt{\frac{1}{N}\sum _{i=1}^N{‖r_{gt}\left(s_i\right)-r\left(s_i\right)‖}^2}.$$ (7)

In-Plane Error (IPE):

the error measured in the natural bending plane of the needle,

$$IPE=\frac{1}{N}\sum _{i=1}^N‖\left(011\right)\cdot \left(r_{gt}\left(s_i\right)-r\left(s_i\right)\right)‖.$$ (8)

Out-of-Plane Error (OPE):

the error measured in the plane orthogonal to the natural bending plane of the needle,

$$OPE=\frac{1}{N}\sum _{i=1}^N‖\left(101\right)\cdot \left(r_{gt}\left(s_i\right)-r\left(s_i\right)\right)‖.$$ (9)

Max Error (MAX):

the maximum error measured along the needle,

$$MAX=\underset{i}{\text{max}}\left(‖r_{gt}\left(s_i\right)-r\left(s_i\right)‖\right).$$ (10)

Note that · above denotes matrix multiplication.

Fig. 5 illustrates the distribution of errors achieved in phantom tissue from the 10 total insertions, 5 with in bevel direction deflection and 5 out of bevel direction deflection. The average RMSE distribution demonstrates that the average shape deviation error are contained within 1 mm for all evaluated insertion depths and insertion configurations. The maximum error deflections achieved by all of these methods are mostly contained within 1 mm, however for the 80 mm insertion depth with in bevel direction deflection, the variation of errors is large and reaches a maximum of about 1.5 mm. The distribution of errors over the in-plane and out-of-plane errors are nearly identical, indicating that there is no imbalance of performance over the sensors inducing any significant error in the shape-sensing in either direction, as well as indicate a good correspondence with the ground truth needle shape.

Fig. 5:

Shape-sensing error statistics for MCF needle insertion into tissue with an air-gap over various deflection directions and full needle insertion depths. Insertions trials where needle deflection was performed towards the natural bending direction are denoted as “in bevel direction” and away from the natural bending direction as “out of bevel direction”.

The in bevel direction deflection results in a straighter needle shape, since the angulation of the needle counters the natural bending occurring during the initial insertion. Since the MCF’s FBG sensors are coaxially-mounted inside a single fiber-optic cable with the needle, the sensors’ sensitivity at small curvatures is quite low, further justified by results from our previous work. Given that the needle shape is straighter, the variation of error is quite large for the in bevel direction deflection due to the lack of sensitivity for straight needle configurations. For the out of bevel direction deflection, the variation of errors decreases significantly, since larger curvatures are induced upon the needle, in a range within the sensors’ sensitivity range. Thus, the results presented are limited to the sensitivity of the experimental device used for insertion. Error variation could be mitigated using a different configuration of the MCF sensor to improve strain transfer from needle to sensor or a single-core fiber setup with circumferentially-mounted FBG sensors.

With the understanding of the used sensors’ sensitivity ranges, it was observed that the extension of our method provides consistent and accurate results comparable with previous shape-sensing results achieved within our previous works without lateral deflection [17], [19]. Thus, the extension of this method is considered to be validated as shape-sensing extension of our method to enable shape-sensing with lateral deflection of the needle for needle control algorithms.

V. CONCLUSIONS

This work presented and validated an extension of our Lie-group theoretic shape-sensing model to handle lateral deflection of the needle enabling autonomous control of needle insertion. Furthermore, a system overview in ROS 2 providing real-time shape-sensing feedback was demonstrated for autonomous needle insertion techniques. Robotic needle insertions were performed in phantom tissue under stereo vision with lateral deflections in the natural bending direction and opposite the natural bending direction of a flexible bevel tip needle embedded with an MCF sensor to validate our model’s extension. Average shape-sensing errors with the extended model realized were under 1 mm, thus justifying the model’s extension. Limitations of this work include the model’s simple approximation of the lateral deflection under small deformation, the MCF sensors’ lack of sensitivity for straight needle configurations and experimental validation in real tissue. Future work includes further development of the extension model to better approximate the needle’s lateral deflection effect upon the needle shape, needle insertion experimental validation in ex vivo tissue for longer insertion depths and evaluation of autonomous needle insertion control with the provided shape-sensing feedback.