Closed-Loop Active Compensation for Needle Deflection and Target Shift During Cooperatively Controlled Robotic Needle Insertion

Abstract

Intra-operative imaging is sometimes available to assist needle biopsy, but typical open-loop insertion does not account for unmodeled needle deflection or target shift. Closed-loop image-guided compensation for deviation from an initial straight-line trajectory through rotational control of an asymmetric tip can reduce targeting error. Incorporating robotic closed-loop control often reduces physician interaction with the patient, but by pairing closed-loop trajectory compensation with hands-on cooperatively controlled insertion, a physician’s control of the procedure can be maintained while incorporating benefits of robotic accuracy. A series of needle insertions were performed with a typical 18G needle using closed-loop active compensation under both fully autonomous and user-directed cooperative control. We demonstrated equivalent improvement in accuracy while maintaining physician-in-the-loop control with no statistically significant difference (p > 0.05) in the targeting accuracy between any pair of autonomous or individual cooperative sets, with average targeting accuracy of 3.56 mm_rms. With cooperatively controlled insertions and target shift between 1 mm – 10 mm introduced upon needle contact, the system was able to effectively compensate up to the point where error approached a maximum curvature governed by bending mechanics. These results show closed-loop active compensation can enhance targeting accuracy, and that the improvement can be maintained under user directed cooperative insertion.

3). Introduction

Positive clinical outcomes for deep percutaneous needle based procedures such as targeted biopsy are dependent on the accuracy of needle tip placement. If a target is not reached with sufficient accuracy a physician often retracts the needle without taking a sample and must restart with a new attempt, lengthening procedure time, increasing cost and causing unnecessary discomfort to the patient. Although intra-operative imaging is sometimes available to assist in this procedure; open-loop insertion trajectories, wherein image-based feedback is not used to adjust the trajectory during the insertion, do not account for unmodeled needle deflection or target shift due to tissue deformation. One approach to compensate for unmodeled error is to robotically control the rotation of a bevel-tipped needle to induce corrective deflection towards the desired trajectory. Paired with real-time needle tip tracking, closed-loop image-guided active compensation can be accomplished. A system controlling robotic needle rotation for compensation also requires accurate control needle insertion position; however, incorporating a robot to actuate needle insertion typically either removes a physician from direct patient interaction through autonomous insertion, or distances them from patient interaction through teleoperation²⁰. Instead, a cooperatively controlled robotic system would enable them to be at the procedure site, maintaining ultimate control while adding robotic accuracy to the final placement of the needle tip. In the presented work, this takes the form of a hands-on cooperative needle insertion, in which the surgical workspace is shared between the physician and the needle placement robot. The physician applies an input force directly onto the robot controlling the needle insertion velocity, while active compensation of the direction through rotational bevel positioning is performed autonomously based on closed-loop image feedback.

Motivation for this work came as our research group performed clinical trials of MRI-guided robot-assisted prostate biopsy using the robot seen in Eslami et al⁷. The system employed a robot for target alignment but was still dependent on manual needle insertion along a robotically aligned axis. As a result, even though initial alignment was performed robotically, the system suffered from loss of targeting accuracy due to unmodeled needle deflection and target shift. Targeting accuracy across all clinical trials was 6.6 ± 5.1 mm²⁶. The overall goal of our research is to implement closed-loop active compensation during cooperative controlled needle insertions under real-time MR imaging, and we have previously reported aspects of this including a needle driver configured for cooperative control and suitable for the MR environment³¹, with needle localization in real-time MR images shown in Patel et al.¹⁸

The maximum needle tip deflection and consequently the degree of compensation attainable is limited by the mechanical properties of the needle. Related work implementing image-feedback for rotational bevel tip positioning has been primarily based on steering thin flexible needles. Modeling of flexible needles has been widely implemented using the kinematic bicycle model³² as well as a joint kinematic and dynamic system²⁴. These were expanded using imaging for feedback in Abayazid et al.¹ for evaluation of two models for flexible needle steering, one kinematics based and one based on needle-tissue interaction forces, both validated with double bend tests. A duty cycled approach for needle steering was introduced by Engh et al.⁶, and expanded in Minhas et al.¹⁴, with Vrooijink et al.³⁰ implementing this for 3D targeting using ultrasound imaging. The duty-cycled approach, although implemented widely with positive results, requires insertion and rotation to be very coupled so as to alternate between pure insertion and rotation during insertion, which would not be feasible in a cooperative approach. Furthermore, a handheld device for seed placement in brachytherapy was introduced by Rossa et al.²¹, rotating the needle autonomously during manual needle insertion. This system can be comparable to the work presented here in that rotation is autonomous while insertion is user directed, however; a cooperative insertion approach adds robotic accuracy to the insertion axis as well.

Robotic insertion can improve accuracy of needle tip placement, but in a clinical scenario a fully autonomous insertion would isolate the patient from the physician and is likely to receive increased regulatory scrutiny²⁰. Several teleoperated systems have been developed to keep the physician in the loop during robotic needle insertion, often employing haptic feedback to restore some of the tactile information typically used by the physician for anatomical localization of the needle tip. For instance, Tse et al.²⁷ introduced a haptic needle device with a feedforward controller to increase response of the slave, while a completely MRI-safe robot was developed by Stoianovici et al.²³, employing manual insertion to a depth stop set robotically using a PneuStep pneumatic stepper motor²⁹. Teleoperated systems offer the benefits of robotic precision for insertion position with the ability to integrate compensation via needle rotation, such as in the system presented by Pacchierotti et al.¹⁷ In this system the user maintains control of the needle insertion via teleoperation, with kinesthetic and vibratory feedback providing guidance to steer the slave robot to the target. This system provides full control to a physician for needle placement, but still distances them from the patient. Cooperative control on the other hand is defined as the direct robotic guidance of a tool that is also held and controlled by the user²⁸. A cooperatively controlled system provides a framework to incorporate the benefits of teleoperation without distancing the physician from the procedure site. Some notable cooperative devices include the PADyC²² used for joint replacement, the ACROBOT¹¹ used in bone surgery and the Steady Hand Robot³, eliminating hand tremor for precision eye surgery. A biopsy robot described to work synergistically with the physician was described by Megali et. al.¹³. This system was considered synergistic in that the procedure had both robotic and manual input components. The alignment of the biopsy needle was done robotically followed by a manual needle insertion, much like the clinical version of our research group's biopsy robot previously described. The system described herein is different in that the cooperative aspect is hands-on synergy between robot and physician during the insertion task.

Finally, in much of the literature present, the specialized thin flexible needles used are not approved by a regulatory agency for biopsy; in fact, they are mostly solid core and too thin to be used in a clinical scenario where a tissue sample must be retrieved. Research contributions on needle steering tend to be focused on obstacle avoidance or following trajectories of a predefined shape to a target location, while teleoperated systems aim for stable control of needle insertion position with haptic feedback. Here the work is focused on clinical translation, with active compensation for deviations from an initial straight-line trajectory using a typical biopsy needle (e.g. that of a relatively stiff 18G biopsy gun). More so, to perform this compensation during cooperatively controlled needle insertions to maintain a biopsy procedure directed by the physician at the procedure site, while providing continuous haptic feedback to restore tactile information typically surrendered in favor of robotic accuracy. The primary contributions of this work are development of a method of closed-loop active compensation for unmodeled needle deflection and target shift under cooperatively controlled needle insertion, and experimental validation of this on an existing robotic needle placement system with results showing the accuracy improvements of autonomous needle placement can be maintained in user directed cooperative insertion.

4). Methods and Materials

A). Workflow

The workflow for retrieving a targeted biopsy core using continuous image-guided feedback is illustrated in Figure 1a. The workflow begins with registration, using a marker or fiducial to transform image-based feature localizations into the robot coordinate frame. After a target feature is defined, the robot positions the needle at the skin surface and aligns the insertion axis towards the target for a straight line insertion trajectory. Insertion begins under cooperative control and continues until the target depth is reached or an error condition occurs, such as the needle sufficiently deflecting into a configuration where the target would not be reachable with suitable accuracy. The tip and target features are localized in imaging during insertion and homogeneous transforms describing their location and orientation are passed to the robot controller. The localization information along with robot kinematics and force sensor readings are used to determine the cooperative insertion velocity as well as the bevel rotation required to compensate for any deviations from the initial straight-line trajectory. Figure 1b illustrates the needle placement manipulator in position to perform a cooperative needle insertion, the imaging stage with phantom, and how an input force is applied to the robot to create a cooperative insertion velocity.

Figure 1:

A) The workflow of biopsy retrieval using a robotic system configured for closed-loop image-guided active compensation through rotational bevel tip position during cooperatively controlled needle insertion. B) The experimental setup, showing a cooperative input to the needle placement manipulator. Cooperative insertions were performed by applying an input force to the robot to insert an 18G clinical biopsy gun.

B). Needle Placement Manipulator

A 6-dof robotic needle placement system designed for in-bore MR image-guided prostate biopsy, similar to that in Wartenberg et al.³¹, was used. It is comprised of the 4-dof alignment base used in the clinical trials previously mentioned, with a 2-dof needle driver configured for cooperative control mounted in place of the manual needle guide. The system collects both user directed input and axial needle forces, with the insertion velocity based on a relationship between the two. All forces were collected using aluminum load cell sensors MLP-10 from Transducer Techniques (Temecula, CA, USA). The system is capable of inserting and rotating a biopsy gun typically used in clinical procedures, the Full-Auto Bx Gun 18G 175 mm from Invivo (Best, Netherlands). Actuation for the base, insertion, and rotation are performed by non-magnetic piezoelectric motors, model USR-60 from the Shinsei Corporation (Tokyo, Japan), and models USR-45 and USR-30 from Fukoku Co Ltd. (Tokyo, Japan), respectively. All position feedback was achieved using optical encoders from US Digital (Vancouver, WA), model EM-1-1250. Figure 2 shows an annotated image of the robotic needle placement manipulator.

Figure 2:

Annotated view of the needle placement robot. The base was previously used in clinical trials, aligning the needle guide for manual insertion. Here, a 2-dof needle driver was mounted in place of that manual needle guide and configured for cooperative controlled needle insertions with active compensation for unmodeled errors through closed-loop image-guided rotational bevel tip positioning.

The control system for this robot is implemented on the sbRIO-9651 from National Instruments (Austin, TX, USA). This module contains an Artix-7 FPGA from Xilinx (San Jose, CA, USA), as well as a Linux Real-Time Operating System (RTOS) running on an ARM Cortex-A9 from ARM Holdings (Cambridge, England, UK). Low level communication is programmed using parallel loops in the FPGA while the cooperative algorithms are implemented in the RTOS using C/C++. The control system is housed inside a custom shielded control box which alongside the robot is suitable for the MR environment and was the same controller used in Nycz et al.¹⁵, where it was shown to perform inside the MR room without degrading image quality.

C). Cooperative Needle Insertion

Tactile forces felt during manual needle insertion are often used by physicians for mental registration of anatomical localization of the needle tip within the body. By collecting both the user input force as well as the forces along the needle, continuous feedback can be provided during cooperative needle insertion by adjusting the insertion velocity as a function of the two. The forces seen along the needle are comprised of three components: the cutting force, the stiffness on the tip, and the friction force along the length of the needle¹⁶. Separating these forces can be possible, for instance by using Fiber Bragg grating⁴, but this is not feasible in a scenario where altering an already regulatory approved clinical biopsy needle is not an option. Instead these forces are collected as one collective measurement at the proximal end of the biopsy gun, Σ F_needle, and used alongside F_input in Equation 1 to show the insertion velocity under the cooperative control scheme.

$$v_{insertion}=\{\begin{array}{cc}{v}_{max}\left(1-0.9e^{-\lambda (F_{input}-\mathrm{\Sigma }{F}_{needle})}\right),\hfill & F_{input}>\mathrm{\Sigma }{F}_{needle}\hfill \\ 0.1v_{max},\hfill & F_{input}<\mathrm{\Sigma }{F}_{needle},F_{input}>0\hfill \\ 0,\hfill & \mathit{\text{Otherwise}}\hfill \end{array}\phantom{\}}$$ (1)

The velocity is based on a decreasing exponential, with the effect of difference in forces in the normal operating range (F_input > Σ F_needle) scaled by the exponential decay constant λ. Increasing the magnitude of λ leads to an increased insertion velocity during initial insertion with higher sensitivity between F_input and Σ F_needle as their values approach each other. The goal is augmented feedback, not transparent haptic sensation. This algorithm gives the user control of insertion speed through a force input, while increasing the sensitivity of tactile feedback as the needle force approaches this input. This situation is seen at the transition between tissue boundaries when needle force spikes, and could allow sensing of boundary transitions that could not have been able to be felt by the user alone. This inherently makes the system non-transparent, instead allowing for additional augmented feedback to be implemented at event detection such as these boundary transitions or forbidden regions corresponding to specific anatomical structures to avoid. Figure 3A illustrates a series of velocity curves for cooperative insertion based on varying the value of λ, and was set to 5 in this work for all insertion trials. The non-zero value in the second condition was added empirically to avoid a discontinuity in movement if, for instance, needle force grew larger than the input force of the user, but an intent of continuous insertion was present.

Figure 3:

A) The cooperative insertion velocity vs. the difference in input and needle forces based on varying the exponential decay constant λ. As λ increases the difference in forces has less influence early in insertion, with increased sensitivity on insertion velocity when the forces approach each other. B) The control loop describing functional methodology for implementing cooperative controlled needle insertion. The two inputs are a force input and a target depth, and the two outputs are collected needle forces and needle tip position.

The cooperative insertion was implemented through admittance based closed-loop velocity control, providing a velocity output to a force input. A block diagram describing cooperative insertion is shown in Figure 3B. There are two inputs, a target depth and a user applied force input. First, Fδ is determined by subtracting the sum of needle forces from the applied input force. Then the current needle tip position, determined via real-time imaging, is subtracted from the target depth to determine if the target depth has been reached. If it has then a zero value is passed to the cooperative controller. If it hasn't then Fδ is passed to the cooperative controller to determine the insertion velocity based on Equation 1. An inner PID control loop with velocity feedback maintains the desired set point insertion velocity based on the cooperative controller output. An insertion velocity produces the increased insertion position that is fed back to make the supervisory control loop decision. Finally, as the needle inserts deeper into the tissue the force seen on the needle increases, and this needle force is fed back to the beginning of the control loop and subtracted from the input force.

D). Feature Localization

The proposed active compensation technique is agnostic to imaging modality, and in previous work we have demonstrated needle tracking in MRI coupled with active scan plane geometry control¹⁸. With the focus of this work being on developing active compensation during cooperatively controlled insertions, for evaluation in the lab we use a standalone software application using two orthogonal USB cameras, models C920 from Logitech (Lausanne. Switzerland); this serves as a proxy for medical imaging to provide real-time 3D coordinates of the needle tip and target within the robot workspace. Each image has a resolution of 640 x 480 pixels and two-dimensional localization of the moving needle tip is found by analyzing pairs of sequential video frames using the dense optical flow algorithm introduced in Farneback⁸. For the purposes of demonstrating the active compensation technique, the target point is either selected manually or segmented by color in the camera images, while target orientation is set to a nominally straight trajectory from the entry point, but it could represent an approach vector for angulated insertions in future work. The output of both localizations are homogeneous transforms for the position and orientation of tip and target.

E). Active Compensation

In this system the control of rotational bevel positioning is completely independent from the imaging modality used. Information of needle tip and target localization is passed one-way downstream from a standalone image-guidance software to the robot controller via the OpenIGTLink communication protocol²⁵, an open network interface for image-guided therapy. Registration is performed prior to any targeting through a marker rigidly attached to the robot and in view of the image frame, and this registration marker along with each transformation used in the following calculations for the benchtop configuration is seen below in Figure 4. When moving to the MRI, registration would be performed with a set of MR visible fiducials in a z-frame configuration rigidly mounted to the robot.

Figure 4:

The coordinate systems and frame transformations used for robot to image guidance registration. The registration marker is rigidly attached to the robot and placed in view of the cameras for registration. Once registration is performed the marker is removed.

This figure illustrates that $T_{\mathit{RobotPose}}^{{\mathit{RobotHome}}^{-1}}{T}_{\mathit{Registration}}^{\mathit{RobotHome}}{T}_{\mathit{Registration}}^{{\mathit{Image}}^{-1}}$ serves as the sequential multiplication of transformations required to register the imaging system’s coordinate frame to the current pose of the robot, $T_{\mathit{Image}}^{\mathit{RobotPose}}$. In this series of multiplications, $T_{\mathit{RobotPose}}^{{\mathit{RobotHome}}^{-1}}$ is the transformation from the current pose of the robot to the robot home coordinate frame, $T_{\mathit{Registration}}^{\mathit{RobotHome}}$ is a constant matrix describing the transformation from the robot home frame to the registration marker, while $T_{\mathit{Registration}}^{{\mathit{Image}}^{-1}}$ is also a constant matrix describing the transformation from the registration marker rigidly attached to the robot to the imaging coordinate frame.

The robot controller continuously receives the localization transforms, $T_{\mathit{Tip}}^{\mathit{Image}}$ and $T_{\mathit{Target}}^{\mathit{Image}}$, throughout insertion and pre-multiplies them by the $T_{\mathit{Image}}^{\mathit{RobotPose}}$ to determine $T_{\mathit{Tip}}^{\mathit{RobotPose}}$ and $T_{\mathit{Target}}^{\mathit{RobotPose}}$, their position and orientation of tip and target with respect to the current pose of the robot at each instance. Finally, $T_{\mathit{Target}}^{\mathit{Tip}}$ is found by the multiplication $T_{\mathit{Tip}}^{{\mathit{RobotPose}}^{-1}}{T}_{\mathit{Target}}^{\mathit{RobotPose}}$, the matrix product of the inverse of $T_{\mathit{Tip}}^{\mathit{RobotPose}}$ with $T_{\mathit{Target}}^{\mathit{RobotPose}}$.

Calculating $T_{\mathit{Target}}^{\mathit{Tip}}$ at each instance provides the information needed to determine desired compensatory effort relative to a reference frame at the needle tip. With insertion defined along the Z-direction, the rotational effort in the plane normal to insertion can be found using the arctangent of the X and Y positions of $T_{\mathit{Target}}^{\mathit{Tip}}$ as shown in Equation 2.

$${\theta }_d=atan2\left(y(T_{Target}^{Tip}),x(T_{Target}^{Tip})\right)$$ (2)

In our case the imaging software is able to estimate needle tip orientation, therefore the rotational component of $T_{\mathit{Target}}^{\mathit{Tip}}$ can be used to employ a variable curvature needle compensation technique, where both direction and magnitude of desired effort can be implemented. A Gaussian based model employing Continuous Rotation and Variable¹² (CURV) curvature was used for all insertion trials; a variation of the duty-cycle approach⁶ wherein the needle continuously rotates with an angle-dependent angular velocity. In this model for rotational bevel positioning, θ is the current rotational angle with θ_d the desired angle of bevel position for compensatory effort. Angular velocity $\widehat{\omega }$ at each rotational position is calculated using Equation 3 with the two parameters c and α corresponding to the Gaussian width and magnitude of compensatory effort respectively.

$$\widehat{\omega }(\theta ,{\theta }_d)=1-\alpha e^{\frac{(\theta -{\theta }_d{)}^2}{2c^2}}$$ (3)

A larger α generates a greater drop from the nominal rotational velocity, leading to more deflection in the desired direction θ_d, while increasing c widens the total range of angular position where rotation occurs below the nominal velocity. Figure 5 illustrates the angular velocity profile for the CURV approach to active compensation. To note, the scope of this work is not to compare different approaches to steering of the needle trajectory; though the authors chose to employ the CURV approach, other bevel tip based curvature models such as the duty cycle⁶ technique could be readily implemented instead.

Figure 5:

Angular velocity, $\widehat{\omega }$ of the needle during Continuous Rotation and variable (CURV) rotational compensation. This Gaussian based model is dependent on the current rotational position, Θ, the desired direction of compensatory effort, θ_d, the desired Gaussian width c, the magnitude of desired compensatory effort, α.

In our case θ_d was calculated throughout insertion using Equation 2 and c was set to 10° for all insertions. The magnitude of compensatory effort α was calculated throughout insertion by the two-dimensional difference between the target point and projection of the needle tip orientation onto the target plane, thus selecting the intermediate value between maximum curvature of a given needle in a given tissue and a straight line trajectory. Euler angles were extracted from the $T_{\mathit{Tip}}^{\mathit{RobotPose}}$ transform to determine the rotations of needle tip orientation about each of the x, y, z axes of the needle insertion coordinate frame, where insertion was performed along the z-direction. The locations of the projected needle tip position on the target plane were found with Equations 4 – 5, using trigonometric relationships of the angles of rotation about the x and y axes normal to the insertion direction, Rot(x) and Rot(y) respectively.

$$X_{projected}=RemainingInsertionDepth\ast {\tan }^{-1}\left(Rot(y)\right)$$ (4)

$$Y_{projected}=RemainingInsertionDepth\ast {\tan }^{-1}\left(Rot(x)\right)$$ (5)

Furthermore, the x and y values of the current needle tip position were used directly as projections of the current needle position on the target plane, and the inplane magnitude of Error_projected was calculated using Equation 6.

$$Erro{r}_{projected}=\sqrt{{\left(X_{Error}+X_{Projected}\right)}^2+{\left(Y_{Error}+Y_{Projected}\right)}^2}$$ (6)

Finally, the instantaneous value of a was set using Equation 7, the ratio of Error_projected to the maximum deflection attainable using our 18G needle, Error_max. For this experimental setup, Error_max was experimentally determined to be 9.31 mm_rms by performing insertions with no needle rotation to a depth of 125 mm, typical of all insertion trials performed as described below in section 5.B. Figure 6 illustrates the components required to calculate Error_projected inside the experimental setup.

Figure 6:

Dynamically determining the magnitude of desired steering compensation α using the needle tip position and orientation. The x and y positions of the needle tip are projected along the Cartesian z-axis to the target plane as the current X_Error and Y_Error between tip and target. Additionally, the instantaneous needle tip orientation is extended to the target plane, where X_projected and Y_Projected are found using the Euler angles determined from the rotation matrix of ${\mathrm{T}}_{\mathrm{Target}}^{\mathrm{Tip}}$ along the remaining insertion depth. The magnitude of total instantaneous Error_projected is determined Equation 6, the two-dimensional Pythagorean theorem of (x_Error + X_Projected) and (Y_Error + Y_Projected)

$$\alpha =\frac{Erro{r}_{projectd}}{Erro{r}_{max}}$$ (7)

5). Results

A). Tissue Phantoms

To produce useful results of targeting accuracy with compensation for needle deflection, the tissue phantoms used must mimic realistic needle forces and induce needle deflection comparable to a clinical scenario. Using an 18G needle, Podder et al.¹⁹ measured an average in vivo needle force of 8.87 N outside the prostate and 6.28 N inside the prostate during brachytherapy. Furthermore, the Young’s Modulus of excised prostate tissue was reported as 16.0 ± 5.7 KPa and 40.6 ± 15.9 KPa for healthy and cancerous samples respectively⁹. In experiments for this work, tissue phantoms were made from a 70%/30% ratio by volume of Plastisol liquid PVC and PVC softener, parts Regular Liquid Plastic and Plastic Softener from M-F manufacturing (Fort Worth, TX, USA). Use of this material for developing tissue phantoms was described in Hungr et al.¹⁰ and also used in Ahn et al² and Elayaperumal et al.⁵ to mimic soft tissue for needle insertion experiments. Constant velocity needle insertions were performed at 2 mm/s, 6 mm/s, and 10 mm/s with average forces at all speeds reaching between 5.5N and 6.5N, which is within the range of in vivo forces presented. Additionally, the Young’s Modulus was tested through ultrasound elastography on a LOGIQ E9 ultrasound machine with probe model C106, both from General Electric (Boston, MA, USA). The Young’s Modulus for the 70%/30% concentration of plastisol and softener was found to be 25 KPa, within the range of reported excised prostate tissue.

B). Accuracy for Stationary Targets

Two targeting experiments were performed, the first on a static target, and then on a shifted target mimicking tissue deformation and target deflection. In all insertion cases the needle was inserted until it reached the target depth based on visual confirmation of tip location using the needle tracking software. For the case of static targets, the robot was positioned to place the needle tip at the desired entry point aligned along a straight line trajectory toward the target location, and insertions were performed in three conditions: 1) autonomous insertion with no rotation to characterize Error_max, the needle deflection without active compensation, 2) autonomous insertion with image-guided active compensation, and 3) hands-on user directed cooperative insertion with image guided active compensation. The hypothesis was that targeting accuracy would improve with active compensation, and that targeting accuracy would not be negatively affected when moving from an autonomous to cooperative insertion. In each trial a different target within the phantom was selected to avoid errors in a needle following a previous insertion track, and the result for all conditions was the position error between the needle tip and target at the target plane, orthogonal to the initial needle alignment axis.

Ten targeted insertions were performed with and without closed-loop image-guided active compensation and the average in-plane targeting error was found to be 9.30 mm_rms and 3.79 mm_rms for the no compensation and active compensation cases, respectively. Subsequently, five participants (3 males, 2 females, average age 33.8 ± 10.32, all right handed, all novice to the task) each performed ten cooperatively controlled needle insertions with active compensation and Table 1 describes the inplane targeting accuracy for each participant. Figure 7 illustrates the comparison of targeting accuracy between the autonomous no rotation, autonomous with active compensation, and all hands on cooperative cases with active compensation.

Table 1:

Stationary targeting accuracy for insertions under cooperative control

	Subject 1	Subject 2	Subject 3	Subject 4	Subject 5
Mean (mm)	3.50	3.15	3.20	2.90	3.04
St. Dev. (mm)	0.71	1.36	0.62	0.95	1.28

Figure 7:

Comparison of targeting accuracy (n=10) between autonomous insertions with no rotational of bevel tip positioning, autonomous insertions with closed-loop image-guided active compensation through rotational bevel tip positioning, and 5 subjects performing hands-on cooperatively controlled needle insertions with active compensation.

For each of the insertion sets with active compensation, paired t-tests were performed to compare the data. Table 2 illustrates there was no statistically significant difference (all p > 0.05) between the targeting accuracy during autonomous insertion with any of the user directed cooperative cases, in addition to no statistical significance between the targeting accuracy seen between each pair of user. Across all insertions the average target depth was 125.23 mm_rms and the average targeting accuracy with active compensation was found to be 3.56 mm_rms.

Table 2:

Statistical outcomes from paired t-tests on stationary targeting accuracy across insertion conditions

	Cooperative Subject 5	Cooperative Subject 4	Cooperative Subject 3	Cooperative Subject 2	Cooperative Subject 1
Autonomous Insertion	0.253	0.085	0.247	0.298	0.514
Cooperative Subject 1	0.383	0.098	0.383	0.443
Cooperative Subject 2	0.803	0.698	0.880
Cooperative Subject 3	0.549	0.448
Cooperative Subject 4	0.802

C). Accuracy for Shifted Targets

The second targeting experiment evaluated the response of the system to target shift during cooperative insertions. As the error seen in the non-compensatory case of the first experiment was 9.30 mm_rms, it was expected the system could compensate with comparable targeting accuracy to target shifts below this value. Twenty targeted cooperative insertions were performed in a similar manner to the first experiment, with a target shift introduced once the needle made contact with the phantom. The shift was introduced virtually in the target plane perpendicular to the insertion axis, with both direction and magnitude assigned randomly using polar coordinates. The direction of shift was open to all 360° and magnitude was randomly assigned between 1 – 10mm.

There was no statistically significant variation among users in the previous experiment, thus a single user was used for this second experiment. Figure 8 shows the targeting accuracy for each insertion with respect to the amount of target shift, sorted by magnitude of the randomly assigned shift.

Figure 8:

Targeting accuracy vs. in-plane target shift magnitude for twenty cooperatively controlled needle insertions with a randomized shift introduced upon needle contact with the phantom. The direction and magnitude of target shift was randomly generated in polar coordinates to be on the target plane perpendicular to the needle insertion axis at a magnitude between 1 – 10 mm.

D). Accuracy of Feature Localization

Finally, validating the accuracy of feature localization must be performed to confidently use the needle tracking outputs for robot control. A series of five controlled needle insertions were performed into a phantom made with the same 70%/30% ratio of plastisol and softener described in section 5. A to test the accuracy of needle localization. Error was found by subtracting a ground truth coordinate found by manual selection of the needle tip in each video frame from the coordinate calculated through the optical flow needle tracking algorithm. With camera resolution 640x480 pixels, the error over the length of all insertions was 10.78 pixels_rms along the needle insertion path and 1.05 pixels_rms perpendicular to the insertion path, corresponding to 2.65 mm_rms and 0.12 mm_rms at the phantom boundary respectively. Error along the axis of needle insertion is greater because the insertion itself creates substantially more motion along that axis as opposed to perpendicular motion from deflection.

6). Discussion

In this work we presented an imaging-agnostic method to provide closed-loop active compensation for unmodeled needle deflection and target shift during cooperatively controlled needle insertions. The controller is structured to determine compensation parameters from streamed homogeneous frame transformations representing needle tip and target received via OpenIGTLink from any capable imaging source. The robot autonomously updates rotational bevel position based on these inputs to direct the needle tip towards the target. As opposed to an open-loop insertion, active compensation is robust to registration error, distortion of images, unmodeled needle deflection and swelling or target shift. Here we performed tests in the lab setting using perpendicular cameras to act as a proxy for multiplanar MR imaging, as we have previously shown needle localization in MRI¹⁸. This system was developed to be used in the lab and is adoptable by other researchers, only requiring two low cost USB cameras to implement with the tracking algorithm available open source on github (https://github.com/WPI-AIM/NeedleTracker).

Full robotic control of needle insertion is expected to be the most accurate way to place a needle tip for biopsy, but this leap from the clinical standard of practice would likely be met with caution from both acting physicians as well as regulatory bodies. Instead, through user directed control via a cooperative insertion paradigm, a physician would maintain full control of the procedure directly at the procedure site, while adding the benefit of robotic accuracy. Continuous augmented haptic feedback is also provided via force sensing of needle interaction with tissue, returning the tactile forces used by the physician for anatomical localization which would typically be lost when migrating to robotic insertion. Furthermore, an enhanced haptic response can be provided during a hands-on cooperative insertion, for instance a scaled force response for event based detection of membrane puncture.

Needle insertions towards stationary targets were performed using image-guided active compensation under both autonomous and cooperative insertions. As was shown in the experiment, active compensation improved accuracy. This experiment was intended to demonstrate that the improvement in accuracy can be maintained even when the physician is still directly in the loop, thus also enhancing the safety of the procedure. In a separate experiment, a case with no compensation was performed under autonomous insertion to illustrate that active compensation does in fact increase overall targeting accuracy, but each participant did not perform the task without active compensation. Figure 6 first illustrates that in-plane targeting error is greatly reduced when using closed-loop active compensation as compared to a non-compensatory no rotation insertion case. The average targeting error was found to be 3.56 mm_rms in all cases with active compensation, considerably less than the average 6.6 ± 5.1 mm error found via open-loop manual needle insertions during clinical trials using the robot base for target alignment²⁶. Furthermore, there was no statistically significant difference in the targeting accuracy between autonomous and user directed cooperative insertions. These results show three things: 1) closed-loop active compensation through autonomous rotational bevel tip positioning can improve the targeting accuracy when compared to open-loop manual needle insertion, 2) targeting accuracy when using closed-loop autonomous rotational bevel positioning does not degrade when moving from an autonomous insertion to a user directed cooperative insertion, and 3) the system is robust to different individuals performing the cooperative insertions.

Regarding cooperative insertions toward shifted targets, the results indicate that the system does compensate for target shift, with accuracy degrading as shift increases towards Error_max. This outcome is understandable for a system using a relatively stiff needle typical of clinical biopsy where aggressive needle poses and s-curve conformations as seen in research of thin flexible needles are not possible. In this case, if a deflection has begun in one direction it is much harder to overcome the initial curvature already present in an undesired direction. This becomes important due to the randomized location of the target shift in the experiment. If the needle entered the phantom with the bevel tip facing a certain way and began deflecting, a random shift to a location opposite the initial trajectory line, to a distance and angle unlikely to occur anatomically, would cause the final targeting accuracy to degrade and not be reflective of what could be possible with this system clinically. Figure 7 suggests targeting accuracy comparable to the stationary case can be achieved for target shifts approximately up to 5 mm, which in a clinical scenario can still lead to more biopsies collected during the first insertion, shortening procedure time and limiting patient discomfort.

In each experiment the depth of the needle was determined using the visual tracking software. Upon extension of this work into the MRI, live imaging would be available to guide the tip to the target depth as was done here. The accuracy would be assessed in transverse images at the target plane, similarly yielding in-plane error results as determined in these experiments.

From a tissue interaction stand point, a cooperative insertion is just a variable velocity insertion, therefore as long as accurate feature localization is provided to update rotational bevel positioning, the targeting accuracy should be comparable whether performed with autonomous or cooperative insertion. This paper showed a method for image-guided closed-loop active compensation of unmodeled needle deflection and target shift, showing no statistical difference between targeting accuracy under autonomous robotic needle insertion versus user directed hands-on cooperative needle insertion. Through analysis of the results, the authors believe overall targeting accuracy could be improved if the same experiments were performed with an increased speed of needle rotation, a more controlled target shift to better represent what is possible anatomically, and ensuring in each trial a biopsy gun with a like-new needle is used, in case any deflection during experiment plastically deformed the needle to have natural curvature.