An X-ray C-arm Guided Automatic Targeting System for Histotripsy

Abstract

Objective:

Histotripsy is an emerging noninvasive, nonionizing and nonthermal focal cancer therapy that is highly precise and can create a treatment zone of virtually any size and shape. Current histotripsy systems rely on ultrasound imaging to target lesions. However, deep or isoechoic targets obstructed by bowel gas or bone can often not be treated safely using ultrasound imaging alone. This work presents an alternative x-ray C-arm based targeting approach and a fully automated robotic targeting system.

Methods:

The approach uses conventional cone beam CT (CBCT) images to localize the target lesion and 2D fluoroscopy to determine the 3D position and orientation of the histotripsy transducer relative to the C-arm. The proposed pose estimation uses a digital model and deep learning-based feature segmentation to estimate the transducer focal point relative to the CBCT coordinate system. Additionally, the integrated robotic arm was calibrated to the C-arm by estimating the transducer pose for four preprogrammed transducer orientations and positions. The calibrated system can then automatically position the transducer such that the focal point aligns with any target selected in a CBCT image.

Results:

The accuracy of the proposed targeting approach was evaluated in phantom studies, where the selected target location was compared to the center of the spherical ablation zones in post-treatment CBCTs. The mean and standard deviation of the Euclidean distance was 1.4 ± 0.5 mm. The mean absolute error of the predicted treatment radius was 0.5 ± 0.5 mm.

Conclusion:

CBCT-based histotripsy targeting enables accurate and fully automated treatment without ultrasound guidance.

Significance:

The proposed approach could considerably decrease operator dependency and enable treatment of tumors not visible under ultrasound.

I. Introduction

Histotripsy is an emerging noninvasive, nonionizing and nonthermal focal cancer therapy that is highly precise and can create a treatment zone of virtually any size and shape [1], [2]. Histotripsy uses a unique physical process (cavitation) to destroy tissue. Very short duration (< 20 μs), high-pressure (> 15 MPa) ultrasound pulses are applied from outside the body and focused inside the target tissue to cause acoustic cavitation (microbubble formation) [3], [4]. The rapid expansion and collapse of the cavitation microbubbles produces high stress and strain to mechanically disrupt tissue at the cellular and subcellular level [5], [6]. Cavitation, and the associated tissue damage, is a threshold phenomenon and only occurs at the focal point of the therapeutic transducer [4], [6]. For the current histotripsy clinical treatment, once a target is selected, the treatment is automatically delivered via a robotic arm integrated into the histotripsy system, which moves the cavitation bubble cloud throughout the desired treatment zone [7]–[11]. Histotripsy spares critical structures with a higher collagen content and mechanical strength (eg, bile ducts and vessels) while otherwise causing uniform tissue destruction and cellular necrosis in the targeted treatment volume [9], [12], [13]. Early rodent work has also demonstrated an immune potentiating effect that could have widespread applications [14]. Clinical trials assessing the safety and efficacy of treating liver tumors with histotripsy are underway (NCT04572633, NCT04573881). Although histotripsy has highly promising therapeutic capability, a major barrier to clinical adoption is the lack of a reliable method for visualizing and targeting tumors.

Histotripsy currently relies on a 2D diagnostic ultrasound transducer coaxially mounted to the therapy transducer to plan, guide, and monitor treatment as well as for initial treatment assessment. However, ultrasound has a number of limitations for targeting and monitoring therapies, including an inability to visualize many locations in the body due to blockage of the beam by bowel, aerated lung, bone, and large body habitus. An inability to adequately visualize and monitor tumors and cavitation could substantially limit the number and location of tumors that can be safely treated. For example, in the first inhuman study (THERESA) [15] tumors in liver segments 7 and 8 were excluded. Additionally, 3 out of 14 patients were screen failures due to inadequate acoustic window and 1 out of 8 patients was mis-targeted. Furthermore, these numbers do not capture the cases, which were never formally enrolled because of known poor ultrasound windows. However, even in cases where the tumor cannot be visualized using ultrasound imaging, it is still possible to perform histotripsy treatments safely if alternative targeting methods were available. Sukovich et al. [16] showed the ability to perform histotripsy treatments through the skull and Knott et al. [17] successfully performed transcostal histotripsy treatments, where ribs and parts of the lung or bowel interfered with the treatment trajectory. In these cases, additional cooling time was added during treatment to minimize body wall heating. Since histotripsy is only recently being investigated for human clinical use, little work has been done on alternative techniques for targeting. The limitations in targeting and monitoring described above are shared by other ultrasound-based treatment techniques such as high intensity focused ultrasound (HIFU). MRI-guidance has been investigated for observation of the heat deposition in tissue during HIFU treatment [18]–[20], which allows observing the heat deposition in tissue during treatment. While this approach is not directly translatable to histotripsy due to the nonthermal nature, previous research suggested that the visualization of mechanically fractionated tissue using MRI is possible [21]–[24]. CT is the most widely used imaging guidance technique for current focal cancer therapies such as percutaneous thermal ablation. CT offers rapid volumetric imaging for target tumor identification, confirmation of probe placement and immediate assessment of treatment success (ie complete coverage of the tumor by the treatment zone). However, closed bore imaging systems such as conventional CT and MRI are not ideal for hepatic histotripsy due to limited space for the treatment head, field of view limitations, and artifacts from the therapeutic transducer and robotic arm. C-arm x-ray fluoroscopy with cone-beam CT (CBCT) is well suited for targeting during histotripsy and an excellent complement to ultrasound for several reasons: 1) The open configuration allows easy patient access and room for the robotic arm and therapeutic transducer, 2) the two imaging capabilities (2D fluoroscopy and CBCT) provide complementary functionality, 3) CBCT produces volumetric CT-like images for target identification, treatment planning and assessment [25]–[27], 4) 2D fluoroscopy provides real-time intra-procedural targeting, and 5) C-arms are ubiquitous world-wide and there is widespread expertise in their use. The techniques presented in this work use 2D fluoroscopic imaging to perform position and orientation (pose) estimation of the histotripsy transducer. X-ray based pose estimation has been proposed for other applications such as ultrasound to x-ray fusion [28]–[30] and 3D localization of implants and devices [31], [32]. Generally, these approaches can be categorized into fiducial-based [28], model-based methods [30], and deep learning approaches [31], [32]. While, for many applications, fiducial tracking provides robust pose estimation, the large size and high density of the histotripsy transducer would make it difficult to attach fiducials such that they are always in the x-ray field of view and do not overlap with the transducer or interfere with the treatment.

The purpose of this work was to: 1) develop a model-based pose estimation approach, which allows the co-registration of the histotripsy transducer and the C-arm coordinate systems, and 2) implement a fully automatic histotripsy targeting system using only C-arm imaging and robotic guidance. Instead of using added fiducial features for the pose estimation, this algorithm relies on fluoroscopic images of the unmodified therapy transducer directly. This study evaluates the accuracy of the transducer focal point location predicted from fluoroscopic images and of the proposed automatic targeting approach.

II. Materials & Methods

Currently, histotripsy is solely guided using an integrated diagnostic ultrasound probe. The conventional workflow (see figure 1) can be summarized in the following steps: 1) The therapy transducer is positioned in a water bath located on the patient for acoustic coupling. The physician can then translate and rotate the transducer manually using the robotic arm to locate the target lesion on live diagnostic ultrasound images. 2) Once the lesion is identified, a treatment plan including the treatment center, diameters, and margin are specified. 3) A calibration step is performed, where a test pulse is generated to determine the offset between geometrical transducer focal point and bubble cloud location due to acoustic aberration. This offset is subtracted from the center of the treatment plan. 4) Histotripsy treatment is performed by moving the bubble cloud (approx. 3 × 3 × 6 mm) through the prescribed region to cover the tumor and margin. While ultrasound imaging is available during treatment, no image feedback is used for treatment purposes. Since tissue destruction for histotripsy can assumed to be instantaneous [33] the transducer is automatically moved along a preprogrammed path relative to the prescribed treatment plan center. 5) An initial assessment of the resulting treatment zone can be performed on the diagnostic ultrasound images to confirm tumor coverage.

Fig. 1.

Conventional ultrasound-based histotripsy targeting workflow: 1) The transducer is inserted in a water bath placed on the patient. Using the robotic arm, the physician can manually translate and rotate the transducer, while the diagnostic transducer provides live ultrasound imaging to locate the tumor. 2) Once the tumor is identified, a treatment plan is created by defining the center and size of the treatment region and margin containing the tumor. 3) A test pulse is generated to calibrate for small offsets between the geometrical transducer focal point and the bubble cloud location due to acoustic aberration. 4) Histotripsy treatment is performed by moving the bubble cloud through the prescribed region. 5) An initial assessment of the resulting treatment can be performed using the diagnostic ultrasound probe.

The proposed C-arm based targeting workflow provides automated targeting based on CBCT images and could replace steps 1 and 2. Additionally, post treatment CBCT images can be used for treatment assessment (step 5). While live imaging of the bubble cloud (for step 3) using CBCT or fluoroscopy is currently not possible, potential workarounds for calibration without ultrasound are discussed in section IV. For this phantom study, no calibration was performed and the geometrical focal point was assumed as bubble cloud location. The first component of a C-arm-based histotripsy targeting platform is the estimation of the histotripsy transducer pose relative to the CBCT coordinate system using 2D fluoroscopic imaging. Knowing the transducer geometry (arrangement of the individual transducer elements) then allows a determination of the geometrical focal point of the transducer corresponding to the position of the bubble cloud. The proposed approach uses a biplane fluoroscopy system with two C-arms, herein referred to as plane A and plane B, for simultaneous x-ray image acquisition from two different angles. However, the same approach could also be implemented for single plane systems by moving the C-arm to different angles. By itself, the pose estimation provides a mechanism to indirectly observe the focal point location (inferred from transducer pose) under live fluoroscopic imaging. However, the proposed system also provides a technique to calibrate the integrated robotic arm (UR5e, Universal Robots, Odense, Denmark) that controls the transducer pose. After co-registering the coordinate systems of the robotic arm and the CBCT, the focal point location can be monitored without live imaging, which would allow the operator to select a target (e.g. a tumor) on the CBCT image and automatically guide the transducer to the target location (see Figure 2).

Fig. 2.

Proposed C-arm based histotripsy targeting workflow using the calibrated robotic arm to automatically target a lesion identified in the C-arm coordinate system: 1) The robot is calibrated to the C-arm based on a series of pose estimations without patient present. 2) A conventional cone beam CT of the patient is acquired and the target lesion identified. 3) The calibrated robotic arm automatically positions the transducer such that the focal spot is aligned on the target. 4) The robotic arm automatically moves the transducer through the prescribed treatment region during histotripsy treatment. 5) A post-treatment cone beam CT is acquired to verify that the lesion was successfully treated.

A. Notation and Nomenclature

Assuming a 3-dimensional space V on a cartesian grid $\overrightarrow{x}=[x,y,z]$, the CBCT can be described by dividing V into isometric cubes (voxels). A grayscale value $V(\overrightarrow{x})$ is assigned to each voxel representing the x-ray attenuation coefficient at point $\overrightarrow{x}$. If $\overrightarrow{x}$ falls between voxel coordinates, $V(\overrightarrow{x})$ represents the tri-linear interpolation. Similarly, the 2D projection image I_α acquired at gantry orientation α can be defined on a 2D cartesian grid with isometric pixels u = [u, v]. The relationship between 3D and 2D space can be described by a projective transform P_α represented by a 4 × 3 matrix using homogeneous coordinates.

$$[u,v,w]=P_{\alpha }\cdot {[x,y,z,1]}^{\top }$$ (1)

Within this work, the notation $H(\overrightarrow{u})$ will be used to describe the homogeneous divide and subsequent drop of the third coordinate such that

$$H([u,v,w])=[u/w,v/w].$$ (2)

In practice, the projection matrices P_α can be determined for selected gantry angles using calibration phantoms with known geometry. [34]

B. Transducer Pose Estimation

The 2D/3D pose estimation approach determines the 3D position and orientation of the transducer in the CBCT coordinate system from two fluoroscopic images acquired from different projection angles. This allows transforming the known location of the geometrical focal point of the transducer from the transducer coordinate system to the CBCT coordinate system. The approach will be used for the calibration of the robotic arm to the C-arm system as described in section II-C. The pose estimation algorithm iteratively translates and rotates a digital model of the transducer to minimize a cost function which depends on the similarity between virtual forward projections of the digital model and the two fluoroscopic images.

1). Digital Model:

A digital point cloud model of the histotripsy transducer was generated to serve as a reference object during pose estimation and to allow the simulation of virtual projection images for the training of deep learning networks (see sections II-B.2 and II-B.4). The point cloud includes structural elements, points representing the absence of x-ray visible parts, and boundaries. In particular, each point $p_i^{(k)}$ is assigned to one of five categories k including the main transducer elements, gaps between transducer parts, inner and outer transducer boundaries, and screws. The main transducer elements are described by a curved surface created by the intersection of a sphere (138.7 mm radius) with cylinders or elliptical cylinders, respectively, where the center of the sphere corresponds to the geometrical focal point and thus the predicted bubble cloud location. In the center of the transducer, a 6.4-cm diameter circular cutout provides space for the diagnostic ultrasound probe. Due to manufacturing constraints, the transducer is divided into four quadrants which are fabricated separately. The resulting gaps between the quadrants create x-ray visible features, which are useful for pose estimation. The points of the digital model for pose estimation were distributed on a regular Cartesian grid with a resolution of 0.47 mm. Additionally, a high resolution (0.125 mm) version was created for the simulation of virtual x-ray projections.

2). Semantic Segmentation:

In order to determine the similarity between the forward projected transducer model and the 2D x-ray images, the same categories assigned to each point in the digital model must be identified in the projection images. To this end, a deep-learning-based semantic segmentation approach was implemented using the U-net architecture [35] with a single input channel (the fluoroscopic image) and 6 output channels representing the background and 5 point categories respectively. In cases of overlapping features, the corresponding pixel was assigned to the category which provides the most important information for pose estimation in the order: (1) screws, (2) gaps between transducer elements, (3) inner transducer boundary, (4) outer transducer boundary, (5) main transducer elements. The network was trained using hybrid images, created by adding virtual forward projections of the high-resolution digital phantom to real x-ray projections from 12 porcine studies. The x-ray images were extracted from 3D CBCT scans using the raw projection images from 304 different gantry angles (3648 images in total) and preprocessed using proprietary software from the C-arm manufacturer (Siemens Healthineers, Forchheim, Germany) to convert the pixel intensities to line integrals of linear attenuation coefficient. Random transducer poses were then generated using Euler angles within the range [0; 360] and translations in the range of [−10; 10] cm in each dimension. A point driven “splatting” approach [36] was used for the forward projection, where each point was projected 16 times with a random offset drawn from the normal distribution 𝒩 (0, 0.5²) for each dimension. Finally, Poisson noise was added to simulate the image quality of projections acquired at fluoroscopic dose levels [37]. The training was performed on randomly cropped 512 × 512 image patches using a gradient descent approach with momentum (0.9), a constant learning rate of 0.001 and a mini-batch size of 5. Training was stopped after no reduction in validation loss was achieved for 3 consecutive epochs (after 50 epochs).

3). Cost Function:

A key element of the proposed pose estimation is the cost function which is inversely related to the similarity between the forward projected digital model and the segmented x-ray images. In particular, the cost function calculates the shortest squared Euclidean distance between each forward projected point and the closest pixels corresponding to the same category. Mathematically, this can be expressed as

$$\begin{gathered} C(R,\overrightarrow{t})=\sum _{\forall \alpha }\sum _{k=1}^5\frac{{\lambda }_k}{n_k}\cdot \sum _{i=1}^{n_k}\underset{j}{\text{min}}{\Vert {\widehat{p}}_i^{(k)}-s_j^{(k)}\Vert }^2,\text{ with } \\ {\widehat{p}}_i^{(k)}=H\left(P_{\alpha }\cdot \left(R\cdot {\overrightarrow{p}}_i^{(k)}+\overrightarrow{t}\right)\right) \end{gathered}$$ (3)

where R and $\overrightarrow{t}$ represent the model parameters as a 3D rotation matrix and translation vector, ${\overrightarrow{s}}_j^{(k)}$ are the segmented 2D point coordinates for category k in homogeneous coordinates, n_k is the number of points in category k and λ_k is a constant weighting factor for each category. In practice, the minimum distances can be interpolated from precalculated distance maps to improve computational efficiency, which can be computed in linear time with respect to the number of image pixels using the Felzenszwalb-Huttenlocher approach [38]. Another advantage is that the distance maps can be calculated prior to the pose optimization making the approach computationally more efficient. In the resulting distance maps calculated for each binary segmentation of category k, each pixel represents the Euclidean distance to the closest pixel corresponding to this category. Thus, the minimum distance term of the cost function can be calculated simply by interpolating a distance map texture at all forward projected points using bilinear interpolation. Special care must be taken for model points outside the field of view when forward projected. Due to the size of the transducer, only a small part of it may be visible in the fluoroscopic images such that many model points would be outside the field of view for the correct transducer pose. Thus, calculating the distance to the closest segmented pixels for those points might cause larger costs than any pose where all points are within the field of view. On the other hand, if the cost for points outside is too low, the algorithm may optimize the pose simply by moving the whole model outside the field of view. To avoid this problem, an empirically determined constant distance value of 0.1 pixels was assigned to all points outside the distance map. The cost function was iteratively minimized using the Nelder-Mead simplex approach [39] with a maximum of 1000 iterations and a function value and parameter change tolerance of 0.01.

4). Deep Learning-Based Initialization:

As for most nonlinear optimization problems, a good initialization of the model parameters (rotation and translation) is crucial to achieve accurate and robust pose estimation results. To this end, a deep-learning-based initialization approach was designed, which addresses the initial estimation of the model parameters in three steps. The first step estimates the azimuth and elevation angles of the transducer relative to each of the 2D projection images using a regression approach. Secondly, the initial transducer translation is estimated using a hierarchical grid search followed by a linear search over axial rotation angles. Due to the rotational symmetry of the transducer, the axial rotation angle has little influence on the appearance of the transducer in 2D projection images. Thus, to limit the number of degrees of freedom for the first step, a deep learning network was designed based on the VGG19 architecture [40], which only estimates azimuth and elevation of the transducer corresponding to the direction of the vector pointing from the center of the transducer to the focal point location. To improve the convergence of the training a two-step approach was chosen, where first, a classification network using the original VGG19 architecture was trained for classification of 36 unique transducer orientations. The number of output classes was chosen empirically, and the corresponding transducer poses were chosen through equidistant sampling of the manifold describing all possible transducer orientations. Due to the similarity between transducer poses when mirrored on the detector plane, the network was trained to only consider transducer directions pointing away from the detector. This ambiguity can then be resolved by combining the information from both image planes. The training was performed using hybrid images, where virtual forward projections of the digital model were superimposed on x-ray images from animal studies to create a large variety of images with random transducer translation and axial rotation for each class. Approval for the animal studies was received from the local Institutional Animal Care and Use Committee (IACUC) of the University of Wisconsin School of Medicine and Public Health on July 11, 2019 (protocol number: M005544-R01-A01). The trained classification network was then converted to a regression network by replacing the softmax layer with a sigmoid layer followed by a fully connected layer with two outputs for the azimuth and elevation angle, respectively. Training was performed using the same hybrid approach as for the classification for a total of 100 epochs (400,000 iterations). Input images were downscaled by a factor of 0.25 to 310×240 pixels prior to applying the network.

To determine the initial parameters for the pose estimation, the trained network is applied to both input images, providing two possible values for the azimuth ϕ and elevation angle θ, where each is relative to the gantry orientation defined by the rotation matrix R_A or R_B, respectively. Each set of angles can be converted to a unit direction vector corresponding to the direction from the center of the transducer to the geometrical focal point and adjusted for the gantry rotation. Since each direction vector may be mirrored on the detector plane, this resulzts in the following 4 possible solutions

$$\begin{gathered} d_{A_{1,2}}=R_A^{-1}\left(\begin{array}{c}\text{cos}{\varphi }_A\cdot \text{sin}{\theta }_A\\ \text{sin}{\varphi }_A\cdot \text{sin}{\theta }_A\\ \pm \text{cos}{\theta }_A\end{array}\right) \\ {d}_{B_{1,2}}=R_B^{-1}\left(\begin{array}{c}\text{cos}{\varphi }_B\cdot \text{sin}{\theta }_B\\ \text{sin}{\varphi }_B\cdot \text{sin}{\theta }_B\\ \pm \text{cos}{\theta }_B\end{array}\right) \end{gathered}$$ (4)

To resolve this ambiguity, the algorithm selects the pair of direction vectors (one from each detector plane) that minimizes the difference between the two vectors. The initial orientation is then calculated by averaging the two vectors. The initial translation of the transducer model is determined using a hierarchical grid search for each detector plane to minimize the cost function $C(R,\overrightarrow{t})$. To reduce the grid dimension, the search was performed separately for each image plane on a 2D grid parallel to the detector plane. In the first stage, 130 grid points on a 40 × 30 cm search grid centered on the isocenter with a step size of 3 cm were evaluated. The grid point which minimizes the solution for plane A was then used as the center of the search grid for plane B. Subsequently, a second pass was performed using a finer search grid with a 3 × 3 cm grid size and 0.5 cm grid point spacing (49 grid points). Finally, the initial axial rotation of the transducer model was determined by evaluating the cost function for 72 equally spaced angles within the interval [0°, 180°].

C. Robotic Arm Calibration

This section describes the calibration between the robotic arm and the C-arm, which will subsequently allow a user to automatically target any position selected on a CBCT image. To this end, the above-described pose estimation approach is used, which provides a mechanism to determine the position and orientation of the transducer relative to the CBCT coordinate system. This relationship can be described by a rigid transform ^hT_c such that the focal point location can be determined by

$$f_c{=}^h{T}_c\cdot f_h,$$ (5)

where f_h is the geometrical focal point of the transducer in the coordinate system of the histotripsy transducer. While this approach by itself would enable visualization of the target location on CBCT under live fluoroscopic imaging, when combined with the integrated robotic arm it provides a system for visual servoing [41] and hand-eye calibration with the eye (C-arm detector) fixed in the world coordinate system. The hand-eye calibration would allow targeting any position in the CBCT coordinate system without additional imaging. The research histotripsy system used for this work is equipped with a 6-jointed, high precision robotic arm (UR5e, Universal Robots, Odense, Denmark) with a pose repeatability of ±0.03 mm. The base of the robotic arm is mounted in a fixed position on the histotripsy cart and the histotripsy transducer is attached to the end effector of the robotic arm. Due to the precise encoding of the robotic arm joints, the rigid transform from robotic arm base to end effector ^rT_e can be calculated for every pose. Likewise, the transform from C-arm to transducer coordinate system ^hT_c can be determined using the proposed pose estimation. The two unknown transforms are the transform from end effector to histotripsy transducer ^eT_h and from robot base to C-arm coordinate system ^rT_c. Figure 3 gives an overview of these transforms and coordinate systems, where the superscript in brackets refers to the index of a specific robotic arm pose. Since the two unknown transforms only depend on the position of the mobile histotripsy cart in the room and the attachment of the transducer to the robotic arm, the calibration can be performed without the patient present and thus requiring no additional radiation exposure. The relationship between the coordinate frames for two robotic arm poses as shown in Figure 3 can be described by the equation

$${}^h{T}_c^{(1)}{\cdot }^e{T}_h{\cdot }^r{T}_e^{(1)}{=}^h{T}_c^{(2)}{\cdot }^e{T}_h{\cdot }^r{T}_e^{(2)}$$ (6)

which can be rewritten as homogeneous Sylvester equation [42]

$$\underset{A}{\underbrace{{\left({}^h{T}_c^{(2)}\right)}^{-1}{\cdot }^h{T}_c^{(1)}}}\cdot \underset{X}{\underbrace{{}^e{T}_h}}=\underset{X}{\underbrace{{}^{e}T{}_h}}{\cdot }^r{T}_e^{(2)}\cdot \underset{B}{\underbrace{{\left({}^r{T}_e^{(1)}\right)}^{-1}}}$$ (7)

Within this work, the dual quaternion approach proposed by Daniilidis [43] was used to analytically solve for ^eT_h requiring at least two motions (3 robotic arm poses). The solution can be determined by calculating the nullspace of the 6n × 8 matrix in equation (8) using the singular value decomposition

$$\left(\begin{array}{ccc}\begin{array}{cc}{\overrightarrow{a}}_0-{\overrightarrow{b}}_0& {\left[{\overrightarrow{a}}_0+{\overrightarrow{b}}_0\right]}_{\times }\\ {\overrightarrow{a}}_0^{\prime }-{\overrightarrow{b}}_0^{\prime }& {\left[{\overrightarrow{a}}_0^{\prime }+{\overrightarrow{b}}_0^{\prime }\right]}_{\times }\end{array}& & \begin{array}{cc}{\mathbf{0}}_{3\times 1}& {\mathbf{0}}_{3\times 3}\\ {\overrightarrow{a}}_0-{\overrightarrow{b}}_0& {\left[{\overrightarrow{a}}_0+{\overrightarrow{b}}_0\right]}_{\times }\end{array}\\ & ⋮& \\ \begin{array}{cc}{\overrightarrow{a}}_n-{\overrightarrow{b}}_n& {\left[{\overrightarrow{a}}_n+{\overrightarrow{b}}_n\right]}_{\times }\\ {\overrightarrow{a}}_n^{\prime }-{\overrightarrow{b}}_n^{\prime }& {\left[{\overrightarrow{a}}_n^{\prime }+{\overrightarrow{b}}_n^{\prime }\right]}_{\times }\end{array}& & \begin{array}{cc}{\mathbf{0}}_{3\times 1}& {\mathbf{0}}_{3\times 3}\\ {\overrightarrow{a}}_n-{\overrightarrow{b}}_n& {\left[{\overrightarrow{a}}_n+{\overrightarrow{b}}_n\right]}_{\times }\end{array}\end{array}\right)\cdot \overrightarrow{q}=0$$ (8)

where a_i and $a_i^{\prime }$ are the real and dual 3 × 1 vector part of the dual quaternion representing the transformation matrix A in equation (7) for the ith motion. Similarly, b_i and $b_i^{\prime }$ are the vector parts of the dual quaternion representing transformation matrix B. The operator [·]_× denotes the cross-product matrix. Finally, $\overrightarrow{q}$ is the dual quaternion describing the transform ^rT_c represented as an 8 × 1 vector. In this work, 4 separate robot poses (n = 3 motions) were used for the calibration.

Fig. 3.

Transforms between robot arm, C-arm and histotripsy transducer for two robot arm poses used for the hand-eye calibration.

D. Experimental Setup

A prototype system was implemented on a research workstation (Precision 5810 XL, Dell, Round Rock, TX.) with a 6 core, 4GHz processor (Xeon E5–1650 v4, Intel, Santa Clara, CA), 64 GB memory and a dedicated graphics card (GeForce GTX 1080Ti, NVIDIA, Santa Clara, CA). The prototype included the proposed transducer pose estimation, the robotic arm calibration, and functionality to navigate the robotic arm. In order to speed up the pose estimation task, the algorithm was implemented on the GPU using C++ and CUDA 10.1 (NVIDIA, Santa Clara CA), while the robotic arm calibration and navigation was implemented in C++ and python 3.7. The workstation was connected to a biplane C-arm system (Artis Zee, Siemens Healthineers, Forchheim, Germany) via Ethernet connection for live streaming of 2D x-ray images to the research workstation. In this study, C-arm angles of −45° and 45° around the posteroanterior angle were used. However, other projection angles can be selected to allow for additional space for the transducer and robotic arm. Additionally, the workstation was connected to the histotripsy system via Ethernet to provide direct control of the integrated robotic arm. The histotripsy system (HistoSonics System, HistoSonics, Plymouth, Minnesota) included a 64-channel therapy sub-system and the robotic arm configured with a 700 kHz therapy transducer with an active area of approximately 24×17 cm, f number of 0.7, 14 cm focal length and excitation sequence to create a 3×3×6 mm bubble cloud. During automated treatment, the robotic arm was motioned through the planned treatment volume comprised of a plurality of discrete focal locations and delivered a predetermined fixed number of histotripsy pulses per focal location. Figure 4A shows an overview of the experimental setup.

Fig. 4.

A. Experimental setup with biplane C-arm gantries aligned on the transducer which is placed to target the phantom in the water bath. B. Layered phantom with alternating layers with and without Barium and homogeneous histotripsy treatment zone in the center.

To test the system, phantom studies were performed to i) determine if the bubble cloud position can be accurately determined using the pose estimation and ii) evaluate the accuracy of the fully automated histotripsy targeting platform. In both cases, an agarose-based phantom (described below) was placed inside a degassed water bath on the patient table of the C-arm system. For the studies assessing the automated targeting solution, the robotic arm was then calibrated to the C-arm by moving the transducer to four different poses and acquiring fluoroscopic images from two views using the two imaging gantries. The pose estimation and subsequent calibration were then performed as described in the previous sections. A CBCT of the phantom was then acquired and a spherical target with a diameter of 2.0 or 2.5 cm selected. Subsequently, the robotic arm was used to automatically position the transducer and perform the histotripsy treatment. A second CBCT was acquired post-treatment to determine the effective treatment volume and compare it to the selected target volume. For the studies without automated targeting (to determine the pose estimation accuracy only), no robotic arm calibration was performed. Instead, the transducer was manually positioned to align the focal point on the center of the phantom under ultrasound imaging guidance. Using the robotic arm, the transducer is then translated by 1.0 or 1.25 cm in each direction along the main robotic arm coordinate axes. These positions describe the margins (top, bottom, left, right, front, and back) of the prescribed treatment volume corresponding to a sphere with 2.0 or 2.5 cm diameter, respectively. Biplane fluoroscopic images were acquired at the center position and each of the margin points to perform pose estimation. Finally, histotripsy treatment was performed followed by a post-treatment CBCT acquisition. The estimated center and margin points using pose estimation were then compared to the actual treatment volume in the post-treatment CBCT.

1). Phantom Design:

The histotripsy treatment effect can only be visualized in the post-treatment CBCTs if there is a change in x-ray attenuation coefficient during treatment. Although clinically this can be visualized as a hypointense region on a contrast enhanced CBCT scan resulting from disruption and perfusional changes, [25], [44] this behavior is challenging to reproduce in a phantom. Therefore, a new multi-layer phantom was developed in which alternating layers of high and low attenuation materials are mixed during histotripsy treatment creating a homogeneous mixture. The mixture of the layers has a medium attenuation value which is easily identifiable in CBCT images. The phantom is a modified version of previously described phantoms for assessing histotripsy under ultrasound [45]: A mixture of 6 g low gelling temperature agarose (Sigma A9045, MilliporeSigma, Burlington, MA) and 400 mL saline were heated to approximately 80 °C and subsequently degassed using a vacuum pump (2561B-50 WOB-L, Welch, Mt. Prospect, IL) and chamber (BVV, Naperville, IL). Once the solution reached a temperature of 43 °C, 20 mL sheep red blood cells (ISHRBC100P100ML, Innovative Research, Inc., Novi, MI) were added and the mixture was poured in a 5×5×5 cm silicone mold to harden. For the layered phantom, the above-described mixture was prepared in two separate batches. The recipe for the high attenuation batch was modified by adding 10 g barium sulfate prior to degassing the saline-agarose mixture. Thin layers (approximately 4 mm) of the two materials were then poured into the silicone mold alternatingly while waiting for the surface to harden before pouring the next layer. The final phantom was then submerged in saline and placed in the vacuum chamber for further degassing until treatment. Figure 4B shows an example of the multi-layer phantom with homogeneously mixed treatment area in the center.

E. Evaluation Metrics

To evaluate the accuracy of the proposed methods, the intermediate results of the 2D transducer segmentation and the deep learning approach were first analyzed before calculating the pose estimation and automatic targeting accuracy. For the 3D results (pose estimation and automatic targeting) it is helpful to break down errors into directional components relative to the transducer orientation. To this end, the direction from the center of the transducer to the geometrical focal point (the direction of ultrasound propagation) will be referred to as depth. The directions in the perpendicular plane will be referred to as top-to-bottom (TB) and left-to-right (LR).

1). Segmentation:

The deep-learning-based transducer segmentation approach was evaluated on 310 manually segmented fluoroscopic images from phantom and pig studies [46]. The reference annotation of the images was performed by manually adjusting the 3D rotation and translation of the digital transducer model to align it with the 2D projection images. To evaluate the results, the average pixel accuracy, intersection over union (IoU), precision and recall were calculated as metrics [47]. Analysis of variance (ANOVA) was used to compare the precision results between different feature classes. Subsequent multiple comparisons were performed using Tukey’s HSD test [48].

2). Initialization:

The deep learning initialization approach was evaluated on manually annotated fluoroscopic images from phantom and pig studies. Evaluation metrics were calculated using the mean absolute error between the predicted and reference azimuth and elevation angles of the transducer. Additionally, the Pearson correlation between deep learning estimation and reference was determined for each network output as a measure of linearity. Subsequently, a linear regression analysis was performed using the Moore-Penrose inverse [49] to determine intercept and slope, which in the ideal case, would be 0 and 1, respectively. The R2 value was used as a metric for the quality of the linear fit.

3). Pose Estimation:

The pose estimation error was calculated in the phantom experiments by comparing the predicted target (center and margin) to the effective treatment zone (target registration error). Quantitative metrics were calculated by manually segmenting the effective treatment region in the post-treatment CBCT images. The accuracy of the predicted treatment center was then compared to the centroid of the segmented region using the 3D Euclidean distance. Similarly, the accuracy of the margin points was calculated as the Euclidean distance between the predicted margin points and the surface of the segmented sphere. Additionally, mean absolute error of the predicted compared to the reference treatment diameter in each direction was calculated. Finally, the precision of the pose estimation was determined by calculating the Euclidean distance between the centroid of the 6 predicted margin points and the predicted center.

4). Targeting:

The accuracy of the fully automated targeting system was evaluated in the same way as the pose estimation error by treating phantoms with histotripsy. The resulting treatment zone (reference) was manually segmented from the post-treatment CBCT and the Euclidean distance between the centroid of the segmented region and the intended target selected in the pre-treatment CBCT calculated.

III. Results

Figure 5 shows examples of the 2D segmentation results for x-ray images of the histotripsy transducer from 6 phantom and 7 animal experiments (56 and 156 images, respectively). The global accuracy for all classes was 93.6±1.9% with a weighted IoU of 89.0±3.21% for all classes. The corresponding average values for precision and recall were 95.6 ± 2.2% and 84.3 ± 3.3%, respectively. A one-way ANOVA test over the precision results for all six class types revealed significant differences between groups (F (5, 1262) = 85.6, p < 10⁻⁵). Tukey’s HSD test for multiple comparisons found that the mean precision value of group screws was significantly lower than the background precision ( p < 10⁻⁵, C.I. = [7.7%, 10.1%] ). No other group had a precision value significantly lower than the background class. Table I lists the metrics for all individual classes.

Fig. 5.

Example deep learning-based transducer segmentation results. A) Input x-ray image, B) manual annotated reference, and C) corresponding color-coded segmentation. Each color represents a different feature class (orange: main transducer elements, blue: outer transducer boundary, purple: inner transducer boundary, green: gaps between transducer elements, yellow: screws).

TABLE I.

Quantitative Segmentation Results

	Precision	Recall	IoU	Accuracy
Background	92.3 ± 3.2%	99.8 ± 0.4%	92.1 ± 3.1%	95.0 ± 1.7%
Transducer	96.2 ± 1.7%	89.2 ± 4.1%	86.1 ± 3.5%	95.1 ± 1.6%
Outer Ring	93.0 ± 8.0%	48.6 ± 4.6%	47.0 ± 5.6%	98.8 ± 0.4%
Inner Ring	97.6 ± 3.0%	41.2 ± 5.0%	40.8 ± 4.9%	99.4 ± 0.2%
Gaps	90.4 ± 11.9%	59.9 ± 5.7%	56.5 ± 8.0%	99.6 ± 0.1%
Screws	82.2 ± 14.5%	68.4 ± 10.3%	59.1 ± 11.5%	99.2 ± 0.4%

IoU = intersection over union

The deep learning-based estimation of azimuth and elevation angles (initialization) was evaluated on the same manually annotated dataset as the segmentation. A strong linear correlation (p < 10⁻⁵) between predicted and reference angles was found (azimuth: 99.1%, elevation: 99.9%). The linear fit for the azimuth angle prediction had a slope of 1.078 and an intercept of −0.136° (R² = 0.983). Similarly, the corresponding results for the elevation angle were a slope of 1.048 and an intercept of −0.101° (R² = 0.999). The mean absolute differences were 1.06±0.71° and 2.34±2.20°, respectively.

The pose estimation accuracy was evaluated in 10 phantom studies using the layered agarose phantoms with barium. The Euclidean distance between the centroids of the manually annotated, effective treatment volumes in the post-treatment CBCT images and the predicted target center estimated from biplane fluoroscopy was 1.38 ± 0.54 mm. This error can be split up into its mean absolute error components along the direction of ultrasound propagation (0.69 ± 0.58 mm) and the perpendicular components 0.52 ± 0.40 mm in TB and 0.81 ± 0.56 mm in LR. The average estimation bias in these three directions was 0.01 mm (depth), 0.30 mm (TB) and −0.77 mm (LR). The average Euclidean distance of the predicted margin points to the annotated treatment boundary was 1.04 ± 0.22 mm.

The mean absolute error of the predicted radius of the spherical treatment region was 0.51±0.24 mm. A breakdown of the margin and radius errors in the individual directions is shown in Table II. Figure 6 shows examples of the pose estimation results as overlays between fluoroscopic image and digital model in estimated pose as well as overlays between post-treatment CBCT with predicted target and margin positions.

TABLE II.

Transducer Pose Estimation Results

	Depth	TB	LR	Overall
Center Bias	0.0 ± 0.9	0.3 ± 0.6	−0.8 ± 0.6	N/A
Center MAE	0.7 ± 0.6	0.5 ± 0.4	0.8 ± 0.6	1.4 ± 0.5
Margin MAE	0.8 ± 0.5	0.9 ± 0.7	1.4 ± 0.7	1.0 ± 0.7
Radius MAE	0.3 ± 0.4	0.6 ± 0.4	0.6 ± 0.6	0.5 ± 0.5

Errors were measured at the target location (transducer focal point) for individual directions along ultrasound propagation (depth), top-bottom (TB), left-right (LR), and combined mean absolute error (MAE) in mm.

Fig. 6.

Pose estimation results examples (columns 1 to 4): 1) Original x-ray image used for pose estimation, 2) x-ray image with digital model in estimated pose superimposed, 3) post-treatment CBCT slice (sagittal plane) with superimposed predicted center (green) and margin points (yellow), 4) post-treatment CBCT slice (transverse plane) with predicted center and margin points superimposed.

Finally, the automated targeting approach was evaluated in 6 phantom studies. The Euclidean distance between the selected target in the pre-treatment CBCT and the centroid of the manually annotated effective treatment region was 1.64±0.34 mm. This result can be broken down into mean absolute errors in the directional components yielding 0.83±0.65 mm along the direction of ultrasound propagation, 0.46 ± 0.29 mm in TB and 1.05 ± 0.58 mm in LR. Similarly, a directional estimation bias of −0.31 mm (depth), 0.11 mm (TB), and −1.01 mm (LR) was observed. Figure 7 shows example overlays of post-treatment CBCTs and the corresponding selected target center location from the pre-treatment CBCT.

Fig. 7.

Automatic targeting results for 4 different phantom experiments showing sagittal, frontal or transverse cut planes through the post-treatment CBCT with selected target from pre-treatment CBCT superimposed as green dot and manual annotation of reference sphere of the effective treatment zone in blue.

IV. Discussion

This work describes a fully automatic targeting system for non-invasive histotripsy treatment of tumors. The system relies only on CBCT and fluoroscopic imaging and thus allows treatment of patients who are not candidates for ultrasound-guided treatment or whose tumors and surrounding anatomy are better imaged by CBCT. The proposed pose estimation approach uses deep learning techniques to segment transducer features from 2D x-ray images and for the initialization of the transducer model orientation followed by iterative minimization of a cost function. The pose estimation approach can then be used to calibrate the robotic arm to the C-arm coordinate system prior to the procedure, thereby allowing fully automatic targeting using a robotic arm simply by selecting the targeted volume (eg, tumor) on CBCT images. Additionally, the pose estimation approach could be used during the procedure e.g. to verify the final transducer position after automatic targeting.

As an emerging focal cancer therapy, histotripsy has considerable potential. Preclinical studies have shown: 1) a decrease in local tumor progression, increased survival and reduced metastases in murine models [14], [50], 2) precise, accurate, and human-sized ablation zones in porcine models [7]–[9], and 3) increased immunogenicity of tumors after histotripsy and synergy with checkpoint inhibitors [14]. In addition, a limited “first-in-human” phase I clinical trial performed in Spain demonstrated early safety and efficacy for treating hepatic tumors [15]. However, a reliance solely on 2D ultrasound to plan, guide, monitor and assess treatments is a significant barrier to widespread clinical adoption. Poor ultrasound windows caused by fatty liver, obesity, lung, bowel gas, ribs, and cirrhotic liver limit the visibility of many tumors. An inability to adequately visualize and monitor tumors and cavitation using ultrasound as the sole imaging modality could substantially limit the number and location of tumors that can be safely treated.

CT is currently the most widely used imaging guidance technique for existing focal cancer therapies such as percutaneous thermal ablation. CT offers rapid, volumetric imaging with high spatial resolution and adequate soft tissue contrast (especially with intravenous contrast) for identifying the target tumor and surrounding critical structures, treatment planning, assessing probe placement, and confirming complete coverage of the target tumor and margins. Due to the size and configuration of the histotripsy system (robotic arm, degassed water bath as a coupling medium, therapy transducer directly over patient), it is not practical to use conventional CT or MRI for procedural imaging. Transcranial MRI-guided histotripsy is being investigated for neurological applications [51], [52], but the impact of respiratory motion, small bore size and equipment compatibility issues will hinder the development and clinical adoption of MRI-guided histotripsy in the abdomen. C-arms are well-suited for this purpose as they are ubiquitous in virtually every hospital and there is widespread expertise and experience with their use. The histotripsy system fits into C-arms due to their open nature, and the therapy transducer and robotic arm can be positioned to minimize image artifacts and allow the C-arm to rotate freely for CBCT acquisitions. In addition, CBCT offers many of the aforementioned advantages of CT, including excellent visualization of tumors, critical structures, and focal treatment zones [25]–[27]. Since the calibration of the robot to the C-arm can be performed prior to the procedure without patient or staff in the room, no radiation exposure is associated with the fluoroscopic imaging. Generally, the proposed workflow includes two CBCT acquisitions: a pre-treatment CBCT for target localization and a post-treatment CBCT to confirm treatment success. The radiation dose for a single CBCT is approximately 3–10 mSv depending on patient size, manufacturer, and imaging protocol [53].

The deep learning segmentation approach was able to reliably extract distinct transducer features from 2D x-ray images acquired during phantom and animal studies independently of the transducer pose and C-arm orientation. For sparse transducer feature classes such as the outer and inner boundary, gaps between elements, and screws, the recall is considerably lower than the precision, suggesting that the number of false negatives is larger than the false positives. This might in part be caused by overlaps between the features with the large transducer elements resulting in only partial segmentations of these features depending on the transducer orientation. However, this can be compensated for by the large number of feature points used during cost function minimization. While the precision is relatively consistent over all feature types, the significantly lower value for the screws class can be explained by the fact that the digital model used for the manual annotation of the evaluation data does not include all screws due to the limited field of view of the CBCT volume. Thus, reference labels for screws further away from the transducer, which might be visible in some x-ray projection orientations, might be incomplete.

The deep learning-based estimation of azimuth and elevation angle showed average errors of 1.06° and 2.34° , respectively. By itself, this would correspond to an average targeting error of 6.28 mm. However, it provides reliable estimates for initialization of the subsequent iterative minimization of the cost function, which can further improve the accuracy. The linear fits for both angles show slopes close to 1 (1.08 and 1.05) and intercepts close to 0 (−0.14 and −0.10) suggesting reliable prediction of the transducer orientation, which is further supported by the R² values of 98.3% and 99.9%, respectively.

The average pose estimation error in the phantom studies was around 1.4 mm corresponding to approximately 4% to 7% of the typical treatment diameters (2 to 2.5 cm) and is considerably smaller than typical ablative margins in alternative more invasive treatment techniques such as microwave ablation [54]. The observed pose estimation bias primarily in the left to right direction can be attributed to minor differences between the digital model and the physical transducer as well as to deviations of the bubble cloud location from the geometrical focal point due to small manufacturing inaccuracies. In practice the accuracy could be improved by including the known bias into the focal point prediction. This can be seen in the even more accurate prediction of the treatment size (0.5 mm on average), which is independent of the bias.

Similar accuracies and biases were achieved for the fully automatic targeting approach with an average accuracy of 1.6 mm. As for the pose estimation accuracy, this could be further improved by including the pose estimation bias during the calibration of the robotic arm. Additionally, compared to alternative treatment approaches, the proposed system and techniques would provide highly accurate targeting even if the tumor cannot be visualized on ultrasound. The described techniques could also be applied to other non-invasive transcutaneous therapies such as HIFU, where similar targeting problems exist.

The limitations of this study include the lack of respiratory motion during treatment, which would be expected in vivo. While motion could reduce the accuracy of the proposed method, the same problem exists for the current ultrasound guided workflow since no image feedback is used during treatment. Several potential solutions for motion compensation or mitigation have been proposed or are currently under investigation. A study by Longo et al. [55] has shown that respiratory motion generally results in an asymmetric elongation of the prescribed treatment region which can be counteracted by reducing the treatment size along the direction of the respiratory motion. Our group has also previously investigated the possibility of generating patient specific respiratory motion models from two CBCT images acquired at different respiratory states [56]. On the other hand, respiratory motion could be minimized by using jet ventilation [57].

Another limitation is the inability of the proposed method to visualize cavitation in real-time. This generally does not limit the ability to perform the treatment since the current ultrasound-based workflow also does not use real-time feedback during treatment. However, adjusting for acoustic aberrations (step 3 in figure 1) would be more challenging in cases where ultrasound imaging is not possible. Several potential options for cavitation detection and localization have been reported or are under investigation and could be used for aberration correction to increase accuracy in-vivo. This includes the possibility of adding transmit-receive capability to the therapy transducer to detect cavitation emission feedback [58] or using acoustic simulation to predict phase aberration [59]. Future in vivo studies will be conducted to investigate motion compensation and aberration correction for the proposed C-arm based technique.

V. Conclusion

Fluoroscopy-based pose estimations of histotripsy therapy transducers are accurate and precise and can be used to monitor the treatment location on cone beam CT images. Fully automated targeting is possible by calibrating the integrated robotic arm of the histotripsy system and registering it to the cone beam CT coordinate system. While the technique needs further development and validation in animal and human studies, the described C-arm-based targeting method promises to help overcome limitations of current ultrasound guidance methods, including operator dependence and poor target tumor visibility.