Deformable registration of PET/CT and ultrasound for disease-targeted focal prostate brachytherapy

Abstract.

We propose a deformable registration algorithm for prostate-specific membrane antigen (PSMA) PET/CT and transrectal ultrasound (TRUS) fusion. Accurate registration of PSMA PET to intraoperative TRUS will allow physicians to customize dose planning based on the regions involved. The inputs to the registration algorithm are the PET/CT and TRUS volumes as well as the prostate segmentations. PET/CT and TRUS volumes are first rigidly registered by maximizing the overlap between the segmented prostate binary masks. Three-dimensional anatomical landmarks are then automatically extracted from the boundary as well as within the prostate. Then, a deformable registration is performed using a regularized thin plate spline where the landmark localization error is optimized between the extracted landmarks that are in correspondence. The proposed algorithm was evaluated on 25 prostate cancer patients treated with low-dose-rate brachytherapy. We registered the postimplant CT to TRUS using the proposed algorithm and computed target registration errors (TREs) by comparing implanted seed locations. Our approach outperforms state-of-the-art methods, with significantly lower ($\text{mean}\pm \text{standard deviation}$) TRE of $1.96\pm 1.29\mathrm{mm}$ while being computationally efficient (mean computation time of 38 s). The proposed landmark-based PET/CT-TRUS deformable registration algorithm is simple, computationally efficient, and capable of producing quality registration of the prostate boundary as well as the internal gland.

1. Introduction

Prostate cancer is the most prevalent malignancy in men and the second leading cause of cancer death in the US with $\sim \mathrm{174,650}$ new cases and 31,620 deaths estimated in 2019.¹ Currently, whole prostate gland treatment is the standard due to concerns regarding uncertainty and inability to define multiple tumor foci within the gland. However, whole-gland treatment may contribute to an increased risk of toxicity. Due to the toll upon patient quality of life from current therapies, there is now substantial interest throughout the urologic oncology community in utilizing disease-targeted focal therapy to mitigate such toxicities.^2,3 The rationale for focal therapy is based on the recognition that whole-gland treatment, regardless of specific modality, is associated with unacceptable toxicity rates, while concurrently it is also realized that patient morbidity and mortality is due to the progression of major foci of high-grade disease, i.e., the index lesion.⁴

Recent advances in multiparametric MR imaging and, more recently, positron emission tomography (PET) imaging of prostate cancer have increased the potential for accurate identification of disease foci within the prostate. Most ablative treatments for prostate cancer, including brachytherapy and cryotherapy, utilize transrectal ultrasound (TRUS) as the primary modality for image-guidance; however, TRUS does not provide accurate diagnostic information regarding the location of the cancer. The recent advent of PET imaging agents, such as prostate-specific membrane antigen (PSMA), is highly promising in prostate focal therapy.⁵ It has proven to be very specific and sensitive in identifying prostate cancer, including intra- and extra- prostatic tumors.⁶ A combination of PET with intraoperative TRUS would allow for precise focal therapy for prostate cancer. Javitt et al. proposed a planned research initiative involving multimodal image fusion for precise diagnosis, staging, and focal therapy of organ-confined prostate cancer and asserted the necessity of multimodal fusion of PSMA PET/CT with real-time TRUS.⁷

While the first and key step toward successful PET-determined focal brachytherapy is an accurate registration of the preoperative PET/CT and intraoperative TRUS images, there are only a few studies addressing this problem. Such paucity of image fusion using PET/CT is mentioned in a recent literature review on multimodal imaging and focal therapy for prostate cancer.⁸ There are several studies in which multimodal image fusions (MR-TRUS or PET-TRUS) were performed using commercially available software platforms.^9–11 Lopci et al.¹⁰ presented a prospective study of PSMA PET-TRUS fusion to detect prostate cancer where the fusion was performed using BioJet™ (GeoScan). Another study on assessing the clinical value of bimodal (choline-PET/mpMRI) and trimodal (PET/mpMRI/TRUS) image fusions for image-guided prostate biopsy was performed where image fusions were carried out using Virtual Navigator (Esaote, Genoa, Italy).¹¹

Bowes et al.¹² presented a comparative study between TRUS-CT and MR-CT fusion for postimplant dosimetry of permanent prostate brachytherapy where they manually co-registered TRUS and CT based on urethral position. This fusion is manual and does not take into account the prostate deformation caused by the TRUS probe. Fei et al.^13–16 performed a retrospective study on multimodal image fusion targeted biopsy and proposed a PET-directed, TRUS-guided biopsy system including an automatic prostate segmentation using wavelet transform and a nonrigid registration. The nonrigid registration was performed by combining point-based registration and volume matching methods, which was then validated by registering pre- and post- biopsy TRUS images.^14,16 Zettinig et al.¹⁷ proposed a system for PSMA-PET/MRI and TRUS fusion for prostate biopsy. This method utilized a coherent point drift algorithm and thin-plate spline (TPS) to perform surface-based deformable registration.

Existing multimodal image registration methods for prostate cancer mostly focus on MR-ultrasound registration as MRI is widely used as a gold standard for prostate biopsy. Since MR and ultrasound do not share much correlative image information, conventional intensity-based registration cannot achieve sufficient registration accuracy. Therefore, most existing MR-ultrasound registration approaches use additional information such as anatomical landmarks to guide the registration process and can be used for the PET/CT-ultrasound registration problem without much modification. Existing registration algorithms can be broadly categorized as intensity-based,¹⁸ surface-based,^16,19 and biomechanical model-based^20–26 approaches.

Intensity-based registration methods produce less accurate results due to poor intensity correlation between MR and TRUS images. Sun et al.¹⁸ proposed an image-based registration of MR-TRUS using a duality-based convex optimization with a multichannel modality independent neighborhood descriptor as a similarity measure. However, this method may produce inaccurate inner-gland registration due to the lack of meaningful intrinsic features. We have previously proposed a PET/CT-TRUS deformable registration algorithm based on a structural descriptor map (SDM) computed by solving a Laplacian equation based on prostate segmentations.²⁷ The computed SDM consists of equipotential surfaces within the prostate that describe smooth transitions from the midline to the boundary of the prostate while preserving the prostate shape and geometry. This SDM-based deformable registration produced accurate registration results with an average target registration error (TRE) of 2.25 mm when tested on five prostate cancer patients’ data.

Surface-based registration methods require segmentations of the prostate glands in both modalities. These methods are able to accurately align the prostate boundary but do not guarantee accurate alignment of internal structures. A feature-based nonrigid registration was proposed where anatomical landmarks are manually selected as salient features and then the distances of corresponding feature points are minimized and, in addition, the overlap ratio of the prostate volume are maximized by a volume matching method.¹⁶ In another study, prostate shapes were represented using three-dimensional (3-D) shape-context descriptors and point matching between the MRI and TRUS was performed¹⁹ using the Hungarian algorithm.²⁸ Validation was performed using implanted brachytherapy seeds as well as external beam radiotherapy fiducials, resulting in an average TRE of 2.56 mm.

Biomechanical model-based registration approaches showed the best performance among existing methods in which deformations are constrained during the registration based on learned physical motion models and tissue properties. Biomechanical tissue properties are modeled using finite element methods (FEM) and incorporated into statistical motion models as a priori knowledge.^22–24 Wang et al.²⁵ proposed a personalized statistical deformable model with FEM-based analysis to simulate the prostate organ deformation where an additional image acquisition, such as ultrasound elastography, was required to measure subject-specific prostate biomechanical parameters. Hu et al.²⁶ proposed a patient-specific statistical shape model for predicting prostate deformation in MR-TRUS registration where organ shape and deformation were derived from a biomechanical model. The TRE for this approach was 2.40 mm using manually selected landmarks. However, these approaches are computationally expensive and often require additional patient-specific training to achieve the best performance, which is cumbersome and may not be feasible for intraoperative use.

Similar surface-based or biomechanical model-based registration approaches, as discussed above, can be used for PET/CT-TRUS registration in which we can use CT images as a surrogate to fuse PET. However, existing algorithms need improvements in terms of registration accuracy and computational efficiency for intraoperative use. In this paper, we propose a deformable image registration algorithm for preoperative PET/CT and intraoperative TRUS image fusion. Our PET/CT-TRUS registration algorithm uses prostate contours drawn on both CT (of PET/CT) and TRUS images, similar to the surface-based approaches described above. To guarantee accurate registration inside the prostate gland, our approach automatically extracts landmarks from both the prostate boundary and inside the gland. A regularized 3-D TPS-based deformable registration is then performed using the extracted landmarks. Our approach is computationally fast and produces accurate registration results for both the prostate boundary and internal gland. While the underlying algorithm of TPS-based image registration has already been presented in the literature, our method attempts to employ the algorithm with necessary modifications in conjunction with an automatic landmark extraction technique to the prostate anatomy to perform accurate and efficient PET/CT-TRUS registration.

The rest of the paper is organized as follows. In Sec. 2, we describe the details of our PET/CT-TRUS registration algorithm and workflow. Registration performance evaluation and numerical results based on 25 prostate cancer patients’ data are presented in Sec. 3. In Sec. 4, we compare the performance of our method with other existing methods and future improvements to our registration process. Finally, the paper concludes in Sec. 5.

2. Methods

The proposed PET/CT-TRUS registration workflow is shown in Fig. 1. The registration process starts with a preprocessing step followed by rigid registration and, finally, deformable registration. In the preprocessing step, the prostate gland needs to be contoured in both CT and TRUS images. The PET/CT images are segmented prior to the brachytherapy procedure, and the TRUS images are segmented during the planning phase of the brachytherapy procedure as a part of a routine clinical procedure. These segmentations are performed slice-by-slice by a physician. The segmented contours are smoothed by a recursive Gaussian filter to reduce stair-like effects caused by abrupt interslice transitions.

Fig. 1.

PET/CT-TRUS registration workflow.

2.1. Rigid Registration

Rigid registration is first initialized by aligning the centers of mass between the CT and TRUS prostate masks followed by rotating the CT prostate mask by an estimated TRUS probe angle to the cranial-caudal axis in the sagittal plane of the CT. This rotation adjustment is desired as the patient position between CT (supine) and TRUS (high lithotomy) are significantly different. We rotate by 15 deg, which is the average rotation observed in our patient data, and is similar to the measurements in other studies.²⁶

Following the initial transformation, the final rigid transformation is computed by maximizing the spatial overlap between the prostate segmentations, measured by a kappa statistics-based similarity metric defined as

$$K_p=2\times \frac{|S_{\mathrm{CT}}\cap S_{\mathrm{TRUS}}|}{|S_{\mathrm{CT}}|+|S_{\mathrm{TRUS}}|},$$ (1)

where $S_{\mathrm{CT}}$ and $S_{\mathrm{TRUS}}$ are prostate segmentations in the CT and TRUS, respectively. $K_p$ is maximized using a stochastic gradient descent optimizer²⁹ to perform the rigid registration.

2.2. Deformable Registration

Landmark-based registration is a commonly used technique in deformable image registration. Its accuracy mostly depends on how the landmarks are extracted. For MR-TRUS or CT-TRUS registration problems, the prostate boundary is commonly used as the landmark as it can be contoured more reliably on both modalities than other anatomical landmarks. However, deformable registration relying only on the prostate boundary cannot produce accurate deformable registration especially inside the gland. In addition to that, manual selection of landmarks inside the gland may be error-prone due to the lack of sufficient soft tissue contrast in CT and TRUS images. Kaplan et al.³⁰ presented a two-dimensional (2-D) rigid registration of MR-TRUS using a set of manually selected fiducials on the boundary of the prostate. Similar landmark-based registration methods can be found where anatomical landmarks are also chosen manually.^31–33

Unlike these approaches that use manually extracted boundary and/or internal gland landmarks, our deformable registration method uses a set of matching landmarks that are automatically extracted from the prostate boundary as well as the internal gland in both CT and TRUS prostate masks. A 3-D TPS-based registration is then computed using the extracted landmarks as control points.

2.2.1. Automatic landmark extraction

To achieve anatomically accurate registration, we extract landmarks from the prostate boundary as well as inside the gland along eight directions that are equally distributed around the base-apex axis as shown in Fig. 2. In the proposed method, eight equidistant boundary landmarks along the prostate circumference determine these eight directions. For each direction, a radial line is computed from the center of the prostate to the boundary landmark. These radial lines are used to extract eight midgland landmarks that correspond to the midpoints between the prostate center and the boundary landmarks.

Fig. 2.

Landmark extraction. An example axial slice with 25 extracted landmarks. In each axial slice, 1 center of mass point, 8 midgland points (red), and 16 prostate boundary points (blue) are extracted. (a) Proposed equidistance method. (b) Equiangular landmark extraction. (c) Extracted 3-D landmark point cloud.

We have previously proposed an equiangular landmark extraction approach where landmarks lie on equiangular radial lines.^34,35 While the equiangular approach does not take into account the prostate deformation, the proposed equidistance landmark extraction extracts boundary landmarks that are uniformly distributed along the prostate circumference. Therefore, these extracted landmarks can better capture prostate deformation and provide accurate registration at the boundary of the prostate.

For the prostate boundary, we add eight additional points between the initially selected eight points in the eight directions. For each 2-D axial slice, a total of 25 landmarks consisting of one center of mass point, 16 prostate boundary points, and 8 midgland points are extracted. The extracted landmarks from all of the slices form a 3-D point cloud as shown in Fig. 2(c). A total of 250 landmarks (25 landmarks per axial slice and 10 slices across the volume) were extracted from each volume.

Since there are greater uncertainties in the prostate contours near the base and apex regions than the midgland (especially in TRUS images), we excluded the first and last 5% of the slices along the base-apex direction. Unlike uniformly distributed landmarks or Euclidean distance-based landmarks, the extracted 3-D landmark point cloud can accurately capture prostate shape and geometric changes in a consistent way in TRUS and CT. Intra-gland landmarks within the prostate ensure a smooth transition from the prostate base-apex midline to the boundary, thus producing physically realistic registration inside the gland.

The detailed steps of our landmark extraction process are listed as Algorithm 1.

Algorithm 1.

Pseudocode of automatic landmark extraction.

For slice 1 to $N$

1. Compute the prostate center of mass, $C_i$.

2. Find eight equidistant boundary landmarks along the prostate circumference (boundary landmarks $B{W}_i$).

3. Find eight radial lines, starting from the center toward the boundary landmarks.

4. For each radial line

Find the mid-point between $C_i$ and $B{W}_i$ (intra-gland landmarks).

5. Find eight midpoints between adjacent $B{W}_i$’s (additional boundary landmarks).

2.2.2. Computing deformation using TPS

TPS is a commonly used technique to perform a smooth deformable registration based on a set of corresponding landmarks. It was originally proposed by Bookstein who used TPS interpolation between two sets of points in 2-D images.³⁶ Rohr et al.³⁷ then proposed an approximating TPS where an anisotropic localization error was introduced. TPS and its variants have been widely used in landmark-based deformable registration.^{32,33,38–40}

Given a set of fixed image landmarks ${\mathbf{p}}_i$ and moving image landmarks ${\mathbf{q}}_i$, where $i=1,2,\dots ,n$, $n$ is the total number of extracted landmarks, and images are of dimension $d$, we obtain a TPS transformation $\mathcal{T}$: ${\mathbb{R}}^d\to {\mathbb{R}}^d$ defined as

$$\mathcal{T}(\mathbf{x})=\sum _{v=1}^d{a}_v{\varphi }_v(\mathbf{x})+\sum _{i=1}^n{w}_{i}U({\mathbf{p}}_i-\mathbf{x}),$$ (2)

where ${\varphi }_v(\mathbf{x})$ represents a set of polynomials on ${\mathbb{R}}^d$, $U(r)$ is a underlying radial basis function defined as $U(r)=r^2\log r^2$, $a_v$ are the 12 affine coefficients of the transformation, and $w_i$ are the TPS weight coefficients.

TPS mapping between the moving and fixed landmarks can be computed by minimizing the bending energy $E_{\mathrm{TPS}}$ over ${\mathbb{R}}^3$, defined as

$$E_{\mathrm{TPS}}={\iiint }_{R^3}[{\left(\frac{{\partial }^2\mathcal{T}}{\partial x^2}\right)}^2+{\left(\frac{{\partial }^2\mathcal{T}}{\partial y^2}\right)}^2+{\left(\frac{{\partial }^2\mathcal{T}}{\partial z^2}\right)}^2+2{\left(\frac{{\partial }^2\mathcal{T}}{\partial xy}\right)}^2+2{\left(\frac{{\partial }^2\mathcal{T}}{\partial yz}\right)}^2+2{\left(\frac{{\partial }^2\mathcal{T}}{\partial xz}\right)}^2\mathrm{d}x\mathrm{d}y\mathrm{d}z.$$ (3)

We take into account the localization error between landmarks by incorporating a quadratic approximation term as an additional constraint.^19,33,37 The final cost function to be minimized is

$$\frac{1}{n}\sum _{i=1}^n\frac{{|q_i-\mathcal{T}(p_i)|}^2}{{\sigma }_i^2}+\lambda E_{\mathrm{TPS}}=0,$$ (4)

where ${\sigma }_i^2$ are variances representing the landmark localization error of corresponding landmarks where ${\sigma }_i^2={\sigma }_{i,p}^2+{\sigma }_{i,q}^2$ and $\lambda$ is a regularization parameter that is empirically set to a small value of ${10}^{-4}$ so that the TPS is well adjusted to local deformations.

The coefficients of Eq. (2) are computed by solving the following linear system of equations:

$$\begin{gathered} (\mathbf{K}+n\lambda {\mathbf{W}}^{-1})\mathbf{w}+\mathbf{Pa}=\mathbf{q} \\ {\mathbf{P}}^T\mathbf{w}=0, \end{gathered}$$ (5)

where elements of $\mathbf{K}$ are $K_{ij}=U(\Vert p_i-p_j\Vert )$ and the $i$’th row of $\mathbf{P}$ is $(1,{\mathbf{p}}_i)$, $\mathbf{q}$ and $\mathbf{w}$ are column vectors from $q_i$ and $w_i$, respectively, $\mathbf{a}$ is a column vector consisting of $a_v$ [see Eq. (2)], and $\mathbf{W}$ is the diagonal matrix of ${\sigma }_i^2$.

3. Experiments and Results

Quantitative evaluation of PET/CT-TRUS registration performance is challenging as it is difficult to obtain ground truth and select anatomical landmarks inside the prostate for TRE computation. Therefore, existing algorithms validated their registration performance based on image or prostate mask similarities, prostate boundary distances, or using a handful of handpicked anatomical landmarks.^18,22,24,25 However, none of these approaches can thoroughly evaluate the registration accuracy inside the prostate. In this paper, for a quantitative assessment of the proposed method, we used implanted radioactive seeds.

We evaluated the proposed registration algorithm on 25 prostate cancer patient datasets treated with low-dose-rate permanent brachytherapy (61 to 107 Pd-103 seeds implanted to the whole gland) at our institution. Each patient had intraoperative TRUS and six x-ray images acquired within a 20-deg cone angle around the anterior-posterior axis of the patient using a mobile C-arm at the end of the seed implantation as well as a postoperative CT taken 1 day after the implantation (day 1) as a routine clinical procedure. Datasets were obtained under a protocol approved by the institutional review board. Transverse TRUS images were acquired at 1 mm intervals with an in-plane pixel size of $0.18\times 0.18{\mathrm{mm}}^2$ and resulting image size of $512\times 444\mathrm{pixels}$. The CT had $512\times 512\times [28-61]\text{voxels}$ with a voxel size of $0.5\times 0.5\times 3.0{\mathrm{mm}}^3$. Note that we used end-of-treatment TRUS/x-ray images as they show implanted brachytherapy seeds that can be used for quantitative evaluation of the registration accuracy. Since planning TRUS images are typically acquired at lower interslice intervals, e.g., 5 mm instead of 1 mm, we used the prostate contour drawn on the planning TRUS that is also aligned to the postimplant TRUS as they were acquired in the same session while the patient pose remained the same. Notice that the prostate contour was updated when the physician observed anatomical changes due to the prostate edema, and the finally available contours were used in our study. Therefore, the registration was validated on a realistic scenario, i.e., registering preoperative PET/CT to intraoperative planning TRUS, while we have means to measure registration errors across the entire prostate gland.

The intraoperative seed locations were computed from the x-ray images and then registered to the TRUS images using our previously developed intraoperative registration of ultrasound and fluoroscopy (iRUF) system.^41–43 The CT was contoured by the same radiation oncologist who contoured the TRUS images and performed the brachytherapy. Although the implanted seeds generated artifacts, the presence of seeds did not affect the quality of CT prostate contouring as they are small and do not globally impact the prostate gland region. The implanted seeds were semiautomatically segmented on the day 1 CT using Variseed brachytherapy treatment planning software (Varian Medical Systems, Palo Alto, California) followed by manual adjustment by an expert medical physicist for postimplant dosimetry. Once the CT was registered to TRUS by the proposed algorithm, we transformed the segmented CT seeds into TRUS space. TREs were then computed by computing Euclidean distances between the TRUS seeds (computed by iRUF) and the transformed CT seeds. Since the implanted seeds were well distributed within the prostate (unlike hand-picked anatomical landmarks), the computed TREs would show the overall performance of the registration across the entire gland. The registration algorithm was implemented using C++ on a workstation with an Intel 2.4 GHz Xeon processor and 32 GB memory.

Example registration results for three cases among 25 test cases are shown in Figs. 3 and 4. Intermediate steps of the proposed deformable registration after the initial rigid alignment are shown in Fig. 3. The deformation field, computed from the TPS registration, was applied to both CT volume and CT segmented prostate mask, as shown in Fig. 3(e). Note that the implanted seeds can be seen (bright white spots) in the registered CT, as shown in Figs. 3(d) and 3(e).

Fig. 3.

Intermediate steps of landmark extraction and deformable registration after the initial rigid registration. Each row shows a different patient. (a) Extracted 3-D landmarks from TRUS (cyan) and CT (red). (b) Extracted landmarks overlaid on an axial slice of the rigidly registered CT (purple) and TRUS (green) masks. (c) TRUS image with prostate contour and extracted landmarks. (d) CT image with prostate contour and extracted landmarks. (e) Registered CT image and CT prostate contour (purple) overlaid with TRUS prostate contour (green).

Fig. 4.

Examples of registration performance validation using implanted seeds. Each row represents a different patient. (a) 3-D surface rendering of TRUS prostate segmentation with implanted seeds (red triangle). TRUS seeds (red triangle) and registered CT seeds (green dots) are superimposed on the (b) registered CT image and (c) TRUS image. TRUS prostate contour (green) and the registered CT prostate contour (purple) are also overlaid.

Figure 4 shows examples of the registered CT and TRUS images overlaid with both TRUS and registered CT seeds for qualitative assessment. The surface renderings of the TRUS volumes are shown in Fig. 4(a) to visualize the distribution of the implanted seeds. Transformed CT seeds by TPS registration along with the TRUS seeds superimposed on the deformed CT and TRUS images are shown in Figs. 4(b) and 4(c), respectively. The bright spots in the registered CT aligned with the TRUS seeds (red triangles) in Fig. 4(b), and, similarly, the transformed CT seeds (green dots) aligned with the bright spots in the TRUS image as shown in Fig. 4(c), qualitatively demonstrating the registration quality. Note that the green dots and red triangles are the centroids of the registered CT and TRUS seeds; therefore, some of them do not appear in the slices shown in Fig. 4.

The resulting TREs for all 25 cases are reported in Table 1. The average TRE of the proposed method with regularized TPS is 1.96 mm with a standard deviation (SD) of 1.29 mm. The average number of implanted seeds is 89, which should be sufficient for properly assessing the accuracy of the registration.

Table 1.

A comparison of TREs among the proposed algorithm, equiangular landmark-based registration method, and distance-map-based b-spline registration for 25 cases. All values are $\text{mean}\pm \mathrm{SD}$ (max) in mm.

Case	Number of seeds	Proposed method	Equiangular landmark-based registration34	Distance-map-based b-spline registration44,45
1	101	$1.51\pm 1.25(6.61)$	$1.50\pm 1.02(4.74)$	$2.32\pm 1.50(7.30)$
2	66	$2.03\pm 1.18(4.52)$	$2.00\pm 1.00(4.25)$	$2.34\pm 1.30(5.23)$
3	78	$2.30\pm 1.30(7.35)$	$2.33\pm 1.56(6.41)$	$2.56\pm 1.62(7.13)$
4	78	$1.64\pm 0.95(6.44)$	$1.68\pm 0.85(5.86)$	$2.86\pm 1.33(7.59)$
5	76	$2.24\pm 1.20(5.71)$	$2.30\pm 1.16(7.26)$	$2.70\pm 1.42(6.04)$
6	104	$1.82\pm 1.25(6.81)$	$1.80\pm 1.22(5.83)$	$1.88\pm 1.10(5.32)$
7	75	$1.81\pm 1.95(7.20)$	$1.92\pm 1.02(4.95)$	$2.68\pm 1.59(5.91)$
8	78	$1.92\pm 1.40(6.30)$	$2.17\pm 1.94(7.60)$	$2.74\pm 1.70(6.32)$
9	61	$2.02\pm 0.95(5.80)$	$2.44\pm 1.42(6.53)$	$2.94\pm 1.49(6.53)$
10	72	$2.35\pm 0.98(4.20)$	$2.47\pm 1.89(8.21)$	$3.14\pm 1.80(7.14)$
11	89	$1.59\pm 1.42(7.83)$	$1.48\pm 1.04(4.45)$	$2.60\pm 1.20(4.56)$
12	74	$2.26\pm 1.50(7.86)$	$2.17\pm 1.67(6.04)$	$2.92\pm 1.70(8.52)$
13	103	$1.92\pm 1.03(5.98)$	$1.74\pm 0.99(5.73)$	$2.78\pm 1.85(4.23)$
14	97	$2.17\pm 1.47(7.45)$	$1.92\pm 1.05(4.83)$	$3.10\pm 1.10(7.36)$
15	107	$1.63\pm 1.28(5.25)$	$1.86\pm 0.97(3.84)$	$3.74\pm 2.04(7.18)$
16	85	$1.71\pm 1.07(7.40)$	$1.82\pm 1.01(5.51)$	$2.48\pm 1.27(6.46)$
17	78	$2.05\pm 1.21(4.82)$	$2.33\pm 1.31(7.68)$	$2.41\pm 1.76(7.77)$
18	82	$1.36\pm 1.31(3.24)$	$1.23\pm 1.16(5.93)$	$3.38\pm 1.85(5.65)$
19	93	$2.16\pm 1.45(7.53)$	$2.21\pm 1.38(5.38)$	$2.77\pm 1.92(6.30)$
20	112	$2.05\pm 1.42(4.84)$	$2.02\pm 1.15(5.43)$	$3.20\pm 1.71(8.94)$
21	105	$2.15\pm 1.27(3.99)$	$1.92\pm 0.94(4.32)$	$2.79\pm 1.80(8.37)$
22	114	$1.41\pm 1.47(4.47)$	$1.88\pm 1.56(6.45)$	$2.78\pm 1.86(9.47)$
23	92	$2.45\pm 1.06(5.20)$	$2.33\pm 1.73(7.84)$	$3.12\pm 2.07(8.50)$
24	110	$2.23\pm 1.41(6.88)$	$2.12\pm 1.56(6.46)$	$2.87\pm 2.50(8.00)$
25	92	$2.10\pm 1.36(5.39)$	$2.19\pm 1.32(6.81)$	$4.02\pm 2.48(7.56)$
Average	89	$1.96\pm 1.29(7.86)$	$1.99\pm 1.28(8.21)$	$2.84\pm 1.68(9.47)$

We compared our proposed method with the equiangular landmark-based registration method that we have previously developed.³⁴ We also compared our proposed method with a widely used distance-map-based b-spline registration algorithm that is available as a module in the 3-D slicer.^44,45 This approach uses signed distance maps of the prostate contours and a b-spline free-form deformation model based on an isotropic grid of six control points. We registered CT and TRUS volumes of the same patient data used for evaluating our method and computed TREs using the same implanted seeds. The registration performance comparison is shown in Table 1 and graphically shown in Fig. 5.

Fig. 5.

Comparison of registration performance between the proposed method and a distance-map-based b-spline registration. Error bars show standard deviation.

The accuracy of the proposed deformable registration may vary depending on the number of landmarks. To determine an optimal number of landmarks, we compared the registration performance in terms of TREs and computation time by varying the number of landmarks. As expected, using more landmarks leads to improved registration accuracy at the cost of increased computation time as shown in Fig. 6. We determined 250 landmarks for the entire prostate gland as the optimal number to use, for the registration performance does not improve much when more than 250 landmarks are used. When using 250 landmarks, the whole registration process including the initial rigid registration, landmark extraction, and deformable registration takes 38 s on average.

Fig. 6.

Performance evaluation in terms of (a) TRE and (b) computation time with increasing number of landmarks.

We also experimented on registration accuracies based on the location of extracted landmarks (boundary versus internal). We observed that internal landmarks significantly improve registration accuracy. Table 2 shows the registration accuracy comparison of the proposed method based on different combinations of boundary and internal landmarks. In all of these experiments, for every slice, a center of mass point was also included along with the boundary and internal landmarks.

Table 2.

Registration performance ($\text{mean}\pm \mathrm{SD}$ TRE) based on different combinations of boundary (BP) and internal landmark points (IP). A center of mass point was included in all cases.

Landmarks on each slice	8 BP	8 BP and 4 IP	8 BP and 8 IP	16 BP and 8 IP
TRE (mm)	$2.90\pm 1.50$	$2.36\pm 1.58$	$2.08\pm 1.35$	$1.96\pm 1.29$

The proposed registration relies on the prostate segmentation, and manual segmentation of CT images is time-consuming. To assess the sensitivity of the proposed algorithm on contouring variability and demonstrate the feasibility of auto-segmentation, we repeated the registration experiment using automatic segmentations of CT by convolutional neural network (CNN). We used 3-D U-Net, which has been widely used in many medical image segmentation problems with very promising results.^46–49 We trained the 3-D U-Net using 336 patients’ CT data (excluding the patients used in our registration validation) who were treated by external beam radiation therapy or brachytherapy at Johns Hopkins Hospital. The trained model was tested on 15 independent patients’ CT data (a subset of the 25 patients used for our registration validation), showing a dice similarity score of 0.86 on average in comparison with the expert’s manual segmentations. An example segmentation result is shown in Fig. 7. We then recomputed the registration for these patients; the registration performance comparison between manual and automatic segmentations are reported in Table 3. The average TRE using autosegmentations (${\mathrm{TRE}}_{\text{auto}}$) was 1.93 mm, which is similar to that with manual segmentation (${\mathrm{TRE}}_{\text{manual}}$). This experiment demonstrates the robustness of the proposed method on potential variability in the prostate segmentation and also the feasibility of using automatic segmentation.

Fig. 7.

An example of automatic segmentation. Blue contours are ground-truth and the shaded regions are automatic segmentations. (a) Axial, (b) sagittal, and (c) coronal views.

Table 3.

Segmentation and registration accuracy using manual and automatic segmentation.

Case	Dice	TREmanual	TREauto
1	0.91	$1.51\pm 1.25(6.61)$	$1.57\pm 1.30(4.24)$
2	0.82	$2.03\pm 1.18(4.52)$	$1.93\pm 1.20(6.22)$
3	0.89	$2.30\pm 1.30(7.35)$	$2.15\pm 0.98(6.06)$
4	0.88	$1.64\pm 0.95(6.44)$	$1.65\pm 1.27(5.31)$
5	0.89	$2.24\pm 1.20(5.71)$	$2.11\pm 1.01(6.43)$
6	0.81	$1.82\pm 1.25(6.81)$	$1.85\pm 1.16(3.46)$
7	0.87	$1.81\pm 1.95(7.20)$	$1.76\pm 1.28(4.54)$
8	0.83	$1.92\pm 1.40(6.30)$	$1.95\pm 1.22(4.16)$
9	0.80	$2.02\pm 0.95(5.80)$	$1.91\pm 1.07(4.40)$
10	0.90	$2.35\pm 0.98(4.20)$	$2.26\pm 1.13(5.66)$
11	0.87	$1.59\pm 1.42(7.83)$	$1.65\pm 1.19(6.17)$
12	0.86	$2.26\pm 1.50(7.86)$	$2.21\pm 1.20(6.64)$
13	0.84	$1.92\pm 1.03(5.98)$	$1.90\pm 1.19(4.62)$
14	0.88	$2.17\pm 1.47(7.45)$	$2.16\pm 1.22(7.26)$
15	0.87	$1.63\pm 1.28(5.25)$	$1.90\pm 0.99(6.27)$
Average	0.86	$1.95\pm 1.27(7.86)$	$1.93\pm 1.16(7.26)$

4. Discussion

In our study, the registration performance was evaluated using radioactive seeds implanted during the brachytherapy procedure instead of using a set of handpicked anatomical landmarks as used in prior studies.^18,22,24,25 Handpicked landmarks may not be consistent between two image modalities and are not well-distributed within the gland; thus, they cannot capture the overall registration accuracy. On the other hand, the implanted seeds are uniformly distributed across the prostate gland, thus enabling more accurate assessment of registration accuracy for both near-boundary regions and the internal gland. Similar validation using implanted seeds was carried out by Mayer et al. on 10 patients where the reported average TRE was 2.56 mm.¹⁹ Another recent study on the feasibility of focal prostate brachytherapy using MRI and TRUS deformable registration reported an average TRE of $2.56\pm 1.32\mathrm{mm}$.⁵⁰ We performed an extensive validation using 25 patient datasets and the average TRE was 1.96 mm, which is within the clinically acceptable range and is superior to the results reported in existing studies.^{19,22,33,50–52} The accuracy of the proposed method using equidistance landmark extraction was higher compared with the equiangular landmark-based registration.³⁵ We also compared the accuracy of the proposed method with a widely used distance-map-based registration method, and it showed superior performance. The registration accuracy of the proposed method was comparable to state-of-the-art biomechanical-model-based approaches^20,22,26 while being computationally more efficient.

In recent years, machine learning-based methods, particularly deep neural networks, have been employed to perform challenging multimodal image registration tasks using acquired knowledge from a training population.^53–58 A similarity metric for multimodal image registration of MR-TRUS volumes was learned using CNN where the average TRE was reported as 3.86 mm.⁵⁶ CNNs in conjunction with generative adversarial networks were used to perform MR-TRUS rigid registration where an average TRE of 3.48 mm was achieved.⁵⁷ Dense displacement fields of multimodal image registration were predicted using CNN where voxel-level transformations were inferred from higher-level anatomical labels.^58,59 This weakly supervised method produces a MR-TRUS registration of the prostate gland yielding a median TRE of 3.6 mm. Deep learning-based methods may overcome limitations of classical pairwise registration such as lack of reliable image similarity or automatic landmark features. However, in most of these cases, they could not achieve the desired level of accuracy needed for focal therapy. Our proposed method, which uses automatically extracted, physically consistent landmarks, achieves significantly lower TRE than these machine learning-based registrations.

One limitation of our proposed algorithm is that it requires segmentations of the prostate in both modalities, and the registration accuracy depends on the quality of the segmentation. Our on-going research includes automatic segmentation of the prostate in both TRUS and CT to fully automate the registration process and avoid the need for time-consuming manual segmentation. We have presented our preliminary results using CNN-based automatic segmentation of the prostate in CT with promising results. The comparable registration performance using automatic segmentation to one with the expert’s manual segmentation shows the feasibility of automating the whole registration process and proves the robustness of our algorithm on segmentation variability. Automatic segmentation of the prostate in TRUS using a deep learning approach is currently under development.

The validation of the proposed registration method involves acquiring implanted seed locations from post-implant images. Seed locations in TRUS images were obtained by reconstructing 3-D seed locations using x-rays and registering them to the TRUS space using the iRUF system. This process has been previously validated, producing an average registration error of less than 2 mm.⁴¹ Although there is uncertainty ($<2\mathrm{mm}$), to our knowledge, this is the best way to establish dense ground truth landmarks across the gland given that there is no other reliable way to identify internal gland anatomical landmarks visible in both TRUS and CT. These dense ground truth landmarks inside the gland help us properly assess the registration accuracy inside the gland, which is crucial for PET/CT-TRUS registration in focal therapy.

Variability in prostate contouring might adversely affect the registration performance, even with our overall registration error of 1.96 mm, which is superior to the existing approaches. Furthermore, we applied a deep-learning-based autosegmentation of the CT prostate to assess the impact of segmentation variability on the registration performance and demonstrated that the registration performance is still comparable to that with manual segmentation. These series of validations based on extensive ground truth establishment (compared with a few handpicked landmarks on the prostate boundary or internal gland used in previous studies), comparison with existing methods, and both manual and automatic segmentations prove that the proposed method can achieve better performance than any existing methods.

Although we focused on PET/CT-TRUS fusion in this paper, the proposed method can be used for MR-TRUS registration for both prostate biopsy and MRI-based focal brachytherapy without any major modification. Since prostate segmentation in MRI is more reliable than in CT, MR-TRUS registration accuracy may be comparable to or better than that of the CT-TRUS registration presented in this paper.

5. Conclusions

This paper proposes a 3-D deformable PET/CT-TRUS registration that allows for the incorporation of PET imaging into a TRUS-guided focal prostate treatment. PET imaging can assist in the identification of intra- and extra- prostatic tumors, thus allowing us to perform disease-targeted focal therapy instead of whole-gland treatment if accurately fused with intraoperative TRUS images. The proposed landmark-based TPS deformable registration algorithm showed superior registration accuracy and improved computational efficiency compared with existing methods. The proposed algorithm is simple and fully automatic except for prostate contouring. With an average registration error of less than 2 mm and an average computation time of 38 s, this method can be seamlessly integrated into the current prostate brachytherapy procedure without adding any additional burden while providing critical information for focal therapy.

Biographies

Sharmin Sultana is a postdoctoral researcher in the Department of Radiation and Oncology at Johns Hopkins University. She received her BS and MS degrees in computer science and engineering from the University of Dhaka in 2009 and 2011, respectively, and her PhD in computational modeling and simulation engineering from Old Dominion University in 2017. Her current research interests include medical image analysis, computer vision, and deep learning.

Daniel Y. Song serves as an associate professor in the Department of Radiation Oncology at Johns Hopkins University. His research focus is on technological innovations for improving the practice of prostate brachytherapy, as well as the conduct of clinical trials in innovative methods of radiotherapy for prostate cancer and other genitourinary malignancies. He performed some of the original research testing the feasibility of hydrogel spacers and establishing their benefit in reducing dose to the rectum.

Junghoon Lee is an associate professor in the Department of Radiation Oncology at Johns Hopkins University. He received his BS degree in electrical engineering and MS degree in biomedical engineering in 1997 and 1999, respectively, from Seoul National University, Republic of Korea, and his PhD in electrical and computer engineering from Purdue University in 2006. He is leading Medical Image Computing and Analysis Lab, and his research interests are in image processing and computer vision with application to medical imaging problems.

Disclosures

No conflicts of interest to report.