Multi-Scale Statistical Deformation Based Co-registration of Prostate MRI and Post-surgical Whole Mount Histopathology

Abstract

Background:

Accurate delineations of regions of interest (ROIs) on multi-parametric magnetic resonance imaging (mpMRI) are crucial for development of automated, machine learning-based prostate cancer (PCa) detection and segmentation models. However, manual ROI delineations are labor-intensive and susceptible to inter-reader variability. Histopathology images from radical prostatectomy (RP) represent the “gold standard” in terms of the delineation of disease extents, e.g., PCa, prostatitis, and benign prostatic hyperplasia (BPH). Co-registering digitized histopathology images onto pre-operative mpMRI enables automated mapping of the ground truth disease extents onto mpMRI, thus enabling the development of machine learning tools for PCa detection and risk stratification. Still, MRI-histopathology co-registration is challenging due to various artifacts and large deformation between in vivo MRI and ex vivo whole-mount histopathology images (WMHs). Furthermore, the artifacts on WMHs, such as tissue loss, may introduce unrealistic deformation during co-registration.

Purpose:

This study presents a new registration pipeline, MSERgSDM, a multi-scale feature-based registration (MSERg) with a statistical deformation (SDM) constraint, which aims to improve accuracy of MRI-histopathology co-registration.

Methods:

In this study, we collected 85 pairs of MRI and WMHs from 48 patients across three cohorts. Cohort 1 (D₁), comprised of a unique set of 3D printed mold data from six patients, facilitated the generation of ground truth deformations between ex vivo WMHs and in vivo MRI. The other two clinically acquired cohorts (D₂ and D₃) included 42 patients. Affine and nonrigid registrations were employed to minimize the deformation between ex vivo WMH and ex vivo T2-weighted MRI (T2WI) in D₁. Subsequently, ground truth deformation between in vivo T2WI and ex vivo WMH was approximated as the deformation between in vivo T2WI and ex vivo T2WI. In D₂ and D₃, the prostate anatomical annotations, e.g., tumor and urethra, were made by a pathologist and a radiologist in collaboration. These annotations included ROI boundary contours and landmark points. Before applying the registration, manual corrections were made for flipping and rotation of WMHs. MSERgSDM comprises two main components: (1) multi-scale representation construction, and (2) SDM construction. For the SDM construction, we collected N = 200 reasonable deformation fields generated using MSERg, verified through visual inspection. Three additional methods, including intensity-based registration, ProsRegNet, and MSERg, were also employed for comparison against MSERgSDM.

Results:

Our results suggest that MSERgSDM performed comparably to the ground truth(p>0.05). Additionally, MSERgSDM (ROI Dice ratio = 0.61, landmark distance = 3.26 mm) exhibited significant improvement over MSERg (ROI Dice ratio = 0.59, landmark distance = 3.69 mm) and ProsRegNet (ROI Dice ratio = 0.56, landmark distance = 4.00 mm) in local alignment.

Conclusions:

This study presents a novel registration method, MSERgSDM, for mapping ex vivo WMH onto in vivo prostate MRI. Our preliminary results demonstrate that MSERgSDM can serve as a valuable tool to map ground truth disease annotations from histopathology images onto MRI, thereby assisting in the development of machine learning models for prostate cancer detection on MRI.

1. Introduction

Multi-parametric magnetic resonance imaging (mpMRI) is becoming increasingly important in prostate cancer (PCa) diagnosis, treatment, and management^1–4. Radical prostatectomy (RP) specimens are considered to be the “gold-standard” for delineation of disease extent on mpMRI⁵, e.g., prostate cancer, benign prostatic hyperplasia (BPH) and prostatitis. While significant efforts have been devoted to developing machine learning models for PCa detection and risk stratification on MRI^6–8, accurate delineation of disease extent on the MRI remains crucial to provide reliable training data for these models. However, manual delineations on MRI are expensive and prone to errors when pathology is unavailable as reference⁹. Additionally, significant variability in PCa detection and segmentation persists due to differences in interpretability among readers and variations in imaging quality across institutions^10–12. Accurate co-registration can efficiently establish the disease annotations on MRI by aligning excised post-RP specimen with pre-RP mpMRI. Such annotations play a pivotal role in training more robust machine learning models and facilitating investigations into correlations between radiographic signatures and pathological phenotypes^9,13–15.

MpMRI comprises sequences of T2-weighted imaging (T2WI), diffusion-weighted imaging (DWI) and dynamic contrast-enhanced MRI (DCE-MRI)¹⁶. Among these modalities, T2WI provides the highest spatial resolution for prostate anatomy¹⁶. As a result, T2WI is commonly chosen as the reference image sequence in MRI-histopathology image registration studies^{9,13–15,17–19}. Registration algorithms optimize the mapping function between reference and target images to spatially align the target images with the reference images. However, MRI-histopathology image co-registration poses significant challenges due to three main reasons. First, T2WI and digitally scanned whole mount histopathology images (WMHs) exhibit substantial differences in appearance and represent prostate tissue at different scales, namely millimeter (mm) and micron scales. Second, the co-registration process needs to accommodate various deformations that occur during tissue preparation, including tissue resection, shrinkage, slicing, and sectioning. Finally, artifacts, such as tissue loss, tearing and folding, further complicate the co-registration, potentially leading to non-physiologic deformation. Consequently, modeling deformations between in vivo T2WI and ex vivo WMHs becomes challenging when using the same mapping function across different image pairs.

Recently, deep learning networks have been designed for image registration with high accuracy and efficiency²⁰. However, collecting paired WMHs and T2WI slices is challenging, making it difficult to train deep learning registration networks with sufficient data. Furthermore, the black-box nature of deep learning models can introduce unrealistic deformations that compromise the registration performance. Other methods also attempt to reconstruct WMHs as 3D volumes and then perform 3D co-registration¹⁴. While these methods eliminate the need for slice-to-slice matching, they come with increased computational costs. Some approaches aim to map both T2WI and WMHs into an alternative representation space where their appearances are more similar than in the original space, thereby improving registration performance^9,13. For example, multi-scale spectral embedding representation registration (MSERg) computes multi-scale alternative representations of both T2WI and WMHs to facilitate registration. However, MSERg experimentally picked the three scales where T2WI and histopathology images are the most similar for constructing alternative representations⁹, which may not be optimal. Consequently, in this study we further optimized MSERg by incorporating the structural similarity index measure (SSIM)²¹ to select the most similar spectral embedding representations between T2WI and histopathology images.

Regularization methods are necessary to smooth out the deformations and constrain unrealistic transformations when pathology images exhibit large deformations or artifacts. For example, it may not be feasible to obtain WMHs due to the limited size of single glass slides when the surgical specimens are large. It also requires more preparation time to slice large specimens thin enough to achieve a high resolution for accurate annotation²². In some cases, large specimens could be cut into fragments and mounted onto multiple slides (Figure 1). The reconstructed pseudo-WMHs contain more artifacts than actual WMHs, such as tissue folding and tissue loss, which require additional regularization during registration. Previous studies demonstrated that a collection of reasonable deformations can be parameterized as a statistical deformation model (SDM) to reflect the deformation distribution and infer anatomical changes between the target and reference images^23–25. By incorporating a prior knowledge of deformations between in vivo T2WI and ex vivo WMHs, an SDM can provide a representation of the deformation distribution and estimate the likelihood of a new deformation being reasonable. Therefore, in this study, we hypothesize that integrating MSERg with SDM (MSERgSDM) can enhance the robustness of MRI-histopathology co-registration.

Figure 1. Illustration of different datasets used for validating MSERgSDM.

1.) 3D-printed mold cohort and clinical cohorts. D₁ contains in vivo T2WI, ex vivo T2WI, and WMHs using a patient-specific 3D-printed mold. The in vivo and ex vivo T2WI co-registration was applied to approximate the ground truth deformation between WMH and in vivo T2WI. 2.) D₂ comprises re-stitched pseudo WMHs and in vivo T2WI. 3.) D₃ also contains WHMs but with ink annotations.

Moreover, most studies were validated on clinically acquired datasets where manual annotations are used for estimating registration errors^{9,13–15,17,18}. However, these validation strategies may not be appropriate due to inherent mismatching between the scans on T2WI and WMHs. Hence, in addition to the clinical datasets, we incorporated patient-specific 3D printed molds to approximate the ground truth deformation between T2WI and WMHs (Figure 1). This approach allowed us to validate the robustness of MSERgSDM in a more accurate and reliable manner. Finally, we compared the registration results of MSERgSDM with other methods, including intensity-based registration²⁶, MSERg⁹ and ProsRegNet¹⁵. This comparative analysis provides a comprehensive evaluation of the performance of MSERgSDM and highlights its advantages over alternative registration techniques.

To summarize, our study introduces a novel registration method, MSERgSDM, with significant contributions:

• Construction of a SDM using prior known reasonable deformations to summarize the deformation distribution between in vivo T2WI and ex vivo WMHs.

• Implementation of the SDM as a regularizer for MSERg, improving registration robustness by constraining unrealistic deformations caused by WMH artifacts.

• Collection of a unique dataset for registration performance validation, comprising 17 corresponding in vivo T2WI, ex vivo T2WI, and WMHs. This dataset enables the approximation of ground truth deformations between ex vivo WMHs and in vivo T2WI (Figure 1).

• Validation of the algorithm on multi-site clinically acquired data (68 pairs of in vivo T2WI and WMHs), including re-stitched pseudo-WMHs (n = 28) with additional artifacts such as tissue shrinkage, tissue loss, and inking (Table1).

• Evaluation of the performance of our method against the recently published state-of-the-art deep learning framework, ProsRegNet¹⁵.

Table 1.

Data description.

Dataset	Number of patients (image pairs)	Modalities	T2WI resolution (mm)	Slice-to-slice correspondence	Evaluation
D1	6 (17)	in vivo T2WI, ex vivo T2WI (with prostate fitted in the 3D printed mold), ex vivo WMHs	0.7×0.7×3	Use ex vivo T2WI as reference to determine the slice-to-slice correspondence between ex vivo WMHs and in vivo T2WI	DSC between registration results and ground truth deformation
D2	21 (28)	in vivo T2WI, ex vivo stitched pseudo WMHs	0.225×0.225×3	Visually determined by a pathologist and a radiologist working together in unison	DSC of anatomical ROIs
D3	21 (40)	in vivo T2WI, ex vivo WMHs	0.4×0.4×3.99	Visually determined by a pathologist and a radiologist working together in unison	RMSE between anatomical landmarks

2. Methods

2.1. Multi-scale Spectral Embedding Registration

Our previously published Multi-scale Spectral Embedding Registration (MSERg) algorithm has demonstrated the capability of accurately aligning WMHs onto corresponding in vivo T2WI⁹. MSERg involves the construction of multi-scale spectral embedding (SE) descriptors $F^{SE}=\{f_1^{SE},\dots ,f_m^{SE}\},f\subset {ℝ}^s,F\subset {ℝ}^{m\times s}$ to decrease the dissimilarity between the s-dimensional fixed image $I_0:{\Omega }_0\subset {ℝ}^s$ and moving image $I_1:{\Omega }_1\subset {ℝ}^s$ and leverages alpha-mutual information (αMI) as the cost function to accommodate for registration in high $m\times s$-dimensional space to achieve better co-registration results⁹. In this experiment, we further introduce SDM as a constraint to increase the robustness of MSERg and avoid unrealistic deformations during optimization. Therefore, we can formulate the registration as an optimization problem:

$$\stackrel{⌢}{d}=\text{arg}\underset{d}{\text{max}}\left(\alpha MI(F_0^{SE},d[F_1^{SE}])+\delta ·\mathit{\text{SDM}}(d)\right).$$ (1)

Where $F_0^{SE}$ and $F_1^{SE}$ represent the SE descriptors extracted from the fixed image $I_0$ and target image $I_1$ respectively. $d:{\Omega }_1\to {\Omega }_0$ denotes the dense deformation field that needs to be optimized so that the similarity between warped moving image $F_1^{SE}(x+d(x))$ and fixed image representations $F_0^{SE}$ can be maximized. $\delta$ is the weighting number for the constraint term SDM.

2.2. Statistical deformation model

SDM is estimated from a set of $n$ vectorized deformation fields $D=[d_1,d_2\dots d_n]$, which produce accurate and reasonable alignment between ex vivo WMHs and in vivo T2WI. We exploited principle components analysis (PCA) to build SDM as it has been commonly used for statistical model construction²⁴. The registration algorithms are applied on the image domain $\Omega \subset {ℝ}^s$ with the same discretization of $q$ degrees of freedom to enable mapping displacement at each position across different deformation fields. Therefore, the vectorized deformation field $d_i\in {ℝ}^{qs}$ is a qs-dimensional vector. As PCA assumes that $d_i$ is sampled from a multivariate normal distribution approximated as ${𝒩}_{qs}(\overline{d},C)$, where the mean deformation field is approximated as arithmetic mean $\overline{d}=\frac{1}{n}\sum _{i=1}^n{d}_i\in {ℝ}^{qs}$, and the covariance matrix $C\in {ℝ}^{qs\times qs}$ as $C={\overline{D}}^T\overline{D}$, where $\overline{D}=D-\overline{d}$.

Then, PCA estimates the principal components of the estimated multivariate distribution ${𝒩}_{qs}(\overline{d},C)$ via calculating the eigenvectors $U=[u_1,u_2,\dots ,u_n]\in {ℝ}^{qs}$ and corresponding eigenvalues ${\lambda }_i$ of the covariance matrix $C\in {ℝ}^{qs\times qs}$. We only retain $\tilde{U}=[u_1,\dots ,u_r]$ so that $\sum _{i=1}^r{\lambda }_i\le 0.98\sum _{i=1}^n{\lambda }_i$, which means that the SDM retains 98% of variance of $D$. The SDM is established as the log likelihood of a multivariate normal distribution as the following equation:

$$\mathit{\text{SDM}}(d)=\text{log}\left(\prod _{i=1}^r\frac{1}{\sqrt{2\pi {\lambda }_i}}\text{exp}\left(-\frac{{\left(u_i^T(d-\overline{d})\right)}^2}{2{\lambda }_i}\right)\right)=\sum _{i=1}^r\left(-\frac{1}{2}\text{log}(2\pi {\lambda }_i)-\frac{{\left(u_i^T(d-\overline{d})\right)}^2}{2{\lambda }_i}\right).$$ (2)

Where $d$ is a vectorized deformation field, $\overline{d}$ is the arithmetic mean of $D$, $u_i$ and ${\lambda }_i$ are the PCA estimated $i_{th}$ eigenvecoter and eigenvalue. Equation (2) can estimate the likelihood of $d$ as a reasonable deformation according to the SDM learned from prior known deformation pool. In this study, the SDM was calculated using 141 eigenvectors that covered 98% of the variance of the deformation pool. Figure 3 presents the pseudo code for the implementation of MSERgSDM and Figure 4 illustrates the deformations along the top three eigenvectors of SDM learned from our dataset, which covered approximately 28.8% variance. Table 2 lists the specific notational descriptions adopted in this study.

Figure 3.

The psesudo-code of the implementation of MSERgSDM consists of two main parts: (1) construction of SDM, where PCA was implemented to estimate the eigenvectors and eigenvalues; and (2) registration, where SDM was added to $\alpha MI$ as the cost function for optimization.

Figure 4.

Illustration of the deformations along the top three eigenvectors with [−3,3] standard deviation of SDM learned from our dataset. Here, u₁, u₂ and u₃ denote the top three eigenvectors of SDM. The blue arrows denote the direction and the magnitude of the deformation spatially. The top eigenvectors represent the main deformation captured from the deformation pool.

Table 2.

notational description.

Symbols	Description	Symbols	Description
$I_0$	Fixed image	$I_1$	Moving image
${\Omega }_0$	Fixed image field	${\Omega }_1$	Moving image field
$F_0^{SE}$	Fixed image multi-scale SE descriptors	$F_1^{SE}$	Moving image multi-scale SE descriptors
$d$	Deformation field	$D$	A set of deformation fields
$\overline{d}$	Mean deformation field	$C$	Cooccurrence matrix
m	Number of SE descriptors	$\delta$	Weighting parameter of SDM
s	Number of dimensions	$q$	Discretization degrees
n	Number of prior known deformation fields	$r$	Number of selected PCA components
$𝒩$	Normal distribution of SDM	$u$	Eigenvectors
$U$	Eigenvector matrix	$\tilde{U}$	Eigenvector matrix with selected eigenvectors
$\Sigma$	Diagonal matrix of eigenvalues	$\lambda$	Eigenvalues

2.3. Other registration methods and performance measurements

Three other registration methods, including intensity-based registration²⁶, MSERg⁹ and ProsRegNet¹⁵, were applied on the same datasets for peformance comparison. Intensity-based registration was a non-rigid registration with mutual information as the cost function. ProsRegNet was first trained with synthetically deformed mono-modal data and then was generalized to register histopathology images onto T2WI. In this study, a pretrained ProsRegNet¹⁵ was applied on the same dataset for head-to-head comparison. MSERg was implemented in autograd image registration laboratory (AIRLab)²⁷ with a similarity measurement αMI²⁸ added.

To evaluate the global alignment accuracy, the Dice similarity coefficient (DSC)²⁹ were applied to calculate the alignment and overlap between prostate capsule of T2WI and transformed WMHs. To evaluate the local alignment accuracy, DSC between the anatomical ROIs or root-mean-squrare error (RMSE) between anatomical landmarks were evaluated. In addition, the anatomical ROIs on transformed WMHs were also compared with ROIs on approximated ground truth using DSC to quantify the errors between the registration results and ground truth.

2.4. Data acquisition

This retrospective study was approved by the Case Western Reserve University, University Hospitals, and the Cleveland Clinic Institutional Review Board (IRB), and is compliant with the Health Insurance Portability and Accountability Act (HIPAA); de-identified data was used, and no protected health information was needed. The need for informed consent from all patients was waived by the IRB.

The study comprised three cohorts of data obtained from different institutes. D₁ consisted of N = 17 pairs of MRI and WMHs from six patients (Table1 and Figure 1). These WMHs were acquired using 3D-printed molds constructed according to the prostate capsule segmentation on T2WI. Each patient in D₁ had previously undergone 3 Tesla (T) pre-operative in vivo T2WI and the prostate capsule was manually segmented for 3D-printed mold design. The surgically removed prostate was fitted into the 3D-printed mold to ensure the in-plane correspondence between sliced surgical specimen and in vivo T2WI. Furthermore, we also collected 3T ex vivo T2WI as an additional reference to determine the slice-to-slice correspondence between ex vivo WMHs and in vivo T2WI in the axial plane. D₂ consisted of N = 28 pairs of MRI and pseudo-WMHs from 21 patients. The pseudo-WMHs were reconstructed using Histostitcher^®22 when the all the sliced specimen quadrants were available for constructing the WMHs. The stitched pseudo-WMHs contained more artifacts and deformation compared to the WMHs, posing greater challenges for co-registration. D₃ comprised N = 40 pairs of MRI and WMHs from 21 patients in the TCIA “Prostate-MRI” dataset³⁰. In this cohort, experts determined the slice correspondence between in vivo T2WI and histopathology images. For the construction of the SDM, N = 200 reasonable deformation fields were generated using MSERg from the datasets of previous studies^15,31, which in turn were collected via visual inspection.

2.5. Evaluation Reference

In D₁, during pre-processing, affine and nonrigid registrations were employed to maximally reduce the deformation between ex vivo WMH and ex vivo T2WI. Affine and nonrigid registrations were sufficient to recover the deformation because the 3D-printed mold constrains the deformation between ex vivo WMH and ex vivo T2WI. Then, ground truth deformation between in vivo T2WI and pre-processed WMH is approximated as the deformation between in vivo T2WI and ex vivo T2WI (Figure 1). In D₂ and D₃, the MSERg registration algorithm was first applied to map the whole-mount pathology images to the MRI images, and then the mapped annotations were reviewed and edited by a pathologist and a radiologist working together in unison. The co-registration was applied to maximally control the subjective error that may be introduced during the annotation. The annotations included regions of interest (ROIs), e.g., prostate cancer, and nodular prostatic hyperplasia, and the landmarks, e.g., urethra and the center of some hyperplasia nodules that are both visually obvious on whole-mount images and MRI images. Before registration was applied, flipping and rotation of WMHs were manually corrected.

2.6. Implementation

For all the RP specimens, the tissue region was cropped and resampled to a resolution of 0.5×0.5mm. Subsequently, the cropped tissue images were padded to a size of 200×200 pixels, with the tissue region centered within the padding. Similarly, the T2WI images were resampled to a resolution of 0.5×0.5mm, and the prostate capsules were cropped according to the manual annotations. The cropped T2WI images were also padded to 200×200 pixels with the prostate region in the center. This process ensured that all prostates were aligned within a consistent coordinate space. To initially align the histopathology images and in vivo T2WI, an affine registration was performed. Subsequently, intensity-based registration, ProsRegNet, MSERg, and MSERg with SDM were applied to register the pre-processed WMH onto the in vivo T2WI. To accommodate ProsRegNet, the images were further padded to a size of 240×240 pixels.

All the registration experiments were performed using an Intel Core i7-10700 CPU (8-Core, 16-Thread, 2.9 GHz (4.8 GHz Turbo)). Intensity-based registration was executed using Elastix toolbox²⁶ with a multi-resolution strategy, mutual information (MI) as the similarity metric, standard gradient descent as the optimizer, and B-spline transform. MSERg and MSERgSDM were implemented in AIRLab²⁷. ProsRegNet and AIRLab were implemented on the GeForce RTX 2060 GPU. AIRLab borrows automatic differentiation of PyTorch³² and thus avoids the need for analytic gradient equation for the objective function. The pseudo-WMH is constructed using Histostitcher^®22, implemented in Matlab 2019a.

3. Results

3.1. Quantitative results

3.1.1. Validation on Mold data

D₁ contains in vivo T2WI, ex vivo T2WI and ex vivo WMHs acquired using 3D-printed mold. The ground truth deformations were approximated via performing intensity-based registration between ex vivo T2WI and in vivo T2WI. These ground truth deformations were then applied to the ex vivo WMHs. The approximated ground truth deformations produced a DSC of 0.98 ± 0.01 for the prostate capsule alignment between in vivo T2WI and ex vivo WMHs, and a DSC of 0.74 ± 0.14 for anatomical ROI alignment. MSERg exhibited a similar overall alignment accuracy compared to the approximated ground truth. However, the DSC for anatomical ROIs was significantly lower than the approximated ground truth. ProsRegNet demonstrated a high mean DSC for anatomical ROIs, comparable to the ground truth, but the overall alignment was significantly lower than the approximated ground truth. Intensity-based registration performed significantly worse than the approximated ground truth. Only MSERgSDM illustrated a comparable registration performance to the approximated ground truth, both in terms of global and local measurements, with p > 0.05 (Table 3).

Table 3. Quantitative registration results on D₁.

The asterisks denote statistically significant differences compared to ground truth.

Methods	Capsule DSC	ROI DSC
Intensity based registration	0.76 ± 0.08*	0.69 ± 0.13*
ProsRegNet	0.95 ± 0.04*	0.69 ± 0.14
MSERg	0.97 ± 0.02	0.68 ± 0.14*
MSERg SDM	0.97 ± 0.02	0.72 ± 0.10
Ground truth	0.98 ± 0.01	0.74 ± 0.14

3.1.2. Validation on Clinically Acquired Data

D₂ comprised N = 28 pairs of in vivo T2WI and pseudo-WMHs containing more artifacts and noise compared to WMHs. MSERgSDM demonstrated significant improvement over MSERg and ProsRegNet in terms of anatomical ROI alignment (Table 4). Furthermore, in D₃ (N = 49), MSERgSDM better aligned the anatomical structures than MSERg and ProsRegNet, suggested by the anatomical landmark distance between transformed WMHs and T2WI (Table 4). These results suggested that the SDM integration in MSERgSDM aided in preserving anatomical structure and avoiding unrealistic deformation between histopathology images and in vivo T2WI. In addition, the results indicated that both MSERg and MSERgSDM can more accurately align the prostate capsule between ex vivo WMHs and in vivo T2WI than intensity-based registration and ProsRegNet.

Table 4. Quantitative registration results on D₂ and D₃.

The asterisks denote statistically significant differences compared to MSERgSDM.

	D2 N = 28		D3 N = 49
Methods	Capsule DSC	ROI DSC	Capsule DSC	RMSD
Intensity based registration	0.84 ± 0.06*	0.5±0.20	0.78±0.09*	3.53±1.20
ProsRegNet	0.91 ± 0.04*	0.56±0.17*	0.94±0.02*	4.00±1.68*
MSERg	0.97 ± 0.01	0.59±0.16*	0.96±0.02	3.69±1.44*
MSERg SDM	0.97 ± 0.01	0.61±0.16	0.96±0.02	3.26±1.45

3.2. Qualitative results

Figure 5 demonstrates the registration results of a pair of in vivo T2WI and WMH with corresponding ex vivo T2WI in D₁. The prostate capsules are highlighted in green and red on WMHs and T2WI, respectively. The corresponding anatomical ROIs were delineated on WMHs or ex vivo T2WI (orange) and T2WI (blue) (Figure 5 (d)-(f)). Additionally, Figure 5 (d) and (e) show the transformed WMHs and ex vivo T2WI based on the approximated ground truth, with the transformed annotations highlighted. MSERgSDM demonstrates comparable registration results to the approximated ground truth, while other registration methods show inferior alignment accuracy for the global capsule or anatomical ROI alignment.

Figure 5.

Registration results on a pair of (a) in vivo T2WI and (b) WMH with corresponding (c) ex vivo T2WI. Images (d)–(f) are the capsule and anatomical ROI annotations. Green and red lines delineate the prostate capsule boundary on histopathology images and in vivo T2WI, respectively. The anatomical ROIs are annotated using blue lines on in vivo T2WI and orange lines on WMH and ex vivo T2WI. (g) and (h) are the ground truth transformation results of WMH and ex vivo T2WI. (i)–(l) are the registration results of MSERgSDM (i); MSERg(j); ProsRegNet(k); and intensity-based registration(l).

Figure 6 illustrated the registration results of two pairs of ex vivo WMHs and in vivo T2WI. Intensity-based registration achieved comparable ROI alignment accuracy to MSERgSDM, but exhibits significantly worse accuracy for global alignment compared to all other methods. ProsRegNet tends to under-deform the WMHs compared to MSERgSDM. MSERgSDM effectively aligns the boundaries of the prostate capsule in both examples and accurately maps tumor extension from WMHs onto T2WI, even in the presence of large artifacts. In addition, MSERgSDM illustrates a better urethral alignment between histopathology image (green dots) and T2WI (red dots) (Figure 6).

Figure 6.

Registration results of (a) two pairs of histopathology image and in vivo T2WI using (b) intensity-based registration; (c) ProsRegNet; (d) MSERg; and (e) MSERgSDM. Green and red lines delineate the prostate capsule boundary on histopathology images and in vivo T2WI, respectively. The tumor extensions are annotated using blue lines on in vivo T2WI and orange lines on histopathology images. The green and red dots denote the urethra on histopathology images and in vivo T2WI.

Furthermore, Figure 7 highlighted the region where MSERg introduces unreasonable, while SDM helps constrain such unnatural deformation. The results show that MSERgSDM generates more robust and accurate registration outcomes compared to MSERg by incorporating SDM to constrain unrealistic deformations. These visualizations and observations support the efficacy of MSERgSDM in achieving improved registration results, surpassing other methods, and providing accurate alignment of anatomical structures between histopathology images and in vivo T2WI.

Figure 7.

Registration results of (a, b) the same pseudo WMH using MSERg and MSERgSDM. The blue arrows visualize the spatial deformation. (a) the dashed line highlights the region of unnatural deformation; (b) the dash line highlights the corresponding region where SDM enables the constraining of non-realistic deformation.

4. Discussion

Accurate delineation of disease extent on MRI is critical to the development of computer-aided prostate cancer detection and characterization models^6–8. However, manual delineations on radiographic images are labor-intensive and prone to subjective variance^14,15. To address this challenge, previous studies have attempted to develop co-registration algorithms to accurately map disease extent from ex vivo WMHs onto in vivo MRI. These approaches could leverage anatomical landmarks¹⁹, create new representations of both image modalities⁹, construct novel cost function^13,33, or implement deep learning networks¹⁵ to facilitate co-registration. However, due to various types of artifacts on WMHs, unrealistic deformations (Figure 7) may be produced during registration, leading to errors in mapping the disease extent on MRI.

In this study, we proposed a novel registration approach, MSERgSDM, which incorporates a statistical deformation model (SDM) as an additional regularizer to constrain deformations and improve registration robustness to noise and artifacts. MSERgSDM consists of two main parts. First, MSERg⁹ allowed for computing alternative representations of WMHs and T2WI to shrink differences in their appearance and thereby improve registration accuracy. In this study, MSERgSDM utilizes the structural similarity index to select the top three most similar representations, reducing computational cost without compromising registration performance. Second, SDM can regularize the new deformation within a reasonable range according to the prior known plausible deformations. SDM is constructed using PCA derived eigenvectors and eigenvalues. As illustrated in Figure 4, the top three eigenvectors capture three main deformations learned from the deformation pool, including the shrinkage and enlargement deformation along the prostate capsule boundaries, and the smooth deformation around urethra. In addition, the corresponding eigenvalues capture the distribution of deformations along the directions of the eigenvectors. During registration iterations, MSERgSDM projects the current deformation onto SDM and estimates its plausibility to further regularize deformation. Although it is difficult to obtain deformations that perfectly transform ex vivo WMHs to match in vivo T2WI, reasonable deformations can be collected from previous results on images with few artifacts using methods such as landmark-based registration¹⁸, non-rigid deformation assisted with novel similarity metrics¹³, or alternative image representations⁹. Furthermore, SDM can be initially built on a few high quality samples²³, and then new SDM-regularized deformations can be accumulated for SDM training to increase its expressiveness.

Additionally, this study collected multi-site cohorts of data to validate the robustness of MSERgSDM, as listed in Table 1. The 3D printed mold used for collecting D₁ was constructed using prostate gland segmentations from in vivo T2WI, which allows the prostate specimen to be cut at approximately the same axial resolution as the in vivo T2WI. Hence, the evaluation of registration performance is more accurate on D₁ compared to D₂ or D₃. Moreover, because both in vivo and ex vivo T2WI correspond to the same imaging modality, the intensity-based registration can accurately approximate their deformations. Furthermore, fewer deformations exist between the ex vivo T2WI and the whole-mount pathology, so the deformation between in vivo and ex vivo T2WI can be approximated as the deformation between in vivo T2WI and the whole-mount pathology images, which is then used as the ground truth. Therefore, we can directly compare the deformation fields estimated by different registration methods with the ground truth to evaluate the registration performance in D₁.In this study, MSERgSDM illustrated comparable performance to the ground truth deformation estimated on D₁. In addition, clinical cohorts D₂ and D₃ were also collected to evaluate the robustness of different registration methods when more artifacts exist. The anatomical ROI and landmark alignment results on D₂ and D₃ corroborated the registration accuracy of MSERgSDM suggesting that SDM could make MSERg more robust to artifacts on WMHs.

Deep learning-based methods have also been explored for co-registration of multi-modal imaging data. For instance, ProsRegNet, a deep learning framework, was recently proposed for registration of prostate MRI and WMHs¹⁵. ProsRegNet achieved similar performance as other the state-of-art registration algorithms¹⁴ while offering faster computation. Our results suggested that MSERgSDM can outperform ProsRegNet and reach better global and local alignment between WMH and in vivo T2WI. ProsRegNet was trained with synthetically deformed mono-modal data and then generalized to register WMHs onto T2WI¹⁵. However, due to the complexity of deformation between WMHs and in vivo T2WI, using randomly deformed image pairs for training may potentially jeopardize the registration, when synthetic deformations cannot fully reflect real deformation distribution between WMHs and in vivo T2WI. Conversely, our method selected representative deformations to construct the SDM as a regularizer while allowing for the flexibility to compensate for deformations not captured by SDM.

A previous study³⁴ found that 13% of clinically significant prostate cancers (CsPCa) were systematically overlooked by mpMRI in the PROMIS cohort and the median maximum cancer core length (MCCL) of undetected CsPCa was 5mm. Therefore, if a registration algorithm can limit the alignment error to within 5 mm, it can effectively assist in identifying CsPCa annotations that may be missed through visual observation on mpMRI. Table 4 provides the landmark distance metrics of various registration algorithms. It shows that the upper bound of the distance error for MSERg is 5.13 mm, MSERgSDM is 4.71 mm, and ProsRegNet is 5.68 mm. Among these, MSERgSDM best fulfills the requirement of controlling the mapping error to within 5 mm. Additionally, MSERgSDM demonstrates significantly higher ROI Dice similarity coefficient (DSC) values compared to MSERg and ProsRegNet. The results highlight the potential of MSERgSDM to provide annotations of CsPCa that may be systematically overlooked by mpMRI for the development of machine learning models to enhance CsPCa detection accuracy.

Although our preliminary results have shown that MSERgSDM can accurately align WMHs onto in vivo T2WI, there are still some limitations. The new MSERg SDM is primarily a 2D registration algorithm, designed to address deformations within each individual slice. However, it is crucial to note that the actual deformations occur in 3D space, and potential plane mismatching between T2WI and WHM slices may negatively impact registration performance. It is also challenging to reconstruct 3D prostate volume using WMH. First, the RP specimens comprise the axial sectioning of the middle gland and the sagittal sectioning of the coned apex and base at 3 mm intervals³⁵. Second, each WMH is a 4 μm section cut from a 3 mm-thick slice. And the WHM may be flipped, randomly rotated, or have tissue folding or loss artifacts during mounting process. Third, the digitally scanned WMH images are typically several gigabytes in size, making the process of image reconstruction cumbersome, labor-intensive and computationally expensive. Beyond the technical and logistical issues associated with the registration, the bigger question that we sought to address in this work was the ability to carefully and spatially map disease extent from the pathology on to the MRI to be able to create accurate training sets for the development of AI algorithms on radiology. With this intent in mind, the 2D approach is reasonable since it enables us to achieve this objective. Additionally, given that prostate lesions typically do not extend across the entire prostate, it is more effective to prioritize local registration performance. Consequently, 2D registration can place greater emphasis on aligning slides that contain prostate lesions. Additionally, this approach also aligns with previously published 2D methods for WMH and in vivo T2WI co-registration^15,17,19,31.

Moreover, MSERgSDM is an iterative method, so the computation will not be as fast as the one-shot deep learning methods, e.g., ProsRegNet. However, in the scenario of WMHs and in vivo T2WI co-registration, accuracy of registration is more important than computation time. Moreover, due to the limited size of the deformation fields used to learn SDM, the current SDM regularizer has limited capacity in constraining the deformation only in the main direction captured by SDM. Thus, more deformation fields should be collected to increase the expressiveness of SDM in the future. Furthermore, we will explore other roles that SDM can play in assisting MRI-histopathology co-registration, such as guiding the generation of synthetic deformations to facilitate the training of deep learning-based registration. In addition, the preprocessing involved manual input, such as flipping and rotation of WMHs, and slice-to-slice correspondences between WMH and T2WI. Thus, in the future, we will attempt to automate the preprocessing steps and explore the integration of SDM as a regularizer with other registration pipelines to accelerate the registration speed.

5. Conclusion

This study presented a novel registration method, MSERgSDM, for mapping whole mount radical prostatectomy surgical specimens onto in vivo prostate MRI. By incorporating a statistical deformation model based on prior knowledge of reasonable deformations, MSERgSDM effectively constrained unrealistic deformations and preserved the anatomical structure on the transformed histopathology images. Our preliminary results demonstrated that MSERgSDM could be employed as a tool to map ground truth disease annotations from histopathology images onto MRI and, in turn, assist in the development of machine learning models for prostate cancer detection on MRI. In addition, such registration methods can also be deployed to establish spatial correspondence between histology and radiology images to facilitate the investigations of correlation between radiographic phenotypes with underlying biological meanings.

Figure 2.

The flowchart of MSERgSDM. MSERgSDM comprises two main parts: (1) multi-scale representation construction; and (2) SDM construction. The top three features with highest structural similarity index were identified as the multi-scale representation $F_0^{SE}$ and $F_1^{SE}$. The SDM was estimated as the gaussian distribution with the orthogonal basis as PCA eigenvectors and variance as the eigenvalues. During $i_{th}$ registration iteration, the current deformation field $d_i$ was projected into SDM and the estimated likelihood $SDM(d_i)$ would be used as the regularizer to constrain the non-physical deformations.

Abbreviations:

ROIs

regions of interest

PCa

prostate cancer

RP

radical prostatectomy

BPH

benign prostatic hyperplasia

WMHs

whole mount histopathology images

MSERg

a multi-scale feature-based registration

SDM

a statistical deformation

MSERgSDM

a multi-scale feature-based registration with a statistical deformation

T2WI

T2-weighted imaging

DWI

diffusion-weighted imaging

DCE

dynamic contrast-enhanced MRI

SWMI

spatial weighted mutual information

MACMI

a multiattribute combined mutual information

CNN

convolutional neural network

AIRLab

autograd image registration laboratory

DSC

Dice similarity coefficient

RMSE

root-mean-square error

MI

mutual information