3D Registration of Pre-Surgical Prostate MRI and Histopathology Images via Super-Resolution Volume Reconstruction

Abstract

The use of MRI for prostate cancer diagnosis and treatment is increasing rapidly. However, identifying the presence and extent of cancer on MRI remains challenging, leading to high variability in detection even among expert radiologists. Improvement in cancer detection on MRI is essential to reducing this variability and maximizing the clinical utility of MRI. To date, such improvement has been limited by the lack of accurately labeled MRI datasets. Data from patients who underwent radical prostatectomy enables the spatial alignment of digitized histopathology images of the resected prostate with corresponding pre-surgical MRI. This alignment facilitates the delineation of detailed cancer labels on MRI via the projection of cancer from histopathology images onto MRI. We introduce a framework that performs 3D registration of whole-mount histopathology images to pre-surgical MRI in three steps. First, we developed a novel multi-image super-resolution generative adversarial network (miSRGAN), which learns information useful for 3D registration by producing a reconstructed 3D MRI. Second, we trained the network to learn information between histopathology slices to facilitate the application of 3D registration methods. Third, we registered the reconstructed 3D histopathology volumes to the reconstructed 3D MRI, mapping the extent of cancer from histopathology images onto MRI without the need for slice-to-slice correspondence. When compared to interpolation methods, our super-resolution reconstruction resulted in the highest PSNR relative to clinical 3D MRI (32.15dB vs 30.16dB for BSpline interpolation). Moreover, the registration of 3D volumes reconstructed via super-resolution for both MRI and histopathology images showed the best alignment of cancer regions when compared to (1) the state-of-the-art RAPSODI approach, (2) volumes that were not reconstructed, or (3) volumes that were reconstructed using nearest neighbor, linear, or BSpline interpolations. The improved 3D alignment of histopathology images and MRI facilitates the projection of accurate cancer labels on MRI, allowing for the development of improved MRI interpretation schemes and machine learning models to automatically detect cancer on MRI.

Graphical Abstract

1. Introduction

Prostate cancer is the second deadliest cancer in men; one in nine men in the United States is diagnosed with prostate cancer during his lifetime (Noone et al., 2018). Early diagnosis and treatment of clinically significant cancer reduces death from prostate cancer (Bill-Axelson et al., 2014). However, conventional prostate cancer diagnosis leads to over-diagnosis and over-treatment of many non-aggressive cancers (Klotz, 2013). The use of Magnetic Resonance Imaging (MRI) is increasing in an effort to preferentially find aggressive cancers that warrant treatment (Murphy et al., 2013). However, the interpretation of MRI is challenging due to subtle differences between benign prostate tissues and cancer, resulting in high variability in interpretation, even among experts (Pickersgill et al., 2020; Sonn et al., 2019). The tedious task of manually delineating cancer on MRI is prone to inaccuracy, often underestimating cancer size and missing cancers that are not clearly visible on MRI (Puech et al., 2012). Accurately defining the location of cancer on MRI may help establish better interpretation schemes and machine learning models.

Patients who have undergone radical prostatectomy for prostate cancer treatment provide a unique opportunity to correlate pre-surgical MRI with whole-mount histopathology images of the resected tissue on which the extent of cancer can be accurately outlined at high-resolution (Humphrey, 2017). Through the registration of digitized histopathology images to their corresponding MRI slices, the extent of cancer can be mapped directly from histopathology images onto MRI, resulting in accurate cancer segmentation, including labels for MRI invisible lesions.

Most current registration methods assume slice-to-slice correspondence between MRI and histopathology images, which can be obtained using custom 3D printed molds for tissue sectioning (Turkbey et al., 2011). However, tissue shrinks and undergoes distortions during formalin fixation, causing movement in the mold and resulting in imperfect correspondence between histopathology images and MRI. Specifically, movements in the apex-base direction result in shifts as large as ~2mm, which is typically half the distance between slices on MRI. Moreover, in the more common scenario where 3D printed molds are not used and tissue is manually sectioned, correspondence between histopathology images and pre-surgical MRI slices is hard to maintain.

One approach to alleviate the lack of perfect slice-to-slice correspondence is to perform the registration of histopathology images to MRI in 3D. However, neither the histopathology image stacks nor the MRI are true 3D volumes. Specifically, Axial T2weighted (T2w) fast spin echo (FSE) MRI has high-resolution in the acquisition plane (axial, in-plane) and low-resolution in the slice select direction (through-plane, sagittal Fig. 1a), resulting in a 2.5-dimensional (2.5D) volume with a voxel size of roughly 0.4×0.4×4.2mm³. Here, we use the term 2.5D to signify volumes where the third dimension (throughplane) has a pixel size that is at least 4 times larger compared to the pixel size of the two other dimensions (in-plane). Multiple histopathology images are acquired as part of processing the excised tissue and are stacked to form a simple 3D representation. Histopathology image stacks have high resolution in the scanning plane (in-plane), typically 0.5 μm at 20x magnification. However, the slices are roughly 4mm apart (through-plane, Fig. 1b).

Fig. 1:

Information between slices in (a) an Axial T2w MRI volume is apparent as pixelation artifacts in the sagittal plane (yellow arrow), and in (b) a histopathology image stack is apparent as empty space between slices.

2. Related Work

2.1. MRI-Histopathology Registration

Multiple automated methods have been developed to register prostate histopathology images with corresponding presurgical T2w MRI. One approach created an image-guided slicing method that sectioned excised tissue in the same plane as pre-surgical MRI and added string fiducials to the tissue to aid registration (Ward et al., 2012). Other approaches used 3D printed molds to optimize correspondence between histopathology and MRI images (Patel et al., 2011; Kalavagunta et al., 2014; Reynolds et al., 2015; Shao et al., 2020; Wu et al., 2019; Rusu et al., 2020). Even after using 3D printed molds, slice-to-slice correspondence remains imperfect due to tissue shrinking during fixation and tissue movement within the mold during sectioning.

Alternative approaches relied on ex vivo MRI as an intermediate between in vivo pre-surgical MRI and sectioned tissue images (Park et al., 2008; Reynolds et al., 2015). However, ex vivo MRI is not acquired as part of a routine workflow and does not address tissue distortion artifacts caused by formalin fixation. Other studies relied on additional information provided by multi-parametric MRI and multi-scale feature extraction, along with custom quantitative measures optimized during registration (Chappelow et al., 2011a; Li et al., 2017). In addition to requiring slice correspondences, such approaches also required careful selection of features and are computationally intensive. Other groups circumvented the imperfect slice-to-slice correspondence problem by reconstructing 3D versions of the stack of histopathology images and MRI in the through-plane using traditional interpolation methods followed by 3D registration (Patel et al., 2011; Losnegård et al., 2018; Wu et al., 2019).

2.2. Medical Image Super-Resolution

Super-resolution methods are particularly valuable to facilitate the reconstruction of histopathology and MR images due to their ability to learn though-plane information that facilitates the use of 3D registration methods. Super-resolution for medical imaging has been used to increase in-plane (Bahrami et al., 2016; Albay et al., 2018; Chaudhari et al., 2018; Chen et al., 2018; Liu et al., 2018; Sanchez and Vilaplana, 2018; Shi et al., 2018; Sood et al., 2018; Zeng et al., 2018; Pham et al., 2019; Lyu et al., 2020) or through-plane resolution (Oktay et al., 2016; Pham et al., 2017; Zhao et al., 2018; Sood and Rusu, 2019). Some approaches learned through-plane information using 3D convolutions (Pham et al., 2017), which can be computationally intensive. Others processed images in 2D either by using high-resolution information from orthogonal volumes (Oktay et al., 2016; Sood and Rusu, 2019) or downsampled versions of in-plane images (Zhao et al., 2018). However, orthogonal volumes may not be available, while downsampling can introduce new artifacts.

Our super-resolution network builds on previous in-plane super-resolution approaches (Ledig et al., 2017), and performs through-plane super-resolution to create a 3D volume by adding information between the slices of the input volume that facilitates the use of 3D registration methods. Unlike approaches based on 3D convolutions, we utilize a multi-image input to allow the efficient learning of spatial information without requiring the optimization of 3D kernels. Moreover, we developed our approach to generalize to two very different datasets: MRI and histopathology.

3. Methods

3.1. Data and Cohort Description

We used a subset of publicly available data from The Cancer Imaging Archive (Cohort P1) (Clark et al., 2013; Bloch et al., 2015; Litjens et al., 2017) and internal data from Stanford Hospital (Cohort P2) (Table 1). Of the multi-parametric MRI sequences available for these patients, we used T2w MRI as it provides the highest soft tissue contrast needed for the alignment of MRI with histopathology images (Stabile et al., 2020). For Cohort P1, only pre-surgical MRI volumes were provided on the Cancer Imaging Archive. Patients in Cohort P1 underwent prostate biopsy to confirm the presence of cancer and it is unknown whether any of these patients underwent radical prostatectomy. All subjects in Cohort P2 underwent radical prostatectomy, from which whole-mount histopathology sections were available for our study. Following surgical resection, the prostate was fixed in formalin and embedded in paraffin. The prostate specimen was then sectioned into 5μm thick slices using a 3D printed mold that was built based on the pre-surgical MRI (Turkbey et al., 2011). The goal of the patient-specific mold is to maintain correspondences between the histopathology slices and T2w images. The whole-mount slices were stained using Hematoxylin & Eosin (H&E) and were digitized at 20x magnification (pixel size 0.5 μm). Fragmentation in the histopathology slices was rare and mostly affected slices at the base of the prostate. Extreme apex and base tissue blocks are cut perpendicular to the axial plane and are discarded in our reconstruction. Figure S2 shows how the prostate tissue is sectioned and a representative histopathology image of perpendicular slices that are discarded in our approach.

Table 1:

Summary of the data used in our study.

Cohort	P1	P2
Source	Cancer Imaging Archive	Stanford University
Dataset	ProstateX; Prostate Diagnosis	-
Variable	MRI	MRI	MRI	Pathology
Modality / Data Type	T2w	3D Gradient Echo T2w	2.5D T2w	whole mount (7 slices per patient)
Manufacturer: Coil Type	Siemens: surface; Phillips: Endorectal	GE: Surface	GE: Surface	-
Number of Patients	237; 78	6	106	106
Pixel Spacing (mm)	0.3-0.6	0.43	0.27-0.94	0.008-0.016
Distance between slices (mm)	3.0-5.0	0.8	3.0-4.2	3.0-4.2
Matrix Size	K,L: 384; M: 19-25	K,L: 512; M: 156	K,L: 256-512; M: 24-43	W,H: 1663-7556; Q: 3-11
Acquisition	TR: 3.8-6.3;	TR: 2.0;	TR: 3.9-6.3;	H&E
Characteristics	TE: 101-120	TE: 101-120	TE: 122-130
Annotations	-	-	Prostate, Landmarks	Prostate, Landmarks

Abbreviations: T2weighted MRI (T2w), Hematoxylin & Eosin (H & E), Relaxation Time (TR) in seconds, Echo Time (TE) in milliseconds; MRI Matrix Size: K × L × M, Histology Matrix Size: W×H×Q. Anatomic landmarks include the urethra and prostate cancer. The matrix size for the 2.5D T2w MRI for subjects in cohort P2 varies across subjects, with the majority (n=98) cases of size 512×512 and pixel spacing of 0.4×0.4 mm², five subjects with matrix size 448×448, and one case with either 256×256, 320×320, or 384×384 pixels, respectively.

Six patients in Cohort P2 had clinical 3D T2w acquisitions in addition to 2.5D T2w MRI. Detailed MRI acquisition parameters of the 3D T2W MRI are provided in Table 1. The additional 3D acquisitions were used exclusively to evaluate the results of the MRI super-resolution and were not included in training. We used the 3D acquisitions because these volumes represent true 3D data and enabled us to evaluate the fidelity of our super-resolved volumes to real 3D volumes. For patients in Cohort P2, NCT and SJCS annotated the prostate on Axial T2w MRI on a slice-by-slice basis for 65 and 41 cases, respectively. NCT and JBW annotated the prostate on histopathology images for 65 cases and 41 cases, respectively. MR (with 10 years of experience reviewing prostate MRI and histopathology images) annotated the urethra on Axial T2w MRI and histopathology images for 25 cases. MR also reviewed and updated all cases to ensure the 3D consistency of all annotations. Inter-reader variability was not evaluated as no case had multiple readers. Cancer on histopathology images was annotated by an expert pathologist (CAK). Mounting the histopathology tissue onto glass slides may induce rotations and flips of the tissue, which were corrected manually on the digital histopathology images prior to further processing.

The through-plane information in MRI volumes and histopathology image stacks is highlighted in Fig. 2. The axial T2w volume has high-resolution in the axial plane (Fig. 2a) and anisotropic resolution (with low resolution in the IS direction) in the sagittal plane (Fig. 2d). In contrast, the sagittal T2w MRI volume, which is acquired as part of the same MRI exam for the same patient, has high-resolution in the sagittal plane (Fig. 2e), but anisotropic resolution in the axial plane (Fig. 2b). Figures 2d–e show the same prostate depicted in the sagittal view. Missing through-plane information in the axial T2w volume is visible as a pixelation artifact (Fig. 2d). Since tissue sectioning was performed in the axial plane, the histopathology image stack lacks information along the orthogonal direction (Fig. 2f).

Fig. 2: (a-c) Axial and (d-e) Sagittal views of the (a, d) 2.5D Axial T2w MRI, (b, e) 2.5D Sagittal T2w MRI volumes, and (c, f) the histopathology image stack of the same prostate.

Note the resolution difference (shown as pixelation artifacts) between the following sets of images: a,b,c and d,e,f.

3.2. Training the Super Resolution Network

3.3. Data Preprocessing

Histopathology:

The glass slides were digitally scanned at 20x magnification, with a pixel size of 0.5μm (Fig. S1). We preprocessed the histopathology images in three steps on a case-by case-basis. First, we extracted a lower-resolution image from the multi-resolution pyramid of the histopathology sections scanned using the Aperio scanner (Leica Biosystems, Wetzlar, Germany). At this lower resolution, the pixel size was either 0.008mm or 0.016mm and provided sufficient resolution to recognize tissue features and landmarks. Second, we corrected the gross rotation and flips that are introduced during the mounting of the tissue section on the glass slide. Third, we applied groupwise registration to find in-plane transformations that align the histopathology slices relative to each other within the image stack. Groupwise registration aligns the histopathology slices by maximizing the similarity between adjacent slices using gradient descent optimization. The slices are iteratively aligned using the following cost function:

$$C\left({\varphi }_1,\dots ,{\varphi }_M\right)=\sum _{i=1}^{M-1}{\Vert S_i○{\varphi }_i-S_{i+1}○{\varphi }_{i+1}\Vert }_2$$ (1)

where S₁, … , S_M is the sequence of M histopathology images within a stack, ϕ₁, … , ϕ_M are in-plane rigid transformations, S_i ○ ϕ_i represents the transformed histopathology slice, and ‖·‖₂ represents the L2 norm. S₁ serves as reference and its transformation was not optimized. By minimizing the sum of intensity differences between each slice and its adjacent slices, our groupwise registration can optimize the transformations {ϕ₁, … , ϕ_M} for all images at the same time.

Other pre-processing steps include applying Gaussian smoothing with σ = 5 pixels and a 101×101 pixel kernel, and rescaling the image intensities between −1 to 1 for the whole histopathology volume. Our histopathology images typically have height and width dimensions ranging well into the thousands of pixels. The slice thickness is 5μm and the distance between slices is that of the MRI (3.0–4.2mm). In order to maintain relatively high in-plane resolution, the histopathology images were only downsampled in-plane by a factor of 8 at test time.

MRI:

We preprocessed the MRI volumes in two steps. First, we enhanced the contrast of the MRI volumes by capping the intensity histogram for each volume to three standard deviations above the volume mean. This intensity capping was applied to the MRI volume as a whole. Second, the MRI volumes were resized to 224×224×M, where M is the original number of slices, and the intensities were rescaled between −1 and 1 using the following formula:

$$I{(x,y,z)}_r=2*\frac{I(x,y,z)-\mathit{\text{min}}}{\mathit{\text{max}}-\mathit{\text{min}}}-1$$ (2)

where I(x, y, z) is the pixel value at indices x, y, z, I(x, y, z)_r is the rescaled pixel value, and max and min are the maximum pixel value and the minimum pixel value in the original image I, respectively.

At test time, we applied the same contrast enhancing procedure and rescaled the volumes using the parameters obtained from the training set. However, the images were not downsampled, resulting in a super-resolved MRI volume with the same in-plane resolution as the original volume.

We trained the miSRGAN in three steps. First, sequences of three images were taken from the preprocessed volumes, with a stride of 1. The first (I^prev) and third (I^next) images in the sequence are input to the generator, while the second (center) image (I^gt) is used as ground truth (Fig. 3). Second, each input image is passed through a residual network, and the features extracted by these networks are concatenated and processed by a third residual network. This residual network produces the generated center image (I^gen), which represents the reconstructed spatial information missing in 3D between the two input images. Finally, I^gt and I^gen are passed to the discriminator, which is a series of convolutional layers followed by densely connected layers for classification and seeks to distinguish the two types of images. We send a single image into the discriminator instead of a triplet containing the generated image as we want the discriminator to evaluate whether the image is realistic and reasonable given the ground truth, not whether it is a valid interpolation. Adversarial training has converged when the discriminator cannot differentiate between I^gt and I^gen. The stopping condition is determined by monitoring the graphs of the generator and discriminator losses. In this study, training was stopped when the plot of the discriminator loss jumps and says steady at 0.5, indicating that the discriminator is classifying images randomly, which implies that the generated and ground truth images are indistinguishable to the discriminator. Both the MRI and histopathology networks were trained using Tensorflow 1.14 and an NVIDIA 2080Ti GPU. The networks were trained with an Adam optimizer, no regularization, and a batch size of 8.

Fig. 3: Multi-Image Super-Resolution Generative Adversarial Network Architecture.

The network contains a multi-image generator and a discriminator. In the generator, I^prev and I^next are processed separately by two residual networks. The extracted features are concatenated and processed by a third residual network to produce I^gen. The discriminator is constructed by a series of convolutional layers, followed by fully connected layers and classifies its input as ground truth (gt) or generated (gen).

Training is driven by the optimization of the generator (l_G) and discriminator (l_D) losses. The generator loss, l_G, has three terms:

$$l_G=l_{MSE}+\beta l_{\mathit{\text{percep}}}+\gamma l_{GAN}$$ (3)

where l_MSE is the mean squared error (MSE), l_percep is the perceptual loss (Johnson et al., 2016), and l_GAN is the generator component of the GAN loss. In this study we set β = 2 * 10⁻⁶ and γ = 10⁻³.

The MSE loss is defined as

$$l_{MSE}=\frac{1}{N}{\Vert I^{gt}-I^{gen}\Vert }_2^2$$ (4)

where N is the number of images in a batch, $\Vert \cdot {\Vert }_2^2$ is the squared L2 norm, and I^gen = G(I^prev, I^next) represents the image produced by the generator (G) from the inputs I^prev and I^next.

The perceptual loss guides the generator to learn edge information by comparing features extracted from the ground truth and generated images by the pretrained VGG19 network (Simonyan and Zisserman, 2015). The perceptual loss is defined as

$$l_{\mathit{\text{percep}}}=\frac{1}{N}{\Vert VGG\left(I^{gt}\right)-VGG\left(I^{gen}\right)\Vert }_2^2$$ (5)

where VGG(·) returns features after the 12th convolutional layer of the VGG19 network.

The loss l_GAN is defined as

$$l_{GAN}=\sum _{n=1}^N\left(-\text{log}D\left(I_n^{gen}\right)\right)$$ (6)

where D represents the discriminator, which returns the probability that the input is a ground truth image. Minimizing l_GAN is equivalent to maximizing the probability that D will misclassify the generated image as ground truth, which happens when the discriminator cannot differentiate between I^gt and I^gen.

The discriminator component of the GAN loss is defined as

$$l_D=\sum _{n=1}^N\left(\text{log}D(I_n^{gt})+\text{log}(1-D(I_n^{gen}))\right)$$ (7)

The goal of l_D is to classify its input images correctly. Note that the second term of l_D has the opposite effect of l_GAN, which makes the joint optimization of l_GAN and l_D drive adversarial learning and push the distribution of generated images toward the distribution of ground truth images.

3.4. Training the MRI Super-Resolution Network

The MRI volumes in P1 were preprocessed and split into training (252 volumes) and validation (63 volumes) sets, which were used to train the version of the miSRGAN that super-resolved MRI volumes (miSRGAN^M). The training was done in two stages. First, the generator was trained independently with β = 0 and γ = 0 for 20 epochs with a learning rate of α = 10⁻⁴ and an exponential learning rate decay of 0.95. The generator’s task of creating new images is more complex than the discriminator’s task of classifying a given image. In order to level the field between the generator and discriminator, independent training ensures that the generator already creates coherent images at the beginning of adversarial training. Second, both the generator and discriminator were adversarially trained for 50 epochs, with β = 2 * 10⁻⁶ and γ = 10⁻³.

3.5. Training the Histopathology Super-Resolution Network

To train the miSRGAN for histopathology super-resolution (miSRGAN^H), we used transfer learning due to the small amount of available data. Transfer learning was performed by finetuning miSRGAN^M with five-fold cross-validation to super-resolve the histopathology image stacks. First, we modified the architecture to work with RGB histopathology images. Specifically, we modified the following layers to have three channels-instead of one channel: the two input layers and the output layer of the generator, and the input layer of the discriminator. Second, we trained the modified layers from random initialization for 30 epochs with α = 10⁻³ while the remaining layers were frozen. Third, all layers were finetuned for 75 epochs with α = 10⁻⁴.

3.6. MRI-Histopathology Registration

We registered histopathology images to the corresponding MRI in three steps, using data from Cohort P2 (Fig. 4). First, consecutive pairs of MR images were fed into the trained generator of miSRGAN^M to create center images. The center images were interleaved with the input images to create the super-resolved volume. The generator is fully convolutional, thus volumes can be super-resolved without resizing. Each pass through the generator creates a 2x improvement in through-plane resolution. In this study, the MRI through-plane resolution was improved from 4.0mm to 1.0mm via two passes through the trained generator.

Fig. 4: Registration Workflow.

Pairs of consecutive images are passed through the trained generators of miSRGAN^M (top) and miSRGAN^H (bottom) producing super-resolved versions of the MRI and histopathology volumes. These volumes are then aligned using rigid, affine and deformable registration.

Second, pairs of histopathology slices were fed into the finetuned generator of miSRGAN^H (Fig. 4). As with the MRI volumes, the histopathology volume through-plane resolution was improved from 4.0mm to 1.0mm. We also super-resolved the masks of the prostate and urethra in both MRI and histopathology volumes as well as the cancer segmentation from the histopathology images. The super-resolution for the prostate, cancer, and urethra masks were performed by passing the binary masks through the miSRGAN^M trained on MRI images, followed by thresholding and morphological image processing. Both the MRI and histopathology masks were super-resolved in the same way. The masks were post-processed by applying thresholding, morphological closing and median filtering.

Third, we performed the 3D registration of super-resolved histopathology volumes to super-resolved MRI (Fig. 4). Prior to registration, the prostate was masked in both MRI and histopathology volumes. We used Simple ITK (Lowekamp et al., 2013; Yaniv et al., 2017) to perform multi-resolution affine and deformable registrations. The affine registration used the prostate masks as the input and the sum of squared differences as the loss function. The deformable registration used the masked images as the input and mutual information as the loss function. The deformable registration was based on free form deformation using a grid size of 12×12×12. The estimated affine and deformable transformations were composed to deform segmentations of the prostate, cancer, and urethra from the super-resolved histopathology volume onto the super-resolved MRI volume. We compared the registration of super-resolved volumes to the registration of unprocessed 2.5D volumes and 3D interpolated volumes. These interpolated volumes were created by projecting the 2.5D volumes onto the super-resolved volume space, using either nearest neighbor, linear, or BSpline interpolation methods.

3.7. Evaluation Metrics

The metrics outlined below were evaluated on the results of our approach and related work, and were compared using the Student’s t-test with Bonferroni multiple comparison correction.

Super-resolution:

We calculated the Peak Signal-to-Noise Ratio (PSNR) between six super-resolved volumes in Cohort P2 and 3D T2w MRI acquisitions from the same patients to evaluate the quality of the super-resolved MRI compared to existing clinical 3D MRI volumes. PSNR is defined as

$$\mathit{\text{PSNR}}\left(V^{gt},V^{gen}\right)=10{\text{log}}_{10}\left(\frac{I_{\mathit{\text{max}}}^2}{{\Vert V^{gt}-V^{gen}\Vert }_2^2}\right)$$ (8)

where I_max represents the maximum possible intensity of the image, V^gt represents the 3D T2w MRI volume and V^gen represents the super-resolved volume.

Registration:

We evaluated the 3D registration with two metrics. We computed the 3D Dice Similarity Coefficient of the registered histopathology images and MRI:

$$\mathit{\text{Dice}}\left(M^M,M^H\right)=\frac{2|M^M\cap M^H|}{|M^M|+|M^H|}$$ (9)

where M^M represents the prostate mask on MRI, M^H represents the prostate mask on the histopathology volume, and | · | is the cardinality of the set. This metric quantifies the overlap of the 3D prostate regions in the MRI and histopathology volumes, where a coefficient of 0 indicates no overlap, and a coefficient of 1 indicates perfect overlap.

We evaluated the alignment of the urethra in the MRI and histopathology volumes using the landmark distance metric:

$$\mathit{\text{Dist}}\left(U^M,U^H\right)=\frac{1}{N}{\Vert C\left(U^M\right)-C\left(U^H\right)\Vert }_2$$ (10)

where U^M represents the urethra segmentation in the MRI, U^H represents the urethra segmentation in the histopathology volume, N represents the total number of slices, and C(·) returns the center of mass.

Similar to Kalavagunta et al. (2014) that used landmarks for the evaluation of 2D registration, we used prostate cancer as a spatial landmark for 3D registration for two reasons. First, prostate cancer is contiguous in 3D space and is therefore suitable for the evaluation of 3D image registration. Second, one major goal of our super-resolved registration is to map the ground truth cancer labels from histopathology images onto MRI.

We evaluate the accuracy of the alignment of the cancer regions by the Dice coefficient and the Average Hausdorff Distance (AVD). The AVD between a cancer mask C^M on MRI and a cancer mask C^H on histopathology is given by:

$$AVD\left(C^M,C^H\right)=(\frac{1}{|C^M|}\sum _{x\in C^M}\underset{y\in C^H}{\text{min}}\Vert x-y{\Vert }_2+\frac{1}{|C^H|}\sum _{y\in C^H}\underset{x\in C^M}{\text{min}}\Vert x-y{\Vert }_2)/2$$ (11)

where | · | denotes the cardinality (number of elements) of a set.

The cancer was manually annotated on histopathology images by our expert pathologist (CAK, 10 years of experience) using the Aperio ImageScope software (Leica Biosystems, Nussloch, Germany). MRI lesions were manually outlined using the software ITK-Snap (Yushkevich et al., 2006) on T2w MRI for 18 patients by two body imaging radiologists (NM - 6 years of experience, AJ - 5 years of experience). The radiologists had access to the whole mount histopathology images. The following exclusion criteria was applied to handle inconsistency between the radiologists and pathologists annotations: (1) the size of two cancer labels of the same region differs by more than 100%, (2) there is no overlap between two cancer labels of the same region, (3) cancer labels are too small (i.e., less than 10 pixels).

4. Results

4.1. MRI Reconstruction via Super-Resolution

The generated images in the super-resolved MRI volumes maintain the in-plane image quality of the input images in addition to visually improving the through-plane continuity of the prostate, including cancer and anatomic landmarks (Fig. 5, Movie M1). Fig. 5a shows two consecutive MR images that have abrupt transitions at the boundary of the prostate, transitional zone, and cancer regions. The generated images provide a clear transition for the internal anatomic landmarks and the prostate boundary, which is missing in the input 2.5D MRI volume. The continuity added by the generated images translates to a through-plane resolution improvement in the super-resolved volumes relative to the input volumes (Fig. 6). The super-resolved through-plane view is visually closer to the T2w sagittal MRI and has diminished pixelation artifacts, contributing to improved edge fidelity.

Fig. 5: MRI Super-Resolution Strategy.

(a) miSRGAN^M takes two input images (I^prev and I^next) and creates center images during each iteration. (b) One image is created during the first iteration and (c) two more images are created during the second iteration. (d) The final super-resolved volume is constructed by interleaving the input and the images generated in iterations 1 and 2. The image boundary color emphasizes the interleaving order.

Fig. 6: Sagittal view of (a) 2.5D T2w Axial MRI, (b) 3D T2w MRI volumes, and (c) super-resolution, (d) nearest neighbor interpolation, (d) linear interpolation, and (e) third-order B-Spline interpolation of (a).

The super-resolution approach (b) reduced the pixelation artifacts present in the input volume (a). Note the relative visual similarity between (b) and (c). The PSNR for (c), (d), (e), and (f) relative to the ground-truth slice (b) are 28.85dB, 28.43dB, 28.47dB, and 28.59dB, suggesting that the super-resolved image is the closest to the ground truth image.

For six cases, a 3D T2w MRI was acquired as part of clinical protocol (Table 1). We used these 3D acquisitions to compare the super-resolved and nearest-neighbor, linear, and BSpline interpolated MRI via PSNR. The PSNR between super-resolved volumes and corresponding 3D T2w MRI volumes was significantly higher than the PSNR between interpolated volumes and 3D MRI (t-test with Bonferroni correction, p < 0.01, Fig. 7). The super-resolved volumes have an average PSNR of 32.15 ± 0.44db compared to an average of 28.69 ± 0.83db, 30.11 ± 1.02db, 30.16 ± 1.01db, for nearest neighbor, linear, and BSpline interpolations, respectively. Higher PSNR indicates that our super-resolved MRI is more similar to 3D T2w MRI compared to interpolated MRI volumes.

Fig. 7:

Peak Signal to Noise Ratio was calculated between nearest neighbor, linear, BSpline interpolated volumes or super-resolved volumes and presurgical 3D T2w MRI obtained for the same patient (N=6, ** p < 0.01).

4.2. Histopathology Reconstruction via Super-Resolution

We reconstructed the information between slices in the histopathology image stacks via super-resolution using miSRGAN^H. The super-resolved histopathology volumes exhibit continuity in major internal features and in the shape of the prostate (Fig. 8, Movie M2). The super-resolved through-plane view of the histopathology volume has continuous edges, which are missing in the through-plane view of the 2.5D volume (Fig. 9). Note the continuous cancer (outlined in yellow in Fig. 9b), which is difficult to discern in the input image stack (Fig. 9a). Unlike MRI, where 3D volumes occasionally exist, tissue sectioning is performed at relatively wide intervals to preserve tissue for future tests and to maintain the practicality of pathologist review. Therefore, resected prostate tissue is typically sectioned at roughly 4mm increments and a true 3D histopathology volume does not exist, preventing PSNR based evaluation of our super-resolved histopathology volume.

Fig. 8: Histopathology Image Super-Resolution Workflow.

The final super-resolved volume is constructed from consecutive histopathology images and generated images from two passes through the finetuned miSRGAN^H.

Fig. 9: Sagittal view of the (a) input histopathology, (b) super-resolved histopathology volume, (c) histopathology upsampled by linear interpolation, and (d) histopathology upsampled by third-order B-spline interpolation.

The yellow outline shows cancer, not easily discernible in (a), (c) and (d), but shown as a continuous region in (b).

Fig. S3 shows an example of a real (ground truth) and a super-resolved histopathology image. Unlike our typical application where we super-resolve images that are only 4mm apart (Fig. S4), to create the results in Fig. S3 we applied miSRGAN^H for two slices that are 8mm apart which allows us to show the ground truth image. Many similarities exist between the ground truth and super-resolved image, yet some differences are also observed. Despite the discrepancies, we show next that the super-resolved histopathology volume facilitates a better alignment between MRI and histopathology images.

4.3. Registration

We applied 3D registration methods to align the MRI and histopathology image stack. Our registration obtained a better alignment when using the super-resolved MRI and histopathology volumes compared to using the unprocessed 2.5D MRI and histopathology image stack (Fig. 10). The improved alignment can be seen at the prostate boundary and in the cancer labels. The cancer label in Fig. 10c appears misaligned because of the partial volume artifacts caused by the 3D registration of the histopathology image stack to the 2.5D MRI volume. These artifacts occur when information from two different histopathology images is projected onto the same MR image. This error is exemplified in Fig. 10c, where the cancer outlines in the lower left and upper right corners of the prostate are obtained from different histopathology slices. The learned through-plane information in the super-resolved volumes prevents such partial volume artifacts and results in cancer labels on the super-resolved MRI that are consistent in 3D (Fig. 10d). Figure 11 shows the super-resolved registration for all whole-mount histopathology slices from the apex to the base of the same subject used in Fig. 10. From the results in Fig. 11, we can see that the registration of the super-resolved MRI and histopathology volumes has accurately aligned both the prostate boundaries and features inside the prostate.

Fig. 10: Visual assessment of multi-modal alignment.

Registration of (a) a histopathology image stack to 2.5D MRI and (b) a super-resolved histopathology volume to super-resolved MRI. The estimated transformation was applied to the cancer label from the (c) histopathology image stack onto MRI (blue) and (d) super-resolved histopathology volume onto MRI (yellow). Note the accurate alignment and labels in (b,d) compared to the partial volume effect (blue arrow in a) and misalignment of the cancer labels in (c).

Fig. 11:

Registration results showing whole-mount histopathology slices from apex to base of a representative subject in Cohort P2 using super-resolved volumes. (Row 1) Input histopathology slices; (Row 2) Histopathology slices registered to T2w MRI; (Row 3) Corresponding T2w MRI; (Row 4) MRI overlaid with registered histopathology images.

We evaluated the 3D Dice coefficient of the prostate gland, distance between the urethra, 3D Dice Coefficient of the cancer regions, and the average Hausdorff distance between cancer regions after registration (Table 2). The prostate Dice coefficient, cancer Dice coefficient, and cancer distance of the super-resolved registration were significantly better than that of the 2.5D registration and other interpolated registrations (p < 0.05, Fig. 12). A previous approach (RAPSODI) resulted in a higher Dice coefficient of the prostate gland than our approach, however this method depends on manually identifying matching histopathology slices to MRI followed by 2D registrations. In addition, our super-resolved registration had the highest sample mean of the Dice coefficient and the lowest sample mean of the average Hausdorff distance of the cancer regions, and smaller sample mean of the urethra distance compared to RAPSODI. These metrics indicate that the proposed super-resolution method yields alignments that are accurate, with regard to prostate boundaries and internal landmarks including prostate cancer.

Table 2:

Evaluation of the registration between histopathology and MR images using various methods.

Method	Dice (Prostate)	Urethra Distance (mm)	Dice (Cancer)	AVD (mm)
2.5D	0.93 ± 0.02	2.79 ± 0.88	0.56 ± 0.15	0.77 ± 0.52
Nearest Neighbor	0.93 ± 0.01	2.74 ± 0.69	0.58 ± 0.11	0.69 ± 0.46
Linear	0.92 ± 0.02	2.63 ± 0.55	0.61 ± 0.08	0.69 ± 0.48
B Spline	0.89 ± 0.02	2.47 ± 0.52	0.58 ± 0.08	0.79 ± 0.50
RAPSODI	0.96 ± 0.01	2.94 ± 0.73	0.65 ± 0.06	0.50 ± 0.15
Super-resolution (ours)	0.95 ± 0.01	2.60 ± 0.57	0.68 ± 0.06	0.43 ± 0.11

AVD: Average Hausdorff Distance (see Eq. (11) for the definition of AVD).

Fig. 12: Registration Evaluation.

Boxplots of (a) 3D Dice Similarity Coefficient for the prostate gland (N=59), (b) Urethra Landmark Distance (N=19), (c) 3D Dice Similarity Coefficient for cancer regions (N = 18), and (d) cancer landmark distance (N = 18). *: p ≤ 0.05, ***: (p < 0.001), ns: p > 0.05.

5. Discussion

We have introduced an approach for the improved registration of histopathology images to MRI, which allows the accurate mapping of cancer labels from histopathology to MRI. Such labels are needed to train machine learning models to detect cancer on MRI (Bhattacharya et al., 2020). Our proposed method addresses two challenges for registration of MRI and histopathology images: missing through-plane information in both modalities and imperfect slice-to-slice correspondence between the two modalities. We overcame these challenges by reconstructing 3D volumes via the proposed super-resolution approach for both MRI and histopathology images prior to applying 3D registration methods. miSRGAN^M learned to generate information between slices in the MRI volume while the finetuned miSRGAN^H learned to generate information between histopathology images. These super-resolved volumes have an improved through-plane resolution, resulting in perceivable 3D continuity without pixelation artifacts. Furthermore, the generated slices in the histopathology volume are anatomically similar to their neighbor slices, since the network generates new slices based on the information from the two adjacent slices, which could be potentially useful in other applications, such as in other registration methods or as strategies for data augmentation. Quantitatively, the improvement brought by super-resolution was also supported by the high PSNR between the super-resolved MRI volumes and corresponding clinical 3D T2w MRI volumes. Additionally, even if 3D clinical MRI are obtained, these volumes still need to be registered to the corresponding histopathology image stacks, which would still need to be super-resolved prior to 3D registration. Finally, our GAN based model produces results with a significantly higher PSNR and SSIM than using the interpolating generator alone, which produces over-smoothed images (Sood and Rusu, 2019).

The improvements in resolution and 3D continuity allowed a more accurate 3D registration between the super-resolved MRI and histopathology volumes. The registration between the super-resolved MRI and histopathology volumes had the best cancer alignment, the second highest Dice coefficient and the second lowest urethra distance, indicating that both the prostate boundary and internal landmarks are well aligned. The 3D registration obviates the need for slice-to-slice correspondence by registering the super-resolved MRI and histopathology volumes as a whole.

Our study has four main limitations. First, we tested our approach in a dataset where the tissue was sectioned using 3D printed molds, which is standard-of-care at our institution. It remains to be tested whether the method can be applied to specimens collected via free-hand sectioning of the excised tissue. Second, our approach requires the segmentation of the prostate gland in both the MRI and histopathology images to facilitate registration. Automatic segmentation of the prostate on MRI and histopathology images will be explored in future work to limit the need for manual labeling. Third, unavailability of code from prior methods prevented us from comparing our results to prior work beyond our own RAPSODI platform (Rusu et al., 2020). Forth, in our study we have not evaluated the effect of the inter-reader variability in segmenting the prostate or cancer regions as none of the segmentations were available from multiple readers. Finally, our data does not have a 3D sequence for each patient to serve as a ground truth for our experiments (only available for 6 subjects included in our study), while the ground truth for histopathology does not exist.

To the best of our knowledge, we are the first to super-resolve histopathology images to create a 3D histopathology volume that captures data otherwise not available due to sectioning. Moreover, the miSRGAN provides a novel approach to through-plane super-resolution in medical images that does not depend on orthogonal acquisitions or additional information. Our approach is manufacturer agnostic in that the MRI volumes used during training and testing were obtained from different manufacturers. The consistency in performance regardless of the source of MRI highlights the strong generalizability of the miSRGAN. Furthermore, although it is not part of our current study, we anticipate that our approach will be applicable in the context of histopathology images prepared as quadrants once they are stitched to create pseudo-whole mounts (Chappelow et al., 2011b).

The accurate registration between the MRI and histopathology images has important clinical implications. The cancer labels from the super-resolved and registered histopathology volume can be projected onto the super-resolved MRI, creating a labeled dataset. This dataset can be used to train a machine learning algorithm to accurately detect and outline clinically significant cancer on MRI.

6. Conclusion

We have introduced a novel super-resolution generative adversarial network to reconstruct MRI and histopathology volumes with higher through-plane resolution. We showed that 3D registration of the super-resolved MRI and histopathology volumes produced more accurate results than other interpolation methods. The accurate cancer labels on MRI from our 3D registration will facilitate the development of radiomic and deep learning approaches to automatically detect prostate cancer on MRI. Such an approach may assist radiologists in their interpretation of MRI, thereby facilitating early detection of prostate cancer, targeting during biopsy, and local treatment.

• Our super-resolution networks can be used to reconstruct information useful for the 3D reconstruction of MRI and histopathology image stacks.

• Accurate registration without the need for slice-to-slice correspondence reduces required human input.

• Accurate 3D registration allows the mapping of cancer outlines from the histopathology images onto the MRI to create labels for downstream learning tasks.