RAPHIA: A Deep Learning Pipeline for the Registration of MRI and Whole-mount Histopathology Images of the Prostate

Abstract

Image registration can map the ground truth extent of prostate cancer from histopathology images onto MRI, facilitating the development of machine learning methods for early prostate cancer detection Here, we present RAdiology PatHology Image Alignment (RAPHIA), an end-to-end pipeline for efficient and accurate registration of MRI and histopathology images. RAPHIA automates several time-consuming manual steps in existing approaches including prostate segmentation, estimation of the rotation angle and horizontal flipping in histopathology images, as well as estimation of MRI-histopathology slice correspondence. By utilizing deep learning registration networks, RAPHIA substantially reduces computational time. Furthermore, RAPHIA obviates the need for a multimodal image similarity metric by transferring histopathology image representations to MRI image representations and vice versa. With the assistance of RAPHIA, novice users achieved expert-level performance, and their mean error in estimating histopathology rotation angle was reduced by 51% (12 degrees vs 8 degrees), their mean accuracy of estimating histopathology flipping was increased by 5% (95.3% vs 100%), and their mean error in estimating MRI-histopathology slice correspondences was reduced by 45% (1.12 slices vs 0.62 slices). When compared to a recent conventional registration approach and a deep learning registration approach, RAPHIA achieved better mapping of histopathology cancer labels, with an improved mean Dice coefficient of cancer regions outlined on MRI and the deformed histopathology (0.44 vs 0.48 vs 0.50), and a reduced mean per-case processing time (51 vs 11 vs 4.5 minutes). The improved performance by RAPHIA allows efficient processing of large datasets for the development of machine-learning models for prostate cancer detection on MRI. Our code will be publicly available at: https://github.com/pimed/RAPHIA.

Graphical Abstract

1. Introduction

Prostate cancer is the most diagnosed cancer among American men and the second leading cause of cancer death across all races (Siegel et al., 2022). Magnetic resonance imaging (MRI) has become a crucial tool for noninvasive detection and localization of prostate cancer. Despite the widespread use of the Prostate Imaging Reporting and Data System (PI-RADS) for standardizing prostate MRI interpretation (Barentsz et al., 2016), it remains challenging due to the subtle differences between normal, benign, and malignant tissues (Ahmed et al., 2017). Consequently, the diagnostic concordance among radiologists is relatively low $(\kappa =0.46-0.78)$, with large variations in reported sensitivity (58%-96%) and specificity (23%-87%) (Ahmed et al., 2017). Radiologists’ performance in interpreting prostate MRI could be improved by using artificial intelligence (AI)- based solutions that automatically and reliably localize clinically significant prostate cancer on MRI. While such automated approaches have been previously proposed (Cao et al., 2019; McGarry et al., 2019; Bhattacharya et al., 2020; Saha et al., 2021; Seetharaman et al., 2021; Bhattacharya et al., 2022c,a), they are often trained with radiologist cancer labels with larger inter-reader variations or pathologist cancer labels from a limited number of subjects. A recent study found that the training of AI models for prostate cancer detection on MRI benefits from using labels of indolent and aggressive cancer mapped from histopathology images onto MRI (Bhattacharya et al., 2022b). Such cancer labels can be obtained by registering whole-mount histopathology image stacks to presurgical MRI and mapping the ground truth cancer labels from histopathology onto MRI for patients who underwent radical prostatectomy (Rusu et al., 2020).

MRI-pathology registration is challenging for various reasons. Firstly, prostate tissue appearance on MRI and histopathology significantly differs in intensity type (grayscale vs. RGB color) and resolution (millimeters vs. micrometers). Secondly, histological processing causes prostate shrinkage and geometric distortion, resulting in nonrigid deformation between MRI and histopathology. Lastly, the registration process often involves numerous manual steps, rendering it labor-intensive, time-consuming, and limited to expert users familiar with both MRI and histopathology. Despite efforts to address these challenges, current registration approaches remain limited. All existing approaches rely on some manual processing steps, including cognitive alignment (Ward et al., 2012; Sumathipala et al., 2018; Cao et al., 2019), prostate gland segmentation (Chappelow et al., 2011; Ward et al., 2012; Rusu et al., 2020; Shao et al., 2021a), annotation offiducial markers(Ward etal., 2012),estimation of histopathology rotation and flipping (Li et al., 2017; Wu et al., 2019; Rusu et al., 2020; Shao et al., 2021a; Zamboglou et al., 2021), and estimation of MRI-histopathology slice correspondences (Chappelow et al., 2011; Rusu et al., 2020; Shao et al., 2021a). Cognitive alignment involves using histopathology images as a reference to mentally map a pathologist’s cancer labels onto MRI scans. However, since the extent of cancer on histopathology images often exceeds what is visible on MRI, this cognitive alignment might result in labels that underestimate the actual extent of cancer Furthermore, many approaches require additional ex vivo MRI (Ward et al., 2012; Reynolds et al., 2015; Li et al., 2017; Wu et al., 2019) as an intermediate modality to facilitate the registration process. Additionally, most registration approaches are computationally expensive due to the use of iterative optimization methods in the registration of new pairs of images (Chappelow et al., 2011; Losnegård et al., 2018; Li et al., 2017; Rusu et al., 2020). Recently, two deep learning approaches (Shao et al., 2021a,b) were developed to expedite MRI-histopathology registration. Unlike traditional methods, trained deep-learning registration networks can register unseen image pairs without additional iterative optimization, boosting execution time by 20 to 60 times. However, the two approaches still involve manual preprocessing steps and may take several minutes to run. Due to the absence of ground-truth point-to-point correspondences between MRI and histopathology images and an effective multimodal similarity metric, previous deep learning models (Shao et al., 2021a,b) used synthetic monomodal image pairs for training, and their performance could be enhanced by training with multimodal image pairs.

Prior methods for registering MRI and histopathology images of the prostate traditionally required several manual steps, making them time-consuming and labor-intensive. The goal of this study is to develop a fully automated pipeline while preserving the accuracy of MRI-histopathology registration. We hypothesize that our registration pipeline can achieve accuracy not inferior to current state-of-the-art methods while significantly reducing processing time In this paper, we have developed the RAdiology PatHology Image Alignment (RAPHIA) pipeline to address the limitations of existing registration approaches. RAPHIA is a human-in-the-loop image registration pipeline that allows human intervention to improve registration results. Users have the flexibility to correct or refine AI predictions at several intermediate steps, including the estimation of rotation angle and horizontal flipping of each histopathology slice, estimation of MRI-histopathology slice correspondences, and prostate segmentation on MRI images. If the user chooses to update AI predictions, our RAPHIA pipeline will rerun all pairwise image registrations based on the updated predictions, without involving optimizations based on traditional algorithms. This feature enables expert users to achieve optimal registration results while significantly reducing the total registration time.

Our RAPHIA approach makes the following major contributions:

• RAPHIA streamlines MRI-histopathology image registration by automating manual steps in existing methods.

• RAPHIA uses a geometry-preserving image-to-image translation network to avoid the need for hand-crafted multimodal image similarity metrics.

• Unlike prior approaches that are limited to expert users, RAPHIA is user-friendly and can assist novice users to achieve expert-level performance.

• RAPHIA is a human-in-the loop pipeline that gives advanced users the flexibility to adjust AI predictions for improved registration results.

2. Methods

2.1. Dataset and Users

The procedures carried out on humans were in accordance with the ethical standards of the World Medical Association (Declaration of Helsinki). This retrospective chart review study was approved by the Stanford University institutional review board. As a chart review of previously collected data, patient consent was waived. This study included 206 patients with biopsy-confirmed prostate cancer who had a presurgical MRI and digital histopathology images from radical prostatectomy. We used 186 patients for training and validation and 20 patients for testing. Table 1 summarizes detailed imaging characteristics of MRI and histopathology.

Table 1.

MRI and histopathology data characteristics.

Modality	MRI	Histopathology
Manufacturer	GE Healthcare	-
Coil Type	32-channel array coils	-
Acquisition Characteristics	Echo time: 0.122–0.13 s Relaxation time: 3.9–6.3 s	3D printed mold H&E staining
Image Height and Width	256 or 512	1663–7556
Number of Slices	24–43	3–11
In-plane Resolution	0.27–0.94 mm	0.0081 or 0.0162 mm
Through-plane Resolution	3.0–5.2 mm	3.0–5.2 mm

The extent of cancer on histopathology images were segmented by a genitourinary pathologist (CK) with more than 10 years of experience. The urethra, visible anatomic landmark (e.g., benign prostate hyperplasia nodules and ejaculatory ducts), and prostate capsule on MRI and histopathology images were manually segmented by our imaging scientists (MR and WS) with 9 years of research experience in image registration. One expert image registration user (WS) and three novice image registration users (SJCS, NG, and IB) with less than 1 year of experience were involved in the evaluation of our RAPHIA registration pipeline.

2.2. Overview of RAPHIA

Given a stack of histopathology images and a T2-weighted (T2-w) MRI volume, our goal is to find a one-to-one correspondence map between each histopathology image and its corresponding MRI slice. Figure 1 gives an overview of our RAPHIA registration pipeline. RAPHIA achieves-MRI-histopathology-registration-in-four-steps First, our deep-learning segmentation networks segment the prostate capsule on the MRI and histopathology images (see Section 2.3). Second, our AI models estimate the rotation angle and horizontal flipping that exist in each histopathology image (see Section 2.4). The estimated rotation and flipping transformations are then applied to the histopathology images and the prostate segmentations. Third, our improved geometry-preserving image-to-image translation network translates the MRI into histopathology images and the histopathology images into MRI(see Section2.5).Our AIalgorithm then determines slice correspondences between the MRI and histopathology images (see Section 2.6). Finally, using our multimodal image registration neural network, each pair of histopathology and MRI images is set as the input and output of an affine transformation and a thin-plate spline transformation (see Section 2.7).

Figure 1:

An overview of our RAPHIA registration pipeline. Refer to Section 2.3 for prostate segmentation (step 1), Section 2.4 for histopathology rotation angle and flipping prediction (step 2), Section 2.6 for slice correspondence estimation (step 3), and Section 2.7 for multi-model registration utilizing image-to-image transition (step 4).

2.3. Prostate Capsule Segmentation

2.3.1. Overview

Current registration methods typically use manual prostate segmentations for MRI and histopathology image registration, but this is labor-intensive and time-consuming. RAPHIA automates prostate segmentation using deep learning approaches. We briefly describe our segmentation networks below.

2.3.2. Prostate Segmentation on T2-weighted MRI

ProGNet, a network for prostate segmentation on MRI, is based on the U-Net architecture (Soerensen et al., 2021). This convolutional network integrates information from three consecutive T2w slices and predicts prostate segmentations on the middle slice, thereby learning the 2.5D volumetric continuity of the prostate on MRI.

2.3.3. Prostate Segmentation on Histopathology Images

We adapted the Dilated Residual-UNet (Vesal et al., 2019) for prostate segmentation on histopathology images. It has four encoder and decoder blocks, separated by a bottleneck block. Each encoder/decoder has two 2D convolutional layers, and a residual connection for enhanced feature extraction. The bottleneck block uses dilated convolution layers (Yu and Koltun, 2015) to capture local and global context, with increasing dilation rates from 1 to 8, summing feature maps to cover the input field of view.

2.4. Prediction of the Rotational Angle and Horizontal Flipping of Histopathology Images

2.4.1. Overview

Mounting tissue sections on glass slides may result in the rotation and horizontal flipping of histopathology images (Cimadamore et al., 2021). To differentiate between the left and right sides of the prostate in these images, pathology preparation involves using black ink to mark the left side and blue ink for the right side during histological preparation. The histopathology image marked with ink explicitly indicates the locations of the left and right sides of the prostate, aiding in determining whether flipping the histopathology image is necessary before aligning it with the corresponding MRI image. Registration methods typically use local optimizers and cannot correct large rotations. To achieve accurate registration results, users often first manually estimate the rotation angle and horizontal flipping and then fine-tune the estimates by rerunning the registration algorithm several times to obtain satisfactory results (Rusu et al., 2020; Shao et al., 2021a,b).

In this paper, we developed an approach to automatically predict the rotation angle and horizontal flipping in histopathology images (see an overview in Fig. 2). During histopathological preparation, the left side of the prostate was marked with blue ink and the right side with black ink to distinguish the sides on the slide. The histopathology image and its ink image were fed into our model to predict the rotation angle (see Section 2.4.1). The estimated rotation transform was then applied to rotate the ink image. There is no need to flip the histopathology image if the center of mass of blue ink pixels in the rotated ink image was on the right side, and vice versa.

Figure 2:

Approach for estimating rotation angle and horizontal flipping in histopathology images.

2.4.2. Ink Extraction

We have developed a traditional approach for ink extraction in two steps. First, we extracted the blue and black inks from histopathology images based on empirically defined thresholds. For blue ink, we used RGB thresholds of < 200, < 200, and > 80, and red/green and red blue ratios of [1.05, 1.20] < 200, and [1.10, 1.40]. For black ink, we used RGB, thresholds of $R<200$, $G<200$, and $G<200$, and red/green and red/blue ratios of [0.95,1.05]. Second, we applied binary dilation operations (1 iteration, kernel size = 10 × 10 pixels, pixel size = 0.016 mm) to the segmented blue and black inks.

Given the computational inefficiency of our traditional ink extraction algorithm, we developed an alternative ink extraction method utilizing the pix2pix model. Training the ink extraction network requires paired histopathology and ink images. However, manually segmenting inks from histopathology images is laborious. As an alternative, we used the traditional extraction algorithm to produce accurate ink images from histopathology samples. We developed an ink extraction algorithm using the pix2pix model for better performance (Isola et al., 2017). The pix2pix model translates images from a source domain to a target domain while maintaining content. We trained it on 1138 histopathology image pairs with ink images from our traditional method. The ink generator was trained by minimizing the adversarial loss and intensity differences between generated images and ground truth.

2.4.3. Rotation Prediction Network

We have developed a deep neural network that takes a histopathology image and an ink image as the input, and outputs the rotation angle of the histopathology image. The ground truth rotation angles (0 degrees) of the histopathology images, used for supervised training of the angle prediction model, were manually determined to match the orientation of MRI during the development and application of our prior RAPSODI registration method (Rusu et al., 2020). Features in the input images are extracted by a pretrained VGG16 network (Simonyan and Zisserman, 2015). The extracted feature vectors are concatenated and fed into a dense layer to regress to the rotation angle $\theta$. To train the network, we proposed a novel cost function that takes into consideration the periodic property of the rotation angle

$$L\left({\theta }_{gd},{\theta }_{est}\right)={\left(\frac{1-\cos \left({\theta }_{gd}-{\theta }_{est}\right)}{2}\right)}^{\frac{1}{4}}$$ (1)

where ${\theta }_{gd}$ is the ground-truth rotation angle and ${\theta }_{est}$ is the smaller the difference between the ground-truth angle and the predicted angle, the smaller the cost $L\left({\theta }_{gd},{\theta }_{est}\right)$.

We implemented our angle prediction network in two stages. In the first stage, images were down sampled to 64×64 for an initial estimate of the rotation angle. In the second stage, we applied the initial estimate to the images and down-sampled them to 224 × 224 to derive a more precise angle estimate. We trained using angle-corrected histopathology and ink images generated by our ink extraction network. We applied random rotations to images in each training batch and used these as input and ground truth. In stage one, angles ranged from −180° to +180°. In stage two, angles ranged from −15° to +15° to refine predictions.

2.5. Geometry-consistent Image-to-Image Translation Network

It is challenging to directly register MRI and histopathology images due to large differences between their imaging characteristics and resolutions. Inspired by the work of Arar et al. (Arar et al., 2020), we tackled this problem by transferring between MRI and histopathology image representations via image-to-image translation. The pix2pix model requires perfectly aligned MRI-histopathology image pairs and there-fore was not suitable. In addition, many image translation networks cannot guarantee the preservation of the geometry in the source images and therefore were not suitable for our image registration task. In this paper, we improved the prior geometry-consistent generative adversarial network (GcGAN) (Fu et al., 2019) (see Fig. 3) that does not assume aligned image pairs to translate MRI to histopathology and vice versa. Detailed descriptions of the GcGAN model can be found in the Appendix.

Figure 3:

Geometry-consistent and cycle-consistent generative adversarial network. The cycle consistency constraint forces the two generators $G_{MH}$ and $G_{HM}$ to be inverses of each other. The geometry consistency constraint forces the two generators to be topology-preserving.

We improved the data augmentation used in the original GcGAN study (Fu et al., 2019) by implementing the transformation $\varphi$ as random affine transformations with $x$ and $y$ scaling within [0.8, 1.2], rotation angle within [−18, 18] degrees, shearing within [−0.05,0.05], and $x$ and $y$ translations within [−10%,10%] of the image size. The extra flexibility in the transformation $\varphi$ yielded more geometry-preserving generators. The GcGAN network has two discriminators $D_H$ and $D_M$ that determine the realism of histopathology and MRI images, respectively. The generator has a standard encoder-decoder architecture, where the encoder consists of 2 convolutional layers with 6 residual blocks and the decoder consists of 2 transposed convolution layers. The discriminator is a convolutional PatchGAN classifier that only penalizes structure at the scale of patches (Isola et al., 2017). During the training, all images were resized to 1024×1024.

2.6. MRI-Histopathology Slice Correspondence

During histopathology imaging, most of the prostate specimen was cut in parallel to the axial plane except for the tissues at the apex and the base of the prostate. Thus, the number of MRI slices containing the prostate exceeds the number of histopathology slices (see Fig. 4). MRI-histopathology registration requires estimating slice correspondences between MRI and histopathology, equivalent to identifying the MRI slice corresponding to the first histopathology slice. Despite using patient-specific 3D printed molds (Turkbey et al., 2011) to cut the prostate in-plane with MRI, estimating MRI-histopathology slice correspondences remains challenging due to tissue distortion and imperfect cuts. Prior registration approaches (Rusu et al., 2020; Shao et al., 2021a,b) often started with a rough correspondence estimate via visual comparison, and then fine-tuned based on the registration results. For traditional approaches, this process could take tens of minutes since rerunning the registration process for each new estimate takes several minutes.

Figure 4:

Illustration of the problem of finding slice correspondences between MRI and histopathology. The contours (light blue) are prostate segmentations.

Since manual estimation of slice correspondences is laborious, time-consuming, and error-prone, we developed an algorithm to automatically determine MRI-histopathology slice correspondences. First, we segmented the prostate gland on MRI and histopathology images using approaches in Section 2.3. We corrected the rotation angle and horizontal flipping in histopathology slices using the approach described in Section 2.4.1. Then, MRI and histopathology images were masked using the prostate masks, denoted by $M_1,\cdots ,M_k$ and $H_1,\cdots ,H_n$, respectively. Our algorithm for estimating the starting MRI slice number is as follows. For each possible MRI slice number, $i$ in the set $\left\{1,2,\cdots ,k-n+1\right\}$, we registered the histopathology stack to a sequence of $n$ MRI slices starting from slice $i$ using our deep learning registration network described in Section 2.7. We measured the accuracy of the estimated slice correspondences between the histopathology stack $H_1,\cdots ,H_n$ and the MRI stack $M_i,\cdots ,M_{n+i-1}$ by the sum of the normalized correlation coefficients between the MRI and registered histopathology. The slice number of the starting MRI slice was chosen to be the one that yielded the highest normalized cross correlation.

2.7. Multi-modal Image Registration

We considered a deep-learning registration network as a function that takes two images, fixed and moving, as the input and directly outputs a geometric transformation that aligns the two images. It is challenging to train a multimodal registration network due to the lack of ground-truth spatial correspondences between the MRI and histopathology images, and the lack of an effective multimodal similarity loss function. Prior deep-learning networks (Shao et al., 2021a,b) were trained with monomodal synthetic image pairs generated by applying known transformations to MRI and histopathology images. Although, the trained networks have achieved descent multimodal image registration results, they are not true multimodal registration networks since they were trained with only monomodal image pairs.

In this paper, we tackled this problem from a different perspective. We used the modified geometry-consistent image-to-image translation network to generate multi-modal MRI-histopathology image pairs for the network training, as shown in Fig. 5. We fed each histopathology image $h_i$ into the generator $G_{HM}$ to translate it into an MRI image. A random transformation $\varphi$ was applied to deform the fake MRI image $G_{HM}\left(h_i\right)$ into $m_i^{\prime }$. Then 3-tuple ($h_i$, $m_i^{\prime }$, $\varphi$) was used as a multimodal training example, with $\varphi$ being the ground-truth transformation between the two images $h_i$ and $m_i^{\prime }$. Similarly, we also created a multimodal training example ($m_i$, $h_i^{\prime }$, $\psi$) for each MRI slice $m_i$ in the training dataset. During the training, each batch consisted of $N$ MRI and histopathology images, and we trained our registration network using the following cost function

$$C=\sum _{i=1}^N\left({\Vert G_{HM}\left(h_i\right)\circ {\varphi }^{\prime }-m_i^{\prime }\Vert }^2+{\Vert G_{MH}\left(m_i\right)\circ {\psi }^{\prime }-h_i^{\prime }\Vert }^2\right)$$ (2)

where ${\varphi }^{\prime }$, and ${\psi }^{\prime }$ are the predicted transformations between $h_i$ and $m_i^{\prime }$, and $m_i$ and $h_i^{\prime }$, respectively. The first term minimizes intensity differences between a real MRI image and the registered fake MRI image, while the second term minimizes differences between a real histopathology image and the registered fake histopathology image. These two terms enable unsupervised learning in our registration framework.

Figure 5:

Generating multi-modal training examples by image translation and random deformations. $h_i$ and $m_i$ are real histopathology and MRI images, $G_{HM}$ is the generator that translates a histopathology image into an MRI image, $G_{MH}$ is the generator that translates an MRI image into a histopathology image, and $\varphi$ is a random deformation.

Inspired by our prior work (Shao et al., 2021b), we implemented a two-stage registration neural network to estimate both the affine and nonrigid transformations between MRI and histopathology (Fig. 6). The input to the registration network was a histopathology image and an MRI image, both masked by their prostate segmentations. The feature extractor consisted of several convolutional layers from a pretrained VGG16 network. The extracted feature maps were fed into a global average pooling (GAP) layer to reduce their dimension from $N\times N$ to 1×1.FeaturevectorsfromtheGAP layers were concatenated and fed into a dense layer to regress to the parameter vector $\theta$. The number of parameters for $\theta$ was 6 for the affine registration and 72 for the deformable registration with a thin-plate spline grid size of 6×6.

Figure 6:

Registration neural network. $\theta$ is the vector that encodes an affine or a deformable transformation. We froze all layers of the VGG16 network except for the last two convolutional layers during the training.

2.8. Prior RAPSODI and ProsRegNet Registration Pipelines

We compared our RAPHIA registration pipeline with two other approaches: RAPSODI (Rusu et al., 2020) and ProsRegNet (Shao et al., 2021a). Both approaches require several manual preprocessing steps. RAPSODI is a traditional registration approach that optimizes a loss function iteratively to find a geometric transformation. ProsRegNet is a deep learning based registration approach. For both approaches, the slice correspondences between MRI and histopathology images were manually estimated based on the registration results. Both approaches allow the users to update the prostate segmentation, slice correspondences, and rotation angle and flipping in histopathology images, and then rerun the registration until desired result is achieved.

Both ProsRegNet and RAPHIA initiate the registration process by extracting image features from MRI and histopathology images using pre-trained convolutional neural networks. The main distinction lies in how these image features are utilized to estimate the transformation parameters. ProsRegNet computes a 4D correlation map between the two image feature maps, followed by the use of a regression network comprising multiple convolutional layers and a fully connected layer to estimate these parameters. In contrast, RAPHIA employs a global average pooling layer to significantly condense the image features to a 1×1 size. Subsequently, these features are concatenated into a vector, followed by a dense layer to regress toward the transformation parameter vector.

2.9. Evaluation Metrics

We used the Dice coefficient to assess the relative overlap between two binary segmentations, the Hausdorff distance to measure the distance between their boundaries. We used the mean landmark error (MLE) to measure how well landmarks on the MRI image were aligned with the corresponding landmarks on the histopathology image. We used the urethra deviation, which is computed as the Euclidean distance between the urethra centers on MRI and histopathology images, to evaluate how well the urethra was aligned.

We evaluated the performance of angle prediction by the following angle estimation error (AEE)

$$AEE=\frac{1}{N}\sum _{i=1}^N\arccos \left(\cos \left(a_i\ast \frac{\pi }{180}-p_i\ast \frac{\pi }{180}\right)\right)\ast \frac{180}{\pi }$$ (3)

where $N$ is the number of histopathology images, and $a_i$ and $p_i$ are the ground-truth angle and the predicted angle (in degrees) of $i$–th histopathology images, respectively.

We evaluated the performance of horizontal flipping prediction by the flipping estimation accuracy (FEA)

$$FEA=\frac{TP+TN}{N}$$ (4)

where TP = true positive, TN = true negative, and $N=128$, the total number of histopathology images.

We defined the slice correspondence estimation error (SCEE) as the mean of the absolute difference between the slice number of the estimated starting MRI slice and the slice number of the ground-truth starting MRI slice

$$SCEE=\frac{1}{M}\left|c_i-c_i^{\prime }\right|$$ (5)

where $M=20$ is the number of subjects, and $c_i$ and $c_i^{\prime }$ are the ground-truth MRI slice number and predicted slice number, respectively.

3. Experiments and Results

3.1. Experiment 1: Prostate Segmentation

In Experiment 1, we trained and evaluated deep learning models that automatically outlined the prostate on both MRI and histopathology. We used prostate segmentations from MRI and histopathology images annotated by two medical imaging experts (MR and WS) for a prior study (Rusu et al., 2020) as the ground truth. Algorithm performance was evaluated relative to expert observers by using the Dice coefficient and Hausdorff distance. Figure 7 shows the prostate segmentation results for a representative subject, ranging from the apex to the base. Both MRI and histopathology segmentation models produced highly accurate segmentations that were consistent across all slices. The high Dice coefficients and the low Hausdorff distances between the ground truth segmentations and the network predictions also demonstrate the high performance of our prostate segmentation models (see Table 2).

Figure 7:

Segmentation results of a representative subject. For space consideration, we show the segmentation result for every other slice. The terms apex and base are used by urologists to describe the prostate, with the apex closer to the pelvic floor and the base adjacent to the bladder. For the MRI, the Dice coefficient = 0.95 and Hausdorff distance = 2.44 mm. For the histopathology images, the Dice coefficient = 0.97 and Hausdorff distance = 1.54 mm.

Table 2.

Accuracy of our deep learning segmentation models.

Modality	Dice Coefficient (mean ± std)	Hausdorff Distance ( mean ± std)
T2w MRI	0.94 ± 0.02	2.88mm ± 1.01mm
Histopathology	0.97 ± 0.01	1.67mm ± 0.95mm

3.2. Experiment 2: Rotation and Horizontal Flipping of Histopathology

In Experiment 2, we trained AI models to estimate the preprocessing parameters for histopathology images, i.e., horizontal flipping and rotation, and evaluated models alone or when assisting human readers with different levels of expertise (expert and novice users). We used the manually estimated rotation angle and horizontal flipping of each histopathology image from a prior study (Rusu et al., 2020) as the ground truth. We first applied our deep-learning algorithm to predict the rotation angle and horizontal flipping in each histopathology image. Without the assistance of our model predictions, the readers read the original histopathology images (see the first row of Fig. 8 for examples) in an image viewer (e.g., Windows Photo Viewer or Irfan View) that provided no guidance, estimated the large rotation angle of a histopathology image, and then determined whether the rotated histopathology image required horizontal flipping based on the location of the blue ink. With the assistance of our AI algorithm, the readers read the deformed histopathology images (see the second row of Fig. 8 for examples) four weeks after reading the images initially and made minor adjustments to the AI predictions. The purpose of the four-week washout interval between the two readings by human readers was to prevent any potential memorization of the cases.

Figure 8:

Representative results of applying rotation and horizontal flipping transforms estimated by our algorithm.

Figure 8 shows the angle predictions of three representative histopathology images, demonstrating the accuracy of RAPHIA when applied to histopathology images with different prostate sizes and stain colors. Results in Fig. 9 demonstrate the high accuracy of our AI tool (rotation angle estimation error = 7°, flipping estimation accuracy = 100%, per-patient processing time = 21 seconds), outperforming all novice users. With the assistance of our AI tool, human readers can achieve a highly accurate estimation of both the rotation angle (4.2° estimation error) and flipping (100% accuracy) of histopathology regardless of their experience.

Figure 9:

Performance of different methods for estimating MRI-histopathology slice correspondences. Triangles are the means.

3.3. Experiment 3: Slice Correspondence

We used the slice correspondences between the MRI and histopathology images from a prior study (Rusu et al., 2020) as the ground truth.The evaluation of 20 testing cases reveals that our AI tool’s slice correspondence estimation error is within 1 slice. Users can effectively utilize our tool to refine slice correspondences by examining the registration results from various starting MRI slice numbers: the AI-predicted slice, one slice before, and one slice after.

In this paper, an expert user and three novice users manually estimated the slice correspondences between MRI and histopathology for each patient with and without AI assistance. Compared to a prior approach (Xiao et al., 2011), we achieved a significantly lower maximal slice correspondence error (1 slice vs 3 slices) as well as a significantly lower mean slice correspondence error (0.25 slices vs 1.2 slices). The results in Fig. 10 show that our AI algorithm outperformed all human users, regardless of their experience. With the assistance of AI, the performance of all users can be improved, and the expert can achieve an error as low as 0 per slice. Another observation is that, although novice users cannot outperform the AI model, even with assistance by the AI, their performance is as good as experts without AI assistance. This demonstrates the challenges in estimating MRI-histopathology correspondences and the effectiveness of our AI algorithm when assisting human users. We also observe that the per-case processing time has been reduced for all users and it only took the AI tool 0.4 seconds to process each case. For expert users, we would recommend refining the slice correspondence predicted by our AI algorithm. However, for novice users, we would recommend full reliance on the AI predictions.

Figure 10:

Performance of different methods for estimating the rotation and flipping in histopathology images. Triangles are the means.

3.4. Experiment 4: RAPHIA vs RAPSODI vs ProsRegNet

We compared the performance of four different registration approaches: expert-user assisted by the traditional RAPSODI pipeline (Rusu et al., 2020), expert-user assisted by the deep-learning ProsRegNet approach (Shao et al., 2021a), our fully automated RAPHIA pipeline, and expertuser assisted by the RAPHIA pipeline. The expert user (WS) had 9 years of experience in medical image registration. Since the hyperparameters of RAPSODI and ProsRegNet had already been fine-tuned for the same patient cohort used in this study, we simply used their default hyperparameters. We quantified image registration accuracy by the accuracy of estimating the MRI-histopathology slice correspondences as well as the accuracy of the alignment of cancer regions. For the registration of each image pair, we calculated the processing time by including the running of the algorithms/models and human intervention. Figure 11 shows the RAPHIA registration results of a representative subject. Rows 1–2 show that large rotation, flipping, affine, and deformable transformations exist in the histopathology images prior to registration. The registration results in the last two rows show that the deformed histopathology images are well-aligned with the fixed MRI images, which can be seen from the alignment of the prostate boundaries, the urethra, and the nodules. Comparing the cancer labels outlined by the radiologist on MRI images (Figure 11, first row) with the ground truth pathologist’s cancer outlines, mapped from histopathology onto MRI images (Figure 11, last row), reveals that the radiologist underestimated the extent of clinically significant prostate cancer on MRI. Firstly, the Dice overlap of radiologist and pathologist cancer labels is only 0.59. Secondly, the radiologist missed the lesion located at the left anterior of the prostate, indicated by the yellow arrow. This underscores the importance of performing MRI histopathology image registration.

Figure 11:

Registration results of a representative subject. Yellow: radiologist’s cancer outline on MRI; green: pathologist’s cancer labels on histopathology; yellow arrow: cancer lesion missed by the radiologist. Urethra deviation = 2.12 mm, landmark error = 1.91 mm, and cancer Dice = 0.59.

Compared to the expert user assisted by RAPSODI and ProsRegNet, our fully-automated RAPHIA pipeline has achieved (see Fig.12) a smaller slice correspondence estimation error (0.55 vs. 0.55 vs. 0.4 slices), larger Dice coefficient of cancer labels (0.44 vs. 0.48 vs. 0.50), and significantly smaller per-patient processing time (50.9 vs. 10.8 vs. 4.5 minutes). With the assistance of RAPHIA, the expert user was able to reduce the slice correspondence estimation error to 0.0 per slice and improve the cancer Dice coefficient by 0.55 within 3 minutes following RAPHIA predictions. It is critical to accurately estimate the MRI-histopathology slice correspondences since even minor errors in estimation can lead to significant errors in registration accuracy, e.g., mapping cancer regions to normal tissues and mapping normal tissues to cancer regions. Therefore, the reduced errors in estimating slice correspondences by RAPHIA has improved the accuracy of cancer alignment. All these results demonstrate that RAPHIA is more user-friendly than RAPSODI and ProsRegNet since it not only allows novice users to easily achieve expert-level registration results without any human intervention, but it can also assist expert users in achieving even more accurate registration results.

Figure 12:

Comparison of the performance of different image registration approaches for estimating MRI-histopathology slice correspondences and aligning cancer regions. The triangles in the boxplots represent the means.

The Dice score of 0.5 for the overlap of cancer labels indicates a high level of accuracy in our RAPHIA method’s MRI-histopathology alignment, primarily for two key reasons. First, radiologists can miss or under-segment prostate cancers due to the subtle differences between normal and cancerous tissues in MRI, impacting the Dice score. Second, the presence of smaller cancer lesions, which are inherently more challenging to detect, typically results in lower Dice scores. Therefore, a Dice score of 0.5, particularly for small lesions, is promising.

4. Discussion

4.1. RAPHIA Pipeline Development

In this paper, we have developed the RAPHIA pipeline to solve the challenging problem of registering a sequence of axial prostate MRI slices with a stack of whole-mount histopathology images. RAPHIA improves MRI-histopathology registration by automating the multiple manual processing steps that are required in existing approaches, including: (1) prostate gland segmentation on both T2-weighted MRI and histopathology imaging which is used to drive the registration, (2) estimation of the rotation and flipping transforms in histopathology that occur when mounting on glass slides, and (3) estimation of MRI-histopathology slice correspondences for pairwise 2D registrations. These manual steps are laborious and time-consuming, and are often repeated several times before users can achieve satisfactory registration results. RAPHIA also accelerates MRI-histopathology registration by using deep-learning registration networks to avoid iterative optimization that takes longer to run. In addition, RAPHIA avoids the need for a sophisticated multi-modal image similarity loss by transferring MRI and histopathology images into the same domain. Finally, we will make our code publicly available, which could potentially improve the impact of our work.

4.2. Limitations of our Study

This study has two primary limitations. First, we employed different convolutional neural networks for similar tasks. For image segmentation, we used ProGNet for prostate MRI segmentation due to its superior performance compared to experienced radiology technicians, while we chose dilated residual U-Net for histopathology segmentation as it yields more accurate results than U-Net (Dice: 0.97 vs 0.96, Hausdorff distance: 1.67mm vs 2.25 mm). For image-to-image translation, we used the pix2pix model for translating histopathology images to ink images, as it requires paired input and output images. In contrast, we applied the GcGAN model for MRI-histopathology translation, as it can handle non-paired images. Second, our evaluation included only 20 patients due to two main factors: (1) the labor-intensive and time-consuming process of manually annotating at least three matching landmarks for each slice (approximately 7 slices/case), and (2) the time-consuming nature of evaluating the previous RAPSODI registration approach, which takes about an hour per case as the algorithm is run multiple times with each run lasting 3–5 minutes. However, the dataset from these 20 patients comprises 128 corresponding T2-weighted MRI and histopathology slices, enabling an accurate evaluation of our study’s approaches. Future studies will aim to include additional patients and datasets from multiple institutions to comprehensively assess our RAPHIA model’s performance and generalizability. Third, our RAPHIA pipeline assumes the existence of MRI-histopathology slice correspondences. In this study, we used patient-specific 3D printed molds to ensure that the histopathology sections were cut parallel to the MRI axial views, thereby maintaining slice correspondences. However, we recognize that such slice correspondences do not always exist due to tissue deformation during free-hand sections. In such scenarios, we may perform a 3D MRI-histopathology registration without the need for slice correspondences, as previously described by us (Sood et al., 2021).

4.3. Modularity and User-Friendly Approach

Training a single multi-task model for MRI-histopathology registration can be complex. In contrast, our step-by-step pipeline streamlines the process, making inference and troubleshooting easier. This approach allows for specific adjustments by the user, like angle corrections, without affecting the entire pipeline, enabling a human-in-the-loop approach. Additionally, it enhances data quality management at each step and provides users with greater flexibility to modify individual components. Overall, this modular and flexible approach yields more reliable and precise outcomes compared to complex, all-in-one models.

Users can execute our RAPHIA model in a Jupyter notebook, which we believe is user-friendly for the following two reasons. First, RAPHIA can run in a fully automated manner, which is friendly to non-expert users since prior methods all require some error-prone and labor-intensive preprocessing steps. Second, RAPHIA allows expert users to easily refine the predictions from RAPHIA at each step (e.g., updating the predicted rotation angle by visualizing the rotated histopathology image, updating the predicted prostate segmentations in 3D Slicer).

4.4. Addressing Challenges in Multi-Modal Registration

he challenge in training a multi-modal registration network primarily arises from two factors: the absence of ground truth spatial correspondences between MRI and histopathology images, and the lack of an effective multimodal similarity loss function. To tackle this, the ProsReg-Net method (Shaoetal.,2021a) utilized synthetic monomodal image pairs created by applying known transformations to the images. This allowed for unsupervised network training using a monomodal loss function. However, this approach has limitations since it does not directly train on multimodal pairs. In contrast, our study introduces a novel approach involving image-to-image translation to transform image representations between MRI and histopathology, thereby creating synthetic multimodal image pairs in the same coordinate system. This method aligns more closely with real-world scenarios as the network is trained on image pairs that mimic true MRI-histopathology image pairs during testing. The primary advantage is the enhanced relevance of the training phase to actual testing conditions.

4.5. Application and Impact of RAPHIA

Results show that our AI tools outperform imaging scientists with limited experience in performing image registration between radiology and pathology images. With the assistance of the proposed RAPHIA pipeline, novice users can achieve expert-level performance. Therefore, novice users can fully rely on our AI predictions without human intervention. We also give advanced users the flexibility to quickly make minor adjustments to AI predictions for achieving more accurate registration results. The improved registration accuracy and processing time by RAPHIA allows us to quickly and efficiently process large amounts of data, facilitating the downstream use of registration results. The resulting pathology cancer labels mapped to MRI can be used to train machine-learning models for selective identification of aggressive and indolent cancers within (Cao et al., 2019; Seetharaman et al., 2021; Bhattacharya et al., 2022c) or outside of (Moroianu et al., 2022) the prostate. The aligned MRI and histopathology images can also be used to learn imaging features from MRI that correlate with histopathology imaging features for better cancer detection on MRI (Cao et al., 2019; Bhattacharya et al., 2022c). This trained cancer-detection deep-learning model has the potential to improve prostate cancer care in three ways: First, it can be used to guide ultrasound-MRI fusion-targeted prostate biopsy (Loeb et al., 2011). Second, it can be used to guide focal therapy to treat small tumors that are localized in the prostate. Third, it provides a new scheme to train radiologists by providing them with ground-truth cancer labels and AI predictions.

5. Conclusion

We have developed the deep learning based RAPHIA pipeline for efficient and accurate MRI-histopathology registration. RAPHIA automates many manual preprocessing steps involved in prior approaches. Compared to existing registration approaches, RAPHIA has achieved more accurate and more rapid registration results. The improved accuracy and processing time by RAPHIA potentially allows users to efficiently process hundreds to thousands of cases for the development of machine-learning models for early prostate cancer detection.

Highlights.

1. This paper introduces RAPHIA, a fully automated pipeline for registering MRI and wholemount histopathology images of the prostate

2. RAPHIA eliminates the need for manual steps required in previous approaches.

3. RAPHIA avoids the need for a hand-crafted multimodal image similarity metric.

4. RAPHIA is designed to be user-friendly and can help novice users achieve expert-level performance.

5. RAPHIA is a human-in-the-loop pipeline that enables advanced users to adjust AI predictions for better registration results.