Coregistration of Preoperative MRI with Ex Vivo Mesorectal Pathology Specimens to Spatially Map Post-treatment Changes in Rectal Cancer Onto In Vivo Imaging:

Abstract

Rationale and Objectives:

The objective of this study was to develop and quantitatively evaluate a radiology-pathology fusion method for spatially mapping tissue regions corresponding to different chemoradiation therapy-related effects from surgically excised whole-mount rectal cancer histopathology onto preoperative magnetic resonance imaging (MRI).

Materials and Methods:

This study included six subjects with rectal cancer treated with chemoradiation therapy who were then imaged with a 3-T T2-weighted MRI sequence, before undergoing mesorectal excision surgery. Excised rectal specimens were sectioned, stained, and digitized as two-dimensional (2D) whole-mount slides. Annotations of residual disease, ulceration, fibrosis, muscularis propria, mucosa, fat, inflammation, and pools of mucin were made by an expert pathologist on digitized slide images. An expert radiologist and pathologist jointly established corresponding 2D sections between MRI and pathology images, as well as identified a total of 10 corresponding landmarks per case (based on visually similar structures) on both modalities (five for driving registration and five for evaluating alignment). We spatially fused the in vivo MRI and ex vivo pathology images using landmark-based registration. This allowed us to spatially map detailed annotations from 2D pathology slides onto corresponding 2D MRI sections.

Results:

Quantitative assessment of coregistered pathology and MRI sections revealed excellent structural alignment, with an overall deviation of 1.50 ± 0.63 mm across five expert-selected anatomic landmarks (in-plane misalignment of two to three pixels at 0.67- to 1.00-mm spatial resolution). Moreover, the T2-weighted intensity distributions were distinctly different when comparing fibrotic tissue to perirectal fat (as expected), but showed a marked overlap when comparing fibrotic tissue and residual rectal cancer.

Conclusions:

Our fusion methodology enabled successful and accurate localization of post-treatment effects on in vivo MRI.

INTRODUCTION

Of the 40,000 patients diagnosed with rectal cancer in 2016 (1), most underwent neoadjuvant chemoradiation therapy before surgery. However, histopathologic analysis of the postsurgical specimens reveals that up to 27% of these patients had pathologic complete response to treatment (2), indicating they could have potentially avoided aggressive surgery. Current preoperative standard-of-care assessment is primarily based on T2-weighted (T2w) magnetic resonance imaging (MRI) for evaluation of treatment response and extent of tumor regression in vivo. Radiologists attempt to restage the tumor based on visual differences in grayscale contrast of soft tissue regions within and around the rectum. Although fibrotic tissue and rectal cancer may be different from a physiological perspective (different T2 relaxation times) (3), regions of benign treatment effects such as fibrosis, inflammation, and mucin pools typically have indistinguishable intensities from residual cancer on T2w MRI (Fig 1) (4). Thus, experts tend to struggle in identifying patients with complete response (no residual disease and only fibrosis) via MRI, leading to possible overstaging in patients with rectal cancer (5).

Figure 1.

Radiology and pathology images for a patient with rectal cancer who underwent chemoradiation therapy before surgical resection of the mesorectum. (a) 3-T T2-weighted magnetic resonance image of the rectum acquired after chemoradiation. (b) Expert radiologist delineation of estimated residual rectal cancer based on visual changes in magnetic resonance imaging pre- and post treatment. (c) Two-dimensional pathology slide of the same patient obtained after surgical removal of the rectum and surrounding the rectal tissue. (d) Expert pathologist delineation of varied treatment effects seen on the pathology image, which are not visually discernible on T2-weighted magnetic resonance imaging. Pathology report indicated that the patient had complete tumor response to radiation therapy (no residual rectal cancer).

To accurately assess the extent of tumor regression on preoperative MRI (6), it is crucial to localize the exact extent of disease and treatment-induced effects on standard-of-care imaging. Currently, this information can only be definitively determined after a total mesorectal excision of the rectum and surrounding fat, when histologic slides of the rectal tissue are evaluated for residual disease and treatment effects (7). If one could coregister these pathologic specimens with corresponding preoperative MRI and bring them into spatial correspondence (8), different pathologic regions (eg, fibrosis, inflammation, and residual disease) could be rigorously mapped onto MRI. This method could, in turn, be utilized to build a definitive “learning set” for machine-based learning of computerized imaging features, which could enable accurate post-treatment evaluation of tumor regression on MRI.

Previous studies have demonstrated that it is possible to visually localize different pathologic features within and around the rectum on magnetic resonance (MR) images (9,10) (such as involvement of the perirectal fat, peritoneum), but did not explicitly attempt spatial coregistration of the two modalities. The use of spatial coregistration methods for mapping pathologic annotations onto imaging has been previously performed by several groups, most popularly in the context of prostate cancer. Applications explored have included mapping disease extent onto diagnostic MRI (8,11–14), as well as mapping treatment effects and residual disease onto post-treatment MRI (15).

Notably, spatial coregistration of radiology and pathology may be significantly more challenging in rectal cancers. Unlike the relatively dense walnut-shaped prostate gland, the rectum is a hollow tube that tends to collapse into the lumen during pathologic sectioning and processing. This makes the rectum more difficult to section as a whole mount. As a result, it is nontrivial to identify corresponding sections and landmarks between the two modalities. Thus, for radiology-pathology coregistration to be reasonably accurate in rectal cancers, significant domain information from both radiology and pathology experts will be required, together with careful nonlinear warping to bring corresponding sections into alignment. In the present study, we present the first attempt at spatially mapping radiation treatment effects and residual disease (annotated on surgical specimens) onto preoperative in vivo imaging via image coregistration, applied to patients treated for rectal cancer.

MATERIALS AND METHODS

Data Description

The institutional review board approved this retrospective study and waived the requirement of informed consent. In the period between January 1, 2012, and January 1, 2013, 22 patients with rectal cancer at University Hospitals Cleveland Medical Center were identified as having undergone neoadjuvant chemoradiation followed by surgery. Of these patients, 10 were identified for whom both rectal pathology sections (from excised specimens after chemoradiation) and preoperative, postchemoradiation MRIs were available. The remaining 12 patients were excluded from our study, as postsurgical specimen data were unavailable for them. All patients had been diagnosed with rectal cancer, treated with standard-of-care neoadjuvant chemoradiation, and had a post-treatment MRI scan and then a total mesorectal excision of the rectum and surrounding fatty tissue. The time between MRI and total mesorectal excision ranged from 1.5 to 5.0 weeks (mean: 3 weeks). Histologic slides were digitized for all 10 patients. Of these 10 patients, only 6 patients had nonfragmented, whole-mount rectal tissue specimens available (all males, age range: 52–82 years, median: 57). Based on pathology reports, our cohort comprised one patient with complete response, four patients with incomplete response, and one patient with poor response to chemoradiation therapy (these groupings were based on pathologic T stage alone, as all patients were N0M0 (16)).

Acquisition of In Vivo MRI

MRI was acquired after chemoradiation therapy and presurgery, using a 3-T T2w turbo spin echo MRI protocol (MAGNETOM Verio; Siemens Healthcare, Malvern, PA) with average repetition and echo times of 1000 and 80 ms, respectively. The T2w volumes were reconstructed with an in-plane resolution of 0.63–1.00 mm (256 × 256 in-plane pixels) and a slice thickness of 3 mm. Note that the MRI field of view used for colocalization and coregistration was selected based on slices being in the plane axial through the tumor.

Digitization of Surgical Specimens and Annotation of Treatment Effects

After total mesorectal excision, rectal tissue specimens were marked for orientation using blue and black inks, fixed in formaldehyde, sectioned in the plane axial to the tumor, and then mounted and stained with hematoxylin and eosin. Pathology slides were scanned at 40× magnification (pixel size: 0.24 μm, 153,470 × 105,207 in-plane pixels). An expert pathologist assessed the tissue for the presence of residual cancer, ulcers, fibrosis, muscularis propria, mucosa, fat, inflammation, and pools of mucin, carefully delineating each region present for each patient (Aperio ImageScope, Leica Biosystems). The annotated area of residual rectal cancer ranged from 1.3 to 170.0 mm² per patient, and the annotated area of treatment effects ranged from 1.4 to 2314.2 mm² per patient.

Identification of Corresponding Slices and Landmarks Between MRI and Pathology Sections

First, a single T2w slice from the entire T2w volume that best corresponded to the annotated pathology slide image was identified jointly by a radiologist (with 17 years of experience) and a pathologist (with 24 years of experience) sitting together. The procedure for this was as follows. Based on surgical and pathologic reports, the pathologist determined the rough location of the tumor—anterior and posterior ends of the rectum, as well as tumor involvement of the anal canal. This technique allowed the radiologist to identify a set of slices within the T2w MRI volume within which the pathology slice might be localized. Two application windows were then opened on the screen—one for each modality. While scrolling through the images in each window, the pathologist and the radiologist visually compared the similarities in appearance, known regions, and structures between every pair of pathology and MR images (within the previously determined MRI volume of interest). They then reached mutual agreement on which slices corresponded between the modalities.

Corresponding landmarks were then identified across the pathology and MRI sections. First, the pathologist identified 12–15 landmarks on the pathology specimen corresponding to edges or folds along the inner wall of the rectum (green and yellow circles in Fig 2a), as well as peaks in curvature along the outer rectal wall where the rectum borders the fat (purple, blue, and red circles in Fig 2a). The radiologist then attempted to localize these same inner folds and outer boundary landmarks around the rectum on the corresponding T2w MRI section as best possible (Fig 2b). At final count, the radiologist was able to confidently localize six to eight of these curvature landmarks per patient. The radiologist and the pathologist then jointly examined the images to identify additional landmarks based on tissue architectures visible on both modalities (eg, glands, lymph nodes, and inflammation). In total, it was ensured that every pair of MRI and pathology sections had 10–12 corresponding landmarks identified. These were then randomly split into two independent sets of landmarks—one set for driving the registration procedure and the other for evaluating the accuracy of registration.

Figure 2.

Visualization and labeling of corresponding landmarks between modalities: (a) pathologist identification of anatomic landmarks on pathology section based on edges and folds along the inner or outer rectal wall, and (b) radiologist localization of corresponding edge and boundary landmarks on magnetic resonance imaging section (indicated by matching pairs of colored circles). (Color version of figure is available online.)

Coregistration of Radiology-Pathology to Spatially Map Detailed Annotations onto MRI

The workflow for radiology-pathology fusion is illustrated in Figure 3, which was applied to all patients considered in the present study.

Figure 3.

Overall workflow for radiology-pathology coregistration to map pathologic annotations of chemoradiation treatment effects and residual disease onto magnetic resonance imaging.

To bring the rectal region of interest on the T2w slice into the same order of magnitude of size (in pixels) with the corresponding region on the pathology slide, digitized pathology slides were down-sampled to 0.5× magnification (pixel size: 18.68 μm, 1643 × 2397 in-plane pixels) and T2w slices were up-sampled by a factor of 6 (in-plane resolution of 0.11– 0.17 mm, 1536 × 1536 in-plane pixels) using bilinear interpolation resampling. Pathology slides were flipped and rotated on a case-by-case basis (based on expert assessment) to ensure gross alignment relative to the T2w slice.

Sources of misalignment between the up-sampled T2w slice and the down-sampled pathology slide included differences in image orientations, as well as artifacts due to pathologic processing of rectal specimens (resulting in tissue folding or missing tissue). To correct for these differences and bring these images into a spatial correspondence, two-dimensional (2D) nonlinear coregistration was applied between these images using the T2w MR image as “fixed” and the pathology image as “moving.” A thin-plate spline-based registration method (17) was used to compute a spatial transformation to bring five to seven corresponding anatomic landmarks into alignment, thus deforming the pathology image into spatial correspondence with the T2w MR image. Coregistration was done using an in-house tool that had been developed based on the Insight Segmentation and Registration Toolkit (ITK) framework.

The transformation matrices obtained through the coregistration procedure were then applied to the corresponding pathology annotations for that patient. This approach allowed the different pathologic annotations (eg, fibrosis, inflammation, and residual disease) to be mapped onto their corresponding spatial locations on the up-sampled T2w slices.

Evaluation of Coregistration Accuracy

The results of image coregistration were qualitatively evaluated by overlaying the registered pathology image and the registered pathology mask onto the “target” T2w image. These registration results were visually inspected by the radiologist and the pathologist together to determine that corresponding regions of the rectum (visible on both imaging modalities) were correctly colocalized (ie, fat, residual cancer, and lumen). They also visually inspected the two sets of landmarks used (one to drive registration and one for quantitative evaluation), in addition to examining mapped annotations to determine the fidelity of landmarks and annotations.

Target registration error (TRE) was quantified as the Euclidean distance between five corresponding anatomic landmarks that had been annotated on each of the pathology and radiology images (Fig 4). Note that all landmarks used to evaluate coregistration were independent of landmarks used to drive registration.

Figure 4.

Strategy to qualitatively and quantitatively evaluate coregistration. Corresponding landmarks on (a) pathology slide and (b) T2w slices selected to drive coregistration. Independent set of five evaluation landmarks on (c) registered pathology and (d) T2w slice, with corresponding labels 1–5. Qualitative inspection of coregistration result involves (e) examining registered pathology slide image overlaid onto T2w slice image for alignment of known structures, (f) spatial comparison of corresponding landmarks used for (h) computing TRE, as well as (g) overlay of registered pathology annotations onto the T2w slice image, which was examined to ensure fidelity of mapped annotations. TRE, target registration error; T2w, T2-weighted.

Additionally, the accuracy of spatial mapping for three pathologic regions relevant to the evaluation of tumor regression (18)—fibrosis, residual disease, and perirectal fat—was also evaluated. Note that these were regions with relatively large annotations, based on the reasoning that, for larger regions, a reasonably accurate mapping of tissue compartments could be projected onto corresponding MRI. For smaller regions (eg, ulceration and mucin pools), there might be less confidence in the fidelity of the spatial mapping. Evaluation involved comparing the T2w intensity distributions between fibrosis, residual disease, and perirectal fat to determine if any imaging trends existed that were reflective of their pathologic characteristics. It was hypothesized that, if the spatial mapping of two unambiguous tissue partitions had been performed accurately, the corresponding T2w intensity distributions with these compartments would be different. A similar approach has been previously employed in the evaluation of coregistration of imaging and pathology specimens (19).

RESULTS

Figure 5 depicts the coregistration results for four patients from our cohort. Overlays of pathology onto radiology (third column from the left) appeared to demonstrate accurate alignment of the interior (lumen) and exterior (fat) edges along the rectal wall for all patients. Note that one patient (row 1) required two additional landmarks to ensure accurate alignment due to cut rectal tissue on pathology being extremely folded. Pathologic annotations of treatment effects were visually determined by the pathologist and the radiologist to have been spatially mapped to their expected locations within the rectal wall on T2w images.

Figure 5.

Results of coregistration for four different patients (one per row). (a, e, i, m) Original pathology slides with landmarks used for registration in different colors. (b, f, j, n) T2w slices and landmarks used for registration (in different colors), corresponding to the pathology slide in the extreme left column. (c, g, k, o) Registered pathology image overlaid onto the T2w slice. (d, h, l, p) Detailed pathology annotations (with region labels in legend) spatially mapped onto the T2w slice. T2w, T2-weighted. (Color version of figure is available online.)

As fat tends to appear distinctly bright on T2w MRI, this tissue region is particularly suited for the evaluation of how well a pathologic annotation overlaps with a visually apparent tissue region on MRI. In addition to a clear visual correspondence between the mapped annotation and the visible region of perirectal fat on MRI, the histogram distributions of the T2w fat intensities were plotted for all six patients, as shown in Figure 6a. Notably, a highly similar T2w intensity profile can be observed for the annotated perirectal fat regions across all patients. The fat region for patient 4 (cyan) can be seen to have a slightly shifted T2w fat intensity profile relative to the other patients. The corresponding mapped annotations for this patient (Fig 5h) reveals that perirectal inflammation appears to have “leaked” into the fatty tissue, which would result in the observed shift in the intensity distribution.

Figure 6.

(a) T2w image intensity distributions for annotated perirectal fat regions for all six patients (in different colors). Five of the six patients show relatively similar fat intensity distributions, which suggest a successful spatial mapping of perirectal fat from pathology onto radiology. (b) Bar plots of average TREs across five evaluation landmarks for each patient, with error bars showing standard deviations. TRE, target registration error; T2w, T2-weighted. (Color version of figure is available online.)

The overall TRE across all six patients was calculated to be 1.50 ± 0.63 mm, using the set of five evaluation landmarks per patient (Fig 6b). As the in-plane resolution of the original MRI images ranged 0.63–1.00 mm, this would correspond to a maximum misalignment of two to three pixels.

Figure 7 shows violin plots of T2w intensity distributions for each patient within mapped regions of fibrotic tissue (red), rectal cancer (green), and perirectal fat (blue), alongside each other on the same y-axis scale. These plots indicate that fibrosis appears to have a markedly different T2w imaging signature compared to perirectal fat across all patients. This finding is to be expected, given the significant tissue differences between these regions, and thus may implicitly indicate that the spatial mapping of these regions onto MRI has been performed accurately. A significant overlap in T2w signal intensities can also be observed between fibrotic tissue (red) and residual cancer (green) distributions. Although this finding is possibly reflective of the overlap in visual appearance between these regions, the distribution trends also suggest the presence of subtler differences, which could be extracted via computerized imaging features. Ignoring patient 4 (who has inflammation affecting the fat distribution), there appears to also be a trend in the relative overlap between T2w intensity distributions for fibrosis, residual disease, and fat. This finding is most evident when comparing the patient with complete response to chemoradiation therapy (extreme left, minimal overlap) to the one with incomplete response to chemoradiation therapy (extreme right, marked overlap).

Figure 7.

Side-by-side violin plots comparing T2w signal intensity distributions from the fibrosis region (red) and rectal cancer (green) to the perirectal fat region (blue) across all patients. The fibrosis region appears to have a signal intensity that is markedly different from that in the fat region, reflective of the accuracy in spatially mapping the pathology annotations onto the T2w images. This difference in signal intensities appears to be harder to discriminate in cases of poor response to treatment (extreme right) compared to complete response to treatment (extreme left). Additionally, the spatially mapped rectal cancer and fibrotic tissue annotations exhibit overlapping T2w signal intensity distributions, reflecting their overlapping visual appearance on T2w MRI. T2w, T2-weighted. (Color version of figure is available online.)

DISCUSSION

The prognostic value of evaluating and analyzing tumor regression in vivo indicates an emerging need for better localization of different treatment effects and residual tumor regions on preoperative MRI. Analyzing these regions in vivo may also enable better and earlier clinical decision making for surgical planning, adjuvant treatment, or active surveillance based on a better assessment of the degree of treatment response than tumor stage alone.

In the present study, we have presented initial results of a radiology-pathology fusion methodology to spatially map annotated regions of treatment effects and residual disease from post-treatment pathologic specimens onto the preoperative, post-treatment MRI in the context of rectal cancer. In-plane registration error was conservatively estimated as under 2 mm across all patients, which corresponds to only two to three pixels (based on the in-plane resolution of the original MRI). This finding is comparable to TREs reported in previous radiology-pathology studies, which implemented landmark-based registration (see Table 1). Notably, we achieved our results using only 5–7 anatomic landmarks (out of a total of 10–12 expert-identified landmarks) per patient to drive the registration for each patient, which is slightly below the average number of landmarks used in comparable studies (12,15). Although we could have used more landmarks to drive the registration, this would have resulted in very few landmarks being available to evaluate registration accuracy. As such, we opted to randomly split our limited set of expert-identified landmarks equally between driving and evaluating the coregistration process.

TABLE 1.

Comparison of Radiology-Pathology Coregistration Results to Relevant Literature

Reference	Organ	Fused Modalities	Coregistrationy Method	Number of Registration Landmarks (per Case)	Number of Evaluation Landmarks (per Case)	TRE (mm)
Kimm et al. (2012) (11)	Prostate	Ex vivo MRI-histology (2D)	Injected fiducials	3	10–25	1.24 ± 0.59
Gibson et al. (2012) (12)	Prostate	Ex vivo MRI-histology (2D)	Injected fiducials	10	3–7	0.71 ± 0.38
Litjens et al. (2014) (15)	Prostate	In vivo MRI-histology (2D)	Thin-plate spline	7	Not performed	Not performed
Rusu et al. (2017) (19)	Lung  nodules	In vivo CT-histology  reconstruction (3D)	Rigid + nonrigid	N/A	5–7	1.06 ± 0.40
This work	Rectum	In vivo MRI-histology (2D)	Thin-plate spline	5–7	5	1.50 ± 0.63

2D, two-dimensional; 3D, three-dimensional; CT, computed tomography; MRI, magnetic resonance imaging; TRE, target registration error.

As a result of spatial mapping pathology onto MRI, we were able to examine the T2w intensity distributions between three clinically relevant subcompartments (Fig 7). We found differences in T2w MRI intensities between spatially mapped regions of fibrosis and perirectal fat, reflecting that these two distinct regions have been likely accurately mapped onto MRI. By contrast, the overlapping intensity distributions of fibrosis and rectal cancer reflect the difficulty in visibly distinguishing these two tissue pathologies using MRI alone (4). Our mapped annotations also support the finding that hypointense MR signal intensity regions may indicate the presence of either fibrosis alone or fibrosis that contains clusters of tumor cells (20). This reduction in T2 signal (together with a decrease in tumor size) may arise because of fibrotic response (replacement of neoplasm by fibrosis) to chemoradiation treatment (21). With the availability of spatially mapped annotations of these pathologies onto MRI, the results of our study may thus have further implications in the development of computerized imaging features to quantitatively capture subtle tissue characteristics on in vivo MRI (22–25).

To our knowledge, the current study is the first of its kind in utilizing spatial mapping of pathology annotations from ex vivo post-treatment surgically resected rectal specimens onto post-treatment imaging. Litjens et al. similarly employed a registration scheme to spatially map post-treatment pathologic annotations (necrosis and residual disease) onto postfocal therapy prostate MRI (15). However, as noted previously, corresponding MRI and pathology slices and anatomic landmarks are considerably easier to localize on the prostate. More recently, Rusu et al. performed deformable coregistration between corresponding 2D images in the z-stack of treatment-naive computed tomography and pathology images of surgically excised malignant lung nodules. Rusu et al. employed threedimensional (3D) reconstruction of the histology volume from 2D sections (19,26). However, Rusu et al. made assumptions about the slice spacing of the pathology sections, which is nonuniform in clinical practice.

We note the fact that there is unavoidable registration error when working with clinical data. First, the possibility of out-of-plane warping of processed pathology sections could result in there being no “perfect” 2D slice correspondence between MRI and pathology. Second, there is uncertainty in how the tissue deforms during coregistration at locations remote from control points due to in-plane warping of registered pathology sections. The latter source of error is often due to the deformation and tissue loss during the specimen sectioning process. Similar to previous work (8,11,12,19,25), we have attempted to account for the in-plane misalignment between T2w MRI and pathology, through careful expert identification of landmarks for driving a nonlinear coregistration procedure.

Our study did have its limitations. Our study was limited by a small sample size, in which only 6 out of 22 patients had postsurgical specimens and post-treatment imaging available deemed to be of good enough quality to be included. Additionally, our cohort included only men within a narrow age range. We were unable to obtain more than one or two histology sections for each case and therefore could not perform a 3D histology reconstruction and attempt a 3D reconstruction as done by Rusu et al. However, we believe that expert evaluation of the six cases evaluated in the current study ensured that all correspondences were within ±1 MRI slice of the pathology section. Our curated data cohort also suffered from significant sectioning and processing differences between the two modalities. Accounting for this would require a prospective study with more controlled pathologic processing of the excised rectum, including sectioning with respect to MRI slice spacing as well as careful slide preparation to mitigate tissue folding artifacts. Creating a 3D mold or a 3D printing of the rectal tissue and surrounding fat (27) before sectioning could also assist in this regard. These steps would lend further confidence in the coregistration results and could be attempted in future work.

In conclusion, this work presented preliminary findings of a radiology-pathology fusion methodology that allowed different pathologic tissue regions (arising because of targeted chemoradiation therapy) to be spatially mapped from ex vivo pathology specimens onto in vivo MRI. This computeraided enrichment of imaging data through spatial correlation with pathology data could enable a more comprehensive, noninvasive, and quantitative assessment of treatment response and tumor regression in vivo. This approach would pave the way to potentially enabling more personalized patient follow-up. Future directions of this work will include utilizing such spatially mapped pathologic regions to rigorously identify computerized imaging features that can characterize and differentiate residual disease from treatment effects on in vivo MRI.

Abbreviations and Acronyms

T2w

T2-weighted

2D

2-dimensional

TRE

target registration error