Deformable Registration for Quantifying Longitudinal Tumor Changes During Neoadjuvant Chemotherapy

Abstract

Purpose

To evaluate DRAMMS, an attribute-based deformable registration algorithm, compared to other intensity-based algorithms, for longitudinal breast MRI registration, and to show its applicability in quantifying tumor changes over the course of neoadjuvant chemotherapy.

Methods

Breast magnetic resonance images from 14 women undergoing neoadjuvant chemotherapy were analyzed. The accuracy of DRAMMS versus five intensity-based deformable registration methods was evaluated based on 2,380 landmarks independently annotated by two experts, for the entire image volume, different image subregions, and patient subgroups. The registration method with the smallest landmark error was used to quantify tumor changes, by calculating the Jacobian determinant maps of the registration deformation.

Results

DRAMMS had the smallest landmark errors (6.05 ± 4.86 mm), followed by the intensity-based methods CC-FFD (8.07 ± 3.86 mm), NMI-FFD (8.21 ± 3.81 mm), SSD-FFD (9.46 ± 4.55 mm), Demons (10.76 ± 6.01 mm), and Diffeomorphic Demons (10.82 ± 6.11 mm). Results show that registration accuracy also depends on tumor versus normal tissue regions and different patient subgroups.

Conclusions

The DRAMMS deformable registration method, driven by attribute-matching and mutual-saliency, can register longitudinal breast magnetic resonance images with a higher accuracy than several intensity-matching methods included in this article. As such, it could be valuable for more accurately quantifying heterogeneous tumor changes as a marker of response to treatment.

INTRODUCTION

Breast cancer is the second most frequent cancer and the second leading cause of death for women in the United States (1). A critical issue in the treatment of locally advanced breast cancer is how to effectively monitor tumor changes during the course of neoadjuvant chemotherapy, and predict the long-term pathologic response. This is needed so that clinicians can make necessary treatment adjustments and optimize treatment effects (2,3). To do so, clinicians often refer to longitudinal breast magnetic resonance (MR) images, acquired in multiple visits, days or weeks apart (a series of longitudinal breast MR images can be found for a representative breast cancer patient in the first row of Fig. 7). Here, a central problem is how to accurately quantify tumor changes over time, and more importantly, how to effectively reveal heterogeneous changes in various subregions within tumors, rather than tumors as a whole, since the heterogeneity may be more indicative of long-term pathologic response (11–16). Toward this goal, recent research has increasingly focused on image registration techniques (17,18). In this article, we evaluate the performance of six image registration methods specifically for the registration of longitudinal breast MR images. One primary objective is to evaluate an attribute-based deformable registration method, DRAMMS (19), and compare it with other intensity-based registration methods. We also demonstrate the feasibility of applying registration as a means to quantify tumor changes over time, which can be used as a potential marker for a breast cancer patient’s response to neoadjuvant chemotherapy.

FIG. 7.

Quantification of the voxelwise longitudinal breast cancer change by the DRAMMS registration algorithm. Results shown in this figure are from an example pathologic partial responder (pPR) patient. Jacobian determinants in the tumor regions are overlaid on the baseline image in the bottom row.

Longitudinal Breast MR Image Registration: Definition and Necessity

By definition, image registration is the process of finding an optimal transformation that spatially aligns two or more images. In the context of longitudinal images, deformable registration can capture how anatomical structures deform over time. This concept is shown in Figure 1. In this figure, a follow-up image (Fig. 1b) is registered to the baseline image (Fig. 1a). The resultant deformation is visualized in Figure 1(d). The deformation is further quantitatively analyzed to derive the Jacobian determinant map, as shown in Figure 1e. The Jacobian determinant value at a voxel indicates the volumetric change ratio (follow-up/baseline): >1 for expansion, <1 for shrinkage, and = 1 for volume preservation. Therefore, if the solid white plate in Figure 1 symbolizes a changing tumor, then the Jacobian determinant map in Figure 1e shows that the tumor has shrunk from the baseline to the follow-up images, while at the same time the neighborhood has expanded to fill into the region that was once tumorous. This conceptually demonstrates that the deformable image registration can quantify longitudinal changes, and that the quantification of changes agrees with the human interpretation.

FIG. 1.

A conceptual illustration of using deformable registration to quantify the volumetric changes of tumors and the surrounding tissue in longitudinal images. The solid white plate symbolizes a tumor that has shrunk at the follow-up visit.

Deformable registration can also reveal the spatial heterogeneity of changes at the voxel level (17,18). In contrast, the standard RECIST rule (20,21) or other segmentation-based approaches (22,23) could only tell how the tumors change as a whole. Indeed, tumors often change heterogeneously in space (24–29), meaning that different regions within a tumor may respond differently to the treatment, and hence they may change heterogeneously over time. Several studies have shown that such heterogeneity within tumors (30–33) and the surrounding tissue (34) can help predict the long-term pathologic response to treatment.

Need for Evaluating Registration Performance for Longitudinal Breast MR Images

Despite its importance, longitudinal registration of breast MRI has been less studied, and the registration performance is less understood than that for other organs [e.g., brain (35,36), lung (37), heart (38), and liver (39,40)].

Prior studies for breast image registration were either on two-dimensional (2-D) mammographic images (41–47), or on “short-interval” longitudinal breast MR images, which are acquired seconds or minutes apart during one visit [e.g., (48–54)]. There, registration is used mainly for motion correction (55–62). In contrast, to quantify the heterogeneous tumor changes, we are dealing with “long-interval” longitudinal breast MR images, which are acquired days or weeks apart in multiple visits. Few registration methods exist for this task. Moreover, a comprehensive evaluation of those methods remains an unexplored topic, which motivates our work in this article.

Outline and Contributions

This article evaluates six deformable registration methods that have been previously reported for the registration of long-interval longitudinal breast MR images. Hereafter, we will use the term “longitudinal breast MR images” as short for “long-interval longitudinal breast MR images,” unless otherwise specified. Out of those six methods, DRAMMS (19) is an attribute-based method (i.e., it finds voxel correspondences by the high-dimensional attributes extracted from the neighborhoods of voxels); while, in contrast, the other five are intensity-based methods (i.e., they find voxel correspondences by maximizing some image similarities defined on the intensity or intensity distributions). A primary purpose of our study is to evaluate whether the use of attributes in DRAMMS for characterizing voxels can help improve the registration accuracy over the intensity-based methods. A second purpose is to show preliminary results on using registration to quantify tumor changes over time as a measure of patient response to treatment. As such, our work presents the following contributions:

1. We evaluate six deformable registration methods specifically for longitudinal breast MR images. They have all been reported as being used for this specific task of longitudinal breast MRI registration. Within this context, we examine the advantages of different registration approaches, with an emphasis on the comparison between attribute- and intensity-based registration algorithms. We also point out potential areas to improve in the future.

2. We quantitatively measure the performance of registration methods based on two breast radiologists independently annotating a total of 2,380 landmarks in longitudinal MRI scans of 14 breast cancer patients.

3. We further separately examine registration errors in tumor versus normal tissue regions. Studies have shown that both types of regions can contribute to the prediction of long-term response to treatment (34). However, little is known about whether we should use them differently. This largely depends on whether the registration performance, and hence the subsequently registration-quantified changes, are different between tumor versus normal tissue regions, which has not yet been investigated in the literature.

4. Finally, we quantitatively examine whether the registration performance is different for patient subgroups having different pathologic responses to treatment. The hypothesis is that, tumors may change more in patients having pathologic complete responses than patients having pathologic partial responses, thereby posing a higher level of challenge to registration.

The rest of this article is organized as follows. In Methods section, we describe our evaluation protocol. In Results section, we show the evaluation results, and show preliminary results of using deformable registration to quantify tumor changes. Finally, Conclusion section discusses and concludes this article.

METHODS

Protocol to Evaluate Registration Accuracy

This section presents the evaluation protocol. In the next four subsections, we will describe our dataset, the registration methods to be evaluated, the parameter settings for those registration methods, and the criterion we used to evaluate the registration accuracy.

Dataset

We have retrospectively collected longitudinal breast MR images (1.5T Siemens Sonata scanner) for 14 women with biopsy-proven T2–3 breast tumors from the ACRIN 6657 I-SPY trial (63,64). All women in this trial underwent the standard neoadjuvant chemotherapy, which, at the time of the study, consisted of four cycles of adriamycin/cytoxan, followed by four cycles of taxotere. At the end of the 3–4 months of chemotherapy, they were evaluated by a pathological analysis. Those who showed the absence of any residual invasive cancer in the breast and the absence of any metastatic cells in the regional lymph nodes were defined as pathologic complete responders (pCR; (65,66)), otherwise as pathologic partial responders (pPR). In our dataset, eight women were classified as pCR and six as pPR.

Imaging parameters were: FOV 18–20 cm, image size 256 × 256 × 64, voxel size 0.70 × 0.70 × 2.0 mm³, pulse repetition time (TR) = 27.0 ms, echo time (TE) = 4.76 ms, flip angle = 45°. Pre-gadodiamide imaging was performed followed by immediate post-gadolinium images (at 2 min) and delayed post-gadodiamide images (at 7 min). The baseline image was the image obtained right before the first chemotherapy. We registered the follow-up image after two rounds of chemotherapy to the baseline image for each patient (average 31 ± 21 days apart). Table 1 lists the information of each patient in our dataset.

Table 1.

Lists of Cases in the Pathologic Complete Responder (pCR) and Partial Responder (pPR) Subgroups in Our Study

	#days (FU-BL)	#LM (tumor)	#LM (normal)	#LM (all)
pPR group
Patient 1	14	63	59	122
Patient 2	29	25	55	80
Patient 3	13	64	60	124
Patient 4	14	22	45	67
Patient 5	14	90	16	106
Patient 6	56	53	55	108
Mean ± stdev	23 ± 17	53 ± 26	48 ± 17	101 ± 23
pCR group
Patient 7	61	42	30	72
Patient 8	14	14	48	62
Patient 9	60	16	45	61
Patient 10	56	8	19	27
Patient 11	14	22	81	103
Patient 12	55	41	20	61
Patient 13	21	51	25	76
Patient 14	14	31	87	118
Mean ± stdev	37 ± 23	28 ± 15	44 ± 27	73 ± 28
All patients
Mean ± stdev	31 ± 21	39 ± 23	46 ± 22	85 ± 29

The time interval between the follow-up and baseline images, and the number of manually annotated landmarks in tumor and normal tissue regions are listed for each patient. (FU: follow-up; BL: baseline; LM: landmarks.)

Two breast imaging radiologists (SW and EC) annotated landmarks independently in the images, blinded to the patient response outcome. First, they agreed on a common set of anatomical/geometric landmarks in the baseline images. Then, they independently defined in the follow-up images the landmarks, which, according to their expert knowledge, were corresponding to the same set of landmarks in the baseline images. The baseline and follow-up images were displayed side-by-side to facilitate expert annotations. The “New VOI”—“Draw Point VOI” function in the MIPAV software (67) was used by the experts to annotate the landmark correspondences (VOI stands for volume of interest). When an expert chose a landmark location by a click of the mouse, the “Draw Point VOI” function placed a cross in the image at the same location. At the same time, a number was associated with the cross, indexing the order in which the landmarks were annotated (i.e., Fig. 2). Locations of the same index numbers were considered corresponding landmark locations defined by experts in the two images. In general, landmarks were scattered in the image space to measure the accuracy of registration in various regions. Landmarks were usually located at nipples, breast boundaries, chest walls, internal milk ducts, vessels, glandular structures as well as tumors. As such, a total of 2,380 landmark pairs (85 ± 29 per patient) were annotated and labeled as being within tumor or normal tissue regions. As Table 1 and Figure 2 show, patients in the pCR and pPR subgroups have similar numbers of landmarks in the normal tissue regions, but pCR patients have significantly fewer landmarks in the tumor regions, highlighting the larger changes in tumors and hence an increased level of ambiguity in finding corresponding landmarks.

FIG. 2.

Example landmarks annotated by human experts in a representative patient (patient #1). Landmarks with the same indexing numbers in different images correspond to each other, according to the expert annotations. Note that the images have been manually cropped for the display purpose. In the automated registration, however, we used the entire images as shown in the first rows in Figs. 7 and 8, where no manual or automated cropping was involved.

Figure 2 shows some example expert-annotated landmark correspondences in a representative subject (i.e., patient #1). Landmarks of the same indexing number in the different images correspond to each other according to expert annotations. Please note that, for the same set of landmarks in the baseline image (in the left panel of the figure), the annotated corresponding landmark locations by expert 1 (in the middle panel) and expert 2 (in the right panel) were slightly different. Such differences can occur in the in-plane x–y locations, and also in the z-direction (see, e.g., the different slice numbers at the upper right corners of subfigures). The distribution of interexpert landmark differences can be found in the box-plots shown in Figure 4. In general, experts differ by 0–5 mm at a majority of (>80%) landmark locations, and they differ by more than 10 mm only at few (<5%) landmark locations.

FIG. 4.

The overall landmark errors in all 14 subjects, for interexpert errors and algorithm-to-expert errors.

All images were converted into the NIfTI image format. Before registration, they all went through the N3-based bias field correction (using the default parameters; 68). Histograms of the follow-up images were matched to histograms of the baseline images, using the histogram matching module of the MIPAV software (67). This was to reduce intensity differences caused by imaging artifacts or by the imaging sequences used at different times.

Registration Methods to be Evaluated

The registration methods included in this evaluation work satisfied two criteria:

1. They were previously reported in the literature to register (long-interval) longitudinal breast MR images for measuring the volumetric changes of tumors, and their corresponding implementation parameters were disclosed.

2. Their software packages are publicly available, therefore, allowing a wide use of them such as in this study.

Through our search of the literature, we found six registration algorithms satisfying these criteria. They have been implemented in three software packages: Demons (69,70), free form deformation (FFD; (71)), and DRAMMS (19). Demons was directly used in (17) and slightly modified in (72) to register breast MRIs acquired weeks or months apart (73). An FFD variant version (74) was used in (18) to register longitudinal breast MR images with known synthetic tumor changes. DRAMMS was used in (19,75) but was compared to FFD for longitudinal breast MRI registration only qualitatively. Below we summarize the registration methods evaluated in our study.

• (Additive) Demons and Diffeomorphic Demons: Demons (69,70) is an intensity-based deformable registration framework. It considers deformable image registration as a nonparametric diffusion process. It matches voxels to their correspondences according to local intensity characterizations. Using the intensity difference as the similarity metric, a force is computed from the optical flow equations to push voxels by some velocity that is iteratively combined with the total displacement (initially zero), which is later smoothed by Gaussian filters for regularization. (Additive) Demons (69) adds the velocity field to the current deformation in each iteration, whereas Diffeomorphic Demons (70) composes the deformation with the exponential of the velocity field, enforcing diffeomorphism. In this article, both (Additive) Demons and Diffeomorphic Demons were included in the evaluation. They are both available in the Demons software, which can be downloaded from http://www.insight-journal.org/browse/publication/154.

• FFD and its variants: FFD is a geometric transformation model introduced to deformable image registration in 1999 (71) and widely used thereafter. It is often used in combination with intensity-based matching criteria, hence intensity-based registration algorithms. In the FFD model, a regular grid of “control points” is superimposed on top of the dense image lattice. The main idea is that the movement of an image voxel is a smooth, cubic-B-spline-based interpolation of the displacements of the neighboring control points. Therefore, the task of finding movements at each voxel is translated into finding the displacements at a much smaller number of regularly spaced control points. In contrast to landmark-based methods, FFD has three properties: (1) control points are regularly spaced in the image, providing guidance throughout the image domain; (2) deformation is regularized by the cubic-B-spline-based interpolation; and (3) a landmark moves by finding its own correspondence, whereas a control point moves by finding the most likely corresponding patches for the image patches it represents. The FFD transformation model can be combined with similarity metrics such as the normalized mutual information (NMI), the correlation coefficient (CC), and the sum of squared difference (SSD) of the two images to be registered. NMI-FFD, CC-FFD, and SSD-FFD were all included in our evaluation. The software package, known as IRTK, is available at http://www.doc.ic.ac.uk/dr/software/.

• DRAMMS: DRAMMS (19) is an attribute-based deformable image registration algorithm. It characterizes voxels by the high-dimensional Gabor texture attributes extracted from the multiscale and multiorientation neighborhoods of voxels. This has the potential to increase the accuracy and reliability of voxelwise matching, especially in the presence of intensity inhomogeneity and contrast differences between longitudinal scans. Besides, DRAMMS uses a newly developed mutual-saliency function, which automatically quantifies the matching reliability at the voxel level. Based on this function, DRAMMS guides the registration with those voxels/regions having higher confidence to establish correspondences across images. DRAMMS uses the cubic-B-spline FFD transformation model ((71); described above) and the discrete optimization strategy (76,77). The DRAMMS software package is available at http://www.cbica.upenn.edu/sbia/software/dramms/.

Parameter Settings in Registration Methods

For (Additive) Demons and Diffeomorphic Demons, the key parameters are (i) the Gaussian smoothing kernel (σ₁) of the velocity field; and (ii) the Gaussian smoothing kernel (σ₂) after having combined the velocity field into the overall deformation field. Larger smoothing kernels σ₁; σ₂ lead to smoother deformations. Study (17) used Demons directly to recover breast changes in longitudinal MR images 3 months apart, which is fairly similar to our dataset with the same objective. In their study, Demons was used with σ₁ = 8 and σ₂ = 1. In another study (73), which registered breast MR images weeks apart, an algorithm similar to Demons was used with parameter σ₁ = 7. In a related study (78) registering breast elastography, which has image sizes and voxel sizes similar to breast MRI, the authors used σ₂ = 1.5. Since the parameters in all three independent studies were similar, we also followed their settings and chose the Demons parameters at σ₁ = 8 and σ₂ = 1.5. In addition, we allowed the maximum number of iterations in three image resolutions to be 30 × 20 × 10, which doubles the default maximum number of iterations and was used in a recent studies (35,80) to ensure the convergence of the program.

For FFD and its variants, the key parameters are (i) the control point spacing δ, with a larger spacing capturing more global deformations; and (ii) the regularization weight λ, with larger weights providing smoother deformations. We set δ = 10 mm and λ = 0.01 following the original FFD paper (71), which, together with other studies from the same group [e.g., (48,60)], found this set of parameters to perform stably and well suit the breast MR image registration task.

For DRAMMS, most parameters are automatically adaptive to the input images (i.e., size, contrast, and histogram), and hence do not need extensive tuning. One parameter that may need to be tuned is the regularization weight g, which is usually set between 0 and 1, with higher weights for smoother deformations, and even greater than 1 for very smooth deformations. We set g = 0.3, which led to slightly smoother deformations than using the default value g = 0.2. This was also the parameter used in its original works (19,75) for longitudinal breast MR image registration experiments.

Note that the parameters mentioned above were not set arbitrarily. Usually, the authors of the aforementioned studies tried a wide range of parameter settings, and reported the ones that led to the most stable and/or most accurate alignment of breast MR images.

Evaluation Criterion for Registration Accuracy

Evaluating the accuracy of registration is a difficult task in general. This is mainly due to the lack of the ground-truth deformation at each and every voxel. In other registration evaluation studies (e.g., for brain, lung, or short-interval longitudinal breast MR images), existing work has used simulated images/deformations [e.g., (41,44,45,51,79)], expert-annotated regions of interest [ROI; e.g., (35,38,43)], or expert-annotated landmarks [e.g., (43,73)]. While simulations provide ground-truth deformations, they do not offer fully realistic deformations as in real images. In our evaluation study, we aimed to measure accuracy directly in clinical longitudinal breast MR images. Approaches based on ROIs have been used more often for well-defined and highly localized structures such as some cortical structures in the brain, whereas the human breast does not necessarily have many such well-defined structures across individuals. For these reasons, registration accuracy in our evaluation study was assessed based on expert-defined landmarks.

As conceptually depicted in Figure 3, we measured interexpert landmark errors as well as algorithm-to-expert landmark errors. Assume that we have a set of expert-annotated landmarks ${\left\{x_n\right\}}_{n=1}^N$ in the baseline image (N is 85 on average, as section Dataset and Table 1 show). Two experts (SW and EC) independently found their corresponding points in the follow-up image, denoted as ${\left\{y_n^{\text{expert}1}\right\}}_{n=1}^N$ and ${\left\{y_n^{\text{expert}2}\right\}}_{n=1}^N$. Conversely, the deformation calculated by a registration algorithm maps the same set of landmarks ${\left\{x_n\right\}}_{n=1}^N$ to locations ${\left\{y_n^{\text{algoritham}}\right\}}_{n=1}^N$. We could, therefore, define interexpert landmark errors = (i.e., the length of the solid line in Fig. 3) as

$$d(y_n^{\text{expert}1},y_n^{\text{expert}2}),\text{for}n=1,2,\dots ,N$$ [1]

and algorithm-to-expert landmark errors (i.e., the average length of the dashed lines in Fig. 3) as

$$\begin{array}{l}\frac{1}{2}\left[d(y_n^{\text{algorithm}},y_n^{\text{expert}1})+d(y_n^{\text{algorithm}},y_n^{\text{expert}2})\right],\\ \text{for}n=1,2,\dots ,N\end{array}$$ [2]

where d(·, ·) is the Euclidean distance between two voxel locations.

FIG. 3.

Depiction of the interexpert landmark errors (the length of the solid line) and the algorithm-to-expert landmark errors (the average length of the dashed lines).

We can further analyze landmark errors for different regions, based on the prior knowledge whether a landmark x_n belongs to tumor or normal tissue regions. We can also analyze landmark errors in patient subgroups having different responses to neoadjuvant chemotherapy, based on the prior knowledge of whether a patient belongs to the pCR or pPR subgroup.

Using Registration to Quantify Tumor Changes During Neoadjuvant Chemotherapy

The registration method obtaining the highest accuracy (i.e., smallest landmark errors) was used to quantify tumor changes in patients undergoing neoadjuvant chemotherapy. We first registered the follow-up images of each patient to their baseline scans. The obtained deformation fields were further used to calculate the Jacobian determinant maps. The Jacobian determinants, computed at the voxel level, show the volumetric change ratio (follow-up/baseline) at each voxel. Mathematically, suppose a 3D baseline image is indexed by three orthogonal axes i; j; k in a Cartesian coordinate system, a voxel is denoted as u = (u_i; u_j; u_k), and a deformation field computed from the registration of follow-up to baseline images is denoted as h(u) = (h_i(u), h_j(u), h_k(u)) at every voxel u. Then the Jacobian determinant at voxel u, denoted as JacDet(u), is defined as

$$\text{JacDet}(\mathbf{u})=\text{det}(\text{Jac}(\mathbf{h}(\mathbf{u}))).$$ [3]

where

$$\text{Jac}(\mathbf{h}(\mathbf{u}))=\left(\begin{array}{ccc}\frac{\partial h_i(\mathbf{u})}{\partial i}& \frac{\partial h_i(\mathbf{u})}{\partial j}& \frac{\partial h_i(\mathbf{u})}{\partial k}\\ \frac{\partial h_j(\mathbf{u})}{\partial i}& \frac{\partial h_j(\mathbf{u})}{\partial j}& \frac{\partial h_j(\mathbf{u})}{\partial k}\\ \frac{\partial h_k(\mathbf{u})}{\partial i}& \frac{\partial h_k(\mathbf{u})}{\partial j}& \frac{\partial h_k(\mathbf{u})}{\partial k}\end{array}\right)$$ [4]

and det (·) is the determinant of a matrix. The Jacobian determinant value, therefore, shows the expansion of a voxel if greater than 1, shrinkage if smaller than 1, and volume-preservation if exactly 1.

RESULTS

Evaluation Results and Observations

This section presents our evaluation results. We measured the landmark-based registration accuracy in the entire image, in different regions (tumor vs. normal tissue), and in different patient subgroups having different pathologic responses to chemotherapy. The results are presented in the next three subsections, respectively. Note that we stratified registration accuracy by regions and pathologic subgroups, since it is our hypothesis that patients in the pCR subgroup may pose higher levels of difficulty to registration due to potentially larger changes in the tumor regions after treatment.

Overall Landmark Errors

The mean and standard deviation errors for different registration methods are listed in Table 2. Furthermore, Figure 4 shows the box-whisker plot of the landmark errors. Several observations can be made:

1. The average landmark error between human experts was 3.12 ± 2.84 mm. This error was higher than the errors in the registration of short-interval longitudinal breast MR images (usually pre- and postcontrast DCE-MRI), where studies have reported subvoxel registration accuracy (50,52). This showed the increased level of registration difficulty in longitudinal breast MR images acquired days or weeks apart. It was perhaps caused by several factors, including the patient repositioning, the soft tissue deformation, and the tumor changes due to treatment.

2. The landmark errors of all the automated registration methods evaluated were larger than the average error between human experts.

3. Among the automated methods included in our evaluation, DRAMMS performed closest to human experts, with landmark errors at 6.05 ± 4.86 mm. FFD and its variants followed, in the order of CC-FFD, NMI-FFD, and SSD-FFD, all with a statistically significant difference from DRAMMS. Demons and Diffeomorphic Demons had the largest landmark errors, with errors at 10.76 ± 6.01 mm and 10.82 ± 6.11 mm, respectively (which are similar to the 11.67 ± 4.81 mm error reported in (73) on a similar dataset). The difference between Demons and Diffeomorphic Demons was not significant (P = 0.0474), if we only consider the differences with P < 0.01 to be statistically significant.

Table 2.

Mean and Standard Deviation (stdev) of Landmark Errors for Different Registration Methods (mm)

	Interexpert	Demons	Diff. Demons	DRAMMS	CC-FFD	NMI-FFD	SSD-FFD
Mean	3.12	10.76	10.82	6.05	8.07	8.21	9.46
Stdev	2.84	6.01	6.11	4.86	3.86	3.81	4.55

Numbers in the bold font indicate either the landmark errors between human experts, or the smallest landmark errors (mean or standard deviation) of automated algorithms as compared with human experts.

Landmark Errors in Different Regions

We further examined landmark errors in tumor versus normal tissue regions. As Figure 5 shows, methods followed the same rank for landmark errors in tumor regions as for landmark errors in the entire image. In tumor regions, DRAMMS had the smallest landmark errors among all automated methods (5.64 ± 4.79 mm), then NMI/CC/SSD-FFD (8.33 ± 3.87, 8.33 ± 3.88, and 8.74 ± 4.38 mm), and then Demons (9.28 ± 5.42 mm) and Diffeomorphic Demons (9.45 ± 5.57 mm). In normal tissue regions, DRAMMS had the smallest landmark errors among all automated methods (6.41 ± 4.90 mm), then CC/NMI/SSD-FFD (8.11 ± 3.76 mm, 7.87 δ3.83 mm, 9.29 ± 4.68 mm) and then Diffeomorphic Demons (12.07 ± 6.32 mm) and Demons (12.09 ± 6.22 mm).

FIG. 5.

Landmark errors in different image regions. The P-values from Student’s two-sample t-test represent the significance of difference between landmark errors in the tumor versus in the normal tissue regions. Purple stars note registration methods which obtained significantly lower errors in the tumor regions compared to errors in the normal tissue regions.

One noteworthy finding is that, DRAMMS, Demons, and Diffeomorphic Demons all had significantly smaller landmark errors in the tumor regions than in the normal tissue regions (evidenced by the P-values provided in Fig. 5). One possible explanation may be that, the soft normal breast tissues exhibit more complicated deformations than the relatively solid tumors. This might be the case especially when considering factors such as patient movements and breast positioning differences in different visits, which may affect the appearances of normal tissue regions more than the tumor regions. Another potential explanation might be that the heterogeneity within the tumor regions may have helped the automated registration algorithms to better locate anatomical correspondences. Conversely, as observed in Figure 6b,c, the errors in tumor regions were actually larger than in normal tissue regions for the pCR subgroup. This suggested that the effect of the treatment may override the comparison of landmark errors in tumor versus normal tissue regions. Finally, whether the landmark differences were significant in different regions seems to also depend on which registration algorithm was used (see Figs 5 and 6). Considering all these possible factors, a larger-scale, carefully controlled study may be needed to fully evaluate the reason why and how landmark errors may differ in tumor versus normal tissue regions.

FIG. 6.

Comparison of landmark errors in the pCR and pPR patient subgroups, in (a) the entire image; (b) the tumor regions; and (c) the normal regions. The differences with P-values smaller than 0.01 are treated as statistically significant, which are noted by the purple stars in the figure.

Landmark Errors and Response to Chemotherapy

The previous subsection showed that registration accuracy may vary by image regions. This subsection will show that registration accuracy may also vary by patient subgroups. Figure 6 compares the landmark errors between the pCR and pPR subgroups, in (a) the entire image, (b) tumor regions, and (c) normal tissue regions. The P-values from the Student’s t-test were provided and the statistical significance (when P < 0.01) was noted by purple stars in the figure. Some observations can be made:

1. The pPR patients almost always had smaller registration errors than pCR patients, regardless of the registration methods being used, and regardless of tumor versus normal tissue regions.

2. The difference of the registration errors between the pPR and pCR patient subgroups were almost always statistically significant in the entire image, except for the expert raters (upper subfigure in Fig. 6). This was most likely driven by the statistically significant difference in landmark errors between the pPR and pCR patients in the tumor regions, except for the expert raters (middle subfigure). Conversely, registration errors in the normal tissue regions remained almost the same, regardless of a patient’s response to neoadjuvant chemotherapy.

Combining these two observations, the larger registration errors in the pCR subgroup as compared with the pPR subgroup may be partly because the tumors changed more in the pCR subgroup than in the pPR subgroup, which posed greater challenges especially for the automated registration methods.

Application to the Quantification of Tumor Changes

Because DRAMMS had higher accuracy than other automated registration methods evaluated, in this section, we show preliminary experiments of using the DRAMMS registration algorithm to quantify tumor changes in patients undergoing neoadjuvant chemotherapy. Examples of the registered images are shown in the second rows of Figures 7 and 8, for a representative pPR patient and a representative pCR patient, respectively. Jacobian determinant maps were computed from the obtained deformation fields, as shown in the third rows in Figures 7 and 8, which are overlaid onto the baseline intensity images. Here, the Jacobian determinant maps are shown only within tumor regions. The tumor regions were obtained in the baseline images by a user-interactive intensity thresholding step, followed by a human expert’s verification of the segmented tumor boundary. Note that the tumor regions were in no means used for image registration or for the calculation of Jacobian determinant maps; they were only used as masks to visualize the Jacobian determinants specifically within the tumor regions. As shown in Figures 7 and 8, the quantified volumetric changes agree with the visual findings of how tumors have changed due to the effect of treatment. Moreover, the Jacobian determinant maps can show the spatially heterogeneous changes within the tumor regions, which may be important features for predicting the long-term pathologic response as suggested in recent studies (30–33). Our future work will be to further validate the automatically computed heterogeneous tumor changes as quantitative imaging biomarkers for predicting the response to neoadjuvant chemotherapy.

FIG. 8.

Quantification of the voxelwise longitudinal breast cancer change by the DRAMMS registration algorithm. Results shown in this figure are from an example pathologic complete responder (pCR) patient. Jacobian determinants in the tumor regions are overlaid on the baseline image in the bottom row.

DISCUSSION

Summary of Work and Findings

Deformable image registration can automatically capture 3-D volumetric changes at the voxel level. It is, therefore, an appropriate choice for quantifying the spatially heterogeneous changes within tumors. Our study evaluated DRAMMS, an attribute-based image registration method, in comparison to five other intensity-based algorithms, for the task of registering long-interval longitudinal breast MR images. We compared the accuracies of registration methods using expert-defined landmarks as references. We also showed preliminary results of using the DRAMMS registration method, which had the smallest landmark errors, to quantify longitudinal tumor changes induced by treatment. As such, our work adds into the literature several important findings:

First, automated registration methods differed greatly in their performances. The attribute-matching component helped DRAMMS to obtain the smallest landmark errors, almost 25–40% smaller than the errors from five other intensity-based methods. The high-dimensional geometric/texture attributes that DRAMMS uses seemed to be less sensitive to the image noise, the intensity inhomogeneity, or the partial volume effects. Because of this, DRAMMS also seemed to require fewer parameter tunings. Besides longitudinal breast images presented in this study, another recent study on the intersubject brain image registration also found a similar trend, that DRAMMS performed relatively accurately, without much need for parameter tunings (80).

Second, our analysis on image regions and patient subgroups showed that, the voxel correspondences were least ambiguous in the tumor regions of patients in the pPR subgroup (where tumors expanded or shrank but did not disappear), then a little more ambiguous in the normal tissue regions (almost equivalent between the pCR and pPR subgroups), and most ambiguous in the tumor regions of patients in the pCR subgroup (perhaps because tumors changed more, or even disappeared, which caused a possible loss of correspondences in some locations). This trend has been observed for both experts and most algorithms, as can be observed from the landmark errors in panels (b) and (c) of Figure 6.

Third, because of the higher accuracy of DRAMMS compared to other registration methods, we further evaluated its ability to quantify longitudinal tumor changes as induced by neoadjuvant chemotherapy for breast cancer. Although preliminary, our results were promising. Those results showed the ability of DRAMMS to capture the changes agreeable to human observations. Furthermore, those results also showed the ability of DRAMMS to characterize the spatially heterogeneous effects induced by treatment, which may be ultimately used as more sensitive markers for evaluating treatment response and predicting patient outcomes.

Future Work

On further evaluating the performance of registering longitudinal breast MR images, our future work will include (a) the quantitative evaluation of registration accuracies on longitudinal images with more than one follow-up time point; (b) the evaluation of the performance of registration in larger datasets and for a broader range of registration algorithms [although not being reported before as being used in this context, many deformable registration algorithms/tools may well serve this specific registration task, such as elastix (81), ANTs (82), ART (83), DROP (76), HAMMER (84), MIND (85), NiftyReg (86), and plastimath (87)]; and (c) the evaluation of the change of registration performance as registration parameters vary. Also, the quantification and better understanding of the genuine ambiguity for human experts to annotate landmarks is another important topic to explore. It may require a substantial amount of work, as it could involve many factors such as varying landmark locations and deriving some precise quantitative measures of the intra-rater ambiguity.

On further improving the accuracy of registering longitudinal breast MR images, our future work will extend the current 3-D registration into 4-D (3-D + time) or groupwise registration [e.g., (54,88–92)]. As tumor changes are often temporally smooth, registering a full 4-D spectrum of longitudinal images at the same time can better maintain the temporal smoothness. In addition, we can incorporate some task-specific prior knowledge to further aid the registration, such as the removal of image background and the chest region [e.g., (93–95)], the automatically detected landmarks [e.g., (96,97)], the segmentation of various breast tissue types [e.g., tumor vs. normal fibroglandular tissue (98,99)], and even the explicit segmentation of tumors [e.g., (100–102)]. Such modifications could further aid the registration to focus on ROI, and could largely reduce the negative impacts from structures that are not of interest to the specific application. Finally, we could also explore methodologies to form a consensus mechanism, where multiple registration methods can complement each other toward more accurately capturing voxelwise volumetric changes [e.g., (103,104)].

Finally, based on our reported results, our future work will also directly evaluate registration-quantified tumor changes. This would include (a) a direct comparison with the RECIST and other segmentation-based approaches, for the accuracy of the quantified morphologic changes; and (b) the use of registration-quantified changes and the registration-revealed heterogeneity of changes to predict long-term pathologic response to treatment. This should have an impact on treatment optimization for breast cancer patients.

CONCLUSION

The registration of longitudinal breast MR images acquired days or weeks apart during the course of treatment has gained increasing interest as a tool, because it can be used to quantify tumor changes, assess treatment effects, and potentially predict long-term response and patient survival. This article performed a comprehensive evaluation of the registration of longitudinal breast MR image, using expert-annotated landmarks as references. Specifically, we evaluated DRAMMS, an attribute-based registration method, and compared it with five other intensity-based registration methods. Our results showed that DRAMMS can achieve significant improvements in registration accuracy over other automated registration methods included in this study. This was possibly due to the use of the high-dimensional voxel neighborhood information to characterize the geometric and anatomical properties of voxels, which seemed to be more robust to the noise, the intensity inhomogeneity and the contrast changes between images. We also showed some preliminary evidence of using DRAMMS to quantify the longitudinal tumor changes, and the spatial heterogeneity, as induced by neoadjuvant chemotherapy. Our ultimate goal is to use registration-quantified tumor changes to more accurately predict long-term pathologic response and patient survival after treatment.