Technical Note: Deformable image registration on partially matched images for radiotherapy applications

Abstract

In radiation therapy applications, deformable image registrations (DIRs) are often carried out between two images that only partially match. Image mismatching could present as superior-inferior coverage differences, field-of-view (FOV) cutoffs, or motion crossing the image boundaries. In this study, the authors propose a method to improve the existing DIR algorithms so that DIR can be carried out in such situations. The basic idea is to extend the image volumes and define the extension voxels (outside the FOV or outside the original image volume) as NaN (not-a-number) values that are transparent to all floating-point computations in the DIR algorithms. Registrations are then carried out with one additional rule that NaN voxels can match any voxels. In this way, the matched sections of the images are registered properly, and the mismatched sections of the images are registered to NaN voxels. This method makes it possible to perform DIR on partially matched images that otherwise are difficult to register. It may also improve DIR accuracy, especially near or in the mismatched image regions.

I. INTRODUCTION

Image registration is a procedure for transforming different image datasets into a common coordinate system so that the corresponding points in the images are matched and allow complementary information from the different image datasets to be used for various diagnostic and therapeutic purposes.¹ Image registration plays increasingly important roles in radiation therapy applications since more and more anatomical images (kVCT, daily megavoltage-CT or cone-beam-CT, MRI, etc.) and functional images (PET, SPECT, fMRI, etc.) are adopted into patient radiation treatment management. Image registration algorithms can be generally grouped into rigid and deformable (nonrigid) registrations.² While rigid registration applies only affine transformations with a limited number of free parameters (up to 12), deformable image registration (DIR) uses a much larger number of parameters to describe tissue deformation.

DIR³ can be computed based on features extracted from the images (e.g., points,⁴ lines,⁵ and surfaces⁶) or based on similarity metrics directly derived from image intensity values [e.g., sum of squared differences (SSD),⁷ mutual information (MI),⁸ cross correlation,⁹ and correlation ratio¹⁰]. There are also different categories of algorithms based on the transformation models and the optimization schemes, for example, the thin-plate spline algorithm,¹¹ B-spline algorithm,^12,13 optical flow algorithms,^7,14,15 diffeomorphic algorithms,^16,17 block matching algorithms,¹⁸ finite element model (FEM) based algorithms,^19,20 etc. Single modality CT-to-CT DIR is especially important for radiation therapy applications such as patient response monitoring, treatment adaptation, dose tracking, and patient motion modeling.^15,21,22

There are many situations that DIR needs to be carried out between two image datasets that only partially match. For example, (1) the daily CT images (kV-CBCT, MV-CBCT, or tomotherapy MVCT) have smaller field of views (FOVs) and superior-inferior coverage than the corresponding treatment planning CT images, as shown in Figs. 1(a)–1(d), (2) two multiple-slice free-breathing ciné mode lung CT scans at the same couch table position but different breathing phases may have different image contents due to breathing motion, as shown in Figs. 1(e) and 1(f). DIR algorithms usually assume that the two images to be registered have matching contents in the whole image domain. If the two images are only partially matched, there will be registration problems associated with the mismatched image contents, as shown in Fig. 2. Currently, a general solution is to crop the images so that the moving image can cover the fixed image entirely in both volume and image content. The disadvantages are (1) registration results will be only defined on the smaller cropped image volumes and (2) useful image information is lost because of cropping.

FIG. 1.

Examples of images partially matching situations. (a)–(d) are the scans of a prostate cancer patient. (a) and (c) are tomotherapy MVCT scans. (b) and (d) are the corresponding conventional kVCT scans. (a) and (b) demonstrate the different superior-inferior coverage between the MVCT and kVCT. The dashed lines in (b) and (d) mark the boundaries of the corresponding MVCT volume. (c) and (d) demonstrate the differences in FOV. The circle in (c) is the tomotherapy MVCT FOV circle. The rectangle marks the maximal cropped region if the MVCT image is cropped to be completely within the FOV. (e) and (f) are two 64-slice axial CT lung scans at the same couch position but at different phases of the breathing cycle. The arrows in (e) demonstrate the directions of tissue motion to match the scan in (f).

FIG. 2.

Demonstration of registration problems on the 64-slice lung CT scans. The CT scan in Fig. 1(e) is used as the moving image and the CT scan in Fig. 1(f) is used as the fixed image. (a) and (d) are the deformed moving image. (b) is the moving image in transverse view. (c) is the fixed image in transverse view. The arrows indicate the most significant registration errors. The motion vectors in (a) and (d) also show the incorrect transverse motions as the diaphragm should move mainly in the superior-inferior direction instead of in the transverse direction.

II. METHOD

In this article, we propose a method to improve the existing DIR algorithms so that DIR can be carried out in such image mismatching situations. The basic idea is to use NaN values to extend the image volumes instead of cropping them and to allow NaN voxels to match to any voxels in the registration computation. As the results, the matched sections between the images are registered and the mismatched sections of the images will be mapped to NaN voxels.

For two images ($I$ and $J$) to be registered, let $I$ be the moving image and $J$ be the fixed image, a deformation vector field (DVF) $V$ registers $I$ to $J$ when the general system energy equation is optimized,

$$E={\int }_{\Omega }S(V\left(I\right),J)d\Omega +{\alpha }^2{\int }_{\Omega }R\left(V\right)d\Omega ,$$ (1)

where $V\left(I\right)$ is the deformed moving image $I$, $S$ is the similarity function, $R$ is the smoothness constraint function, $\Omega$ is the image domain, and $\alpha$ is a configurable constant. To allow mismatched images, we slightly modify the system energy equation to

$$E^{\prime }={\int }_{{\Omega }^{\prime }}{S}^{\prime }(V\left(I^{\prime }\right),J^{\prime })d{\Omega }^{\prime }+{\alpha }^2{\int }_{{\Omega }^{\prime }}R\left(V\right)d{\Omega }^{\prime },$$ (2)

where $I^{\prime }$ and $J^{\prime }$ are the images with their volumes extended with NaN values, ${\Omega }^{\prime }$ is the extended image domain, the similarity measurement function $S^{\prime }$ is slightly modified to take only valid (non-NaN) voxels. For example, if SSD is used as the similarity metric, the similarity term can be written as

$${\int }_{{\Omega }^{\prime }}{S}^{\prime }(V\left(I^{\prime }\right),J^{\prime })d{\Omega }^{\prime }={\int }_{{\Omega }^{\prime }}Z\left({(V\left(I^{\prime }\right)-J^{\prime })}^2\right)d{\Omega }^{\prime },$$ (3)

where the NaN value checking function $Z\left(x\right)$ is defined as

$$Z\left(x\right)=\{\begin{array}{ll}0& x\text{is}\mathrm{Na}\mathrm{N}\\ x& x\text{is}\text{not}\mathrm{Na}\mathrm{N}.\end{array}\phantom{\}}$$ (4)

However, adding the function $Z$ into the system energy equation (2) makes Eq. (2) not differentiable because $Z$ is not differentiable and there is no simple way to write it in a differentiable formulation. One approximation is to defer the NaN checking from the system energy equation to iterative solutions by which the system equation is solved. To demonstrate such an approximation, we use the Horn–Schunck¹⁴ (HS) optical flow algorithm which computes $DVF$ using the following iterative equation:

$$V_{\mathrm{k}+1}={\overline{V}}_{\mathrm{k}}-\frac{(J-I+{\overline{V}}_{\mathrm{k}}\cdot \nabla I)\nabla I}{{\alpha }^2+{|\nabla I|}^2},$$ (5)

where $V_k$ is the $DVF$ at iteration $k$ and ${\overline{V}}_{\mathrm{k}}$ is $V_{\mathrm{k}}$ averaged in the neighborhood of every voxel. $V_0$ is initialized to all 0 for $k=0$. To check the NaN values in the iteration, this equation is modified to

$$V_{\mathrm{k}+1}={\overline{V}}_{\mathrm{k}}-Z\left(\frac{(J^{\prime }-I^{\prime }+{\overline{V}}_{\mathrm{k}}\cdot \nabla I^{\prime })\nabla I^{\prime }}{{\alpha }^2+{|\nabla I^{\prime }|}^2}\right).$$ (6)

Even though there are only slight modifications in the system energy equation and the iterative solution, there are several aspects that require further explanation. First, the reason to use NaN instead of other floating values for undefined voxels is that NaN is transparent to all arithmetic operations that DIR computations are based on. The results of arithmetic operations, such as addition, multiplication, etc., will be NaN if any operand of an arithmetic operation is a NaN. The undefined voxels need to be tracked through out the DIR iterative computations. NaN values are appropriate as the indicators for these voxels while other floating-point values, such as 0 or negative values, could not straightforwardly serve the same purpose because they could not maintain their values throughout arithmetic operations as NaN values do.

The second issue is whether allowing NaN voxels to be registered to any voxels (whether or not they are NaN voxels) would cause inaccurate registrations. Image intensity information is not available for undefined voxels; so there is no way to determine the image motion for these voxels. Assigning NaN values to these voxels and allowing them to match to any voxels is equivalent to not computing image motion on these undefined voxels. Global smoothing mechanisms of the DIR algorithms help to propagate motion from the valid image regions to the NaN region. Therefore, image motions on the undefined voxels are estimated using image motion information from the neighboring voxels containing valid intensity values.

For the modified HS algorithm, the function $Z$ checks the $DVF$ adjustment [the operand of function $Z$ in Eq. (6)], denoted as $\Delta V$, at every iteration. $\Delta V$ will be NaN for the undefined voxels as well as neighboring voxels because of image gradient and deformation computations that use neighboring voxel intensities. How a NaN voxel affects $\Delta V$ in its neighborhood depends on the actual implementation of image gradient computation and image deformation. In our implementation, the image gradient is computed in a way similar to $(I_{\mathrm{n}+1}-I_{\mathrm{n}-1})∕2$ and trilinear interpolation is used for image deformation. A NaN voxel will affect $\Delta V$ in the neighboring voxels but not further. One should note that the $V$ in Eq. (6) will not have NaN values after the function $Z$ is used. This ensures that global smoothness mechanisms applied during the iterations will continue to work and will diffuse the DVF into the undefined image regions.

Finally, an image boundary should be extended far enough to cover the maximal image displacement at the boundary. Smoothing operations, such as Gaussian low-pass filtering, will propagate NaN values into the neighboring voxels. Necessary image smoothing should be carried out before the image volumes are expanded with NaN voxels.

III. RESULTS AND VALIDATION

Figure 3 shows the results of registration between two 64-slice axial CT scans with mismatched image contents. Both CT images were $512\times 512\times 64$ with voxel size of $0.9765\times 0.9765\times 0.625{\mathrm{mm}}^3$. They were acquired at the same couch table position but different breathing tidal volumes. This 4DCT registration situation has been an interested problem for respiratory motion studies^23,24 but has never been successfully solved previously. The proposed method in this paper makes a better solution for this problem. As shown in Fig. 3, we can directly register these two 4DCT scans by extending them in the superior and inferior directions by eight slices $\left(5\mathrm{mm}\right)$ with NaN voxels before registration. Compared to the results in Fig. 2, the results in Fig. 3 are visually better.

FIG. 3.

New registration results using the NaN voxel extension. Both CT scans are extended in the superior-inferior and lateral directions with NaN voxels. (a) and (f) are the moving image. (b) and (g) are the fixed image. (c) and (h) are the deformed moving image. The dotted lines in (c) mark the original CT scan boundary before NaN voxel extension. (d) is the different image before registration. (e) is the different images after registration. (i) is the checkerboard image after registration. The arrows in (c), (h), and (i) indicate the deformed NaN voxels. The transverse slice shown in (f) to (i) is marked by the dashed line in (c).

We have performed landmark-based validation for this 4DCT example. For 20 manually selected corresponding landmarks (lung-bronchi bifurcation points) in both CT scans, the average distance between the corresponding landmark pairs was $4.3\mathrm{mm}$ before registration. The registration errors, which are defined as the absolute distances from the registered landmark points in the deformed moving image to the target landmark points in the fixed image, were $0.8\pm 0.3\mathrm{mm}$.

We have also performed quantitative validation for the MVCT and kVCT volume and FOV mismatching situations shown in Fig. 1. We used a digital phantom (a kVCT image defined by a known $DVF$) with an artificial FOV cutoff. We carried DIR using the original kVCT as the moving image and the digital phantom as the fixed image. We quantitatively compared the registration errors against those from a regular DIR without using NaN extension. The results listed in Table I suggest that the proposed method could reduce registration errors, comparing to regular DIR results for the FOV mismatching situations.

TABLE I.

Registration results comparison for the MVCT FOV truncation situations.

Image region	Ground truth imagedisplacement(mm)	Registration errors (mm)after setting voxelsoutside FOV to NaN	Registration errors(mm) after settingvoxels outside FOV to 0
Entire image	$8.2\pm 4.8$	$3.0\pm 3.2$	$8.6\pm 10.8$
Inside FOV	$9.7\pm 4.2$	$2.9\pm 3.1$	$4.1\pm 5.0$
Outside FOV	$7.3\pm 5.2$	$5.7\pm 4.9$	$13.3\pm 12.9$
Within $10\text{pixels}$ insideFOV	$9.2\pm 4.7$	$2.8\pm 2.5$	$6.5\pm 7.2$
Within $10\text{pixels}$ outsideFOV	$8.5\pm 4.3$	$4.8\pm 3.5$	$19.7\pm 17.1$

IV. DISCUSSION AND CONCLUSION

The proposed method is able to estimate the DVF in the image mismatching image regions. Therefore, it has the advantage to allow DIR computation in larger image volumes comparing to the regular approach which crops the image volumes before DIR computation. Allowing the NaN voxels to match to any other voxels is also the primary limitation of the proposed method. The DVFs computed on the undefined voxels are not obtained by image intensity matching, but by propagating the DVF from the neighborhood valid voxels. While DVF propagating is a reasonable method for estimating DVF on the undefined voxels where image information is not available, the accuracy cannot be guaranteed.

In conclusion, we have proposed a method to improve the existing DIR algorithms to handle partially matched images. The core concept is to extend the image volumes with NaN voxels and to allow such NaN voxels to match to any voxels in the other image. The proposed method could be applied to many radiotherapy applications similar to the examples used in this paper. It is also possible to apply this method in similar ways to other algorithms that are based on different similarity metrics, including mutual information and cross correlation, even though we have to leave out such discussions from this short paper.