A Method for Assessing Ground-Truth Accuracy of the 5DCT Technique

Abstract

Purpose

To develop a technique that assesses the accuracy of the breathing phase-specific volume image generation process by patient-specific breathing motion model using the original free-breathing CT scans as ground truths.

Methods

16 lung cancer patients underwent a previously published protocol in which 25 free-breathing fast helical CT scans were acquired with a simultaneous breathing surrogate. A patient-specific motion model was constructed based on the tissue displacements determined by a state-of-the-art deformable image registration. The first image was arbitrarily selected as the reference image. The motion model was used, along with the free-breathing phase information of the original 25 image datasets, to generate a set of deformation vector fields (DVF) that mapped the reference image to the 24 non-reference images. The high-pitch helically acquired original scans served as ground truths because they captured the instantaneous tissue positions during free breathing. Image similarity between the simulated and the original scans was assessed using deformable registration that evaluated the point-wise discordance throughout the lungs.

Results

Qualitative comparisons using image overlays showed excellent agreement between the simulated and the original images. Even large 2 cm diaphragm displacements were very well modeled, as was sliding motion across the lung-chest wall boundary. The mean error across the patient cohort was 1.15±0.37 mm, while the mean 95^th percentile error was 2.47±0.78 mm.

Conclusion

The proposed ground truth based technique provided voxel-by-voxel accuracy analysis that could identify organ or tumor-specific motion modeling errors for treatment planning. Despite a large variety of breathing patterns and lung deformations during the free-breathing scanning session, the 5DCT technique was able to accurately reproduce the original helical CT scans, suggesting its applicability to a wide range of patients.

Introduction

Respiratory motion presents a significant challenge for radiation therapy because of its irregularity and the consequential uncertainty of tumor and normal organ positions, leading to larger than necessary target volumes and potentially systematic errors in radiotherapy [1]. Four dimensional computer tomography (4DCT) [2-4] has become an indispensable tool that measures and characterizes breathing-induced motion, providing breathing phase-specific images for defining target volumes and motion mitigation strategies in radiotherapy. The incorporation of 4DCT into treatment planning provides the spatial and temporal information that explicitly accounts for respiratory motion [5-9].

Accurate motion estimation by 4DCT holds the potential of improving local control rates and normal tissue sparing in high-dose conformal lung radiotherapy [10-12]. However, due to breathing irregularities inherent in human quiet respiration, 4DCT is susceptible to sorting artifacts, which can cause target delineation errors [13-16]. Techniques have been proposed for retrospective sorting-based approaches with the aim of reducing motion artifacts in 4DCT images [15, 17-23]. Of these, respiratory motion model based techniques have shown promising improvement in mitigating sorting artifacts [18, 19, 24, 25], besides the additional attractive qualities of their predictive power. In particular, Thomas et al [25] demonstrated the generation of motion artifact-free 4DCT images at user-selected breathing phases based on a fast helical CT acquisition technique and a deformable lung motion model. The model was based on Low et al [26], employing two surrogates, breathing amplitude and rate, as independent variables. Such models are termed here as 5D. The model-generated 5DCT images, being sorting artifact-free, would not convey obvious indications as to whether the motion model prediction accurately reflected the phase-specific breathing motion. Ideally, the accuracy of 5DCT images would need to be evaluated prior to their clinical utilization to determine optimal motion management strategy.

In this paper, we present a technique that characterizes the accuracy of the 5DCT acquisition and motion modeling process described by Thomas et al [25] and Low et al [27]. We proposed to use the motion model to reconstruct the original free-breathing helical CT scans for a direct comparison in the image domain. The original CT scans were considered as ground truths in that they were acquired using quantitative imaging techniques that scanned each location within approximately 0.23s, so the localization uncertainty of any tissue landmark was minimal. Using the motion model and the breathing surrogate signals measured during the image acquisition, the original CT scan geometries were simulated by deforming the reference scan image with the model-derived DVF. The original and simulated scan geometries were compared throughout the lungs using a previously validated DIR technique. Both qualitative and quantitative analyses were presented for 16 cancer patients with either primary or metastatic lung tumors.

Methods

Image Acquisition and Patient Lung Model Construction

Fig. 1 describes the workflow for the acquisition of the free-breathing scans and the reconstruction of these scans using the 5D respiratory motion model. Sixteen patients were imaged under an IRB approved protocol using 64-slice CT scanners (Somatom Definition Flash and Biograph TruePoint PET·CT; Siemens Healthcare, Forchheim, Germany). One patient (that would have been the 17^th enrolled) was removed from our analysis because a rib was missing and the motion model was unable to manage the unusual deformation near the missing rib. To adequately sample the patient anatomy for lung model construction, 25 acquisitions of the free-breathing thoracic images were performed according to the fast helical CT protocol previously reported in [25, 27] with simultaneous recording of a breathing surrogate signal. A pneumatic accordion-shaped bellows tube was wrapped around the patient's abdomen. The tube was sealed and the air pressure in the tube monitored using a pressure transducer sampled at 100 Hz. The bellows air pressure was used as the breathing surrogate amplitude, which decreased and increased as the patient inhaled and exhaled, respectively. Using the shortest rotation period of 0.28s and pitch of 1.2, the lung anatomy was scanned through in 2.5s. Each location on the CT couch was imaged for 0.23s, short enough to be considered as essentially static representations of the patient's geometry at the time of scanning each slice. This allowed us to assign a breathing phase to each slice. The 25 helically acquired scans are termed here as the original scans.

Figure 1.

Flowchart of the simulation of the original scans and their comparison against the original scans.

Each scan spanned a large fraction of the breathing cycle and the scans together reflected the extent of motion the lung tissues underwent. The 5DCT approach incorporated the 25 lung tissue position samples into a patient-specific lung motion model [25-27]. Lung tissue displacements were determined using DIR, where the first original scan was arbitrarily chosen as the reference image. In this common geometry, the resulting 24 deformation vector fields (DVF) were related on a per-voxel basis to the bellows-measured breathing amplitude, ν, and its time derivative, the breathing rate, f, by solving the following linear equation for the lung motion model parameters [26]:

$$\overrightarrow{X}=\overrightarrow{X_0}+\overrightarrow{\alpha }\cdot \nu +\overrightarrow{\beta }\cdot f,$$ (1)

where X⃗ the voxel displacement, X⃗₀ the voxel position at a zero amplitude and rate, α⃗ described the motion due to breathing amplitude and β⃗ described the motion due to the rate-induced motion hysteresis effect.

Deformable Image Registration

The DIR algorithm employed for the study was deeds [28-30], based on a multi-level B-spline algorithm. Contrary to conventional thoracic registration algorithms, which relied on segmentation to model the shear motion at the lung boundary, a unique feature of deeds was the preservation of the sliding motion using image-adaptive minimum spanning trees [29, 31]. The incorporation of deeds-derived DVF into our motion modeling thus allowed for a straightforward tracking of the voxel movement on both sides of the pleural cavity. The image similarity evaluation was based on self-similarity context, which was a highly discriminative descriptor that contained structural information in the neighborhood of the voxel of interest [28, 30]. This metric was also shown to be robust in identifying anatomical features in the presence of local varying contrast and image noise during respiration, as was typically encountered in the images obtained by our low-dose, free-breathing acquisition protocol. In a recent EMPIRE 10 study [32], deeds achieved top-ranked accuracy in terms of landmark correspondence and had also been validated using the publicly available DIR-LAB data [29]. The use of such highly accurate DIR was essential for reducing the uncertainty in voxel position localization and contributing to the overall model parameter determination. In this work, deeds was used both in the initial registration of the free-breathing scans and the evaluation of matching between the simulated and original scans.

Reconstruction of Free-Breathing Scans by the Lung Motion Model

We proposed that the 5DCT image accuracy be assessed by comparing between the original images and the set of simulated images generated by deforming the reference image to the same non-reference breathing phases as recorded during image acquisition. The original images and their corresponding surrogate breathing curves represented a series of respiratory states that the patient underwent during quiet respiration. Using the motion model parameters and the breathing phase information consisting of breathing amplitude and rate, the breathing phase-specific DVFs were computed according to equation (1) at each voxel of dimension 1 mm³. The model DVFs pointed from the reference scan geometry to the other 24 non-reference original scans, as did the vectors obtained by the DIR process. To allow for the generation of the simulated images through resampling the reference image, the model DVFs were subsequently inverted using the fixed-point algorithm by Chen et al [33]. To compensate for the lung voxel intensity changes during breathing, the Hounsfield units (HU) were linearly interpolated using the sampled surrogate voltages and the resulting HU correction table was applied to the lung region. The quality of the DVF inversion was assessed by evaluating the fraction of the negative Jacobian values, which corresponded to singularities in the model-generated deformation [32, 33]. Lung model generated non-reference images were termed simulated images.

Statistical Analysis

For accuracy determination, the simulated images were deformably registered to their original counterparts and the resulting deformation vector magnitude of each lung voxel in the original images was used as misalignment error. For each patient, the model error was characterized using the grand mean, defined as

$$M_{\mathit{\text{GM}}}=\frac{{\sum }_{i=1}^N{M}_i}{N},$$ (2)

where M_i was the mean misalignment error of the i^th scan and N the total number of scans. To provide an estimate of the worst model prediction, M_G95 was similarly defined as the grand mean of the averaged 95^th percentile misalignment error of all scans. To address the potential bias towards a larger proportion of lung tissues of little motion, voxels were tracked throughout the 25 scans and ratios of the mean error versus the mean displacement, defined as misalignment error ratio, were formed for each lung voxel. Two-dimensional histograms that displayed the misalignment error and the misalignment error ratio distributions of the lung voxels with respect to their motion were generated to analyze the motion model simulation performance. The above analysis was performed using in-house MATLAB (Natwick, MA) and C++ programs. The approximate computation times were 15 minutes for model parameter generation, 10 minutes for each deformable registration, and 3 minutes for a helical scan simulation.

Results

The model errors in terms of the grand mean and the grand mean of the averaged 95^th percentile errors are summarized in Table 1. The averaged grand mean error across the 16 patients was 1.15±0.37 mm, with the averaged grand mean of 95^th percentile error estimated at 2.47±0.78 mm. Of the evaluated Jacobian values of the model generated DVFs, the mean negative Jacobian fraction was less than 0.05% across all patients, suggesting that the inverted model DVFs were well-defined for each transformation.

Figure 2 illustrates an example of the proposed model-guided image simulation technique. For an initial misalignment between the reference scan and target original scan geometries characterized by their respective breathing waveforms (Fig 2(a)), the relative deformation was found by registering the reference scan to the target original scan. In this expanding lung deformation marked by large diaphragm displacements (Fig 2(b)), large and complex deformation was observed towards the base of the lung, where the two lungs expand by different amounts (Fig 2(c)). Subsequently, the motion model-derived DVF specific to the target breathing phase was inverted and applied to the reference image for simulating the lung deformation (Fig 2(d)). Fig 2(e) shows that the model generated deformation pattern displays a remarkable resemblance to that generated by DIR (Fig 2(c)). The excellent alignment between the original scan and the simulated one can be appreciated in Fig 2(f). Lastly, Fig. 2(g) shows the residual differences found by forming the magnitude of the deformation vectors between the simulated and the original images, as contrasted from the initial large deformation.

Figure 2.

(a) The bellows breathing amplitude between the reference and target original scan image in patient 2. Surrogate signals of more negative voltage represents inhalation, and vice versa for exhalation breathing phases. The same convention applies to all other figures; (b) Initial misalignment between the image pair. The reference scan is displayed in green, the target scan in magenta. (c) Colormap showing the large and locally varying magnitude of inverse mapping required to create the target scan according to registration; (d) Inverted model deformation vectors overlaid on the reference image and (e) the corresponding deformation magnitude; (f) Overlay of the simulated scan onto the original scan, and (g) Residual misalignment in terms of registration deformation vector magnitude between the model simulated image and the original image.

Figure 3 demonstrates inter-patient examples where the 5D model accurately reproduced the tissue motion for large displacements. For 2.5s scans under free-breathing conditions, diaphragm displacements of large 2cm relative to the reference scan geometry were accurately modeled as can be visualized by the image overlay between the simulated and the original scan. In addition, detailed lung anatomical features, such as nodules, bifurcations, lobe fissures and lung boundary, were also very well aligned. Small misalignments showed regional variations in the lung motion and the model simulation accuracy often suffered at the inferior posterior lung boundary. While the quantitative analysis concentrated on tissues within the lungs, the bony structures in the thorax, including the spine and ribs, were also correctly aligned due to the employed image-wide DIR technique.

Figure 3.

For patient 1 undergoing large inhalation of about 2 cm diaphragm displacement, (a) the breathing traces between the reference and target original scan; (b,c,d) the initial misalignment, the excellent alignment after applying model deformation to the reference scan, and the residual error in the central coronal slice; (e,f,g) the same in right sagittal view. Similar figures for patient 5 (h-n) and patient 13 (o-u), in large expanding deformations.

An example of the shear motion near the lung boundary is shown in Figure 4. Fig 4(a, b) show the positions of a lung vessel in the reference and the target original images where the target original scan was selected to reflect tidal inhalation near the diaphragm. In the close-up region, the lung vessel moved inferiorly as the lung underwent expansion. The shear motion of the cross-marked vessel within the lung boundary was shown by the deformation vectors pointing in inferior direction (Fig 4(e)) while the vectors outside showed little motion magnitude, displaying a motion discontinuity at the chest-wall boundary. The alignment of the lung vessel in the image overlay indicated that the sliding motion on either side of the lung was well reproduced by the motion model.

Figure 4.

(a) In a patient 9 scan, where the lung is expanding relative to the reference geometry, the same lung vessel (in red box) moves in the inferior direction. A close-up of the boxed area and the position marks of the vessel (x) with respect to the neighboring rib position (+) in the reference (b) and in the target (c) and together in the overlay image (d). (e) The model deformation vectors, plotted in 4mm grid, are overlaid on the reference image. (f) The overlay image of the model simulated image on the target original image.

Figure 5(a) provides a histogram of the lung voxels binned by their misalignment errors and mean voxel displacement and 5(b) a histogram of the misalignment error ratios and mean voxel displacement for patient 10. Figure 5(c) shows the distribution of the misalignment errors in patient 10 over the total number of original scans. In 5(d), a quantitative description of the error statistics for the sixteen patients was given. In all patient scans examined, the grand mean over the lung region was below 2 mm, while the grand mean of averaged 95^th percentile misalignment was just slightly over 4 mm.

Figure 5.

Histogram in a base-10 logarithmic color scale displaying for patient 10 (a) the misalignment error magnitude versus the mean motion magnitude and (b) the lung voxel misalignment error ratios versus their mean motion; (c) boxplot showing mean (x), median (red line), interquartile range (blue box), and 5^th and 95^th percentile misalignment error (whiskers) for patient 10; (d) distribution of the grand mean (red) and the grand mean of averaged 95^th percentile (blue) error of all scans per patient, with the error bar showing the standard deviation.

Discussion

A technique for characterizing the accuracy of CT images generated by the 5D model has been proposed and was tested in sixteen patients. As illustrated in Figure 5(a) and 5(b), histograms show that the vast majority of the lung voxels exhibit mismatching error on the order of one millimeter. This suggested that 5DCT technique accurately reproduced the dynamic lung deformation associated with the variety of the sampled patient free-breathing states during the 2.5-minute scanning session. In particular, our results showed robustness in simulating the large deformation of deep breaths using an arbitrary reference scan.

Quality assurance of 4DCT scanning is important for minimizing systematic errors in radiotherapy of mobile tumors. Hurkmans et al evaluated 4DCT imaging accuracy using a programmable phantom that moves with periodic breathing cycle and a known motion range [34]. The present work assesses the 5DCT workflow accuracy by comparing to the high-pitch helical “snapshot” free-breathing scans as ground truth. Such accuracy assessment would be valuable in determining the appropriate motion management strategy for treatment. To the best of our knowledge, the proposed 5DCT protocol is the first respiratory-gated CT technique that explicitly estimates the accuracy of the generated images using patient ground truth data.

The validity of the original scans as ground truth was limited by the motion-induced artifacts due to high tissue motion velocities. Such artifacts appeared in some images, but consisted largely of small structure blurring (<3 mm) and ghosting of the diaphragm boundary. The position of the blurred structures, however, would generally reflect the average position of that structure during the 0.23s that the location had been scanned. We examined the CT scans and determined that the motion-induced artifacts led to relatively minor errors in the average positions of the lung tissues.

One of the hallmarks of not using time as an explicit variable was that the complexity of human breathing motion could be isolated into time-dependent and time-independent variables, where the time-dependent variable was physically measured and the time-independent variables described the spatial characteristics of the motion. This allowed breathing irregularity to be explicitly managed by the 5D motion model. The ability of the 5D motion model to reproduce lung tissue hysteresis was previously demonstrated in Thomas et al and Low et al. [25, 26]. Building on this unique characteristic, the lung modeling realism was here further enhanced by accounting for sliding motion along the lung periphery through the use of a lung-segmentation-free DIR technique. This particular modeling feature may be useful for analyzing pathologies where lung tumor is attached to the pleura and holds the potential for increased precision of highly conformal radiotherapy.

While the 5D formulation accurately described lung tissue motion in the examined cohort, the accuracy of the simulated tissue alignment was specific to the patient, the breathing phase, and also showed local variations throughout the lungs. For example, the larger misalignment errors observed in patient 11 could be attributed to the fact that most of the image data were captured during exhalation phases. The model fitting was therefore biased towards exhalation, so the model quality at other phases suffered. A prospective CT acquisition technique that assured adequate breathing phase sampling for each location in the lung would address this issue. The measurement noise in the surrogate signal was also found to be correlated with the increased motion model error, particularly in patients 6, 7, 11, and 13. The high frequency components, presumably electronic noise, in the recorded physiological signal introduce errors in the model parameter model fitting and degraded the model accuracy. Careful placement of the abdominal belt to maximize the voltage difference between inhalation and exhalation and improving the pressure transducer readout circuit design could improve the surrogate measurement and subsequent motion model accuracy.

The model accuracy also suffered towards the posterior lung boundary, likely because the tissue motion tended to be greatest in this region with subsequent motion-induced blurring in the images acquired during inspiration or expiration. The blurred images would be more poorly registered relative to images acquired during peak inhalation or exhalation. The image acquisition-related issues of posterior edge misalignment and artifact errors could in the future be minimized by scanning at higher pitches, faster rotation speeds, using more sophisticated image reconstruction, or the development of motion sensitive deformable registration algorithms.

Conclusion

We have demonstrated a method to evaluate the 5DCT accuracy by comparing the model-simulated images against the ground-truth original scanned images. The demonstrated similarity agreement for the examined cohort during free-breathing imaging suggested the applicability of the proposed 5DCT technique for a wide range of patients. The proposed ground truth based analysis is unique in CT-based breathing motion modeling for radiation therapy and will provide uncertainty estimations in the model-based 4DCT breathing motion estimate of tumors and normal organs.

Summary.

A quality assurance method was developed for a model-based 4DCT technique that employs fast helical free-breathing scans for constructing the breathing motion model. Using deformable image registration, the accuracy was measured by comparing the motion model reconstructed to the ground-truth original scans. The original scans were treated as ground truths because of the employed fast helical CT acquisition technique. Results were presented for 16 patients and the average error was just over 1 millimeter.