Task-based validation and application of a scanner-specific CT simulator using an anthropomorphic phantom

Abstract

Background:

Quantitative analysis of CT images traditionally utilizes real patient data which can pose challenges with replicability, efficiency, and radiation exposure. Instead, Virtual Imaging Trials (VITs) can overcome these hurdles through computer simulations of models of patients and imaging systems. DukeSim is a scanner-specific CT imaging simulator that has previously been validated with simple cylindrical phantoms, but not with anthropomorphic conditions and clinically relevant measurements.

Purpose:

To validate a scanner-specific CT simulator (DukeSim) for the assessment of lung imaging biomarkers under clinically relevant conditions across multiple scanners using an anthropomorphic chest phantom, and to demonstrate the utility of virtual trials by studying the effects or radiation dose and reconstruction kernels on the lung imaging quantifications.

Methods:

An anthropomorphic chest phantom with customized tube inserts was imaged with two commercial scanners (Siemens Force and Siemens Flash) at 28 dose and reconstruction conditions. A computational version of the chest phantom was used with a scanner-specific CT simulator (DukeSim) to simulate virtual images corresponding to the settings of the real acquisitions. Lung imaging biomarkers were computed from both real and simulated CT images and quantitatively compared across all imaging conditions. The Virtual Imaging Trial (VIT) framework was further utilized to investigate the effects of radiation dose (20 – 300 mAs) and reconstruction settings (Qr32f, Qr40f, and Qr69f reconstruction kernels using ADMIRE strength 3) on the accuracy of lung imaging biomarkers, compared against the ground truth values modeled in the computational chest phantom.

Results:

The simulated CT images matched closely the real images for both scanners and all imaging conditions qualitatively and quantitatively, with the average biomarker percent error of 3.51% (range 0.002%–18.91%). The VIT study showed that sharper reconstruction kernels had lower accuracy with errors in mean lung HU of 84 to 94 HU, lung volume of 797 to 3785 cm³, and lung mass of −800 to 1751 g. Lower tube currents had the lower accuracy with errors in mean lung HU of 6 to 84 HU, lung volume of 66 to 3785 cm³, and lung mass of 170 to 1751 g. Other imaging biomarkers were consistent under the studied reconstruction settings and tube currents.

Conclusion:

We comprehensively evaluated the realism of DukeSim in an anthropomorphic setup across a diverse range of imaging conditions. This study paves the way towards utilizing VITs more reliably for conducting medical imaging experiments that are not practical using actual patient images.

Introduction

Computed Tomography (CT) provides a non-invasive means of assessing diseases. In the United States alone, about 70 million CT scans are performed each year, and this number continues to rise over time¹. The assessment and optimization of CT images for quantitative analysis traditionally involves working with real patient data which can be challenging, expensive, and ground-truth limited². With a growing number of CT scans performed every year, these limitations will become more significant.

Alternatively, these assessments can be performed using realistic virtual simulations of the CT imaging process. Imaging simulations can be conducted using Virtual Imaging Trials (VITs)³ which involve using computational models of the patient and imaging systems as opposed to real patients imaged with clinical scanners. A general schematic of a VIT is shown in Figure 1. VITs can overcome some of the challenges associated with conventional imaging trials. Since they are performed via computer simulations, they are easily replicable and cost efficient⁴. Additionally, VITs include the ground truth as the exact structural anatomy and material of the patient model is known⁵. As such, VITs enable multiple image acquisitions of virtual patients without potential issues with patient data privacy and radiation exposure⁴.

Figure 1.

A depiction of a Virtual Imaging Trial platform. A computer-generated human phantom is shown depicting a virtual patient (left) which is virtually scanned (middle) and a coronal view corresponding to the simulation result (right) using window level of −600 HU and width of 1500 HU.

For virtual trials in CT, a major component is realistic simulation of the CT image acquisition process. DukeSim is a rapid and scanner-specific imaging simulator that produces CT sinograms of any computational phantom. DukeSim works with voxelized phantoms and a material file including the density and elemental compositions of materials in the phantoms. DukeSim accounts for various scanner specific features including poly-chromatic x-ray source, bowtie filter, focal spot wobbling, tube current modulation, and poly-energetic detector response. To generate sinograms, DukeSim has a ray-tracing and a Monte Carlo module to account for both primary and scatter signals^6–8. In previous work, DukeSim has been validated against experimental data acquired from various scanners and imaging conditions with cylindrical phantoms of various diameters using measurements of image contrast, noise texture, and spatial resolution^{6, 9}. However, CT images of cylindrical-shaped phantoms do not reflect in a way similar to the varying structures of the human chest, thereby indicating a need for further investigations regarding the accuracy and reliability of DukeSim in terms of physiological measurements for more realistic and anthropomorphic phantoms. For lung imaging applications in particular, a CT simulation needs to be validated against a corresponding anthropomorphic chest phantom including ribs, spine, mediastinum, and lungs, and in terms of the quantifications that are relevant to the pulmonary imaging applications^{10, 11}.

The purpose of this study was to validate the accuracy of DukeSim for lung imaging trials by simulating images of an anthropomorphic chest phantom and comparing them to real images acquired with clinical scanners in terms of lung imaging biomarkers. We further demonstrated the utility of the virtual framework by analyzing the effects of varying imaging tube current and iterative reconstruction kernels on the accuracy of biomarker measurements.

Methods

Anthropomorphic chest phantom

We utilized a version of the commercially available physical phantom (Kyoto Kagaku Chest Phantom N1 “LUNGMAN” (PH-1)), customized by the manufacturer for investigators at the University of Iowa^12–14. In this chest phantom, a urethane resin surrounded the model of the right and left lungs and thoracic bones. As part of the customization, three long tubes capable of containing sample inserts were placed vertically into the lung models, with one 59 mm diameter tube passing through the right lung and two 28 mm diameter tubes passing through the left lung. Three different sets of experimental sample inserts were placed into the vertical tubes, with each set being labeled either as Configuration 1, Configuration 2, or Configuration 3 (Figure 2). The lung samples were cored from porcine lung fixed at an inflation of 15 cmH2O (low) or 20cmH2O (mid) following methods discussed by Vasilescu et al.¹⁵. Synthesized lung in the form of standardized foams acquired from the National Institute of Standards (NIST) were also included. A schematic of the chest phantom and its dimensions, along with the identities of the three configurations of tube inserts, are shown in Figure 2. Inserts, other than the lung samples were produced by the Phantom Laboratory (Salem, NY).

Figure 2.

Schematics for the physical Kyoto-Kagaku chest phantom, and Configurations 1, 2, and 3 of the three tube inserts. Note that the region marked as “MTF Bead” in Configuration 3 is a tungsten carbide bead surrounded by low contrast. PMP is polymethylpentene polymer.

Real CT Data

The Kyoto-Kagaku anthropomorphic chest phantom was imaged with two clinical CT scanners (Siemens SOMATOM Definition Flash (n=15 images) and Siemens SOMATOM Force (n=13 images)) at 120 kV, reconstructed with iterative and weighted filtered back projections (wFBP) algorithms¹⁶. Table 1 lists an overview of the CT image sets and their associated imaging parameters.

Table 1.

CT scanners (rows) used to image the physical Kyoto-Kagaku Chest Phantom and imaging conditions (columns) utilized.

	Number of Images	CTDIvol Range (mGy)	kV	Reconstruction Algorithm	Reconstruction Kernel	Pixel Spacing (mm2)	Slice Thickness (mm)
Siemens SOMATOM Definition Flash	15	3.71– 7.42	120	SAFIRE 3, SAFIRE 5, wFBP	Q30f, B35f	0.51 × 0.51	0.75
Siemens SOMATOM Force	13	1.15–7.35	120	ADMIRE 3, ADMIRE 5, wFBP	Qr40d, Bf40d	0.51 × 0.51 0.98 × 0.98	0.75

Virtual Imaging Trial

A computational model for the Kyoto-Kagaku chest phantom was created in a voxelized format. To model the anatomy of the phantom, we used the CT images of the physical Kyoto-Kagaku phantom (acquired with FORCE at 7.42 mGy and reconstructed with iterative reconstruction, SAFIRE 3). The images were then manually segmented using an open-source software (Seg3D)¹⁷ to delineate masks representing various regions in the phantom (e.g., lungs, thoracic bones, inserts, etc.). The final mask was up-sampled to a higher resolution phantom file with a voxel size of 0.25 mm. A mid-level region of this mask is shown in Figure 3.

Figure 3.

Computational Chest Phantom seen from an axial (left), sagittal (middle), and coronal (right) view. Each shade represents a different one of the 21 regions, though not all 21 regions are visible from each view.

Linear attenuation coefficients were calculated for each material in the phantom using an x-ray library database¹⁸, given the corresponding elemental compositions and densities that were obtained from the phantom manufacturer. For the low contrast material, which had an unknown elemental composition, a similar liver tissue was utilized. The density of the synthetic lungs (a urethane foam made from a urethane-based resin) was found by tuning the HU values in the simulated images to find a reasonable estimate since an accurate reference value was unknown. This was corroborated by computing the ratio of the lung mass and volume across multiple real CT images of the Kyoto phantom.

We simulated the acquisitions replicating the real imaging conditions used in the real acquisitions (Table 1). The modeled computational phantom, corresponding material information, and scanner specifications were input to DukeSim to generate CT projection data. The simulated projection data were reconstructed using a vendor-specific reconstruction software (ReconCT, version 15.0.53098.0) using the same reconstruction algorithm and imaging conditions of the real data (Table 1).

Validations and analysis

Comparisons between the real and simulated images were made by computing various lung imaging biomarkers: lung density (average HU values in the lungs), LAA −950 (percentage of lung voxels with an HU ≤ −950, which relates to airflow obstruction), LAA −856 (percentage of lung voxels with HU ≤ −856, a measure of gas trapping), Perc 15 (HU value at the 15th percentile of all lung voxels, used for measuring emphysema progression), lung volume, and lung mass (defined as voxel size multiplied by $\sum _{i=1}^N\left(\frac{{HU}_i+1024}{1024}\right)$, where N is the number of total lung voxels)¹⁹. The mean and standard deviation of the HU values were computed in each sample region of the tubes for Configurations 1, 2, and 3. In addition, slice sensitivity profiles were calculated in the images around the highly dense (~15 g/cm³) tungsten carbide bead surrounded by low contrast (Configuration 3). These profiles depicted normalized HU value as a function of vertical slice position^{8, 20}.

Noise characteristics (noise magnitude and noise power spectrum) were also compared between real and simulated data. Noise magnitude was computed following the work of Christianson et al²¹, and noise power spectrum (NPS) was computed following previous literature^{6, 22}. The average frequency, peak frequency, and root mean square error (RMSE) of the NPS curves were extracted to compare the simulated and real results.

All the measurements from the simulated images were compared against the real data in terms of the percent relative difference between the real and simulated biomarker measurements, considering the real data measurements as the reference.

A VIT study

We conducted a VIT study to provide an example of how our developed and validated virtual framework can be used for imaging studies. In this VIT study, we focused on the effects of radiation dose and image reconstruction techniques on the accuracy of lung imaging biomarkers and HU values.

The same chest phantom was imaged using DukeSim based on the geometry and settings of the Siemens SOMATOM Force CT scanner at a tube voltage of 120 kV and pitch of 1.0. The images were acquired at three different tube current levels of 20, 100, and 300 mAs, which correspond to CTDI_vol values of 1.315, 6.575, and 19.725 mGy respectively. All sinogram data were reconstructed using the same software used in the validation study (ReconCT, version 15.0.53098.0). The images were reconstructed with an iterative algorithm (ADMIRE with the strength 3) at three different kernels (Qr32f, Qr40f, and Qr69f) with a pixel size of 0.51 × 0.51 mm and a slice thickness of 0.75 mm. Imaging biomarkers (similar to measurements in the validation section) were measured across all the images.

For each imaging biomarker, the measurements were compared against the ground truth values that were defined prior to the imaging. The ground truth HU values were computed from the computational phantom defined as the linear attenuation coefficients of the materials at the effective energy (67 keV) that the images were acquired.

Results

Qualitatively, the simulated images had high visual realism. Figure 4 shows representative reconstructions of the simulated images alongside the corresponding real CT images.

Figure 4.

Axial slice of real and simulated images shown at window level of −437.5 HU and width of 1173 HU. The top left image is the real CT Image from Siemens SOMATOM Definition Flash, and the top right is the corresponding simulated image. The bottom left is the real CT Image from Siemens SOMATOM Force, and the bottom right is the corresponding simulated image.

Figure 5 shows the measurements for both real and simulated images across the two scanner models. Each bar corresponds to particular biomarkers averaged across all images acquired from that scanner with the error bars corresponding to their standard deviation. The simulated measurements were close to real data with an average relative difference of 3.51% (range 0.002% to18.91%). The Force scanner generally had a higher magnitude for HU biomarkers across both real and simulated data compared to Flash. The similar trend in the real and simulated data demonstrates that DukeSim is sensitive to the differences between the scanner models.

Figure 5.

Comparison of the Biomarker in Siemens SOMATOM Definition Flash and Siemens SOMATOM Force split by units of the biomarkers: HU (top), Percentage of lung voxels under LAA thresholds (middle), and mass & volume (bottom). RT = Right tube, LPT = Left Posterior Tube, LAT = Left Anterior Tube

Note that the “Water” insert results are not included since in the actual phantom the insert had extra materials that were unknown to us. In addition, the Lung Samples “A” through “F” were not evaluated since the inserts had textured materials that were unknown to us.

Figure 6 shows the slice sensitivity profiles for both scanners measured from real and simulated data for one sample image. Each plot was produced by finding the slice sensitivity profile (maximum HU voxel across each slice in the z-direction) for each image in each scanner in the real and simulated case and then normalizing from zero to one.

Figure 6.

Slice Sensitivity profiles for each scanner comparing real and simulated data for one sample image

Slice sensitivity results showed a strong overlap of the peaks in the curves between real and simulated data (Figure 6). Table 2 summarizes the full width at half maximum (FWHM) results, with the errors of 0.4 and 0.15 mm for the Flash and Force scanners, respectively.

Table 2.

Summary of the FWHM characteristics (in mm) of the Slice Sensitivity Profiles. Each entry is the mean across all relevant images using standard deviation as the uncertainty.

	Real CT Data	Simulated Data
Siemens SOMATOM Definition Flash	1.46 ± 0.05 mm	1.85 ± 0.16 mm
Siemens SOMATOM Force	1.46 ± 0.07 mm	1.61± 0.04 mm

Figure 7 shows the noise magnitude measurements for both real and simulated images. Each bar corresponds to the mean noise magnitude measured across all relevant images, with error bars corresponding to standard deviations. The simulations had images with noise magnitudes close to the real data with an average relative difference of 1.88% and 2.17% in for Force and Flash acquisitions, respectively. Figure 8 shows an example of real and simulated noise power spectra set of imaging and reconstruction conditions. The NPS plots are normalized to area of one to only reflect the shape of the curve (the magnitude is compared in Figure 7). Across all images, the simulations provided noise power spectra close to the real measurements with a mean difference of 0.03 ± 0.02 mm⁻¹ and 0.03 ± 0.01 mm⁻¹ in terms average frequency, 0.04 ± 0.03 mm⁻¹ and 0.04 ± 0.02 mm⁻¹ in terms of peak frequency for the Force and Flash acquisitions, respectively. The normalized NPS curves (normalized to area of one) had RMSE of 0.27 ± 0.17 mm and 0.26 ± 0.06 mm, for the Force and Flash acquisitions, respectively.

Figure 7.

Noise magnitude (HU) across all images for Real vs. Simulated images for both scanner

Figure 8.

Normalized noise power spectrum (NPS) of the real and simulated images for one image condition of each scanner (5.80 mGy CTDI_vol, Q30f kernel, 0.75 mm slice thickness, and 0.51 × 0.51 mm pixel spacing for Flash and 7.35 mGy CTDI_vol, Qr40d kernel, 0.75 mm slice thickness, and 0.51 × 0.51 mm pixel spacing for Force). For these two examples, RMSE of the normalized NPS were 0.18 mm (Flash) and 0.07 mm (Force).

VIT Results:

Figure 9 shows reconstructed images of the combinations of the dose levels (20, 100, and 300 mAs) and reconstruction kernels (Qr32f, Qr40f, and Qr69f) that were virtually aqcuired in this study. Qualitatively, the images demonstrate increased noise magnitude in the lower dose levels and sharper kernels, as expected. In addition, higher spatial resolution renditions were observed in the sharper kernels.

Figure 9.

Mid-level axial views of all combinations of tube current and reconstruction kernels used in this phase of the study, shown at Window level −600 and Width 1500. The sharper kernels and lower dose images yield to noisier renditions.

Figure 10 shows plots of the quantitative errors for each imaging biomarker with respect to the corresponding ground truth value across the dose levels and reconstruction kernels. In general, the absolute errors changed for the lower dose levels and sharper kernels on average by 11.6 HU (range 3.1 to 25.5 HU) and 15.3 HU (range 0.3 to 100.0 HU), respectively. Under the conditions that were studied, the results demonstrated that reconstruction kernels had higher impact on the biomarker accuracy compared to the tube current (Figure 10). For the low-attenuating objects, the measured values overestimated the ground truth values, whereas the high-attenuating objects underestimated the ground truth values; both are likely due to the effects of scattering and beam hardening. The errors for the air tubes in the right and left lungs were also different (e.g., in the case of 300 mAs and Qr32f kernel, errors were 5.5 and 45.3 HU, respectively), demonstrating that these errors depend on the location and shape of the imaged object.

Figure 10.

HU Biomarker errors (difference between the measurements and the ground truth values) for each region as a function of tube current using reconstruction kernel Qr32f (top), Qr40f (middle), and Qr69f (bottom).

Figure 11 shows a heat map visualizing the effects of tube current and reconstruction kernel on the corresponding accuracy of the biomarker measurements for “lung volume” and “lung mass” biomarkers. Similar to the density-based biomarkers (Figure 10), higher tube currents and softer kernels had lower errors in measuring these biomarkers. In particular, sharper kernels yielded errors as large as 797 to 3785 cm³ and lung mass errors of 800 to 1751 g. Lower tube currents yielded errors in lung volume as large as 66 to 3785 cm³ and lung mass errors of 170 to 1751 g. For the sharpest kernel (Qr69f), the images were so noisy (noise magnitude of approximately 176.17 HU) that the segmentation algorithm failed to segment some of the lung regions, resulting in large errors in the “lung volume” and “lung mass” biomarkers.

Figure 11.

Differences between the measured and the ground truth values of the “lung mass” and “lung volume” biomarkers, at different tube current and reconstruction kernels.

Discussion

This study validated the accuracy and reliability of virtual simulations in terms of producing scanner-specific CT images of anthropomorphic models. The measured imaging biomarkers were highly close to the real measurements (3.51% on average) across the imaging conditions and the scanner models utilized in this study. The noise characteristics between real and simulated data were close to each other with ~2% error in the noise magnitude. Further, the simulated images had slightly higher NPS average frequency (0.03 mm⁻¹) and peak frequency (0.04 mm⁻¹). Our simulations were also sensitive to various imaging settings, parameters, and biomarkers which enables efficient and comprehensive virtual imaging studies for various imaging applications.

While the differences between the real and simulated measurements were generally small (3.51% on average), there were larger differences for certain conditions. For the density-based biomarkers, the accurate knowledge of the materials’ density and elemental composition is crucial. While we knew those values for most materials, the values for the manufactured versions might be slightly different, resulting in differences in the final HU values. Further, there were various simulation modules that affected simulation accuracy (e.g., detector model, approximation for the primary and scatter signal, beam hardening correction). While our simulation modules are state-of-the-art, they have some inherent approximations. It should be noted that there is also an inherent variability across different scanners of the same model. For instance, Sieren et al.²³ have observed HU differences of up to 20 HU in water measured from same scanner models at different sites. The simulation errors in this study were within that range with errors of 12.7 ± 11.1 HU for Siemens Flash (Range: 0.02 to 29.4 HU) and 14.7 ± 10.4 HU for Siemens Force (Range: 1.3 to 34.3 HU). Future studies may quantify whether our simulation errors are within the range of scanner-to-scanner variabilities for various inserts and materials. Further, the magnitude of simulation errors in the lung materials does not affect the current diagnostic criteria for COPD that is recommended by COPDGene²⁴ (i.e., LAA-950 > 5% in inspiratory images and LAA-856 > 15% in expiratory images).

In terms of the slice sensitivity, we observed FWHM errors of 0.4 and 0.15 mm. We approximated the tungsten beam to have a size of one voxel (0.25 mm) which may affect the accuracy of slice sensitivity measurements. The 0.25 mm voxel size has shown to be an effective resolution for density-based imaging applications. However, for high-resolution tasks (e.g., airway measurements), the simulations would be more accurate with smaller voxel sizes which make the overall simulations more computationally expensive.

This work extended validation of virtual imaging trials under anthropomorphic conditions and clinical tasks. While this study mostly focused on density-based evaluations, it would be insightful to further evaluate the utility of virtual CT platforms for radiomics studies. Such validations require knowledge of the exact heterogeneities in the textured areas of the physical phantom which was not known to us in this study.

The VIT study suggested that higher tube current and smoother kernels yield more accurate density-based quantifications. This is likely due to the reduced noise that is achieved with these conditions. The results are in line with previous research studies, as others have found that after a certain point, use of a sharper kernel greatly increases CT image noise for small changes in sharpness²⁵. In particular, the Qr69 kernel dramatically shifted the biomarker measurements from the ground truth values, suggesting that this condition could result into unreliable lung imaging quantifications. Such VIT studies could be further tailored for task-specific optimizations where the goal is to identify imaging parameters that provide acceptable image quality (e.g., lowest absolute error for relevant imaging biomarkers) while minimizing the radiation dose.

We also observed that in the case of the sharpest kernel (Qr69f), the Otsu based segmentation algorithm failed to properly segment the whole lung, resulting in a large difference between the measured biomarkers and their corresponding ground truth. These results highlight that the biomarker accuracy depends on both imaging conditions and image processing algorithms (in this case segmentation method). This emphasizes the significance and need to explore optimum image processing algorithms towards accurate and robust CT quantifications.

Conclusion

VIT technology provides scientists and clinicians a more efficient, replicable, and robust method of evaluating clinical disease without the risk of unnecessary radiation exposures commonly seen with traditional imaging methods. We comprehensively evaluated the realism of a CT simulation platform (DukeSim) across a diverse range of imaging conditions. This study paves the way towards utilizing VITs more reliably for conducting medical imaging experiments which may not be practical with actual patients.