Examining the influence of digital phantom models in virtual imaging trials for tomographic breast imaging

Abstract.

Purpose

Digital phantoms are one of the key components of virtual imaging trials (VITs) that aim to assess and optimize new medical imaging systems and algorithms. However, these phantoms vary in their voxel resolution, appearance, and structural details. We investigate whether and how variations between digital phantoms influence system optimization with digital breast tomosynthesis (DBT) as a chosen modality.

Methods

We selected widely used and open-access digital breast phantoms created with different methods and generated an ensemble of DBT images to test acquisition strategies. Human observer performance was evaluated using localization receiver operating characteristic (LROC) studies for each phantom type. Noise power spectrum and gaze metrics were also employed to compare phantoms and generated images.

Results

Our LROC results show that the arc samplings for peak performance were $\sim 2.5\mathrm{deg}$ and 6 deg in Bakic and XCAT breast phantoms, respectively, for the 3-mm lesion detection task and indicate that system optimization outcomes from VITs can vary with phantom types and structural frequency components. In addition, a significant correlation ($p<0.01$) between gaze metrics and diagnostic performance suggests that gaze analysis can be used to understand and evaluate task difficulty in VITs.

Conclusion

Our results point to the critical need to evaluate realism in digital phantoms and ensure sufficient structural variations at spatial frequencies relevant to the intended task. Standardizing phantom generation and validation tools may help reduce discrepancies among independently conducted VITs for system or algorithmic optimizations.

1. Introduction

Rapid advances in medical imaging technologies and methods make it impractical to evaluate and refine their design for optimal clinical use through clinical trials alone. These trials can take extensive resources and long duration. Virtual imaging trials (VITs) are an alternative approach to assess the potential of an imaging system, software, or specific system/software combinations.^1–8 VITs are based on in silico methods, where digital phantoms replace patients, the software platform mimics the imaging process, and virtual interpreters represent the human readers.

As VITs are becoming more accurate, realistic digital phantom development has drawn much attention from the medical imaging community. Our group recently examined the contribution of anatomical and quantum noise in signal detection and performance for human observers in such imaging trials.^9,10 Using VITs, we have also examined in the past the development of novel visual search observer models to match humans,^11–13 system and algorithm optimization questions,^3,4,14–17 understanding image texture features as relevant to human observer performance,^18,19 and radiomics variability.²⁰ As in other groups’ work, all of these studies were conducted with one breast phantom type.

Here, we will evaluate what implications a VIT study outcome may have by changing the phantom type. We once again take digital breast tomosynthesis (DBT) as the modality of choice to examine this critical question and use two widely accepted types of breast phantoms—both considered anthropomorphic. For the last few years, many computational breast phantoms have been developed using different methods.²¹ Some commonly used breast phantoms include power-law-based phantoms,^22–25 anthropomorphic phantoms,^26–31 modified patient tomographic data,^32–36 and mastectomy specimens.^37–39 These phantoms vary in their resolution, model, structures, and details. Power spectrum analysis has been used as a method to assess phantom structures and realism.^{27–30,38,40,41} Cockmartin et al.⁴¹ observed in 2013 that none of the evaluated phantoms matched with the patient data in terms of power-spectrum parameters. Few studies extended the realism assessment with trained humans’ (physicists and radiologists) rating.^27,30,32,34 Badano et al.⁴² however argued that realism is subjective and simulating relevant properties for the task is sufficient. However, it is unclear which properties are relevant and what level of realism is sufficient for the task of tomographic imaging system/algorithmic optimizations.

Past work by various groups has used different phantom types for evaluating DBT system configurations.^22,37,43,44 Because of the differences in phantoms, configurations, and interpreters among the studies, there is little agreement on the optimal configuration. Specifically, the contribution of phantom differences in these inconsistencies is not well understood. In the preliminary analysis, Park et al.⁴⁵ and Zhao et al.⁴⁶ observed that phantom differences (uniform versus structured) influence tomographic system optimal configurations. However, differences resulting from various available structured phantoms (when they are each considered realistic by the research community) have never been explored.

We explore this critical question and examine whether and how phantom structure variability would influence the study outcome of a VIT such as for DBT acquisition parameters. In addition, we also investigated the influence of task difficulty due to changes in the phantom structures using analysis of observer gaze patterns and interpretation processes when viewing images generated with different phantoms in the VIT studies. We examine how variations in phantom design modulate visual search strategies and cognitive processes during image interpretation. Another key objective of this research is to assess whether specific gaze metrics can serve as reliable indicators of task difficulty in virtual trials. We present results from our study examining the influence of breast phantom types in VIT for DBT when all other aspects remain the same. To the best of our knowledge, ours is the first study to evaluate this critical aspect.

To accomplish this, we selected two widely used and open-access breast phantom types which were generated using different procedures. Further, human observers analyzed simulated in-plane DBT images of the selected phantoms for similar configurations. Our results show a comparison of predicted system optimizations between VITs using these two phantom types.

2. Materials and Methods

The study methods are summarized below. First, the selected phantoms are listed. Second, our simulation methodology to generate both abnormal and normal cases of DBT images is described. Next, the procedure to characterize phantom structures using a power spectrum is described. Finally, the experimental method to estimate human observer performance and gaze pattern is discussed.

2.1. Phantom Selection

We selected two types of digital breast phantoms for this study: three-dimensional anthropomorphic breast models generated by Bakic et al.²⁶ at the University of Pennsylvania (hereafter referred to as Bakic phantom) and XCAT breast phantoms generated at Duke University using compressed volumes of patient breast CT data³³ (hereafter referred to as XCAT breast phantoms). Bakic phantoms are based on mathematical models that define the breast structural variability. These phantoms can be manipulated easily through the model configurations to simulate changes in anatomy. XCAT breast phantoms are based on patient breast CT images. These phantoms may appear more realistic, but changing their anatomical variations or resolution is difficult.⁷ For our analysis, six digital phantoms of a 5-cm thickness for each type were selected. The phantoms of each type were categorized based on volumetric glandular fractions (VGFs) with 25% density and the other three with 50% density. Bakic and XCAT breast phantoms have a voxel resolution of ${(0.2\mathrm{mm})}^3$ and $(0.25\mathrm{mm}{)}^3$, respectively. The selected case numbers of XCAT phantoms of $\sim 25\%$ density are CTA1608, CTA0357, and CTA1326 and of $\sim 50\%$ density are CTB6013, CTA1284, and CTA1285.

Figure 1 shows sample slices of both types of phantoms on the top. The transitions in XCAT breast phantoms from 100% glandular to adipose tissue have intermediately dense (25%, 50%, and 75%) voxels, whereas these transitions are sharp (100% dense to adipose) in Bakic phantoms. The XCAT breast phantoms lack small structures such as Cooper’s ligaments. Erickson et al.³³ noted that the “lack of very fine-detail structures similar to Cooper’s ligaments can negatively impact the realism of digital phantoms.” References 34, 35, and 47 tried different methods to improve XCAT breast phantoms, but these phantoms were not released publicly.

Fig. 1.

Sample slice of 25% VGF Bakic (top left) and XCAT breast (top right) phantoms and the corresponding reconstructed DBT slices with a 3-mm spherical lesion on the bottom.

2.2. Image Generation

The DBT images used in this study were generated using a simulation platform based on a serial cascade model.^48,49 The simulation platform modeled generic DBT systems with both source and detector rotating geometry, which was detailed in our previous work^9,49,50 and is described briefly here. The X-ray spectrum modeled a 30-kVp molybdenum anode source with a 0.7-mm-thick Al filter and the X-ray fluence scaled to provide a 1.5-mGy mean glandular dose to a 5-cm-thick compressed breast. This total dose was evenly distributed at each projection angle, and the X-ray fluence per projection was determined based on the breast dosimetry data (Dgn coefficients) generated by a Monte Carlo simulator.⁵¹ Focal-spot blurring with a 0.1-mm focal-spot size was modeled using a Gaussian modulation transfer function. The detector was modeled as a 0.1-mm-thick CsI-based a:Si flat panel detector with a 0.1-mm pixel size. The scintillator blurring was modeled using an empirically measured pre-sampling MTF. Quantum noise was modeled by a Poisson distribution for each keV (at the absorption of X-rays within the scintillator), whereas additive electronic noise followed a Gaussian process with a standard deviation of 2200 electrons. Scatter was not modeled in this simulation.

The lesion targets were homogeneous spherical masses with a 3-mm diameter. Although this is smaller than the average lesion sizes detected by the current DBT systems, VITs are also aimed at evaluating future imaging system designs. This signal size of 3 mm was also chosen to achieve sufficient challenge without requiring artificially altering attenuation values of the signal and to capture the human observer performance trends more accurately for a range of breast densities. An abnormal case was generated by substituting the lesion into the randomly selected location in the glandular region prior to the projection imaging. The lesion contrast or local glandularity was not matched between phantom types, whereas selecting lesion locations as absolute performance was not relevant for this study. The simulated mass was assumed to have an energy-dependent attenuation coefficient for invasive ductal carcinoma as reported by Johns and Yaffe.⁵² Eight abnormal cases and one normal case were formed for each phantom. The projections were acquired using Siddon’s ray tracing method⁵³ to model X-ray transmission through the breast. Two different sets of projections were generated with different phantom types. To filter the random noise, an adaptive Wiener filter-based denoising algorithm was applied to each projection.^54,55 The denoising algorithm was shown to reduce the noise effectively in our prior work.⁹ Each data set was acquired over an angular span of 60 deg with a projection number $P\in \{\mathrm{3,7},\mathrm{11,15},\mathrm{19,21},\mathrm{25,31},\mathrm{35,41},45\}$ by keeping the total dose steady at 1.5 mGy for each DBT acquisition. Feldkamp filtered back-projection algorithm⁵⁶ was used for image reconstruction. A three-dimensional Butterworth filter with a cutoff of frequency 0.25 cycles/pixel was applied on reconstructed volumes. In-plane DBT images of 1-mm thickness were produced by applying a boxcar averaging. Eight lesion-present (abnormal) and eight lesion-absent (normal) images were created for each phantom for the human observer studies as described in our earlier work.⁴ A set of 96 images was produced from the six phantoms of each phantom type for a given projection number. Figure 1 shows sample 1-mm abnormal DBT slices of both types of phantoms on the bottom.

2.3. Noise Power Spectrum

Noise power spectrum (NPS) based analysis has been used in literature to quantify the similarity between phantom structures and clinical data.^28,41 Our NPS estimation methods are briefly summarized here and can be found in detail elsewhere.⁹ We selected multiple regions of interest (ROIs) of size $2.43\mathrm{cm}\times 2.43\mathrm{cm}$ from each lesion-absent DBT slice from the breast region. The mean of each ROI was subtracted from the corresponding ROI, and a Hann tapering window was applied to each ROI to reduce the edge artifacts. The 2D NPS was calculated by ensemble averaging the square of the magnitude of the discrete Fourier transform of each tapered ROI, and radial averaging of the 2D NPS was performed, resulting in a 1D NPS. A linear regression fit (from lower frequency ranges between 0.1 and $0.15{\mathrm{mm}}^{-1}$ to higher frequency ranges between 0.4 and $0.7{\mathrm{mm}}^{-1}$) was estimated to the natural logarithm of the 1D NPS that maximizes the coefficient of determination ($R^2$). The NPS parameter $\beta$ is estimated as the slope of linear fit.

2.4. Human Observer Study

Three non-radiologists took part in the localization ROC (LROC) experiments. An LROC study entails both detection and localization of a 3-mm spherical lesion. The task in our study is the search and localization/detection of spherical mass in simulated in-plane DBT images, thus justifying the non-radiologists as observers. Physicists and engineers who participated as observers had the same level of experience in reading simulated images. The human observer experimental method was the same as that described in our prior work^3,4,11,12 and summarized briefly here. In this study, in-plane DBT slices of the Bakic phantom set correspond to five acquisition protocols reconstructed with projection number $P$ of 7, 11, 19, 25, and 35 and the XCAT breast phantom set corresponds to six acquisition protocols reconstructed with projection number $P$ of 3, 7, 11, 19, 25 and 35 were evaluated. The 96 images per set (48 pairs of abnormal/normal images) were divided into 72 test images (36 pairs) and 24 training images (12 pairs). All sets included an initial training session followed by a test session. Each observer thus read 10 image sets. Observers were asked to select the lesion location, and a four-point ordinal scale was used to collect the confidence rating. Localizations were considered correct when the observer selected the location within a 2-mm radius of the lesion center (radius of spherical lesion $+0.5\mathrm{mm}$ additional radius for human selection error). The observers’ performance was quantified with the area under LROC curves (AUC). The estimate of AUC for a given observer and protocol was obtained with a Wilcoxon-based non-parametric ranking method. To assess the consistency of the AUC values between the observers, the intraclass correlation coefficient (ICC) was calculated using Python (v3.7.6) software.⁵⁷

2.5. Eye Gaze Collection

Numerous studies have demonstrated that gaze metrics correlate with diagnostic performance and can reveal human observers’ interpretation process.^58–61 To evaluate the differences in the interpretation process due to the change of phantom type, we collected gaze data using a screen-based eye-tracking system (Tobii pro X3-120 with EPU). An additional three non-radiologists participated in the eye-tracking study. The eye gaze data were collected for a single acquisition protocol of 35 projections for both types of phantoms. Both presentation of stimuli and eye-tracking were controlled by a Lenovo ThinkPad p52s laptop using an in-house built Python software. Calibration and validation procedures were performed before starting a session. The images were displayed on a standard Dell 23.8 in. LCD monitor with a resolution of $1920\times 1080$. Once the software collected and stored the gaze logs, post-processing was applied to estimate fixation locations, fixation durations, and saccade durations using an I-VT filter based on Tobii pro white paper.⁶² Figure 2 shows a sample gaze pattern of an observer for the task of searching and locating a 3-mm lesion. Each vertex indicates the fixation center with numbers indicating fixation order and lines indicating the saccade paths and lengths.

Fig. 2.

Example gaze pattern with fixations and saccades on DBT slices of 25% dense (top) and 50% dense (bottom) phantoms.

We computed five gaze metrics to characterize the gaze pattern. Namely, we found the total time spent on each image, the total number of fixations made on each image, the time taken to first fixate on the lesion region (first hit time), the number of fixations on the lesion region, and the accumulated lesion dwell time. These gaze metrics were estimated using MATLAB2018a (The MathWorks, Inc., Natick, United States) with in-house-built scripts. Spearman rank correlation was estimated between the average values of each gaze metric and the corresponding AUC values of the six observers.

3. Results

3.1. Power Spectrum Analysis

To characterize the structural variations of the selected phantom types, we estimated the NPS of the simulated DBT images. We selected the NPS of in-plane DBT images corresponding to the acquisition configurations of 35 projections over a 60-deg arc span for comparing the phantoms’ structures. Figure 3 shows the averaged NPS of both the phantom types in the log-log scale along with linear fits. These plots show that Bakic phantom backgrounds have higher spectral densities at higher frequencies (considered the anatomical region) than XCAT breast phantom backgrounds. We note that this high anatomic noise is not reflected in the calculated $\beta$ values. The estimated values of $\beta$ were 2.52 and 3.32 for Bakic and XCAT breast phantom backgrounds, respectively. This result is contradictory to the popular belief that lower $\beta$ values indicate lower anatomical noise and is in agreement with observations made in our prior work.⁹

Fig. 3.

Power spectra analysis of DBT slices of both types of backgrounds for a sample acquisition protocol of a 60-deg arc span, 35 projections suggest that lack of small and sharp structures in XCAT breast phantoms resulted in lower spectral density at higher frequencies than that of Bakic phantoms.

3.2. Human Observer LROC Results

In our studies, each human observer was tasked with search and localization of a 3-mm lesion within the displayed DBT image slice (similar to the ones shown in Fig. 1). Figure 4 shows sample regions of DBT slices with a lesion at the center of the region to illustrate the changes in the visibility of the signal for the varying number of projections in both types of phantom backgrounds. The signal is better visible in the Bakic phantom background for the acquisition configuration of 35 projections, whereas the signal is better visible in the XCAT breast background for the acquisitions of 11 to 35 projections. Figure 5 presents LROC plots for the three observers, for a sample acquisition of 35 projections over a 60-deg arc span, in both phantom backgrounds. The $y$-axis of the LROC represents the joint probability of correctly localizing a lesion in a case reported as positive. Therefore, the curve reaches up to the percentage of cases with correct lesion localization. The LROC AUC values above zero are considered better than guessing as the likelihood of guessing the lesion’s location is zero. Figure 6 shows the average performance of the three observers in both Bakic and XCAT breast phantom images. Error bar lengths indicate twice the standard error of the three observers’ AUC values. We observed a greater improvement in performance up to 11 projections in XCAT breast phantom backgrounds and a steady performance thereafter. This corresponds to an arc sampling of approximately 6 deg between adjacent projections for peak performance. In Bakic phantom backgrounds, observers’ performance improved up to 25 projections and required finer arc sampling of $\sim 2.5\mathrm{deg}$ to achieve peak performance. We also plotted the detection performance separately for 25% dense and 50% dense slices, where 25% dense indicates an easy level of task difficulty while 50% dense indicates a higher level of task difficulty. Regardless of phantom type, both levels of task difficulty show similar trends and suggest that optimization may not change with the task difficulty, which is in accordance with earlier observations by Zeng et al.⁴⁴ and Mackenzie et al.⁶³ We also noticed in Figs. 5 and 6 that observers had overall slightly lower performance in XCAT breast backgrounds than in Bakic backgrounds in particular for 25% dense images. The inter-observer agreement, quantified using the ICC, ranged from 0.92 to 0.95 with the average AUC values, suggesting a strong agreement between the observers.

Fig. 4.

Sample lesion presents regions of DBT slices of both phantoms acquired with different numbers of projections. The black arrows indicate the lesion position.

Fig. 5.

Human observer performance plotted as LROC curves for a sample acquisition of 35 projections over 60 deg in a 3-mm mass detection study in Bakic phantom (top) and XCAT breast phantom (bottom) backgrounds.

Fig. 6.

Human observer performance plotted as the area under LROC (AUC) against the number of projections in a 3-mm mass detection study in Bakic phantom (top) and XCAT breast phantom (bottom) backgrounds. The results suggest the optimal configurations do not change with the task difficulty but change with the background structure type.

3.3. Gaze Analysis

From the eye-tracking data, the total time spent as well as the total number of fixations made on images (both lesion present and lesion absent images) were estimated. For lesion present images, we also estimated the first hit time (on the lesion), lesion dwell time, and the number of fixations on the lesion. Each of these gaze metrics was averaged across all images for a given phantom type and breast density for each observer. Figure 7 shows the influence of phantom type on two of the gaze metrics plotted separately for lesion absent and lesion present images. Observers spent a longer time and made more fixations to diagnose lesion-absent images than lesion-present images, which is in line with previous findings.^64,65 The most striking result to emerge from this data is that observers spent a longer time and made more fixations on images with the Bakic phantom backgrounds than XCAT breast backgrounds. This difference is significant ($p\text{-}\text{value}<0.01$) in lesion-absent images only.

Fig. 7.

Average amount of time spent and the average number of fixations made on images (includes lesion absent and lesion present images) plotted for both phantom types. Observers spent a longer time and made more fixations to make decisions on images with Bakic phantom backgrounds in comparison to those with XCAT breast backgrounds.

Spearman rank correlation coefficient was computed between the average values of each gaze metric and the AUC values of both phantom types and densities. Table 1 shows that all of the estimated gaze metrics have a good correlation with diagnostic performance. Our observation of first hit time, lesion dwell time, and number of fixations on the lesion showed that the observers took longer to first fixate on the lesion, spent less cumulative time on the lesion, and had fewer fixations on the lesion as the task difficulty increased due to higher breast density. No significant difference was observed in these gaze metrics due to a change of phantom type in lesion-present images. First hit time showed a strong negative ($-0.85$) correlation with the AUC values as shown in Fig. 8. This result suggests that the quicker an observer fixates on the lesion the better the diagnostic performance which is in accordance with the observation made by Kundel et al.⁶⁰ The positive correlation of lesion fixations and dwell time with AUC suggests that observers fixate the lesion longer and multiple times to locate accurately. All five gaze metrics showed a significant correlation ($p\text{-}\text{value}<0.01$) with AUC values.

Table 1.

Spearman rank correlation between the AUC values of the six observers and the corresponding average values of the five gaze metrics. All five metrics show a good and significant correlation with the diagnostic performance.

Gaze metric	Correlation coefficient ($\rho$)	$p$-value
Total number of fixations	−0.52	0.0085
Total time	−0.6	0.0019
Lesion dwell time	0.67	0.0003
Lesion number of fixations	0.68	0.0003
First hit time	−0.85	1.2e−7

Fig. 8.

Average value of first hit time, lesion dwell time, and number of fixations on lesion were plotted against the AUC value of each observer (total of six) for both phantom types and two densities.

4. Discussion and Conclusion

This work evaluated the influence of variations between digital breast phantoms on DBT optimization and interpretation process for a small lesion localization task. Our results indicate that the phantoms should have adequate structures at spatial frequencies that are relevant to the signal size and the intended task for sufficient realism. We observed that the optimal number of projections for peak detection performance could change with the structural complexity of the phantoms. In addition, we observed that the number of projections required to achieve maximum performance is smaller for XCAT breast phantoms than for Bakic phantoms. Our power spectrum analysis revealed that the complex structures in Bakic phantoms contribute to high frequencies, whereas the high-frequency content of XCAT breast phantoms showed lower amplitudes. These high-frequency structures resulted in more aliasing artifacts in Bakic phantom backgrounds compared with XCAT breast phantom backgrounds under similar sparse sampling conditions. Hence, more projections were required to resolve these high-frequency structures in Bakic phantom backgrounds. The difference in observers’ performance between the two phantom backgrounds in particular for 25% dense images (see Figs. 5 and 6) cannot be attributed to phantom structures necessarily as the local densities in the region where lesions are inserted could also influence these LROC AUC. The magnitude of AUC values is less relevant in this particular study as only relative changes in AUC values with changing system parameters (such as a number of projections in this study) were used to deduce the final conclusions.

Our gaze analysis revealed intriguing insights into the influence of anatomical variations on visual search strategies. Observers spent significantly more time and made more fixations when inspecting images with Bakic phantom backgrounds compared with XCAT breast backgrounds. A possible explanation for this difference may be that the greater anatomical noise of Bakic phantom backgrounds than the XCAT breast backgrounds (see Fig. 3) made observers less confident and hence spent a longer time to make decisions. Notably, this effect of anatomical complexity was most pronounced in lesion-absent cases, where observers had to rely solely on the background characteristics to guide their search. By contrast, when a lesion was present, the visibility and salience of the target became the dominant factor driving gaze patterns, overriding the influence of background complexity. Observers exhibited longer first hit time, reduced cumulative lesion dwell times, and fewer lesion-directed fixations as the breast density, and hence task difficulty, increased. This suggests that lesion visibility or task difficulty plays a pivotal role in shaping visual attention allocation when a target is present, whereas anatomical complexity exerts a greater influence on search strategies in the absence of a defined target. All five gaze metrics showed a good correlation with diagnostic performance, which is in accordance with the observations made by Voisin et al.⁶¹ in mammography. This result indicates that gaze metrics and interpretation processes were influenced by the task difficulty. Hence, these gaze metrics can be used to understand the task difficulty in VIT imaging.

In this study, we evaluated the optimal number of projections in an arc span of 60 deg, which is wider than the arc ranges used in the clinical DBT systems. The arc samplings between the adjacent projections for peak performance were $\sim 2.5\mathrm{deg}$ and 6 deg in Bakic and XCAT breast phantoms respectively for the 3-mm lesion detection task. For other arc spans, we expect a similar amount of aliasing artifacts for the similar arc sampling. Thus, our conclusions should hold for other arc spans used in clinical DBT prototypes that have not been examined here in our VIT study.

The difference in the optimal arc sampling from VIT conducted with two widely used phantom types raises the need to standardize and unify the frequency contents and complexity of digital phantoms if results published from multiple groups need to be compared. The goal of our study was not to determine if one of the two phantoms is favorable or better than the other for use in VIT for breast. It is likely that the perceived realism and correlation with observer performance for VIT with actual clinical imaging trials would show results in favor of one of these phantoms based on the chosen task and the signal type.

One of the limitations of our study is that only three observers participated in estimating AUC trends. This number is smaller than the number of observers who participated in DBT optimization studies in the literature.^44,66–68 However, the observers had higher agreement with each other which is backed by our estimation of ICC value of greater than 0.92. In addition, we also estimated the correlation between the average AUC trends of Bakic and XCAT breast phantoms. The Spearman rank correlation of 0.2 suggests that the two trends are not similar.

Furthermore, the study was conducted only for the task of searching and locating a 3-mm spherical lesion as conducting human observer studies for different signal sizes and types is time-consuming and expensive. Prior studies suggest that smaller (high frequency) lesions dictate the optimal number of projections because their visibility is affected more by both aliasing artifacts (at a lower number of projections) and random noise (mainly at a higher number of projections).^23,43 Many studies also used 3-mm^22,23,37 and similar sizes^44,67–69 of lesions as a signal target in optimization studies. We believe that for larger signals the differences in optimal arc sampling due to differences in the selected phantom structures may be less evident than that of smaller signals. We chose a sphere lesion as a target because detecting a sphere lesion is a relatively easy task compared with a complex spicule lesion study, hence requiring minimal training. In addition, the performance of non-radiologists and radiologists was shown to be similar for relatively easy tasks⁷⁰ and multiple studies have used spherical lesions in VITs.^{22,23,37,43,44} On the other hand, for high-contrast smaller signals such as micro-calcification, random noise was shown to be the dominating factor in determining the optimal configuration rather than the background structures.⁴³ Future studies will include signals of different sizes and types.

Another limitation of our study is that only six phantoms (to generate a much larger number of independent cases in each study set) were selected for each phantom type, and this sample may not represent the entire population of each phantom type. Our goal was not to compare Bakic and XCAT phantoms but to validate how optimization estimation changes with phantoms that were generated differently. One may choose six phantoms in their studies in evaluating optimal configurations as some of the studies in literature chose around two to nine phantoms.^{11,15,66–68} If the selected new phantoms have similar NPS properties, we anticipate similar differences in the optimal number of projections. Instead, if the selected phantoms have different NPS properties and result in different optimal numbers of projections, this strengthens our argument that phantom structures influence the estimation of system optimal configurations. The selected phantoms (25% and 50% dense) are in the higher range of breast densities observed in the clinical data. The average AUC trends (see Fig. 6) did not change due to changes in breast density, suggesting that AUC trends may remain the same for other density phantoms. These results also point to the need to include more phantoms with variations in VIT studies.

In conclusion, our results indicate that both the structural complexity and the relevant spatial frequency magnitudes in digital phantom structures can influence the estimation of optimal system configurations. Our results highlight the importance of accurate modeling of phantoms to resemble the patient’s anatomy and the importance of assessment for their realism (for chosen tasks) before use in VITs. These results can be generalized to digital phantoms used for multiple imaging modalities. As a final note, our goal in this particular study was not to discuss the superiority of one phantom type over the other based on the results shown here. The key aspect is for the VIT and medical physics community to be mindful that optimization or other results shown using one “realistic” breast phantom may not always agree with results when using another “realistic” phantom unless additional standardization efforts are pursued.

Biographies

Amar Kavuri received his PhD in biomedical engineering from the University of Houston in 2022. He is currently working as a postdoctoral associate at Duke University. His current research interests include medical imaging, image processing, image analysis, image perception, visual attention, and virtual imaging trials. He is a member of SPIE – the international society for optics and photonics.

Mini Das is a Moores professor of physics, electrical engineering, and biomedical engineering at the University of Houston. Her research interests span broad areas of applied physics, optical physics, system development, quantitative imaging, inverse problems, detector physics, image science, and image perception to solve outstanding challenges in medicine and biology. She is a fellow of SPIE – the international society for optics and photonics, and is keen to develop educational and outreach activities to advance and promote interdisciplinary research.

Disclosures

The authors declare that the research was conducted without any commercial or financial relationships that could be construed as a potential conflict of interest.

Code and Data Availability

Both code and data could be made available upon written request to the authors.