Impact of breast structure on lesion detection in breast tomosynthesis, a simulation study

Abstract.

This study aims to characterize the effect of background tissue density and heterogeneity on the detection of irregular masses in breast tomosynthesis, while demonstrating the capability of the sophisticated tools that can be used in the design, implementation, and performance analysis of virtual clinical trials (VCTs). Twenty breast phantoms from the extended cardiac-torso (XCAT) family, generated based on dedicated breast computed tomography of human subjects, were used to extract a total of 2173 volumes of interest (VOIs) from simulated tomosynthesis images. Five different lesions, modeled after human subject tomosynthesis images, were embedded in the breasts and combined with the lesion absent condition yielded a total of $6\times 2173$ VOIs. Effects of background tissue density and heterogeneity on the detection of the lesions were studied by implementing a composite hypothesis signal detection paradigm with location known exactly, lesion known exactly or statistically, and background known statistically. Using the area under the receiver operating characteristic curve, detection performance deteriorated as density was increased, yielding findings consistent with clinical studies. A human observer study was performed on a subset of the simulated tomosynthesis images, confirming the detection performance trends with respect to density and serving as a validation of the implemented detector. Performance of the implemented detector varied substantially across the 20 breasts. Furthermore, background tissue density and heterogeneity affected the log-likelihood ratio test statistic differently under lesion absent and lesion present conditions. Therefore, considering background tissue variability in tissue models can change the outcomes of a VCT and is hence of crucial importance. The XCAT breast phantoms have the potential to address this concern by offering realistic modeling of background tissue variability based on a wide range of human subjects, comprising various breast shapes, sizes, and densities.

1. Introduction

Advanced imaging techniques and systems are constantly under development and examination to improve the screening and diagnosis of breast cancer. The advent of digital X-ray detectors has facilitated three-dimensional (3-D) X-ray imaging of the breast. Digital breast tomosynthesis and dedicated breast computed tomography (CT) are two promising 3-D modalities that are rapidly growing in their development and application.¹

Detecting a lesion in tomosynthesis images is confounded not only by several factors, such as system quantum noise, scattering, and other physical artifacts, but also by background tissue variation, which can lead to masking of an actual lesion as well as giving the misleading impression of a lesion where none exists. It is therefore important to consider all of these factors when studying lesion detection. One attribute of uncertain background tissue is its average density, the volume fraction of fibroglandular tissue, which can be considered as a first-order statistic. The background tissue average density is referred to as background tissue density for simplicity hereafter. Another important attribute of uncertain background tissue is its spatial distribution and texture, which reflects higher-order statistics and is referred to as heterogeneity hereafter.

The focus of this study is to characterize the effect of background tissue density and heterogeneity on the detection of irregular masses in digital breast tomosynthesis in a simulation study. The presented study expands upon initial results reported in Ref. 2 and was made possible through modeling the population, the imaging system, and the observers. The population was represented via several virtual breast phantoms from the extended cardiac-torso (XCAT) family,^3–7 with lesion models extracted from human subject tomosynthesis images. The geometry, beam spectrum, and noise and scatter characteristics of the imaging system were modeled after a prototype tomosynthesis imaging system to simulate the image formation chain. Finally, a doubly composite hypothesis signal detection theory paradigm, where both the $H_0$ and $H_1$ hypothesis have uncertain parameters,⁸ was devised and implemented to evaluate and characterize the performance of observers in signal known exactly (SKE) or signal known statistically (SKS) and background known statistically (BKS) detection paradigms. The implemented likelihood ratio detection paradigm, a Bayesian ideal observer, takes advantage of the volumetric data available in tomosynthesis images, incorporates both lesion and background uncertainty in decision-making, and uses an optimal decision metric with the help of Monte Carlo integration techniques.⁹ In a doubly composite hypothesis paradigm, the numerator and the denominator of the likelihood ratio are each typically modeled mathematically by joint probability density functions (pdfs) of the data, conditional to unknown parameters (the realizations of the ensemble), which are then weighted by the prior probabilities of the parameters and integrated. An aspect of our approach is that it obtains the ensemble of realizations needed for the Monte Carlo integration directly from the ensemble of virtual breast phantoms. Initial exploratory human observer studies were conducted and receiver operating characteristic curves (ROCs) were obtained to get insight into whether this might also model the detection performance of experienced human observers who have some memory of previous images and decision performance. This study considers the particular task of deciding the presence or absence of a lesion in which there is uncertainty in both the structure of the lesion as well as the background.

2. Methods

Detecting a lesion in tomosynthesis images in a location known exactly sense can be modeled as classifying a given volume of interest (VOI) as corresponding to only background tissue or to background tissue plus a lesion. This detection/classification problem can be formulated in terms of a hypothesis test, where the null hypothesis indicates the sole presence of noisy and uncertain background tissue and the alternative hypothesis indicates the presence of a lesion in noisy and uncertain background tissue in a VOI in the tomosynthesis image set. In other words, the detector, emulating a radiologist reader, is trying to decide whether a VOI in tomosynthesis images contains a lesion or is simply normal breast tissue. The readers are usually trained by studying images of numerous cases and their corresponding pathological appearance. For each modality, the readers have practically formed a training library of statistically known background tissue variations with and without lesions. In the case of tomosynthesis, the readers have access to slices through the reconstructed volume of a compressed breast, which they can scroll through, look at in cine, or look at via their maximum intensity projection rendering. In this study, we considered the detector to have access to the tomosynthesis reconstructed slices in a VOI, forming such a “training” library. The following elaborate the design and implementation of this detection paradigm and its performance analysis.

2.1. Population Modeling

The XCAT breast phantoms were generated through processing and segmenting dedicated breast CT images of a large number of human subjects acquired at the University of California, Davis, as part of an institutional review board approved study using a prototype dedicated breast CT scanner.^3–7 A group of 20 different breast phantoms from the XCAT family was selected to represent the population with different shapes, sizes, and densities. Each phantom voxel was assigned to adipose (0% dense) or three fibroglandular classes of 80%, 90%, and 100% to create a realistic “feathering effect” at the transitions between adipose and fibroglandular tissue. The mesh models for these breast phantoms were compressed to 4 cm thickness in the craniocaudal direction using a simplistic mathematical technique.³ The breast phantoms can also be compressed using finite-element compression techniques.¹⁰ Figure 1 shows the middle axial slice through the 4-cm-thick compressed volume of the 20 breast phantoms. The gray values correspond to different fibroglandular tissue classes based on density. The various anatomical distributions of the tissues, the different shapes, sizes, and densities of the breasts, can be appreciated in this figure.

Fig. 1.

Middle slice through the 4-cm-thick compressed volume of the 20 breast phantoms used in the study. The gray values correspond to different fibroglandular tissue classes based on density. Note the various shapes, sizes, and densities of the phantoms.

2.2. Lesion Modeling

The lesion models for this study were generated from the tomosynthesis images of five different human subjects with biopsy-proven cancers. These masses were segmented and used to inform the fitting of a mathematical volume with a Gaussian edge profile¹¹ to be bound in a $5\times 5\times 5\mathrm{mm}$ cube. Figure 2 depicts the middle slice through the tomosynthesis images of the human subjects and the corresponding models generated based on them.

Fig. 2.

Five lesion models: middle tomosynthesis slice through the lesions in human subject images (top), volume rendering of the generated models (middle), and the three fibroglandular classes after scaling the lesions to fit in a $5\times 5\times 5\mathrm{mm}$ bounding box.

A grid with 200 positions 1 cm apart was designed to embed lesions in the central depth of each breast phantom. This arrangement was designed to take advantage of as much of the breast volume as possible. Only locations falling within the uniformly compressed 4-cm-thick region of each breast were used for fair comparison. Figure 3 depicts the lesion grid with one of the five lesion models in all of its 200 positions. Using this technique, a total of 2173 grid positions were identified for lesion placement within the 20 breast phantoms.

Fig. 3.

Lesion grid with 200 positions 1 cm apart filled with the same lesion model.

2.3. Image Formation

The geometry, spectrum, and physical characteristics of a tomosynthesis unit (Siemens MAMMOMAT Inspiration, Erlangen, Germany) were used to simulate the tomosynthesis images. Twenty-five projection images spanning a 45-deg arc were simulated by ray tracing through the mesh models of the breast phantoms, implemented on a graphics processing unit (GPU) cluster. A conventional mammographic spectrum with W/Rh at 30 kVp was used for generating the projection images. Pixels were $4\times 4$ binned to $340\mu \mathrm{m}$ size to limit computational complexity. Considering the spatial extent of the imaging features, this process was expected to have negligible effects on the results.

Noise and scatter were simulated and added to the projection images. The scatter contribution was generated by convolution of the scatter point spread function with the primary image,¹² then scaled by the empirically measured scatter-to-primary ratio for this thickness and energy. The noise magnitude as a function of binned pixel value and the noise power spectrum (NPS) of the tomosynthesis system were estimated based on a 50% glandular, 4-cm uniform phantom (Computerized Imaging Reference Systems Inc., Norfolk, Virginia). To add the noise to the projection images, a Gaussian noise map, which is a good approximation of the Poisson noise in the projection images, was created and filtered by the square root of the NPS curve. This map was then multiplied by an intensity-to-noise magnitude map defined by the measured trends to give the overall noise pattern.¹³ The magnitude of the added noise corresponded to an average glandular dose of 1 mGy.

A standard filtered back-projection algorithm with a cosine filter implemented on a GPU cluster was used to reconstruct the tomosynthesis slices at 1 mm slice spacing and $340\mu \mathrm{m}$ in-plane resolution.^14,15 To estimate the noise in the tomosynthesis reconstructed images, a requirement of our likelihood ratio calculations, projection images of a uniform phantom were simulated with added noise and scatter and reconstructed. The first-order pdf of the reconstructed noise was found to correspond to a zero-mean Gaussian noise, but presumed not to be white. Figure 4 shows simulated projection and reconstructed images of a breast phantom with various lesions embedded in its middle depth, with and without simulated noise and scatter.

Fig. 4.

Central projection image of a breast phantom with lesions embedded in its middle depth (a) without and (b) with noise and scatter. The middle slice through the tomosynthesis reconstructed volume of the same breast phantom (c) without and (d) with noise and scatter.

2.4. Observer Modeling

2.4.1. Detection paradigm

It is commonly known that dense parenchymal background tissue may obscure lesions, leading to degradation in lesion detection.^16–20 At the same time, parenchymal tissue can mimic lesions, leading to a high number of false positives.^16–20 In this study, we aimed to quantify the impact of background tissue heterogeneity on lesion detection and put that in comparison with background tissue density. As such, we evaluated the dependence of detection performance on both background tissue density and heterogeneity.

The detection problem can be formulated as a doubly composite signal detection theory paradigm⁸ where the null hypothesis indicates only noisy and uncertain background tissue, whereas the alternative hypothesis indicates the presence of a lesion in noisy and uncertain background tissue. The optimum detector for this situation forms the likelihood ratio, which incorporates the uncertainties in an optimal way. In the context of observer modeling, the likelihood ratio detector is sometimes called a Bayesian ideal observer since the ROC using this optimal approach provides a realistic upper bound on detection performance.

To reiterate, the detector is presented with a VOI in reconstructed tomosynthesis images and processes the information so as to decide whether there is a lesion in the VOI or if it is solely background tissue. Each VOI has $K$ voxels. The detector has access to $N$ different realizations of a VOI of this size in its training dataset, which represents the uncertainty in the background tissue variations. $S^0$ is the set of these $N$ background-only VOIs. The detector also has access to a copy of each VOI from $S^0$ with a given lesion model, $l$, embedded in the same background tissue for all lesion models $l=1,\dots ,L$. $S_{1,\dots ,L}^1$ is the set of these $L\times N$ VOIs. Therefore, the detector training dataset consists of $S^0\cup S_{1,\dots ,L}^1$. The detector classifies a vectorized test VOI $x_v=(x_{v,1},\dots ,x_{v,k})$ as belonging to either hypothesis,

$$\begin{gathered} H_0:x_v=s_i^0+n,s_i^0=(s_{i,1,}^0,\dots ,s_{i,K}^0),i=1,\dots ,N,s_i^0\in S^0\text{or} \\ {H}_1:x_v=s_i^1(l)+n,s_i^1(l)=[s_{i,1}^1(l),\dots ,s_{i,K}^1(l)],i=1,\dots ,\mathrm{N},l=1,\dots ,L,s_i^1(l)\in S_{1,\dots ,L}^1, \end{gathered}$$ (1)

where $n=(n_1,\dots ,n_k)$, $n_i\sim N(0,\sigma )$ represents the noise. Note that noise refers to the aggregate deteriorating effects of system quantum noise, scatter, and other physical characteristics on reconstructed tomosynthesis images. To assess the effect of the background tissue on detection performance, noise was modeled as an additive zero-mean white Gaussian noise with a standard deviation estimated from the tomosynthesis images.

The likelihood ratio is formed by calculating the joint pdf of a VOI conditional to each of the two hypotheses. In this work, a Monte Carlo integration technique is used to approximate the marginal pdfs. This results in a form of the likelihood ratio that uses the available realizations of a VOI in the training dataset.⁹ This direct use of actual data realizations is a major feature of this approach as opposed to trying to capture the characteristics of the data through multivariate Gaussian modeling. Equations (2)–(4) show the derivation of the likelihood ratio, ${\lambda }_v$, for a given test VOI, $x_v$, for the doubly composite hypothesis signal detection theory problem, namely where there is background uncertainty present under both hypotheses, and lesion uncertainty under the alternative hypothesis. In particular, the likelihood ratio directly incorporates the uncertain background information by capturing its dependencies and correlations. It was also assumed that the training dataset does not include the exact background tissue as in the test VOI, with or without a lesion embedded

$$\begin{gathered} {\lambda }_v=\frac{p(x_v|H_1)}{p(x_v|H_0)}=\frac{\sum _{l=1}^L\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^{N}p[x_v|H_1,s_i^1(l)]·p[s_i^1(l)|H_1]}{\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^{N}p(x_v|H_0,s_i^0)·p(s_i^0|H_0)}, \\ =\frac{\sum _{l=1}^L\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^N\prod _{j=1}^{K}p[x_{v,j}|H_1,s_i^1(l)]·p[s_i^1(l)|H_1]}{\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^N\prod _{j=1}^{K}p(x_{v,j}|H_0,s_i^0)·p(s_i^0|H_0)}. \end{gathered}$$ (2)

Substituting the conditional probability distribution functions of the test voxels under each hypothesis based on Eq. (1) in the likelihood ratio results in

$${\lambda }_v=\frac{\sum _{l=1}^L\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^N\prod _{j=1}^K\frac{1}{\sigma \sqrt{2\pi }}\exp \{-\frac{{[x_{v,j}-s_{i,j}^1(l)]}^2}{2{\sigma }^2}\}·p[s_i^1(l)|H_1]}{\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^N\prod _{j=1}^K\frac{1}{\sigma \sqrt{2\pi }}\exp [-\frac{{(x_{v,j}-s_{i,j}^0)}^2}{2{\sigma }^2}]·p(s_i^0|H_0)}.$$ (3)

The VOIs in the training dataset were assumed to be equiprobable. Therefore,

$${\lambda }_v=\frac{\sum _{l=1}^L\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^N\exp \left(\sum _{j=1}^K\right[-\frac{{[x_{v,j}-s_{i,j}^1(l)]}^2}{2{\sigma }^2}\left]\right)·\frac{1}{LN}}{\sum _{\begin{array}{l}i=1\\ i\ne v\end{array}}^N\exp \left\{\sum _{j=1}^K\right[-\frac{{(x_{v,j}-s_{i,j}^0)}^2}{2{\sigma }^2}\left]\right\}·\frac{1}{N}}.$$ (4)

Note that the final results are joint pdfs of the VOI conditional to each of the hypotheses and are typically dependent and in general not multivariate Gaussian and hence not representable by a mean vector and covariance matrix. Furthermore, this likelihood ratio goes beyond simple cross correlation by including the signal energy terms in both its numerator and denominator. The likelihood ratio is calculated for every VOI in the test dataset. The likelihood ratio may then be thresholded to calculate the probability of detection (sensitivity), $P_{\mathrm{D}}$, and the probability of false alarm (1-specificity), $P_{\mathrm{F}}$, to obtain the ROC, or the resultant area under the curve (AUC). AUC values and their associated standard error and 95% confidence interval were computed with ROCkit software (University of Chicago, Chicago, Illinois, 2011).

2.4.2. Test and training datasets

Two different sets of tomosynthesis images were generated to constitute the test and training data under the two hypotheses. Under $H_0$, from the images of the 20 breast phantoms, without inclusion of any lesions, 2173 VOIs of dimensions $1\times 1\times 1\mathrm{cm}$ were extracted at the locations that fall within the uniformly 4-cm-thick region of each breast. These VOIs constituted $S^0$ with $N=2173$. Under $H_1$ on the other hand, tomosynthesis images were generated with the inclusion of a lesion grid. A set of 2173 VOIs of the same dimensions was extracted from the same locations as in $S^0$ for each of the five lesion models, resulting in a total of $5\times 2173$ VOIs. Different subsets of these $5\times 2173$ VOIs could be used to constitute data for problems with different hypotheses for $H_0$ and $H_1$. The subscript in $S_l^1$ specifies the lesion model(s) used in the VOIs in the set; for instance $S_l^1$ consists of 2173 VOIs with lesion model 1 embedded in them, and $S_{\mathrm{2,3},\mathrm{4,5}}^1$ consist of the 2173 VOIs with lesion models 2, 3, 4, and 5 embedded in them, respectively, constituting $4\times 2173$ VOIs in total. Figure 5 shows sample VOIs from one breast with and without lesions. The test and training datasets were used in a leave-one-out fashion, meaning that each testing background was excluded from training. The density of every VOI was calculated from the corresponding $1\times 1\times 1\mathrm{cm}$ region in the voxelized breast phantom as a fraction of fibroglandular tissue in the volume. Figure 6 shows the distribution of the VOI density across the 2173 VOIs. The mean VOI density per breast for the 20 breast phantoms was 0.23, in a range of 0.02 to 0.77.

Fig. 5.

Middle slice through adjacent tomosynthesis reconstructed VOIs from one breast (a) with a lesion model embedded in them and (b) without any lesions. Notice that the lesions are invisible in certain VOIs as a result of tissue superposition, noise, and scattering.

Fig. 6.

Histogram of the distribution of VOI densities across 2173 VOIs.

2.5. Detection Analysis

2.5.1. Effect of background tissue density uncertainty

To study the effect of background tissue density on detection performance, the 2173 background realizations were divided equally into five categories based on the density of the VOI, denoted by $d(·)$. The effect of background tissue density uncertainty was evaluated in two different paradigms described below. Note that under both paradigms, the detector had to classify a test VOI in a given density category as either background-only or background plus a known lesion. The background was only known statistically.

SKE-BKS: First, it was assumed that the detector is trained in all density categories but is tested in a certain density category. The test dataset in each density category contained one fifth of the cases in the training dataset. For each density category, performance was evaluated as a function of the test and training sample size.

SKE-BKS-density known statistically (SKE-BKS-DKS): Next, it was assumed that the detector knows the background tissue density category of the test VOI but not its particular background tissue density, which is part of the underlying uncertainty. The detector was both tested and trained in the same density category. This paradigm examined the effect of background uncertainty with the knowledge of background density category.

2.5.2. Effect of lesion uncertainty

To characterize the effect of lesion uncertainty on detection performance, two different paradigms were considered. The background was only known statistically.

SKE-BKS-across lesions: First, it was assumed that the detector exactly knows the lesion; this situation was implemented for every one of the five lesion models. This paradigm examines the effect of lesion variability on detection performance.

SKS-BKS: Next, the detector was assumed to only have a statistical knowledge of the lesion in the sense that it is trained with a finite number of similar lesion possibilities. The purpose of this paradigm was to examine the effect of lesion uncertainty in addition to background uncertainty when the detector is trained with several lesion models but tested on a new lesion model.

2.5.3. Effect of background tissue heterogeneity uncertainty

Up to this point, the data collected from the 20 different breast phantoms were combined to evaluate the detection performance. To evaluate the detection performance in an individual breast, the lesion known exactly, background known statistically paradigm was selected for each VOI and the detector assumed the VOIs for a given breast were statistically independent.

The trends in log-likelihood ratio under $H_0$ and $H_1$ at the 2173 VOI locations across the 20 breast phantoms were compared with the trends in background tissue density to examine the contribution of background tissue density and heterogeneity on detection performance.

2.6. Human Observer Study

The implemented detector was validated with a human observer study. The observers were presented with images from the simulated tomosynthesis images one by one to score their confidence in the presence of a lesion in the images. The midslices of 200 tomosynthesis VOIs were presented to the observers in random order. These images were displayed to the scale at $1\mathrm{cm}\times 1\mathrm{cm}$ on a grayscale standard display function-calibrated, 5-megapixel display. The image set contained 20 VOIs from each of the five density categories, with and without a lesion embedded in them. The images included just one lesion model. The observers were comprised of two medical physicists (J.Y.L. and E.S.) with many years of experience evaluating image quality for tomosynthesis and five students familiar with medical imaging. There was no significant difference in their performance. Since the detection task was relatively simple and no clinical diagnosis was required, it was sufficient to rely upon physicists and students for this study. The performances of the observers were averaged in each density category, and interobserver error bars were calculated. Figure 7 shows sample images presented to the observers.

Fig. 7.

Sample midslice images of the VOIs in the five density categories (density increasing from top to bottom), (a) with embedded lesion, or (b) background-only.

3. Results

3.1. Human Observers Performance

Figure 8 shows the average ROC curve of all observers across the five density categories. The AUC values are also presented in a bar chart in Fig. 8. Detection performance changes with background density; as density increases, the detection performance deteriorates.

Fig. 8.

Average observer performance with respect to background density; error bars indicate interobserver variation. Note that the detection performance deteriorates as the background density increases.

3.2. Effect of Background Tissue Density Uncertainty

Figure 9 shows the ROC curve of the implemented detector under the SKE-BKS paradigm. Although the lesion is known exactly, the detector is faced with background uncertainty and does not know the density.

Fig. 9.

ROC curve for the SKE-BKS paradigm, where the lesion is known exactly but the background is only known statistically. The AUC is $0.87\pm 0.006$ with 95% confidence interval [0.854, 0.876].

Under the same conditions, the ROC curves were generated for each density category separately to evaluate the effect of density uncertainty on the detection performance in each density category in the SKE-BKS paradigm, Figs. 10(a) and 10(b). As density was increased between 0, 0.005, 0.049, 0.142, 0.332, and 1.0 (density category bounds), it is seen that the AUC was decreased by 1%, 7%, 13%, and 19%, respectively. In other words, the detection performance deteriorated as the density increased. The concordance of these performance trends with those of human observers serves to validate the performance of the detector and the tools and models used in this study.

Fig. 10.

(a) ROC curves for different test density categories in the SKE-BKS paradigm, where the detector is trained on all density categories, and (b) AUC of each curve as a function of density. (c) ROC curves for different density categories in the SKE-BKS-DKS paradigm, where the detector is tested and trained on a specific density category, and (d) AUC of each curve as a function of density. The change in the density training paradigm increased the AUC by an average of 5% in each density category. Error bars denote standard error in AUC values.

With the test and training datasets divided into the five density categories in the SKE-BKS-DKS paradigm, the effect of more detailed knowledge of the density category on detection performance and the AUC are presented in Figs. 10(c) and 10(d). The results suggest that the knowledge of the density category improves the performance of the detector by an average of 5% increase in the AUC in each density category, compared to the SKE-BKS paradigm.

Under the paradigm of training on all density categories, the SKE-BKS paradigm, the effect of varying the number of VOIs included in the test and training datasets on the AUC was studied to ensure the stability of the above findings. Note that when different values were used, the number of VOIs in each density category was not necessarily equalized. It appears that a minimum of 400 independent VOIs needed to be included in the training dataset for the results to approach the steady state. This finding ensured that the number of VOIs included in the study was sufficient.

3.3. Effect of Lesion Uncertainty

First, the effect of lesion variability was examined on the detection performance in the SKE-BKS across lesions paradigm. The use of the five lesion models led to similar performance in terms of density dependence. The ROC curves corresponding to these paradigms are shown in Fig. 11(a). As the density was increased over the same density categories as before, the AUC dropped. The AUC varied between 2% and 9% across the lesion models in the same density category. The standard deviation of the AUC values across the five lesions was increased as the density was increased over the same categories as before.

Fig. 11.

(a) ROC curves for the SKE-BKS across lesions paradigm and (b) the SKS-BKS paradigm, along with the SKE-BKS paradigm in solid lines for comparison.

Next, the ROC curves for the case of lesion and background known statistically in the SKS-BKS paradigm are shown in Fig. 11(b) in each density category. The AUC dropped between 2% and 7% in the same density categories in comparison with the SKE-BKS paradigm. These results are in accordance with the expectation that not knowing the exact characteristics of the lesion can make the detection harder. This last paradigm can be considered to be a preliminary result closer to a situation for a radiologist reader; the radiologist has seen many sample lesions, but as in this work not the exact lesion in the current patient. Table 1 shows the AUC value for each of these paradigms along with its standard error and 95% confidence interval.

Table 1.

$\mathrm{AUC}\pm$ standard error values and 95% confidence intervals for different detection paradigms. Equalized density categories are denoted by bounds on density, $d$.

Paradigm	$0\le d<0.005$	$0.005\le d<0.049$	$0.049\le d<0.142$	$0.142\le d<0.332$	$0.332\le d<1$
SKE-BKS (lesion 1)	$0.98\pm 0.004$ [0.975, 0.991]	$0.97\pm 0.005$ [0.957, 0.978]	$0.90\pm 0.011$ [0.883, 0.925]	$0.78\pm 0.016$ [0.753, 0.815]	$0.63\pm 0.017$ [0.595, 0.663]
SKE-BKS (lesion 2)	$0.98\pm 0.004$ [0.977, 0.992]	$0.97\pm 0.005$ [0.961, 0.982]	$0.91\pm 0.010$ [0.886, 0.924]	$0.78\pm 0.016$ [0.752, 0.813]	$0.61\pm 0.019$ [0.573, 0.647]
SKE-BKS (lesion 3)	$0.99\pm 0.004$ [0.979, 0.993]	$0.97\pm 0.005$ [0.964, 0.983]	$0.93\pm 0.008$ [0.917, 0.947]	$0.81\pm 0.014$ [0.780, 0.835]	$0.63\pm 0.019$ [0.596, 0.670]
SKE-BKS (lesion 4)	$1\pm 0.002$ [0.991, 0.999]	$0.99\pm 0.003$ [0.984, 0.996]	$0.96\pm 0.007$ [0.945, 0.972]	$0.86\pm 0.014$ [0.838, 0.890]	$0.66\pm 0.031$ [0.596, 0.716]
SKE-BKS (lesion 5)	$0.99\pm 0.002$ [0.988, 0.998]	$0.99\pm 0.004$ [0.979, 0.994]	$0.95\pm 0.007$ [0.941, 0.966]	$0.85\pm 0.012$ [0.829, 0.877]	$0.62\pm 0.019$ [0.582, 0.656]
SKE-BKS-DKS (lesion 1)	$1\pm 0.000$ [1, 1]	$0.99\pm 0.003$ [0.987, 0.999]	$0.96\pm 0.006$ [0.952, 0.977]	$0.84\pm 0.014$ [0.815, 0.868]	$0.61\pm 0.019$ [0.5876 0.651]
SKS-BKS	$0.96\pm 0.006$ [0.944, 0.968]	$0.93\pm 0.009$ [0.908, 0.943]	$0.83\pm 0.013$ [0.806, 0.858]	$0.73\pm 0.018$ [0.699, 0.768]	$0.59\pm 0.019$ [0.551, 0.625]

Figure 12 shows the concordance of the most representative detection paradigm (SKE-BKS) with observer data as a function of density. The results indicate that the detector provides a density dependency similar to that of human observers but not identical. This difference should remain as an area of future exploration; however, it can potentially be attributed to the differences in the design of the detector and the human observer study. This data, however, can be used to calibrate the detector data to observer performance.

Fig. 12.

Scatter plot of AUC against density for the implemented detector and the human observers. The SKE-BKS paradigm, which most closely relates to the human observer study, was selected for this comparison. Each density category is presented by its mean value. The standard error bars are presented for the SKE-BKS data points.

3.4. Effect of Background Tissue Heterogeneity Uncertainty

Finally, performance across different VOIs is analyzed to determine the effect of tissue heterogeneity. As suggested by the AUC values for individual breasts (mean: 0.81; standard deviation: 0.148), detection performance varies substantially across different breasts and it tends to be closely related to the breast density.

To further visualize the location-wise dependence of detection performance within every breast, Fig. 13 aims to show the correspondence between background tissue density, background tissue heterogeneity, the log-likelihood ratio under $H_0$, and the log-likelihood ratio under $H_1$ at every VOI location across the 20 breast phantoms. The corresponding log-likelihood ratio under $H_0$ and log-likelihood ratio under $H_1$ at the VOI locations as calculated per Eqs. (2)–(4) are presented in the form of two heat maps. Comparing the same VOI location across the middle tomosynthesis slice and the heat maps suggests that in general the log-likelihood ratio under $H_0$ follows the density trend and the log-likelihood ratio under $H_1$ follows the opposite density trend; generally, the log-likelihood ratio under $H_0$ is higher where the density is higher, and the log-likelihood ratio under $H_1$ is higher where the density is lower. However, there are several locations for which the reverse is observed. This is not too surprising since it is the uncertain heterogeneity of the background that is being modeled using actual realizations of real tomosynthesis reconstructed images, for which the density is just one parameter that only partially characterizes the uncertainty in the background.

Fig. 13.

(a) Middle slice through the tomosynthesis reconstructed volume, (b) the corresponding heat map of extracted VOI density, and (c and d) log-likelihood ratio under $H_0$ and $H_1$ maps per Eqs. (2)–(4) of the 20 breast phantoms. The AUC of the SKE-BKS paradigm in a given breast varied considerably across the 20 breasts (mean: 0.81; standard deviation: 0.148).

Figure 14 shows a scatter plot of the log-likelihood ratio under $H_0$ and the log-likelihood ratio under $H_1$ against density across all VOIs. The data suggest that the relationship between density and the log-likelihood ratio under $H_0$ or $H_1$ is not linear and can rather be approximated by a quadratic fit. As a result of density variation from 0 to 1, the quadratic fit to the log-likelihood ratio under $H_0$ varied by 3.36, and the quadratic fit to the log-likelihood ratio under $H_1$ varied by 17.63. This suggests that the density has a more significant effect on the log-likelihood ratio under $H_1$ than on the log-likelihood ratio under $H_0$.

Fig. 14.

Distribution of the log-likelihood ratio under $H_0$ (a) and $H_1$ (b) per Eqs. (2)–(4) versus density and their corresponding quadratic fits. Dotted vertical lines indicate the density categories with equal number of samples, and solid vertical lines at mid-density points indicate standard deviation of the log-likelihoods in the density category.

Furthermore, the fact that the same density results in multiple possible values for both the log-likelihood ratio under $H_0$ and the log-likelihood ratio under $H_1$ emphasizes that density is not the only factor playing a role in detection performance.

Background tissue heterogeneity, though yielding the same average VOI density, can significantly change the detection performance. As a result of background tissue heterogeneity, an average standard deviation of $\pm 7.61$ was measured in the log-likelihood ratio under $H_0$ and $H_1$ within each equalized density category, as separated by vertical dotted lines in Fig. 14. These results suggest that, based on the metrology of this study alone, background tissue heterogeneity can affect the log-likelihood ratio under $H_0$ $\sim 4.5$ times as much as background tissue density, while background tissue density seems to be affecting the log-likelihood ratio under $H_1$ slightly more than background tissue heterogeneity.

4. Discussion

Given the recent advances in computation and modeling, virtual clinical trials (VCTs) can be carefully designed and carried out to inform, orient, or potentially replace clinical trials. VCTs involve simulation of the patient population, image formation, and the observers. In this study, we elaborated on the employment and advancement of the sophisticated tools and models that were developed in previous studies^3–7 and can potentially be used in a VCT.

This study focused on the detection of irregular masses in uncertain heterogeneous background tissue as perceived from tomosynthesis reconstructed images. The detection task was defined as deciding whether a given VOI includes a lesion or not, under lesion known exactly or statistically and background known statistically paradigms. The detection performance analysis was carried out rigorously through sensitivity and specificity analyses, characterizing the effects of background tissue density and heterogeneity uncertainty and lesion uncertainty on the detection performance.

The observer model used in this study surpasses the models used in previous studies by simultaneously considering several factors: (1) it took advantage of the 3-D data provided by tomosynthesis, (2) it incorporated the uncertainty resulting from realistic background tissue variations in the detection task, (3) it incorporated the uncertainty resulting from lesion variation in the detection task, (4) it took advantage of the realistic data realizations in evaluating the decision metric rather than constructed mathematical uncertainty, and (5) it is ideal in the sense of using likelihood ratios as its decision metric.

The performed human observer study served to validate the observer model used in this study. The human observers demonstrated deteriorating detection performance as the background density increased. The concordance of these trends with those of the implemented detector validated the application of the detector and the tools and models used to perform this simulation study.

Similar to the human observers, the detector performance results suggest that the detection performance deteriorates as the background tissue density increases. However, the knowledge of the background tissue density category seems to have an effect as well. The additional information about background tissue density category can somewhat improve the detection performance in the presence of uncertain background heterogeneity. The close agreement of these results with those reported previously^16–20 may serve as a first-order validation of the simulation platform for possible VCTs.

The stability of the AUC values after inclusion of enough VOI realizations in the training dataset suggests that the total number of realizations used in the analysis were sufficient for drawing conclusions. Furthermore, the distribution of background tissue density was in agreement with reported average breast density in recent literature.²¹

When using the same lesion model in the test and training datasets, the different lesion models resulted in similar performance trends with respect to density. When using different lesion models in the training dataset than the lesion model in the test dataset, the AUC dropped in comparison with using the same lesion model in the test and training datasets. These preliminary results are in accord with the expectation that not knowing the exact characteristics of the lesion can make the detection harder and that the more samples of lesions in the training dataset can help with the detection.

Furthermore, it was hypothesized that the detectability is not only affected by background tissue density uncertainty but also by background tissue heterogeneity uncertainty. In fact, the density is a partial characterization of the background uncertainty. This latter was observed through comparing the log-likelihood ratio under $H_0$ and $H_1$ with density. It was seen that not only did the likelihoods vary significantly across different breasts, but also they did not necessarily correspond to the background tissue density. This suggests that in addition to background tissue density, background tissue heterogeneity also plays a role in the detection performance. Although the test VOIs were not designed specifically to avoid background tissue density and heterogeneity interdependence, it can be deduced that both background tissue density and heterogeneity affect detection performance. Given the substantial role that uncertainty in the background tissue heterogeneity and density play in the detection performance, it seems critical to include heterogeneous anatomical backgrounds with enough variation when performing VCTs.

The results reported in this study, though informative and promising, can be further refined by including a larger number of breast phantoms and lesion models, as well as including other lesion shapes, for better representation of the population, and by simulating higher-resolution images at the cost of computational resources. In that regard, it should be noted that both the breast tissue and lesion definitions included the inherent image properties of the original systems from which they were extracted (dedicated breast CT and digital breast tomosynthesis, respectively). Future work may include deblurring processes for better spatial definitions. The dose condition for the simulations targeted an average radiation condition corresponding to an average glandular dose of 1 mGy. However, given the density variability across the breast models, that condition corresponded to different glandular dose imaging conditions in different breasts. Future work may include a systematic evaluation of effect of dose on detectability. More advanced modeling of the noise in projections images as well as reconstructed images can be carried out to account for underlying physical phenomena and noise correlations. Background tissue heterogeneity metrics can be devised and evaluated to better characterize the effects of background tissue density and heterogeneity with respect to each other. The effects of rotation, translation, and scale of training lesions can also be quantified. In the future, the performance of the devised detection paradigm can be compared with the performance of real observers.

5. Conclusion

This study focused on the detection of both certain and uncertain irregular masses in uncertain breast background tissue as perceived from tomosynthesis reconstructed images. Twenty breast models from the XCAT family with various shapes, sizes, and fibroglandular densities, representing a wide range of the population, were used to generate 2173 VOIs. Five irregular mass lesion models were generated from tomosynthesis images of human subjects and embedded in these VOIs. Tomosynthesis volumes were generated by reconstructing simulated projection images with added noise and scatter. The detection task was defined as identifying whether a given VOI includes a lesion or not, under lesion known exactly or statistically and background known statistically paradigms. The detection performance analysis was done rigorously through sensitivity and specificity analyses, which provide more insight into the performance compared to traditional figures of merit such as signal-to-noise ratio, contrast-to-noise ratio, and so on. Similarity of the trends in detection results with those of human observers served to validate the observer model. The differences between them, although they could be attributed to the differences in the study design, should remain an area of future exploration. The results suggested that both background tissue density uncertainty as well as higher-order heterogeneity uncertainties can affect detection performance. Given the significant role that uncertainty in the background tissue plays in the detection performance, it seems critical to include heterogeneous anatomical backgrounds with enough variation when performing VCTs. In the future, the framework presented in this work can be advanced to be applied to real patient images to compute ROC maps that may aid the diagnosis by human observers.

Biographies

Nooshin Kiarashi received her doctoral degree in electrical and computer engineering from Duke University while she was a member of the Carl E. Ravin Advanced Imaging Laboratories. Currently, she serves as a lead scientific reviewer at the Center for Devices and Radiological Health at the US FDA. Her research interests include development and application of advanced modeling and computing techniques to realize virtual clinical trials for optimization and evaluation of medical imaging systems.

Loren W. Nolte is a professor of electrical and computer engineering at Duke University. His research interests include developing Bayesian approaches to optimal signal detection, classification, localization and decision fusion in numerous applications. In collaboration with the Medical Physics Group at Duke, his research includes approaches that directly incorporate statistical information from real clinical data to improve sensitivity and selectivity in the detection of uncertain cancer tissue structures in the presence of background imaging uncertainties.

Joseph Y. Lo is a professor and associate vice chair for research of the Department of Radiology, Duke University School of Medicine. He also serves as director of the Carl E. Ravin Advanced Imaging Laboratories. His research focuses on development of anthropomorphic breast phantoms for virtual clinical trials, as well as radiogenomics for improved management of breast cancer.

W. Paul Segars is an associate professor of radiology and biomedical engineering and a member of the Carl E. Ravin Advanced Imaging Laboratories (RAILabs) at Duke University, Durham, North Carolina. He is among the leaders in the development of simulation tools for medical imaging research where he has applied state-of-the-art computer graphics techniques to develop realistic anatomical and physiological models.

Sujata V. Ghate is an associate professor of radiology specializing in mammography, tomosynthesis, breast US and MRI. She practices breast imaging in a university setting, teaching residents, fellows and medical students and collaborating on research projects with the Department of Medical Physics and Biomedical Engineering. She is currently a fellow of the society of breast imaging, and on the advisory board for the CDC Breast and Cervical Cancer Early Detection and Control Advisory Committee.

Justin B. Solomon received his doctoral degree in medical physics from Duke University in 2016 and is currently a medical physicist in the Clinical Imaging Physics Group (CIPG) at Duke University Medical Center’s Radiology Department. His expertise is in x-ray computed tomography imaging and image quality assessment.

Ehsan Samei DABR, FAAPM, FSPIE is a tenured professor at Duke University and the director of the Duke Medical Physics Graduate Program and the Clinical Imaging Physics Program. His interests include clinically relevant metrology of imaging quality and safety for optimum interpretive and quantitative performance. He strives to bridge the gap between scientific scholarship and clinical practice by meaningful realization of translational research and the actualization of clinical processes that are informed by scientific evidence.