Assessment of Volumetric Dense Tissue Segmentation in Tomosynthesis Using Deep Virtual Clinical Trials

Abstract

The adoption of artificial intelligence (AI) in medical imaging requires careful evaluation of machine-learning algorithms. We propose the use of a “deep virtual clinical trial” (DeepVCT) method to effectively evaluate the performance of AI algorithms. In this paper, DeepVCTs have been proposed to elucidate limitations of AI applications and predictions of clinical outcomes, avoiding biases in study designs. The DeepVCT method was used to evaluate the performance of nnU-Net models in assessing volumetric breast density (VBD) from digital breast tomosynthesis (DBT) images. In total, 2,010 anatomical breast models were simulated. Projections were simulated using the acquisition geometry of a clinical DBT system. The projections were reconstructed using 0.1, 0.2, and 0.5 mm plane spacing. nnU-Net models were developed using the center-most planes of the reconstructions with the respective ground-truth. The results show that the accuracy of the nnU-Net improves significantly with DBT images reconstructed with 0.1 mm plane spacing (78.4×205.3×40.1 mm³). The segmentations resulted in Dice values up to 0.84 with area under the receiver operating characteristic curve of 0.92. The optimization of plane spacing for VBD assessment was used as an exemplar of a DeepVCT application, allowing us to interpret better the input parameters and outcomes of the nnU-Net. Thus, DeepVCTs can provide a plethora of evidence to predict the efficacy of these algorithms using large-scale simulation-based data.

Graphical Abstract

1. Introduction

The adoption machine learning in clinical care demands a thorough evaluation of algorithms and evidence-based methods to confirm the effectiveness and safety of artificial intelligence (AI) applications [1]. “Black box” systems pose legal and ethical concerns, requiring substantial evidence for their accuracy in clinical applications [2]. Explainable AI (XAI) methods enable understanding and reproduction of learning and knowledge extraction, especially beneficial for weakly supervised and unsupervised learning [3]. A key trade-off exists between performance and explainability [4]; often, machine learning is considered less transparent, while easily interpretable algorithms are less accurate. Understanding the impact of input data and parameters on AI models is crucial. This study introduces an XAI approach named “deep virtual clinical trial (deepVCT).”

VCTs were developed to design, validate, and optimize imaging systems through fast simulations of human anatomy, image acquisition, and interpretation [5]. Combining simulated images from anatomical models (phantoms) with machine-based image interpretation, VCTs have evolved to tackle more complex tasks. Today’s VCTs enhance imaging system optimizations for clinical tasks and predict human reader performance [6]. This manuscript introduces the use of “DeepVCTs” to assess AI methods for the assessment of breast density, an imaging biomarker associated with the risk of masking cancers [7]. Masking can hinder early diagnosis and worsen outcomes. Moreover, a significant correlation exists between volumetric breast density (VBD) and breast cancer risk [7].

Automated tools for VBD assessment [8] show significant variance from radiologists’ qualitative evaluations, with discrepancies up to 14% in dense tissue classification [9]. To address these inconsistencies, machine learning offers state-of-the-art performance in VBD estimates [10]; however, without clear interpretability. DeepVCTs present a potential solution to evaluate and support machine learning algorithms for clinical usage.

In this study, we assess the performance of nnU-net [11] models trained for VBD segmentation. Simulations of digital breast tomosynthesis (DBT) images were used to provide ground-truth labels for the nnU-Nets. Our evaluation of nnU-net models focused on the segmentation accuracy influenced by breast density levels and plane spacing, using partial volume maps from both ground-truth and reconstructed DBT images.

2. Materials and Methods

2.1. Computer Simulations

2.1.1. Anthropomorphic Breast Phantoms

The OpenVCT framework was used to simulate tissue compartments randomly within a semi-ellipsoidal breast model [12]. The compartments represent glandular (i.e., dense) and adipose tissue bounded by fibrous Cooper’s ligaments [6]; dense compartments affect the percentage of VBD. The resulting phantoms consist of ground-truth labels that represent various tissue types and air (Fig. 1a and 1b). The glandular-tissue labels can be extracted from the breast phantoms (Fig. 1c) and used as binary-volume masks (Fig. 1d). The binary-volume masks correspond to the ground truth of air and simulated tissue. The simulation parameters used are shown in Table 1.

Figure 1:

(a) Ground-truth labels that represent each voxel-material: (I) background, (II) skin, (III) adipose, (IV) glandular, (V-VIII) Cooper’s ligaments. (b) Volume view of internal structures of the 3D breast models (skin is excluded). (c) Segmented glandular-tissue mask and (d) Volume view of the ground-truth VBD.

Table 1:

Summary of (A) phantoms and (B) acquisition parameters simulated.

(A) Phantom
Number of Simulations (#)	2,010
Fraction of Dense Compartments (%)	0, 10, 20, 30, and 50
Volume (mL)	700
Breast Compression (50% compression in z)	Medio-Lateral
Dimensions (x×y×z, mm)	78.4×205.3×63.3
Voxel Size (mm)	0.10
Number of Compartments (#)	425, 850, 1275, and 1700
Compartment Shape	{(0.1;1;1;2), (0.01; 1; 1; 4)}
Ligament Material, # ([thickness], mm)	4 [0.1, 0.3]
Skin Thickness (mm)	0.15
(B) Acquisition
Anode/Filter Material	W/Al
Filter Thickness, mm (focal spot, mm)	0.7 (0.3)
Projections, # [Angular Range°]	15 [−7.5, +7.5]
Tube Motion	Continuous
Detection Material	a-Se
Detector Element Size # (size, mm)	2048×1664 (0.140×0.140)
Detector Size (mm)	286.72×232.96
Source-to-Image Distance (mm)	700.0
Number of Planes, # (spacing, mm)	633 (0.1), 317 (0.2), and 127 (0.5)

2.1.2. Imaging Acquisition

Each tissue label corresponds to a combination of materials in the DBT projections, created by simulating x-rays tracing from the source through the phantom volume to the detector elements [6]. This process accounts for blurring from the size of detector elements and tube motion, combined with radiographic and electronic noises incorporated in the acquisitions. The acquisitions (Table 1) are simulated using the automatic exposure control of a clinical system (Selenia Dimensions, Hologic Inc., Bedford, MA) [13]. The projections are processed and reconstructed using a commercially available software library that allows customization of reconstruction parameters [14].

2.1.3. Data Pre-processing

The DBT acquisition parameters (Table 1) and phantom in-plane dimensions (784×2,053) were used to crop the DBT images. The DBT images and binary masks were resized using bilinear interpolation (97×256). The image aspect ratio, number of bits, and pixel values were preserved. The top and bottom planes (11.6 mm each) were excluded from each volume to eliminate skin and planes predominantly adipose.

The DBT reconstructions and binary masks were both used for training and testing the nnU-Net models. The number of images used as input to the models was held constant, thus the thickness of the central region varied with the learning model [selected from 5.1, 10.2, 25.5 mm (approaches I-III), and 40.1 mm (approach IV) - more details presented in section 2.2.1].

2.2. nnU-Net Architecture

We used the 3D U-Net structure from the nnU-Net framework [11] (Fig.2, top), which supports four encoder and four decoder layers. Each encoder layer performs two convolutions and one max pooling with leaky ReLU activation. The decoder layers begin with upsampling and a convolution, followed by two additional convolutions with leaky ReLU. This setup allows the encoder to pass feature maps to the decoder, enhancing image class prediction through the integration and activation of these features.

Figure 2:

(Top) nnU-Net architecture implemented for VBD assessment using customized plane spacing: 0.1, 0.2, and 0.5 mm (where any plane can assume the role of A-C). (Bottom) Approaches tested to evaluate the performance of the nnU-Net models, *considering “A” the best-performance model.

2.2.1. Training and Evaluating nnU-Net Models in VBD Assessment

VBD level and plane spacing were parameterized to evaluate changes in performance of the nnU-Net models. Plane spacing was evaluated to determine the best approach for segmentation of the fibroglandular tissue. Four approaches were used to evaluate the performance of the nnU-Net models as shown in Fig. 2 (bottom):

1. Independently trained models. The effect of plane spacing was evaluated using nnU-Net models trained for segmentation. Three models are trained independently, using binary masks and the respective DBT images reconstructed with plane spacing with sizes of 0.1, 0.2, and 0.5 mm. The optimal plane spacing was defined based upon the performance of these nnU-Net models (Fig. 2A–C)

2. Z-resolution agnostic model. Changes in segmentation were evaluated using the best model trained in (I) and testing this model with input volumes simulated with different plane spacings.

3. Fine-tune the best model. Changes in segmentation were evaluated using learned weights obtained from the best model in (I) to initialize the training of a fine-tuned model that was further trained with an input simulated with different plane spacings.

4. Refine the best model. Finally, the best performance model evaluated in approaches I-III was trained using a greater number of images (D), as shown in Table 2. This approach was designed to refine the nnU-Net model, test the method with a more representative phantom volume, and test the reproducibility of the proposed method.

Table 2:

Summary of image set and nnU-Net hyperparameters used for training and test the approaches described in the study. For approach IV, D is determined from the optimal plane spacing. Phantoms simulated using 0.1% of fraction of dense compartments were excluded from the test sets.

Approach	I	II	III	IV
nnU-Net Step	Training/Testing	Training /Testing	Fine-tuning/Testing	Training/Testing
Phantoms (#) Per Fraction of Dense Comp. (0.1, 10, 20, 30, and 50%)	1,000/260 200/65	−/260 −/65	750/260 150/65	1,000/260 200/65
nnU-Net Models (#)	3	0	2	1
Volume (w×h×d)	97×256×51	97×256×51	97×256×51	97×256×D
Epochs (#)	60	-	60	60
Batch Size (#)	6	-	6	6

A controlled group of phantoms was used for training and a separate controlled group of phantoms was used for testing each model (Table 2) evaluated in approaches I-IV. The control group consist of independent phantoms with different anatomical structures to avoid overfitting. To avoid computational burden, the approaches were evaluated using partial phantom volumes as input for the nnU-Net. The input (I-III) consists of a reconstruction slab with 51 central planes (97×256×51, voxels) with different plane spacing. Plane spacing with sizes of 0.1, 0.2 and 0.5 mm were evaluated, spanning the center 5.1, 10.2 and 25.5 mm of the phantom, respectively. In approach IV, a larger phantom volume was used as input of the model. D represents 40.1 mm of the central phantom volume at the optimal plane spacing; D is determined from the best performing model of approach I. The summary of parameters used for training and testing the nnU-Net is shown in Table 2.

The nnU-net models were saved every five epochs (total = 60 and batch size = 6); the loss and accuracy values were monitored to avoid overfitting. The models were trained for binary segmentation using the focal Tversky loss (FTL) [15] and the Adam optimizer with an initial learning rate of 3×10⁻⁴ (default). The FTL was used to improve the segmentation by shifting the optimal solution to minimize the number of false-negative predictions. FTL uses three hyperparameters (α, β, and γ) to improve recall rates in case of large class imbalance (positives and negatives). In this study, we calculated the values of α and β based upon the distribution of fibroglandular and non-fibroglandular voxels (excluding background) from the entire sample phantom (α=0.88 and β=0.12). The hyperparameter γ=3/4 was selected based upon prior work [15].

Approaches I-IV were evaluated using a RedHat Enterprise Linux-based cluster [16]. This high-performance computing cluster is installed and connected to the University of Pennsylvania Hospital System network.

2.2.2. Evaluation Metrics

VBD was calculated from the nnU-Net segmentation of fibroglandular tissue from non-fibroglandular tissue. Classically in segmentation, the object (or foreground) is segmented from the background and evaluated using Jaccard and Dice similarity coefficients. These two coefficients assign the same weight for false negative (FN) and false positive (FP) errors; they do not consider the imbalance in the number of positive and negatives. In this study, positives and negatives represent the ground-truth glandular and non-glandular voxels, respectively. The Tversky index (TV) (Eq. 1) [15] was used as an evaluation metric for the nnU-Net segmentations. This metric is an asymmetric similarity index where

$$TV=\frac{TP}{\left(TP+\alpha FN+\beta FP\right)},$$ (1)

and (TP) is true positive. The parameters α and β are considered weighting factors for the TV calculations. By setting values of α > β, FN will be penalized more than FP. Note that α = β = 0.5 and α = β = 1 simplifies TV into Dice and Jaccard coefficients, respectively. In this study, TV was calculated using the following (α, β) pairs: (0.5, 0.5), (0.88, 0.12), and (1, 1). The pair (0.88, 0.12) were selected based upon the distribution of fibroglandular and non-fibroglandular voxels of the phantom population (n=1,020). TV was used to evaluate the VBD assessment performed by the nnU-Net models, categorized by the fraction of dense compartments.

It is also reasonable to treat segmentation as a binary classification task in which voxels are classified as being fibroglandular or non-fibroglandular [17]. As such, the sensitivity (TPR), specificity (TNR), and efficiency (EFF), were calculated. Receiver operating characteristics (ROC) curves were constructed to compare the nnU-Net models. The area under the curve (AUC) of each model was calculated using the concatenated test set. The standard deviation (SD) in the AUCs across the test set was also calculated. These metrics were used to analyze the differences between the AUCs and explore nuances in the classification task.

Dice ratios (α = β = 0.5) and differences in AUCs (Δ) were used to compare changes in performance between models. Volumetric colormaps depicting TP, true negatives (TN), FP, and FN illustrate the accuracy (ACC) of each nnU-Net model (I-IV). For the best-performing model (IV), examples of volumetric maps of the nnU-Net confidence were also reported.

Finally, to evaluate changes in performance of the nnU-Net models, non-parametric Kruskal-Wallis test by rank and Mann-Whitney-Wilcoxon test were used to estimate differences in means of the segmentation metrics and to analyze significant differences using pairwise comparisons. The identifiers ‘ns’, ‘*’, ‘**’, ‘***’, and ‘****’ represent the significance code for p-values >0.05, <0.05, <0.01, <0.005, and <0.001, respectively.

3. Results

3.1. Approach I: Independently Trained Models

The summary of metrics evaluated for the three independent nnU-Net models is shown in Fig. 3. The model trained using volumes reconstructed with 0.1 mm plane spacing (reference) achieved the best overall performance, in particular for denser breasts (Fig. 3a). By comparing the reference model with the model trained using 0.2 mm plane spacing (Fig. 3b), the performance improvement yields a mean Dice increase of 33%, 15%, 10%, and 6% (P<0.001) for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments, respectively. The overall performance of the reference model is also higher than the model trained with volumes reconstructed using 0.5 mm plane spacing (Fig. 3c), offering a mean Dice increase of 45%, 26%, 17%, and 20% (P<0.001), for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments, respectively. Similarly, these changes in performance can also be noted in the mean Jaccard coefficient.

Figure 3:

Boxplot of metrics used to evaluate the performance of a nnU-Net model trained using volumetric images (97×256×51 voxel) reconstructed with (a) 0.1 mm, (b) 0.2 mm, and (c) 0.5 mm plane spacing.

Note that for the nnU-Net model developed using volumes reconstructed with 0.5 mm plane spacing, the trend of TV (α=0.88, β=0.12) and fraction of dense compartments decreases with increased fraction of dense compartments. This metric penalizes FNs more than FPs in its calculation (α > β), controlling for the imbalanced number of fibroglandular voxels.

Examples of segmentation using the nnU-Net models (approach I) are shown in Fig. 4. Note the number of FP and FN voxels is reduced substantially in the VBD segmentation using the nnU-Net model trained with input volumes reconstructed with 0.1 mm plane spacing (Figs. 4b and 4f).

Figure 4:

Examples of (a-e) reconstructed DBT images and respective volumetric colormap of the nnU-Net segmentation (approach I). Input volumes tested using plane spacing size of (b-f) 0.1 mm, (c-g) 0.2 mm, and (d-h) 0.5 mm.

3.2. Approach II: Z-resolution agnostic model

The summary of metrics evaluated using a pre-trained nnU-Net model (0.1 mm plane spacing) is shown in Fig. 5. The reference model did not improve the segmentation from volumes reconstructed using larger plane spacing (0.2 mm and 0.5 mm). The overall performance of the nnU-Net did not improve (Fig. 5a) when compared with the performance of models trained with 0.5 mm voxel size (Fig. 3c). Fig. 5a shows changes in the mean Dice coefficient of −5% (P=0.33), 4% (P<0.01), 4% (P<0.01) and −8% (P<0.001), for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments, respectively. Fig. 5b shows a mean Dice increase of 41% (P<0.001), 13% (P<0.001), 8% (P<0.001), and 4% (P<0.001), for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments. Examples of segmentation using the nnU-Net model tested in approach II are shown in the next section for comparison with approach III.

Figure 5:

Boxplot of metrics used to evaluate the performance of a nnU-Net model tested using volumetric images (97×256×51 voxel) reconstructed with (A) 0.2 mm and (B) 0.5 mm plane spacing.

3.3. Approach III: Fine-Tuned Model

The summary of metrics evaluated using fine-tuned nnU-Net models is shown in Fig. 6. The performance of the nnU-Net for segmentation can be improved with the use of learned weights acquired from the reference model (0.1 mm plane spacing). The segmentation slightly improved with the fine-tuned model trained with volumes reconstructed with 0.2 mm plane spacing. By comparing the results acquired with fine-tuned model (Fig. 6a) with the model trained with volumes reconstructed with 0.2 mm plane spacing (Fig. 3b), the improvement in performance represents a mean Dice increase of 39%, 12%, 6%, and 3% (P<0.001) for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments, respectively. In contrast, by comparing the results of the fine-tuned model (Fig. 6b) with the model trained with volumes reconstructed with 0.5 mm plane spacing (Fig. 3c), the performance of the fine-tuned model represents a variation in mean Dice of −28%, 2%, 19%, and 31% (P<0.05), for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments, respectively. Note that the fine-tuned models improve the performance in segmentation only if the input volumes of the pre-trained model have minor differences in reconstructed plane spacing (e.g., 0.1 mm and 0.2 mm) or if the input volumes have higher breast density (e.g., >30%).

Figure 6:

Boxplot of metrics used to evaluate the performance of a nnU-Net model fine-tuned using volumetric images (97×256×51 voxel) reconstructed with (A) 0.2 mm and (B) 0.5 mm plane spacing.

Examples of segmentation using the nnU-Net models developed in approaches II and III are shown in Fig. 7. Note that the fine-tuned model trained with volumes reconstructed with 0.5 mm plane spacing (Fig. 7f) did not improve the models’ performance. In addition, no major differences in segmentation are seen with the fine-tuned model trained with volumes reconstructed with 0.2 mm plane spacing (Fig. 7c).

Figure 7:

Examples of (a-e) reconstructed DBT images and respective volumetric colormap of the nnU-Net segmentation (approach I). Input volumes tested using plane spacing with size of (b-f) 0.1 mm, (c-g) 0.2 mm, and (d-h) 0.5 mm.

3.4. Approach IV: Refined Learning Model

Finally, the best-performing of the above approaches (approach I with 0.1 mm plane spacing) was refined to use the entire phantom volume (40.1 mm) as input for training. As predicted, the overall performance of the segmentation did not vary significantly when comparing the results calculated using the model trained with small volumes (Fig. 3a) and with larger volumes (Fig. 8a). The improvement in performance represents a mean Dice variation of 0% (P=0.77), −1% (P=0.38), −2% (P=0.06), and −1% (P=0.06), for phantoms simulated using 10%, 20%, 30%, and 50% of dense compartments, respectively. Larger volumes did not affect the overall performance of the nnU-Net model (approach IV), demonstrating stability and consistency in convergence for the best-performance model to the optimal solution.

Figure 8:

(a) Boxplot of metrics used to evaluate the performance of a nnU-Net model refined using volumetric images (97×256×401 voxel) reconstructed with 0.1 mm. (b) ROC curves of segmentation models tested in approaches I and IV.

ROCs and AUCs were also calculated to show the changes in performance of the nnU-Net models (Fig. 8b) developed with volumes reconstructed with 0.1, 0.2, and 0.5 mm plane spacing (approach IV – approach I). The ΔAUC between models 0.1 mm - 0.1 mm, 0.1 mm - 0.2 mm, and 0.1 mm - 0.5 mm was 0.04, 0.04, and 0.09, respectively. These results show that larger volumes improve the overall performance in the classification task (approach IV), demonstrating higher overall accuracy in the classification of simulated glandular and non-glandular tissue.

However, it is important to mention that the nnU-Net models were trained with an imbalance of positive and negative data (i.e., background and foreground). The FTL function shifts the optimal solution of each model, affecting the operating points and shape of the ROC curves. Asymmetric evaluation metrics should be considered for analyzing changes in performance between models. Table 3 summarizes the evaluation metrics used to analyze changes in performance of every nnU-Net model (approaches I-IV). Note that there are differences in performance when the metrics used for segmentation and classification tasks are compared. For instance, the mean Jaccard and mean Dice (segmentation) shown in Table 3D and 3F do not show the changes in performance shown in the AUC (classification). However, the mean TV metric (0.88, 0.12) shows a consistent change in performance when compared with the binary classification metrics.

Table 3:

Summary of (A) segmentation and (B) binary classification metrics collected using nnU-Net models developed in approaches I-IV.

Task	(A) Segmentation
plane spacing, mm (Approach)	TV (1.00, 1.00) (mean ±SD)	TV (0.88, 0.12) (mean ±SD)	TV (0.50, 0.50) (mean ±SD)
A. 0.1 (IV)	0.52 ±0.11	0.80 ±0.05	0.67 ±0.10
B. 0.1 (I)	0.51 ±0.12	0.75 ±0.09	0.77 ±0.10
C. 0.2 (I)	0.41 ±0.14	0.75 ±0.08	0.57 ±0.15
D. 0.5 (I)	0.34 ±0.11	0.63 ±0.06	0.50 ±0.13
E. 0.2 (II)	0.49 ±0.11	0.71 ±0.09	0.65 ±0.10
F. 0.5 (II)	0.35 ±0.13	0.54 ±0.14	0.50 ±0.14
G. 0.2 (III)	0.48 ±0.11	0.74 ±0.09	0.64 ±0.10
H. 0.5 (III)	0.28 ±0.04	0.44 ±0.07	0.44 ±0.05

Task	(B) Binary Classification
plane spacing, mm (Approach)	%TPR (mean ±SD)	%TNR (mean ±SD)	%EFF (mean ±SD)	AUC (conc ±SD)
A. 0.1 (IV)	0.85 ±0.04	0.92 ±0.01	0.90 ±0.02	0.92 ±0.02
B. 0.1 (I)	0.78 ±0.09	0.93 ±0.02	0.87 ±0.03	0.88 ±0.04
C. 0.2 (I)	0.87 ±0.04	0.85 ±0.02	0.87 ±0.02	0.88 ±0.02
D. 0.5 (I)	0.71 ±0.07	0.86 ±0.03	0.80 ±0.02	0.81 ±0.03
E. 0.2 (II)	0.74 ±0.09	0.93 ±0.02	0.86 ±0.02	0.86 ±0.06
F. 0.5 (II)	0.56 ±0.14	0.91 ±0.03	0.78 ±0.03	0.74 ±0.09
G. 0.2 (III)	0.78 ±0.09	0.91 ±0.03	0.86 ±0.02	0.88 ±0.05
H. 0.5 (III)	0.44 ±0.09	0.94 ±0.01	0.75 ±0.04	0.63 ±0.05

Finally, examples of segmentation using the nnU-Net model developed in approach IV are shown in Figs. 9b and 9e. We can observe that most errors (FP and FN) are in voxels located where the model is the least confident (Figs. 9c and 9f), particularly in regions where there is a transition between glandular-adipose tissue (i.e., boundaries).

Figure 9:

(a-d) Examples of reconstructed DBT images, and (b-e) respective volumetric colormap and (c-f) confidence map (softmax) resulted from the nnU-Net segmentation (approach IV).

4. Discussion

Many machine learning algorithms have been developed and used for breast cancer detection [18] and for cancer risk estimates [19]. However, for most systems there is insufficient evidence to predict algorithm efficacy in large-scale population-based studies. Most AI research is conducted using small sets of retrospective data, based on selective image acquisition at a single site with a single vendor’s imaging products, and is presented with minimal detail regarding the development of the AI model [20]. We posited that DeepVCTs can help build a set of evidence-based strategies to elucidate limitations in studies such as biases in training or test data, and misinterpretation of the clinical outcomes. DeepVCTs provide full control of patient and imaging acquisition parameters, allowing for the evaluation of independent variables (e.g., plane spacing or breast density) which can affect the performance of machine learning models [12]. The following observations were made regarding the nnU-Net models when tested with DeepVCTs.

4.1. Study Results:

Plane spacing affects segmentation accuracy, and hence VBD; the nnU-Net performance can be improved by using input DBT images with smaller plane spacing. Compared with the high in-plane spatial resolution of DBT, z-resolution is poor [21]. This is used to justify reconstructing images with coarse plane spacing. Conversely, from analysis of oblique reconstructions, it is known that DBT images are able to capture fine structure with minute z-spacing differences [22]. Fig. 3 shows how segmentation performance is affected by plane spacing. As the plane spacing decreases from 0.5 mm to 0.1 mm, the TV values increase, and overall performance is improved. This trend may be attributed in part to partial volumes of the glandular signal being averaged into predominantly adipose-signal regions within the reconstruction. In addition, limited-angle tomographic x-ray techniques are prone to cone-beam and out-of-plane reconstruction artifacts [23]. These artifacts impede breast tissue segmentation or classification tasks.

Breast density affects accuracy in VBD estimation; the nnU-Net performance improved significantly (Kruskal-Wallis, P<0.001) using DBT images acquired on phantoms simulating denser breasts. However, the improvement in performance should be carefully analyzed, since the training and test sets contain imbalanced data. Without balancing the labels in the dataset, the learning process tends to converge to a point that is strongly biased towards the class with the majority labels (i.e., non-fibroglandular labels or negative class). We originally used a nnU-Net loss function (cross entropy with sum of Dice) that did not handle imbalanced data to converge the models for the optimal solution, resulting in extremely high %TNR and %EFF. To resolve the imbalanced data in the segmentation task, an asymmetric evaluation metric along with asymmetric loss function are required to penalize segmentation errors (FP and FN) differently. In this study, TV was chosen to handle the imbalanced data (α > β). Preliminary experiments showed that TV did not improve with breast density (Kruskal-Wallis, P=0.2) when α and β is determined on an individual basis. Thus, the observation that the nnU-Net performance is improved with denser breasts is biased, especially for nnU-Net models trained using DBT reconstructions with larger plane spacing.

Testing is sensitive to z-resolution (plane spacing); the nnU-Net performance slightly improved the segmentation metrics when the model trained with 0.1 mm plane spacing was tested with data having 0.2 mm plane spacing. However, the overall performance did not change substantially (ΔAUC=0.02). Testing the reference model (0.1 mm plane spacing) with 0.5 mm plane spacing reduces significantly the overall performance (ΔAUC=−0.07). Thus, z-resolution of images used for training and testing should be consistent or exhibit only small differences in z-resolution for the best performance of nnU-Net models. This is important to consider when learning systems are intended for use with a variety of DBT models in which plane spacing may differ.

Fine-tuning is sensitive to plane spacing; the nnU-Net performance changed when the models (for 0.2 and 0.5mm plane spacing) were fine-tuned based on a model pre-trained with a smaller reconstructed voxel size (0.1 mm plane spacing). For a fine-tuned model trained with 0.2 mm plane spacing, the nnU-Net improved significantly, especially for images that simulated less dense phantoms (39% increase in Dice, P<0.001). In the US population, most patients who participate in our screening mammography have large and low-density breasts (<15% in breast density) [24]. A fine-tuning approach may be worth considering for screening populations that consist of predominantly adipose breasts. However, low-density breasts are usually associated with women at low-risk of breast cancer. Fine-tuning models for this specific population may not be necessary. Based on our results, the fine-tuning approach was only beneficial when the differences in plane spacing for the training volumes were not large.

The best-performing method is reproducible; the nnU-Net model shows consistent results when trained with a greater number of DBT images simulated with a reconstructed voxel size of 0.1 mm³. The model is efficient and accurate (Table 3), achieving mean TV (0.88, 0.12) values of 0.80 and an AUC value of 0.92. Most segmentation errors were observed in regions near breast tissue boundaries and for compartments with admixtures of dense and adipose tissue (Fig. 9).

4.2. Study Implications:

DeepVCTs can play a key role in explainable-AI research, helping developers to understand the effect of changes in the input data on the model’s output. However, there is an inherent tension between the benefits of Deep-VCT methods and the costs associated with using computer simulations for clinically relevant tasks, such as medical image segmentation. The transition from simulation-based to patient-based methods is certainly not easy. However, this transition is not a problem unique to breast imaging research. There are many instances in science in which simulations may be the only avenue for researchers to advance a field, from evaluating the impact of various climate change mitigation strategies to modeling nuclear explosives while adhering to the test ban.

There are a variety of methods that can be used to validate simulation-based models [25]. The most common validation method uses results obtained from other well-established software. However, at best this seeks to demonstrate concordance, rather than measure quantitative accuracy. In the current work, this strategy would require either the comparison of the breast density software against other comparable code, or the comparison to other ground-truth VBD data. The former was judged invalid as our breast density code was trained solely on simulation data while other such software is trained solely on clinical data; the latter was not pursued as these data are not widely available today. That said, breast anatomy simulations are simplified representations of real mammary parenchymal. The complex and fine structures found in the mammary parenchymal are not simulated perfectly, which could alter the segmentation performance of the available software. Additionally, the simulation parameters were constrained to create phantoms with the same dimensions to simplify the overall segmentation task. This may limit the generalizability of the study results to other AI software; however, the deepVCT testing method is still applicable.

Another validation method is based on parameter variability [also known as sensitivity analysis] [25]. This method consists of varying the parameter values of the input data to determine the effect of these variations on the output of the simulation-based models. This controlled input data is used to evaluate changes in magnitudes of outputs quantitatively and qualitatively. While it is not possible to create ground-truth VBD data using human images, it is possible to repeat and reproduce our studies with other breast models, having different simulation strategies to assess the robustness of this method. In fact, deepVCT methods would allow precise control of the input parameters in a sensitivity analysis study.

Alternatively, the ground truth could be extracted from other imaging methods, such as magnetic resonance images (MRI) to measure to accuracy of estimating VBD in DBT. Breast MRI is a three-dimensional imaging modality with superior tissue contrast when compared to DBT. However, this supplemental modality is recommended only for high-risk women, generally with dense breasts. Additionally, there is no easy way to match the corresponding volumes of breast tissue in MRI and DBT images. That said, it would be beneficial to investigate the use of nnU-Net models trained using human data and clinical DBT images in the future.

Training and/or testing machine learning algorithms with datasets which combine patient and simulated images have been proposed [26]. That study showed that data augmentation with synthetic images did not result in increased performance of their AI algorithm. In fact, the sensitivity of the algorithm was reduced by inclusion of a large number of synthetic images [26]. Nuances in the image properties [e.g., power-spectrum [27], non-gaussian properties [28], acquisition noise [29], etc.] between real and simulated images can be detected algorithmically [30]. However, as we have shown, training and testing with currently available simulated data lead to a better understanding of how machine learning algorithms might work in clinical practice.

Previous VCT studies have focused on specific clinical tasks [5]; they have demonstrated that VCTs can represent a broad range of human imaging conditions using reliable and simple anatomical models. Thus, allowing the outcomes to be better understood, optimized, and reproduced. In prior work [31], we postulated that variations in reconstructed voxel size can significantly affect lesion detection in DBT. DBT acquisition geometries that use smaller reconstructed voxels (in-plane) improved the detection of clinically significant lesions, particularly microcalcifications. Based on the results presented in this study, the voxel size used for reconstructing DBT images can also affect the segmentation of breast tissue. We can potentially improve dense tissue segmentation for DBT by varying additional acquisition parameters such as source motion and detector element size [31].

VCTs have been shown to be beneficial for the optimization of new technologies prior to production and clinical testing, making the design process more cost-effective and rapid [5]. We would argue that DeepVCTs can play a similar role, but for development and optimization of AI algorithms, while also maintaining a high level of machine-learning performance (prediction accuracy) and human-learning interpretation (explainability). Such methods can help us to verify the appropriateness of an algorithm applied to clinically relevant tasks (e.g., image segmentation).

An application developed using DeepVCTs should not be considered a “final product”. Testing models fully developed with VCT-based data are not likely to succeed on real world domain (and vice-verse). Instead, the proposed methods should be incorporated as part of the development and optimization of an AI-based system. Drawing analogy to our work in VCT-based systems design, we would propose the following three steps: 1) using simulation data, evaluate a large variety of networks and algorithms; 2) select a limited number of the most accurate algorithms for training and testing with clinical data preferably with quantitative correlates; 3) repeat the testing of the best algorithm(s) with de novo data from multiple sites and vendors.

We have also investigated VCT methods to optimize and personalize acquisition geometries for improving breast density segmentation [12]. Custom DBT acquisition geometries that incorporate x-ray source motion in the posteroanterior direction can improve adipose-fibroglandular tissue classification. In a previous study [17], we demonstrated that binary classification (adipose or glandular) can increase d’ (sensitivity index) by up to 7%. These results demonstrate that x-ray source motion also affects breast tissue classification. We posit that acquisition geometry optimization can improve the input to deep-learning breast density segmentation algorithms.

The investigation of additional machine learning architectures trained for breast tissue segmentation should be considered, including the traditional U-Net. A preliminary study has shown that the U-Net architecture can also segment fibroglandular tissue using slice-wise methods [32]. Similarly, the best-performing U-Net models are usually trained and tested using the central portion of the DBT data as input. The prior results showed a mean Jaccard of 0.62 for fibroglandular segmentation with a positive correlation with breast density (Person’s coefficient of 0.60). The results presented here are consistent with this prior work and support the use of machine learning algorithms with data reconstructed with non-linear imaging modalities.

Reconstructing tomographic projections can be time-consuming; finer plane spacing increases the reconstruction time and increases the size of the DBT dataset. However, most machine learning applications do not require full-resolution images and volumes for training and testing [20]. We performed cropping and resizing operations during the reconstruction process to reduce the computational burden and accelerate the acquisition of DBT images; execution time and data storage are not critical factors in this work.

5. Conclusion

In this study, nnU-Net models were optimized using DeepVCT methods to segment or classify fibroglandular from non-fibroglandular tissue, and thus evaluate VBD using simulated DBT images. The performance of the models was evaluated using a large database (n=2,010) of synthetic x-ray images with full ground truth of the breast tissue. Four training approaches were considered in the study to: (I) select the optimal plane spacing, (II) evaluate the influence of plane spacing in VBD assessment, (III) measure performance changes from using a fine-tuning method, and (IV) refine and test the best-performing method with a greater number of volumetric images.

These approaches were designed to understand the nnU-Net outcomes and allow the optimization of DBT reconstruction parameters to improve VBD assessment. Plane spacing and distribution of fibroglandular tissue significantly affect the performance of the nnU-Net models. Smaller plane spacing improves the segmentation of fibroglandular from non-fibroglandular tissue, especially for breasts simulated with high breast density.

Some limitations exist in terms of the breast size, volume, and textural complexity of mammary parenchymal in the anatomical models. However, the anatomical models were intentionally constrained to simplify the segmentation task and optimize the plane spacing for volumetric segmentation of fibroglandular tissue.

Importantly, this study showed that VBD assessment requires the use of an asymmetric evaluation metric and loss function to handle imbalanced data. The Tversky index was demonstrated to be an appropriate metric for determining thresholds for α and β based on the distribution of breast density in a specific patient population. Although the Tversky index is less sensitive to breast density than Jaccard or Dice, the choice of α and β required careful analysis and interpretation of the nnU-Net outcomes.

More importantly, this study showed that DeepVCTs provide methods to investigate optimal imaging parameters to train machine learning algorithms for clinically relevant tasks. Various approaches can be designed and tested to help interpret the performance of these algorithms, providing VCT-based evidence to predict the efficacy of applications in medical imaging using large-scale simulation-based data.

For the next step of this work, we need to evaluate the effectiveness and practical role of DeepVCTs through domain adaptation. We plan to explore BI-RADS classification of specific patient populations matched with the VBD calculated using virtual phantoms. Customized models can be fine-tuned using a specific clinical dataset, weights can be calibrated, and ultimately, the domain of virtual models can be adapted (i.e., the effective transition from virtual to real world) to better segment breast density.

Highlights.

• DeepVCT provides simulation-based evidence to evaluate and optimize AI algorithms

• nnU-Net can be optimized to segment fibroglandular from adipose tissue

• Fraction of density affects volumetric breast density assessment

• Plane spacing also affects the volumetric breast density assessment

• Asymmetric evaluation metrics are necessary to handle imbalanced data

Biographies

Bruno Barufaldi is a Research Associate of Radiology at the University of Pennsylvania. His research foci (AI and VCTs) are driven by the interest in applying state-of-the-art computer methods to Radiology. Bruno is a junior AAPM member and has received the AAPM Research Seed Funding Grant to support this work.

Jordy Vieira has received his B.Sc. at the Federal University of Paraiba (UFPB) and member of the Artificial Intelligence Applications Laboratory (ARIA). His research interest is in the development of AI methods to medical imaging applications.

Telmo Silva Filho is an Adjunct Professor at UFPB, where he is one of the coordinators of the Artificial Intelligence Applications Laboratory (ARIA). His research focuses on machine learning evaluation, probability calibration, optimization, and data mining with complex data types.

Thaís Gaudêncio is an Adjunct Professor at the UFPB. She also coordinates the Artificial Intelligence Applications Laboratory (ARIA). Her research is focused on the development of machine learning approaches in different areas, including bioinformatics and medical imaging.

Yuri Malheiros is an Adjunct Professor at the UFPB Informatics Center, where he teaches artificial intelligence and data science courses. He also coordinates the Artificial Intelligence Applications Laboratory (ARIA). His research foci are on machine learning, neural networks, computer vision, medical imaging, natural language processing, and data analysis.

Trevor Vent has received his Ph.D in the department of Bioengineering at the University of Pennsylvania. His research consists of the development and evaluation of novel x-ray imaging systems. His research interest is in the development of digital x-ray imaging methods and analysis, in particular for 3D breast imaging systems.

Aimilia Gastounioti is an Assistant Professor of Radiology in the Mallinckrodt Institute of Radiology at the Washington University School of Medicine in St. Louis. She is interested in artificial intelligence with a primary focus on cancer imaging phenotypes related to risk prediction. She has co-authored 22 journal articles, 3 book chapters, and 50 conference publications. Aimilia is an Associate AACR member.

Andrew Maidment is Associate Professor of Radiology at the University of Pennsylvania and chief of Physics for Radiology in the Hospital of Pennsylvania Health System. His expertise is development of 3D breast x-ray imaging systems and metrics of image quality. Andrew is a fellow of the AAPM and the SBI.