Test-retest repeatability of a deep learning architecture in detecting and segmenting clinically significant prostate cancer on apparent diffusion coefficient (ADC) maps

Abstract

Objectives

To evaluate short-term test-retest repeatability of a deep learning architecture (U-Net) in slice- and lesion-level detection and segmentation of clinically significant prostate cancer (csPCa: Gleason grade group > 1) using diffusion-weighted imaging fitted with monoexponential function, ADC_m.

Methods

One hundred twelve patients with prostate cancer (PCa) underwent 2 prostate MRI examinations on the same day. PCa areas were annotated using whole mount prostatectomy sections. Two U-Net-based convolutional neural networks were trained on three different ADC_m b value settings for (a) slice- and (b) lesion-level detection and (c) segmentation of csPCa. Short-term test-retest repeatability was estimated using intra-class correlation coefficient (ICC(3,1)), proportionate agreement, and dice similarity coefficient (DSC). A 3-fold cross-validation was performed on training set (N = 78 patients) and evaluated for performance and repeatability on testing data (N = 34 patients).

Results

For the three ADC_m b value settings, repeatability of mean ADC_m of csPCa lesions was ICC(3,1) = 0.86–0.98. Two CNNs with U-Net-based architecture demonstrated ICC(3,1) in the range of 0.80–0.83, agreement of 66–72%, and DSC of 0.68–0.72 for slice- and lesion-level detection and segmentation of csPCa. Bland-Altman plots suggest that there is no systematic bias in agreement between inter-scan ground truth segmentation repeatability and segmentation repeatability of the networks.

Conclusions

For the three ADC_m b value settings, two CNNs with U-Net-based architecture were repeatable for the problem of detection of csPCa at the slice-level. The network repeatability in segmenting csPCa lesions is affected by inter-scan variability and ground truth segmentation repeatability and may thus improve with better inter-scan reproducibility.

Introduction

In recent years, deep learning (DL)-based convolutional neural networks (CNNs) have gained tremendous attention in medical imaging especially using MRI for various applications such as organ segmentation [1, 2], cancer detection and diagnosis [3–5], and characterization [6, 7]. However, MRI images might be influenced by different sources of noise variations such as scanner acquisition noise [8] and motion artifacts [9]. These variations not only affect the visual quality of an image but may also interfere with downstream analysis of MRI images [10].

Prostate Imaging-Reporting and Data System (PI-RADS) has standardized the diagnosis of PCa using MRI and has shown to be effective in characterizing PCa [11]. However, it has been found that PI-RADS-based scoring has only moderate to good inter- and intra-reader variability [12, 13]. Recently, much attention has been drawn to machine learning (ML) models built using radiomics-derived representations on MRI for PCa detection and characterization [14, 15]. However, the sources of variation in MRI acquisition and reconstruction [16–19] have shown to influence these representations [10]. Therefore, lately, there has been an increasing interest in applying test-retest analysis to rank order radiomics features based on their repeatability and discriminability, and build ML classifiers based on most stable features [20, 21]. In contrast, although several DL approaches have been presented for PCa segmentation [22, 23], detection [24, 25], and characterization [26, 27], to the best of our knowledge, none of them has been explicitly evaluated in the context of test-retest repeatability.

A unique test-retest data of monoexponential fitted prostate apparent diffusion coefficient (ADC_m) maps was used in this study. Two MRI scans were taken approximately 15 min apart for each patient. We used only ADC_m maps and not bi-parametric MRI (T2W MRI and ADC_m) since T2W MRI were not available for two time points. The evaluation of repeatability of DL models trained on ADC_m maps taken at such short time span allows us to evaluate stability of DL models against variations with respect to acquisition of images. Additionally, it is also safe to assume that changes in tissue biology are negligible over such a short time span.

Due to increasing popularity of the deep learning architecture, U-Net [26, 28–30] in segmentation, detection, and classification tasks, we use U-Net-based architecture in our study. U-Net [31] is a fully convolutional network designed for semantic segmentation tasks with two components, an encoder and a decoder. The U-Net decoder combines both local information and the contextual information which is required to predict a good segmentation map. Additionally, since there is no dense layer involved in the architecture, images of different sizes can be given as input.

Therefore, in this study, we evaluate test-retest repeatability of convolutional neural networks using a U-Net-based architecture on three different ADC_m b value settings for (a) slice-and (b) lesion-level detection and (c) segmentation of clinically significant prostate cancer (csPCa: Gleason grade group (GGG) > 1). A 3-fold cross-validation was performed on training set (N = 78 patients) and evaluated for performance and repeatability on testing data (N = 34 patients).

Materials and methods

MR imaging and data

This retrospective study was compliant with Health Insurance Portability and Accountability Act (HIPAA) and approved by institutional review board. All patients, N = 115, with diagnosed PCa signed informed consent and underwent prostate MRI before robotic-assisted laparoscopic prostatectomy between March 2013 and February 2016 [17, 32]. All patients underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart following repositioning on MR scanner table [19, 32]. The scans were performed using a 3T MR scanner (3 Tesla Philips Ingenuity PET/MR). DWI was performed using a single-shot spin echo–based sequence with monopolar diffusion gradient and an echo-planar read out. Summary acquisition parameters are provided in Table E1 (supplementary), while detailed acquisition protocol was described previously [17]. We evaluated ADC_m maps at the voxel level with DWI data for three different b value settings: (a) four b values in the range of 0–900 s/mm², B_4b900 (0, 300, 500, 900 s/mm²) [33], (b) four b value distribution which was previously suggested as being a potentially optimal distribution, B_4b2000 (0, 900, 1100, 2000 s/mm²) [16], and (c) two b values in the range of 0–1300 s/mm², B_2b1300 (0 and 1300 s/mm²). The third option was considered to evaluate a setting with minimal number of b values for signal-to-noise-ratio and contrast trade-off in the context of CNN-based classifications [16]. Three patients were excluded due to the presence of severe motion (n = 1) and/or susceptibility artifacts (n = 2). Figure E1 (supplementary) shows the flow chart of inclusion/exclusion criteria of the patients and splitting of the data into training and test sets. The data splits were the same as reported by Merisaari et al [32].

Prostate capsule and lesion segmentation

A radiologist with 9 years of prostate MRI experience in consensus with a board-certified staff urogenital pathologist (10 years of experience in urogenital pathology) delineated the prostate capsule and cancerous regions on DWI with whole mounts prostatectomy sections as ground truth using the Carimas (version 2.9) software. Demographic information (age, PSA), lesion distribution in different zones (peripheral zone, central/transitional zone), GGG categories (1–5), and the distribution of csPCa and non-csPCa (GGG = 1, benign) patches is shown in Table 1.

Table 1.

Demographic and lesion characteristics of N = 112 patients. All patients underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart following repositioning on MR scanner table

n = patients	112
Age, medium (IQR), years	64.5 (60–68)
PSA, medium (IQR), ng/ml	9.3 (6.6–12.25)
n = lesions	170
Lesion zones, n (%)
Peripheral zone (PZ)	124 (72.94%)
Transitional zone (TZ)/central zone (cz)	46 (27.06%)
Gleason grade group, n (%)
1 (Gleason score 3 + 3)	28 (16.47%)
2(Gleason score 3 + 4)	77 (45.29%)
3 (Gleason score 4 + 3)	31 (18.23%)
4 (Gleason score 4 + 4.3 + 5.5 + 3	17 (10%)
5 (Gleason score 4 + 5.5 + 4)	17 (10%)
n = 2D axial images, n (%)	SA = 1261		SB = 1253
	Atrain = 875 (69.4%)	Atest = 386 (30.6%)	Btrain = 870 (69.4)	Btest = 383 (30.6%)
non-csPCa(GGG = 1, benign)	594 (47.1%)	265 (21.0%)	583 (46.5%)	262 (20.9%)
csPCa (GGG ≤ 1)	281(22.3%)	121 (9.6%)	287 (22.9%)	121 (9.7%)

Axial images refer to each 2D axial slice of the whole prostate volume. Each image is considered as positive if a clinically significant prostate cancer (csPCa) lesion exists on the image or is considered as negative

IQR inter-quantile range

U-Net architecture

U-Net [31] is a fully convolutional network designed for semantic segmentation tasks with two components, a descending encoder path and an ascending decoding path. The modified U-Net consists of 5 encoder blocks and 5 decoder blocks. Each of the encoder blocks and decoder blocks consists of two convolutional layers except for the last decoder block with only one convolutional layer accounting for a total of 19 convolutional layers. The decoder and the encoder paths consist of batch normalization layers and drop-out layers in between the convolutional layers, with max pooling in the decoder blocks and up-sampling in the encoder blocks. The model consists of a total of 7,852,002 trainable parameters. Figure E2(b) shows the architectural diagram of U-Net.

U-Net training

The details of data preprocessing and data augmentation are described in the supplementary section (S1). We define the problem of slice-wise detection of clinically significant prostate cancer (csPCa) regions as a classification task. Each slice with prostate voxel was defined either as containing csPCa or non-csPCa (Gleason grade grouping (GGG) = 1/benign). We defined the ground truth labels by considering each extracted patch from ADC_m, with the presence of csPCa lesion (GGG > 1) as a positive exemplar, all others were deemed as negative. We used a modified network architecture (U-Net_m) for the classification task. The network architecture for U-Net_m is shown in Fig. E2(a).

For csPCa lesion detection and segmentation, the manually annotated lesion delineations done with whole mount prostatectomy sections as reference were used as ground truth. A transfer learning strategy was used to initialize the encoder weights of the U-Net by transference of weights from the model U-Net_m trained for detection of csPCa at the slice-level. The network architecture of U-Net is shown in Fig. E2(b). We used the segmentation maps outputted by the networks to evaluate lesion detection. We defined a lesion as being detected if ≥ 0.2 DSC overlap existed between the network segmentation map and the ground truth delineation of that corresponding lesion. Figure 1 depicts the training of U-Net_m, U-Net. Other related implementation details are provided in the supplementary section (S2). For the source code for the network training and evaluation of repeatability, see https://github.com/amogh3892/Test-retest-repeatability-of-U-Net-in-detecting-segmenting-clinically-significant-prostate-cancer.

Fig. 1.

a Training process of a modified U-Net (U-Net_m) for detecting 2D patches with presence of clinically significant prostate cancer (csPCa). The input to the network is a 2D patch extracted and cropped to the prostate boundary using the manual segmentation drawn over the prostate capsule. ADC_m patches with the presence of csPCa lesion (GGG > 1) were considered as positive samples and others were marked negative. b Training process of U-Net for segmentation of csPCa regions. csPCa regions delineated by an experience radiologist was used as ground truth for lesion segmentation

Evaluation metrics and statistical analysis

Area under the receiver operator characteristic curve (AUCs), sensitivity and positive predictive value (PPV), and dice similarity coefficient (DSC) were used to evaluate the performance of slice- and lesion-level detection and segmentation of csPCa on ADC_m respectively. Similarly, intra-class correlation coefficient (ICC(3,1)), proportionate agreement, and DSC were used to evaluate repeatability of the networks for slice- and lesion-level detection and segmentation of csPCa on ADC_m respectively. Ninety-five percent confidence intervals were calculated wherever necessary and cross-validation results were reported as mean ± standard deviation. Further details and definitions of the performance metrics are presented in the supplementary section (S3).

Experiment 1: Repeatability of U-Net_m in slice-level detection of clinically significant prostate cancer on prostate apparent diffusion coefficient maps

For all three b value settings (B_4b900, B_4b2000, and B_2b1300), U-Net_m was trained for slice-level detection of csPCa regions with 3-fold cross-validation setting on the training sets A_train (networks C_A1, C_A2, and C_A3 trained on the three folds of A_train) and B_train (networks C_B1, C_B2, and C_B3 trained on the three folds of B_train), and was evaluated for performance in terms of AUC. The ensemble of classifiers from 3-fold cross-validation C_A (average predictions from C_A1, C_A2, and C_A3) and C_B (average predictions from C_B1, C_B2, and C_B3) was used to evaluate the (a) performance in terms of AUCs and (b) repeatability of the network predictions in terms of ICCs on the test set S_test (A_test + B_test). Additionally, other performance metrics such as accuracy, sensitivity, and specificity were reported by calculating the optimal cutoff through Youden index [34]. We combined the test sets A_test and B_test since S_A and S_B were not co-registered with respect to each other and registration of the scans would lead to additional registration artifacts. Figure 2 shows the overall experimental design for evaluating the repeatability of the network outputs.

Fig. 2.

Experimental design for evaluating repeatability of CNNs for (a) slice-level classification of clinically significant prostate cancer (csPCa: Gleason grade group (GGG) > 1) and non-csPCa regions and (b) lesion-level detection and segmentation of csPCa regions. N = 112 patients scheduled for prostatectomy underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart. The scans, S_A and S_B, were divided into training set (A_train and B_train), N = 78, and test set (A_test and B_test) N = 34. Two different instances N_A and N_B were trained on scans A_train and B_train, respectively, and evaluated on a combined test set, S_test (A_test + B_test). The outputs P₁ and P₂ by N_A and N_B, respectively, on S_test are used to calculate repeatability of the CNNs

Experiment 2: Repeatability of U-Net in segmentation and detection of clinically significant prostate cancer lesions on prostate apparent diffusion coefficient maps

For all three b value settings (B_4b900, B_4b2000, and B_2b1300), U-Net was trained with a 3-fold cross-validation setting on the training set, A_train (networks D_A1, D_A2, and D_A3 trained on the three folds of A_train) and B_train (networks D_B1, D_B2, and D_B3 trained on the three folds of B_train) for segmenting csPCa lesions on ADC_m maps. The ensemble of segmentation networks from 3-fold cross-validation D_A and D_B (D_A: Logical “OR” of segmentations from D_A1, D_A2, D_A3; and D_B: Logical “OR” of segmentations from D_B1, D_B2, and D_B3) was used to obtain final segmentation maps on the test set, S_test. We post-process the output segmentations in order to remove some false positives. The details of post-processing of the lesion segmentations are provided in the supplementary (S4).

We use the output segmentation maps to assess csPCa lesion detection performance by evaluating the sensitivity and positive predictive value of the networks D_A and D_B. The repeatability of csPCa lesion detection was assessed by evaluating the proportionate agreement between the networks D_A and D_B.

We further evaluate (a) segmentation performance and (b) repeatability of segmentations in terms of DSC for the detected lesions on S_test. We also assess the repeatability of network segmented volumes and mean ADC_m value in the lesion with respect to ICC and compare them with ground truth delineations.

Agreement of inter-scan ground truth segmentation repeatability and U-Net’s segmentation repeatability in segmenting csPCa lesions

We co-registered the scans A_test and B_test and chose only the csPCa lesions that are detected on both A_test and B_test for the analysis. The details of registration are provided in the supplementary section (S5). The agreement between repeatability of ground truth delineations and repeatability of segmentation maps obtained by D_A and D_B was illustrated using Bland-Altman plots. No systematic bias as a function of the evaluated signal was found to be present in the Bland-Altman plots.

Results

Experiment 1: Repeatability of U-Net_m in slice-level detection of clinically significant prostate cancer on prostate apparent diffusion coefficient maps

Table 2 shows the performance metrics of slice-level detection of csPCa on cross-validation and testing cohorts for networks trained on A_train and B_train for three different b value settings (B_4b900, B_4b2000, and B_2b1300). For all the b value settings, we can observe that the networks yielded an AUC of 0.81–0.85 for the cross-validation on A_train and B_train. The ensemble of classifiers from 3-fold cross-validation, C_A and C_B, resulted in an AUC of 0.78–0.85 in S_test. A DeLong test [35] between the cross-validation AUCs and AUCs on S_test did not show significant difference between the results obtained (p > 0.11). Figure 3 shows the receiver operator characteristic (ROC) curves of the networks for slice-level detection of csPCa on prostate ADC_m maps on B_4b900 for cross-validation on A_train and B_train and evaluation on S_test.

Table 2.

Cross-validation and hold-out test set performance metrics (AUC, accuracy, sensitivity, specificity) of networks C_A (trained on A_train) and C_B (trained on B_train) in detecting clinically significant prostate cancer slices, evaluated on three different b value settings (a) B_4b900 (0, 300, 500, 900 s/mm²), (b) B_4b2000 (0, 900, 1100, 2000 s/mm²), and (c) B_2b1300 (0 and 1300 s/mm²). N = 112 patients scheduled for prostatectomy underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart. The scans, S_A and S_B, were divided into training set (A_train and B_train), N = 78, and test set (A_test and B_test) N = 34

b value setting	Cross-validation	AUC (95% CI)	Accuracy	Sensitivity	Specificity	p value (AUC)
B4b900	CA	0.81 (0.79–0.83)	74%	86%	69%	p = 0.64
	CA	0.82 (0.79–0.84)	76%	70%	79%
B4b2000	CA	0.82 (0.79–0.84)	78%	69%	82%	p = 0.18
	CB	0.80 (0.78–0.83)	71%	77%	69%
B2b1300	CA	0.85 (0.83–0.87)	78%	79%	77%	p = 0.39
	CB	0.84 (0.80–0.86)	76%	76%	76%
b value setting	Hold-out test set (Stest)	AUC (95% CI)	Accuracy	Sensitivity	Specificity	p value (AUC)
B4b900	CA	0.78 (0.75–0.81)	70%	88%	69%	p = 0.58
	CB	0.79 (0.76–0.82)	76%	75%	76%
B4b2000	CA	0.80 (0.77–0.84)	76%	81%	74%	p = 0.11
	CB	0.78 (0.75–0.82)	74%	74%	73%
B2b1300	CA	0.84 (0.81–0.87)	79%	72%	82%	p = 0.58
	CB	0.85 (0.83–0.88)	80%	77%	81%

Fig. 3.

Receiver operating characteristic (ROC) curves for U-Net_m in slice-level detection of clinically significant prostate cancer regions on apparent diffusion coefficient ADC_m (B_4b900: b values (0, 300, 500, 900 s/mm²) maps. The test-retest dataset (S_A and S_B) of 112 patients were divided into training set (A_train and B_train) and test set (A_test and B_test). a 3-fold cross-validation was performed on the training sets A_train (C_A1, C_A2, C_A3 trained on the three folds of A_train) and B_train (C_B1, C_B2, C_B3 trained on the three folds B_train). a Mean ROC curves, A_m and B_m, for 3-fold cross-validation on A_train and B_train respectively. b ROC curves for ensemble classifiers C_A (average probabilities of C_A1, C_A2, C_A3) and C_B (average probabilities of C_B1, C_B2, C_B3) on hold-out test set, S_test (A_test + B_test)

The probability scores of the ensemble classifiers C_A and C_B are used to evaluate repeatability on S_test. The U-Net_m yielded an ICC of 0.83, 95% CI (0.80–0.85); 0.80, 95% CI (0.77–0.83); and 0.83, 95% CI (0.80–0.85) on B_4b900, B_4b2000, and B_2b1300, respectively, in detecting clinically significant prostate cancer regions on ADC maps.

Experiment 2: Repeatability of U-Net in segmentation and detection of clinically significant prostate cancer lesions on prostate apparent diffusion coefficient maps

Table 3 depicts the csPCa lesion detection performance on the cross-validation set and S_test for different b value settings (B_4b900, B_4b2000, and B_2b1300). The networks resulted in a sensitivity of 55–60% and a PPV of 51–53% on the cross-validation set. The networks D_A and D_B had proportionate agreement of 66–72% in detecting csPCa lesions on S_test and the corresponding sensitivity and PPV was in the range 63–66% and 45–57% respectively.

Table 3.

Cross-validation and hold-out test set performance metrics (true positives, false negatives, false positives, sensitivity, and positive predictive value) of networks D_A (trained on A_train) and D_B (trained on B_train) in detecting clinically significant prostate cancer lesions evaluated on three different b value settings (a) B_4b900 (0, 300, 500, 900 s/mm²), (b) B_4b2000 (0, 900, 1100, 2000 s/mm²), and (c) B_2b1300 (0 and 1300 s/mm²). N = 112 patients scheduled for prostatectomy underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart. The scans, S_A and S_B, were divided into training set (A_train and B_train), N = 78, and test set (A_test and B_test) N = 34

b value setting	Cross-validation	True positives	False negatives	False positives	Sensitivity	Positive predictive value
B4b900	DA	64	41	61	61%	51%
	DB	68	37	63	65%	52%
B4b2000	DA	66	39	60	63%	52%
	DB	67	38	58	64%	54%
B2b1300	DA	63	42	57	60%	53%
	DB	58	47	56	55%	51%
b value setting	Hold-out test set (Stest)	True positives	False negatives	False positives	Sensitivity	Positive predictive value
B4b900	DA	52	27	52	66%	50%
	DB	51	28	62	65%	45%
B4b2000	DA	50	29	46	63%	47%
	DB	52	27	56	66%	48%
B2b1300	DA	51	28	39	65%	53%
	DB	51	28	45	65%	53%
b value setting	Repeatability (proportionate agreement)		Agreement (%)		Disagreement (%)
B4b900	DA and DB		90 (66%)		46 (34%)
B4b2000	DA and DB		83 (72%)		32 (28%)
B2b1300	DA and DB		79 (69%)		35 (31%)

Table 4 illustrates the csPCa lesion segmentation performance of the networks on detected csPCa lesions for B_4b900, B_4b2000, and B_2b1300. The networks D_A and D_B resulted in DSC of 0.47–0.54 on the cross-validation set and 0.58–0.64 on S_test respectively. The DSC between the network segmentations (repeatability) was in the range 0.68–0.72.

Table 4.

Cross-validation and hold-out test set performance of by networks D_A (trained on A_train) and D_B (trained on B_train) in segmenting detected clinically significant prostate cancer lesions evaluated on three different b value settings (a) B_4b900 (0, 300, 500, 900 s/mm²), (b) B_4b2000 (0, 900, 1100, 2000 s/mm²), and (c) B_2b1300 (0 and 1300 s/mm²). N = 112 patients scheduled for prostatectomy underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart. The scans, S_A and S_B, were divided into training set (A_train and B_train), N = 78, and test set (A_test and B_test) N = 34

Dice similarity coefficient (DSC)	B4b900		B4b2000		B2b1300
	DA	DB	DA	DB	DA	DB
Cross-validation	0.47 ± 0.20	0.48 ± 0.19	0.50 ± 0.21	0.54 ± 0.19	0.49 ± 0.22	0.49 ± 0.20
Hold-out test set (Stest)	0.61 ± 0.11	0.60 ± 0.12	0.60 ± 0.18	0.64 ± 0.16	0.58 ± 0.22	0.60 ± 0.22
Repeatability (DSC) DA and DB	0.68 ± 0.21		0.72 ± 0.22		0.70 ± 0.20

Figure 4 shows the overlaid segmentation maps on the ADC_m (B_4b900) images with DSC reported in 3D. We can observe that, although some of the lesions are poorly segmented by the networks, the repeatability in terms of DSC between the networks is high.

Fig. 4.

Clinically significant prostate cancer (csPCa) lesion segmentation maps on ADC_m (B_4b900: b values (0, 300, 500, 900 s/mm²) of networks D_A (trained on A_train) and D_B (trained on B_train). a Full field of view of monoexponential fitted prostate apparent diffusion coefficient (ADC_m) maps. b Overlaid ground truth delineation (GT) and segmentation maps of D_A and D_B. c Dice similarity coefficient (DSC) between GT and D_A. d DSC between GT and D_B. e DSC between D_A and D_B. csPCa lesions (1,2) have high DSC overlap both between the ground truth and between the networks. csPCa lesions (3,4) have high DSC overlap between the networks even though they poorly segment the lesions. All DSC reported are evaluated in 3D

Table 5 shows the repeatability of volume measurement and mean ADC_m values of ground truth delineations and U-Net-based segmentations on S_test for different b value settings (B_4b900, B_4b2000, and B_2b1300). U-Net obtained an ICC score of 0.89–0.92 and 0.84–0.87 for volume and mean ADC_m value, respectively, while compared with ground truth delineations with an ICC score of 0.92 for volume and 0.86–0.98 for mean ADC_m.

Table 5.

Repeatability of volume, mean apparent diffusion coefficient (ADC) value of clinically significant prostate cancer (csPCa) lesion in terms of intra-class correlation coefficient (ICC(3,1)) for ground truth delineations, and segmentation maps obtained by U-Net in the hold-out test set (S_test) for ADC_m of three different b value settings (a) B_4b900 (0, 300, 500, 900 s/mm²), (b) B_4b2000 (0, 900, 1100, 2000 s/mm²), and (c) B_2b1300 (0 and 1300 s/mm²). N = 112 patients scheduled for prostatectomy underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart. The scans, S_A and S_B, were divided into training set (A_train and B_train), N = 78, and test set (A_test and B_test) N = 34

Repeatability ICC (95% confidence interval)	B4b900		B4b2000		B2b1300
	Volume	Mean ADC value	Volume	Mean ADC value	Volume	Mean ADC value
Ground truth delineations	0.92 (0.86–0.96)	0.86 (0.77–92)	0.92 (0.86–0.96)	0.89 (0.81–0.93)	0.92 (0.86–0.96)	0.98 (0.97–0.99)
U-Net segmentations (DA and DB)	0.89 (0.79–0.94)	0.87 (0.69–0.95)	0.92 (0.84–0.96)	0.87 (0.71–0.95)	0.89 (0.79–0.94)	0.84 (0.63–0.93)

Agreement of inter-scan ground truth segmentation repeatability and U-Net’s segmentation repeatability in segmenting csPCa lesions

Figures 5 a and b show the agreement of repeatability of ground truth delineations and network segmentations using a Bland-Altman plot on B_4b900. The mean of repeatability between ground truth delineations and network segmentations in terms of DSC are plotted against the difference between ground truth delineations and network segmentation DSC. The plots suggest that the ground truth delineations are in moderate agreement with network-based segmentations in most of the cases, with a few outliers. We can also observe that repeatability of ground truth delineations is slightly better than network-based segmentation repeatability. The Bland-Altman plots for B_4b2000 and B_2b1300 expressed similar agreement between the ground truth and the network results; these are illustrated in the supplementary figure (Fig. E3).

Fig. 5.

Bland-Altman plots between variation of ground truth delineations (DSC) and variation between network segmentations (DSC) on ADC_m (B_4b900: b values (0, 300, 500, 900 s/mm²). N = 112 patients scheduled for prostatectomy underwent two prostate MR examinations (S_A and S_B) performed on the same day approximately 15 min apart. The scans, S_A and S_B, were divided into training set (A_train and B_train), N = 78, and test set (A_test and B_test) N = 34. U-Net architecture-based networks D_A and D_B trained on A_train and B_train respectively. a Bland-Altman plot between ground truth delineations and segmentation maps of D_A with B_test co-registered to A_test.. b Bland-Altman plot between ground truth delineations and segmentation maps of D_B with A_test co-registered to B_test

Discussion

In this study, we evaluated repeatability of U-Net for (a) slice-and (b) lesion-level detection and (c) segmentation of clinically significant prostate cancer (csPCa: Gleason grade group (GGG) > 1) on prostate apparent diffusion coefficient (ADC_m) maps with three different b value settings (B_4b900, B_4b2000, and B_2b1300). The U-Net-based architecture was found to be repeatable (ICC of 0.8–0.83) for slice-level detection of csPCa regions, and moderately repeatable in detecting (proportionate agreement of 66–72%) and segmenting (DSC of 0.68–0.72) csPCa lesions.

High predictive power from single time point with low test-retest repeatability might be misleading. While number of studies have looked at repeatability of radiomics features [32, 36, 37], relatively little work has been done in the context of deep learning (DL), specifically convolutional neural networks (CNNs) [38, 39]. To the best of our knowledge, none of the previous studies has analyzed repeatability of CNNs for detection and segmentation for csPCa. Cole et al [38] analyzed the repeatability of a 3D-CNN in predicting brain age from N = 20 T1-Weighted MRIs and showed that the model was able to predict brain age with high repeatability (ICC 0.9–0.98). Honsy et al [39] showed that their 3D-CNN predictions on CT for lung cancer prognostication had high ICC of 0.91. They analyzed repeatability of a trained network by evaluating the network on test-retest scans. However, in this work, we evaluated test-retest repeatability of a CNN on the testing data by training two separate models on test and retest data. This allowed us to analyze the robustness of parameter learning of the networks.

The patients with csPCa lesions are recommended to undergo treatments such as radiation therapy and surgery [40, 41] while others are recommended an active surveillance strategy. However, invasive biopsies still remain the only standard to determine GGG and identify csPCa. Therefore, in this work, we detected slices with csPCa regions and obtained an AUC of 0.78–0.85 on testing data. Although a few previous studies [42] presented an end-to-end framework to identify csPCa images on multi-parametric MRI, none of these works has analyzed the repeatability of these networks. In our study, the availability of test-retest data enabled us to analyze the robustness of the networks and we showed that the U-Net-based architecture is repeatable in slice-level detection csPCa regions. Additionally, the training of a network for slice-level detection of csPCa regions further helped in initializing the network weights for csPCa lesion segmentation. Figure 6 depicts activation maps of the networks C_A and C_B calculated using Grad-CAM [43] on four samples, two being csPCa regions and the other two being non-csPCa (GGG = 1/benign) regions on prostate ADC_m maps (B_2b900). We may observe that the networks C_A and C_B focus on similar regions to drive decisions.

Fig. 6.

Activation maps of C_A and C_B on (a, b) clinically significant prostate cancer (csPCa: Gleason grade group (GGG) > 1) regions of ADC_m (B_4b900: b values (0, 300, 500, 900 s/mm²) maps and (c, d) non-csPCa (GGG = 1/benign) regions. The activation map shows that networks look at a darker ADC_m region for csPCa regions compared with non-csPCa regions where the network looks at a brighter area. Additionally, when activations are compared between C_A and C_B, we may observe that the models focus on similar regions

Detection and segmentation of csPCa regions is vital for performing lesion-wise analysis of PCa and stratifying patients according to different risk categories. Kohl et al [44] used adversarial networks to detect and segment csPCa lesions and obtained a sensitivity of 55% in detecting csPCa lesions and a dice similarity coefficient (DSC) of 0.41 in segmenting csPCa lesions. Our model yielded a sensitivity of 63–65% in detecting csPCa lesions and a DSC of 0.68–0.72 in segmenting csPCa lesions. Additionally, the availability of test-retest scans allowed us to evaluate the repeatability of segmentations and detection of csPCa lesions in terms of DSC and proportionate agreement respectively.

In order to obtain good network reproducibility, it is essential to assess the variability in the test-retest scans themselves. Therefore, we evaluated the repeatability of volume and mean ADC_m of the csPCa lesions in terms of ICC and found that they were highly repeatable for all three b value settings (volume: ICC = 0.92; and mean ADC_m: ICC = 0.86–0.98). The repeatability of segmentations provided by the networks may also depend on inter-scan ground truth segmentation variability in segmenting lesions and prostate capsule. The Bland-Altman plots suggest that inter-scan ground truth segmentation variability is in moderate agreement with network-based segmentations.

We acknowledge that our study did have its limitations. Our work is limited to only ADC_m maps since T2W MRI were not obtained for two time points. However, few previous studies have shown that there are no significant differences between segmentations on T2W and ADC [45]. The number of patients in the study was small and data was obtained from a single institute. Since a single experienced radiologist working in consensus with pathologist delineated the lesions, incorporating multiple reader annotations was considered to be beyond the scope of the current study. However, we have evaluated inter-scan ground truth segmentation variability in segmenting csPCa lesions and its effect on networks’ performance. We have performed detection of csPCa regions and classification of other GGG groups is left for future work. Additionally, future studies are required to evaluate the performance of networks using different functions and/or models for prostate DWI since we have used only one model (diffusion-weighted imaging fitted with monoexponential function). We evaluated repeatability of a single network architecture, and comparison with other network architectures remains to be performed in future studies.

Conclusions

For the three ADC_m b value settings, U-Net-based architecture was repeatable in slice-level detection of csPCa regions. The network repeatability in segmenting csPCa lesions is affected by inter-scan variability and ground truth segmentation repeatability and may thus improve with better inter-scan reproducibility.

Key Points.

• For the three ADC_m b value settings, two CNNs with U-Net-based architecture were repeatable for the problem of detection of csPCa at the slice-level.

• The network repeatability in segmenting csPCa lesions is affected by inter-scan variability and ground truth segmentation repeatability and may thus improve with better inter-scan reproducibility.

Abbreviations

ADC

Apparent diffusion coefficient maps

ADC_m

Apparent diffusion coefficient maps obtained using monoexponential fit of DWI signal decay

AUC

Area under the receiver operating characteristic curve

CNN

Convolutional neural network

csPCa

Clinically significant prostate cancer

DL

Deep learning

DSC

Dice similarity coefficient

FN

False negatives

FP

False positives

ICC

Intra-class correlation coefficient

ML

Machine learning

MRI

Magnetic resonance imaging

PCa

Prostate cancer

PI-RADS

Prostate Imaging-Reporting and Data System

PPV

Positive predictive value

TP

True positives