PARALLEL ROUTE COHERENT MIXUP DEEP LEARNING FOR SINGLE SOURCE TO MULTI-DOMAIN GENERALIZED PROSTATE CANCER SEGMENTATION FOR MR GUIDED ADAPTIVE RADIOTHERAPY

Abstract

Magnetic resonance imaging (MRI) guided adaptive radiotherapy for prostate cancer (PCa) can allow to adapt treatment every day to the geometry of tumor and organs to deliver precise treatments while sparing organs of radiation. However, manual delineation of tumors and organs limits the adoption of this technology. Deep learning (DL) methods, though promising for in-distribution (ID) data that resembles training, deteriorate in performance when applied to out-of-distribution (OOD) datasets. Hence, we developed a novel parallel and coherent feature mixup (PaRC-mix) training approach for single source to multi-domain generalization. PaRC-mix creates feature augmentations at multiple network layers through matching linear combination of features extracted across training batches. PaRC-mix training was implemented in UNet++ to segment PCa from apparent diffusion coefficient images and compared to other mixup approaches. Models were trained on 2,029 samples from 3.0T GE MRI and tested on 557 PCa samples from 3 datasets used for diagnosis, treatment planning, and during radiation scans acquired using 3T Siemens, 3T Philips, and 1.5T Elekta Unity MR-Linac scanners from public and institutional datasets. PaRC-mix training led to significantly more accurate tumor detection (p < 0.001) and segmentation (p < 0.001) compared to other methods.

1. INTRODUCTION

Accurate segmentation of prostatic cancers (PCa) from magnetic resonance imaging (MR/MRI) is essential for MRI-guided radiotherapies. Current commercial and clinical solutions have seen great success in segmenting the whole prostate gland, but performance is much worse for PCa. Unlike prostate gland, segmentation of PCa is challenging and prone to inter-rater delineation variability because of poor tumor-tissue differentiation, small lesion sizes (under 2 cm³) as well as variable appearance in the peripheral, central, and transition zone due to complex anatomy of the prostate gland[1], [2], [3]. Hence, consistently accurate localization and delineation of PCa is essential for accurate delivery of biologically effective dose for safer and effective adaptive radiation treatments. Even when models are capable of segmenting PCa, the generalizability of these models to other institutions and datasets, particularly to clinical datasets like MR-simulation scans for planning and MR-Linac scans for treatment, is dubious. This is the problem of out of distribution data (OOD).

Multi-source domain adaptation mitigates accuracy degradation on OOD datasets by including multi-institution datasets into training, which is not practical for most applications due to limitations in transferring patient datasets from patient privacy concerns, as well as difficulty of obtaining manually delineated datasets from various institutions[3]. Another set of approaches employ generative adversarial networks (GAN) to synthesize different domain datasets and increase data diversity[4], [5], [6], [7]. Whereas the latter approach alleviates the need for labeled multi-domain datasets, it still requires unlabeled multi-institutional datasets for GAN training. Models trained on a single source and then refined through few-shot fine tuning have shown to be effective for prostate gland segmentation from ultrasound images. Alternatively, domain invariant features as organ shapes have shown to be effective to achieve multi-domain generalization. Whereas geometry is a useful feature for organs, cancerous lesions that depict considerable appearance and shape changes even in same patient undergoing treatment are less effective as a feature to inform segmentation.

A clinically practical approach is to train on a single source and with direct applicability to multi-target domain datasets by training models to extract domain invariant image features18. This approach can also be combined with other approaches to enhance the benefits of training when multiple domain datasets are available. A manifold feature mixup strategy was implemented into UNet++[8] to simultaneously regularize features extracted at different image resolutions across training batches and reduce feature variability to image differences.

2. METHODS

Feature mixup augmentation for training deep networks: The coherent mixup strategy used in PaRC-mix performs a non-linear mixup by sequentially varying features at all the feature layers occurring in the backbone as well as intermediate layers from deep skip and dense feature streams to produce a rich representation of the computed features. Hence, this approach allows extraction of a more comprehensive variation of the data at various feature resolutions and semantic context levels. Coherent feature mixup at any layer k in the network will involve a set of features from residual and skip layers that input to that feature layer ${\mathrm{X}}_{\mathrm{k}}=({\mathrm{x}}_{\mathrm{k}1},{\mathrm{x}}_{\mathrm{k}2},\dots {\mathrm{x}}_{\mathrm{k}\mathrm{n}})$ along with the corresponding labels y. For example, in Figure 1, the features at a given layer are defined by the feature streams (lines and arrows) that intersect with the mixup block (green box) defining mixup at that layer. To execute coherent mixup we combine features and labels from two intermediate minibatches (${\mathrm{X}}_{\mathrm{k}},{\mathrm{y}}_{\mathrm{k}}$) and (${\mathrm{X}}_{\mathrm{k}}\mathrm{’},{\mathrm{y}}_{\mathrm{k}}\mathrm{’}$):

$${\mathrm{X}}_{\mathrm{m}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}=\mathrm{X}\mathrm{*}\mathbf{\lambda }+\mathrm{X}\mathrm{’}\mathrm{*}(1-\mathbf{\lambda })$$

$${\mathrm{y}}_{\mathrm{m}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}}=\mathrm{y}\mathrm{*}\mathbf{\lambda }+\mathrm{y}\mathrm{’}\mathrm{*}(1-\mathbf{\lambda })$$

Figure 1.

The architecture for UNet++ with PaRC-mix. PaRC-mix can include any combination of layers for mixup. For each feature stream rs_k passing through a mixup layer the features in each sample of the batch are interpolated using the weighting parameter (drawn from the beta distribution ꞵ(ɑ, ɑ)), with another random sample from the batch is held constant for all concurrent interpolations for each feature stream and the targets for each sample (y_i). Dense skip connections are not depicted here to improve readability.

Incoherent mixup operates only on a subset of the feature streams, most often the layers occurring in the network backbone, e.g., ${\mathrm{X}}_{\mathrm{k}}\_\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{b}\mathrm{o}\mathrm{n}\mathrm{e}=\left({\mathrm{x}}_{\mathrm{k}\mathrm{k}}\right)$. This results in incoherence between the omitted features and the rest of the network, including the target labels.

In our implementation we create the minibatch (X’k,y’k) by randomly shuffling the rows of the batch (Xk,yk), resulting in a random pairing of the samples in the superbatch. Our mixup approach allows a flexible mixup strategy spanning multiple feature layers, creating a non-linear mixup of more than a single pair of batch elements

2.1. Implementation

The method was applied to a standard UNet++ architecture [8] as shown in Figure 1. The model had a depth of five nodes, corresponding to four UNet++ subnetworks. Mixup training does not alter the network architecture, and, following training, the default network was used for testing.

We investigate 4 different training methods which differ from each other only in mixup strategy: (1) No-mixup, wherein the network is trained with only standard augmentations such as image rotations, scaling, color jittering, etc. (2) Input-mix, wherein the input images and labels to the network are subject to mixup using the inputs from different training batches. (3) Incoherent-mix, which incorporates feature mixup along the feature layers occurring in the network backbone, and (4) PaRC-mix, which incorporates coherent feature mixup along all parallel paths or features layers in the network.

Models were implemented in Pytorch[9]. Networks were trained with Adam using a learning rate of 0.0001. Models were initially trained with a fixed learning rate for 20 epochs, followed by a linearly decaying learning rate until a maximum of 600 epochs. The label smoothing hyperparameter ϵ, batch size, and mixup hyperparameters were determined empirically by best segmentation performance defined as the model with the highest normed score on the validation dataset. Label smoothing was applied with ϵ=0.001. Batch size was set to eight. Mixup weight defined using the distribution (1,1), which corresponds to a uniform distribution.

2.2. Patient characteristics and datasets

Patients with PCa from 4 different datasets (see Table 1) were analyzed and consisted of: (a) Training, institutional ID dataset of patients who underwent diagnostic imaging using 3 Tesla GE with endorectal coil followed by surgical resection[10], [11], (b) ProstateX, an open source OOD dataset[12], [13] acquired on 3 Tesla Siemens diagnostic scanners with manual PCa delineations[14], (c) MR-Sim, internal OOD dataset of patients with PCa acquired on 3 Tesla Philips MR scanners for simulating radiation treatment plans prior to MR guided RT, and (d) MR-Linac, internal OOD dataset of patients with prostate cancer who underwent hypofractionated adaptive RT on 1.5 Tesla Philips MR-Unity (Elekta) with onboard imager and Linac system, with images acquired daily prior to treatment. Internal data was collected in accordance with a protocol approved by the MSKCC institutional review board. ProstateX data was used retrospectively as provided in The Cancer Imaging Archive.

Table 1.

Number of intermediate to high risk (IHR) lesions for each dataset (counting lesions contained in separate scans multiple times), and the corresponding lesion characteristics in terms of volume and ADC.

Dataset	Scanner	Number of IHR or PIRADS>3 Lesions (GS ≥ 7) /Total	Median Lesion Volume (cc)	Median Lesion ADC (×10-3 mm2/s)	Resolution (mm3)
Training/ Validation	Internal diagnostic w/ EC, 3T GE	147/191	0.84 [0.39–1.3]	0.95 [0.87–1.11]	0.625×0.625×3 and 0.78×0.78×3.6
ProstateX	External diagnostic, 3T Siemens	76/298	1.1 [0.51–2.6]	0.96 [0.88–1.04]	2×2×3 and 2×2×4
MR-Sim	Internal, planning, 3T Philips	59/59	0.58 [0.26–1.8]	0.81 [0.69–0.94]*	1.125×1.125×4
MR- Linac	Internal, treatment, 1.5T Philips	200/200	1.52 [0.99–2.4]*	1.43 [1.28–1.53]*	1.92×1.92×4

Note that the images in the training and validation set used an endorectal coil (EC). Significant differences in lesion volume or ADC from the training set are marked with an *.

3. RESULTS

Example segmentations for each dataset and model can be seen in Figure 2. PaRC-mix had the highest generalization accuracy, outperforming other methods for both DSC and HD95. Table 2 contains the median DSC for each dataset and method, along with the relative recall and precision for each approach. As a general trend PaRC-mix and incoherent-mix improved precision (fewest false detections) over no-mix or input-mix, while PaRC-mix and input-mix improved recall (more lesions detected) and lesion DSC over no-mix and incoherent-mix.

Figure 2.

Representative examples from analyzed MR scanners used in the testing datasets showing segmentation results produced by various training strategies. Manual delineation is depicted using solid “white” contour while model-generated segmentation is shown in dotted “cyan” color. The two columns of MR-Sim and MR-Linac show matching patients from MR-Sim and the first fraction of treatment on the MR-Linac.

Table 2.

(left) The median Dice scores comparing all lesions of GS ≥ 7 detected by at least one model.

(A)	ProstateX, 3T Siemens	MR-Sim, 3T Philips	MR-Linac, 1.5T Philips	All
	Dice Score (diff)	Dice Score (diff)	Dice Score (diff)	Dice Score (diff)
No-mixup	0.55 (↓0.09**)	0.42 (↓0.04*)	0.30 (↓0.04**)	0.34(↓0.07***)
Input-mix	0.59 (↓0.05)	0.44 (↓0.02)	0.30 (↓0.04**)	0.39 (↓0.02***)
Incoherent-mix	0.63 ↓0.01)	0.35 (↓0.11)	0.30 (↓0.04***)	0.37 (↓0.04***)
PaRC-mix	0.64	0.46	0.34	0.41
(B)	ProstateX, 3T Siemens	MR-Sim, 3T Philips	MR-Linac, 1.5T Philips	All
	Precision	Precision	Precision	Precision
No-mixup	0.21*	0.18	0.16	0.18*
Input-mix	0.24	0.18	0.14	0.19
Incoherent-mix	0.29	0.20	0.17	0.22
PaRC-mix	0.27	0.21	0.17	0.21
(C)	ProstateX, 3T Siemens	MR-Sim, 3T Philips	MR-Linac, 1.5T Philips	All
	Recall	Recall	Recall	Recall
No-mixup	0.87*	0.69	0.78	0.86
Input-mix	0.96	0.75	0.69**	0.85**
Incoherent-mix	0.99	0.68	0.72	0.85**
PaRC-mix	1	0.8	0.78	0.91

Models under-performing PaRC-mix are indicated with blue down arrow and those over-performing PaRC-mix are indicated by red up arrow. (B) shows precision and (C) shows recall metrics for each model and dataset. Models with significantly different scores compared to PaRC-mix are indicated by * for p < 0.05, ** for p < 0.01, *** for p < 0.001.

3.1. Disaggregated analysis of models’ segmentation accuracies with varying lesion characteristics

Lesion level GS were available for ProstateX; 35 lesions had GS 4+ 3 or higher, 41 had GS 3+4, and 36 had GS 3+3. Disaggregated analysis found both PaRC-mix and incoherent-mix training achieved higher median accuracy than no-mixup or input-mix.

Lesion level zone designations were available for a subset of ProstateX, MR-Sim, and MR-Linac datasets . 38 lesions were designated in the peripheral zone (PZ) and 97 lesions in the transition zone (TZ). Lesions occurring in the anterior stroma were grouped with the TZ lesions.

Performance enhancement was most pronounced for the most difficult lesions. Lesions in the TZ were segmented with significantly higher accuracy compared to those occurring in the PZ [1], [15]irrespective of the training approach (no-mixup p<0.001, input-mix p=0.006, incoherent-mix p<0.001, PaRC-mix p = 0.003). Performance gap analysis comparing normed scores for lesions occurring in TZ and PZ showed the largest gap for no-mixup training (0.23) The performance gap decreased to 0.11 for input-mix, 0.16 for incoherent-mix, and 0.14 for PaRC-mix (which had the highest median DSC in both categories).

When assessed in terms of Gleason score the least aggressive lesions tended to be the hardest to segment. Performance gap was computed as the difference in the median normed score between intermediate or highly aggressive (GS 3+4 or higher) and less aggressive (GS 3+3) lesions for the individual methods. Models trained with no-mixup and input-mix resulted in the largest DSC performance gaps (0.059 and 0.071), indicating that these models were not as robust to variations in lesion aggressiveness. Mixup improved segmentations scores and feature mixup training reduced the performance gap, with a performance gap of 0.035 for incoherent-mix, and 0.006 for PaRC-mix.

These results indicate that mixup training, especially the feature mixup methods like PaRC-mix and incoherent mix improved the robustness of the models to variations in lesion aggressiveness or spatial location within the prostate gland.

3.2. Fine-tuning further enhanced accuracy for clinical cases on the MR-Linac dataset

Finetuning of the PaRC-mix model via a 5-fold cross validated division of the 40 MR-Linac patients resulted in a significant increase in DSC from 0.33 (0.18–0.43) to 0.38 (0.18–0.55), p<0.001, and decrease in HD95 from 7.76 mm (5.73 mm −11.84 mm) to 6.41 mm (4.75 mm - 9.82 mm), p<0.01. This was accompanied by a statistically significant drop in false positives, with precision increasing from 0.17 to 0.25 (p<0.001), but recall did not change significantly (0.78 vs 0.81, p = 0.36).

4. DISCUSSION

To our knowledge, this is the first PCa lesion segmentation approach that evaluated the generalization capability of a single source training method across a reasonably wide range of scanners and for patients scanned at different stages of disease and treatment, including diagnosis, planning simulation, and during hypofractionated radiation treatment. Importantly, our work also studied why generalization occurs with different training approaches. Prior works typically used data acquired from a single institution[1], [15]. A few prior works used data from different institutions for evaluation but were acquired with the same scanner manufacturer[16], [17], [18]. In our prior work, we evaluated UNet++ and multiple other methods on ProstateX and a subset of the MR-Sim scans[10]. In general, prior works, including ours, studied the capability of the models applied to diagnostic quality MRI as opposed to images acquired for treatment simulation and across the course of radiation treatment. Of note, prior works typically focused on segmenting the whole prostate gland, easier to segment than the intra-prostatic lesions due to the inherent difficulty in detecting the lesions and ensuing variability even between clinicians in delineating such lesions[1], [2], [3]. Furthermore, segmenting whole prostate gland is insufficient for dose escalation therapies that boost the dose to the dominant lesions, shown to be effective in reducing biological recurrence and increasing overall survival by clinical trials such as the multi-institutional FLAME trial.[19], [20]

Prior methods for domain invariant segmentation often used geometric priors, which are applicable to organs but not tumors due to their highly variable appearance and shapes. Adversarial perturbation of image intensities[21] and image style to alter underlying first and second order texture statistics[22] can increase data diversity. However, such methods only vary the intensity statistics of the images themselves, which may not lead to sufficient diversity in the extracted features. Alternative methods such as feature-level augmentation implemented by linearly mixing input features from different domains[23] build on the input mixup methods[24]. Previously, Jung et al[25] demonstrated enhanced accuracy by implementing feature level mixup separately for input, encoder, decoder, bottleneck and skip level layers of a UNet. However, this approach did not implement any measures to ensure coherence between manifold mixup in the network backbone and the long skip connections. Other feature-level mixup approaches employed feature mixup at each skip connection and output of the bottleneck layer[26] or randomly manipulated the feature layers used for mixup augmentation[27] implemented into the UNet in the context of semi-supervised learning. PaRC-mix improves over prior methods by performing feature level mixup coherently into encoder, decoder, residual and skip features within a single network. Our results show that PaRC-mix that uses coherent mixup of features for all paths results in better generalization compared to the incoherent mixup approach when considering both networks.

There was a drop in performance when the models were directly applied to the MR-Linac datasets, possibly due to factors beyond image distribution shift such as differing norms for delineating lesions for radiation treatments relative to diagnosis by radiologists. We found that these segmentations can be improved by fine-tuning for radiation treatments. Further improvements could be achieved with patient-specific fine-tuning as needed to achieve highly accurate segmentation in consequent treatment fractions, reducing delineation requirements for radiation oncologists. This work focused on assessing the geometric accuracy of the AI generated segmentations compared to clinical delineations, but not in assessing if the AI segmentations further reduced clinical variability. Proper and detailed assessment of the same requires a larger and focused study and is planned future work. Our model was limited to using only the ADC instead of combining T2-weighted MRI to reduce impact of potential variations from T2-weighted imaging and limited accuracy improvement for combining the two modalities seen in our preliminary evaluation. Nevertheless, our results show reliable performance on multiple analyzed datasets with varied imaging acquisitions.

5. CONCLUSIONS

PaRCmix significantly improved the segmentation generalization to scanners and acquisitions outside the training distribution.

Fig. 3.

Disaggregated analysis depicts the DSC for the analyzed methods with respect to Gleason scores and lesion location categorized as peripheral zone or PZ and transition zone or TZ.