PSIGAN: Joint probabilistic segmentation and image distribution matching for unpaired cross-modality adaptation based MRI segmentation

Abstract

We developed a new joint probabilistic segmentation and image distribution matching generative adversarial network (PSIGAN) for unsupervised domain adaptation (UDA) and multi-organ segmentation from magnetic resonance (MRI) images. Our UDA approach models the co-dependency between images and their segmentation as a joint probability distribution using a new structure discriminator. The structure discriminator computes structure of interest focused adversarial loss by combining the generated pseudo MRI with probabilistic segmentations produced by a simultaneously trained segmentation sub-network. The segmentation sub-network is trained using the pseudo MRI produced by the generator sub-network. This leads to a cyclical optimization of both the generator and segmentation sub-networks that are jointly trained as part of an end-to-end network. Extensive experiments and comparisons against multiple state-of-the-art methods were done on four different MRI sequences totalling 257 scans for generating multi-organ and tumor segmentation. The experiments included, (a) 20 T1-weighted (T1w) in-phase mdixon and (b) 20 T2-weighted (T2w) abdominal MRI for segmenting liver, spleen, left and right kidneys, (c) 162 T2-weighted fat suppressed head and neck MRI (T2wFS) for parotid gland segmentation, and (d) 75 T2w MRI for lung tumor segmentation. Our method achieved an overall average DSC of 0.87 on T1w and 0.90 on T2w for the abdominal organs, 0.82 on T2wFS for the parotid glands, and 0.77 on T2w MRI for lung tumors.

I. Introduction

Magnetic resonance imaging (MRI) is rapidly emerging as the modality for image-guided adaptive radiation therapy treatments[1] due to its better soft tissue contrast compared with computed tomography (CT) scans. However, a critical requirement for MR-guided radiotherapy is fast, accurate, and consistent segmentation of target and surrounding normal organs at risk (OAR) [2].

Deep learning-based methods have shown remarkable success in diverse image analysis tasks when they can be trained using large annotated datasets. However, acquiring large expert-segmented medical image datasets is difficult. This is because, slice-by-slice delineations of several organs is highly time consuming, and such delineations require a domain expert, such as a radiologist or a radiation oncologist.

Domain adaptation [4], [5], [6], [7] is a commonly used approach to overcome the issue of learning from limited and unlabeled target modality datasets, where a model for the target domain is trained by using an existing labeled dataset from a different modality, called the source domain. In unsupervised domain adaptation (UDA) based segmentation, the focus of this work, no target domain labeled data is available for training.

UDA segmentation has been accomplished by either using feature-level or pixel-level adaptation. In feature-level adaptation, the encoder networks are adversarially trained to extract domain-invariant features, such that a single segmentation model trained on source data is applicable to both domains [4], [5], [8], [9]. In pixel-level adaptation, generative adversarial networks (GAN) model the complex inter-modality anatomical relationships and compute image to image (I2I) translations [6], [10]. The target modality model is then learned by using the source to target transformed images. Cyclical consistency [11] and feature disentanglement [12], [13] losses are often used in unpaired I2I translation methods to circumvent the lack of corresponding source and target modality images [3], [14], [7]. Hybrid methods combine feature and pixel-level adaptation [15], [16], [17] in order to ensure good pixel-level I2I transformations and the preservation of low-level edge and mid-level textural characteristics of the target modality images.

A major issue while performing UDA is mode collapse [18], wherein multiple distinct inputs are mapped to a same output. Modality hallucination is a related manifestation of this issue that commonly occurs in medical image I2I translations, wherein distinct organ characteristics like geometry (or overall shape) and appearance (or intensity distribution) are ill-preserved or are removed [10] in translated images. This is because the commonly used losses based on matching global or marginal intensity statistics of whole images cannot sufficiently constrain the generator to model the local organ/tumor geometry and appearance statistics. Feature disentanglement methods have been reported to reduce the aforementioned issues by using shared content and domain-specific attribute encoders [12], with demonstrated success in medical image applications [13]. Nevertheless, the lack of explicit conditioning of losses with respect to the geometry and appearance of the various structures of interest (SOI), may not yield the desired results for the output task.

Prior works have used output segmentation as an adversarial loss to constrain UDA and improve segmentations [14], [19], [20], [21], [10]. As shown in [15], task-specific losses can improve training stability and reduce chances of mode collapse. However, adversarial losses computed only using the segmentation output can constrain the geometry but not necessarily the appearance of SOIs in I2I translation.

A key difference of our approach compared to prior works is the use of joint distribution matching adversarial loss. The joint distribution is represented as a channel-wise concatenation of images and their voxel-wise segmentation probability maps. Prior works have employed similar joint-distribution matching to constrain bi-directional mapping between images and a low dimensional latent distribution vector [22], [23] or images and scalar output categories [24]. Ours on the other hand constrains pixel-level relationships between I2I translations and segmentations. To our best knowledge, this is the first approach to compute joint distribution matching of images and segmentation probabilities for UDA segmentation (Fig. 1). This approach also leads to a cyclical optimization where both segmentation and generator outputs constrain each other. In prior approaches like [21], [8], segmentation outputs do not constrain generator network gradient computation.

Fig. 1.

Difference between traditional UDA [3] and PSIGAN segmentation. In both (a) and (b), the translation network T parameterized by θ_T produces labeled pseudo target data {$X_c^m$, Y_c} from unpaired source X_c and target X_m data and trains the segmentation (S) network, parameterized by θ_S. PSIGAN also uses a structure discriminator D_struct to match the joint distribution of image-segmentation probability maps {x_m, ψ_m}, {$x_c^m$, ${\psi }_c^m$} to further optimize θ_T.

Our contributions are:

• Organ geometry and appearance constrained unpaired cross-modality adaptation. We introduced an UDA approach that constrains organ geometry and appearance in I2I translations by computing adversarial losses to minimize the mismatch in the joint distribution of images and their segmentation probability maps.

• A cyclic feedback based UDA segmentation. In our approach, the generator outputs are used to train the segmentation network, while the segmentation outputs are used to compute losses for the generator.

• Comprehensive performance comparisons were done against multiple state-of-the-art methods using three datasets for multiple organs and tumor segmentation. Extensive ablation experiments were done to measure the impact of joint distribution matching on both segmentation and I2I translation.

II. Related Works

A. Feature-level UDA segmentation

Feature-level UDA segmentation methods extract a domain-invariant feature encoding, such that a model trained on source data is applicable to both source and target domains. This method is often used for performing domain adaptation between related image sequences. Example applications of this include prostate gland segmentation from MRIs acquired from different scanners [5], and brain tumor segmentation from similar MRI contrasts [4]. Joint adversarial training strategy combining a domain critic network with a segmentation network has been used for the more challenging cross-modality adaptation between CT and MRI in [25]. However, as shown in [25], these methods may require different number of feature layers to be adapted for segmenting various organs. Hence, network feature-adaptation depth is an important hyper-parameter. Such a depth tuning can require computationally intensive training when generating segmentation of large number of organs for radiation therapy applications. This issue has been avoided by using joint latent space learning with variational autoencoders [7] and disentanged feature representations [26]. The work in [27] employed output feature matching for producing scanner invariant estimation of cardiothoracic ratio (or heart size) from chest X-rays. Segmentation probability maps produced from softMax layer [28] and entropy maps indicative of pixel-wise classification uncertainties [21] have been used for domain invariant (synthetic and camera-acquired) natural image segmentation.

B. Pixel-level UDA segmentation

Pixel-level domain adaptation and segmentation methods model the inter-modality relationships and compute I2I image translations. The translated images are used to train a segmentation model for the target domain [29], [30], [3], [14]. Two-step training consisting of I2I translation (e.g., CT to MRI) followed by segmentation network training has been used for MRI lung tumor [10] and fundus image segmentation [31]. I2I translation and segmentation were combined into one network to segment cardiac structures from CT [14], [17], abdominal organs from CT and MRI [3], and knee structures from MRI [30]. Multiple works [32], [30], [3] have combined cyclical consistency losses of the CycleGAN [11] with a segmentation network to train with unpaired source and target modalities. Cyclical consistency loss shrinks the space of possible mappings in GANs by computing global intensity distribution mismatches. But this loss alone is insufficient to preserve the SOI geometry and appearance in I2I translation of medical images [33]. Inclusion of style and perceptual losses have shown improvements in I2I translations [34]. Geometry preserving losses implemented by backpropagating segmentation losses [14] to the generator and high-level segmentation features matching losses have also shown improvements for both tumor [10] and semantic segmentation of real-world images [16]. However, none of the aforementioned losses provide constraints to sufficiently control both geometry and appearance of the SOIs in I2I translation. We address this problem by combining images and their segmentation probability maps as a joint density for adversarial learning.

III. Method

Goal:

Learn MRI multi-organ segmentation models by using unpaired expert-segmented CT and unlabeled MRI images.

Notation:

CT (X_c, Y_c) is the source domain that consists of images x_c ∈ X_C and expert-segmentations y_c ∈ Y_C for training. MRI is the target domain that is provided with only MRI images x_m ∈ X_M for training. As the learning optimization involves a finite set of examples, the probability distribution of CT and MRI images are represented as p(x_c) and p(x_m). The probability distribution of pseudo MRI $x_c^m$ and pseudo CT $x_m^c$ resulting from I2I translations of CT to MRI and MRI to CT are represented as $p\left(x_c^m\right)$ and $p\left(x_m^c\right)$, respectively. The joint probability distribution of the real MR and pseudo MR images and their probabilistic segmentations are represented as p(x_m, ψ_m) and $p\left(x_c^m,{\psi }_c^m\right)$, respectively.

A. Background

In supervised learning with finite set of training examples N, given a joint probability distribution of inputs and outputs p(x, y) and a model parameterized by θ, a chosen loss function L_c(.) is used to compute the empirical risk in predicting the outputs y from inputs x:

$$\mathbb{E}[L(x,y,\theta )]=\underset{\theta \in \Theta }{\mathit{\text{argmin}}}\sum _{i=1}^{N}p\left(x_i,y_i\right)L_c\left(x_i,y_i,\theta \right),$$ (1)

In pixel-level UDA segmentation, we seek to minimize the empirical risk of training with pseudo target examples, obtained through a model $\varphi :x_c\to x_c^m$, by assuming that $p\left(x_c^m\right)\approx p\left(x_m\right)$:

$$\mathbb{E}\left[L\left(x_c^m,y,\theta \right)\right]=\underset{\theta \in \Theta }{\mathit{\text{argmin}}}\sum _{i=1}^{N}p\left(x_{c_i}^m,y_i\right)L_c\left(x_{c_i}^m,y_i,\theta \right).$$ (2)

where $p\left(x_c^m,y\right)$ is the joint distribution on an intermediate representation or pseudo domain. However, ϕ does not produce a perfect mapping, that is $p\left(x_c^m\right)\ne p\left(x_m\right)$. Therefore, an additional domain translation loss L_t needs to be added to (2). With $p\left(x_c^m,x_m\right)$ as the joint probability distribution over pseudo and real target samples, the risk is computed as:

$$\begin{gathered} \mathbb{E}\left[L\left(x_c^m,x_m,y,\theta ,\varphi \right)\right]=\underset{\text{Domain translation }}{\underbrace{\underset{\varphi \in \Phi }{\mathit{\text{argmin}}}\sum _{j=1}^{M}p\left(x_{c_j}^m,x_{m_j}\right)L_t\left(x_{c_j}^m,x_{m_j},\varphi \right)}} \\ +\underset{\text{Segmentation }}{\underbrace{\underset{\theta \in \Theta }{\mathit{\text{argmin}}}\sum _{i=1}^{N}p\left(x_{c_i}^m,y_i\right)L_c\left(x_{c_i}^m,y_i,\theta \right).}} \end{gathered}$$ (3)

Equation (3) can be optimized by training the domain translation and segmentation networks either sequentially [10], [31] or jointly [3], [14], [16]. However, this optimization ignores the co-dependency of domain translation and segmentation. For instance, no explicit constraint exists to preserve any inherent target modality appearance or geometric characteristics of the SOIs that distinguishes them from the background in the I2I translated images, which is a cause of sub-optimal performance. We model this co-dependency by computing adversarial losses using the joint distribution of images and their segmentation probability maps as a pair obtained from the translated $\left(\left\{x_c^m,{\psi }_c^m\right\}\right)$ and real target images ({x_m, ψ_m}). In other words, we require $p\left(x_c^m,{\psi }_c^m\right)\approx p\left(x_m,{\psi }_m\right)$.

B. Probabilistic segmentation and image matching GAN (PSIGAN)

The overview of PSIGAN is shown in Fig. 2. PSIGAN consists of a CT to MRI generator $G_C^M:x_c\to x_c^m$, global intensity discriminator D_M, a structure discriminator D_struct, and a target domain segmentor S: x_m → {ψ_m, y_m}, where ψ_m is the predicted map of voxel-wise segmentation probabilities for x_m and y_m is the K-organ segmentation for x_m. We also include a MR to CT generator $G_M^C:x_m\to x_m^c$ and global intensity discriminator D_C to implement cyclical consistency losses for unpaired I2I translation. D_M computes a global adversarial loss to penalize mismatches in the marginal intensity distribution of pseudo and real MRI images in order that $p\left(x_c^m\right)\approx p\left(x_m\right)$:

$$\underset{D_M}{\text{max}}\underset{G_C^M}{\text{min}}{L}_{\mathit{\text{adv}}}^{C\to M}\left(G_C^M,D_M\right)={\mathbb{E}}_{x_m~p\left(x_m\right)}\left[\mathit{\text{log}}\left(D_M\left(x_m\right)\right)\right]+{\mathbb{E}}_{x_c~p\left(x_c\right)}[\mathit{\text{log}}(1-\left(D_M\left(G_C^M\left(x_c\right)\right)\right].$$ (4)

Fig. 2.

Approach overview. Generator $G_C^M$ converts CT image x_c into pseudo MR image $x_c^m$, which is used to train the segmentor S. S uses split encoders E_M, $E_C^M$ and shared decoder DE to separate the gradient flows for D_struct and $G_C^M$ through sub-networks S_M and $S_C^M$. D_struct computes joint distribution matching ({$x_c^m$, ${\psi }_c^m$}, {x_m, ψ_m}) adversarial loss $L_{\mathit{\text{struct}}}^D$, where ${\psi }_c^m$ and ψ_m are produced by S_M. The corresponding adversarial loss $L_{\mathit{\text{struct}}}^G$ for $G_G^M$ uses {$x_c^m$, ${\overline{\psi }}_c^m$}, where ${\overline{\psi }}_c^m$ is produced by $S_C^M$. D_M and D_C are global intensity discriminators for MRI and CT domain; $G_M^C$ converts MRI to CT images for enforcing cyclically consistent I2I translations.

D_struct computes a joint distribution adversarial loss to reduce SOI appearance (intensity distribution) and geometry (overall shape) mismatches between pseudo $x_c^m$ and real MRIs x_m. It accomplishes this by matching the joint distributions p(x_m, ψ_m) and $p\left(x_c^m,{\psi }_c^m\right)$, implemented by concatenating the images and their segmentation probability maps. The segmentor S (Fig. 2) produces the segmentation probability maps ψ_m, ${\psi }_c^m$ from real x_m and pseudo MRI $x_c^m$. D_struct loss is computed by using both real and pseudo MRI as:

$$\underset{D_{\mathit{\text{struct}}}}{\text{max}}{L}_{\mathit{\text{struct}}}^D={\mathbb{E}}_{\left(x_m,{\psi }_m\right)~p\left(x_m,S\left(x_m\right)\right)}\left[\mathit{\text{log}}\left(D_{\mathit{\text{struct}}}\left(x_m,{\psi }_m\right)\right)\right]+{\mathbb{E}}_{\left(x_c^m,{\psi }_c^m\right)~p\left(x_c^m,S\left(x_c^m\right)\right)}\left[\mathit{\text{log}}\left(1-D_{\mathit{\text{struct}}}\left(x_c^m,{\psi }_c^m\right)\right)\right].$$ (5)

The inclusion of segmentation probability maps in this loss constrains the SOI geometry similar to prior works [3], [14], [31], [17]. The inclusion of images in this loss additionally constrains the appearance. The generator $G_C^M$ is optimized by adversarial loss, which is computed using the pseudo MR images $x_c^m=G_C^M\left(x_c\right)$ as:

$$\underset{G_C^M}{\text{min}}{L}_{\mathit{\text{struct}}}^G={\mathbb{E}}_{\left(x_c^m,{\psi }_c^m\right)~p\left(x_c^m,S\left(x_c^m\right)\right)}\left[\mathit{\text{log}}\left(1-D_{\mathit{\text{struct}}}\left(x_c^m,{\psi }_c^m\right)\right)\right].$$ (6)

ψ_m, ${\psi }_c^m$ are produced by aggregating label assignment probabilities e_{i, j} at location i, j for K SOIs from K channels as generated by the softMax layer of the network S as:

$${\psi }_{i,j}=\sum _{n=2}^K{e}_{i,j}^n.$$ (7)

The first channel corresponds to background and is thus ignored in the aggregation shown in (7). The map ψ has continuous values in the range [0, 1], where higher values indicate higher likelihood of a voxel corresponding to a SOI, while lower values are indicative of background.

The segmentor S can be trained using the generated pseudo MRI and the associated expert-segmentations available on the source modality $\left\{x_c^m,y_c\right\}$ and optimized using cross-entropy losses. The output segmentation is computed using an argmax function.

Given a fixed $G_C^M$ and S, the optimal D_struct at any point in the optimization of (5) is given by $\frac{p\left(x_m,{\psi }_m\right)}{p\left(x_m,{\psi }_m\right)+p\left(x_c^m,{\psi }_c^m\right)}$. The global equilibrium is achieved if and only if the joint distribution of $p\left(x_c^m,{\psi }_c^m\right)$ and p(x_m, ψ_m) are matched, i.e., $p\left(x_c^m,{\psi }_c^m\right)=p\left(x_m,{\psi }_m\right)$. Thus, unlike methods in [3], [14], [10], [31], [17] that only constrain geometry, ours constrains both organ geometry and appearance.

The joint distribution formulation leads to a cyclical optimization of the generator $G_C^M$ and the segmentation S networks. Concretely, S requires the output $x_c^m$ of $G_C^M$ for its training, while $G_C^M$ requires the output of S, namely ${\psi }_c^m$ for gradient computation. The gradient flow relation between D_struct, $G_C^M$ and S is illustrated in Fig. 3(a)¹.

Fig. 3.

Gradient flow between $G_G^M$, D_struct, and segmentors using (a) single segmentor S, and (b) split segmentors $S_C^M$ and S_M.

The gradient flow resulting from a network using a single segmentor S is shown in Fig. 3(a), where the outputs of $G_C^M$ and S are used to update each other. Concretely, $x_c^m$ is used to train S, and $\left\{x_c^m,{\psi }_c^m\right\}$ is used to compute the joint distribution adversarial loss for $G_C^M$. As a result, the outputs of $G_C^M$ and S can co-adapt to facilitate easy segmentation, but without $x_c^m$ matching the target distribution. In this case, D_struct can easily distinguish $p\left\{x_c^m,{\psi }_c^m\right\}$ from p(x_m, ψ_m), allowing it to reach a stable local minima well before $G_C^M$ reaches its local minima, and possibly prevent GAN convergence. This is because, GAN optimization hinges on achieving Nash equilibrium, whereby all players (generator and discriminator) achieve equal payoffs (reach local minima at similar times). This would also lower the generalization accuracy of S on real MRI. We address this potential issue by separating the networks used for producing the segmentation probability maps for $G_C^M$ and D_struct. Thus, segmentation probability maps from network S_M is used to compute D_struct loss, and segmentation probability maps from $S_C^M$ is used for to compute $G_C^M$ loss. The modified configuration of gradient flow is shown in Fig. 3 (b).

C. Split segmentor network

The split segmentation network consists of two encoder subnetworks, called E_M and $E_C^M$ with a shared decoder network (DE) (Fig. 2), from which two segmentations are generated via S_M = E_M ○ DE and $S_C^M=E_C^M○DE$. Shared decoder is used because the high-level contextual features for segmentation should be same between pseudo and real MRI. It also reduces the number of parameters required for training S_M and $S_C^M$. The networks S_M, $S_C^M$ are trained using pseudo MRI data with label ($x_c^m$, y_c) and optimized with cross-entropy losses $L_{seg}^M$ (first part of summation in (8)) for S_M and $L_{seg}^{\overline{M}}$ (second part of summation in (8)) for $S_C^M$, respectively. The overall loss $L_{seg}=L_{seg}^M+L_{seg}^{\overline{M}}$ is computed as:

$$L_{seg}=\underset{L_{seg}^M}{\underbrace{{\mathbb{E}}_{x_c^m~p\left(x_c^m\right),y_c~Y_c}[\mathit{\text{log}}P\left(y_c\mid S_M\left(x_c^m\right)\right]}}+\underset{L_{seg}^{\overline{M}}}{\underbrace{{\mathbb{E}}_{x_c^m~p\left(x_c^m\right),y_c~Y_c}\left[\mathit{\text{log}}P\left(y_c\mid S_C^M\left(x_c^m\right)\right)\right]}}.$$ (8)

The network S_M is used to produce ψ_m and ${\psi }_c^m$ from x_m (solid blue arrow Fig. 2) and $x_c^m$ (solid orange arrow Fig. 2), respectively to compute the gradients for D_struct. Gradient flow with respect to $G_C^M$, S_M, and D_struct is shown in Fig. 3 (b). The modified loss $L_{\mathit{\text{struct}}}^D$ to optimize D_struct, obtained by replacing S in (5) with S_M is computed as:

$$\underset{D_{\mathit{\text{struct}}}}{\text{max}}{L}_{\mathit{\text{struct}}}^D={\mathbb{E}}_{\left(x_m,{\psi }_m\right)~p\left(x_m,S_M\left(x_m\right)\right)}\left[\mathit{\text{log}}\left(D_{\mathit{\text{struct}}}\left(\left[x_m,{\psi }_m\right]\right)\right)\right]+{\mathbb{E}}_{\left(x_c^m,{\psi }_c^m\right)~p\left(x_c^m,S_M\left(x_c^m\right)\right)}\left[\mathit{\text{log}}\left(1-\left(D_{\mathit{\text{struct}}}\left(\left[x_c^m,{\psi }_c^m\right]\right)\right)\right)\right].$$ (9)

The network $S_C^M$ is used to produce ${\overline{\psi }}_c^m$ from $x_c^m$ (dotted orange arrow Fig. 2) in order to compute the gradients for $G_C^M$. We use ${\overline{\psi }}_c^m$ to differentiate from ${\overline{\psi }}_c^m$ that is generated using S_M in (9). The loss $L_{\mathit{\text{struct}}}^G$ to optimize ${\overline{\psi }}_c^m$ is now computed by replacing S in (6) with $S_C^M$ and $x_c^m=G_C^M\left(x_c\right)$:

$$\underset{G_C^M}{\text{min}}{L}_{\mathit{\text{struct}}}^G={\mathbb{E}}_{\left(x_c^m,{\overline{\psi }}_c^m\right)~p\left(x_c^m,S_C^M\left(x_c^m\right)\right)}\left[1-\mathit{\text{log}}\left(D_{\mathit{\text{struct}}}\left(\left[x_c^m,{\overline{\psi }}_c^m\right]\right)\right)\right].$$ (10)

The network S_M, which is never used in generator update is used to segment target datasets at test time. We found that this network produces more accurate MRI segmentations than the $S_C^M$ as shown in results.

D. Additional losses for unpaired mapping

In order to train with unpaired CT and MR datasets, we enforce consistent reverse mapping by computing adversarial penalties from target to source generator $G_C^M$ and global intensity discriminator D_C. This loss is expressed as:

$$\underset{D_C}{\text{max}}\underset{G_M^C}{\text{min}}{L}_{\mathit{\text{adv}}}^{M\to C}\left(G_M^C,D_C\right)={\mathbb{E}}_{x_c~p\left(x_c\right)}\left[\mathit{\text{log}}\left(D_C\left(x_c\right)\right)\right]+{\mathbb{E}}_{x_m~p\left(x_m\right)}[\mathit{\text{log}}(1-\left(D_C\left(G_M^C\left(x_m\right)\right)\right].$$ (11)

The global adversarial loss is then computed as $L_{\mathit{\text{adv}}}\text{=}{L}_{\mathit{\text{adv}}}^{C\to M}+L_{\mathit{\text{adv}}}^{M\to C}$. Cyclical consistency loss [11] is computed to account for lack of any pixel-level correspondence between the source and the target domain images in the I2I translation $(G_{C\circlearrowleft M}=G_M^C\left(G_C^M\left(x_c\right)\right)$; $G_{M\circlearrowleft C}=G_C^M\left(G_M^C\left(x_m\right)\right))$ as:

$$L_{\mathit{\text{cyc}}}\left(G_C^M,G_M^C\right)={\mathbb{E}}_{x_c~p\left(x_c\right)}\left[{\Vert G_{C\circlearrowleft M}\left(x_c\right)-x_c\Vert }_1\right]+{\mathbb{E}}_{x_m~p\left(x_m\right)}\left[{\Vert G_{M\circlearrowleft C}\left(x_m\right)-x_m\Vert }_1\right].$$ (12)

The total loss is expressed as:

$$L_{\mathit{\text{total}}}=L_{\mathit{\text{adv}}}+{\lambda }_{\mathit{\text{cyc}}}{L}_{\mathit{\text{cyc}}}+{\lambda }_{\mathit{\text{struct}}}{L}_{\mathit{\text{struct}}}+{\lambda }_{\text{seg}}{L}_{\text{seg}},$$ (13)

where L_struct corresponds to either $L_{\mathit{\text{struct}}}^D$ or $L_{\mathit{\text{struct}}}^G$ depending on whether this loss is for D_struct ((9)) or $G_C^M$ ((10)).

The generators ($G_C^M$, $G_M^C$), discriminators (D_M, D_C, and D_struct), and segmentors (S_M, $S_C^M$) are updated with the following gradients, $-{\Delta }_{{\theta }_G}(L_{\mathit{\text{adv}}}+{\lambda }_{\mathit{\text{cyc}}}{L}_{\mathit{\text{cyc}}}+{\lambda }_{\mathit{\text{struct}}}{L}_{\mathit{\text{struct}}}^G+{\lambda }_{seg}{L}_{seg}^{\overline{M}})$, $-{\Delta }_{{\theta }_{D,D_{\mathit{\text{struct}}}}}\left(L_{\mathit{\text{adv}}}+{\lambda }_{\mathit{\text{struct}}}{L}_{\mathit{\text{struct}}}^D\right)$ and $-{\Delta }_{{\theta }_S}\left(L_{seg}\right)$, respectively. The algorithm for the proposed method is shown in Algorithm 1.

E. Implementation and network structure

All networks were implemented using the Pytorch library and were trained on Nvidia GTX V100 with 16 GB memory. Training was done using ADAM algorithm[35] with an initial learning rate of 1e-4 and batch size of 2. We set λ_cyc=10, λ_struct=0.5 and λ_seg=5 in the training. The appropriate values for these hyper-parameters were selected empirically from the T2w MR parotid dataset set (see Supplementary document Sec. I). The learning rate was kept constant for the first 30 epochs and decayed to zero in the next 30 epochs.

The generator architectures were adopted from DC-GAN [36]. The generators ($G_C^M$ and $G_M^C$) consisted of two stride-2 convolutions, 9 residual blocks and two fractionally strided convolutions with half strides and used tanh activation following the last convolutional layer to produce output images. The discriminator networks (D_M, D_C and D_struct) were composed of 5 convolutional layers with a kernel size of 4 ×4 pixels that resulted in feature maps of size 64, 128, 256, 512, 1 in each layer of these networks. Discriminators were implemented as 70 × 70 pixels patchGAN [37].

The segmentation network was based on the U-net [38] with batch normalization added after each convolution filter. Feature encoders, E_M and $E_C^M$ of the segmentation network were implemented using 2 successive operations of convolution using kernels of size (3×3), batch normalization and ReLu activation followed by max pooling. Four max-pooling operations were used in the encoder structure for subsampling the image feature maps leading to the feature sizes of 64, 128, 256 and 512. Skip connections from both the encoder networks layers are used to combine with the decoder layer features to prevent segmentation blurring. The decoder network (DE) was implemented using four unpooling operations to upsample the features back to the original image resolution. We have made our code available².

F. Evaluation Metrics

Per-organ segmentation accuracies were measured using Dice similarity coefficient (DSC), computed from voxel-wise true positives (TP), false positives (FP) and false negatives (FN) $\left(DSC=\frac{2\times TP}{FP+2\times TP+FN}\right)$, and Hausdorff Distance at 95^th percentile (HD95) as suggested in [39]. We also computed an overall DSC that is an average DSC over all the structures segmented in a given scan.

G. Compared methods

We compared our method against multiple UDA segmentation methods including the CycadaGAN[16], segmentation structure matching (SA)[28], segmentation entropy matching (ADVENT)[21], SynSeg-Net[3], and SIFA[17]. Also, we compared against the CycleGAN[11] with a U-net[38] and the UNIT[40] with a U-net. SynSeg[3] and SIFA[17] are methods that have been developed and applied to medical image segmentation, while all other compared methods were developed for analyzing natural images. Both SA and ADVENT methods employed additional segmentation matching based adversarial losses for domain adapted segmentation training. We also computed the performance on a model trained with only CT (source) images without domain adaptation. Moreover, we compared performance against supervised MRI segmentation model to ascertain the accuracy upper limit. The supervised MRI segmentation training was done by using the MRI validation set as the training set.

Default implementation of the various networks as available from the authors’ were used. All networks, including ours, were trained and tested using identical datasets with the same hyperparameter settings (learning rates and batch size).

IV. Experiments and Results

We evaluated multiple organ and tumor segmentation from four different MRI sequences arising from two external institution (with two MR sequences) and two private and internal institution datasets. All networks were trained using 256×256 CT and MRI image patches enclosing the SOIs. All MR scans were standardized to remove patient-dependent signal intensity variations [41] and normalized to a range [−1, 1] for meaningful computation of global and joint distribution adversarial losses.

Ablation experiments were done to assess the utility of the proposed structure discriminator $\left(L_{\mathit{\text{struct}}}^D\right)$, the global intensity discriminator (L_adv) and cycle losses (L_cyc). We also evaluated alternative network design choices for computing the structure adversarial GAN losses by using (a) only segmentation probability maps, (b) joint distribution representation using multi-channel segmentation probability maps, (c) a single encoder-decoder segmentation network, and (d) SOI-specific discriminators.

A. MRI abdomen organs dataset

1). Data:

Twenty MRIs (T1-DUAL in-phase and T2w spectral pre-saturation inversion recovery or SPIR) acquired on a 1.5T Philips machine from the Combined Healthy Abdominal Organ Segmentation (CHAOS) challenge data [42] were used to generate segmentation of liver, left and right kidneys, and spleen. Ten MRIs were used for training (without labels) and validation and the remaining 10 scans were used for testing. Both MRI sequences were acquired to perform fat suppression. These data sets were acquired using a 1.5T Philips MRI, with an image resolution of 256 × 256 pixels, slice thickness that ranged between 5.5mm to 9mm (average of 7.84mm). Additional details of the MR sequences are in supplemental document Sec.I.A.

Thirty contrast-enhanced portal venous phase CT scans with expert segmentations were obtained from a completely different dataset [43] for UDA training. The CT images had a resolution of 512×512 pixels, an in-plane resolution ranging between 0.54mm ×0.54mm to 0.98mm ×0.98mm, and slice thickness ranging between 2.5mm to 5.0mm. CT images were cropped to contain only body region and then resampled to 256×256 to have the same resolution as the MRI images. Following histogram standardization, T1w and T2w MRI were clipped to the range of [0, 1136] and [0, 1814] using the 95th percentile of the reference MRI intensity values, respectively. Separate segmentation networks were trained for the T1w and T2w MRI using 256×256 pixels image patches. These patches were extracted from 8000 T1w, 7872 T2w MRI slices, and 14038 CT slices.

We also evaluated the feasibility of our method to segment T2w MRIs acquired on different scanners with various repetition times (TR), echo times (TE), with and without fat suppression from external institution TCIA-LIHC [44] dataset. Six patients were downselected from a total of 97 patients; exclusion criteria were only CT scans (N=57), absence of T2w FSE/TSE MRI (N=22), motion artifacts (N=6), and large liver tumors(N=6). Table I shows the sequence details from the CHAOS and the six patients from TCIA-LIHC dataset.

TABLE I.

T2w MRI scanning parameters used in the analysis. ETL: Echo train length; TE: Echo time; TR; repetition time; FatSup: Fat suppressed; FSE: Fast spin echo; TSE: Turbo spin echo; RT: Respiratory trigger; Nav: navigator; FatSat: Fat saturation

Dataset	Series	Manufacturer	Magnet	ETL	TE ms	TR ms	Flip angle
CHAOS	SPIR FatSup	Philips	1.5T	24	70	1930	90°
DD-A4NJ	FSE RT	Siemens	1.5T	21	89	5233	180°
DD-A1ED	FSE FatSat RT	GE	3T	12	83.90	10000	90°
DD-A4NF	TSE Nav	Siemens	1.5T	17	76	3386	150°
DD-A113	FSE	GE	1.5T	10	78.9	7500	90°
DD-A114	FSE	GE	1.5T	16	86.8	6000	90°
K7-AAU7	FSE RT	Philips	1.5T	85	80	563	90°

2). Volumetric segmentation accuracies:

The volumetric DSC segmentation accuracies for the multiple abdominal organs generated from the T2w and T1w MRI sequences on the testset are shown in Table II. A model trained with only CT images (w/o adaptation) was unable to produce clinically usable segmentations on MRI. PSIGAN achieved better overall average (computed over all organs) DSC of 0.90 on T2w and 0.87 on T1w MRI and a HD95 of 7.80mm on T2w and 6.90mm on T1w MRI than all other methods except supervised MRI segmentation. PSIGAN accuracy was only slightly lower than the fully supervised MRI segmentation.

TABLE II.

Overall segmentation accuracy on CHAOS dataset. Liver-Lv, Spleen-Sp, Left kidney-LK, Right kidney-RK. Overall average (Avg) is also shown.

Method		T2w-SPIR MRI (fat suppressed) (N=10)										T1w-DUAL in phase MRI (fat suppressed) (N=10)
		DSC ↑					HD95 mm ↓					DSC ↑					HD95 mm ↓
		LV	SP	LK	RK	Avg.	LV	SP	LK	RK	Avg.	LV	SP	LK	RK	Avg.	LV	SP	LK	RK	Avg.
W/o Adaptation	Avg.	0.08	0.23	0.26	0.01	0.15	64.52	71.12	47.48	72.70	63.96	0.00	0.00	0.00	0.00	0	97.03	141.26	89.47	108.45	109.05
	Std.	0.12	0.15	0.26	0.02		34.21	22.23	19.29	10.85		0.00	0.00	0.00	0.00		36.47	42.04	50.05	25.06
Supervised	Avg.	0.92	0.87	0.92	0.91	0.91	11.13	6.26	4.78	4.26	6.61	0.92	0.87	0.85	0.86	0.88	7.24	5.44	5.64	4.67	5.75
	Std.	0.03	0.07	0.03	0.03		8.31	2.19	5.34	1.68		0.04	0.04	0.14	0.18		2.48	1.18	3.28	1.90
CycleGAN	Avg.	0.86	0.75	0.88	0.87	0.84	24.59	13.51	14.26	6.08	14.61	0.82	0.83	0.63	0.61	0.72	10.40	9.45	10.72	20.26	12.71
	Std.	0.09	0.17	0.03	0.05		11.50	16.09	10.36	1.74		0.07	0.05	0.14	0.24		2.93	10.07	5.01	9.16
UNIT	Avg.	0.87	0.76	0.91	0.88	0.86	15.85	15.45	7.47	5.87	11.16	0.89	0.81	0.64	0.62	0.74	11.39	10.64	11.45	12.59	11.52
	Std.	0.08	0.23	0.03	0.04		9.63	15.80	7.46	1.68		0.02	0.07	0.17	0.23		6.62	7.84	5.66	5.82
CycaDA	Avg.	0.88	0.77	0.86	0.86	0.84	12.34	12.24	6.86	8.56	10.00	0.85	0.73	0.71	0.70	0.75	11.46	12.75	9.13	17.13	12.62
	Std.	0.08	0.18	0.03	0.06		15.82	14.70	1.43	1.80		0.04	0.12	0.21	0.19		9.79	13.77	3.10	4.31
SA	Avg.	0.86	0.80	0.89	0.89	0.86	18.18	12.32	10.09	9.14	12.43	0.89	0.73	0.72	0.79	0.78	11.67	11.33	10.30	10.33	10.91
	Std.	0.12	0.07	0.07	0.03		11.47	11.84	5.25	4.67		0.03	0.10	0.10	0.10		2.86	9.67	2.62	2.61
ADVENT	Avg.	0.89	0.79	0.81	0.80	0.82	12.58	13.76	11.50	11.65	12.37	0.87	0.84	0.76	0.79	0.82	14.55	11.72	10.22	10.62	11.78
	Std.	0.08	0.03	0.03	0.12		4.81	12.21	2.26	5.26		0.03	0.08	0.06	0.08		6.08	10.32	1.78	1.44
SynSeg	Avg.	0.88	0.77	0.89	0.85	0.85	21.04	12.63	10.23	6.28	12.55	0.89	0.85	0.73	0.70	0.79	8.98	8.76	9.13	11.95	9.71
	Std.	0.08	0.19	0.03	0.10		11.97	13.08	9.05	2.44		0.03	0.05	0.09	0.19		2.82	5.01	3.10	5.51
SIFA	Avg.	0.89	0.77	0.90	0.89	0.86	19.20	13.56	7.28	5.78	11.46	0.90	0.85	0.77	0.78	0.83	9.55	7.45	8.35	12.67	9.51
	Std.	0.09	0.22	0.02	0.03		13.01	16.53	1.21	1.44		0.02	0.05	0.12	0.07		2.92	4.02	1.76	7.55
PSIGAN	Avg.	0.91	0.87	0.91	0.90	0.90	11.15	8.34	6.81	4.88	7.80	0.92	0.87	0.83	0.84	0.87	7.41	5.87	7.62	6.70	6.90
	Std.	0.03	0.02	0.03	0.06		3.38	6.67	4.66	1.35		0.02	0.03	0.10	0.06		2.76	1.95	1.78	1.44

Fig. 4 shows example segmentations generated by the various methods from T1w and T2w MRI. Segmentations produced without adaptation and supervised method are shown in Supplementary Fig.2. As shown, the PSIGAN segmentations are nearly indistinguishable from the expert’s segmentations.

Fig. 4.

Segmentation performance of different methods on mdixon T1w and T2w MRI. The overall DSC computed for all organs on this patient is shown in the top right corner of images.

3). Evaluation on TCIA-LIHC dataset:

PSIGAN produced an average DSC accuracy of 0.87 and an average HD95 accuracy of 8.59mm on this dataset, which is close to that achieved on the CHAOS dataset (Table. II). PSIGAN segmentations were highly similar to expert delineations on all six cases with highly varying MR tissue contrasts (Supplementary Fig. 5).

4). MR to CT UDA segmentation:

We also evaluated our and other approaches to segment CT images by using either T1w or T2w MRI as the source modality. Validation was done using 15 CTs and testing was done on remaining 15 cases. Fig. 5 shows segmentations computed on a representative CT case. The corresponding segmentations without adaption and with supervised segmentation are shown in Supplementary Fig. 6. The DSC and HD95 accuracies of all methods computed from the test set are in Supplementary Table I. PSIGAN produced an overall average DSC of 0.90 and HD95 of 10.35mm when performing T2w to CT UDA and an average DSC of 0.89 and HD95 of 10.30mm when performing T1w to CT UDA segmentation. The next closest method SIFA produced a lower average DSC of 0.86 from T2w to CT and 0.85 from T1w to CT UDA segmentations and higher average HD95 of 14.26mm and 16.20mm for T2w to CT and T1w to CT UDA segmentations, respectively.

Fig. 5.

Representative segmentations produced by different methods on CT when performing T1w to CT (top row) and T2w to CT UDA segmentation. The overall DSC accuracies for each method are also shown.

B. MRI parotid glands dataset

1). Data:

A private institution head and neck MRI dataset consisting of 162 T2w fat suppressed (T2wFS) head and neck MRIs and obtained from 57 patients who were scanned before and every week during radiation therapy was analyzed. Eighty five MRIs from 30 patients that resulted in 14000 MRI 2D slices were used in training (expert-segmentations removed) and validation (with expert-segmentations added). Remaining 77 MRIs from 27 patients were used for testing. Ninety six head and neck CT scans combining 48 private and 48 opensource public domain database for computational anatomy (PDDCA) [45] were used as the source domain. MRI images were clipped to the range of [0, 1651] using the 95th percentile of the intensity values of the reference scan following image standardization. Two-dimensional patches of size 256 × 256 pixels containing the head and neck regions obtained after cropping portions outside of the body, resulting in 15000 CT and 14000 MRI images were used for training. Ablation experiments were done using this dataset, because it had the most MRIs available.

2). Volumetric segmentation accuracies:

Table. III shows the segmentation accuracies for the left and right parotid glands using the various methods including, supervised MRI training and MRI segmentation obtained using a CT network trained without domain adaption. The CT model trained without any domain adaptation was unable to generate segmentation on MRI. PSIGAN produced an average DSC of 0.82 and the lowest H95 of 3.06mm, which was slightly lower than supervised method’s accuracy with DSC of 0.84 and HD95 of 2.26mm. The next best method SIFA reached an average DSC of 0.72 and HD95 of 4.99mm. Fig. 6 shows segmentations generated on a representative test case using the compared methods; the corresponding segmentations produced without adaptation and with the supervised method are in Supplementary Fig. 4. As seen, PSIGAN segmentations were nearly indistinguishable from the expert contours.

TABLE III.

Overall segmentation accuracy on MRI parotid test dataset. Left parotid - LP, right parotid - RP.

Method	Test T2wFS MRI (N=77)
	DSC ↑			HD95 mm ↓
	RP	LP	Avg.	RP	LP	Avg.
W/o Adaption	0.00±0.00	0.00±0.00	0.00	87.81±18.89	85.45±19.94	86.63
Supervised	0.84±0.06	0.84±0.04	0.84	2.24±0.48	2.28±1.31	2.26
CycleGAN	0.55±0.09	0.51±0.11	0.53	8.22±4.81	7.38±2.29	3.90
UNIT	0.66±0.06	0.62±0.10	0.64	6.91±5.35	5.91±1.89	6.41
CycaDA	0.70±0.08	0.69±0.09	0.70	5.59±2.40	4.82±1.44	5.21
SA	0.74±0.07	0.71±0.07	0.73	5.05±1.74	4.94±1.58	5.00
ADVENT	0.73±0.12	0.71±0.11	0.72	4.66±2.16	5.06±1.65	4.86
SynSeg	0.67±0.09	0.65±0.09	0.66	5.27±3.56	6.08±2.73	5.68
SIFA	0.73±0.08	0.71±0.06	0.72	4.95±1.55	5.03±1.47	4.99
PSIGAN	0.82±0.03	0.81±0.05	0.82	2.98±1.01	3.14±1.17	3.06

Fig. 6.

Segmentation performance of different methods on T2wFS MRI. Red contour indicates the manual segmentation and the yellow contour indicates the algorithm segmentation. The overall DSC is shown in the top right corner of the images.

C. MRI lung tumor dataset

1). Data:

A private institution dataset with 75 T2-weighted turbo spin-echo (T2wTSE) MRIs obtained from 27 non-small cell lung cancer (NSCLC) cancer patients scanned before and every week during conventional fractionated external beam radiotherapy of 60 Gy was analyzed. Motion-robust two-dimensional axial images were acquired by using respiratory triggering on a 3T Philips Ingenia scanner (Medical Systems, Best, Netherlands). This is the same dataset as used in our prior work [10]. MRI images were clipped to the range of [0, 1198] using the 95th percentile of the reference MRI intensity values following image standardization. Training was done on 9696 unlabeled 2D image patches containing lung tumor extracted from 35 longitudinal MRI scans of 5 patients, while independent testing was done on the remaining 40 MRI scans from 22 patients. The CT source domain data was obtained from 377 expert-segmented CT scans of NSCLC patients and available from the Cancer Imaging Archive (TCIA) [46]. Training was done using 32, 000 image patches of size 256×256 pixels containing lung tumor.

2). Volumetric segmentation accuracies:

Table IV shows the lung tumor segmentation accuracies achieved by the various methods. The CT segmentation model without domain adaptation failed to generate segmentations on MRI. PSIGAN was more accurate than all except the supervised MRI segmentation method. Both tumor-aware [10] and SIFA produced a lower accuracy than PSIGAN. Fig. 7 shows two representative examples with algorithms’ segmentation together with expert delineations. The corresponding segmentations without adaption and with supervised segmentation are in Supplementary Fig. 3.

TABLE IV.

Segmentation accuracy on T2wTSE MRI (Lung tumor) test set.

Method	Test (N=40)
	DSC	HD95 mm
W/o Adaption	0.00±0.00	∞
Supervised	0.80±0.09	7.05±3.66
CycleGAN	0.64±0.20	17.83±15.43
UNIT	0.70±0.16	14.31±12.26
Cycada	0.70±0.18	14.17±12.91
SA	0.71±0.15	12.42±11.38
ADVENT	0.72±0.18	12.30±11.93
SynSeg	0.69±0.20	15.70±11.98
SIFA	0.72±0.15	11.97±6.12
Tumor-aware	0.72±0.16	12.88±11.08
PSIGAN	0.77±0.10	7.90±4.40

Fig. 7.

MRI lung tumor segmentation with DSC accuracies of multiple methods. Red contour is expert’s, yellow is algorithm segmentation.

D. Differences in feature maps extracted using global and joint distribution discriminator

Fig. 8 shows four randomly chosen feature maps of the first convolutional layer from the global intensity discriminator that used only images (Fig. 8(c)), a discriminator that used only segmentation probability maps (Fig. 8(d)), and structure discriminator that used a concatenation of image and aggregated segmentation probability map (Fig. 8(e)). We visualized the first layer features due to their proximity to the input images and to ascertain what low-level features were relevant for the discriminator. As seen, the image intensity matching global intensity discriminator extracts features without a clear focus on any part of the image (Fig. 8(c)), while the segmentation probability matching discriminator amplified features at the SOI boundaries (Fig. 8(d)), thereby, emphasizing organ geometry. On the other hand, feature responses are higher both within and at SOI boundaries, with a slight emphasis on some background structures (e.g. bottom row Fig. 8(e)) when using the structure discriminator. As a result, our method allows the structure discriminator to focus on both geometry and the content within the SOIs, while also preserving some background features.

Fig. 8.

Feature maps from first convolutional layer and computed with global intensity discriminator using (c) image, (d) aggregated segmentation probability, and (e) structure discriminator using joint distribution of image and aggregated segmentation probability map. The T2w SPIR MRI (a) and the aggregated segmentation probability map (b) are also shown. Feature response values are normalized to [0, 1] for visualization.

E. PSIGAN network design experiments

1). Structure discriminator:

We evaluated segmentation performance under the following structure discriminator network designs: (i) multi-channel segmentation probability only, (ii) aggregated segmentation probability map only, (iii) channel-wise concatenation of image and multi-channel segmentation probability, (iv) SOI specific structure discriminators that used channel-wise concatenation of image and segmentation probability map for each SOI, and (v) default PSIGAN, which used channel-wise concatenation of image and aggregated segmentation probability maps. These tests were done on both the MRI parotid and the T1w SPIR MRI abdomen dataset (CHAOS). Separate networks were trained in each one of these settings from scratch. The main difference between settings (i) and (ii) vs. the rest was the use of segmentation probability only in (i) and (ii) vs. joint-distribution matching of images and segmentation probabilities in (iii), (iv), and (v). Both (iii) and (v) used a single structure discriminator as opposed to K structure discriminators in (iv). Finally, whereas K channels for each organ was used to represent segmentation probability map in (iii), a single aggregated segmentation probability map was used in (v). All other losses including the global intensity discriminator and cycle consistency losses as used in PSIGAN were used all experiments.

As shown in Table V, the settings (iii), (iv), and (v) produced more accurate segmentations than (i) and (ii). Furthermore, the default setting of PSIGAN that aggregates the segmentation probabilities into a single map produced more accurate segmentation than all other methods. SOI-specific discriminator setting (iv) was similarly accurate as the setting using multi-channel segmentation probability maps in (iii). Segmentations produced on a representative case from the CHAOS dataset by all these methods is shown in Fig. 9. Segmentations on a case from the MR parotid dataset is in Supplementary Fig.9.

TABLE V.

Impact of using segmentation probability maps for adversarial training on segmentation accuracy.

	T1wFS MR Parotid (N = 77)			T1w SPIR MR Abdomen (N = 10)
Discriminator	Parotid right	Parotid left	Avg.	Liver	Spleen	Kidney left	Kidney right	Avg.
i. Multi-channel segmentation prob	0.74±0.07	0.73±0.06	0.74	0.89±0.02	0.84±0.05	0.79±0.12	0.80±0.07	0.83
ii. Aggregated segmentation prob	0.75±0.05	0.75±0.06	0.75	0.89±0.03	0.83±0.09	0.80±0.11	0.81±0.09	0.83
iii. Multi-channel segmentation prob + Image	0.78±0.05	0.77±0.05	0.78	0.90±0.03	0.85±0.05	0.80±0.10	0.82±0.07	0.84
iv. SOI-specific segmentation prob + Image	0.79±0.03	0.78±0.04	0.79	0.91±0.02	0.85±0.08	0.81±0.10	0.82±0.07	0.85
v. Aggregated segmentation prob + Image	0.82±0.03	0.81±0.05	0.82	0.92±0.02	0.87±0.03	0.83±0.10	0.84±0.06	0.87

Fig. 9.

Segmentation results for a representative case when using different ways of combining segmentation for computing adversarial loss. SOI-specific discriminator combines segmentation probability map for individual SOI with the images.

2). Segmentation network:

We evaluated performance when using single or split segmentor networks (Fig. 3). The structure discriminator and generator losses were computed using (5) and (6) for single segmentor, as opposed to using (9), (10) for the split segmentor case. In addition, we also measured the accuracy of segmentor $S_C^M$ for generating MRI segmentations instead of the default S_M network used in PSIGAN. Both split and single segmentor networks were trained from scratch till convergence with identical training, validation, and testing sets for both parotid and T1w SPIR abdomen organs segmentation.

The single segmentor configuration was less accurate than the split-segmentator (Table. II, and Table. III) with an average DSC of 0.85 on the T1w SPIR abdomen segmentation, and an average DSC of 0.80 on the parotid segmentation, respectively. The network parameters were slightly higher for the split-segmentor (42.97M) compared with single segmentor (38.99M). Adversarial losses for $G_C^M$ and D_struct during training are shown for split and single network configurations in Supplementary Fig. 10. As shown, the gap in losses for $G_C^M$ and D_struct stabilized faster for the split than the single segmentor configuration.

The $S_C^M$ network produced a much lower accuracy than both S_M and single-segmentor networks with an average DSC of 0.84 on the T1w SPIR abdomen segmentation, and an average DSC of 0.77 on the parotid segmentation, respectively. Organ-specific accuracies are in Supplementary Table II. Segmentations on representative examples using these three networks are shown in Supplementary Fig.7 and Fig.8 for both datasets.

F. Ablation experiments

1). Contribution of structure discriminator, global intensity discriminator, and cycle losses on accuracy:

The goal of this experiment was to evaluate the contribution of each loss on segmentation performance. Both structure and global intensity discriminator compute an adversarial loss and can train the generator independent of each other. Separate networks were trained from scratch until convergence using identical training, validation, and testing sets from the T2wFS MRI parotid datasets with the following loss settings:

1. CT to MR global intensity discriminator loss $L_{\mathit{\text{adv}}}^{C\to M}$ ( (4) ): Only a global adversarial loss was computed using the discriminator D_M for CT to MRI I2I translation.

2. CT to MR, MR to CT global intensity discriminators, and cycle losses $L_{\mathit{\text{adv}}}^{C\to M}+L_{\mathit{\text{adv}}}^{M\to C}+L_{\mathit{\text{cyc}}}$ ( (4), (11), (12) ): Cycle consistency, global adversarial losses computed using D_M and D_C (for MR to CT translation) were used.

3. Structure discriminator loss L_struct ( (9), (10) ): Joint distribution (image and aggregated segmentation probability map) matching adversarial loss was computed from D_struct.

4. Structure and CT to MR global intensity discriminator losses $L_{\mathit{\text{struct}}}+L_{\mathit{\text{adv}}}^{C\to M}$ ( (9), (10), (4) ): Losses from setting 1 and 3 were combined.

5. Structure, MR to CT global intensity discriminator, and cycle losses $L_{\mathit{\text{struct}}}+L_{\mathit{\text{adv}}}^{M\to C}+L_{\mathit{\text{cyc}}}$ ( (9), (10), (11), (12) ): Cycle loss and loss from setting 3 were combined.

6. Structure, CT to MR, MR to CT global intensity discriminators, and cycle losses $L_{\mathit{\text{struct}}}+L_{\mathit{\text{adv}}}^{C\to M}+L_{\mathit{\text{adv}}}^{M\to C}+L_{\mathit{\text{cyc}}}$: Default PSIGAN.

As shown in Table VI, L_struct loss alone leads to a clear performance improvement compared with the combination of global adversarial and cycle losses $\left(L_{\mathit{\text{adv}}}^{C\to M}+L_{\mathit{\text{adv}}}^{M\to C}+L_{\mathit{\text{cyc}}}\right)$. Addition of either the CT to MR global adversarial (setting 4) or cycle losses (setting 5) to the L_struct loss resulted in equivalent performance improvements. PSIGAN, which combines all the losses produced the most accurate segmentation. Segmentations produced on a representative case by the various networks trained in the aforementioned settings are shown in Supplementary Fig. 11.

TABLE VI.

Impact of each component in PSIGAN. RP: Right parotid; LP: Left parotid.

Setting	$L_{\mathit{\text{adv}}}^{C\to M}$	$L_{\mathit{\text{adv}}}^{M\to C}$	Lcyc	Lstruct	DSC
					RP	LP
1)	✓	✕	✕	✕	0.53±0.10	0.47±0.08
2)	✓	✓	✓	✕	0.65±0.09	0.63±0.10
3)	✕	✕	✕	✓	0.75±0.06	0.74±0.06
4)	✓	✕	✕	✓	0.77±0.05	0.77±0.04
5)	✕	✓	✓	✓	0.77±0.04	0.77±0.06
6)	✓	✓	✓	✓	0.82±0.03	0.81±0.05

2). Impact of structure discriminator on I2I translation:

Fig. 10 shows I2I translations produced by networks trained without and with structure discriminator on two example images, first one from the T2w SPIR MRI abdomen and the second from the T2wFS MRI parotid dataset. The source CT image is shown for reference. As shown, the addition of structure discriminator improved the contrast of SOIs with respect to background Fig. 10(c) and more accurately modeled the internal characteristics like the regularity in the organization of blood vessels in the liver, as appearing on real MRI. Additional I2I translation results are in Supplementary Fig. 12.

Fig. 10.

Impact of D_struct on the CT to MRI translation on parotid and abdomen T2w dataset.

Quantitative comparison of the distribution of MR signal intensities within the SOIs between the pseudo and real MRIs were computed using Kullback-Leibler (KL) divergence metric³. The method trained without the structure discriminator produced a higher KL-divergence of 1.5 within the parotid glands and 0.14 within liver. Whereas the method trained using structure discriminator produced a K-L divergence of 0.05 for parotid glands and 0.018 for liver.⁴

G. Evolution of segmentation probability maps during training

Fig. 11 shows the evolution of segmentation probability maps produced on representative examples from the three analyzed datasets during early epochs in training. As seen, the various organs and the tumor are correctly detected and the segmentation probabilities improve, becoming sharper and more focused with training. The higher probabilities are indicated by red color while low probabilities correspond to blue color. These maps clearly indicate their usefulness to constrain I2I translation after training only for a few epochs.

Fig. 11.

Example PSIGAN segmentation probability maps produced during the early training epochs. First column shows expert segmentations on MRI. The structure segmentation probabilities (blue - low probability, and red - high probability) steadily increase with training.

V. Discussion

We introduced PSIGAN, a joint distribution matching method for unsupervised domain adaptation-based multiple organ segmentation. Our method produced highly similar accuracy as supervised method on three different datasets, indicating its ability to learn without requiring target modality labeled image sets. Our method also showed feasibility to segment CT scans when performing T1w to CT and T2w to CT UDA segmentation.

Joint distribution matching has previously been used to constrain the space of GAN mappings by learning the bidirectional mapping from image to a scalar latent variable [23], [22] or from image to vector of class categories [24]. To our knowledge, ours is the first to perform joint distribution matching using pairs of images for UDA segmentation.

We conducted ablation and network design experiments to determine the utility of the joint distribution matching structure discriminator for both I2I translation and segmentation. Our results show clear performance improvements when using joint distribution matching adversarial losses. Consistent with prior work that computed adversarial losses using segmentation [21], [8] or with a joint translation-segmentation network [3], [14], we also found that features extracted using adversarial discriminators using only the segmentation maps showed a strong preference for SOI geometry, by emphasizing features on SOI boundary. On the other hand, the joint-distribution matching discriminator heavily weighted features corresponding to both the geometry and the appearance within the organs.

We also found that joint distribution formulations that used aggregated segmentation probability maps yielded more accurate segmentations than formulations using either multi-channel segmentation maps or SOI-specific structure discriminators. Performance improvement in the aggregated case could have resulted from increased contextual information from the other organs that was available to the structure discriminator.

Finally, the split segmentor showed a small improvement in accuracy over the single segmentor. However, the choice of the split segmentation network for MRI segmentation clearly impacted accuracy. More specifically, the network that was used for computing discriminator gradients was more accurate than the one used for computing the generator gradients.

Our idea of using segmentation probability maps to guide the translation is similar in principle to attention-guided translation methods [47], [48], which iteratively focus the domain translation network on to regions of interest and produce the desired translation. The main difference is that our method handles simultaneous translation of multiple target and background structures while attention-guided methods are typically restricted to transfiguring a single foreground. Also, as the optimization of the segmentation network is done in a supervised manner, pre-specified image to target relationships can be easily extracted through a fast converging network to constrain translation. Deriving unsupervised attention information on the other hand, would require pre-training of the self-attention network for several epochs before it can be combined with the generator as shown in [48].

A deficiency of our method as is common to most UDA methods is the inability to handle expert delineation variabilities across modalities that may arise as visibility of structures can vary across modalities. Preliminary evaluation of our method on an external dataset with a variety of MR contrasts and scanning parameters indicates that it is possibly robust to MR contrast variations. However, extensive validation and potential extension to handle large MR contrast variations on much bigger cohorts are needed and is work for future. Nevertheless, our method outperformed multiple state-of-the-art methods.

VI. Conclusion

We developed and evaluated a new unpaired domain adaptation segmentation approach using joint distribution matching structure discriminator for multiple organ segmentation on MRI datasets. Our approach outperformed multiple state-of-the art methods and demonstrated the value of structure discriminator in improving I2I translation and segmentation.