Dual Adversarial Attention Mechanism for Unsupervised Domain Adaptive Medical Image Segmentation

Abstract

Domain adaptation techniques have been demonstrated to be effective in addressing label deficiency challenges in medical image segmentation. However, conventional domain adaptation based approaches often concentrate on matching global marginal distributions between different domains in a class-agnostic fashion. In this paper, we present a dual-attention domain-adaptative segmentation network (DADASeg-Net) for cross-modality medical image segmentation. The key contribution of DADASeg-Net is a novel dual adversarial attention mechanism, which regularizes the domain adaptation module with two attention maps respectively from the space and class perspectives. Specifically, the spatial attention map guides the domain adaptation module to focus on regions that are challenging to align in adaptation. The class attention map encourages the domain adaptation module to capture class-specific instead of class-agnostic knowledge for distribution alignment. DADASeg-Net shows superior performance in two challenging medical image segmentation tasks.

I. INTRODUCTION

Automatic segmentation of medical images has attracted increasing attention in modern computer-aided diagnosis (CAD) [1], [2]. Although great advancements have been made with the emergence of deep learning techniques, building a reliable segmentation model for medical images remains challenging. The appearance of medical images can vary significantly with imaging modalities or acquisition parameters. Moreover, manually annotating ground truth for medical images is typically challenging and requires professional skills or knowledge. The lack of pixel-wise annotated data is a longstanding issue in medical image segmentation.

To address this problem, a feasible solution is to exploit the annotations from another modality via domain adaptation at the image level. By learning an image-to-image translation model between the target modality and a label-rich source modality, realistic data can be synthesized to train the segmenter for the target modality without pixel-wise annotations [3]–[5]. For instance, Huo et al. [3] presented SynSeg-Net to learn image-level translations between a target domain without annotations and a source domain with annotations. Synthetic data in the target domain are coupled with annotations in the source domain to train the segmentation model. Li et al. [5] proposed a simplified unsupervised image translation method (SUIT) for domain adaptation in semantic segmentation. Another promising solution is to align the hidden representation spaces of different domains via adversarial learning, i.e., feature-level domain adaptation [6]–[9]. For example, Dou et al. [6] introduced PnP-AdaNet to adapt segmentation networks between different modalities with a plug-and-play module.

Most of the above-mentioned domain adaptation methods concentrate on matching the global marginal distributions of different domains at the image or feature level. Such strategy treats all spatial positions equally in a class-agnostic fashion and (i) ignores the fact that some regions can be more easily misaligned than others; (ii) disadvantages minor classes; and (iii) discards domain-specific information that preserves discriminability and contains meaningful information for downstream tasks.

In this paper, we present a dual-attention domain-adaptative segmentation network (DADASeg-Net) to adapt the knowledge learned from a source domain with annotations to a target domain without annotations. This is realized by aligning the feature-level distributions of the source and target domains via spatial and class attention mechanisms, which regularize the domain adaptation module by adaptively weighting the adversarial loss at the feature level. The former forces the domain adaptation module to focus on the spatial regions that tend to be misaligned. The latter encourages the capturing of class-specific rather than class-agnostic knowledge for adaptation. We confirmed the efficacy of our method based on two challenging cross-modality medical image segmentation tasks: skull segmentation from MRI and cardiac substructure segmentation from CT.

II. RELATED WORK

A. Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) is broadly used to address the distributional shift issue between a source domain, where labels are available, and a target domain, where labels are unavailable. UDA transfers knowledge between two domains, with the assumption that the class labels across domains are consistent. UDA is of particular significance in medical image segmentation, where annotations are typically scarce. Earlier UDA methods include reweighting methods for reweighting source samples by their significance [10], [11], iterative methods for iteratively generating pseudo labels for target samples [12]–[14], common-representation methods for building a common feature space for two domains, and hierarchical Bayesian methods for factorizing domain-dependent latent representations [15], [16]. Inspired by generative adversarial networks (GANs) [17], recent UDA methods focus on distribution alignment via adversarial learning at data-level [3], [4], [18], [19] or feature-level [6], [9], [20], [21]. For example, Huo et al. [3] proposed to adapt the visual appearance of two domains with a CycleGAN [22]. Ouyang et al. [20] learned a common domain-agnostic representation space to obtain distribution-matched latent semantic features. Distribution alignment performed jointly at both image and feature levels further improves performance [7], [19]. Chen et al. [21] proposed to hierarchically harmonize transferability and discriminability through feature reweighting for cross-domain object detection. Luo et al. [23] introduced a category-level adversarial learning framework for semantic consistent domain adaptation.

B. Attention Mechanism

Attention mechanism, originally developed for natural language modeling [24], has been widely adopted for computer vision tasks, e.g., image captioning [25]–[28], semantic segmentation [23], [29]–[32], and object detection [33]–[35]. Attention mechanism is usually realized by selective reweighting of features based on their relevance to a specific task. For instance, Chen et al. [25] proposed SCA-CNN to incorporate spatial-wise and channel-wise attention in a CNN for image captioning. Fu et al. [32] presented a dual attention network to adaptively integrate local features with their global dependencies for scene segmentation. Wang et al. [36] introduced a non-local operation to capture the long-range spatial dependencies by computing an attention map. Huang et al. [29] proposed CCNet to adaptively capture long-range contextual information from full-image dependencies with a criss-cross attention module. Zhao et al. [37] introduced PSANet to leverage the contextual information for each individual position in a hidden representation space.

III. METHOD

A. Overview

For a source domain image set ${𝒳}_{\text{S}}$ with annotation ${𝒴}_{\text{S}}$ and a target domain image set ${𝒳}_{\text{T}}$ without annotation, our goal is to construct an auto-segmentation model for the target domain by leveraging the knowledge learned from the source domain.

To this end, a widely used strategy is to adapt the features captured from target domain images to those from source domain images by matching the entire hidden representation spaces of two domains [6], [7], [38], [39]. However, such strategy often extracts domain agnostic representations and discards domain specific information that not only preserves discriminability but is also meaningful for downstream tasks [23]. In this work, we encourage the domain adaptation module to capture meaningful information by calculating two attention maps: (i) a spatial attention map, and (ii) a class attention map. The former assigns high weights for spatial positions that appear to be poorly aligned in domain adaptation. The latter adjusts the weights for each class individually for discriminative distribution alignment. These two attention maps are finally combined and incorporated into a domain adaptation adversarial loss function, as introduced in Section III-B.

Based on proposed dual adversarial attention mechanism, we train our cross-modality segmentation model with source domain annotations, which will be detailed in Section III-C. The network architecture and training details will be elaborated in Section III-D and III-E, respectively.

B. Dual Adversarial Attention Mechanism

The proposed dual adversarial attention mechanism is illustrated in Figure 1. We use a segmenter to perform semantic segmentation for images from either the source or target domain, as well as a discriminator to match the distributions of semantic representations for different domains.

Fig. 1:

The proposed dual adversarial attention mechanism.

Specifically, the segmenter consists of two encoders $S_{\text{E}}^{\text{S}}$ and $S_{\text{E}}^{\text{T}}$ to respectively extract latent representations from the source and target domain images, and a shared decoder S_D to produce the segmentation masks from extracted features. The discriminator D adapts the semantic features produced by S_D to a common feature space via adversarial learning. However, unlike conventional domain adaptation methods that treat all elements of semantic features equally, our domain adaptation module focuses on specific regions determined by the spatial and class attention mechanisms.

The spatial attention mechanism encourages the domain adaptation module to focus on the regions that tend to be misaligned in the spatial dimension. To realize this, we incorporate a CycleGAN [22] into our framework to learn the image-level translations between two domains. Ideally, the semantic features extracted from a real image in one domain and its synthetic counterpart in another domain should be consistent. We realize this requirement with the following feature consistency loss function:

$$L_{\text{fc}}={\Vert S_{\text{E}}^{\text{S}}\left(x_{\text{s}}\right)-S_{\text{E}}^{\text{T}}\left({\widehat{x}}_{\text{s}\to \text{t}}\right)\Vert }_2^2+{\Vert S_{\text{E}}^{\text{T}}\left(x_{\text{t}}\right)-S_{\text{E}}^{\text{S}}\left({\widehat{x}}_{\text{t}\to \text{s}}\right)\Vert }_2^2,$$ (1)

where $x_{\text{s}}\in {𝒳}_{\text{S}}$ and $x_{\text{t}}\in {𝒳}_{\text{T}}$ denote the real images of source domain and target domain, respectively. ${\widehat{x}}_{\text{s}\to \text{t}}$ and ${\widehat{x}}_{\text{t}\to \text{s}}$ are the synthetic images translated from x_s and x_t by CycleGAN, respectively. In practice, however, the semantic features of real and synthetic images can still differ substantially at some spatial positions. We thus calculate the spatial attention map based on the inconsistency of semantic features captured from real and synthetic images at all spatial positions. The larger the inconsistency, the higher the attention weight. Let $S_{\text{D}}^{-2}$(●) denote the feature map generated by the layer before the output layer of segmenter, we calculate the spatial attention map with the operator f_spatial defined as

$$f_{\text{spatial}}\left(x_t\right)=\varphi \left(S_{\text{D}}^{-2}\left(S_{\text{E}}^{\text{T}}\left(x_t\right)\right),S_{\text{D}}^{-2}\left(S_{\text{E}}^{\text{S}}\left({\widehat{x}}_{\text{t}\to \text{s}}\right)\right)\right)),$$ (2)

where $x_{\text{t}}\in {𝒳}_{\text{T}}$ is a real target domain image, and ø(●,●) measures the dissimilarity between two semantic features across all spatial positions. In this work, ø(●,●) is defined as the cosine distance.

The class attention mechanism encourages a more discriminative distribution alignment. It produces an attention map for each class individually, which is calculated based on the probability of class occurrence. This is realized by using the operator f_class defined as

$$f_{\text{class}}\left(x_t\right)=\text{Softmax}\left(S_{\text{D}}\left(S_{\text{E}}^{\text{T}}\left(x_{\text{t}}\right)\right)\right).$$ (3)

Specifically, for each class, we calculate a class attention map based on the Softmax prediction of segmenter output across all spatial positions. The greater the likelihood a position is estimated as belonging to the c-th class, the higher the weight at this position is in the c-th attention map. In this way, the discriminator is guided to capture class-specific rather than class-agnostic knowledge in adaptation, which encourages more discriminative feature alignment as well as reduces the bias caused by inter-class discrepancy.

The spatial and class attention maps are combined and incorporated into our domain adaptation adversarial loss function. Specifically, the spatial attention map is tiled and then element-wise multiplied with the class attention map to generate a final activation map. This activation map is then multiplied with the intermediate output of D to produce the final output with a size of C × H × W, where C denotes the number of classes, and H and W indicate the height and width of training images, respectively. The output of discriminator can be interpreted as C individual slices, where the values of c-th slice represent whether the corresponding voxel of input features are from the source or target domains for c-th class. Following [40], we measure the discrimination and adversarial domain adaptation losses with the least squares loss function as follows:

$$\begin{array}{l}{L}_{\text{dis}}^{\text{f}}(D)={\left[D\left(S_{\text{D}}^{-2}\left(S_{\text{E}}^{\text{S}}\left(x_{\text{s}}\right)\right)\right)-1\right]}^2+\\ {\left[D\left(S_{\text{D}}^{-2}\left(S_{\text{E}}^{\text{T}}\left(x_{\text{t}}\right)\right)\right)+1\right]}^2,\end{array}$$ (4)

$$L_{\text{adv}}^{\text{f}}\left(S_{\text{E}}^{\text{T}},S_{\text{D}}\right)=\left[f_{\text{spatial}}\left(x_{\text{t}}\right)\otimes f_{\text{class}}\left(x_{\text{t}}\right)\right]\otimes {\left[D\left(S_{\text{D}}^{-2}\left(S_{\text{E}}^{\text{T}}\left(x_{\text{t}}\right)\right)\right)-1\right]}^2,$$ (5)

where $\otimes$ represents the element-wise multiplication operator.

C. Domain Adaptive Segmentation

The domain adaptation module forces the segmenter encoders $S_{\text{E}}^{\text{S}}$ and $S_{\text{E}}^{\text{T}}$ to embed the images of either the source or target domain to a common feature space. Naturally, the source domain annotations can be used to train the $S_{\text{E}}^{\text{S}}$ and S_D with the following loss function:

$$L_{\text{seg}}^{\text{s}}\left(S_{\text{E}}^{\text{S}},S_{\text{D}}\right)=\text{CE}\left(S_{\text{D}}\left(S_{\text{E}}^{\text{S}}\left(x_{\text{s}}\right)\right),y_{\text{s}}\right),$$ (6)

where $x_{\text{s}}\in {𝒳}_{\text{S}}$ and $y_{\text{s}}\in {𝒴}_{\text{S}}$ are respectively the source domain image and the corresponding annotation, and CE represents the cross-entropy loss function.

To avoid the segmenter being biased to the source domain, the synthetic target domain images are coupled with the source domain annotations to learn the $S_{\text{E}}^{\text{T}}$ and S_D. The loss function is formulated as

$$L_{\text{seg}}^{\text{t}}\left(S_{\text{E}}^{\text{T}},S_{\text{D}}\right)=\text{CE}\left(S_{\text{D}}\left(S_{\text{E}}^{\text{T}}\left({\widehat{x}}_{\text{s}\to \text{t}}\right)\right),y_{\text{s}}\right),$$ (7)

where ${\widehat{x}}_{\text{s}\to \text{t}}$ is the synthetic target domain image that is translated from the source domain image x_s.

D. Network Architecture

Our domain adaptation module consists of two segmenter encoders ($S_{\text{E}}^{\text{S}}$ and $S_{\text{E}}^{\text{T}}$), a shared segmenter decoder S_D and a discriminator D. The segmenter encoder contains three encoding blocks. Each encoding block consists of three convolutional layers, where the final one is with the stride of 2 to gradually reduce the resolution of feature maps. This is followed by three residual blocks [41] to increase non-linear modeling capability. The segmenter decoder is comprised of three residual blocks, three decoding blocks and an additional convolutional layer to produce the final output. Each decoding block is comprised of a transposed convolutional layer with a stride of 2, followed by two convolutional layers. For discriminator D, we adopt the U-Net [42] architecture. In addition, we append group normalization [43] after each convolutional layer and transposed convolutional layer. For activation function, we choose ReLU for the segmenter and Lealy ReLU with slope 0.2 for the discriminator as suggested in [44].

E. Training Details

Our framework consists of a CycleGAN [22] (which consists of two generators and two image discriminators), a segmenter and a feature discriminator. We split all subnetworks into 3 groups and trained them alternately. The training strategy is detailed in Algorithm 1. Specifically, at each iteration we first trained all discriminators with loss function $L_{\text{dis}}^{\text{I}}+L_{\text{dis}}^{\text{f}}$, where $L_{\text{dis}}^{\text{I}}$ is the image discrimination loss in the CycleGAN. Then we trained the segmenter with loss function $L_{\text{seg}}^{\text{s}}+{\lambda }_1{L}_{\text{seg}}^{\text{t}}+{\lambda }_2{L}_{\text{adv}}^{\text{f}}$. Finally, the generators in the CycleGAN were trained with loss function $L_{\text{cyc}}+{\lambda }_3{L}_{\text{adv}}^{\text{I}}+{\lambda }_4{L}_{\text{fc}}$ where L_cyc and $L_{\text{adv}}^{\text{I}}$ are the cycle-consistency and image adversarial losses in the CycleGAN.

We trained the model with mini-batches. Each batch contain 4 source domain images and 4 target domain images that randomly selected from the ${𝒳}_{\text{S}}$ and ${𝒳}_{\text{T}}$, respectively. We adopted the Adam algorithm [45] to train our model with a learning rate of 0.0001. Other hyper-parameters were set as λ₁ = 0.5, λ₂ = 0.01, λ₃ = 0.01, and λ₄ = 0.1.

Algorithm 1.

Training strategy

Input: Image sets ${𝒳}_{\text{S}}$ and ${𝒳}_{\text{T}}$, and annotation set ${𝒴}_{\text{S}}$

1: for t = 1 to T do

2:  Train the discriminators with loss $L_{\text{dis}}^{\text{I}}+L_{\text{dis}}^{\text{f}}$.

3:  Train the segmenter with loss $L_{\text{seg}}^{\text{s}}+{\lambda }_1{L}_{\text{seg}}^{\text{t}}+{\lambda }_2{L}_{\text{adv}}^{\text{f}}$.

4:  Train the generators with loss $L_{\text{cyc}}+{\lambda }_3{L}_{\text{adv}}^{\text{I}}+{\lambda }_4{L}_{\text{fc}}$.

5: end for

IV. EXPERIMENTS

A. Experimental Settings

To evaluate the efficacy of DADASeg-Net, we first tested it on a skull segmentation task based on MRI. In this task, we aim to automatically annotate bony structures for MR images by utilizing the bony annotations from CT. To this end, we built the training dataset with 50 CT and 50 MRI scans, which were respectively collected from the CQ500 database [46] and the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database [47]. Besides, the testing dataset includes 8 pairs of CT-MRI scans that collected from the ADNI database. For data preprocessing, we linearly aligned all CT and MRI scans by using FMRIB’s Linear Image Registration Tool (FLIRT) [48]. The aligned scans were then cropped to a same size of 176×184×146. Then, we linearly scaled the intensity values of all scans into [–1, 1]. The skull annotations of both training and testing data were acquired from CT images by intensity-based thresholding, and followed by manual corrections as needed. For paired CT-MRI data in the testing dataset, the bony annotations of CT scans can be shared with their MRI counterparts for quantitative evaluation.

In addition, we also tested DADASeg-Net on cardiac substructure segmentation based on CT. In this task, we aim to automatically delineate four cardiac substructures [the ascending aorta (AA), the left atrium blood cavity (LAC), the left ventricle blood cavity (LVC) and the myocardium of the left ventricle (MYO)] in CT images, while the annotations are defined in MRI modality. The experimental data came from the Multi-Modality Whole Heart Segmentation (MMWHS) Challenge 2017 dataset [49], which includes 20 CT and 20 MRI unpaired scans. We adopted the preprocessed data as described in [6]¹. Specifically, the training dataset contains 9,600 CT and 8,400 MRI samples, which were respectively cropped from 16 randomly selected CT and MRI scans. The rest 4 CT scans were used to build the testing dataset. The intensity values of all experimental data were z-score normalized, i.e., with zero-mean and unit variance.

To confirm the efficacy of our DADASeg-Net, we compared it with 5 state-of-the-art unsupervised domain-adaptative medical image segmentation methods, including SynSeg-Net [3], PnP-AdaNet [6], Ouyang et al. [20], SIFA [7] and ARL-GAN [19]. Segmentation performance was quantitatively measured with the Dice similarity coefficient (DSC) and the average symmetric surface distance (ASSD).

B. Segmentation Results

For skull segmentation, we first trained our method and all competing methods with 50 unpaired CT and MRI scans. Then all trained models were tested on 8 testing MRI scans. Table I shows the quantitative performance of all comparison methods. DADASeg-Net yields the best performance, outperforming the second best by 3.02 and 0.09 in terms of DSC and ASSD, respectively. Note that SIFA and ARL-GAN have similar frameworks with our method. They also perform the domain adaptations in image level and feature level simultaneously with a cross-modality image synthesis submodule and an adversarial feature discriminator. However, the feature space alignment of our framework is guided by the proposed dual adversarial attention mechanism, which is a key distinction over existing methods. The performance improvement shown in Table I clearly indicates the efficacy of DADASeg-Net in comparison with other methods.

TABLE I:

Quantitative results of skull segmentation. Best results are marked in bold.

Methods	DSC(%)	ASSD
SynSeg-Net	78.32 ± 2.58	1.48 ± 0.17
PnP-AdaNet	74.01 ± 2.31	1.79 ± 0.15
Ouyang et al.	78.64 ± 3.27	1.52 ± 0.23
SIFA	76.60 ± 1.74	1.67 ± 0.09
ARL-GAN	81.05 ± 1.13	1.27 ± 0.08
DADASeg-Net	84.07 ± 1.51	1.18 ± 0.11

Fig. 2 shows some representative results of all comparison methods (b)-(h), as well as the (a) input images and (g) ground truth. Compared with other methods, the skull masks produced by DADASeg-Net are more complete and with less artifacts.

Fig. 2:

Visual results for skull segmentation. Compared with other methods, DADASeg-Net produces smoother skull segmentation mask.

We also compared DADASeg-Net with other methods in cardiac substructure segmentation. In this task, we trained all methods on unpaired cardiac CT-MRI data with manual annotations that only defined in MRI modality. Then we tested the trained models on 4 testing CT scans. In addition, we trained a U-Net [42] in a fully supervised fashion (i.e., with manual annotations for CT) for reference.

Table II summarizes the quantitative comparison results. Our DADASeg-Net yields the best performance in most cases and on average. However, the superior performance of U-Net implies that the unsupervised domain adaptive segmentation models still have room for improvement. The representative segmentation results shown in Fig. 3 also exhibit the superiority of DADASeg-Net. Compared with other algorithms, DADASeg-Net is able to produce more complete and smooth segmentation masks.

TABLE II:

Quantitative results of cardiac segmentation. Best results (except reference) are marked in bold.

		SynSeg-Net	Ouyang et al.	PnP-AdaNet	SIFA	ARL-GAN	Proposed	U-Net
DSC (%)↑	AA	71.6 ± 13.9	82.4 ± 6.3	74.0 ± 20.9	81.3 ± 27.6	71.3 ± 8.6	87.0 ± 2.1	69.2 ± 36.8
	LAC	69.0 ± 28.0	74.2 ± 16.2	68.9 ± 14.4	79.5 ± 9.0	80.6 ± 9.3	80.7 ± 9.1	91.7 ± 3.3
	LVC	51.6 ± 22.7	61.1 ± 26.9	61.9 ± 15.2	73.8 ± 6.4	69.5 ± 18.1	77.9 ± 9.0	89.4 ± 6.0
	MYO	40.8 ± 17.1	75.8 ± 8.7	50.8 ± 17.6	61.6 ± 4.8	81.6 ± 7.9	61.2 ± 9.1	90.6 ± 5.1
	AVG	58.2 ± 22.3	73.4 ± 21.4	63.9 ± 18.6	74.1 ± 18.8	75.7 ± 12.9	76.7 ± 12.4	85.2 ± 21.0
ASSD (mm)↓	AA	11.7 ± 5.0	3.3 ± 1.1	12.8 ± 12.6	7.9 ± 9.3	6.3 ± 2.4	4.5 ± 3.0	4.4 ± 4.6
	LAC	7.8 ± 2.1	3.9 ± 1.8	6.3 ± 3.5	6.2 ± 1.9	5.9 ± 2.6	5.6 ± 2.7	2.0 ± 0.6
	LVC	7.0 ± 1.8	9.8 ± 5.3	17.4 ± 4.5	5.5 ± 1.3	6.7 ± 3.9	4.7 ± 1.6	2.6 ± 1.3
	MYO	9.2 ± 4.3	5.6 ± 1.2	14.7 ± 4.5	8.5 ± 0.7	6.5 ± 1.7	5.5 ± 2.3	1.9 ± 0.6
	AVG	8.9 ± 3.8	5.7 ± 3.9	12.8±7.6	7.0 ± 5.7	6.4 ± 2.0	5.1 ± 2.5	2.7 ± 2.6

Fig. 3:

Visual results for cardiac segmentation. The objects with colors yellow, green, blue and red respectively indicate AA, LAC, LVC, MYO.

C. Ablation Study

We conducted the ablation study to further analyze the role of several key components of proposed framework. Specifically, we trained 2 variants of the our model by removing the feature consistency loss $L_{\text{consis}}^{\text{f}}$ (V1) and the feature adversarial learning losses $L_{\text{adv}}^{\text{f}}+L_{\text{dis}}^{\text{f}}$ (V2). In addition, to measure the effect of proposed dual adversarial attention mechanism, we trained 3 additional variants by removing the class attention (V3), spatial attention (V4) and both of them (V5), respectively.

Table III summarizes the evaluation results. We can observe that V2 achieves the worst performance, which confirms the effectiveness of feature-level adaptation in cross-modality segmentation. Moreover, the feature consistency constraint also contributes to the performance improvement by preserving the anatomical structure information in image-level adaptation. On the other hand, the results also suggest that conducting spatial or class attention separately (V3 and V4) achieves only marginal improvements. However, by integrating two attention mechanisms together, our attention-based feature adversarial learning improves the segmentation accuracy effectively, which further confirms the efficacy of our dual adversarial attention mechanism.

TABLE III:

Effects of loss functions and proposed dual adversarial attention mechanism.

Methods	DSC(%)↑	ASSD(mm)↓
V1 (w/o $L_{\text{consis}}^{\text{f}}$)	81.72 ± 1.85	1.33 ± 0.14
V2 (w/o $L_{\text{adv}}^{\text{f}}+L_{\text{dis}}^{\text{f}}$)	79.81 ± 1.60	1.34 ± 0.13
V3 (w/o class attention)	82.08 ± 2.84	1.24 ± 0.19
V4 (w/o spatial attention)	81.53 ± 1.60	1.42 ± 0.12
V5 (w/o spatial & class attention)	81.03 ± 1.65	1.35 ± 0.13
Proposed	84.07 ± 1.51	1.18 ± 0.11

D. Attention Map Analysis

To further study the role of proposed dual adversarial attention mechanism, we visually analyzed the generated attention maps. Figure 4 and 5 show the representative attention maps calculated in skull segmentation and cardiac substructure segmentation tasks, respectively. The higher the attention weight, the warmer the color. We can observe that the high weights in the spatial attention map are mainly distributed around the boundaries of segmentation targets. These regions have the tendency to be misclassified, which deserve more attention in adaptation. On the other hand, the class attention maps basically cover the segmentation targets, which encourage the domain adaptation module to capture class-specific information from the region of interest for each class individually. Such class-specific knowledge not only preserve the discriminability, but also contains meaningful information that would be beneficial to downstream tasks.

Fig. 4:

Representative attention maps in skull segmentation.

Fig. 5:

Representative attention maps in cardiac substructure segmentation.

E. Model Analysis

In this section, we further analyze several properties of proposed method.

1. Parameter Analysis: As shown in Algorithm 1, there are four hyper-parameters involved in model training. We empirically set λ₁ = 0.5, λ₃ = 0.01, and investigated the selection of λ₂ and λ₄ on the validation set of cardiac substructure segmentation task. Specifically, we searched for the optimal λ₂ and λ₄ from {10⁻¹, 10⁻², 10⁻³}. Figure 6 shows the grid search results, according to which we chose λ₂ = 0.01 and λ₄ = 0.1.

2. Sample Complexity Analysis: We performed sample complexity analysis by training the proposed method by using only a fraction of the training samples. As shown in Figure 7, the skull segmentation performance of DADASeg-Net stabilizes with 40%∼50% training samples. In contrast, the performance for cardiac segmentation keeps increasing as more training samples are used.

3. Feature Visualization: To further study the effectiveness of DADASeg-Net in reducing the domain gap between different imaging modalities, we extracted the source and target domain features respectively using $S_{\text{E}}^{\text{S}}$ and $S_{\text{E}}^{\text{T}}$, and visualized the feature distribution in 2D space with t-SNE [50]. As shown in Figure 8, the domain discrepancy between the source and target modalities is effectively reduced, and the distribution of features from different domains is gradually aligned during model training.

Fig. 6:

Grid search results for hyper-parameters λ₂ and λ₄.

Fig. 7:

Performance of DADASeg-Net on skull and cardiac segmentation tasks with respect to training samples.

Fig. 8:

Evolution of source (red) and target (blue) domain features during model training, visualized using t-SNE.

V. CONCLUSION

In this work, we presented an unsupervised domain adaptive medical image segmentation framework. The major contribution that differentiates our work from others is its novel dual adversarial attention mechanism. Unlike the commonly used global alignment strategy [6]–[9], DADASeg-Net forces the domain adaptation module to focus on meaningful regions rather than treating all positions equally. This is realized by adaptively weighting the adversarial loss based on the proposed dual attention mechanism. Compared with prior methods with dual attention in spatial and channel dimensions [25], [32], DADASeg-Net aims to adaptively reduce the domain gap between the source and target domains by regularizing the domain adaptation module space-wise and class-wise. Specifically, the spatial attention module concentrates on spatial positions that tend to be misaligned in adaptation, judging based on the inconsistency of semantic features during image-level adaptation. The class attention module encourages the domain adaptation module to capture class-specific knowledge for more discriminative feature-level adaptation. The efficacy of DADASeg-Net is confirmed with experimental results based on challenging cross-modality skull and cardiac substructure segmentation tasks.