Dynamic Cross-Task Representation Adaptation for Clinical Targets Co-Segmentation in CT Image-Guided Post-Prostatectomy Radiotherapy

Abstract

Adjuvant and salvage radiotherapy after radical prostatectomy requires precise delineations of prostate bed (PB), i.e., the clinical target volume, and surrounding organs at risk (OARs) to optimize radiotherapy planning. Segmenting PB is particularly challenging even for clinicians, e.g., from the planning computed tomography (CT) images, as it is an invisible/virtual target after the operative removal of the cancerous prostate gland. Very recently, a few deep learning-based methods have been proposed to automatically contour non-contrast PB by leveraging its spatial reliance on adjacent OARs (i.e., the bladder and rectum) with much more clear boundaries, mimicking the clinical workflow of experienced clinicians. Although achieving state-of-the-art results from both the clinical and technical aspects, these existing methods improperly ignore the gap between the hierarchical feature representations needed for segmenting those fundamentally different clinical targets (i.e., PB and OARs), which in turn limits their delineation accuracy. This paper proposes an asymmetric multi-task network integrating dynamic cross-task representation adaptation (i.e., DyAdapt) for accurate and efficient co-segmentation of PB and OARs in one-pass from CT images. In the learning-to-learn framework, the DyAdapt modules adaptively transfer the hierarchical feature representations from the source task of OARs segmentation to match up with the target (and more challenging) task of PB segmentation, conditioned on the dynamic inter-task associations learned from the learning states of the feed-forward path. On a real-patient dataset, our method led to state-of-the-art results of PB and OARs co-segmentation. Source code is available at https://github.com/ladder-xjtu/DyAdapt.

I. Introduction

As a standard treatment of localized prostate cancer, radical prostatectomy is a surgical procedure that removes the entire prostate gland and some surrounding small tissues. For better prognosis, adjuvant and salvage radiotherapy after the surgery is recommended for patients with high-risk pathological features and those who develop a rising prostate-specific concentration during post-surgery follow-up, respectively [4]. The efficacy of radiotherapy requires carefully designed planning, including the accurate delineation of the clinical target volume (CTV) and surrounding organs at risk (OARs), e.g., from computed tomography (CT) images, to maximize target irradiation while minimizing side effects [5], [6]. The CTV for post-prostatectomy radiotherapy is mainly the prostate bed (PB), i.e., a residual part of the operatively resected prostate gland encompassing microscopic cancerous tissues. Particularly, in contrast to typical CTVs expanded from gross tumor volumes, PB is a specific virtual target featuring invisible boundaries in CT images, which brings much more challenges for both their manual contouring by clinicians [7], [8] and automatic segmentation by advanced learning-based methods [9], such as the examples shown in Fig. 1.

Fig. 1.

Typical examples of prostate bed (PB) and organs at risk (OARs) segmentation, where the blue and yellow masks denote the bladder and rectum (i.e., OARs), respectively, and the red contour stands for PB. We can see that PB is a virtual target without visible boundaries, which brings particular challenges for general deep networks learning segmentations from imaging appearances, e.g., U-Net [1] with residual blocks [2] (i.e., ResU-Net) and MTAN [3]. Our method explicitly models the spatial reliance of PB on OARs in training an end-to-end co-segmentation network, leading to much more consistent results with the manual delineations by experienced clinicians (i.e., ground truth).

In practice, clinicians typically delineate the contour of PB according to its spatial positions to adjacent OARs (i.e., the bladder and rectum) with much more clear boundaries. Such a complex operation requires the joint consideration of published consensus guidelines [7], [10]–[12] and patient characteristics, resulting in very large inter-operator and intraoperator variations due to its heavy reliance on a clinician’s personal knowledge/experience [8]. To tackle this challenge, automatic workflows are highly desired in the clinic. However, although automatic methods for image segmentation has been comprehensively studied in both the communities of computer vision and medical image computing, only a few of them were proposed for PB delineation. These existing PB segmentation methods can be categorized into two groups, i.e., conventional atlas-based approaches [13], [14] and modern deep learning-based approaches proposed very recently [15], [16].

The atlas-based segmentation is commonly used in commercial software for radiotherapy planning [14], which leverages deformable image registration techniques [17] to align the label map(s) from template/atlas image(s) onto the image under segmentation. Although having promising performance in contouring organs, these atlas-based methods have under-satisfactory accuracy for non-contrast PB delineation due to their strong reliance on image intensity information [15], [16], [18]. Along with the explosive development of deep learning, diverse automatic methods, e.g., those adopt convolutional encoder-decoder architectures [1] as the backbone, have been proposed for medical image segmentation, demonstrating successful applications in delineating male pelvic organs [19]–[23]. However, the general architectures used by these organ segmentation networks are hard to obtain promising results in segmenting virtual CTV (e.g., PB), since they typically learn feature representations for segmentation from imaging appearances. More recently, inspired by the clinical workflow of clinicians, a few studies proposed to explicitly leverage the spatial associations between PB and adjacent OARs to design more accurate deep networks for fully automatic PB delineation [15], [16]. Roughly in an asymmetric multi-task learning setting [24], these existing methods alternatively adopt OAR segmentation results as one part of the input of PB segmentation network or select a part of the hierarchical feature representations learned by the OAR sub-network to assist PB segmentation. Such a direct copy of OAR segmentation information into PB segmentation improperly ignores a natural gap between the two asymmetrically dependent but fundamentally different tasks, i.e., the desired feature representations for accurate PB and OAR delineations should be consistent but not strictly identical.

In this paper, we propose a deep encoder-decoder network integrating dynamic cross-task representation adaptation (i.e., DyAdapt) for more effective asymmetric multi-task learning, with application to accurate and efficient co-segmentation of PB and OARs in one-pass from CT images. As shown in Fig. 2, a series of DyAdapt modules are positioned along the forward path of our network, acting as the bridges between the OAR and PB branches to hierarchically transfer local-to-global feature representations across tasks in an end-to-end fashion. Specifically, based on the idea of learning-to-learn [25], [26], the DyAdapt module at a specific position of the PB branch learns to adapt the corresponding feature representation from the source task (i.e., OAR segmentation) to maximize its compatibility with the target task (i.e., PB segmentation), conditioned on the dynamic inter-task dependency and discrepancy learned from the learning states of the network’s preceding layers. The proposed method was thoroughly evaluated on a real-patient dataset of 186 post-prostatectomy CT images [15]. The obtained results demonstrate that it outperformed state-of-the-art multi-task deep networks by a relatively large margin in the end-to-end co-segmentation of PB and OARs.

Fig. 2.

Schematic diagram of our end-to-end deep network for PB and OARs (i.e., bladder and rectum) co-segmentation from CT images. The network uses a head block to capture detailed local information from the high-resolution input image (size: 256 × 256 × 3), a back-bone encoder-decoder branch (i.e., a series of Conv, Res Conv, Res DeConv and DeConv blocks) for OARs segmentation, and another encoder-decoder branch relying on the backbone for PB segmentation. The PB segmentation branch leverages a series of DyAdapt modules to perform dynamic cross-task feature representation adaptation, by which the OAR segmentation knowledge is transferred to assist the more challenging task of PB contouring. Notably, the Conv and DeConv blocks here are not detailed as they are the same to Res Conv and Res DeConv block except the residual connection.

The rest of the paper is organized as follows. Related work, including multi-task learning in deep networks and pelvic organ segmentation are briefly reviewed in Section II. Section III describes the proposed method. The experimental results on the real-patient dataset and discussions regarding the method’s efficacy/limitations are presented in Sections IV and V, respectively. We conclude the paper in Section VI.

II. Related Work

A. Multi-Task Learning in Deep Networks

Multi-task learning (MTL) is a machine learning paradigm to learn a unified model by integrating the complementary information from multiple related tasks to improve generalization. Deep neural networks typically perform MTL via feature transformation (or selection) across tasks [24], under the assumption that related tasks share a common set of feature representations. Capitalizing on task-oriented feature learning and cross-task regularization, multi-task deep neural networks have been successfully applied in both the communities of computer vision and medical image computing.

For example, Misra et al. [27] proposed a cross-stitch module to linearly transform feature representations between the subnetworks implemented for any two complementary tasks, e.g., joint semantic segmentation and surface normal estimation. Liu et al. [3] assume there is a common feature pool shared across tasks, from which task-specific features are selected by attention mechanisms for multiple tasks of image-to-image prediction and classification. Liu et al. [28] design a selective integration module to dynamically select and integrate features from a shared backbone for each specific task, with application to concurrent saliency, edge and skeleton detection in natural images. Xue et al. [29] proposed a deep multi-task relationship learning network to learn jointly the cardiac phase and correlations between different left ventricle indices from magnetic resonance (MR) sequences. Lian et al. [30] proposed a multi-task transformer network to perform joint feature selection and transformation from an encoder-decoder backbone for concurrent bony structure segmentation and landmark localization from dental cone-beam CT images. Zhang et al. [31] proposed a 3D MTL network integrating scale-aware attention for common feature learning and task-aware attention for task-specific feature transformation, with application to concurrent tumor segmentation and lymph node classification in CT images. Uslu et al. [32] design an attention-based encoder-decoder network for left atrium segmentation and edge regression from cardiac MR images.

Although multi-task deep learning has shown promising performance in various applications, these existing studies typically implement deep neural networks under a symmetric setting, which assumes that all related tasks are equally important or have comparable difficulties in learning. However, such a symmetric assumption cannot always hold in real-world applications, since the inter-task correlations/dependencies could also be asymmetric, considering that some tasks may naturally rely on other tasks, e.g., the case of PB and OAR co-segmentation in this paper. Notably, to improve the learning of end-to-end deep neural networks for asymmetric MTL, a potentially feasible strategy is to explicitly consider the dependency between tasks in regulating cross-task knowledge transfer, which has interesting commons/relateness with some other machine learning paradigms, including transfer learning [33] and learning-to-learn (or meta-learning) [25], [26].

B. Learning-based Pelvic Organ Segmentation

Accurate segmentation of prostate gland (or PB) and OARs (i.e., the bladder and rectum) from CT images is an essential step of prostate cancer radiotherapy planning [19]. Considerable deep learning-based methods have been proposed in the literature for the co-segmentation of male pelvic organs (i.e., the prostate gland, bladder, and rectum) [19]–[22], [34]–[36]. Although they have presented promising performance in both the cases of organ delineations from CT and MR images, these existing methods are not fully applicable in the specific task PB and OARs co-segmentation, due to the virtual nature of PB that should be addressed by task-specific methodological designs. Also, to refine the automatic delineation of organ contours, many deep learning-based methods proposed in recent years implement their segmentation models in the MTL framework, e.g., by designing auxiliary tasks to assist the target of organ segmentation. For example, [20]–[22] adopt different ways to include prostate boundary regression as an auxiliary task to boost prostate gland segmentation. Similar to the related work introduced in Section II-A, these learning-based methods for pelvic organ segmentation work in a symmetric MTL setting to learn segmentation-oriented feature representations from CT imaging appearances. That is, they may not be suitable for the more challenging task of non-contrast PB delineation relying on adjacent OARs.

More recently, a few asymmetric multi-task deep networks have been proposed to explicitly leverage the knowledge from OAR segmentation to guide PB segmentation. Specifically, Balagopal et al. [16] design a two-stage framework, which trains first the OAR segmentation sub-networks, based on whose outputs another sub-network is further trained for PB segmentation. In contrast, Xu et al. [15] proposed an end-to-end architecture, which adopts attention mechanisms to hierarchically select and transfer the multi-scale feature representations from the OAR branch to the PB branch. Although achieved better results than atlas-based methods (i.e., commercial software for radiotherapy planning) and state-of-the-art learning-based methods, due to the direct duplication of OAR segmentation information in PB segmentation without any adaptation, [15] and [16] improperly neglect the gap between the optimal feature representations needed for contouring PB and OARs, two clinical targets that are asymmetrically dependent but also fundamentally different.

III. Method

As the schematic diagram shown in Fig. 2, we propose an asymmetric multi-task network for PB and OARs co-segmentation from CT images. In the framework of learning-to-learn for cross-task knowledge transfer, our method adopts an encoder-decoder backbone to conduct the source task of OARs segmentation. Parallel to the backbone, another encoder-decoder sub-network is designed for the target task of PB contouring. It adopts a series of dynamic cross-task representation adaptation (DyAdapt) modules to adapt the feature representations from the source task to maximize their compatibility with the target task, conditioned on the states of inter-task associations dynamically changing along the forward path of the end-to-end network.

A. Network Architecture

Given a high-resolution image (e.g., with the size of 256 × 256 × 3), our asymmetric multi-task network performs PB and OARs co-segmentation in an end-to-end fashion. It first applies a Head block to extract local image details, which are shared between two tasks, from the original input. Such a Head consists of three convolutional layers, with the first two performing general 3×3 convolution (with unit stride and zero-padding) for feature extraction and the last one performing depth-wise separable 2 × 2 convolution (2 strides without padding) for downsampling. Besides the Head block, our network adopts a deep encoder-decoder architecture to learn and integrate local-to-global feature representations for multi-task image-to-image predictions. It uses a U-Net with residual blocks (i.e., ResU-Net) as the backbone for the source task of OARs segmentation, based on which another encoder-decoder branch is designed for the target task of PB segmentation.

The backbone ResU-Net uses the downsampled CT image as the input and starts with two 3 × 3 convolutional layers to extract local features, which are further fused with the downsampled features from the Head block to form multi-scale representations of image details, assisting the learning of subsequent convolutional (Conv) and deconvolutional (DeConv) blocks. Each Conv block in the encoding path of the backbone contains two 3 × 3 convolutions with residual connection, and a 2 × 2 pooling layer is applied between any two Conv blocks to halve the size of feature maps. Symmetric to the encoding path, each DeConv block in the decoding path starts with the bilinear up-sampling of input features, followed by the skip connection with the corresponding Conv block in the encoding path, and then two 3 × 3 convolutions with residual connection. Finally, the output of the backbone is combined with the high-resolution image details (from the 2nd convolutional layer of the Head) to produce the probability maps of OARs segmentation.

The encoder-decoder branch for PB delineation is designed in terms of the hierarchical feature representations from the backbone, considering that the contouring of non-contrast PB highly depends on its spatial locations to adjacent OARs with much more clear boundaries. To begin with, the PB branch adopts the backbone output as input. The blocks in the encoding/decoding path of the PB branch are one-to-one correspondent with those from the backbone. That is, each DyAdapt block in the PB branch adaptively transfers the corresponding feature representations from a specific block of the source task (i.e., OAR segmentation) to maximize their compatibility with the target task (i.e., PB segmentation). Similar to OAR segmentation, the output of the PB branch is further combined with the high-resolution image details to produce the probability maps of PB segmentation.

B. Dynamic Cross-Task Representation Adaptation

To perform cross-task knowledge transfer in the end-to-end network for asymmetric MTL, we make three intuitive assumptions. 1) The feature representations for PB contouring are originated from those for OARs delineation, as the former task strictly relies on the latter one. 2) A gap exists between the feature representations needed by the two segmentation tasks, as they are dependent but also fundamentally different. 3) The inter-task associations are not static along the feed-forward path of the end-to-end network, i.e., between the hierarchical feature representations, thus desiring dynamic cross-task adaptations in a layer-by-layer fashion.

Based on these assumptions, the DyAdapt modules in the PB branch conduct hierarchical cross-task representation adaptations by leveraging the idea of learning-to-learn. Specifically, let ${\mathbf{O}}_1^{\left(s\right)}$ and ${\mathbf{O}}_2^{\left(s\right)}$ be the feature representations from the sth encoding block of the backbone, produced by the first and second convolutional layers, respectively; correspondingly, ${\mathbf{P}}^{(s-1)}$ denotes the downsampled output of the preceding encoding block of the PB branch. As the diagram shown in Fig. 2, the DyAdapt module at the sth scale adapts the respective backbone representations to maximize its compatibility with PB segmentation in two steps:

1). Dynamic convolution-based transfromation:

The backbone representation ${\mathbf{O}}_2^{\left(s\right)}$ (with $C_{\text{in}}$ channels) is transferred into the PB branch via the dynamic convolution parameterized by ${\mathbf{O}}_1^{\left(s\right)}$ and ${\mathbf{P}}^{(s-1)}$, such as

$${\widehat{\mathbf{O}}}_2^{(s)}={\mathfrak{f}}_{1\times 1}\left({\mathbf{O}}_2^{(s)}\mid {\mathbf{O}}_1^{(s)},{\mathbf{P}}^{(s-1)}\right),$$ (1)

where ${\mathfrak{f}}_{1\times 1}(\cdot \mid \cdot ,\cdot )$ denotes the 1 × 1 convolution, and ${\widehat{\mathbf{O}}}_2^{\left(s\right)}$ is the transferred feature representation with $C_{\text{out}}$ channels. That is, the feature space at a specific scale of the target segmentation branch is spanned by the corresponding feature representations from the source segmentation branch, conditioned on the dynamic inter-task dependency and discrepancy inferred from the learning states of preceding layers, i.e., in terms of the convolutional parameters predicted by fusing ${\mathbf{O}}_1^{\left(s\right)}$ and ${\mathbf{P}}^{(s-1)}$. Considering that predicting all learnable parameters is computationally infeasible even for a 1 × 1 layer (i.e., $C_{\text{in}}\times C_{\text{out}}$ parameters), the dynamic convolution defined in (1) can be factorized in an analog of SVD [37]. Accordingly, ${\mathfrak{f}}_{1\times 1}(\cdot \mid \cdot ,\cdot )$ can be approximated by

$${\widehat{\mathbf{O}}}_2^{\left(s\right)}\approx {\mathbf{U}}_{1\times 1}^{\left(s\right)}\mathrm{*}{𝒲}_{1\times 1}\left({\mathbf{O}}_1^{\left(s\right)},{\mathbf{P}}^{\left(s-1\right)}\right){\mathrm{*}}_c{\mathbf{V}}_{1\times 1}^{\left(s\right)}\mathrm{*}{\mathbf{O}}_2^{\left(s\right)},$$ (2)

where $\mathrm{*}$ denotes the general 1 × 1 convolution, and ${\mathbf{U}}_{1\times 1}^{\left(s\right)}$ and ${\mathbf{V}}_{1\times 1}^{\left(s\right)}$ are the corresponding parameter matrices with $C_{\text{out}}$ and $C_{\text{in}}$ output channels, respectively. In contrast, ${\mathrm{*}}_c$ stands for the efficient depth-wise separable 1 × 1 convolution, and ${𝒲}_{1\times 1}(\cdot ,\cdot )$ is a light-weight kernel adapter, which adopts the concatenation of ${\mathbf{O}}_1^{\left(s\right)}$ and ${\mathbf{P}}^{(s-1)}$ as the input, to predict a diagonal matrix with only $C_{\text{in}}$ parameters for ${\mathrm{*}}_c$.

2) Attention-based fusion:

The transferred representation ${\widehat{\mathbf{O}}}_2^{\left(s\right)}$ is further fused with ${\mathbf{P}}^{(s-1)}$ to produce the encoding representation ${\mathbf{P}}^{\left(s\right)}$ at the sth scale of the PB branch. To this end, an attention map ${\mathbf{A}}^{\left(s\right)}$ with the same size as $[{\widehat{\mathbf{O}}}_2^{\left(s\right)};{\mathbf{P}}^{\left(s\right)}]$, i.e., the concatenation of ${\widehat{\mathbf{O}}}_2^{\left(s\right)}$ and ${\mathbf{P}}^{\left(s\right)}$, is first defined as

$${\mathbf{A}}^{\left(s\right)}={\mathfrak{f}}_{\mathrm{a}\mathrm{t}\mathrm{t}}\left({\widehat{\mathbf{O}}}_2^{\left(s\right)},{\mathbf{P}}^{\left(s\right)}\right).$$ (3)

Thus, in a self-supervised fashion with back-propagation [3], [15], the element-wise multiplication ${\mathbf{A}}^{\left(s\right)}\odot [{\widehat{\mathbf{O}}}_2^{\left(s\right)};{\mathbf{P}}^{\left(s\right)}]$ selects spatially and channel-wise important features from both ${\widehat{\mathbf{O}}}_2^{\left(s\right)}$ and ${\mathbf{P}}^{\left(s\right)}$ to further enhance the compatibility between the source task information and target task information. Finally, ${\mathbf{P}}^{\left(s\right)}$ is output by

$${\mathit{P}}^{\left(s\right)}={\mathit{W}}_{3\times 3}^{\left(s\right)}\mathrm{*}\left({\mathit{A}}^{\left(s\right)}\odot [{\widehat{\mathit{O}}}_2^{\left(s\right)};{\mathit{P}}^{\left(s\right)}]\right),$$ (4)

where ${\mathbf{W}}_{3\times 3}^{\left(s\right)}$ stands for a general 3 × 3 convolution (with unit stride and zero-padding).

As the illustrations in Fig. 2, the DyAdapt blocks in the decoding path of the PB segmentation branch have similar operations with those in the encoding path. The only exception is that, instead of feature down-sampling, the decoding DyAdapt starts with the bilinear up-sampling of input feature representations and concatenate them with those from the corresponding encoding block via skip connection.

C. Implementation

1). Implementation details:

In our implementation, the network input is a set of 3 adjacent CT image slices (with the size of 256 × 256 × 3), and the outputs are the OARs and PB probability maps of the central slice. The Head block contains three convolutions, with each of them attached with batch normalization (BN) and ReLU activation. Using the downsampled image as the input (with the size of 128 × 128 × 3), the ResU-Net backbone for OARs segmentation contains five encoding blocks and four decoding blocks. All encoding blocks consist of two successive 3 × 3 convolutions attached with BN and ReLU and the residual connection, and the first four of them are followed by 2 × 2 max-pooling. Similarly, all decoding blocks also contain two successive 3 × 3 convolutions attached with BN and ReLU and the residual connection, and the first three of them starts with 2 × 2 bilinear up-sampling.

The PB segmentation branch adopts the backbone output as the input. Its encoding and decoding paths consist of five and four DyAdapt modules, respectively, with each of them bilaterally connected to the corresponding encoding/decoding block in the backbone. Each DyAdapt module is a combination of the operations defined by (1) to (4). In (2), ${𝒲}_{1\times 1}(\cdot ,\cdot )$ is implemented as a light-weight block starting with a 1 × 1 convolution, followed by global average pooling (GAP), another two 1 × 1 convolutions, and finally $L_2$ normalization, where the first two convolutions are attached with BN and ReLU. In (3), the operator ${\mathfrak{f}}_{\text{att}}(\cdot ,\cdot )$ consists of two general 1 × 1 convolutions followed by BN, and ReLU and sigmoid activations, respectively. The general convolutions in (2) and (4) are all followed by BN and ReLU.

2). Training & inference details:

The CT slices with varying sizes and resolutions were bilinearly resampled and center-cropped to have a uniform spatial resolution (i.e., 1 × 1 mm²) and size (i.e., 256 × 256). The image intensities were first clipped between [−200, 800] Hounsfield Unit (HU), followed by min-max normalization to be scaled between [0, 1]. The training images were augmented on-the-fly by combining random translations and rotations. The end-to-end network was trained by minimizing the loss function defined as

$$ℒ={ℒ}_{\text{Dice}}^{\mathrm{P}\mathrm{B}}+{ℒ}_{\text{Dice}}^{\mathrm{O}\mathrm{A}\mathrm{R}}+{ℒ}_{\text{Dice}}^{\mathrm{d}\mathrm{P}\mathrm{B}}+{ℒ}_{\text{Dice}}^{\mathrm{d}\mathrm{O}\mathrm{A}\mathrm{R}},$$ (5)

where the first two terms are the Dice losses [38] to quantify the PB and OARs segmentation errors in the input space, while the last two terms are the auxiliary Dice loss functions quantified in the down-sampled image space (i.e., by using the output of the backbone and PB branch without the fusion with the Head block). Briefly, given a segmentation probability map produced by the network $\tilde{\mathbf{Y}}$ and the corresponding one-hot ground truth $\mathbf{Y}$, the Dice loss ${ℒ}_{\text{Dice}}(\tilde{\mathbf{Y}},\mathbf{Y})$ is defined as

$${ℒ}_{\text{Dice}}(\tilde{\mathbf{Y}},\mathbf{Y})=1-\frac{1}{C}\sum _{c=1}^C \frac{2\sum _{m=1}^M {\tilde{\mathbf{Y}}}_{cm}{\mathbf{Y}}_{cm}+ϵ}{\sum _{m=1}^M ({\tilde{\mathbf{Y}}}_{cm}^2+{\mathbf{Y}}_{cm}^2)+ϵ},$$ (6)

where $C$ and $M$ denote the numbers of classes and pixels, respectively, and $ϵ>0$ is a small scalar to stabilize numerical calculation. In (5), as a supplementary to the main Dice losses (i.e., ${ℒ}_{\text{Dice}}^{\mathrm{P}\mathrm{B}}$ and ${ℒ}_{\text{Dice}}^{\mathrm{O}\mathrm{A}\mathrm{R}}$), the auxiliary Dice losses (i.e., ${ℒ}_{\text{Dice}}^{\mathrm{d}\mathrm{P}\mathrm{B}}$ and ${ℒ}_{\text{Dice}}^{\mathrm{d}\mathrm{O}\mathrm{A}\mathrm{R}}$) help refine the learning of more representative features in the down-sampled image space for fine-grained segmentation. We updated the network parameters by using the Adam optimizer with an initial learning rate of 0.001 and mini-batch size of 72. The network was implemented by using the PyTorch framework on a Ubuntu workstation with two NVIDIA GeForce RTX 3090 graphic cards. Overall, for a training set and a validation set with more than 15, 000 and 5, 000 CT images, respectively, the training of our network by 100 epochs took around 350 mins. The final network was saved as that with the best accuracy on the validation set. The inference on a test CT volume can be conducted in seconds and the network predictions were interpolated back into the original physical space of the input CT volume.

IV. Experiments

A. Dataset

The dataset used in this study consists of 186 post-prostatectomy patients collected between the year of 2009 and 2019. Each patient has one planning CT image, with the in-plane spacing ranging from 0.81 to 1.37 mm, the slice thickness being 3.00 or 1.50 mm isotropically, and the number of slices ranging from 98 to 270. For these images, the ground-truth segmentation masks of PB and OARs (i.e., bladder and rectum) were manually delineated by one of three expert physicians and modified/verified by the rest two physicians to mitigate the inter-observer variability. In total, the dataset contains 26, 133 image slices with a uniform size of 512 × 512.

B. Competing Methods

To comprehensively evaluate the efficacy of our method, we compared it with six different methods, including the U-Net [1] with residual blocks [2], i.e., ResU-Net, for single-task PB segmentation, three symmetric MTL networks, i.e., MT-U-Net, and the state-of-the-art MTAN [3] and DFI [28], and two asymmetric MTL networks, i.e., Cas-U-Net and the state-of-the-art AMTA-Net [15]. These competing methods are briefly described below.

1. ResU-Net [1], [2]: The network has the same architecture with the backbone in Fig. 2. It was trained to conduct the single task of PB contouring by minimizing the Dice loss.

2. MT-U-Net: A a commonly used modification of ResU-Net for MTL, MT-U-Net contains a encoding branch shared between the tasks of PB and OARs segmentaton, and two parallel decoding branches specified for these two different tasks. Specifically, MT-U-Net shares the same encoder architecture with ResU-Net and the backbone in Fig. 2. It has two parallel decoders (the same as that in ResU-Net), which produce the segmentations of OAR and PB, respectively.

3. MTAN [3]: A state-of-the-art network consisting of a common backbone and two sibling task-specific sub-networks for symmetric MTL. It adopts an ResU-Net as the backbone shared between the tasks of PB and OARs segmentation. The common feature representations from the backbone are selected for each task via task-oriented attention mechanisms.

4. DFI [28]: Based on the hierarchical feature representations learned by a share backbone, a set of task-adaptive attention modules (TAMs) are designed to dynamically select and integrate multi-scale backbone features for each specific task, i.e., PB segmentation and OARs segmentation. Notably, in this study, the original DFI model in [28] was slightly modified by adjusting the output layers for two-task segmentations. Also, its backbone was accordingly updated to have comparable learnable parameters compared with other competing methods.

5. Cas-U-Net: Similar to [16], Cas-U-Net was implemented by cascading two ResU-Nets, where the first one performs OARs segmentation, and the second one performs PB segmentation by using the OAR probability maps as the supplementary input in addition to the original image.

6. AMTA-Net [15]: A state-of-the-art asymmetric MTL network specified for PB and OARs co-segmentation. In the end-to-end fashion, the feature representations learned for OARs segmentation are hierarchically selected for the more challenging task of non-contrast PB delineation.

For a fair comparison, all competing methods were adjusted to have a comparable number of learnable parameters, and they were trained by using the same strategy and environment introduced in Section III-C.2.

C. Experimental Settings

The segmentation performance of different competing methods was compared under 5-fold cross-validation, i.e., the patients were randomly divided as five folds, and in each iteration three folds were used as the training set, while the remaining two folds were used as the validation and testing sets, respectively. Using the manual annotations as the ground truth, we quantify the segmentation accuracy in terms of two complementary metrics, including 1) the Dice similarity coefficient (DSC) and 2) the average symmetric surface distance (ASD). Let $\tilde{\mathbf{V}}$ and $\mathbf{V}$ be the network prediction and one-hot reference segmentation, respectively. Then, the DSC value is calculated as

$$\mathrm{D}\mathrm{S}\mathrm{C}(\tilde{\mathbf{V}},\mathbf{V})=\frac{2\sum _{m=1}^M {\tilde{\mathbf{V}}}_m{\mathbf{V}}_m}{\sum _{m=1}^M {\tilde{\mathbf{V}}}_m+\sum _{m=1}^M {\mathbf{V}}_m},$$ (7)

where $M$ is the number of elements in each mask. Let $\tilde{\mathbf{S}}$ and $\mathbf{S}$ be the surfaces of $\tilde{\mathbf{V}}$ and $\mathbf{V}$, respectively. The ASD value is quantified as

$$\mathrm{A}\mathrm{S}\mathrm{D}(\tilde{\mathbf{S}},\mathbf{S})=\frac{\sum _{\bar{s}\in \tilde{\mathbf{S}}} dist(\tilde{s},\mathbf{S})+\sum _{s\in \mathbf{S}} dist(s,\tilde{\mathbf{S}})}{|\tilde{\mathbf{S}}|+|\mathbf{S}|},$$ (8)

where $dist(\tilde{s},\mathbf{S})$ denotes the shortest distance from the point $\tilde{s}$ on $\tilde{\mathbf{S}}$ to $\mathbf{S}$, $|\tilde{\mathbf{S}}|$ is the total number of points on $\tilde{\mathbf{S}}$, and vice versa. Notably, these two metrics are commonly used in medical image segmentation.

D. Results

The quantitative segmentation results in terms of DSC and ASD are summarized in Table I, from which we can have at least three observations. First, compared with the mono-task ResU-Net, all other deep networks implemented in the MTL setting led to better accuracy of PB segmentation. Consistent with clinical workflow, this observation justifies the benefit of leveraging OARs segmentation as a source task to assist the more challenging task of non-contrast PB delineation. Second, on average, the asymmetric MTL methods (especially AMTA-Net and our method) have relatively better performance than the symmetric MTL methods (i.e., MT-U-Net, MTAN, and DFI) in both the segmentations of OARs and PB. It suggests that, compared with the adaptation from OARs to PB, PB delineation could not bring too much guidance to regulate OARs segmentation due to the asymmetric inter-task dependency in this specific application. The explicit modeling of such asymmetric dependency in network design, such as AMTA-Net and our method, could more effectively transfer the guidance from OAR segmentation to regulate the sub-network for PB segmentation, without sacrificing the performance of the source OAR segmentation task. Also, compared with Cas-U-Net, AMTA-Net and our method are two asymmetric MTL networks brought obviously more consistent improvements, implying that how we transfer the representations between the two asymmetrically dependent but fundamentally different tasks is critical for PB and OARs co-segmentation. Third, our method outperformed the state-of-the-art asymmetric deep networks (i.e., Cas-U-Net and AMTA-Net) by a significantly large margin (with $p<5e-5$) in the challenging task of delineating the virtual target of PB. It implies the efficacy of the proposed DyAdapt modules in adapting the feature representations from the source task (i.e., OARs segmentation) to maximize their compatibility with the target task (i.e., PB contouring) for their end-to-end learning. On the other hand, it is worth mentioning that the improvements brought by our method for OARs segmentation were not as significant as that for PB segmentation. That is, according to Table I, we can see that our method significantly outperformed (with $p<5e-5$) all other competing methods in PB delineation, while the improvements for OAR contouring is not so significant (with $p<5e-2$ or even larger). It further confirms that our asymmetric dependency assumption is technically sound in PB and OARs co-segmentation, as PB needs more guidance effectively transferred from OARs.

TABLE I.

Segmentation results (mean±standard deviation) quantified under 5-fold cross-validation, in terms OF DSC (in %) and asd (in mm), for six state-of-the-art DEEP learning-based methods and our method. These methods can be grouped as three Different groups, i.e., a single-task network (resu-net), symmetric multi-task networks (mt-u-net, mtan, and dfi), and asymmetric multi-task networks (cas-u-net, amta-net, and our method).

Target	Metric	ResU-Net	MT-U-Net	MTAN	DFI	Cas-U-Net	AMTA-Net	Our method
Bladder	DSC	-	90.19 ± 7.12	89.72 ± 7.20*	89.94 ± 7.93	90.43 ± 7.61	90.54 ± 6.52	90.84 ± 5.99
	ASD	-	1.14 ± 0.94	1.26 ± 1.10*	1.16 ± 0.97	1.14 ± 1.09	1.10 ± 0.90	1.08 ± 0.84
Rectum	DSC	-	80.61 ± 9.99*	80.37 ± 9.12*	81.31 ± 7.68	80.79 ± 9.43	81.85 ± 7.55	82.08 ± 6.60
	ASD	-	2.61 ± 1.06	2.61 ± 2.11	2.45 ± 2.16	2.57 ± 2.06	2.35 ± 1.88	2.29 ± 1.59
PB	DSC	75.21 ± 7.09**	75.26 ± 7.23**	75.65 ± 7.40**	75.45 ± 7.33**	75.56 ± 7.11**	75.94 ± 7.12**	77.87 ± 6.64
	ASD	2.53 ± 1.05**	2.50 ± 1.07**	2.49 ± 1.12**	2.46 ± 1.11**	2.46 ± 1.05**	2.44 ± 1.08**	2.16 ± 1.04

THE SYMBOLS ** AND * STAND FOR SIGNIFICANT DIFFERENCES BETWEEN THE RESPECTIVE METHOD AND OUR METHOD WITH $p<{5e}^{-5}$ AND $p<{5e}^{-2}$, RESPECTIVELY.

Corresponding to the quantitative evaluations, we also visually compared the OAR and PB contours produced by different automatic methods, with four representative examples presented in Fig. 3. We can see that, regarding the experts’ manual delineations as the reference, our method consistently yielded better segmentation contours, especially of PB, across the four different subjects. In addition, the improvements by our method are visually more significant at the region inbetween the bladder and rectum, such as indicated by the yellow arrows. It is worth noting that precise PB delineation at this location is critical in clinical practice, since it is a precondition to make the corresponding radiation dose being sharp to protect the OARs. On the other hand, compared with other locations, the segmentation of PB at this location is also more straightforwardly dependent on the relative position between PB and adjacent OARs. The promising segmentation results produced by our method at this location further justify the efficacy of the proposed DyAdapt modules in explicitly modeling such strong spatial dependency/reliance for cross-task feature representation adaptation in PB and OAR co-segmentation. These visual evaluation results shown in Fig. 3 are consistent with the quantitative evaluations in Table I, suggesting the practical usage of our method for fully automatic PB and OARs co-segmentation in planning CT images.

Fig. 3.

Segmentations by the six competing methods (i.e., ResU-Net, MT-U-Net, MTAN, DFI, Cas-U-Net, and AMTA-Net) and our method for four representative examples, with each row corresponding to a specific patient. The dark blue, dark yellow, and dark red contours denote the ground-truth masks of the bladder, rectum, and PB, respectively, i.e., the manual delineations by experts. Accordingly, the light blue, light yellow, and rose red contours stand for the results of bladder, rectum, and PB automatic segmentations produced by different deep learning-based methods.

V. Discussions

A. Efficacy of Dynamic Convolution

As described in Section III-B.1, our DyAdapt module adopts the dynamic convolutions conditioned on the states of inter-task associations to perform hierarchical feature transformation from the backbone of OAR segmentation to the branch of PB segmentation. In this group of experiments, we performed an ablation study by removing the dynamic convolution from the original DyAdapt module to evaluate the influence on PB and OARs co-segmentation. The corresponding segmentation results in terms of DSC and ASD are summarized in Fig. 4, where w/o dynamic conv stands for DyAdapt without dynamic convolutions, i.e., using solely attention mechanisms. By comparing w/o dynamic conv with DyAdapt in Fig. 4, we can see that the inclusion of the dynamic convolution-based feature transform effectively improved the PB delineation accuracy, e.g., the DSC value was increased by 2.15% (from 75.72% to 77.87%) without sacrificing the OAR accuracy (i.e., the DSC and ASD values are comparable between DyAdapt and w/o dynamic conv). These results further justify the effectiveness of the dynamic convolution-based transform in cross-task representation adaptation for accurate PB and OARs co-segmentation in an end-to-end fashion.

Fig. 4.

Ablation studies to evaluate the effectiveness of dynamic convolution-based feature transformation and attention-based feature selection in our DyAdapt module, where w/o dynamic conv and w/o attention denote the variants of DyAdapt after removing these two components, respectively.

B. Efficacy of Attention Mechanism

By leveraging the attention mechanism in our DyAdapt (i.e., Section III-B.2), the transformed source-task representation is fused with the target-task representation at each specific scale to further improve their compatibility for PB segmentation. In this group of experiments, we performed an ablation study by removing the attention operation from the original DyAdapt module, such a variant is denoted as w/o attention, to evaluate the respective influence on PB and OARs co-segmentation. According to the results shown in Fig. 4, we can observe that the utilization of the attention mechanism following the dynamic convolution-based cross-task feature transform help to further refine the PB segmentation accuracy, suggesting its positive role in our end-to-end PB and OARs co-segmentation network. On the other hand, by comparing w/o attention with w/o dynamic conv in Fig. 4, it can be seen that the inclusion of attention-based fusion brings less segmentation improvements than dynamic convolution-based cross-task transformation. This observation further implies that a gap could exist between the feature representations required for accurate OAR and PB segmentation, which could not be comprehensively handled by attention-based feature selection for cross-task knowledge transfer, such as in AMTA-Net and the w/o dynamic conv variant of our method.

C. Influence of Image Resolution & Head Conv

Typically, both high-resolution image information and semantic contexts are critical for accurate segmentation. To this end, our network attaches a Head block onto the deep encoder-decoder architecture. Given a relatively large mini-batch of high-resolution inputs for training (i.e., 72 CT images with the spatial resolution of 1 × 1 mm² and size of 256 × 256 × 3, respectively), the Head block can efficiently combine local details with global context extracted from the high-resolution input. Such a design is different to the most related work (i.e., AMTA-Net) [15] that works on down-sampled images (size: 128 × 128; resolution: 2 × 2 mm²).

To check the value brought by the high-resolution input as well as the Head Conv block, we implemented two variants of our network following the setting in [15], i.e., by removing the Head block and training directly the remaining encoder-decoder architecture with the image size of 128 × 128 × 3 and 256 × 256 × 3, respectively, for end-to-end segmentation. Notably, for the 256 × 256 × 3 variant, the mini-batch should be significantly reduced to control memory consumption during the training under the same condition. The comparison between these variants and our original network is detailed in Table II. From Table II, we can see that increasing the input quality (i.e., with Head 1 × 1 mm² vs. w/o Head 2 × 2 mm²) consistently boosted the segmentation accuracies for both PB and OARs. Also, Table II implies that the Head block is an effective way for increasing the input quality but preserving training efficiency, as compared with w/o Head 2 × 2 mm², w/o Head 1 × 1 mm² even led to worse accuracies due to training inefficiency caused by much larger GPU memory requirement.

TABLE II.

The segmentation results in terms of DSC obtained by our method with or without the head conv block, respectively. accordingly, 2 X 2 MM² AND 1 X 1 MM² denote the input image resolution, and the corresponding image size is 128 X 128 and 256 X 256, respectively.

	Resolution	PB	Bladder	Rectum
w/o Head	2 x 2 mm2	76.08 ± 7.13	89.75 ± 7.78	81.20 ± 7.72
	1 x 1 mm2	75.78 ± 7.12	7.78 89.20 ± 7.76	7.72 80.56 ± 7.76
with Head	1 x 1 mm2	77.87 ± 6.64	90.84 ± 5.99	82.08 ± 6.60

On the other hand, it is also worth mentioning that our re-implementation of AMTA-Net in Table I (by increasing the input resolution and size) also effectively improved its performance according to the comparisons with the original results reported in [15], e.g., the DSC values for PB, bladder, and rectum segmentation were improved from 75.67 ± 6.56, 88.40 ± 8.72, 80.35 ± 9.36 to 75.94 ± 7.12, 90.54 ± 6.52, and 81.85 ± 7.55, respectively. All these results indicate that accurate segmentation by deep networks desires an effective combination of high-quality image details and contextual information, such as the Head Conv design in our study. In addition, we can see that the variant of our method with low-resolution input performed as well as AMTA-Net with high-resolution input, further justifying the efficacy of our method.

D. Efficacy of Auxiliary Losses

In network training, we quantified the losses in both the original image space and down-sampled image space for deep supervision. Such as shown in (5), ${ℒ}_{\text{Dice}}^{\mathrm{d}\mathrm{P}\mathrm{B}}$ and ${ℒ}_{\mathrm{D}\mathrm{i}\mathrm{c}\mathrm{e}}^{\mathrm{d}\mathrm{O}\mathrm{A}\mathrm{R}}$ attached onto the outputs of the last ResDeConv and DyAdapt blocks, respectively, help refine the representations learned in the down-sampled image space, based on which the final segmentations in the original image space (after the final DeConv blocks) can be improved. To further evaluate the influence of these auxiliary loss terms, we conducted another group of experiments by removing these terms from (5). We compared the obtained results (in terms of DSC and ASD) with the original network trained by using (5). Such comparisons are summarized in Fig. 5, where with and w/o Auxiliary represent the networks trained with and without the auxiliary loss terms, respectively. From Fig. 5, we can observe that these auxiliary losses consistently helped refine the segmentations of PB, bladder, and rectum in our experiment, which in some sense suggests the efficacy of such a specific training strategy.

Fig. 5.

Ablation studies to evaluate the influence of auxiliary losses (quantified in the down-sampled image space) in training, where with Auxiliary denotes training with the complete loss function (5), while w/o Auxiliary stands for training without the last two terms in (5).

E. Limitations and Future Work

Although our method achieved state-of-the-art performance in the challenging task of PB and OARs co-segmentation from CT images, its practical usage is still need to be further improved by addressing several limitations. First, the current study is on a dataset acquired from a single hospital. To more comprehensively evaluate the generalizability of our method, more subjects are needed, especially those from other imaging centers, acquired by various protocols, and different imaging manufactures etc. Second, our method explicitly leverages the spatial reliance of PB on adjacent OARs to design the co-segmentation network, where the feature representations for OARs segmentation are adaptively transferred for PB segmentation, showing promising results. In our current study, the PB and OARs co-segmentation network was trained end-to-end by minimizing the Dice loss functions. Although out of the scope of this paper, to further refine the segmentation details, one promising way is to include other auxiliary tasks as guidance, e.g., the regression of target boundaries or inter-target displacements maps, a common strategy has been successfully validated by various learning-based methods for pelvic organ segmentation [16], [20], [22]. Third, in our current implementation, DyAdapt implicitly models the state of the inter-task associations at a specific layer/scale via the fusion of both tasks’ feature representations from the preceding layers, to parameterize the dynamic convolution for cross-task representation adaptation. An alternative strategy to more explicitly quantify such inter-task dependency and discrepancy may further improve the efficacy of DyAdapt.

VI. Conclusion

In this work, we have proposed to concurrently contour PB and adjacent OARs in a fully automatic fashion from CT images for post-prostatectomy radiotherapy planning. An end-to-end asymmetric multi-task network has been proposed, which leverages dynamic cross-task representation adaptation (DyAdapt) modules designed in the learning-to-learn framework to adaptively transfer hierarchical feature representations from the source task of OARs segmentation to maximize their compatibility with the target task of non-contrast PB segmentation, under an intuitive assumption that the latter challenging task relies on but is also fundamentally different to the former one. We have evaluated our method on a dataset of post-prostatectomy CT images acquired at clinical practice. The corresponding results have shown that our method outperforms the state-of-the-art deep networks in both the settings of symmetric and asymmetric multi-task learning.