Deep‐learning convolutional neural network: Inner and outer bladder wall segmentation in CT urography

Abstract

Purpose

We are developing a computerized segmentation tool for the inner and outer bladder wall as a part of an image analysis pipeline for CT urography (CTU).

Materials and Methods

A data set of 172 CTU cases was collected retrospectively with Institutional Review Board (IRB) approval. The data set was randomly split into two independent sets of training (81 cases) and testing (92 cases) which were manually outlined for both the inner and outer wall. We trained a deep‐learning convolutional neural network (DL‐CNN) to distinguish the bladder wall from the inside and outside of the bladder using neighborhood information. Approximately, 240 000 regions of interest (ROIs) of 16 × 16 pixels in size were extracted from regions in the training cases identified by the manually outlined inner and outer bladder walls to form a training set for the DL‐CNN; half of the ROIs were selected to include the bladder wall and the other half were selected to exclude the bladder wall with some of these ROIs being inside the bladder and the rest outside the bladder entirely. The DL‐CNN trained on these ROIs was applied to the cases in the test set slice‐by‐slice to generate a bladder wall likelihood map where the gray level of a given pixel represents the likelihood that a given pixel would belong to the bladder wall. We then used the DL‐CNN likelihood map as an energy term in the energy equation of a cascaded level sets method to segment the inner and outer bladder wall. The DL‐CNN segmentation with level sets was compared to the three‐dimensional (3D) hand‐segmented contours as a reference standard.

Results

For the inner wall contour, the training set achieved the average volume intersection, average volume error, average absolute volume error, and average distance of 90.0 ± 8.7%, −4.2 ± 18.4%, 12.9 ± 13.9%, and 3.0 ± 1.6 mm, respectively. The corresponding values for the test set were 86.9 ± 9.6%, −8.3 ± 37.7%, 18.4 ± 33.8%, and 3.4 ± 1.8 mm, respectively. For the outer wall contour, the training set achieved the values of 93.7 ± 3.9%, −7.8 ± 11.4%, 10.3 ± 9.3%, and 3.0 ± 1.2 mm, respectively. The corresponding values for the test set were 87.5 ± 9.9%, −1.2 ± 20.8%, 11.9 ± 17.0%, and 3.5 ± 2.3 mm, respectively.

Conclusions

Our study demonstrates that DL‐CNN‐assisted level sets can effectively segment bladder walls from the inner bladder and outer structures despite a lack of consistent distinctions along the inner wall. However, even with the addition of level sets, the inner and outer walls may still be over‐segmented and the DL‐CNN‐assisted level sets may incorrectly segment parts of the prostate that overlap with the outer bladder wall. The outer wall segmentation was improved compared to our previous method and the DL‐CNN‐assisted level sets were also able to segment the inner bladder wall with similar performance. This study shows the DL‐CNN‐assisted level set segmentation tool can effectively segment the inner and outer wall of the bladder.

1. Introduction

The American Cancer Society estimates that in 2017, 76 900 people will be diagnosed with bladder cancer (58 950 in men and 18 010 in women) and that bladder cancer will be responsible for 16 390 deaths (11 820 in men and 4570 in women).¹

Early detection of bladder cancer is possible using a multidetector row CT (MDCT) urography exam, which can image the bladder, kidneys, and ureters in a single scan.^{2, 3, 4, 5, 6} Unfortunately, the interpretation of a CT urography (CTU) exam is a time‐intensive process. A CTU scan has on average 300 slices in each exam with a range of 200 to 600 slices. Since multiple lesions are possible in different areas of the scan, the entire set of slices must be carefully examined by the radiologist to determine if a lesion is present, which involves frequent adjustment of images in order to best view each slice. Additionally, CTU scans often have many urinary anomalies and the radiologist must determine the likelihood that each anomaly is benign or cancerous. These challenges in CTU interpretation lead to large variance among radiologists in detecting bladder cancer. Studies showed that the sensitivity ranges from 64% to 97% among radiologists.^{7, 8}

Due to the large inter‐radiologist variability and the substantial chance that a radiologist may miss a potentially cancerous lesion, it is imperative to develop techniques that make diagnosing bladder cancer easier and more accurate. To accomplish this goal, we are developing a computer‐aided detection (CAD) system to assist with diagnosis of urothelial neoplasms including asymmetrical bladder wall thickenings. A critical step in developing a CAD system is the segmentation of the bladder and bladder wall as it determines the search region for the subsequent steps. If the bladder is under‐segmented such that parts of the bladder are not included within the segmented region, the search may miss the lesions that appear outside the segmented region. Without segmentation or with an over‐segmented bladder, nonbladder structures may be identified as lesions and also distract the detection of true lesions, thereby increasing the chance of both false positives and false negatives. Additionally, lesions typically manifest in the wall of the bladder. For these reasons, accurate segmentation of the bladder wall from the interior of the bladder and the outer structures is a crucial step in the development of a CAD system for urinary cancer.

Segmentation of the bladder wall in CTU is a challenging task due to a number of confounding factors. Bladders may be partially or completely filled with intravenous contrast material or have no contrast material at all. The different conditions create an inconsistent boundary between the wall of the bladder, the interior, and the outside structures. The boundary between the contrast material inside the bladder, the bladder wall, and the surrounding structures is a weak boundary and the CTU images are often very noisy. A common difficulty in segmenting the bladder wall is the prostate in male patients. For some cases, the prostate protruding into the bladder has a similar appearance as the bladder wall. Often the interface between the bladder wall and the prostate is difficult to distinguish. A bladder wall segmentation method based on simple thresholding tends to leak into the portions of the prostate and the surrounding structures. The boundary between the noncontrast and contrast regions of the bladder interior strongly resembles a wall, further confounding the segmentation process. Adding to these problems is the irregular sizes and shapes of the bladder. Even healthy bladders can be shaped oddly or appear to have anomalies, making it extremely difficult to accurately segment the bladders with any conventional methods.

Previous researchers have attempted to segment the bladder. Li et al.⁹ and Duan et al.¹⁰ used magnetic resonance (MR) images to segment the bladder wall. Duan et al.¹¹ used a window setting scheme to segment and detect tumors in MR images. Han et al.¹² used a Markov random field model and level sets to segment MR images. Chai et al.¹³ developed a segmentation system for cone beam CT, but relied on population data as prior knowledge which can cause errors in cases that deviate from the training set. These methods have the advantage of being relatively fast on modern hardware, but these studies all employed small data sets of no more than 22 patients. Other researchers have attempted to use level sets to segment the bladder wall. Ma et al.¹⁴ used level sets to segment the inner wall in MR images, but relied on a separate Chan–Vese model to segment the outer wall. Chi et al. used coupled level sets to segment the bladder wall in MR images, but only on a small data set of 11 patients.¹⁵ Hadjiiski et al.^{16, 17} initially proposed a level set segmentation method for bladders in CTU. Hadjiiski et al.¹⁸ subsequently developed a new system based on the characteristics of bladder CTU images called conjoint level set analysis and segmentation system (CLASS). Cha et al.¹⁹ further improved the segmentation accuracy of the CLASS method and expanded the data set to the one used in the current study. This method showed improved results over previous attempts to segment the bladder, but relied on two manually marked bounding boxes for the contrast‐filled and noncontrast‐filled regions, respectively human inputs as a starting point.

In this study, we explored the use of a deep‐learning convolutional neural network (DL‐CNN) to segment the bladder wall. Convolutional neural network (CNN)²⁰ was first applied to medical image pattern recognition by Lo et al.²¹ for lung nodule detection and has been used for the detection of breast lesions^{22, 23, 24, 25, 26, 27, 28, 29, 30} since the early 1990's. These early applications used relatively shallow CNN and typically trained with smaller data sets due to the extensive training time and limited availability of medical images. As graphical processing units (GPUs) become more commonly available and powerful, in combination with the design of better convergence and regularization techniques, the application of CNN with large number of layers and trained with larger data sets to complex pattern recognition tasks becomes more practical. Krizhevsky et al.^{31, 32} demonstrated the effectiveness of a DL‐CNN in image classification on the ImageNet ILSVRC‐2010 and ILSVRC‐2012 data sets,³³ and the CIFAR‐10 data set.³⁴ Since then, there have been strong interests in applying DL‐CNN to all kinds of medical imaging problems.³⁵ We explored the use of a DL‐CNN in the segmentation of the entire bladder³⁶ and the bladder wall in previous pilot studies³⁷. In Cha et al., a DL‐CNN with level sets was shown to be more effective in segmenting the bladder than previous level set systems or other standard image segmentation techniques. In the current study, we further investigated the segmentation of the bladder wall as a step toward bladder lesion detection. The DL‐CNN in this study was trained to distinguish the bladder wall and the output of the DL‐CNN was used to guide level set segmentation of the inner and outer bladder wall.

This paper is organized as follows. First, the data set for this study is described. Second, the details and use of the DL‐CNN is described. Third, the level set segmentation method is presented and lastly, the results are reported and discussed.

2. Materials and methods

A DL‐CNN was trained to distinguish between ROIs inside the bladder wall and ROIs not within the bladder wall. The DL‐CNN outputs a likelihood map where the brightness of a given pixel represents the likelihood that the pixel falls within the bladder wall. The likelihood map is then used as a gradient image in cascaded level sets to segment the inner and outer bladder wall. The process is shown in the flowchart in Fig. 1.

Figure 1.

Flowchart of deep‐learning convolutional neural network (DL‐CNN) segmentation that shows generation of a bladder wall likelihood map and use of level sets to obtain inner and outer wall contours. The DL‐CNN likelihood map was used as a gradient image in the energy equation of a cascaded level sets method. [Color figure can be viewed at wileyonlinelibrary.com]

2.A. Data set

With IRB approval, a data set of 172 patients who had undergone CTU was collected for this study. The cases were collected retrospectively from the Abdominal Imaging Division of the Department of Radiology at Michigan Medicine. All cases were acquired with established CTU protocol during the patients’ clinical care. Within this set of 172 cases, we designated 81 as part of the training set and the remaining 91 as part of the test set. The cases were balanced in difficulty by appearance and shape of the bladder between the training and the test set.

In the training set, 42 bladders contained focal mass‐like lesion, 21 had wall thickenings and 18 were normal. Forty of the bladders that contained focal mass‐like lesion were malignant. Sixteen of the wall thickened bladders were also malignant. Sixty‐one bladders in the training set were partially filled with IV contrast material, 8 were fully filled, and 12 were not filled. In the test set, 42 bladders contained focal mass‐like lesion, 36 had wall thickenings and 13 were normal. Forty‐two of the bladders that contained focal mass‐like lesion and 23 of the bladders with wall thickenings were malignant. Eighty‐four of the test set bladders were partially filled with contrast material, four were fully filled, and three were not filled.

The inner and outer bladder walls were hand‐outlined by an abdominal radiologist with more than 20 yr of experience for both the training and the test sets using an in‐house developed graphical user interface that we named MiViewer. The hand outlines of the training set were used to classify the regions of interest (ROIs) for the DL‐CNN training. The outlines were also used as the reference standard for the evaluation of the segmentation performance (see Section 2.D). Both the outer bladder wall and the inner bladder wall were outlined on each two‐dimensional (2D) slice to form a three‐dimensional (3D) surface contour of both bladder walls. The 172 bladders contained a total of approximately 16 000 slices, which equates to about 100 slices per bladder.

2.B. Bladder likelihood map generation using deep‐learning convolutional neural network (DL‐CNN)

We applied the DL‐CNN called Cuda‐Convnet developed by Krizhevsky³¹ to the classification of voxels of 2D CTU slices as being inside the bladder wall or outside the bladder wall. The DL‐CNN was trained using ROIs extracted from the 2D slices of the training cases and labeled as within or not within the bladder wall. After training, the DL‐CNN was applied to all pixels of the CTU images in the training and test set to estimate the likelihood that an ROI was within the bladder wall. The output was then assigned to the center pixel of the ROI, resulting in a likelihood map where brighter pixels were more likely to be within the bladder wall and the stack of the resulting 2D likelihood maps on the CT slices formed a 3D likelihood map for the bladder wall.

2.B.1. DL‐CNN components

The components and layers of the DL‐CNN are described here. More information on these components can be found in the literature.^{31, 36}

Neurons

A neuron is defined by the following function,

$$f(x)=max(0,x)$$ (1)

where x is the sum of the weighted inputs to the neuron. The activation function used in the DL‐CNN is a RE‐LU [Eq. (1)], which converges faster than the typically used sigmoid activation function.³⁸

Convolution layer

The convolutional layer uses convolutional kernels to convolve with the input ROI. The resulting values are then input to the corresponding neurons in the kernel maps within the convolutional layer. The output of these neurons is determined by the activation function in Eq. (1).

Locally connected convolutional layer

The locally connected convolutional layer performs the same set of operations as the convolutional layer, but different convolutional kernels are applied at each location of the input image and then collected into the corresponding neurons instead of using a single convolutional kernel.

Fully connected layer

The fully connected layer connects all inputs from the previous layer. Every map element multiplied by a trained weight is used as input. The fully connected layer outputs values, which are passed onto the next fully connected layer or finally to a Softmax layer to obtain the final output of the DL‐CNN. There can be one or more fully connected layers before the Softmax layer in a DL‐CNN architecture.

Pooling layer

Pooling layers summarize and reduce the outputs of groups of neighboring neurons. Our pooling layers use overlapping pooling due to its tendency to reduce overtraining.

Local response normalization layer

Local normalization layers aid in the generalization of training by normalizing input parameters to subsequent layers of the DL‐CNN.³⁹ The output of neurons was normalized by the following equation:

$$b_{x,y}^i=\frac{a_{x,y}^i}{{\left(1+\frac{\tau }{N}{\sum }_{j=max\left(0,i-\frac{N}{2}\right)}^{min\left(n,i+\frac{N}{2}\right)}{\left(a_{x,y}^j\right)}^2\right)}^{\epsilon }}$$ (2)

where $b_{x,y}^i$ is the response‐normalized neuron activity, $a_{x,y}^i$ is the neuron activity computed by applying the kernel $i$ at the coordinates (x, y), $n$ is the number of kernel maps, and $N$, $\tau$, and $\epsilon$ are constants. Our implementation of the DL‐CNN used N = 9, τ = 0.001, and ɛ = 0.75, which was demonstrated to be effective.^{31, 36} These parameters were taken from Krizhevsky et al.³¹ and were further reinforced as ideal choices from our own experimentation.

2.B.2. DL‐CNN architecture

The architecture of the DL‐CNN used in this study consists of five main layers: two convolutional layers, two locally connected convolutional layers, and one fully connected layer, in that order (Fig. 2). The first two convolutional layers are followed by a pooling and by a normalization layer before proceeding to the next layer.

Figure 2.

Block diagram showing our deep‐learning convolutional neural network (DL‐CNN) architecture. The input layer is the classified ROIs and the output of the trained DL‐CNN creates a likelihood map.

The first two convolutional layers consist of 64 kernels of size 5 × 5. An ROI is input to the first convolutional layer. The output of the first layer is pooled and normalized and then used as input in the second convolutional layer. After another pooling and normalization layers, the output of the second convolutional layer is sent to a locally connected convolutional layer with 64 kernels of size 3 × 3. The second locally connected convolutional layer contains 32 kernels of size 3 × 3. The fully connected layer outputs two values used as input to the Softmax layer defined by the function:

$$f(x_i)=\frac{e^{x_i}}{{\sum }_j{e}^{x_j}}$$ (3)

where x_i is each input value. The layer outputs a single value in [0,1], which represents the likelihood of the input ROI being in the bladder wall.

2.B.3. DL‐CNN training

The DL‐CNN was trained using the 81 cases in the training set. Each 2D axial slice of the CTU scan was divided into overlapping ROIs of M x M pixels. Following the inner and outer wall of the bladder of each slice hand‐outlined by a radiologist, the ROIs were labeled as within or outside the bladder wall. Three ROI sizes, M = 8, 16, and 32 were studied and the size of 16 × 16 pixels was chosen. For each 16 × 16 ROI, the central 8 × 8 pixel image was used in the labeling criterion. If 70% of the inner 8 × 8 pixel image fell within the bladder wall, that is, the pixels were located between the hand‐outlined contours of the inner wall and the outer wall, the ROI was labeled as inside the bladder wall. Seventy percent was chosen to ensure that a large fraction of the ROI area was within the wall, while not making the classification requirement so strict as to having too few ROIs that could be labeled as within the wall because some of the walls were very thin. If 95% of the inner ROI fell within the interior of the bladder, or if less than 10% of the ROI was located inside the bladder delineated by the outer wall contour, the ROI was labeled as not within the bladder wall. These classification requirements were chosen to properly label ROIs along the wall as not within the wall while still obtaining a sufficient number of ROIs within the wall. Figure 3 shows a bladder with the ROI images superimposed on top.

Figure 3.

Regions of interest (ROIs) superimposed on a CTU slice. The darker boxes are ROIs determined to be within the wall and the lighter boxes are ones determined not to be within the wall. The number of ROIs is balanced, so that the training ROIs contain exactly half within each category. [Color figure can be viewed at wileyonlinelibrary.com]

Approximately, 240 000 ROIs were extracted from the training cases after balancing. Exactly half of the ROIs were labeled as within the bladder wall and the other half as not within the bladder wall. Subsets of the ROIs are presented in Fig. 4.

Figure 4.

16 × 16 pixel regions of interest (ROIs). (a) ROIs labeled as being within the wall; (b) ROIs labeled as not within the wall.

The DL‐CNN was trained for 1500 iterations based on our previous experience³⁶ on the training set to produce the bladder likelihood maps. The training process typically took about 7 to 8 h to complete with a Nvidia Tesla K20 GPU.

2.B.4. Bladder likelihood map generation

We applied the trained DL‐CNN to the CTU scans to estimate the likelihood of a given voxel being within the bladder wall. For each scan, a rectangular box enclosing the bladder was drawn to denote the volume of interest (VOI). The trained DL‐CNN was applied to each voxel within the VOI. For each voxel, a 16 × 16 pixel ROI of the axial slice centered at that voxel was input to the DL‐CNN, which output a likelihood score between 0 and 1 that represented the likelihood that the center pixel of the ROI was within the wall. The scores for each ROI were assigned to the center voxel in order to create a map of the output scores from the DL‐CNN. The collection of likelihood scores for all voxels in the VOI provided the 3D bladder wall likelihood map. Examples of bladder wall likelihood maps on the axial slices are shown in Fig. 5.

Figure 5.

Bladder wall likelihood maps (b), (d) shown with the CTU slices (a), (c) from which they were generated. The brighter pixels represent a higher likelihood of that pixel occurring in the wall.

2.C. Segmentation of bladder walls using level sets

After the generation of the bladder likelihood maps, we developed a system to segment the inner and outer bladder wall using level sets. The system takes in the 3D bladder wall likelihood map and incorporates it within the level set energy functions to guide the segmentation and creates a contour of the inner or outer wall depending on which set of parameters is used. The system consists of four stages: (a) preprocessing, (b) initial segmentation, (c) 3D level set segmentation, and (d) 2D level set segmentation.

The first stage consists of preprocessing techniques to create a set of gradient vector images. Smoothing, anisotropic diffusion, gradient filters, and the rank transform of the gradient magnitude are applied to slices within the VOI. The vector images created at this stage are used during the propagation of the 3D level sets.

The second stage generates the initial segmentation surface and consists of three parts. First, we apply a threshold to the bladder wall likelihood maps to create a binary mask. The mask DL _Mask is generated according to the following criterion:

$$D{L}_{\mathit{Mask}}(x,y)=\left\{\begin{array}{cc}1,\hfill & D{L}_{\mathit{Score}}\left(x,y\right)\ge \theta \hfill \\ 0,\hfill & D{L}_{\mathit{Score}}\left(x,y\right)<\theta \hfill \end{array}\right.$$ (4)

where x and y are coordinates in the image, DL _Mask is the value of the resulting binary mask at the given coordinate, DL_Score is the value of the likelihood map score at the given coordinate, and θ is a chosen value for the threshold. The value of θ was determined experimentally³⁶ by histogram analysis. By using the training cases, a histogram of the DL‐CNN likelihood scores for the pixels inside and outside of the bladder within the VOIs was generated. The threshold of 0.85 provided a good separation of the two classes and the best contours that did not leak to the outside of the bladder wall while closely approaching the hand segmentation for cases in the training set.

The second part of the initial segmentation process is the creation of an ellipsoid as the object region for segmentation. An ellipsoid with major and minor axis 1.5 times the width and height of the VOI, respectively, is placed at the centroid of the bladder mask. The intersection between this ellipsoid and bladder mask is the object region. This ellipsoid prevents the segmentation from leaking into the surrounding structures outside the bladder by limiting the area to which the contours can expand. This limited object region prevents interference from nonbladder structures and solves the problem of high likelihood scores on other regions of the CT scan such as the pelvic bone, which often receives high likelihood scores.

Finally, neighboring components in the object region are connected using a morphological dilation filter with a 2‐voxel‐radius spherical structuring element, a 3D flood fill algorithm, and a morphological erosion filter with a 2‐voxel‐radius spherical structuring element. This process allows for the extraction of an initial segmentation surface ${\varphi }_0(\overrightarrow{x})$ from the bladder wall likelihood map for use in the level set propagation.

The third stage of the system propagates the initial segmentation toward the inner or outer bladder wall using cascading level sets. The level set contours are generated according to the chosen level set equation:

Chemical Formula: $$\mathrm{\Psi }(\mathit{x},n=0)={\varphi }_0(\overrightarrow{x})$$

α, β, and γ are the coefficients for the advection, propagation, and curvature terms, respectively. ($\mathrm{A}(\overrightarrow{x})$ is a vector field image where each voxel in the image is a vector. ($\mathrm{P}\overrightarrow{x}$) is a scalar speed term between 0 and 1.²³ κ($\overrightarrow{x}$) is the mean curvature of the level set at the point $\overrightarrow{x}$ and can be defined as

(6).

Chemical Formula: $$\kappa (\overrightarrow{x})=\text{div}(\mathrm{\nabla }\mathrm{\Psi }(\overrightarrow{x})/|\mathrm{\nabla }\mathrm{\Psi }(\overrightarrow{x})|)$$

The symbol ∇ denotes the gradient operator, div is the divergence operator, ${\varphi }_0(\overrightarrow{x})$ is the initial segmentation, and n is the number of iterations. The bladder wall likelihood map is used as the $\mathrm{\Psi }(\overrightarrow{x})$ term in the fourth level set as an integral part of the advection, propagation, and curvature terms, while the original CTU volume is used for the first three level sets.

The advection term drives the contour toward regions of high gradient according to the $\mathrm{A}(\overrightarrow{x})$ vector. The propagation term controls the expansion of the contours according to local pixel information. The curvature term causes the contour to maintain relative shape and curvature while expanding.

Four different 3D level sets are applied sequentially. The first level set expands the initial contours slightly and smooths edges. The second level set brings the contours toward sharp edges and draws it slightly in regions of low gradient. The third level set finishes drawing the contour toward edges. The fourth level set draws the contours toward the inner and outer bladder walls, using the bladder wall likelihood map for the $\mathrm{\nabla }\mathrm{\Psi }(\overrightarrow{x})$ term in Eq. (5). The level set equations are run twice, once for the outer contour and once for the inner contour, using different parameters.

(5).

Chemical Formula: $$\frac{\partial }{\partial t}\mathrm{\Psi }(\overrightarrow{x})=-\alpha A(\overrightarrow{x})\mathrm{\nabla }\mathrm{\Psi }(\overrightarrow{x})-\beta P(\overrightarrow{x})\left|\mathrm{\nabla }\mathrm{\Psi }(\overrightarrow{x})\right|+\gamma \kappa (\overrightarrow{x})\left|\mathrm{\nabla }\mathrm{\Psi }(\overrightarrow{x})\right|,$$

As the final step in the level set system, 2D level sets are applied to each slice of the 3D segmented object, using the 3D level set generated contours on each slice as an initial contour, to further refine the contours. Further details on the methods of cascading level sets and the necessity of 2D level sets to refine segmentation can be found in our previous publications.^{16, 36, 40}

Table 1 shows the coefficient information for each round of level sets. The differences in the runs for outer and inner wall segmentation are the parameters α, β, and γ and the number of iterations (n) in the fourth level sets. Notably, the propagation coefficient β is positive when running the level set equations for the outer wall, so the contour expands to the outer wall and negative for the inner wall, so the contour propagates inwards. The coefficients used for the first 3 rounds of level sets are the same as those used in our previous bladder segmentation.³⁶ The parameter q in Table 1 is defined by the linear function $\sigma M+\varphi$, where M is the 2D diagonal distance of the VOI in mm, $\sigma =0.06,\text{and}\varphi =-0.11$ as shown in previous work.³⁶

Table 1.

Level set parameters. The parameters listed in the “Fourth” row are listed as inner wall parameter/outer wall parameter

Level sets	α	β	γ	n
First	1	2	1	10
Second	1	0.6	q	150
Third	0	1	0	10
Fourth	4.5/3	−2.5/2	2/2	150/100
2D slices	4	0.2	0.5	100

2.D. Evaluation methods

Segmentation performance was evaluated by comparing the generated contours to 3D hand‐segmented contours. The inner and outer walls were each independently compared to the hand‐outlined inner and outer wall contours. We calculated the volume intersection ratio, the volume error, the absolute volume error, and the minimum distance as performance metrics, defined below. Additionally, we compared the area between the inner and outer wall contours to that of the hand‐outlined contours using the volume intersection, the volume error, and the absolute volume error.

The volume intersection is the ratio of the intersection between the given volume enclosed by the contour generated by the level sets and the reference volume enclosed by the reference contour, to the reference volume:

$$R^{3D}=\frac{V_R\cap V_U}{V_R}$$ (7)

where V_U is the given volume and V_R is the reference volume.

The volume error is defined as the ratio of the difference between the reference volume and the given volume to the reference volume:

$$E^{3D}=\frac{V_R-V_U}{V_R}$$ (8)

Since the given volume is segmented by the level sets, a positive error represents under‐segmentation of the bladder wall, while a negative error indicates over‐segmentation. We also calculate the absolute volume error |E^3D| to show the average deviation from the reference contours. Other performance indicators can be derived from the volume intersection ratio and the volume error.⁴¹

AVDIST is the average distance between the closest points on the reference and the given contours:

$$\begin{array}{cc}& AVDIST=\hfill \\ & \frac{1}{2}\left(\frac{{\sum }_{x\in R}min\left\{d\left(\mathit{x},\mathit{y}\right):\mathit{y}\in U\right\}}{N_R}+\frac{{\sum }_{y\in U}min\left\{d\left(\mathit{x},\mathit{y}\right):\mathit{x}\in R\right\}}{N_U}\right)\hfill \end{array}$$ (9)

R is the reference contour, U is the given contour, and N_R and N_U are the number of voxels along each respective contour. The function d is the minimum Euclidian distance between a given voxel x on the contour R to a voxel $\overrightarrow{\mathit{y}}$ on the contour U. The minimum distances of all points along R are calculated and then averaged. The minimum distances of all points along U are also calculated and averaged by repeating the process with the roles of R and U switched. The overall average distance AVDIST between the two contours is then calculated from the two average minimum distances.

The volume intersection ratio, the volume error, and the absolute volume error are also calculated for the region between the inner and outer wall contours. These metrics are calculated in the same way as for the outer and inner wall contours, except that V_R and V_U are the volume of the shell enclosed between the outer wall contour and the inner wall contour.

The volume intersection ratio is also calculated for the lesions and the outer wall contour to determine what percent of the lesions is successfully enclosed by the segmentation.

3. Results

3.A. Segmentation performance of inner and outer wall contours

Examples of segmentation results from the test cases are shown in Fig. 6. The DL‐CNN bladder wall likelihood maps that were used to guide the segmentation are also shown.

Figure 6.

(a) and (c) are bladder wall likelihood maps generated from CTU scans from the test set (b) and (d), respectively. The level set contours are overlaid on the CTU slices. [Color figure can be viewed at wileyonlinelibrary.com]

Histograms for the training and test sets for the volume intersection ratio, the volume error, and the average distance are shown in Fig. 7.

Figure 7.

Histograms of the volume intersection % (a, b), volume % error (c, d), and average distance (e, f) for the training and test sets, respectively.

The average segmentation performances for the training and test sets using the different performance metrics are shown in Table 2.

Table 2.

Summary of the performance metrics for the inner and outer bladder walls

	Volume intersection (%)	Volume error (%)	Absolute volume error (%)	Average distance (mm)
Training
Inner	90.2 ± 8.7	−4.3 ± 18.2	12.6 ± 13.7	3.0 ± 1.6
Outer	93.2 ± 5.8	−7.2 ± 12.3	10.4 ± 9.6	3.0 ± 1.2
Test
Inner	87.2 ± 10.5	−5.3 ± 28.2	15.6 ± 24.0	3.2 ± 1.7
Outer	89.5 ± 9.8	−6.2 ± 20.5	14.6 ± 15.6	3.5 ± 2.0

For the training set, the inner wall contour achieved average volume intersection ratio, average volume error, average absolute volume error, and average distance of 90.2 ± 8.7%, −4.3 ± 18.2%, 12.6 ± 13.7%, and 3.0 ± 1.6 mm, respectively. For the test set, the inner wall achieved values of 87.2 ± 10.5%, −5.3 ± 28.2%, 15.6 ± 24.0%, and 3.2 ± 1.7 mm, respectively. For the training set, the outer wall contour achieved the values of 93.2 ± 5.8%, −7.2 ± 12.3%, 10.4 ± 9.6%, and 3.0 ± 1.2 mm, respectively. For the test set, the outer wall contour achieved values of 89.5 ± 9.8%, −6.2 ± 20.5%, 14.6 ± 15.6%, and 3.5 ± 2.0 mm, respectively.

3.B. Performance of combined wall contours

For the segmented bladder wall, the training set achieved volume intersection ratio, volume error, and absolute volume error of 61.0 ± 11.3%, −13.7 ± 49.1%, and 34.5 ± 37.3%, respectively. The test set achieved values of 54.6 ± 10.4%, 10.7 ± 28.0%, and 25.1 ± 15.8%, respectively.

Due to the small volume of the wall compared to the entire bladder volume enclosed by either the inner or the outer walls, small differences between the contours lead to large fluctuations in the measured values, which is the main reason that the performance metrics are worse for the segmented wall than the individual inner and outer wall contours.

3.C. Lesion intersection

To determine if the DL‐CNN with level sets enclosed bladder lesions within the contours, we evaluated the volume intersection between the lesion as the reference contour and the outer wall as the given contour. Histograms of the result are shown in Fig. 8. For the DL‐CNN‐outlined walls, the average volume intersection ratio is 80.3 ± 23.8% for the training set and 81.6 ± 16.6% for the test set. Of the lesions, 70.2% have a volume intersection ratio over 75% with the DL‐CNN‐assisted level sets contours compared to 89.1% for the hand‐outlined contours.

Figure 8.

Histogram of the lesion volume intersection. The graph (a, b) shows results both for the radiologist hand‐outlined contours and the deep‐learning convolutional neural network with level set contours for the training and test set, respectively.

3.D. Level set method comparison

Table 3 shows a comparison between performing segmentation by using the likelihood map as the $\mathrm{\Psi }(\overrightarrow{x})$ term in only the fourth level set and by using the likelihood map as the $\mathrm{\Psi }(\overrightarrow{x})$ term in all four stages of the cascading level sets. When the likelihood map was used only in the fourth level set, the volume intersection index was significantly better (P < 0.01) for the outer wall in both the training and test set. For the inner wall, the difference was not significantly different in either the training or the test set.

Table 3.

Comparison between using the likelihood map for all level sets and using the likelihood map for only the fourth level set

		Volume intersection (%)	Volume error (%)	Absolute volume error (%)	Average distance (mm)
Training
Inner	All level sets	93.5 ± 4.5	−14.0 ± 25.0	16.5 ± 23.3	3.0 ± 1.4
	4th level set only	90.2 ± 8.7	−4.3 ±  18.2	12.6 ± 13.7	3.0 ± 1.6
Outer	All level sets	81.8 ± 8.0	13.3 ± 9.8	14.1 ± 8.5	3.8 ± 1.5
	4th level set only	93.2 ± 5.8	−7.2 ± 12.3	10.4 ± 9.6	3.0 ± 1.2
Test
Inner	All level sets	88.5 ± 10.3	−8.0 ± 27.7	16.8 ± 23.4	3.3 ± 1.7
	4th level set only	87.2 ± 10.5	−5.3 ± 28.2	15.6 ± 24.0	3.2 ± 1.7
Outer	All level sets	76.1 ± 11.9	18.0 ± 15.5	19.8 ± 13.0	4.7 ± 2.4
	4th level set only	89.5 ± 9.8	−6.2 ± 20.5	14.6 ± 15.6	3.5 ± 2.0

To further demonstrate the effectiveness of the level set method with the DL‐CNN, we compared the DL‐CNN with level sets method to contours created using the DL‐CNN without level sets (Table 4). The DL‐CNN without level sets tended to over‐segment the inner wall but under‐segment the outer wall. With the refinement by the level sets, both the over‐segmentation of the inner wall and under‐segmentation of the outer wall were reduced significantly (P < 0.01), as indicated by the volume error, absolute volume error, and average distance.

Table 4.

Comparison between bladder contours generated by using the DL‐CNN with level sets and contours generated directly from the DL‐CNN likelihood map without applying level sets

		Volume intersection (%)	Volume error (%)	Absolute volume error (%)	Average distance (mm)
Training
Inner	DL‐CNN without level sets	97.6 ± 2.1	−33.8 ± 36.1	33.8 ± 36.1	4.0 ± 1.9
	DL‐CNN with level sets	90.2 ± 8.7	−4.3 ± 18.2	12.6 ± 13.7	3.0 ± 1.6
Outer	DL‐CNN without level sets	72.8 ± 8.7	24.3 ± 8.8	24.3 ± 8.8	5.2 ± 1.6
	DL‐CNN with level sets	93.2 ± 5.8	−7.2 ± 12.3	10.4 ± 9.6	3.0 ± 1.2
Test
Inner	DL‐CNN without level sets	94.2 ± 8.0	−29.3 ± 32.6	30.9 ± 31.1	4.1 ± 2.2
	DL‐CNN with level sets	87.2 ± 10.5	−5.3 ± 28.2	15.6 ± 24.0	3.2 ± 1.7
Outer	DL‐CNN without level sets	76.5 ± 12.0	17.7 ± 15.2	19.5 ± 13.0	4.6 ± 2.5
	DL‐CNN with level sets	89.5 ± 9.8	−6.2 ± 20.5	14.6 ± 15.6	3.5 ± 2.0

4. Discussion

We developed a segmentation method to extract the bladder wall from the interior of the bladder and the surrounding structures. The method uses a deep‐learning convolutional neural network combined with a series of cascading level sets to detect the contours for the inner and outer walls of the bladder in CTU scans. Segmentation of the wall presents many challenges; some are associated with segmentation of the bladder in general, and others are unique to the segmentation of the wall specifically. The training and test sets of CTU scans contain bladders completely filled with contrast material, partially filled, and not filled. The boundary between the wall and the interior of the bladder is drastically different depending on whether contrast material is present. The combination of the DL‐CNN and level sets is able to account for the difference in the regions, demonstrating that the DL‐CNN can overcome the strong barrier between regions, even for narrow regions such as the bladder wall.

The small thickness of the bladder wall presents other problems for segmentation as well. Bladder walls can vary in thickness drastically compared to the full bladder. Thin walls can be only few pixels wide compared to bladder walls with urothelial thickenings. The DL‐CNN often has difficulties with this variation while the level sets are able to account for this difference. The DL‐CNN and the level sets play complementary roles in segmenting the bladder wall and generating the likelihood maps.

The presence of lesions also presents another confounding factor for segmentation of the wall. Our DL‐CNN‐assisted level sets method often generates inner wall contours through the lesions rather than around them (Fig. 9), although the bladder wall likelihood maps typically include the lesions accurately during the estimation of the bladder wall.

Figure 9.

Segmented contours on a CTU slice. The inner contour goes through the lesion rather than around it. The small contours in the lower right of the image are ignored. [Color figure can be viewed at wileyonlinelibrary.com]

The 16 × 16 ROI size was chosen after experimenting with ROIs of 8 × 8, 16 × 16, and 32 × 32 pixels in size. The 8 × 8 pixel ROIs tended to generate too much noise in the bladder wall likelihood maps, specifically along the barrier between the contrast and noncontrast region. The 8 × 8 pixel ROIs also had the tendency to create gaps in parts of the wall that were extremely thin [Fig. 10(b)]. The 32 × 32 pixel ROIs resulted in walls that were far too thick in the bladder likelihood maps. Due to the small thickness of the wall, ROIs of 32 × 32 pixels were too large to accurately represent the wall [Fig. 10(d)]. The 16 × 16 pixel ROIs were chosen as a middle ground because they provided bladder wall likelihood maps that did not have as many gaps or as much noise as those obtained with the 8 × 8 pixel ROIs, but provided thinner walls that resulted in more accurate contours than the 32 × 32 pixel ROIs [Fig. 10(c)].

Figure 10.

Likelihood maps of the CTU slice shown in (a). (b) was generated using 8 × 8 regions of interest (ROIs), (c) by 16 × 16 ROIs, and (d) by 32 × 32 ROIs. All likelihood maps were shown with the same brightness and contrast window settings.

To further prevent holes in the bladder wall likelihood maps, we developed a method for the generation of the ROIs that can capture the thin regions of the bladder wall, while still using an ROI size large enough to accurately capture the wall features in thicker walls. In this method, a 16 × 16‐pixel ROI is used to train the deep‐learning algorithm, but only the central 8 × 8‐pixel area of the ROI is used for the labeling of the ROI, referred to as 8 × 8 jittering. If 70% of this inner 8 × 8 pixel area falls between the hand‐outlined outer and inner wall contours, the ROI is labeled as within the bladder wall; otherwise, the ROI is classified as not within the bladder wall. Using the central smaller area rather than the whole ROI allows more ROIs to be correctly labeled as within a thin wall, while preventing more ROIs outside the wall from being mislabeled as within the wall.

This jittering technique and the added level set equation were important methodological changes from the method used in Cha et al.³⁶ The addition of another round of level sets that incorporate the DL‐CNN likelihood map in the equation allows the level sets to segment both the inner and outer bladder wall as opposed to just the entire bladder as a whole.

The only small drawback in the use of 16 × 16 ROIs is a slight increase in the training time; an increase from 5.5 to 6.5 h due to an increase in the number of training ROIs compared to the use of 32 × 32 ROIs. However, both 16 × 16 and 32 × 32 ROI sizes take about 4 min to generate a bladder likelihood map during deployment, so this drawback is a minor one. The ROI size does not affect the time required for running the level sets either, which takes about 2 to 5 min per contour.

The likelihood map was chosen to be used only in this fourth level set based on our experimental results (Table 3). When the likelihood map was used only in the fourth level set, the segmentation results were more accurate. The fourth level set is crucial to differentiating between the outer wall and the inner wall for the level set segmentation.

In the cascaded level sets pipeline, each level set shows progressively improved results over the previous level set, justifying the need for each subsequent level set. Table 5 shows the volume intersection ratios from the third to the last level sets.

Table 5.

The volume intersection ratios after the 3rd level set, the 4th level set, and the full level set cascade. The segmentation by the full cascaded level sets achieved the best performance

	After 3rd level set (%)	After 4th level set (%)	Full level set cascade (%)
Outer training set	77.7 ± 7.9	89.7 ± 6.8	93.2 ± 5.8
Inner training set	77.7 ± 7.9	79.3 ± 7.0	90.2 ± 8.7
Outer test set	72.4 ± 11.8	84.3 ± 11.4	89.5 ± 9.8
Inner test set	72.3 ± 11.5	74.9 ± 10.6	87.2 ± 10.5

The inclusion of lesions within the bladder wall contours is important for the bladder wall segmentation as the segmented wall defines the search region for bladder lesions in the subsequent steps of a CAD system. A segmentation method that can reliably include all lesions is therefore critical for the CAD system to identify cancerous lesions or find wall thickenings. The results of this study show that our proposed method is promising for this application.

Compared to the study by Cha et al.³⁶ on the same data set, the current study showed better performance. The purpose of the study by Cha et al.³⁶ was to segment the entire bladder, so that they only needed to segment the outer bladder wall and also used a DL‐CNN with level sets. The volume intersection ratios for the training set and test set achieved in the current study were significantly better (P < 0.01) than those in the previous study. The volume intersection ratios were 93.7% and 89.5% for the training and test sets, respectively, in this study compared to 84.2% and 78.0%, respectively, by Cha et al.³⁶ The differences in the absolute percent volume errors for the training set and test set are not significant, however.

In order to demonstrate the effectiveness of our proposed deep‐learning approach to this problem, we compared the performance for the outer wall segmentation from this study with the previous attempt at segmenting the same data set using an adaptive level set system (CLASS).¹⁶ We also applied a commonly used image segmentation tool, ITK‐SNAP, to a subset of our CTU data as it was used by Ma et al.^{13, 14} to segment the bladder walls in T2‐wieghted MR images. The performance of the DL‐CNN with level sets is significantly better (P < 0.01) than that of ITK‐SNAP and CLASS for the metrics of volume intersection and absolute volume error. Table 6 shows the comparison of the performances by CLASS and DL‐CNN. CLASS is a level set system that does not use a DL‐CNN to assist with segmentation. The improved results of the DL‐CNN with level sets show the value of a deep‐learning approach to bladder segmentation. Table 7 shows the average values for a smaller data set of 30 bladders used for ITK‐SNAP alongside values from the same data set for CLASS and DL‐CNN. Even with the limited scope of previous experiments, the performance of the DL‐CNN with level sets is significantly better than that of ITK‐SNAP and CLASS with a P < 0.01 for the metrics of volume intersection and absolute volume error on the smaller data set.

Table 6.

Comparison of the outer wall segmentation using CLASS versus the proposed DL‐CNN‐assisted level set method

	Volume intersection (%)	Volume error (%)	Absolute volume error (%)	Average distance (mm)
DL‐CNN	89.5 ± 9.8	−6.2 ± 20.5	14.6 ± 15.6	3.5 ± 2.0
CLASS	84.0 ± 11.4	8.2 ± 17.4	13.0 ± 14.1	3.5 ± 1.9

Table 7.

Comparison of the outer wall segmentation by the proposed DL‐CNN‐assisted level set method to those using the ITK‐SNAP and CLASS on 30 cases

	Volume intersection (%)	Volume error (%)	Absolute volume error (%)	Average distance (mm)
DL‐CNN	94.4 ± 3.2	−8.5 ± 9.8	10.0 ± 8.3	3.0 ± 1.2
ITK‐SNAP	78.8 ± 16.3	8.3 ± 33.1	24.2 ± 23.7	5.2 ± 2.6
CLASS	79.0 ± 8.2	16.1 ± 16.3	19.9 ± 11.1	3.5 ± 1.3

One limitation of our current segmentation method was observed for CT scans with thick slices and an unusually small bladder. For small bladders, the level sets cannot change fast enough to account for the rapid differences between slices when slices are 5 mm thick. This problem could potentially be overcome by selecting a different set of parameters that is optimized for thick slices and the small bladders. An automated preprocessing triage stage that can recognize small bladders with thick slices has to be developed and call for the specific set of parameters for the segmentation of these outlier cases. We will continue to optimize the parameters of the level sets in future studies.

The small thickness of the wall also presents problems in the evaluation of the DL‐CNN and level set contours. While the outer and inner contours individually achieved good results compared to the reference standard, the overlap of the wall between the inner and outer contours achieved much lower results in terms of the volume intersection and the volume error. Since the bladder wall can be very thin, small deviations in the wall contours can lead to massive errors that are not observed when comparing the bladder as a whole. Cases in both the training and the test sets with good performance for the inner and outer wall contours had noticeably poorer performance in the wall between the two contours due to these small deviations. Examples of bladder wall segmentations are presented in Fig. 11.

Figure 11.

Comparisons between the hand‐outlined contours and the computer‐segmented contours. (a), (c), and (e) show the outer wall contour, and (b), (d), and (f) the inner contour. The dark and light contours in each image represent the hand‐outlined contour and the computer‐segmented contour, respectively. The 2D area intersection ratio of the thin wall between the inner and outer wall contours (a) 50.0%, (c) 53.0%, and (e) 55.9%. [Color figure can be viewed at wileyonlinelibrary.com]

Another limitation associated with the method is the reliance on hand outlines from a single radiologist. Generating outlines from multiple radiologists and taking the average will reduce the bias for the reference standard, which will allow better estimation of the DL‐CNN performance. However, due to the excessive time required to generate manual hand outlines for such a large data set (a total of approximately 16 000 slices of bladder outlines), only one set of hand outlines was obtained and used for evaluation.

5. Conclusions

Our results in this study show that the DL‐CNN‐assisted level set method is useful for segmentation of the bladder wall in CTU scans. This method can specifically segment the bladder wall by detecting both the inner and outer wall contours of the bladder. The DL‐CNN can accurately differentiate the bladder wall from the interior of the bladder and the surrounding structured background. The segmentation of the wall is an important first step for automated lesion detection and bladder analysis. Further studies are underway to improve the performance of the wall segmentation process, especially for the inclusion of lesions within the wall contours. This study lays the important groundwork for further efforts to develop computerized decision support systems for diagnosis and treatment of bladder cancer.

Conflict of interest

The authors have no conflicts to disclose.