Learning Image Context for Segmentation of Prostate in CT-Guided Radiotherapy

Abstract

Accurate segmentation of prostate is the key to the success of external beam radiotherapy of prostate cancer. However, accurate segmentation of prostate in computer tomography (CT) images remains challenging mainly due to three factors: (1) low image contrast between the prostate and its surrounding tissues, (2) unpredictable prostate motion across different treatment days, and (3) large variations of intensities and shapes of bladder and rectum around the prostate. In this paper, an online-learning and patient-specific classification method based on the location-adaptive image context is presented to deal with all these challenging issues and achieve the precise segmentation of prostate in CT images. Specifically, two sets of location-adaptive classifiers are placed, respectively, along the two coordinate directions of the planning image space of a patient, and further trained with the planning image and also the previous-segmented treatment images of the same patient to jointly perform prostate segmentation for a new treatment image (of the same patient). In particular, each location-adaptive classifier, which itself consists of a set of sequential sub-classifiers, is recursively trained with both the static image appearance features and the iteratively-updated image context features (extracted at different scales and orientations) for better identification of each prostate region. The proposed learning-based prostate segmentation method has been extensively evaluated on 161 images of 11 patients, each with more than 9 daily treatment 3D CT images. Our method achieves the mean Dice value 0.908 and the mean ± SD of average surface distance (ASD) value 1.40 ± 0.57 mm. Its performance is also compared with three prostate segmentation methods, indicating the best segmentation accuracy by the proposed method among all methods under comparison.

1 Introduction

The prostate is a walnut-sized gland that forms part of the male reproductive system. Prostate cancer is one of the most common cancers in males and is a leading cause of male cancer death in US [1]. The external beam radiation treatment can provide a non-invasive and effective therapy for prostate cancer, which mainly consists of two subsequent stages. First, in the planning stage, a planning CT image is acquired from a patient for a clinical expert to delineate prostate and nearby critical structures manually. In this way, a patient-specific treatment plan, including the intensities and shapes of the radiation beams to be delivered to the prostate, can be designed based on the planning image and its segmented prostate. Second, in the treatment stage, dose is delivered to the prostate in daily fractions during a period of eight to nine weeks, guided by the location of prostate identified from each treatment image. Specifically, by localizing prostate in the treatment image, the treatment plan designed in the planning image space can be transformed and applied to the patient at the treatment day, with the goal of maximizing the dose delivered to the prostate and minimizing the dose delivered to the surrounding areas such as bladder and rectum. Therefore, the success of image-guided radiotherapy (IGRT) highly depends on the accurate localization of prostate in the treatment CT images. Moreover, accurate prostate segmentation also has important applications such as dose tracking and plan correction.

However, accurate segmentation of prostate in CT images is challenging mainly due to the four factors. First, the image contrast between prostate and its surrounding tissues is generally very low in CT images. This can be visually inspected from Fig. 1 (a), where the red contours denote the manually-delineated prostate boundaries. Second, the shape and size of prostate among different patients of different ages vary significantly [2]. Third, the prostate motion/deformation (mainly caused by both water in the bladder and fillings/air in the rectum) is complex among different treatment days [3]. This factor can be also illustrated by Fig. 1 (b) which shows the prostate regions of two treatment images (in white and gray, respectively) of a same patient after rigid alignment based on their pelvic bone structures. It can be observed that, even after rigid alignment based on bone structures, the relative prostate motion/deformation is still large. Fourth, the uncertain presence of rectum gas makes image appearance change dramatically across different treatment days. This factor can be again visualized by Fig. 1 (a), which shows large variations of the shape and appearance of the rectum gas (black) in different treatment images.

Fig. 1.

(a) Slices from the three treatment images showing the low contrast between prostate and surrounding organs (i.e., bladder, rectum, and muscles), as well as large variation of the shape and appearance of the rectum gas (black). Here, the three red curves denote the prostate boundaries manually delineated by a clinical expert. (b) Overlapping display of the two prostate shapes (white and grey) from two treatment images of the same patient, showing large motion/deformation even after bone-based rigid alignment.

So far, several categories of methods have been proposed for prostate segmentation from CT images. The first and the most popular category of prostate segmentation methods is the deformable-model based segmentation methods [4–8]. For instance, Broadhurst et. al. [4] proposed a deformable model based on histogram-statistics appearance model at multiple scales for prostate segmentation. Freedman et al. [5] proposed a segmentation algorithm based on matching the probabilistic distributions of photometric variables that incorporates the learned shape and appearance models for the objects of interest. Also, Feng et al. [7] presented a deformable model based on the statistical shape and local image appearance information which leverages both population and patient-specific statistics to segment prostate in the CT images. Chen et al. [8] presented a prostate segmentation algorithm in which both the anatomical constraints from bones and a learned appearance model are incorporated under the Bayesian framework. Rousson et al. [6] proposed a Bayesian framework to impose a shape constraint on the prostate and couple its extraction with that of the bladder, where the prostate and bladder are represented by two level set surfaces with non-overlapping constraint. Overall, these deformable-model based segmentation methods, which consider the statistical information extracted from both organ shape and image appearance, have been proven effective in segmenting prostate from CT images. However, this category of methods normally requires a good initialization and a specific optimization strategy in order to obtain accurate and robust segmentation results.

The second category of prostate segmentation methods is the registration-based segmentation [9, 10] methods. For instance, Davis et al. [9] develloped a registration-based prostate localization approach for CT images by combining a fluid-based image regiistration algorithm and a bowel gas deflation algorithm. Wang et al. [10] implemented a three-dimensional (3D) deformable registration algorithm to aid dose tracking for the targeted prostate radiotherapy. However, the uncertain presence of water in the bladder and filllings/air in the rectum makes the accurate registration of two treatmment images difficult.

Besides, other methods have also been developed based on mathematical morpphology [11], combination of deformable model and registration [12], and different optimization strategies [13, 14]. Note that, besides segmenting prostate from CT images, there are also other methods segmenting prostate from other imaging modalities such as MR [15, 16] and US[17]. In this paper, we focus on developing a learning-based method for segmenting prostate from 3D CT images.

The context and high-level information plays an important role in object recognition and scene understanding, especially for medical image analysis [18, 19]. Several models in statistical classification field, such as Markov Random Field (MRF) [20] and Conditional Random Field (CRF) [21], have been proposed to capture the context information using a neighborhood (clique) structure with a limited and fixed topology. However, this topology limits the modeling capability. Therefore, other alternative methods were proposed to extract context information more effectively. For instance, Jin et al. [22] proposed a framework called composition machine to construct the hierarchical probabilistic image models in order to accommodate arbitrary contextual relationships. Loog et al. [23] proposed an iterative contextual pixel classification method to segment the posterior ribs in chest radiographs, in which the classification of each pixel was iteratively updated based on the context information. Hupse et al. [24] developed a set of context features that represent suspiciousness of tissue for detection of malignant masses in mammograms and achieved performance improvement when multiple views were used for computing context features. Recently, similar to [23], a supervised machine learning method with context information has also been proposed in [25]. In this work, the classification probability maps created by the learned classifiers were used as context information in a recursive manner to improve classification result. However, due to the use of a single global classifier, the boundary of object in their classification result, where the image appearance and context have large variation, is still blur.

To deal with difficulty in segmentation of prostate in CT images, we propose a patient-specific classification method to segment prostate from 3D CT images by incorporating both appearance features (section 2.3.1) and context features (section 2.3.2) into an online-learning framework. Our method avoids negative side-effect of inaccurate initialization and improves the boundary segmentation accuracy. In particular, first, we learn a different classifier for each local region of the prostate, rather than a single global classifier for the whole prostate. These local classifiers are named as location-adaptive classifiers in this paper. Second, we employ patient-specific information and the online learning strategy. In other words, we improve the segmentation accuracy by learning information from the previous-segmented treatment images of a patient, thus achieving accurate segmentation for the new treatment images of the same patient in the following days. Third, we extend the auto-context method of [25] by introducing context information from both original image and classification probability. Different from the traditional learning-based methods, each location-adaptive classifier itself consists of a series of sub-classifiers. This means a series of learning processes are performed iteratively to produce a set of sequential sub-classifiers to compose a location-adaptive classifier. By using this learning process, the classifiers can effectively gather information locally and propagate it to other places. Last, context features are collected from a large number of locations far-away from a pixel of interest. By using the context features, the partial shape information can also be learned by our classifiers.

Specifically, in our method, two sets of location-adaptive classifiers along the two coordinate directions are trained within a region of interest (ROI) around the prostate which is predefined, as shown in Fig. 2. The ROI is separated into parallel rectangular blocks in two coordinate directions. Each rectangular block has a corresponding location-adaptive classifier. Each location-adaptive classifier is trained only on 2D slices within its corresponding rectangular block. The classification probability maps produced by the two sets of location-adaptive classifiers will be stacked along each direction firstly and then fused into a single 3D classification probability map for the daily image under segmentation. Next, this single 3D classification probability map is further refined by a 3D level-set algorithm [26] for final segmentation. This two-coordinate-directions based learning strategy encodes different local regions of the whole ROI in the two directions. It enables our location-adaptive classifiers to learn the complementary information about the prostate in 3D space in a more effective way. Note that the two coordinate directions (i.e., axial and coronal directions) are selected based on the anatomical symmetry of pelvis and the knowledge of observed prostate motion (i.e., along superior/inferior and anterior/posterior directions [3]). Along these two directions, the symmetric shape information (e.g., bones and muscles) and motion information can be both learned by our classifiers based on the features designed in our method.

Fig. 2.

Illustration of location-addaptive classifiers placed along the two coordinate directions (Y and Z) around the prostate region in the extracted ROI. A 3D ROI around a prostate (red object) is shown in (a), where each magenta/green rectangle represents the place where we will place a location-adaptive classifier. (b) and (c) are the sample slices displayed in the views of axial (Z coordinate) and coronal (Y coordinate) directions, respectively. The red contours represent prostate boundaries. The magenta/green straight line represents where a location-adaptive classifier will be placed.

Notably, our proposed method for prostate segmentation is designed under the scenario of clinical daily radiotherapy. So, the whole method can be further optimized by an offline verification process, which allows the physicist to check the accuracy of automatic segmentation by our method. In this way, only those accurate segmented previous images will be used to retrain our location-adaptive classifiers to gradually improve the accuracy of automatic segmentation for the following daily images. Furthermore, our method requires only the first several treatment days to be segmented manually to initialize the whole training procedure in our method. Compared to the requirement of segmenting all treatment images (i.e., about 45 images) in the course of a patient's radiotherapy in the current clinical application, our method can significantly reduce manual efforts and thus make the whole process more efficient.

Our method has been extensively evaluated on 11 patients, each with more than 9 3D treatment CT images of prostate. It is further compared with three state-of-the-art prostate segmentation algorithms for CT images. Experimental results show that our method can consistently achieve the highest segmentation accuracy among all the methods under comparison. We note that a preliminary version of our method was previously reported in a conference paper [27]. The present paper provides a complete description of the method and further presents more experimental results to evaluate its performance.

The rest of this paper is organized as follows. Section 2 introduces the proposed algorithm. Section 3 provides and analyzes the experimental results. Section 4 provides some discussions on the proposed method and concludes this paper.

2 Methodology

Our method mainly consists of an online training stage and a segmentation stage in the course of radiotherapy for a patient. The flowchart of the proposed method is shown in Fig. 3. Hereafter, we denote and index all images starting from number 1 for the sake of simplicity (e.g., the planning image is denoted by image 1 and the first treatment image is denoted by image 2).

Fig. 3.

Flowchart of the proposed patient-specific prostate segmentation algorithm for 3D CT images. The black solid arrows indicate the training process, while the red solid arrows indicate the segmentation of a new treatment image acquired at the subsequent treatment day.

Training stage: Two sets of location-adaptive classifiers are initially trained by the patient's first m daily scans (e.g., m = 3 in this paper) with manual segmentations. The classifiers will be further retrained and updated based on the n latest-acquired scans (e.g., n = 4) of the same patient and their corresponding segmentation results produced by the proposed method during the radiotherapy. That is, with more and more subsequent treatment images acquired and automatically segmented, our classifiers can be updated with the automatic segmentation results of the n latest-acquired treatment images, along with the first m daily scans. In this way, our trained classifiers can track the up-to-date patient-specific image context and appearance information to improve the classification accuracy for the next daily treatment image. In this paper, we set m = 3, which means that we use only the first 3 images and their corresponding manual segmentations to initialize our training process in the whole course of radiotherapy for a specific patient. Then, the classifiers are updated by using the automatic segmentation results (with or without offline verification) of the subsequent daily treatment images produced by the pre-trained classifiers. Our method can work automatically for segmenting a treatment image after retraining classifier for next day, which is important for a segmentation method to be practical for clinical application.

Specifically, patient's poses in these images need to be aligned firstly to remove whole body rigid motion. Therefore, the pelvic bone structures of all training images are rigidly aligned to the pelvic bone structures in the planning image of the same patient. After alignment, the remaining movement and shape variation of the prostate in each training image are mainly caused by other facts such as water in the bladder and gas/fillings in the rectum. The goal of our classifiers is to capture these movements and shape variations in all aligned training images.

Next, a 3D region of interest (ROI) in the planning image is determined around the prostate (e.g., shown in Fig. 2) to reduce computation burden in both training and segmentation stages. The position of ROI is determined based on the manual segmentation of the planning image, and the size of ROI (130mm×130mm×180mm) is set large enough to include the whole prostate in all daily scans. With this defined ROI, we can use it to extract a same-size subvolume from all rigidly-aligned training images, which can used to form the training data set for training our location-adaptive classifiers.

To train each location-adaptive classifier, the training pixels are first sampled from the corresponding slices for which this classifier is responsible. Then, we use features of the training pixels and the respective segmentation labels to train each classifier by using a machine learning technique [25]. Each trained classifier is retrained when the training data set is updated by a new treatment image and its automatic segmentation result.

Segmentation stage: Each new treatment image is first rigidly aligned onto the planning image space based on the pelvic bone structures. Then, the ROI defined in the training stage will be used to extract the target region in the new aligned treatment image, which will be further classified/labeled by the two sets of our trained location-adaptive classifiers (along the two coordinate directions). The two prostate classification probability maps produced by the two sets of classifiers are stacked along their directions respectively to achieve two 3D classification results. Then the two 3D classification results can be finally fused into a single 3D classification result and further segmented by a 3D level-set based segmentation technique extended from [26] to produce the final binary segmentation result for the new treatment image.

The above training stage and the segmentation stage are recursively repeated in the whole course of radiotherapy of a patient with the aid of an automatic or manual result verification process. For example, if the period of radiotherapy for a patient consists of 45 days and the number of images required for training initialization is set to 3, the clinician needs to manually label the first 3 images to start our method. Then, the classifiers for segmenting the 4th daily treatment image are trained based on the segmentations of the first 3 images. After automated or manual verification, the segmentation result of the 4^th daily image can be added into our training data set, and our classifiers for the 5^th daily image are retrained based on the updated training data set. This procedure is repeated in all following treatment days. Since the retraining process can be done overnight, the updated classifiers can be used for prostate segmentation for the next treatment day. Due to this repeated retaining procedure, our method is thus called as online-learning method.

2.1 Auto-context classification formulation

In this section, we present the formulation of auto-context classification and notations used in this paper. More details can be referred to [25]. Here, a 2D/3D image is represented as a function X = (x₁,…,x_N) mapping from spatial domain $\Omega \subset {\mathbb{R}}^2∕{\mathbb{R}}^3$ to an intensity value in $\mathbb{R}$, where N is the number of pixels in the image, For each image X, its corresponding ground-truth segmentation/label image y = (y₁,…,y_N) is also available (i.e., manual segmentation result in the first several scans), with label value y_j of each pixel x_j(j =1,…,N) belonging to {0,1}.

We aim to train a classifier to find an optimal classification result through Maximum a Posteriori (MAP) given a new image X, i.e., argmax_Y(p(Y|X)). Here, the marginal distribution is employed to calculate p(Y|X). To better approximate the marginal distribution is employed to calculate p(Y|X). To better approximate the marginal distribution an auto-context model is proposed, in which context features are extracted from the classification probability map F^t produced by a traditional classifier such as SVM or AdaBoost at each iteration t and further integrated into the training set recursively to provide context information. The iterative algorithm updates p^(t)(y_j|B_j(X),F^t−1) to approach p(y_j|X) asymptotically without doing explicit integration, where B_j(X) denotes the pixels in an image patch centered at the j-th pixel. This iterative process can be expressed by Equation 1:

$$p^{\left(t\right)}(y_i\mid B_j\left(X\right),F^{t-1})\to p(y_j\mid X)=\int p(y_j,Y_{-j}\mid X)d{Y}_{-j},$$ (1)

where Y_−j denotes the group of segmentation labels of all the pixels except y_j

2.2 Location-adaptive classifiers

In this section, we describe how to train each location-adaptive classifier. First, the corresponding slices used to train a specific location-adaptive classifier are selected from all ROIs in the training data set and further denoted as X_i(i = 1…M), where M ≥ 1 is the number of the slices. For example, suppose that there are in total 4 training ROIs available and each location-adaptive classifier is responsible for 5 slices, then the number of slices, M, used to train each local-adaptive classifier is M = 4 * 5 = 20.

Details of the training instance sampling procedure are summarized as follows. First, different types of organs and tissues such as bone and air in each prostate training image can be identified based on an intensity thresholding process (i.e., more details of this step can be referred to Section 3). Then, we can randomly draw samples from regions belonging to different types of organs and tissues with the following constraints enforced: 1) half of the training instances are constrained to be positive instances belonging to the prostate region; 2) 50% of the negative instances are constrained to lie within 20 mm around the prostate, 30% of the negative instances are constrained to lie in bones and air, and 20% of the negative instances are constrained to lie in other regions. Figure 4 illustrates this sampling procedure.

Fig. 4.

An illustration of the training instance sampling procedure. The red curve is the boundary of prostate. Pixels of different color represent the training instances drawn from different regions belonging to different types of organs and tissues (i.e., prostate (blue), region within 20 mm around the prostate (magenta), bone and air (green), and other regions (yellow)).

For a specific location-adaptive classifier, assume that K_i training instances are obtained for each training slice X_i. All training instances (i.e., sampled training pixels) are denoted as {x_ij, i =1…M, j = 1…K_i}. Then, the training set can be represented as S = {(y_ij, υ_ij), i= 1…M, j =1…K_i} for each for location-adaptive classifier, where υij is the corresponding feature vector of x_ij. the image appearance features (refers to section 2.3.1) of each training pixel x_ij are A_ij(i =1…M, j =1…K_i}. Also, the corresponding context features (described in section 2.3.2) of x_ij at iteration t are denoted as $C_{ij}^t(i=1\cdots M,j=1\cdots K_i,t=0\cdots T)$, which are computed from the slice classification map $F_i^t$ and X_i. Accordingly, the feature vector υ for pixel x_ij at iteration t is now represented by both appearance features and context features, i.e., ${\nu }_{ij}=(A_{ij},C_{ij}^t)$. Then the training set becomes $S_t=\left\{(y_{ij},A_{ij},C_{ij}^t)\right\}$ at iteration t. Finally, we can iteratively train our location-adaptive classifier using the AdaBoost algorithm [28]. Fig. 5 outlines our training process for a location-adaptive classifier. At the end of the training procedure, we get a sequence of classifiers L_t(t = 0…T), namely the sub-classifiers, for each location-adaptive classifier, where T denotes the last iteration and totally T+1 iterations are used. Note that, for the first sub-classifier, we use the default uniform classification probability map (i.e., $F_i^0$ is a uniform classification probability map), since the new classification probability map is not computed yet.

Fig. 5.

Recursive training process of a location-adaptive classifier.

2.3 Feature extraction

Two types of features are employed in our learning-based segmentation algorithm. The first type of features is image appearance features, which are calculated from intensity images. The second type of features is context features, which are computed both from intensity images and classification probability maps. These two types of features construct our feature pool.

2.3.1 Appearance features

The appearance feature used in this paper consists of both Haar features [29] and histogram of oriented gradient (HOG) features [30]. The two types of appearance features are employed to capture the local information around each voxel in a CT image. Haar features are widely used in object recognition, which can be computed at different scales of interest and with high speed by using integral images. A Haar-like feature considers adjacent rectangular regions at a specific location in a detection window, sums up the pixel intensities in these regions, and calculates the difference between them. In this paper, we extract Haar features from different scales of interest to facilitate our prostate classification. Fig. 6(a) shows some 2D Haar-like descriptors used in our method.

Fig. 6.

(a) Examples of some 2D Haar-like descriptors. (b) Context sample pattern.

The other appearance features adopted in this paper are the histogram of oriented gradient (HOG) features. The HOG features can be extracted by dividing the image into small connected regions, called as cells, and then compiling a histogram of gradient directions for the pixels within each cell. To improve the segmentation accuracy, the local histograms are contrast-normalized for correcting intensity-inhomogeneity in the prostate CT images by calculating the mean of the intensity (i.e., average intensity) across a larger region of the image, called as a block, and then this value is used to normalize all cells within the block. Our HOG features consist of 9 cells by 9 histogram bins. The size of the block is set to be the size of image patch used for computing appearance features. In this paper, the size of the image patch is set empirically to be 21×21. Finally, the coordinates of each pixel are also adopted as the appearance features in our method to learn the movement information of prostate within the patient's pelvis.

2.3.2 Context features

Context features are used to capture global scene information around each pixel in a CT slice. A context sample pattern is employed to extract information at different pixel around each pixel as its context features. Different from the local appearance features, the context features of a pixel are extracted from a large region centered at the pixel rather than a small neighborhood region. Specifically, for a pixel, eight rays in 45-degree intervals are extended out from the current pixel and then we sparsely sample the context locations on these rays at different radii, as illustrated in Fig. 6(b). In our paper, a radius sequence R = {R_u, u = 1…15} = {4, 5, 6, 8, 10, 12, 14, 16, 20, 25, 30, 40, 50, 60, 80} is used for choosing context locations on each ray. Furthermore, the location on each of the current pixel and pixels in its 3×3 neighborhood are also included in the whole context sample pattern. For prostate segmentation, the relative coordinates (α_uw, β_uw) i.e., (0,0) is the center pixel, of context (except 3×3 neighborhood of the center pixel) is constructed with following formulation.

$$\{\begin{array}{cc}{\alpha }_{u,w}=\hfill & \lfloor R_u\cos \left(\frac{\pi }{4}\ast w\right)+\frac{1}{2}\rfloor \hfill \\ {\beta }_{u,w}=\hfill & \lfloor R_u\sin \left(\frac{\pi }{4}\ast w\right)+\frac{1}{2}\rfloor \hfill \end{array}\phantom{\}},u=1\cdots 15,w=0\cdots 7$$

where ⌊·⌋ gives the floor of real number.

For each of the above context locations in the context sample pattern, its classification probability and the mean probability within a 3×3 window are adopted as the context feature like Tu et al.'s work [25]. However, different from their work, we also extract context information from intensity image at these selected context locations rather than from the classification probability only. In other words, we consider that there is important context information included in the intensity variation of image at these selected context locations. Therefore the absolute intensity variations between the context locations and the current pixel of interest are also included as context features in our algorithm. The construction and update of the context features is shown in Fig.7. Thus, the location-adaptive classifiers can learn the location information from the air or bone regions (which often have large intensity variation from the current pixel of interest) through our designed context features. These context features implicitly represent the shape and configuration information of the prostate. Due to the nature of iteratively learning context information, our location-adaptive classifiers can achieve better performance. This can be seen from Fig.7, where the prostate region in the sequential classification maps becomes more and more prominent and the boundary between the prostate and background becomes sharper and sharper. Most importantly, our proposed algorithm is general which can incorporate other types of features for possible improvement of prostate segmentation.

Fig. 7.

Illustration of construction and update of the features used in our method. The red rectangle in top right image represents the patch used for calculating appearance features. The small rectangles located along rays (i.e., context sample pattern) are positions where context features are extracted. In the column of classification probability map, classification map is iteratively updated from a uniform probability map (top) to a better and better classification result (bottom). Note that the classification probability map in the iteration is a default uniform classification probability map

3 Experimental Results

In this section, several experiments were conducted to evaluate the proposed method. Since our classifiers were designed to learn the patient-specific information, all training and segmentation processes were done on the images from the same patient. In the following experiments, each treatment image was firstly rigidly aligned to the planning image based on pelvic bone structures. The alignment process was performed by using the FLIRT toolkit in [31]. The bone and air regions in each image were detected according to their values in Hounsfield units (HU) by using the predefined thresholds: +400 and −150 (i.e., regions with intensity of more than +400 were determined as bones, while regions with intensity of less than −150 were determined as air). Given the segmentation ground-truth of the planning image, we defined a ROI with size of 130mm×130mm×180mm around prostate, which was used in the training and segmentation stages to trim out ROIs from the aligned treatment images. Note that the segmentation ground-truth of each image is provided by one clinical expert in this study, and different clinical experts could provide different segmentation ground-truths. However, the corresponding location-adaptive classifiers will be trained consistently with the segmentation ground-truths provided by the specific clinical expert.

According to the statistics of position variation of prostates in different daily images based on our data set, we found that the prostate mainly moves in Y (coronal) and Z (axial) directions [3]. So the location-adaptive classifiers can learn the most important movements of prostates when they are trained on slices along Y and Z directions. Adding another set of location-adaptive classifiers, i.e., trained along X (sagittal) direction, may slightly increase the performance of our method. However, for saving computation time, we do not include classifiers along X direction. The numbers of the location-adaptive classifiers along Y and Z directions were determined based on the resolutions of patient image along the two directions and also the size of ROI determined in the planning image. For our data set, 12 and 20 classifiers were placed along Y and Z directions, respectively. Each location-adaptive classifier includes a series of 3 sub-classifiers, each of which is an AdaBoost-based classifier, consisting of 100 3-level decision trees as weak classifiers.

The number of the features used in the experiments was 568 in total. The image appearance features consisted of 118 Haar features and 81 HOG features which were all calculated from the 21×21 image patch of each pixel. The number of context features was 369. The parameters of 3D level-set based segmentation technique used in our paper were set as follows: μ = 0.02 * 255², ν = 0,λ₁ = λ₂ = 1. The first two parameters (i.e., μ, ν) denote the weightings given to surface area and inside volume of the level set surface for regularization. The parameters λ₁ and λ₂ are the weightings given to the two fitting terms in the energy function [26].

3.1 Data and evaluation measures

Our data consists of 11 patients, each with more than 9 daily prostate CT scans, with total image count of 161. Each image has XY size 512×512 and 34 to 97 slices with voxel resolution of 1mm×1mm×3mm. For each patient, the first 3 images and their manual segmentations were used to initialize the patient-specific learning process. To validate the proposed algorithm quantitatively, the expert manual segmentation results for each image were used as ground-truth. Three quantitative measures were used to evaluate the performance of our method, as listed below:

1. The Dice similarity coefficient (Dice) [32]. The Dice is defined as (2 * V_a⋂g)/(V_a + V_g), where V_a⋂g denotes the number of voxels classified as the prostate by both the proposed algorithm and by the ground-truth, respectively.

2. The average surface distance (ASD) between the automated and manual segmentation results. ASD is calculated as the average distance between the surfaces of the ground-truth and the automatically-segmented prostates along 300 rays evenly directed to the spherical surface from the ground-truth centroid.

3. The centroid distance (CD) is calculated to measure the segmentation accuracy with respect to the centroid measurement, since the centroid is of particular interest for a radiation oncologist when designing a treatment plan. We compare the distance between the centroid of the ground-truth with the centroid of the automatic segmentation results. The differences in the lateral (X), anterior-posterior (Y), and superior-inferior (Z) directions are measured separately.

3.2 Location-adaptive classifiers versus global classifier

To justify that our location-adaptive classification scheme is better than using a single global classifier, we designed two experiments. In the first experiment, we compared our method with a 3D population-based global classifier trained on the population images (i.e., different patients' images were used for training). For training the 3D global classifier, similar features in 3D image space (such as Haar, HOG, and context features) were employed. Then 80 images out of our data set were randomly selected to train the 3D global classifier. Some typical results produced by our method and the 3D global classifier are shown in Fig. 8 for visual comparison. As indicated by Fig. 8, the results by our method are significantly better than those by the 3D global classifier. This indicates that the prostate cannot be segmented accurately if using only the population-based global classifier. It also indicates that the information learned from the population images cannot improve the classification performance due to large variations of image appearance across different patients.

Fig. 8.

Classification results produced by our proposed method and a single 3D global classifier trained on the population images. From left to right column, intensity slices, classification results of our classifiers, classification results of the 3D global classifier, and intensity slices overlapped by ground truth (red curves), segmentation results of our method (blue curves), and segmentation results of the 3D global classifier (green curves) are displayed.

In the second experiment, we compared our method with a 3D patient-specific global classifier, trained on the first 5 images from the same patient, which has totally 16 images. The similar features as the 3D population-based global classifier were employed again. Then 11 images from the same patient (except the first 5 images used for training) were used as testing images. Some typical results are shown in Fig. 8. As indicated by Fig.9, our method also performed much better than the 3D patient-specific global classifier, especially around the boundary of prostate. This approves again the difficulty of training only a global classifier for accurate prostate segmentation. On the other hand, we can observe that the 3D patient-specific global classifier is better than the 3D population global classifier, which means that the performance of classifiers can be improved by using the information learned from images of the same patient. In Fig. 10, Dice values of 11 segmentation results by our method and by the 3D patient-specific global classifier are displayed. Dice values of our method are significantly (p<0.001) greater than those of the 3D patient-specific global classifier.

Fig. 9.

Classification results produced by our proposed method and a 3D patient-specific global classifier trained on images from the same patient. From left to right column, intensity slices, classification results of our proposed method, classification results of the 3D patient-specific global classifier, and intensity slices overlapped by ground truths (red curves), segmentation results of our method (blue curves), and segmentation results of the 3D patient-specific global classifier (green curves) are displayed.

Fig. 10.

Dice values of segmentation results of 11 images from the same patient by our location-adaptive classifiers and the 3D patient-specific global classifier.

3.3 Role of context features

By using image context features, our classification results of image slices have more uniform high values in the region of prostate and low values in the background. In training the location-adaptive classifiers, the context features play an important role in training the 3 sequential sub-classifiers used in this paper. In our method, the context features of each pixel consist of two parts at each context location (see Section 2.3.2). Fig. 11 shows an example about the numbers of the two parts of context features used by the 3 sequential sub-classifiers in a learned location-adaptive classifier. From Fig.11, it can be observed that different sets of context features are selected by each sub-classifier. It can also be observed that features from the two parts of context are used in sub-classifiers jointly. However, the number of context features from the second part is larger than that from the first part, indicating that the absolute intensity variations in context features are more important than other context features in our method.

Fig. 11.

An example illustrating the locations of two parts of the context features learned by the 3 sub-classifiers of a location-adaptive classifier at 3 different iterations. (a), (b), and (c) illustrate the context features learned by the 1^st, 2^nd, and 3^rd sub-classifiers. The green circle, blue diamond, and red `+' represent the center of context features, context features calculated from the classification maps and the absolute intensity difference between the current pixel of interest and the context location, respectively.

3.4 Fusion of results of location-adaptive classifiers

To illustrate the effectiveness of the fusion of the classification probability maps produced by the two sets of classifiers along the two coordinate directions, a visual comparison between results of fusion, Y direction, and Z direction is given in Fig.12. Also, a Dice ratio comparison between their results for all images of 11 patients is also shown in Fig.13. It can be observed from both Fig. 12 and Fig 13 that the segmentation results after classification fusion is better than the segmentation results directly based on the classification along each direction.

Fig. 12.

Demonstration on the superior classification results obtained after fusing the classification results produced by the two sets of classifiers placed along the two coordinate directions. (a) Original image; (b) Classification result by the classifiers placed along the z coordinate direction; (c) Classification result by the classifiers placed along the y coordinate direction; (d) Fusion of (b) and (c); (e) Ground-truth of the prostate contour (red), and the estimated prostate contour (blue) by our proposed segmentation algorithm.

Fig. 13.

(a) Dice measure of segmentation results by using only y direction classifiers (y_d), using only z direction classifiers (z_d), and using fusion of y and z directions classifiers together (Fusion). The horizontal lines in the box plots represent the lower quartile, median, and upper quartile values, respectively. The whiskers show the extent of the rest of the data. Outliers, which fall outside 1.5 times the interquartile range, are denoted with the `+' symbol.

3.5 Segmentation accuracy of the proposed algorithm against different number of training images

The number of the available training images is also a critical factor affecting the segmentation accuracy of the classifiers. Intuitively, by increasing the number of available training images, the segmentation accuracy of the classifiers can be improved as more information can be learnt from the training images. However, in real world clinical application, the number of available training images is limited. Therefore, the robustness of the segmentation algorithm against the number of available training images is also interesting to explore. In this paper, we tested the segmentation accuracy of the proposed algorithm using different number of training images. In particular, we selected 3 patients (patient ID 2, 3 and 8) with large variations of segmentation accuracy across different treatment images (i.e., these three patients have large variations of Dice measures compared to other patients (see Fig. 17(a))). Next, we selected one particular treatment image from each of the three selected patients (i.e., image 11 of patient 2, image 12 of patient 3, and image 14 of patient 8). Then, we performed the prostate segmentation on each of these 3 images by using our location-adaptive classifiers trained by the latest 2 to 6 treatment images of the same patient. The corresponding Dice measures are shown in Fig.14. It can be observed that the Dice values become steady when the number of training images is 4 or larger. Therefore, the number of the training images was set to 3~7 for the tradeoff between training time and segmentation accuracy in all our experiments, as well as considering the increasing available treatment images for training and clinical feasibility.

Fig. 17.

(a) Dice measure of segmentation results, and (b) average surface distance (ASD) of segmentation results.

Fig. 14.

Segmentation accuracy of our proposed algorithm with respect to different number of available training images. Here, “PA2_IM11” means the image 11 of patient 2.

The improvement of the segmentation accuracy by increasing the number of training images is visually demonstrated in Fig.15. From Fig. 14, it can be observed that segmentation result is improved as the number of training images is increased, i.e., the prostate boundaries of the automatic segmentation results gradually approach to the prostate boundary of the ground-truth.

Fig. 15.

An example illustrating the improvement of the segmentation accuracy by increasing the number of training images (image 14 of patient 8). (a) is original image slice; (b), (c), and (d) are segmentation results obtained by the classifiers trained with 2, 4, and 6 training images, respectively. The red curves are the ground-truth prostate contours. The blue curves are the prostate contours of the automatic segmentation results.

3.6 Evaluation of the online learning mechanism

The online learning mechanism can avoid the trained classifiers to be possibly overfitted on some training data by introducing more new training data. As a result, the classifiers trained by online learning mechanism can produce good results robustly and consistently as demonstrated above. In this experiment, we used the first 3 images with manual segmentations and the 4 latest treatment images with automatic segmentation results (produced by the proposed algorithm) to train our classifiers. When the number of available training images is less than 7, all the available images were used for training in order to avoid the lack of sample problem. The number of the images with manual ground-truth segmentation (i.e., 3 in this experiment) used for initializing the training process is considerably less than the total amount (about 45) of the treatment images in the whole radiotherapy period of one patient. With the online learning mechanism, our location-adaptive classifiers can learn the most up-to-date patient-specific information and also reserve the information from the manual segmentation results from the first 3 images. Fig. 16 shows an example of the segmentation results on a patient, using our location-adaptive classifiers trained along the two coordinate directions. We applied our method to the serial images of 11 patients respectively. The prostate segmentation for each patient started from the 4^th image. The online learning based segmentation results on the images of the 11 patients are summarized by the whisker plot in Fig. 17(a) for Dice measures and Fig. 17(b) for ASD measures, respectively. Also, the centroid differences in lateral (x), anterior-posterior (y), and superior-inferior (z) directions are shown in Fig. 18. Our method achieved the mean Dice value 0.908 and the ASD (mean ± SD) of 1.40 ± 0.57 mm.

Fig. 16.

(a) Segmentation results on evenly-spaced slices of image 11 of patient 5 (Dice 0.906). The red contours show the manually delineated boundaries of prostate by a radiation oncologist, while the blue contours show the prostate boundaries determined by the proposed algorithm. (b) 3D comparison of two segmentation results.

Fig. 18.

(a), (b) and (c) are the centroid differences in lateral (x), anterior-posterior (y), and superior-inferior (z) directions, respectively. All symbols in the figure are the same as those in Fig. 16.

Furthermore, we demonstrated the advantage of our online retraining method by running a comparison experiment, in which the classifiers were trained by only the first 3 images, without further including the 4 latest treatment images (with automatic segmentations). In particular, four patients with 16–18 treatment images were selected. Fig. 19 shows that the Dice ratio of each image obtained by the classifier trained with only the first 3 images and our proposed online-learning method, respectively. It can be observed that our proposed online learning method can achieve much higher Dice ratios. For a typical example, i.e., the 13^th image of Patient 8, the low Dice ratio was produced by the classifiers trained with only the first 3 images, indicating that this image is significantly different from the first 3 images of the same patient. However, it can be satisfactorily segmented by our method which incorporates the latest information from the latest segmented treatment images.

Fig. 19.

Demonstration of our online learning mechanism. “PA2-manual” means that the images of patient 2 are segmented by the classifiers trained with only the first 3 images (with manual segmentations). “PA2-online” means that the images of patient 2 are segmented by our proposed online-learning method.

3.7 Comparison with other methods

Three state-of-the-art prostate segmentation algorithms [7–9] were compared with our algorithm in this section. The best segmentation performance of the methods reported in [7–9] were used to compare with our algorithm. These three existing methods are deformable-model-based or registration-based, while our algorithm is classification-based. The method in [7] employs the patient-specific and online learning strategy as our method. The method in [8] exploits population-based deformable model with anatomical constraints from pelvic bones. The method in [9] uses patient-specific registration. The comparisons of the above 3 methods with our method are given in Table 1. Specifically, values of the five segmentation performance measures are provided in Table 1, in which entries marked with N/A mean that the corresponding measures were not reported in the respective papers. Our mean Dice measure and mean ASD on all patients are 0.908 and 1.40 mm, respectively, which are better than the mean Dice measure of 0.893 and the mean ASD of 2.08 mm reported in [7]. Paired t-test on the Dice measures shows that our method is significantly (p<0.002) better than the method in [7]. Our mean centroid differences are 0.18 mm, −0.02mm, and 0.57 mm along x, y, and z directions, which are comparable to the best result of −0.26 mm, 0.35 mm, and 0.22 mm reported in [9], while our mean Dice measure 0.908 is significantly higher than the value 0.82 in [5]. Our median probability of detection (PD) and false alarm (FA) are 0.90 and 0.10, respectively, which are much better than 0.84 and 0.13 reported in [8]. However, the mean ASD 1.1 mm reported in [8] is better than the value 1.4 mm of our method. Furthermore, our method only uses 3~7 images of one patient for training, while other methods use much more images from same patients or other patients for training through a leave-one-out strategy.

Table 1.

The comparison of different methods wth respect to five performance measures.

Method	Method dscription	Patients	Images	Mean Dice	Mean ASD (mm)	Mean CD (x/y/z) (mm)	Median PD	Median FA
Feng et al. [7]	Online, patient specific, deformable model	24	306	0.893	2.08	N/A	N/A	N/A
Davis et al. [9]	Patient specific, registration-based method	3	40	0.820	N/A	−0.26/0.35/0.22	N/A	N/A
Chen et al. [8]	Anatomical constraints, deformable model	13	185	N/A	1.10	N/A	0.84	0.13
Our method	Online, learning-based classification	11	161	0.908	1.40	0.18/−0.02/0.57	0.90	0.10

N/A: the value was not reported in the paper.

3.8 Online update of training data

One potential problem in our online learning method is that if a low quality automatic segmentation result is used to update the training data, the following retrained classifiers will be not reliable. The low quality automatic segmentations are mainly owing to a significant variation of image appearance around the prostate in daily treatment images of some patients [33]. To illustrate this case, images of the first 4 days of patient 3 are shown in Fig. 20. It can be observed that the new treatment image (the 4^th image) is significantly different from the first 3 images used for the training images, thus the previous trained classifiers cannot effectively identify prostate in the new treatment image (the 4^th image).

Fig. 20.

Illustration of large variation of position and image appearance of prostates in the first 4 images of patient 3 during the radiotherapy course. Each column represents slices of a 3D CT image scanned at a certain day. Each row represents the same slices of rigidly-aligned images of 4 different treatment days. The red curves are ground truth.

To address this problem, our method is designed in a more flexible manner which ensures only the well-segmented images are used to update the training data, through an image dissimilarity indicator (IDI). Also, our method allows users to further check the automatic segmentation results offline, which is clinically feasible. Specifically, we define an image dissimilarity indicator (IDI) to determine whether the automatic segmentation can be used to update training data, as well as tell users whether the checking of the automatic segmentation result is needed. Here, the role of IDI is to estimate the quality of segmentation and assign a confidence degree to the automatic segmentation result of the current treatment image. When IDI has a high value, the confidence degree is low and in this case the manual checking is needed since less accurate segmentation result is probably obtained. On the contrary, if the IDI value is low, the confidence degree is high and in this case the manual checking is not needed. By using this IDI measure, we can reduce the time to manually check the automatic segmentation results and meanwhile eliminate the negative impact of including low accurate automatic segmentation results to retrain the classifiers.

IDI is defined as a ratio of the mean dissimilarity between the new treatment image and all training images to the mean dissimilarity between each pair of training images, where dissimilarity is measured based on the features learned by our location-adaptive classifiers. When a new image comes in, it is first rigidly aligned onto the planning image and further trimmed out with a 3D ROI (whose size and position are defined in the training process). Then, the dissimilarity between image I_a amd image I_b is computed within the ROI by the following steps:

• 1)Select some evenly-spaced slices at the same positions within the ROIs of both I_a and I_b, and further select evenly-spaced in-plane pixels from those selected slices.
• 2)For each selected slice, its corresponding location-adaptive classifier is known. Next, features and weights used by the first sub-classifier of the corresponding location-adaptive classifier are obtained. Then, values of the above features, including image appearance features and image context features, are calculated for each sampled pixel in the current slice respectively. (Note that, for the first sub-classifier, we can use only the default uniform classification probability map, since the new classification probability map is not computed yet.)
• 3)Finally, the square root of weighted square sum of feature values at all selected pixels of selected slices is calculated as the dissimilarity between I_a and I_b.

Assume that the number of training images is P The dissimilarities between the new image and each training images $d_p^{\mathit{new}}(p=1,\cdots ,P)$, as well as each pair of training images, $d_q^{\mathit{train}}(q=1,\cdots ,Q,Q=C_P^2=P\times (P-1)∕2)$, is calculated through Step 1) to Step 3), respectively. Finally, we can get the IDI through the following formulation:

$$IDI=\frac{\mathrm{mean}(d_1^{\text{new}},\cdots ,d_P^{\text{new}})}{\mathrm{mean}(d_1^{\text{train}},\cdots ,d_Q^{\text{train}})},$$

where mean denotes the average operator.

In Fig. 21, the IDI value with respect to the Dice values of automatic segmentation results of the images for 11 patients are shown. From Fig. 21, it can be observed that all IDIs of low Dice value images (e.g., Dice value less than 0.80) have larger value (e.g., larger than 1.5). It can also be observed that some images with high Dice value also have large IDI value (e.g., larger than real value 1.5). In other words, our location-adaptive classifiers (each consisting of sequential sub-classifiers) can still produce high accurate segmentation results even there exists large dissimilarity between new image and training images in some cases. In general, the most important point is that low accurate automatic segmentation results will not be missed, which really need manual checking by using the proposed IDI measure, in order to keep high reliability of our online leaning classifiers. In our application on 11 patients, the total number of the IDI with value larger than the pre-defined threshold (e.g., 1.5) is actually small (i.e., 12.5% of the total number of the images used in this paper), indicating that little manual checking time is required in the whole radiotherapy based on the proposed IDI.

Fig. 21.

The relationship between the image dissimilarity indicators (IDIs) and the Dice values of automatic segmentation results, for all images of 11 patients.

4 Discussion

In this section, important properties and possible alternative schemes of the proposed method are discussed. Our proposed algorithm is flexible which can incorporate other types of classifiers and features for possible improvement of prostate segmentation. First, since each location-adaptive classifier consists of a sequence of sub-classifiers, the choice of the sub-classifiers also influence the overall segmentation performance of prostate. Although other learning algorithm, such as SVM and random forest, can also be used as sub-classifiers, boosting has many appealing properties [34]: (1) It is a natural feature selection and fusion process; (2) It can deal with a large number of features from a considerable amount of training data; (3) Features do not need to be normalized; (4) It is less susceptible to the over-fitting problem than most other learning algorithms; (5) It can be efficiently trained and used. Therefore, we use AdaBoost as the sub-classifier in this paper. Second, different types of feature can be used in our algorithm. One reason to use the Haar and HOG features as the image appearance features in this paper is due to their computational efficiency. Another reason is that HOG features provide complementary information to the Haar features [30]. On the other hand, the prostate region in the CT image has almost homogeneous intensity values except for some intermittent weak boundary. Therefore, texture descriptors seem not be effective in this application. As for the context feature, different context sample pattern can be designed for integrating context information more effectively, as well as the features at each context sample point.

Currently, our method was implemented with Matlab2010 on an Intel Xeon 2.26-GHz CPU, and the typical computation time for the training stage, i.e., for training totally 32 location-adaptive classifiers (with 12 and 20 classifiers along two directions, respectively), was about 2 hours. After training, the average time for segmenting a new image was about 3 minutes. However, our algorithm can be optimized and implemented using C/C++ in the future, for significant improvement of segmentation speed. Thanks to the strategy of the location-adaptive classifiers in two coordinate directions, the program can be further improved by using parallel programing techniques to reduce the time of training and segmentation.

5 Conclusion

In this paper, we have presented a patient-specific, online-learning, and location-adaptive classification method, based on image context information, to segment prostate in the 3D CT scans during radiotherapy. Our method uses only the previous treatment images of the same patient as training samples to specifically learn the patient-specific information to enhance the segmentation accuracy. This is different from the conventional deformable-model and image-registration based methods that generally use the population information to guide the segmentation. Our trained classifiers can be updated online by the latest-acquired treatment images, thus better adapting to the prostate change of the current patient during the treatment. So, the location-adaptive classifiers make prediction more accurate. Experimental results on our data set show that our proposed method can consistently outperform the 3 state-of-the-art segmentation algorithms under comparison, using 5 different performance measures.

In summary, we have presented a patient-specific, online-learning, and location-adaptive classification method, based on image context information, to segment prostate in the 3D CT scans during radiotherapy. We applied the method to serial images of 11 patients. To evaluate the performance of the proposed method, we evaluated it using manually labeled prostates as the ground truth. The results show that the algorithm is able to segment prostate efficiently and accurately, achieving mean Dice value 0.908 and mean ASD value 1.40.

Future works are summarized in the following aspects. First, shape priors or more effective constraints of prostate shape can be integrated into our framework to get more accurate and applausive results, although the context features already contain some partial shape information implicitly. Second, the bladder and rectum can be segmented simultaneously with the prostate using a multi-class learning framework, to provide more information of their correlation. Third, clinical feedbacks will be acquired from the clinicians to further improve the proposed method.