A Segmentation Editing Framework Based on Shape Change Statistics

Abstract

Segmentation is a key task in medical image analysis because its accuracy significantly affects successive steps. Automatic segmentation methods often produce inadequate segmentations, which require the user to manually edit the produced segmentation slice by slice. Because editing is time-consuming, an editing tool that enables the user to produce accurate segmentations by only drawing a sparse set of contours would be needed. This paper describes such a framework as applied to a single object. Constrained by the additional information enabled by the manually segmented contours, the proposed framework utilizes object shape statistics to transform the failed automatic segmentation to a more accurate version. Instead of modeling the object shape, the proposed framework utilizes shape change statistics that were generated to capture the object deformation from the failed automatic segmentation to its corresponding correct segmentation. An optimization procedure was used to minimize an energy function that consists of two terms, an external contour match term and an internal shape change regularity term. The high accuracy of the proposed segmentation editing approach was confirmed by testing it on a simulated data set based on 10 in-vivo infant magnetic resonance brain data sets using four similarity metrics. Segmentation results indicated that our method can provide efficient and adequately accurate segmentations (Dice segmentation accuracy increase of 10%), with very sparse contours (only 10%), which is promising in greatly decreasing the work expected from the user.

1. INTRODUCTION

Medical image segmentation refers to the procedure of partitioning a medical image into separate regions. In particular, this paper focuses on the segmentation of anatomical objects from medical images. Segmentation is a crucial step in neuroimaging research that uses shape-based features to test various hypotheses about the pathology of different brain disorders. It is also a crucial step in applications like image guided surgery, radiation therapy or diagnostic imaging. A broad array of various segmentation techniques are currently available that vary in the level of user interaction. Manual segmentation performed by experts remains the gold standard; however, it is time-consuming and suffers from inter-observer variability. Although many automatic segmentation algorithms have been developed, they still fail in cases where the target appearance is different from the norm, which makes automatic segmentation unreliable to produce adequate segmentation in every situation. In such cases, the use of interactive segmentation techniques becomes necessary.¹ Segmentation editing is a special case of interactive segmentation that typically starts with an initial segmentation that the user has to correct slice by slice in a time-consuming contour manipulation procedure.^{2, 3} The initial segmentation is given by a dedicated automatic or semi-automatic algorithm that can be independent of the editing tool.⁴ An effective segmentation editing tool would be able to quickly provide intuitive modifications in 3D using few contours added by the user (see Figure 1).

Figure 1.

General segmentation editing problem

Segmentation editing methods can be classified into (i) image-based and (ii) image-independent techniques. Heckel et al.⁵ proposed an image-based method that iteratively simulates the sketch-based user input on the neighboring slices using a block matching followed by the shortest path approach on gradients within the image. Egger et al. ⁶ proposed a manual refinement method based on a graph-based approach by restricting the graph-cut using additional user-defined seed points to set up fixed nodes in the graph. Grady et al.⁷ proposed an energy minimization formulation of the segmentation editing problem that was solved using graph cuts or the random walker 3D segmentation algorithms based on image seeds added by the user. Although image-based editing techniques are relatively simple and able to provide accurate results in a timely manner, they usually fail in cases where the image information is not sufficient to produce a good segmentation such as segmenting infant brains that have greatly reduced contrast and low signal-to-noise ratio in magnetic resonance (MR) images. Moreover, results obtained from image-based editing methods cannot be guaranteed to respect the object geometry. On the other hand, image-independent editing techniques do not rely on image information and are usually used when the image does not provide enough information for segmentation. For instance, Silva et al.⁸ proposed a method that allows adding and removing voxels in 3D by a spherical brush with a user-defined radius that was applied on left ventricle segmentation. Heckel et al.⁴ proposed a sketch-based segmentation editing of 3D segmentation of compact objects in the context of tumor segmentation in computed tomography (CT) based on an object reconstruction using variational interpolation. However, surface models and drawing tools are not intuitive in 3D. Moreover, surface based correction methods can result in unexpected modifications.

In this work, we aim to develop a segmentation editing tool that is capable of providing accurate segmentation with low required user interaction. For that, we propose a novel framework that utilizes sparse manually segmented contours, produced by the user, and object shape change statistics to improve the failed automatic segmentation in an efficient manner (see Figure 2). The proposed techniques were used to improve failed automatic segmentations of a single object, namely the brain caudate, extracted from infant MR brain images. In particular, this paper will rely on the rich geometry information provided by a particular form of quasi-medial skeletal model representation called the s-rep.^{9, 10} The proposed framework starts with an s-rep fit to a failed automatic segmentation together with the sparse contours added by the user. The objective is to find an s-rep that agrees with the generated contours and represent a probable change from the failed automatic segmentation.

Figure 2.

The proposed segmentation editing framework

In overview, the proposed framework utilizes shape change statistics that were generated to capture the object deformation from the automatic failed segmentation to its corresponding correct segmentation. An optimization procedure based on conjugate gradient method is used to search the shape space in order to minimize an energy function that consist of two terms, an external contour match term and an internal geometric typicality term. Details of the proposed segmentation editing framework are provided in the next section.

2. METHOD

2.1 Data Collection and Simulation

The proposed methods were tested on the Infant Brain Imaging Study (IBIS) dataset. The study participants are 24-month-old infants at high risk of developing autism spectrum disorder (ASD). 10 T1-weighted MR brain scans were acquired on 3T Siemens Tim Trio scanners using the following parameters: field of view: 224 × 256 mm, number of slices: 160, slice thickness: 1 mm, voxel resolution: 1 × 1 × 1 mm³, TR: 2400 ms, TE: 499 ms, and flip angle: 120. Each caudate nucleus was segmented both automatically using the AutoSeg multi-atlas segementation software pipeline¹¹ and manually by an MR expert (see Figure 3). This paper focuses on a single object segmentation. Based on the extent of automatic segmentation errors, we chose the right caudate (see Figure 3) for our investigation.

Figure 3.

An example of from the IBIS dataset with right caudate shown in yellow

To test our method, shape change statistics, between the automatically segmented caudates and their manually segmented ones, needed to be generated and applied to many cases. Because of the limited number of cases available, simulated test cases needed to be generated to produce our shape change statistics and test the proposed segmentation editing methods. Given each image in the given dataset, the deformation to a chosen reference subject was generated using a nonlinear registration method, as shown if Figure 4(a). Then principal component analysis (PCA) was used to extract modes of variation that in turn was used to produce the required additional random samples. The complete steps for generating the required random samples are given in Algorithm 1. As demonstrated in Figure 4(b), most of the variation can be explained by the first 5 modes of variation. The mean shape change and the selected modes of variation were then used to generate 100 new random pairs of automatic and manual segmentations, which was then used in the following shape training process.

Figure 4.

Deformation field statistics generation

Algorithm 1.

Summary of the steps used to generate the simulated cases.

Register k data sets images Ik to a selected reference image Iref, with available labeled image Lref, using a deformable model registration algorithm to produce corresponding displacement fields Dk. Perform PCA on generated Dk to produce mean deformation field D̄, eigenvalues λi, and eigenvectors ϑi. Generate D̂ using random coefficients αi drawn from a normal distribution with zero mean and unit variance. $$\widehat{D}=\overline{D}+\sum _{i=1}^n{\alpha }_i\sqrt{{\lambda }_i}{\vartheta }_i$$ (1) Apply the sampled deformation to both Iref and Lref (automatic and manual) to generate the required random image samples (Î and L̂). $$\widehat{I}=\widehat{D}\circ I_{\mathit{ref}},\widehat{L}=\widehat{D}\circ L_{\mathit{ref}},$$ (2)

2.2 Shape Change Statistical Analysis

In this paper, a skeletal extension to the medial model¹² called the s-rep was used to represent the object shape.¹⁰ It has been shown in various publications to be more powerful in capturing the interior of most non-branching anatomic objects and providing efficient shape statistical summaries when compared to boundary point distribution models (PDMs).^{10, 13–15} An s-rep fitting procedure starts with an initial template model of the object and a distance map that was computed from the binary image of the target object. The fitting process is based on thin plate spline transformation of a spherical harmonic representation of the surfaces of the template (initial caudate model) as well as the target objects.⁹ This fitting procedure was applied to all the simulated training data set and corresponding s-reps were generated.

Instead of modeling the shape of the object, we are here interested in modeling shape change differences from a failed automatic segmentation to its corresponding manual segmentation ground truth. In order to statistically model these shape differences in an effective manner, careful attention must be given to the process by which the objects and their differences are examined. In the area of statistical shape analysis, complex shape representations such as s-reps are best thought of as non-linear entities that live on curved manifolds. Because the space of shapes is curved, the difference between a pair of shapes in the space will be a curve along the manifold, such as a geodesic connecting the two shapes. The problem then becomes estimating a probability distribution of geodesics in this curved space. While Euclidean statistical methods such as PCA are broadly applied in the analysis of PDMs, this approach is ill-suited for direct application to s-reps since modeling manifold data using linear subspaces yields nonmeaningful results (data outside the original manifold). Several techniques have been proposed to analyze data directly on their manifold.^{16, 17} However, to avoid complications associated with estimating probability distributions on curved manifolds, a technique called composite principal nested spheres (CPNS)¹⁰ was applied to the union of all training shapes to compute a polar system that can be used to create Euclideanized versions of the input s-reps. In Euclidean space, the differences between the Euclideanized s-rep pairs are line segments that can be analyzed using standard PCA yielding eigenmodes e_i of the shape change and their associated standard deviations σ_i (see Figure 5). For a new test case, we start from an initial model R created by adding the mean shape change to the failed automatic segmentation Euclideanized s-rep fit. Then, in Euclidean space, the best deformation M of the initial model R into each test image is computed by optimizing over the space of coefficients b_i of the selected eigenmodes e_i:

$$M=R+\sum _i{b}_i{e}_i,$$ (3)

where the values of b_i were selected based on the associated standard deviations σ_i:

$$-3{\sigma }_i\le b_i\le 3{\sigma }_i.$$ (4)

Figure 5.

Proposed shape change statistical analysis

Using the polar system generated using CPNS, a Euclideanized shape can be returned to its original ambient space by undoing these projections to recover the original s-rep representation.

2.3 Generating Test Case Data

After generating a new simulated test case according to Algorithm 1, a sparse set of manually segmented contours were placed (< 10% of the number of slices) by the user to enhance the failed automatic segmentation. Moreover, we wish to assess a candidate model based on the relative distance to these added contours. To accomplish this, an initial distance map needs to be produced. First, a shape-based interpolation¹⁸ method was used to create an initial 3D distance map by computing a 2D signed distance image for the manually segmented contours. Then, to produce a dense distance map a simple linear interpolation is performed between the surfaces in these slices. Next, the generated surface is smoothed based on the flow of Laplacian of curvature on the signed distance image in a way that constrains the modification to maintain the original manually segmented contours.¹⁹ In particular, for a time step Δt, an update for the distance image Φ based on the Laplacian of curvature ∇²κ as follows:

$${\mathrm{\Phi }}^{i+1}={\mathrm{\Phi }}^i+\mathrm{\Delta }t{\nabla }^2\kappa$$ (5)

After this update, each voxel where a constraint is provided is checked to make sure it has not moved more than a small threshold τ from its initial value Φ⁰. If this happened to a certain coxel, it is set to be Φ⁰ ± τ instead, with the sign determined by the sign of the update. If the slices adjacent to a voxel where the constraint was enforced are non-constrained, the amount by which the voxel was corrected is applied to the update values in the neighboring slices with a Gaussian falloff, so that directly neighboring slices are more affected than those more distant. This process is iterated until either a certain number of iterations is reached or the surface no longer moves.

2.4 Optimization Framework

During the optimization procedure, we search for M that minimizes an error E, which is the weighted sum of two terms, an external contour match term (E_c) and an internal shape typicality term (E_s):

$$E={\mu }_c{E}_c+{\mu }_s{E}_s.$$ (6)

The contour match term is computed as the sum of squared distances, with weights Ψ between the boundary voxels B of the label image I (implied by the distance image of the initial surface) and the continuous boundary surface Ω implied by a candidate model M. This is equivalent to thinking of each surface point or boundary voxel as a weighted landmark based on the relative distance to the manually segmented contours to reflect the confidence level of the distance calculation at this voxel:

$$E_c(M,I)~\sum _{{\omega }_i\in \mathrm{\Omega }}{\psi }_i{d}^2({\omega }_i,B).$$ (7)

Under Gaussian assumptions, given a mean and the variances of the principal directions of deformation, this method provides a shape-normalized Mahalanobis distance from any particular shape to the mean shape. Using this as the geometric prior in optimization tends to keep the candidate models clustered near the mean where our statistical shape model is most likely to be valid:

$$E_S(M,R)~\sum _i{(\frac{b_i}{{\sigma }_i})}^2.$$ (8)

The proposed optimization framework uses the quadratically convergent conjugate gradient descent algorithm, which selects the successive direction vectors as a conjugate version of the successive gradients obtained as the method progresses. Thus, the directions are not specified beforehand but rather are determined sequentially at each step of the iteration. At each step, one evaluates the current negative gradient vector and adds to it a linear combination of the previous direction vectors to obtain a new conjugate direction vector along which to move.²⁰ The conjugate gradient algorithm guarantees convergence in number of steps no more than the dimensionality of the problem, with a slight complication over steepest descent. In addition, because the directions are based on the gradients, the process makes good uniform progress toward the solution at every step. However, because conjugate gradient search performs best given a relatively isotropic global minimum, some experimentation is required to fine tune the weighting factors (μ_c and μ_s) to new shape studies.

3. RESULTS

Performance assessment of the segmentation editing results was carried out by applying the proposed techniques on 10 additional simulated infant MR brain data sets that were not used in the training process. The performance of the proposed segmentation framework was evaluated using four performance metrics: (i) the Dice similarity coefficient (DSC), (ii) the Jaccard similarity coefficient (JSC), (iii) the Hausdorff distance (HD), and (iv) the mean absolute distance (MAD).^{21, 22} Detailed segmentation results for the initial automatic segmentation and the proposed segmentation editing approach are given in Table 1 and Table 2, respectively. Also, a comparison summary of the obtained results, represented by the mean±standard deviation values, are provided in Table 3. Figure 7 shows an example of 3D results of the proposed segmentation editing approach.

Table 1.

Automatic segmentation approach Results

ID/Metric	DSC	JSC	HD	MAD
TS1	0.7668	0.0986	0.6217	0.6217
TS2	0.6507	0.0789	0.0289	0.4822
TS3	0.6934	0.0945	0.0294	0.5307
TS4	0.8182	0.0880	0.0173	0.6923
TS5	0.8018	0.0580	0.0180	0.6692
TS6	0.7806	0.0547	0.0223	0.6401
TS7	0.7558	0.0694	0.0219	0.6074
TS8	0.5377	0.1103	0.0440	0.3677
TS9	0.7301	0.0719	0.0266	0.5750
TS10	0.7978	0.0696	0.0190	0.6636

Table 2.

Proposed segmentation editing approach Results

ID/Metric	DSC	JSC	HD	MAD
TS1	0.8209	0.0945	0.0191	0.6961
TS2	0.8014	0.0525	0.0141	0.6686
TS3	0.8512	0.0603	0.0120	0.7409
TS4	0.8513	0.0696	0.0148	0.7411
TS5	0.8255	0.0868	0.0185	0.7029
TS6	0.8488	0.0679	0.0144	0.7374
TS7	0.8075	0.0887	0.0212	0.6772
TS8	0.8372	0.0771	0.0146	0.7200
TS9	0.8503	0.0699	0.0157	0.7395
TS10	0.8427	0.0827	0.0167	0.7282

Table 3.

Summary of the segmentation results represented by the mean±standard deviation values

ID/Metric	DSC	JSC	HD	MAD
Automatic	0.7333 ±0.0860	0.5850±0.1003	0.0794±0.0181	0.0849±0.1888
Proposed	0.8337±0.0188	0.7152±0.0274	0.0750±0.0133	0.0161±0.0028
p-value	0.0020	0.0001	0.5445	0.2642

Figure 7.

3D visualization for a segmentation editing result

As demonstrated by the t-test in Table 1, two out of the four performance measurements (DSC and JSC) show statistically significant differences (p-value<0.05) between automatic segmentations and their corresponding edited segmentations, which confirms the usefulness of the proposed segmentation editing techniques. Adding new test samples will probably drive the MAD results to be significant as well. In the future, the HD results will be replaced with its 95-percentile modified HD in order to make it less sensitive to segmentation outliers, which is the reason that it was found to be insignificant. As shown in Figure 7, most of the segmentation errors are due to the sharply narrowing tail of the caudate. This problem is mainly because of the quasi-slabular s-reps inability to capture shape information in tubular section of the caudate. Currently, my colleague Junpyo Hong is working on a new s-rep primitive that will allow for an accurate representation of the caudate tail.

4. DISCUSSION AND FUTURE WORK

In this paper, we have described a complete pipeline for editing automatic segmentations efficiently. The proposed segmentation editing technique demonstrates that the integration of shape change statistics with few manually segmented contours is promising in enhancing current automatic segmentation results. The proposed framework still needed to be applied with different segmentation tools and a variety of real datasets to fully test its accuracy. On the other hand, the proposed technique allows users to freely and quickly improve automatic segmentations in a way that is applicable to various kinds of clinical applications, at least for the many objects that are well modeled by s-reps.

The main contribution of this paper is the development of a method using s-reps for computing probability distributions of how shape changes from one state to another. A Euclideanization technique was used to to transform shape objects into a space where classical Euclidean statistical methods can be used effectively to analyze the difference between two states of the same object without having to deal with difficulties associated with shapes living on their original curved spaces.

Currently, we are investigating the use of low-rank decomposition techniques²³ to represent the failed automatic segmentation s-rep using the database of manually segmented s-reps. The surface implied by this low rank model can be used to create the weighted distance image required in the proposed algorithm (Figure 8). On the other hand, low rank decomposition can remove outliers before performing PCA on the shape change differences in the proposed framework. As shown in Figure 9, a compact PCA scree plot can be obtained, which promises to preserve the shape properties of the targeted object. Finally, the next phase of this project aims to study the effects of editing the segmentation of one object to neighborhood object segmentations.

Figure 8.

Low rank decomposition example

Figure 9.

PCA analysis of low rank representations

Figure 6.

Shape change statistical analysis results