Keypoint Transfer for Fast Whole-Body Segmentation

Abstract

We introduce an approach for image segmentation based on sparse correspondences between keypoints in testing and training images. Keypoints represent automatically identified distinctive image locations, where each keypoint correspondence suggests a transformation between images. We use these correspondences to transfer label maps of entire organs from the training images to the test image. The keypoint transfer algorithm includes three steps: (i) keypoint matching, (ii) voting-based keypoint labeling, and (iii) keypoint-based probabilistic transfer of organ segmentations. We report segmentation results for abdominal organs in whole-body CT and MRI, as well as in contrast-enhanced CT and MRI. Our method offers a speed-up of about three orders of magnitude in comparison to common multi-atlas segmentation, while achieving an accuracy that compares favorably. Moreover, keypoint transfer does not require the registration to an atlas or a training phase. Finally, the method allows for the segmentation of scans with highly variable field-of-view.

I. Introduction

Is atlas-based segmentation without dense correspondences possible? Dense correspondences, i.e., correspondences for each location in the test image to the training images, are computed by common registration- and patch-based segmentation methods [1]–[8]. The computation of dense correspondences via deformation fields or the identification of similar patches can become computationally intense for images with a large field-of-view. As an alternative, we introduce an approach for image segmentation based on sparse correspondences by identifying distinctive locations in the image: key-points. Keypoints are automatically computed as local optima of a saliency function [9], contrary to manually selected landmarks [10]. We match keypoints between test and training images to establish correspondences for a sparse set of image locations. Based on these correspondences, the segmentation masks of entire organs are transferred and fed into a probabilistic fusion algorithm. The segmentation accuracy compares favorably to common multi-atlas techniques, while working with sparse correspondences leads to a computationally efficient algorithm, offering orders of magnitude of speed-up.

We outline the keypoint transfer segmentation algorithm in Fig. 1 and as animation in the supplementary material. Keypoints are extracted at salient image regions and described by their geometry and a descriptor based on a histogram of local image intensity gradients. Following keypoint extraction, we segment an image in three steps. First, we match keypoints in the test image to keypoints in the training images based on the geometry and the descriptor. Second, keypoint labels are voted on based on matches. In the example in Fig. 1, the keypoint receives two votes for right kidney and one for liver, resulting in a majority vote for right kidney. Third, the label mask is transferred for the entire organ for each match that is consistent with the majority label vote. The organ map from one training image is possibly transferred multiple times if more than one match is available for this training image. Keypoint transfer also integrates the certainty in the keypoint label voting and computes the intensity similarity between scans. The algorithm’s capability in approximating the organ shape can further improve with a growing number of manually labeled scans, where additional images can be included in the training set without the need for a dedicated training stage.

Fig. 1:

Overview of the keypoint transfer algorithm for whole-body segmentation. First, keypoints (white circles) are matched between training and test images (arrow). Second, the organ label of the test keypoint is voted on, based on the identified matches (r.Kidney). Third, training keypoints with r.Kidney as label transfer the surrounding organ map to the test image, creating a probabilistic segmentation. The manual segmentation is shown for comparison.

In addition to being fast, keypoint transfer is beneficial for segmenting images with varying field-of-view. In our experiments, we use manually annotated whole-body scans to segment images with a limited field-of-view. Such scans are commonly acquired in clinical practice by focusing on the region of diagnostic interest, and thereby reducing scanning time and radiation dose. The intensity-based registration between images with a limited field-of-view and full abdominal images is challenging, particularly when anatomical structures are initially not approximately aligned. Keypoint matching is robust to such variations in field-of-view and therefore offers an efficient and practical tool to deal with the growing number of clinical images. A preliminary version of this work was presented at a conference [11]. Major changes in this version are: an updated presentation of the algorithm in the method’s section including an algorithmic perspective of the keypoint segmentation, the addition of experiments for whole-body MR and contrast-enhanced MR on the gold corpus, an evaluation of the matching criteria on segmentation performance, and additional experiments on all four contrasts on the silver corpus.

A. Related Work

For the segmentation of large field-of-view scans, several methods have previously been proposed. A combination of discriminative and generative models [12] and entangled decision forests [13] have been explored for the segmentation of CT scans. Authors in [14] proposed simultaneous segmentation of multiple organs by combining global and local context. Marginal space learning was described for organ detection in [15]. A combination of multi-object recognition and iterative graph-cuts with active shape model was introduced in [16]. Random decision forests for patch-based segmentation were proposed in [17]. Contrary to previously demonstrated methods, keypoint transfer does not need an extensive training on manually annotated images.

We use the publicly available Visceral dataset [18], [19] for evaluating keypoint transfer. Methods based on multi-atlas segmentation have been applied on the Visceral data [20]–[25], which we employ as a baseline in our evaluation. Multi-atlas segmentation with atlas selection and label fusion was proposed for 12 abdominal structures on clinically acquired CT in [26]. We use a 3D extension [27] of the popular scale invariant feature transform (SIFT) [9] for the extraction and description of keypoints. Next to image registration, 3D SIFT features were also used for studying questions related to neuroimaging [28] and for efficient big data analyses of medical images with approximate nearest-neighbor search [29]. Contrary to previous applications of the 3D SIFT descriptor, we use it to propagate information across images.

II. Method

Given training images¹ $\mathcal{I}=\{I_1,\dots ,I_n\}$ and corresponding segmentations $\mathcal{S}=\{S_1,\dots ,S_n\}$, where S_i(x) ∈ {1, …, η} for η labels, our aim is to infer segmentation S for test image I. To this end, we automatically identify keypoints in the images and employ them to create sparse correspondences. This is in contrast to atlas-based segmentation, where images are aligned with deformable registration. Keypoints are extracted by locally maximizing a saliency function. Following the SIFT descriptor, we use the difference-of-Gaussians [9]

$$\left\{(x_i,{\sigma }_i)\right\}=\text{local}\underset{x,\sigma }{\arg \max }\mid f(x,\kappa \sigma )-f(x,\sigma )\mid ,$$ (1)

where x_i and σ_i are the location and scale of keypoint i, f(·, σ) is the convolution of the image I with a Gaussian kernel of variance σ², and κ is a multiplicative scale sampling rate. Keypoints are located at distinctive spherical image regions that show a local extrema in scale-space. For the descriptor of the keypoint F^D, we employ a 3D extension of the image gradient orientation histogram [27] with 8 orientation and 8 spatial bins, which is scale and rotation invariant and further robust to small deformations. Working with image gradients instead of intensity values makes the descriptor more robust to intensity variations and therefore offers advantages in comparing descriptors across subjects.

The keypoint F of a salient image region is described by the 64-dimensional histogram F^D with the keypoint location $F^x\in {\mathbb{R}}^3$ and keypoint scale $F^{\sigma }\in \mathbb{R}$, resulting in a compact 68-dimensional representation. Let F_I denote the set of keypoints identified in the test image I and let ${\mathcal{F}}_{\mathcal{I}}=\{{\mathcal{F}}_{I_1},\dots ,{\mathcal{F}}_{I_n}\}$ denote the set of keypoints identified in the training images $\mathcal{I}$. Script letters are used for denoting training data and non-script letters for denoting testing data. Each keypoint is assigned an organ label ${\mathcal{F}}_{I_i}$ according to the organ it is located in, $\mathcal{L}=S_i\left({\mathcal{F}}^x\right)$ for $\mathcal{F}\in {\mathcal{F}}_{I_i}$. Keypoints that are located in the un-segmented background, are discarded. For the keypoints in the test image, the organ label L is unknown and inferred with a voting algorithm as described later in this section. Table I summarizes the notation used in the article.

TABLE I:

Summary of notation used in the article.

I, $\mathcal{I}=\{I_1,\dots ,I_n\}$	Test and training images
S, $\mathcal{S}=\{S_1,\dots ,S_n\}$	Test and training segmentations
Sl	Segmentation probability map for label l
$\widehat{S}$	Inferred segmentation
F, $\mathcal{F}$	Keypoint in test and training image
Fx, ${\mathcal{F}}^x$	Location of test and training keypoint
Fσ, ${\mathcal{F}}^{\sigma }$	Scale of test and training keypoint
FD, ${\mathcal{F}}^D$	Descriptor of test and training keypoint
FI, ${\mathcal{F}}_{I_i}$	All keypoints from test and training image i
L, $\mathcal{L}$	Label of test and training keypoint
$\widehat{L}$	Inferred label for test keypoint
m	Match between a test and training keypoint
m(F)	Corresponding training keypoint matched to F
$\mathcal{M}\left(F\right)$	All matches for test keypoint F
${\mathcal{M}}_i$	All matches for training image Ii
${\mathcal{I}}_m$, ${\mathcal{S}}_m$	Training image and segmentation of match m

A. Keypoint Matching

In the first step, we match each keypoint in the test image with keypoints in the training images. To ensure high quality matches, a two-stage matching procedure is proposed to improve the reliability of the matches by including additional constraints. In the first stage, a match $\mathcal{M}{\left(F\right)}_i$ is computed between a test keypoint F ∈ F_I and keypoints in a training image ${\mathcal{F}}_{I_i}$. To this end, we find the nearest neighbor based on the similarity of the descriptor and scale constraints

$$\mathcal{M}{\left(F\right)}_i=\underset{\mathcal{F}\in {\mathcal{F}}_{I_i}}{\arg \min }\Vert F^D-{\mathcal{F}}^D\Vert ,\text{s.t.}{\epsilon }_{\sigma }^{-1}\le \frac{F^{\sigma }}{{\mathcal{F}}^{\sigma }}\le {\epsilon }_{\sigma },$$ (2)

where a loose threshold on the scale allows for variations up to a factor of ε_σ = 2. The distance ratio test is employed to discard keypoint matches that are not reliable [9], where we compute the ratio between descriptors of the closest and second-closest neighbor. All matches with a distance ratio of greater than 0.9 are rejected.

In the second stage, we improve the matches by additionally imposing loose spatial constraints, which requires an approximate alignment. For our datasets, accounting for translation was sufficient at this stage due to consistent patient orientation; as an alternative we could use an efficient keypoint-based pre-alignment [27]. The most likely translation t_i suggested by the matches ${\mathcal{M}}_i$ is computed with the Hough transform [30]². Mapping the training keypoints with t_i leads to an approximate alignment of the keypoints and allows for an updated set of matches with an additional spatial constraint

$$\begin{array}{cc}\hfill & \mathcal{M}{\left(F\right)}_i=\underset{\mathcal{F}\in {\mathcal{F}}_{I_i}}{\arg \min }\Vert F^D-{\mathcal{F}}^D\Vert ,\hfill \\ \hfill & \text{s.t.}{\epsilon }_{\sigma }^{-1}\le \frac{F^{\sigma }}{{\mathcal{F}}^{\sigma }}\le {\epsilon }_{\sigma },{\Vert F^x-{\mathcal{F}}^x-t_i\Vert }_2<{\epsilon }_x,\hfill \end{array}$$ (3)

where a spatial threshold ε_x is selected to keep 10% of the closest matches. As previously, matches that do not fulfill the distance ratio test are discarded. Note that the estimated global translation between training and testing image is only used for improving matches.

At this stage, each keypoint F in the test image is matched to at most one keypoint $\mathcal{M}{\left(F\right)}_i$ per training image I_i, denoted as match m. We quantify the reliability of a match m by constructing a distribution over matches p(m), see Fig. 2. We build the distribution based on the translation of a match, i.e., matches that propose a translation that is not proposed by other matches receive a low probability. Hence, the probability p(m) for a match m defines the translational consistency of the match with respect to other matches. This translational consistency only holds for a training and test image pair, so that a separate distribution is estimated for each training image. For the estimation, we use matches ${\mathcal{M}}_i$ between keypoints in the test image and those in the i-th training image and kernel density estimation. For notational ease, we write p(m), although there are actually separate distributions pⁱ(m) for each training image i; the selection of the corresponding distribution is evident from the training keypoint involved in the match. The non-parametric model accommodates multi-modal distributions, which is helpful for whole-body scans, as keypoints in the upper abdomen may suggest a different transformation than those in the lower abdomen.

Fig. 2:

Illustration of the construction of the distribution p(m) over matches m. Arrows indicate matches between training and test image. Each match proposes a translation, illustrated as dots in 2D. Based on these samples the distribution is estimated.

B. Keypoint Voting

After computing matches for keypoints in the test image, an organ label L is inferred for each keypoint in the test image based on the generative model illustrated in Fig. 3. The latent variable m represents keypoint matches from the previous step. With keypoint labeling it is possible to get a coarse localization of organs in the image. Further, keypoint labels are employed to guide the image segmentation as described in the following section. For inferring keypoint labels, we use the factorization from the graphical model in Fig. 3 and marginalize over the latent random variable m

$$p(L,F,\mathcal{L},\mathcal{F})=\sum _{m\in \mathcal{M}\left(F\right)}p(L,F,\mathcal{L},\mathcal{F},m)$$ (4)

$$=\sum _{m\in \mathcal{M}\left(F\right)}p(L\mid \mathcal{L},m)\cdot p(F\mid \mathcal{F},m)\cdot p\left(m\right),$$ (5)

where $\mathcal{M}\left(F\right)$ includes matches for keypoint F across all training images. Working with a sparse set of matches makes the marginalization computationally efficient. The label probability is defined as

$$p(L=l\mid \mathcal{L},m)=\{\begin{array}{cc}1\hfill & \text{if}{\mathcal{L}}_{m\left(F\right)}=l,\hfill \\ 0\hfill & \text{otherwise},\hfill \end{array}\phantom{\}}$$ (6)

where ${\mathcal{L}}_{m\left(F\right)}$ is the label of a training keypoint that the match m assigns to the test keypoint F. Based on the descriptor, the keypoint probability is defined as

$$p(F\mid \mathcal{F},m)=\frac{1}{\sqrt{2\pi {\tau }^2}}\text{exp}\left(-\frac{{\Vert F^D-{\mathcal{F}}_{m\left(F\right)}^D\Vert }_2^2}{2{\tau }^2}\right),$$ (7)

where we set ${\tau }^2={\max }_m{\Vert F^D-{\mathcal{F}}_{m\left(F\right)}^D\Vert }_2^2$. The most likely organ label is assigned to be the label of the keypoint $\widehat{L}$

$$\widehat{L}=\underset{l\in \{1,\dots ,\eta \}}{\arg \max }p(L=l\mid F,\mathcal{L},\mathcal{F})$$ (8)

$$=\underset{l\in \{1,\dots ,\eta \}}{\arg \max }p(L=l,F,\mathcal{L},\mathcal{F}).$$ (9)

To summarize, each training keypoint $\mathcal{F}$ that was matched to test keypoint F votes for the label L with the organ it is located in $\mathcal{L}$. If we would set the probabilities $p(F\mid \mathcal{F},m)\propto 1$ and p(m) ∝ 1 to constant values this would result in a majority vote for the keypoint label. To increase robustness, we weight the contribution of each training keypoint by the similarity of the descriptors $p(F\mid \mathcal{F},m)$ and the probability of the match p(m).

Fig. 3:

Graphical model for keypoint voting with the match m and the keypoint label L being latent random variables. The labels of training keypoints $\mathcal{L}$, the test keypoint F and the training keypoints $\mathcal{F}$ are observed, illustrated as shaded nodes.

C. Keypoint segmentation

The keypoint segmentation is based on keypoint matching and voting from the previous sections. In [11], we derived the segmentation method from a generative model with marginalization over latent random variables. Here, we provide an algorithmic perspective in Algorithm 1 instead, to emphasize the simplicity of transferring entire organs maps. The method uses the extracted keypoints F, the identified matches m, and the voted label of the keypoint $\widehat{L}$, as presented in previous sections. The objective is to estimate the segmentation

Algorithm 1.

Keypoint transfer segmentation. Organ segmentations from the training images are transferred to the test image via identified matches.

1: ∀l ∈ {1, …, η}, Sl = 0

2: for all F ∈ FI do

3:   for all $m\in \mathcal{M}\left(F\right)$ do

4:    if $\widehat{L}\ne {\mathcal{L}}_m$ then continue

5:    end if

6:    W = 0

7:    for all x : ${\mathcal{S}}_m\left(x\right)=\widehat{L}$ do

8:      $W\left(x\right)=\frac{1}{\sqrt{2\pi \nu }}\text{exp}\left(-\frac{{\left(I\right(x)-{\mathcal{I}}_m(x\left)\right)}^2}{2{\nu }^2}\right)$

9:    end for

10:    $S^{\widehat{L}}=S^{\widehat{L}}+W\ast p(L=\widehat{L})\ast p\left(m\right)$

11:   end for

12: end for

13: $\forall x:\widehat{S}\left(x\right)=\arg {\max }_{l\in \{1,\dots ,\eta \}}{S}^l\left(x\right)$

14: return $\widehat{S}$

$\widehat{S}$ based on probability maps for each of the organs S^l with l ∈ {1, …, η}. In short, the maps are computed by going through all the matches and transferring entire organ label maps from the training images, where the transformations are implied by the matches. Fig. 4 illustrates the procedure.

Fig. 4:

Illustration of keypoint transfer segmentation for the example of liver. The crosses indicate keypoints in the training and test images with the match m, illustrated as arrow. For the label transfer to take place, the label of the training keypoint ${\mathcal{L}}_m$ and the voted label of the test keypoint $\widehat{L}$ have to be liver. The segmentation map of the training image ${\mathcal{S}}_m$ = liver is then transferred and the probability map for liver S^liver is updated. To increase the robustness and accuracy of the segmentation, we weigh the transferred segmentation according to the certainty in (i) the keypoint label voting, (ii) the match, and (iii) the local intensity similarity of the test and training image.

In details, we let ${\mathcal{I}}_m$ denote the training image identified with match m after the transformation implied by the match has been applied. ${\mathcal{S}}_m$ is similarly defined to be the selected and transformed segmentation map. We initialize the probability maps S^l = 0, and then iterate through all the keypoints in the test image and associated matches. We only allow for those training keypoints to transfer their segmentation whose votes are consistent with the majority vote in Eq. (9), $\widehat{L}={\mathcal{L}}_m$. Preventing keypoints with inconsistent label from transferring the segmentation improves the robustness of the algorithm. Instead of directly transferring the binary organ map, we modulate it with the local similarity between test and training image, $I\left(x\right)-{\mathcal{I}}_m\left(x\right)$. Locations with similar intensities in test I and training ${\mathcal{I}}_m$ image obtain a higher weight than locations with larger intensity differences. We use a Gaussian distribution for obtaining the weights W from the image differences, where ν² is the intensity noise variance. The probability map of the organ $\widehat{L}$ is then incremented by the weights. In the increment, we also consider the certainty in the label voting and the match, by multiplying with the label probability $p(L=\widehat{L})$ and the distribution over matches p(m), respectively. Finally, we select the most likely label in the final segmentation $\widehat{S}$, where we account for not transferring the background surrounding the organ by assigning the background label if the maximal probability in the voting is below 15%.

In the presented algorithm, keypoints can only transfer segmentations that have the same label, e.g., a liver keypoint can only transfer liver segmentation maps. This may be overly restrictive because the transformation resulting from a good match may also be usable for neighboring organs. Transferring the organ segmentation more often may lead to a better approximation of the target organ shape. In our experimental evaluation, we therefore also investigate the transfer of organ segmentations that are different from the keypoint labels

We further study the potential benefit of accounting for affine organ variations across subjects. To this end, we estimate an organ-specific affine transformation when there are at least three matches for an organ between one training image and the test image. We use the random sample consensus (RANSAC) algorithm [31] to find the transformation parameters with the highest number of inliers. In our experiments, we have not observed a robust improvement of segmentation accuracy with the organ-wide affine transformation and therefore do not report it in the results. We believe that the reason for the affine transformation to not improve the results lies in the multiple transfer of organ labels per scan for different translations; this, in combination with probabilistic weighting, already accounts for much of the organ variability.

III. Results

We segment 10 anatomical structures (liver, spleen, aorta, trachea, left/right lung, left/right kidney, left/right psoas major muscle (PM)) and perform experiments on four different contrasts from the Visceral dataset resampled to 2mm isotropic voxels [18], [19]: contrast-enhanced CT (ceCT), whole-body CT (wbCT), contrast-enhanced MR (ceMR), and whole-body MR (wbMR). The dataset contains 20 images for each of the contrasts with manual annotations that we refer to as gold corpus. Image dimensions are roughly 217 × 217 × 695 for wbCT, 200 × 200 × 349 for ceCT, 252 × 87 × 942 for wbMR, and 195 × 108 × 240 for ceMR. All of the 10 structures are annotated on ceCT and wbCT scans, and most on wbMR. On ceMR, only liver, spleen, aorta, kidneys, and PM have sufficient annotations to allow for an evaluation. In addition to the gold corpus, a silver corpus exists with 65 ceCT, 62 wbCT, 71 ceMR, and 37 wbMR scans. The silver corpus does not have manual annotations but labels were created by fusing the results of several segmentation methods that were submitted to the Visceral challenge [19]. The fusion was done with the SIMPLE approach [5] and resulted in more accurate segmentations than any of the individual algorithms, but may not be as accurate as manual annotations [19]. Following [27], we set σ = 1.6 voxels and $\kappa =\sqrt[3]{2}$ for the keypoint localization. We use 10 bins along each dimension in the Hough transform and for the distribution p(m), where kernels with sigma 0.2 are employed.

On the gold corpus, we perform leave-one-out experiments, using 19 images for training and one image for testing. On the silver corpus, we use the 20 images from the gold corpus as training set and the fused segmentations as reference. We set ν = 50 for all organs, except for lungs and trachea on the CT scans, where we set ν = 300. We compare our method to multi-atlas segmentation with majority voting (MV) [3], [4] and locally-weighted label fusion (LW) [6] using ANTS [32] for deformable registration³. We quantify the segmentation accuracy with the Dice volume overlap between reference and automatic segmentation.

A. Gold Corpus

Statistics for the voting on keypoint labels are displayed in Table II. As expected, the average number of keypoints varies across organs, also influenced by an organ’s size. Keypoints that do not receive reliable matches due to spatial constraint and distance ratio test are not labeled. Since it is possible that certain keypoints in the test image do not appear in the training set, the focus on reliable keypoints improves the performance of the algorithm. We observe a high voting accuracy for keypoints that are labeled. Exceptions with lower accuracies are aorta and psoas major muscles on MR. Since we do not include background keypoints in the training set, all of the votes on background keypoints in the test image are incorrect. However, only about one third of the CT background keypoints and about one half of MR keypoints received labels. As long as there is no bias in transferring label maps to a specific location, the remaining background keypoints have limited impact on the segmentation outcome.

TABLE II:

We report keypoint voting statistics per organ for ceCT, wbCT, ceMR, and wbMR: the average number of keypoints per organ, the average fraction of keypoints that get labeled, and the average fraction of correct keypoints labels. If there exists no reliable match, keypoints are not assigned labels.

	Organs	Liver	Spleen	Aorta	Trachea	r.Lung	l.Lung	r.Kid	l.Kid	r.PM	l.PM	Bckgrnd
ceCT	# Keypts	13.6	4.0	7.6	3.0	29.7	24.7	12.1	12.2	2.5	3.0	526.0
	% Labeled	73	89	98	100	95	92	98	99	94	92	33
	% Correct	87	91	97	99	100	100	98	100	99	93	0
wbCT	# Keypts	6.0	2.6	5.6	4.4	28.2	24.0	6.7	9.0	2.5	2.5	637.2
	% Labeled	93	98	100	100	98	98	98	99	98	100	35
	% Correct	82	87	92	100	99	99	98	96	100	93	0
ceMR	# Keypts	25.1	4.0	-	-	-	-	9.5	11.8	2.5	1.8	562.9
	% Labeled	92	95	-	-	-	-	99	100	91	82	59
	% Correct	96	89	-	-	-	-	92	97	70	53	0
wbMR	# Keypts	22.1	4.1	7.1	1.9	3.2	2.9	9.3	10.8	2.6	3.0	709.8
	% Labeled	93	97	96	100	93	95	99	99	95	89	51
	% Correct	93	83	49	99	87	87	90	93	75	65	0

Fig. 5 reports segmentation results for all contrasts with keypoint transfer and multi-atlas segmentation. Across all anatomical structures, locally-weighted voting outperforms majority voting. Keypoint transfer segmentation leads to segmentation accuracy comparable to that of locally-weighted voting for most structures. Statistically significantly higher accuracy (two-tailed, paired t-test) was achieved for kidneys in ceCT (about 15 Dice points, p < 0.02), wbCT (about 20 Dice points, p < 0.001), and wbMR (about 20 Dice points, p < 0.005). Further significant improvements exist for aorta in wbCT (p < 0.05) and wbMR (p < 0.001), spleen in wbCT (p < 0.05), trachea in wbMR (p < 0.001), and psoas major muscles in wbMR (p < 0.01). Keypoint transfer is significantly worse than locally-weighted voting for liver in ceMR (p < 0.005) and lungs in ceCT (p < 0.005). The transfer of label maps that are different from the keypoint label did not yield a robust improvement in these experiments. Figs. 6 to 8 illustrate segmentation results for all the contrasts.

Fig. 5:

Segmentation accuracy on the gold corpus for different organs on ceCT, wbCT, ceMR, and wbMR images for majority voting, locally-weighted voting, and keypoint transfer. Bars indicate the mean Dice and error bars correspond to standard error.

Fig. 6:

Example segmentation results on coronal views for wbMR overlaid on the intensity images, shown for manual, keypoint transfer, locally-weighted multi-atlas.

Fig. 8:

Example segmentation results on coronal views for ceCT (left) and wbCT (right) overlaid on the intensity images. Each series shows segmentations in the following order: manual, keypoint transfer, locally-weighted multi-atlas.

The average segmentation result for ceCT scans when varying the number of training scans from 5 to 15 is shown in Fig. 9; the evaluation is done on the five images not included in the training set. As we increase the number of training images, the segmentation accuracy generally increases. We note a slight decrease for spleen, left lung and right PM for 15 scans, which may be due to the composition of the training set. Averaging over segmentations of a larger number of subjects therefore assists in recovering the true shape of the organ. In the future, the growing number of large datasets may therefore further improve the segmentation results. Moreover, keypoints may support the efficient implementation of an atlas selection scheme to only transfer organs from overall similar subjects.

Fig. 9:

Segmentation results for varying the number of training images from 5 to 15. Results are shown for ten organs on ceCT images with keypoint transfer. Bars indicate the mean Dice over five test images and error bars correspond to standard error.

The runtime of keypoint transfer segmentation and multi-atlas label fusion is compared in Fig. 10, with keypoint transfer being about three orders of magnitude faster than multi-atlas segmentation. For ceCT scans, as an example, the extraction of keypoints takes about 17s and the segmentation transfer takes 16s, yielding a segmentation time for ten organs that is about half a minute. We implemented the segmentation transfer in Matlab without parallelization. The pairwise deformable registration takes most of the runtime for multi-atlas segmentation. To reducing computational costs for atlas-based segmentation, we also experimented with creating a probabilistic atlas. However, the iterative estimation of the atlas is also expensive and the high anatomical variability of the abdomen makes the summarization challenging.

Fig. 10:

Average segmentation runtimes (in minutes) per image for keypoint transfer and multi-atlas label fusion with ten organs. The time is plotted on the logarithmic scale.

We further evaluate the impact of the different matching constraints from Section II-A (scale, spatial, and distance ratio) on the segmentation accuracy of ceCT data. We measure that 12.5% of matches are changed due to scale, 52.6% of matches are changed due to the distance threshold, and 60.9% of matches are discarded due to distance ratio. Fig. 11 shows the segmentation accuracy for the standard keypoint matching algorithm and variations by not using scale constraints, spatial constraints, or the distance ratio test. Not considering spatial constraints yields to the largest decrease in accuracy. For scale and distance ratio, there is no unanimous picture across all organs, but the overall Dice is highest by including all of the matching constraints.

Fig. 11:

Segmentation results for removing the individual matching constraints. Bars indicate the mean Dice over five test images and error bars correspond to standard error.

We plot the distribution over matches p(m) projected on the x-z axes in Fig. 12. We observe that translations proposed by the matches do not follow a uni-modal distribution.

Fig. 12:

Visualization of the distribution over matches p(m) for 2D (x-z).

Limited field-of-view scans

Next to the segmentation of abdominal and whole-body scans, the segmentation of scans with limited field-of-view was also evaluated. In clinical practice, such partial scans are frequently acquired because of a specific diagnostic focus. For evaluating the performance of the algorithm, we cropped ceCT and wbCT images around the kidneys and the spleen, as shown in Fig. 13. Specific for the segmentation of the spleen in the limited field-of-view scans, we noted a substantial improvement by transferring organ segmentations that are different from the keypoint label for spleen images. In this case, we also let lung and liver keypoints transfer the segmentation of spleen. Fig. 13 shows segmentation results for kidneys and spleen. In comparison to segmenting the whole scans, we observe a slight decrease in segmentation accuracy for partial scans. Overall, however, the key-point transfer is robust to variations in the field-of-view and enables segmentation without modifications of the algorithm. We do not show results for the multi-atlas segmentation in this experiment because the registration between the cropped images and the training images failed. Since the initial alignment does not yield a rough overlap of the target regions, it is a very challenging registration problem. While it may be possible to design initialization techniques that improve the alignment, we view it as a major advantage of the keypoint transfer that no modification is required to handle limited field-of-view scans.

Fig. 13:

Segmentation results for scans with limited field-of-view. We show coronal views of images with the kidneys and the spleen for ceCT and wbCT scans. Bars indicate the mean Dice and error bars correspond to standard error. We denote the use of lung and liver keypoints to transfer spleen segmentations as ‘Spleen Across’. For comparison, the mean Dice scores on the entire images for ceCT were 0.87 (r. kidney), 0.86 (l. kidney), and 0.76 (spleen), and for wbCT 0.75 (r. kidney), 0.81 (l. kidney), and 0.68 (spleen)

B. Silver Corpus

Fig. 14 reports segmentation results on the silver corpus for all contrasts, comparing keypoint transfer to multi-atlas segmentation. As for the gold corpus, locally-weighted voting outperforms majority voting for all anatomical structures. Further, we see similar results in the comparison of keypoint transfer and locally-weighted voting for the silver corpus and for the gold corpus. Key-point transfer achieves higher accuracy on kidneys for both CT contrasts. Liver, spleen and aorta are segmented more accurately on ceCT; aorta and trachea have higher Dice overlap for both whole-body modalities. Liver and spleen show lower accuracy for keypoint transfer than locally-weighted voting for both MR contrasts.

Fig. 14:

Segmentation accuracy on the silver corpus for different organs on ceCT, wbCT, ceMR, and wbMR images for majority voting, locally-weighted voting, and keypoint transfer. Bars indicate the mean Dice and error bars correspond to standard error.

IV. Discussion

Overall, we note a higher segmentation accuracy for CT than for MR scans. We think that the standardization with Hounsfield units in CT scans is potentially beneficial at several stages of the method. The construction of matches as well as the keypoint voting are influenced by the descriptor similarity across scans. Further, the weights W (x) in the keypoint segmentation are based on image intensity differences between training and test images. Studying the voting statistics in Table 1, we see that the voting accuracy for MR scans is comparable to CT scans for most organs. These good results for MR support the robustness of the descriptor to intensity variations, since the voting and matching is only based descriptor similarity. In contrast, the weights W (x) are computed on image intensity differences between training and test images, where the variations can have a larger impact. Approaches for correction of intensity inhomogeneity in MRI [33] may be helpful to increase the segmentation accuracy in the future.

In contrast to multi-atlas methods, keypoint transfer does not estimate a dense deformation field between scans. Instead, keypoints between organs are matched and the implied transformation is then used to transfer the organ segmentation. A less constrained deformation model, like this one, can have advantages in situations, where it is complicated to estimate image-wide deformation fields. We have shown this for the segmentation of scans with variations in the field-of-view, where keypoint transfer enabled the segmentation of partial scans. While we do incorporate spatial consistency of matches as weighting term p(m), this can handle multi-modal distributions and is less restrictive than typical regularization constraints imposed on deformation fields, particularly interesting for whole-body scans with wide inter-subject variation.

We have presented results on a gold and a silver corpus. The silver corpus does not have manual annotations but provides an opportunity to evaluate the segmentation methods on a larger dataset. Overall, the segmentation results across both corpora were similar. Since many of the segmentation techniques submitted to the Visceral challenge are based on multi-atlas techniques [19], there may be a bias in the fused segmentation of the silver corpus, favoring multi-atlas methods in the comparison.

The keypoint transfer segmentation relies on the identification of keypoints in the organs. While we did not experience this problem in our experiments, it may be a limitation for small organs without salient texture. An alternative in such scenarios could be to use keypoints from neighboring organs to transfer the segmentation. As shown for the spleen on the limited field-of-view scans, the transfer from neighboring organs can improve the segmentation result. However, we also performed experiments with using neighboring organs to transfer the label map on whole-body images and it did not lead to an improvement. A potential reason is that some organs are quite large so that keypoints can be fairly distant from the target organ. It may be a promising research direction in the future to restrict the transfer to keypoints that are close by.

Keypoint transfer is a very fast segmentation approach. Depending on the image type, the segmentation of a single scans takes between 10 and 84 seconds. This is orders of magnitude faster than multi-atlas techniques. In addition, there is no training stage in the algorithm that may require additional time. The only preparation for a new scan to be included in the training dataset is the extraction of keypoints. Keypoint transfer segmentation is therefore a highly scalable approach for large training and test sets.

V. Conclusion

We proposed an approach for image segmentation with keypoints that transfers label maps of entire organs. The algorithm relies on sparse correspondences between keypoints in the test and training images, which increases the efficiency of the method. We have further demonstrated that keypoint matches are robust to variations in the field-of-view, which allowed for the segmentation of partial scans. We evaluated the method on two CT and MR contrasts from gold and silver corpora. The accuracy of keypoint transfer segmentation compares favorably to multi-atlas segmentation, while being about three orders of magnitude faster. Since a segmentation can be obtained very quickly with keypoint transfer, the produced segmentation may be used as additional input for other segmentation approaches, e.g., based on deep learning.

Fig. 7:

Example segmentation results on coronal views for ceMR overlaid on the intensity images, shown for manual, keypoint transfer, locally-weighted multi-atlas.