Separating Touching Cells using Pixel Replicated Elliptical Shape Models

Abstract

One of the most important and error-prone tasks in biological image analysis is the segmentation of touching or overlapping cells. Particularly for optical microscopy, including transmitted light and confocal fluorescence microscopy, there is often no consistent discriminative information to separate cells that touch or overlap. It is desired to partition touching foreground pixels into cells using the binary threshold image information only, and optionally incorporating gradient information. The most common approaches for segmenting touching and overlapping cells in these scenarios are based on the watershed transform. We describe a new approach called pixel replication for the task of segmenting elliptical objects that touch or overlap. Pixel replication uses the image Euclidean distance transform in combination with Gaussian mixture models to better exploit practically effective optimization for delineating objects with elliptical decision boundaries. Pixel replication improves significantly on commonly used methods based on watershed transforms, or based on fitting Gaussian mixtures directly to the thresholded image data. Pixel replication works equivalently on both 2-D and 3-D image data, and naturally combines information from multichannel images. The accuracy of the proposed technique is measured using both the segmentation accuracy on simulated ellipse data and the tracking accuracy on validated stem cell tracking results extracted from hundreds of live-cell microscopy image sequences. Pixel replication is shown to be significantly more accurate compared to other approaches. Variance relationships are derived, allowing a more practically effective Gaussian mixture model to extract cell boundaries for data generated from the threshold image using the uniform elliptical distribution and from the distance transform image using the triangular elliptical distribution.

I. Introduction

In biological microscopy, cell segmentation delineates individual cells within each image. This is one of the most important, complex and error-prone tasks in biological microscopy image analysis. While there are seemingly endless combinations of algorithms for cell segmentation, there are two broad categories of tasks common to many segmentation algorithms [1, 2]. The first task is thresholding, converting the image to regions of foreground and background pixels (or voxels in 3-D). The second task is object partitioning. Given a set of foreground pixels that touch each other, or “connected components,” the object partitioning problem seeks to identify which pixels belong to which objects. This object partitioning step is a key challenge and is the focus of the present work. Here we describe a novel approach for the object partitioning task called pixel replication (PR). Pixel replication transforms data from the threshold image to better approximate data generated by a Gaussian mixture model (GMM), and then uses widely available and generally robust optimization techniques to fit the GMM. The contributions of the present work include the development and evaluation of the PR algorithm and also a derivation of variance equivalences that define the relationship between the GMM isocontours to the actual object boundaries in the input image.

PR proceeds as follows. First, the distance image is computed, with each foreground pixel storing the distance to the nearest background pixel. Next, the spatial location of each foreground pixel is entered repeatedly in a list – the “replication” step – with the number of repeats based on the distance image value. Segmentation then proceeds as with fitting a GMM to the spatial locations of foreground pixels in the threshold image, but instead it is fit to the pixel replication list. PR significantly improves on approaches based on watershed algorithms, and on using GMs fit to the threshold image. The data generated by PR is more accurately described as a nonlinear mixture of elliptical triangular distributions [3], but fitting a Gaussian mixture to the PR data is significantly more accurate compared to fitting a Gaussian mixture model directly to the image data.

The most common approaches to object partitioning are techniques based on watershed catchment basins [4]. These first transform the threshold image into a distance image where each foreground pixel value represents the distance to the nearest background pixel. Basins are identified by regional maxima within the distance transform image. The boundaries separating basins containing regional maxima in the distance image are used as cell boundaries. Watershed techniques are generally effective in the absence of noise in the threshold image used to create the distance transform, and when objects are relatively circular. However, watershed techniques are very susceptible to over-splitting and become less accurate in the case of elliptical cells or when the threshold boundaries are perturbed by significant noise.

Several edge fitting approaches have also been used for cell partitioning [5]. Edge or boundary geometry algorithms such as active contours and concave point detection require reliable edge and gradient information to accurately partition connected components of foreground pixels into cells. Particularly in 2-D transmitted light microscopy (e.g. phase, DIC, brightfield) images, gradient between overlapping cells may be unreliable or unavailable, making these approaches inaccurate. Lin et al. integrated gradient magnitude information with a standard watershed transform to provide better separation when gradient information was available [6]. Following the same approach, the PR approach presented here optionally uses edge gradient information, along with fluorescent information for multi-channel images, to weight the distance transform used in pixel replication to assist in partitioning cells. These data fusion techniques have the advantage of remaining robust even when minimal gradient information is available. Another common set of algorithms used for segmentation are based on level set contours, fit by minimizing an energy functional [7]. Cell segmentation applications in histopathology have recently achieved excellent results using techniques based on the repulsive level set that uses multiple level sets with a repulsive energy term for separating touching cells [8, 9]. These techniques, like other edge fitting approaches, generally require accurate gradient estimates, but can also incorporate some shape specific modeling. Recently, density estimation techniques have been proposed for clustering and cell separation [10, 11]. These techniques use assumptions about the concentration of fluorescent proteins inside each cell to assign voxels to individual cells. Density based approaches require that fluorophore concentration increases towards the center of cells. This occurs e.g. when using nuclear fluorescent markers. These approaches can be very effective in 3-D environments where fluorescent protein density is concentrated in the nucleus and the cells are spherical. However, density-based approaches cannot be used for clustering in 2-D images or 3-D fluorescence images where the fluorescent protein concentrations are variable throughout and between cells.

Another approach is to fit a weighted sum of Gaussian distributions to the spatial locations of each threshold pixel [12–14]. Gaussian mixture models (GMMs) have been used extensively in fitting color or intensity histograms; less common is their application directly to the spatial coordinates of the pixels. There are two advantages to using GMM compared to watershed and edge-based techniques for partitioning touching pixels into objects. First, the GMM fit is optimized over every pixel in the foreground, while the watershed considers only regional maxima of the distance image, and edge-based techniques utilize only boundary information. This makes the GMM more robust for noisy images. Second, Gaussians have a natural elliptical boundary [15] and ellipsoids are a good model for the morphology of cell nuclei [16], and for mouse and human neural and hematopoietic stem cells. However, the mass distribution of a Gaussian is very different from a “flat” threshold image, making the direct applications of GMM less accurate because of this difference in the underlying distributions. This difference between the Gaussian distribution and the flat threshold image is what motivates the present pixel replication algorithm.

Beyond the techniques described here, many methods for separating touching cells have been proposed. Curvature-based approaches are generally intended for 3-D images, relying primarily on high-curvature regions where the cells do not overlap to provide the separation information [17, 18]. Similar to curvature-based approaches are seeded-partitioning approaches [19]. Other techniques are based on eroded shape features [20, 21]. These approaches attempt to improve on the watershed algorithm by adding shape information via either the h-maxima transform or from a graph-cuts optimized cost function.

A key aspect of the cell partitioning task is in finding the number of cells in a foreground region. The parameter describing the number of cells in each component is often referred to as ‘K’. The approach proposed here is to separate the difficult question of determining K from the question of optimally partitioning pixels into K regions. We are not aware of any general purpose methods to reliably estimate K for different types of cells and imaging modalities. Instead, most approaches rely on application specific solutions. The watershed algorithm estimates K directly by using the number of basins found in the foreground region. This is not a robust estimator for K since watershed basins vary greatly with the eccentricity of the data and the level of noise in the threshold image. Watershed approaches are known for over segmentation, significantly overestimating K. An imaging and object dependent smoothing must generally be applied as part of the watershed algorithm to alleviate this problem. For watershed approaches, the question of estimating K can be separated from the partitioning task using seeded algorithms or merge strategies.

Shape-based methods such as marker-controlled watershed or two-stage graph-cuts use eroded shape features to identify the number of overlapping components. These approaches provide additional shape information for components and can be quite effective for cells with low overlap. However, erosion approaches lead to significant under-estimation of K when cells are highly overlapping [20, 21]. Clustering algorithms such as PR and Gaussian mixture models can also estimate K from the data, but goodness of fit measures for clustering algorithms tend to improve monotonically as the number of clusters K grows, encouraging over-segmentation. This has led to the development of criteria that can be used for identifying the most appropriate number of clusters in a region. Popular methods that can directly estimate K from a foreground region include the Aikake information criterion (AIC), Bayesian information criterion (BIC), and the gap statistic [15].

Many problem-specific techniques exist for estimating K. The repulsive level-sets algorithm implemented for comparison here uses a seed-finding algorithm based on gradient voting [8]. Similarly, our initial neural stem cell segmentation also uses a morphological gradient along with cell-size constraints to identify K [22]. However, gradient information often is insufficient in a single frame to accurately determine the number of cells in a cluster, particularly in high-overlap scenarios. For the data presented here, we incorporated temporal context from the tracker [14, 23, 24] and population context from the cellular lineage [12, 25–27], to determine the correct number of cells per-frame. This technique of using temporal context from a tracking algorithm for identifying the number of cells in a connected component of foreground pixels has proven very effective in several applications and was integral to acquiring the validated neural progenitor cell (NPC) data used here for tracking-based comparison of the watershed, PR, and mixture of Gaussians algorithms fit directly to the threshold image.

The following sections detail the implementation of the pixel replication algorithm. A distribution sampling approach is also discussed to reduce the memory and time required in the case of large connected components. The theory of general elliptical distributions [3] is then used to model threshold and distance transformed images for the purpose of deriving covariance relationships between Gaussian, uniform elliptical and triangular elliptical distributions. This enables the isocontours representing the elliptical boundaries to be computed using Gaussian mixture models applied to either threshold image (GMM) or PR data. Finally, the new pixel replication technique is compared to approaches based on watershed transforms, methods based on repulsive level sets, and methods that threshold image fit Gaussian mixtures. Each method is evaluated over thousands of 2-D and 3-D images of synthetic overlapping ellipses where the correct segmentation results are known exactly. The approaches are also compared by measuring tracking algorithm performance with different partitioning strategies against manually validated ground truth for hundreds of phase contrast movies of neural stem cells with 108,178 different partitioning cases. PR is shown to be significantly more accurate compared to the other methods. The open source PR software is described in section V.

II. Methods

A. Selecting K

Identifying K, the number of cells in a connected component of foreground pixels is a challenging task. For the present manuscript, our interest is in optimal partitioning into cells given K as input. For synthetic evaluation, data is generated with known K and known pixel/voxel ground truth. For the NSC image data, and for the sample image data shown as well, K was identified first by an initial segmentation algorithm specific to imaging modality. Next, the value of K was automatically refined using tracking information from time-lapse data and population information established by the lineage tree for cell death and mitotic events [28]. Finally, the value of K was manually validated by human observers in each image frame and any errors corrected [29, 30].

B. Pixel Replication

The implementation of pixel replication (PR) takes as input a threshold image and the number of cells K in each connected component of foreground pixels. The threshold image contains ones at each foreground pixel location and zeros at each background pixel location. We use the term ‘pixel’ here to refer to both pixels and voxels and below omit the explicit z, but the application of PR is independent of spatial dimension and is well-suited for 3-D data. PR starts with a threshold image T. The list of (x, y) coordinates of the foreground pixels of T is denoted as T_xy ={(x, y)₁...(x, y)_N}, with N being the total number of foreground pixels in T. The distance transform of image T is taken, and d (x, y) denotes the distance from pixel location (x, y) to the nearest background pixel. This value is rounded to the nearest integer and scaled if desired (see section II.B below). Next, a list $\stackrel{\rightharpoonup }{\mathbf{X}}=\{{(x,y)}_1^{d{(x,y)}_1}\dots {(x,y)}_N^{d{(x,y)}_N}\}$ is generated, where the notation (x, y)^{d(x, y)} means to repeat location (x, y) in the list d (x, y) times. Finally, a Gaussian mixture model (GMM) with K components is fit to $\stackrel{\rightharpoonup }{\mathbf{X}}$ and the segmentation result is obtained by clustering the locations T_xy using that GMM. A summary and example of the PR for two overlapping ellipses is shown in Figure 1.

Figure 1. Overview of pixel replication (PR) algorithm.

PR takes a threshold image (A) and the number of cells K in each connected component as input. The Euclidean distance transform, representing the distance from each foreground pixel to the nearest background pixel is computed (B). A list $\stackrel{\rightharpoonup }{\mathbf{X}}$ is generated by entering the spatial coordinates of each foreground pixel repeatedly. Each coordinate is repeated a number of times equal to the distance value at that pixel (C). The replicated coordinates are fit to a Gaussian mixture model (D). Each Gaussian component represents a single cell. Pixels are separated into individual cells using a maximum likelihood classifier. Elliptical cell boundaries are recovered using the covariance relationship between the Gaussian and elliptical triangular distributions derived in the present work (E), for 2-D the cellular boundary corresponds to the isocontour at a Mahalanobis distance of $\sqrt{20/3}$.

Combining different imaging modalities with PR is accomplished by thresholding each channel separately and combining the distance transforms from each channel using addition. Combining edge, or gradient, information with PR can be done in two ways. One is to threshold the gradient image and combine that logically with the thresholding of the intensity image. This approach has been used previously with the morphological gradient [12, 27]. An alternative approach is to combine the gradient image with the distance transform to produce an edge-weighted distance transform [6]. This is the approach adopted here. Section III details specific examples.

C. Pixel Replication with Distribution Sampling

The memory and time requirements for pixel replication, given a d-dimensional connected component of maximum radius r, is O (r^d+1). A d-dimensional sphere has volume proportional to r^d and the additional r factor is due to the linear increase of the distance transform up to r at the center of the d-sphere. There are two approaches to make pixel replication more efficient for large connected components. The first alternative would be to reduce the distance transform image by a constant scale factor before replicating. In practice we have found this to produce good results, but it may produce structured quantization errors. The second alternative is to treat the distance transform image as an empirical probability distribution and sample a specified number of points from this distribution. The method described here is a variant of the well-known rejection technique [31], modified to be more efficient for distributions with small support.

First, randomly choose a point (x, y) on the connected component. Next, generate a uniformly distributed acceptance probability P on [0,1]. The point (x, y) is accepted if P ≤ f (x, y) / cu (x, y) where function f (x, y) is the probability distribution estimated from the distance transform image and u (x, y) is the constant uniform probability on the nonzero region of f. The value c is the marginal expected number of rejections per point computed as max_x,y{ (f x, y) / u(x, y)}. When the generated points are not pixel aligned, we approximate f using local linear interpolation. This method is repeated until a specified number of points is generated. The accepted points from this process will follow distribution f.

D. Generation and Evaluation of Simulated Overlapping Ellipse Data

The performance of pixel replication was evaluated first using the accuracy of the segmentation at each pixel on simulated overlapping elliptical regions. Randomly generated ellipses were created with principal axis radii uniformly distributed on [10,30]. The ellipses were then rotated by a uniformly distributed angle on [0,2π]. This was done one thousand times each for two, three, four and five ellipses. To combine ellipses for an overlap image, points on each ellipse are picked at random, and the centroids of the ellipses are translated by the vector connecting the two randomly chosen points. This allows us to evaluate the accuracy of different partitioning approaches for every pixel. The same process was repeated for a 3-D dataset. Each of the four thousand simulated ellipse images was segmented using (1) pixel replication, (2) Gaussian mixtures fit directly to the spatial locations of the threshold image with no pixel replication, (3) a waterfall algorithm using a region merging step with the watershed approach to partition to K regions, (4) a seeded watershed that uses the centroids from the pixel replication segmentation result as seeds together with a geodesic distance measure, (5) an implementation of the repulsive level sets (RLS) algorithm described by Qi et al. using their seed detection technique but keeping only the best K seeds [8], and finally (6) using the same repulsive level sets algorithm seeded with the centroids found through pixel replication (RLS+PR).

The first watershed approach, often referred to as the waterfall algorithm, iteratively merges regions until only K regions remain [32]. Waterfall uses a simple merge strategy that nevertheless produces good results in many cases. The basin minimum to edge height is computed for all regions and the edge with the smallest difference is merged iteratively. An analogy to this merge approach is to raise the “water level,” beginning at each basin minimum, at the same rate. The first basin region that overflows will be merged into the receiving neighbor and the process is repeated iteratively until the specified number of regions are left. Pixel replication was also compared to a seeded watershed approach [33]. The seeded watershed algorithm uses centroids found by pixel replication as the seeds. These seed points are treated as basin minima and the standard watershed transform is applied to cluster the foreground to the nearest points.

The repulsive level set methods follow the approach described in [8]. The first step is seed point detection from boundary pixel voting. Local gradient information at each edge pixel contributes to seed point estimation weighted by a Gaussian distribution centered halfway between the minimum and maximum radii, orthogonal to the local surface estimate. Our approach differs from the original formulation in that instead of using mean shift to determine the actual number of seed points, we use the K highest voted seed points as detected by the algorithm, where K is specified as input. For each seed point, we create a level set functional, evolved to minimize the sum across all K level sets in the connected component. We use the same parameters as described in [8], with curve evolution performed as in [9]. Note that the RLS algorithm requires six parameters, compared to a single parameter (K) for PR.

A balanced error measure was used to compare the techniques. The balanced error rate is computed as the average across each ellipse of the number of misclassified pixels divided by the total area in each ellipse that could potentially be misclassified. Regions of the image that are overlapped by all ellipses cannot be misclassified and are excluded from the error rate calculation. The balanced error rate penalizes percentage of ellipse regions misclassified equally across all ellipses. This is important when partitioning overlapping ellipses of very different size. By contrast, direct misclassification rates encourage completely misclassifying a small ellipse in favor of correctly classifying all pixels of a large neighboringellipse. Since partitioning is invariant to permutations of the cluster numbering, all label permutations for each technique were evaluated and the lowest error rate labeling was chosen. Appendix B describes the balanced error rate computation in more detail.

E. Comparing Tracking Results with Different Segmentation Algorithms

Algorithms for object partitioning were next evaluated based on how many tracking errors the different partitioning approaches caused. The tracking errors were evaluated on time-lapse microscopy image sequences of a mixture of several kinds of proliferating cells derived from the adult mouse forebrain neural stem cell zone, here referred to collectively as neural stem cells (NSCs). The NSC data, captured in eight different imaging experiments from 232 movies, contained 379 clones or family trees, 1,130,496 total segmentations and 108,178 segmentations that required partitioning. This data was manually validated using the LEVER program [25] with any tracking errors corrected. The fully validated dataset, including image data and segmentation, tracking and lineaging results, can be explored using the interactive CloneView tool, http://n2t.net/ark:/87918/d9wc73/nscProcesses/index.html?Adult.

The performance evaluation for tracking accuracy starts by segmenting each connected component in the entire dataset that required partitioning using each of the six different algorithms. Each algorithm is provided with the correct value of K from the ground truth data. The re-partitioned cells are matched with cells in the manually corrected ground truth data by maximal overlap between ground truth segmentation and the partitioned cell. The tracking for all cells in each image frame is then recomputed, using the same multi-frame tracking algorithm (MAT) used by the LEVER program [34, 35]. All tracking connections that differ from the ground truth tracking are counted, and then reset to the correct value so that errors do not accumulate. This process is repeated for every image frame in the sequence. A similar method was used in previous work for comparing tracking performance with the full segmentation results to tracking performance using only the segmentation centroids [12].

F. Modelling the Threshold and Distance Transform Images with Elliptical Distributions

The Gaussian distribution belongs to a family of distributions known as elliptical distributions. The Gaussian distribution has a number of properties that make it highly effective for partitioning cells from regions of connected foreground pixels. First, the isocontours of the Gaussian distribution are elliptical. Second, techniques to fit mixtures, or linear combinations of Gaussians based on expectation maximization, are widely available and practically effective [15]. The idea behind pixel replication (PR) is to make the data obtained from the threshold image better suited to fitting with a Gaussian mixture. This section describes the use of the uniform elliptical distribution as a model for the threshold image, and the triangular elliptical distribution as a model for the distance transform of the threshold image. Equivalence relationships among the covariance and shape matrices for the Gaussian, the uniform and triangular elliptical distributions are derived in Appendix A. This allows us to determine the equivalent cell boundaries for data modelled via uniform or triangular elliptical distributions directly from the covariance matrix obtained from the Gaussian fit. This also gives a theoretical basis for the improved performance of pixel replication.

The uniform elliptical distribution is a constant-valued probability density function that is zero everywhere outside of an elliptical boundary. This accurately models the threshold image for cells with elliptical shape. The triangular elliptical distribution is also zero everywhere outside an elliptical boundary, but has a probability value that increases linearly towards the center of the ellipse. The triangular elliptical distribution is an accurate model of the distance transform of a threshold image for circular cells. For non-circular shapes the triangular elliptical distribution provides an effective analytical approximation.

The basis of our approach is that every elliptical distribution has a shape matrix ∑ whose eigenvalues and eigenvectors provide the magnitude and direction for the principal axes of the bounding ellipse for that distribution. In the special case of the Gaussian distribution the shape matrix is equal to the covariance matrix. For the uniform elliptical distribution in d-dimensions, the relationship between shape and covariance is given by $\mathbf{Cov}[X]=\raisebox{1ex}{$\mathrm{\Sigma }$}\!\left/ \!\raisebox{-1ex}{$(d+2)$}\right.$. Using a Gaussian fit to uniform data a multiplier of $\sqrt{d+2}$ on the principal radii obtained from the eigenvalues of the covariance matrix would represent the boundary of the uniform data. Figure 2 shows a standard 1-D uniform elliptical distribution with zero mean and unit variance. Note that the radius of the distribution (boundary distance on the x-axis) is $\sqrt{3}$. The covariance for the triangular elliptical distribution is related to the shape matrix ∑ of the elliptical boundary by $\mathbf{Cov}[X]=\frac{d+1}{(d+2)(d+3)}\mathrm{\Sigma }$. Scaling the principal radii obtained from the eigenvalues of the covariance matrix by a factor of $\sqrt{\frac{(d+2)(d+3)}{d+1}}$ ($\sqrt{20/3}$ in 2-D) exactly specifies the boundary radii for a triangular elliptical distribution. These scale factors provide a method for visualizing ellipse fits based on Gaussian mixture models or pixel replication. Figure 2 shows the standard triangular elliptical distribution with zero mean and unit variance, with radius of the distribution given by $\sqrt{6}$ (d=1).

Figure 2. Comparison of 1-D Gaussian, uniform and triangular elliptical distributions with zero mean and unit variance.

The uniform elliptical distribution models the threshold image. The triangular elliptical distribution models the distance transform image and is more similar to the Gaussian. Pixel replication generates data from the distance transform of the threshold image following more closely the triangular distribution and improving the fit by Gaussian mixtures.

G. Identifying Elliptical Decision Boundaries from Pixel Replicated Data

As described above, the elliptical triangular distribution is an effective model for the pixel replication data for a single cell. Accurately modelling multiple overlapping cells requires a non-linear mixture of elliptical triangular distributions. The parameters of this mixture model can be found using nonlinear optimization techniques, but in practice we have found that fitting Gaussian mixtures has better convergence properties. Gaussian mixtures have infinite support and are smooth, facilitating the search for optimal solutions. Gaussian mixtures are generally fit using expectation maximization (EM) [15]. Implementations of this are available in most image analysis platforms, including MATLAB, OpenCV, Java and Python.

For our purposes, we repeat the EM algorithm five times with different randomly chosen initial values. The replicate with the maximum overall log-likelihood is selected as the best partitioning. A maximum of 100 iterations is allowed for each replicate, and the algorithm is considered to converge if the log-likelihood changes by less than 10^-6.

III. Results

The accuracy of the pixel replication approach was compared on both simulated overlapping elliptical data and real adult and embryonic stem cells imaged using phase contrast microscopy. For both the simulated and real data pixel replication was compared against a spatial Gaussian mixture fitting as well as two watershed transform methods and two repulsive level set methods. The seeding algorithm used to provide initialization for RLS was not able to reliably adapt to the lack of gradient information in the real or simulated images, and in particular on highly overlapping elliptical cells. As an alternative seeding approach we formulated a version of RLS that uses the centroids obtained from the results of the pixel replication algorithm as seeds (RLS+PR). Likewise, the seeded watershed (SW) algorithm used the cell centroids returned by the pixel replication as seed locations.

Segmentation pixel accuracy was first evaluated directly using simulated overlapping ellipse data. Random configurations were clustered using all six algorithms. Data containing 2, 3, 4, and 5 elliptical components was generated for both 2-D and 3-D data. A balanced error rate was used for evaluation of accuracy. Figure 3 shows a sample of each clustering method applied to simulated ellipse data in both 2-D and 3-D, balanced error rates on these examples are noted below each technique in parentheses. Each algorithm was given the correct number of clusters K for each configuration. Pixel replication significantly outperformed both watershed techniques, as well as the RLS and RLS+PR, as well as the direct application of GMM to the threshold pixel locations (p-values using two sample t-test between classifiers as in [36]). Figure 4A summarizes the results of the simulated data comparison. PR was consistently better as measured by segmentation accuracy on the synthetic data (p<10⁻¹⁵).

Figure 3. Visualizing the performance of algorithms for segmenting touching cells.

Example random configuration of three ellipses in 2-D. Ground truth is shown as black outlines. The partitioning into 3 components by each algorithm (GMM=Gaussian Mixture Model, WF=waterfall, SW=seeded watershed, RLS=repulsive level set) is shown as (magenta, cyan, yellow) colored pixels. Incorrectly segmented pixels are marked with black crosshatching. Balanced error rates, averaged per component, are shown for each method in parenthesis.

Supplementary Movie 1 shows a similar example with a 3-D rendering to illustrate the performance and evaluation of the algorithms on 3-D simulated data.

Figure 4. Simulated and edit-based error rates of algorithms for segmenting touching cells.

Error rates for 1,000 each of [2, 3, 4, 5] randomly overlapping ellipses (A). Example synthetic ellipse shown in Figure 3. Pixel replication (PR) was compared to a waterfall algorithm that merges to K basins, to a seeded watershed initialized with the centroids found by PR, to a Gaussian mixture model (GMM) fit to the spatial locations of foreground pixels in the threshold image, and to a repulsive level set method initialized with a seed finding algorithm (RLS) and also initialized with seeds obtained from PR centroids (RLS+PR). PR significantly outperformed other methods (p<10⁻¹⁵). Error rates are shown by K (B) as well for each algorithm. Performance comparison using tracking error rates caused by different segmentation algorithms (C). A neural stem cell (NSC) dataset containing over one hundred thousand touching cells and one million total segmentations was processed with each algorithm. Tracking errors for each algorithm were identified using manually established ground truth. Combining PR with an RLS refinement did not significantly increase accuracy (p=0.6). PR was significantly more accurate compared to all other methods (p<0.03 for GMM). PR reduces tracking errors by more than a factor of two compared to the commonly used watershed approaches. Error bars are 95% confidence intervals.

Second, tracking accuracy was compared against manually validated tracking results from an adult neural stem cell (NSC) image dataset consisting of 8 experiments with 232 movies and a total of 1,130,496 tracking connections. Tracking errors make an excellent functional performance metric for a segmentation algorithm since single tracking errors can impact a large portion of the data being analyzed. A single tracking error can corrupt all of the subsequent time-sequence data that is erroneously connected. It is generally possible for analysis of segmentation and tracking data to be robust to segmentation errors, but not to tracking errors [37]. Many applications require that tracking errors be manually corrected. Segmentation correction is also highly user dependent making it difficult to establish a trusted ground truth. Each cluster of segmentations in the adult NSC datasets was reevaluated using all six partitioning methods then tracked frame-by-frame, counting tracking errors for each method. An example segmentation from this dataset is shown in Figure 5A-C. Each method was given the correct number of cells, K, for each segmentation.

Figure 5. Example segmentations using pixel replication (PR).

Phase contrast image of mouse adult neural stem cells undergoing division (A-C); threshold image used as input to PR (A), distance transform of threshold image rendered in 3-D gray overlaid with Gaussians obtained from PR colored in magenta and yellow (B). Original image with two cells and segmentation convex hull overlaid (C). Combined fluorescence (FUCCI) and brightfield mouse T lymphocytes (D-F); threshold image showing brightfield (white) combined with FUCCI threshold (green) (D); superimposed distance transforms in 3-D of threshold images for green (FUCCI) and brightfield channels colored by segmentation (E), original brightfield and FUCCI combined image with segmentation overlaid (F). Fluorescent human embryonic stem cells using H2B nuclear markers (G-H); threshold image (G), segmentation overlaid on original image (H). Note that the segmentation was recovered purely from the morphological information in the threshold image. Human embryo images (I-J); original image (I), isocontours of Gaussian obtained from PR colored in magenta and yellow (J). 3-D mouse embryonic neural stem cell image (K-L). Cells are labeled with a GFAP-GFP marker (K), and segmentation overlaid (L). The same PR implementation was used for all of these different cell types and imaging modalities. Scale bars 5μm.

PR was significantly more accurate compared to all other techniques (p<0.03). Combining pixel replication with a sub-sequent repulsive level set refinement initialized with the PR centers did not significantly improve upon the accuracy of the PR results (p=0.6). Figure 4B summarizes the tracking comparison results. For real NSC data and 3-D simulated data there is less overlap compared to 2-D simulated data and the cells show more eccentricity. Increasing the eccentricity of the 2-D simulated data increases the accuracy of the GMM and PR algorithms relative to the watershed approaches because the Gaussian boundaries are elliptical while the watershed uses Euclidean or geodesic distance. PR cuts error rates by more than a factor of two compared to the watershed algorithms for both real and synthetic data.

Pixel replication has been successfully applied to a wide variety of cell types and imaging modalities. Several examples of PR segmentation results are shown in Figure 5. These examples were chosen because they are moderately difficult, yet the correct partition is visually apparent. Panels A-C show mouse adult neural stem cells. There are two cells in the image. Despite the lack of gradient information between the two cells PR still separates the cells effectively using only the morphology implied by the distance transform, overlaid as a gray surface in Panel B. The individual Gaussian components fit for each adult neural stem cell are also shown as magenta and yellow surfaces. Panels D-F show mouse T lymphocytes imaged using both brightfield and fluorescence ubiquitination cell cycle indicator (FUCCI) [38]. The FUCCI signal is not present in every cell. PR is able to naturally fuse information from the fluorescence and brightfield threshold images by combining the depth transform images from each channel. Panels G-H show human embryonic stem cells labeled with a fluorescent nuclear marker (H2B). PR was able to accurately recover the segmentation based only on the morphology from the threshold image, without requiring gradient information.

Panels I-J show PR applied to segmenting cells within a human embryo during its progression to blastocyst. Here, edge gradient information is incorporated to better separate the highly overlapping cells; a weighted linear combination of the Hough accumulation array with the distance transform image is used as input to PR [6]. Finally, in panels K-L, 3-D image data of mouse neural stem cells labeled with GFAP-GFP are shown. In 3-D PR is used in exactly the same manner as 2-D. The key point is that the same PR implementation, with no additional logic, gets excellent results for these diverse applications.

IV. Discussion

Segmentation algorithms are often the first step in automated analysis of biological image data, and many downstream results are dependent on the accuracy of the segmentation. As such, segmentation algorithms are one of the most important components of an image processing pipeline. Often a simple thresholding approach will effectively separate foreground regions of cells from the background, however these regions will generally contain clusters of cells that touch. The most common approaches to separating these touching cells are based on watershed transforms. Pixel replication is a new segmentation technique for identifying objects, such as cells in a thresholded connected component. The key idea of pixel replication is that foreground regions often represent a cluster of objects and must be partitioned into multiple components. Pixel replication uses the distance transform of foreground regions to improve the fit of a spatial Gaussian mixture model to threshold pixel coordinates. PR is robust to noise in the threshold boundary and effectively fits objects with elliptical or approximately elliptical shape. PR is notable in being applicable to both 2-D and 3-D images, for both phase and fluorescence multi-channel images.

Unlike the watershed transform, pixel replication does not estimate the number of cells in each region K. The knowledge of K is an important parameter for any segmentation algorithm. The question of estimating K is most often addressed with application specific solutions. Several techniques, e.g. AIC, BIC, gap statistic, etc. can be used to estimate K directly from the fit data [15]. Temporal context has also been used to accurately estimate the number of cells in each foreground cluster [12]. In future work, it will be useful to evaluate different approaches to establishing K. Moving forward, investigating reliable and general-purpose techniques for estimating K that are applicable in both 2-D and 3-D images and across different imaging modalities will be an important area for future research.

The pixel replication algorithm significantly improves on the segmentation accuracy of watershed techniques as well as spatial Gaussian mixtures and repulsive level sets for separating the pixels of simulated overlapping elliptical data. PR also significantly improved tracking accuracy on validated adult neural stem cell data compared to watershed techniques and to repulsive level sets without seeds derived from the PR algorithm. The data was comprised of 8 experiments and 232 data sets. Using the watershed or repulsive level set segmentations would produce thousands more tracking errors that would require correction before further statistical analysis. This is a key improvement because many algorithms are robust to segmentation errors but require accurate tracking to effectively identify behavioral differences among cells [37].

Pixel replication assumes objects with elliptical boundaries. This is reasonable for many cell types and morphologies, particularly stem cells. However, this is not a practical technique for fitting tubular structures such as blood vessels. PR also requires solid objects suitable for computing a distance transform. In cases where objects contain large holes or extreme internal irregularities that make the distance transform unreliable, segmentation may be more accurate using a non-replicated GMM. A compelling area for future research would investigate how effectively the distance transform distribution is fit by the GMM model. This could lead to a hybrid approach for partitioning elliptical objects with varying morphology

V. Open Source Software

Software implementations for pixel replication are available in MATLAB, Python, and in Java as an ImageJ plugin. The simulation software for comparing synthetically generated ellipses using pixel replication, Gaussian mixtures on the spatial locations from the threshold image, the RLS, the seeded watershed and the waterfall routines are also available. The software is free and open source (BSD license) and can be downloaded from http://n2t.net/ark:/87918/d9mw2w.

Appendix A. Modelling the Threshold and Distance Transform Images with Elliptical Distributions

Gomez et al. provide a comprehensive survey of the family of elliptical distributions, and describe a technique for calculating their moments [3]. Following their approach, we derive here the relationship between the covariance matrices for the Gaussian, uniform and triangular elliptical distributions. The approach starts with a shape matrix ∑ whose eigenvalues and eigenvectors define the shape of the elliptical isocontours of the distribution. A d-dimensional elliptically distributed random variable X ~ E_d (μ, ∑, g) has centroid μ, shape matrix μ and g(t),t ∞ [0,∞) a non-negative function describing the relative distribution of mass along the squared radius of the ellipse. For example, g(t) = e^-t/2 corresponds to the Gaussian distributions parameterized by μ and ∑ where the covariance matrix and the shape matrix are equivalent. Constraining the domain of g to [0,1] for the uniform and triangular elliptical distribution allows the boundaries to be completely defined by ∑. A valid elliptical distribution must also have finite radial mass [3],

$$M_d={\int }_0^{\infty }{t}^{d/2-1}g(t)dt<\infty .$$

We construct an elliptically distributed random variable X ~ E_d (μ, ∑, g) stochastically by sampling from the compound random variable X= μ+A^T RU ^(d) where ∑=A^T A, U^(d) is a random variable distributed uniformly on the unit sphere in ℝ^d and R is a random variable with probability density function

$$h(r)=\frac{2}{M_d}{r}^{(d-1)}g\left(r^2\right),r\in [0,\infty ).$$

This representation is convenient for simulating from any elliptical family and it also provides a form for relating the statistical properties of X to the moments of R. In particular, if E[R]and E[R²] exist then E[X] μ [3] and

$$\mathbf{Cov}[X]=\frac{1}{d}\mathbf{E}\left[R^2\right]\mathrm{\Sigma }.$$ (1)

To compute the Gaussian, uniform and triangular elliptical distributions with the equivalent covariances then requires only the second moments of R. For the uniform elliptical distribution in d-dimensions let g(t) = 1for t𝜖[0,1] and zero otherwise. The radial mass is then written as

$$\begin{gathered} M_d={\int }_0^{\infty }{t}^{d/2-1}g(t)dt \\ ={\int }_0^1{t}^{d/2-1}dt \\ =\frac{2}{d}. \end{gathered}$$

Since M_d is finite and non-zero this is a valid elliptical distribution, and the second moment E[R²] is

$$\begin{gathered} \mathbf{E}\left[R^2\right]={\int }_0^{\infty }{r}^{2}h(r)dr \\ =\frac{2}{M_d}{\int }_0^1{r}^{d+1}dr \\ =\frac{d}{d+2}. \end{gathered}$$ (2)

Substituting eqn. (2) into eqn. (1), $\mathbf{Cov}[X]=\frac{1}{d+2}\mathrm{\Sigma }$ for the uniform elliptical distribution in d-dimensions. Using a Gaussian fit to uniform data a multiplier of $\sqrt{d+2}$ on the principal radii obtained from the eigenvalues of the covariance matrix would represent the boundary of the uniform data. Figure 2 shows a standard 1-D uniform elliptical distribution with zero mean and unit variance. Note that the radius of the distribution (boundary distance on the x-axis) is $\sqrt{3}$.

Similar to above we can derive the scale factor for a triangular elliptical distribution, setting g(t)=2(1- t^1/2) for t𝜖[0,1] and zero otherwise,

$$\begin{gathered} M_d={\int }_0^{\infty }{t}^{d/2-1}g(t)dt \\ =2{\int }_0^1{t}^{d/2-1}-t^{(d+1)/2-1}dt \\ =2\left(\frac{2}{d}-\frac{2}{d+1}\right) \\ =\frac{4}{d(d+1)}. \end{gathered}$$

This is a valid elliptical distribution with second moment E[R²],

$$\begin{gathered} \mathbf{E}\left[R^2\right]={\int }_0^{\infty }{r}^{2}h(r)dr \\ =\frac{4}{M_d}{\int }_0^1{r}^{d+1}-r^{d+2}dr \\ =d(d+1)\left(\frac{1}{d+2}-\frac{1}{d+3}\right) \\ =\frac{d(d+1)}{(d+2)(d+3)}. \end{gathered}$$ (3)

Substituting eqn. (3) into eqn. (1), the covariance for the triangular elliptical distribution is related to the shape matrix ∑ of the elliptical boundary by $\mathbf{Cov}[X]=\frac{d+1}{(d+2)(d+3)}\mathrm{\Sigma }.$

Appendix B –The balanced error rate for synthetic data

The balanced error rate is used to compare segmentation accuracy for different partitioning algorithms on overlapping ellipses. This approach is used for synthetically generated ellipses, where the precise ground truth is known. The ellipses are generated by choosing a uniformly distributed random length for the two principal axes, and then choosing uniformly distributed random angle on [0, 2π]. As each ellipse is added to the composite overlap image, a single point on each ellipse is chosen as the overlap origin, and the two ellipses are aligned at these points.

The segmentation algorithms return a label image. Each pixel is labelled with an identifier on [1, K] corresponding to the ellipse that pixel was assigned to. There are two challenges to measuring the performance of the algorithms. First, in regions where multiple ellipses overlap, it is acceptable to assign the pixels with the label of any overlapping ellipse. Second, the label values are not deterministic. any permutation of the label values is equally correct. What matters is that groups of pixels with the same label refer to the same underlying ellipse.

The error rate is computed by averaging the number of incorrectly assigned pixels (or voxels for the 3-D version) across all K ellipses. In regions where multiple ellipses overlap, points are only considered incorrect if they are assigned to an ellipse that does not overlap at that point. The ellipse generation routine identifies exactly which subset of ellipses each pixel belongs to, and then considers all permutations of a given labelling to compute the error rate. The permutation corresponding to the lowest number of incorrectly labelled pixels is selected. The error rate for each ellipse is then computed as the number of incorrectly labelled pixels divided by the total number of pixels (area) for that ellipse, and this value is averaged across all K ellipses to determine the final balanced error rate for the algorithm.