VOTING-BASED SEGMENTATION OF OVERLAPPING NUCLEI IN CLARITY IMAGES

Abstract

New tissue-clearing techniques and improvements in optical microscopy have rapidly advanced capabilities to acquire volumetric imagery of neural tissue at resolutions of one micron or better. As sizes for data collections increase, accurate automatic segmentation of cell nuclei becomes increasingly important for quantitative analysis of imaged tissue. We present a cell nucleus segmentation method that is formulated as a parameter estimation problem with the goal of determining the count, shapes, and locations of nuclei that most accurately describe an image. We applied our new voting-based approach to fluorescence confocal microscopy images of neural tissue stained with DAPI, which highlights nuclei. Compared to manual counting of cells in three DAPI images, our method outperformed three existing approaches. On a manually labeled high-resolution DAPI image, our method also outperformed those methods and achieved a cell count accuracy of 98.99% and mean Dice coefficient of 0.6498.

1. INTRODUCTION

With the advent of tissue clearing techniques and large-scale microscopy imaging methods, it is possible to image large tissue samples while sampling at a sub-micron level. Improvements in acquisition rate have introduced new challenges when analyzing these large datasets. Methods that can streamline image analysis will be of particular value to reduce processing burden. Segmentation of cell nuclei in microscopy images has been a topic of heightened interest because of its utility in biological studies. Algorithms such as watershed and Laplacian filtering can be inaccurate when applied to more complex, high cell-density images [1, 2, 3]. Recent segmentation approaches have addressed this problem using more state-of-the-art methods such as active contours, deep learning, and convex decomposition [4, 5, 6, 7, 8]. More general approaches for segmentation have been developed through analysis toolkits such as FARSight [9] and CellProfiler [10]. While both packages are limited to processing of 2-D images, CellProfiler provides a set of modular tools that can be used to create a custom segmentation pipeline.

We present a method that combines a 3-D extension of the Iterative Radial Voting algorithm [11, 12] with the Random Sample Consensus (RANSAC) algorithm [13]. Iterative Radial Voting, a discriminative model, predicts cell locations and candidate cell nucleus boundary points. RANSAC, a generative model, is then used at each location to fit a set of ellipsoid parameters to the region’s local nucleus shape.

2. METHODS

We focus on identification of cell nuclei in neocortex samples stained using 4,6-diamidino-2-phenylindole (DAPI), a fluorescent stain that binds to nucleic acids and fluoresces blue when exposed to UV light. Specifically, DAPI binds to the double-stranded DNA within the tissue, which is found only in the nucleus. DNA is distributed throughout the nucleus, which enables nuclear boundaries to be imaged using DAPI. Other cellular structures, such as axons and dendrites, do not bind to DAPI. As a result, only nuclei are visible in the microscopy image, and they present varied intensities within the nuclei, corresponding to non-uniform DNA concentrations.

Within a typical image (see Fig. 1a), there may be hundreds of nuclei intersecting the plane of section. The density of the nuclei within the image is high relative to those acquired from other types of tissue, for which more rudimentary segmentation approaches, e.g., watershed, may be effective. Because the imaged sections are thick relative to the nuclei dimensions and the distances between nuclei, the boundaries of nuclei within an image may overlap, even though they do not intersect in physical space (see Fig. 1b). It is thus important that we perform this segmentation task in 3D and identify nuclei shapes across the image slices. We do this using a locally generative method, which models the imaged nuclei as a collection of ellipsoids of varying sizes, positions, and orientations. Once nuclei positions have been identified, the generative method seeks to identify the optimal set of ellipsoid parameters that best matches the shape of the nucleus.

Fig. 1: CLARITY image.

(a) A 2048×2048 image of DAPI-stained PCW17 human neocortex showing numerous cell nuclei. (b) Close-up of boxed region. Nuclei appear to have intersecting boundaries because of thick imaging sections.

Algorithm.

We use three descriptors to characterize cell nuclei. The cell count c is a scalar representing the total number of cells in the stack. For each nucleus, indexed by i, we define its location, l_i, as a point in voxel space and its shape, s_i, as the orientation and axis lengths of an ellipsoid. The collection of nuclei is thus described by c, the set of nucleus positions L = {l₁, l₂,…,l_c}, and the set of nucleus shapes. The objective of our algorithm is to find the set of parameters that best fit the input image. We first use Iterative Radial Voting to estimate the cell count and a set of approximate locations for each of these nuclei. By determining the cell count and defining a range of possible ellipsoid axis lengths, we reduce the solution space for our generative method to a more tractable size. We then apply RANSAC around each location l_i to find a set of ellipsoid parameters most representative of the local nuclear shape.

Estimating cell count and nucleus locations.

Iterative Radial Voting [11, 12] identifies spherical objects using radial symmetry and is able to identify overlapping objects. It operates by iteratively tallying votes at each voxel based on image gradients, which contribute votes based on their magnitudes and directions. At each iteration, voting directions are adjusted to focus the predicted object centers in the voting landscape and improve the distinction between overlapping objects. The iterative radial voting algorithm works as follows:

1. Calculate the image’s normalized gradient G_n and Euclidean distance transform D. Define a gradient threshold and a pixel subset s = {p₀ | |G_n(p₀) > γ }. Initialize the voting .direction Γ(p₀) to be G_n(p₀).

2. For each pixel p₀ in subset s initialize a cone C(p₁; ϕ, h, p₀) with height h and radius r = h × arctan(ϕ) at the pixel’s location p₀ and oriented along its voting direction Γ(p₀).

3. Update the voting landscape.

$$V{\left(p_0\right)}^{\prime }=V\left(p_0\right)+\frac{1}{\sqrt{2\pi {\sigma }^2}}\text{exp}\left(\frac{-{\left|p_1-p_0\right|}^2}{2{\sigma }^2}\right)\times \left|G_n\left(p_0\right)\right|\times D\left(p_1\right)$$

1. Update the voting direction Γ(p₀).

$$p_1^{\prime }=\underset{p_1\in C\left(p_1;\varphi ,h,p_0\right)}{\text{arg max}}V\left(p_1\right)$$

$$\Gamma \left(p_0\right)=\frac{p_1^{\prime }-p_0}{\Vert p_1^{\prime }-p_0\Vert }$$

1. Decrement the cone’s angular range ϕ_new = ϕ − Δϕ and repeat steps 2 to 5 until ϕ ≤ ϕ_stopping.

Once the iterations have stopped, V (p₀) is thresholded and pixel clusters are identified. The nucleus centroids L are calculated from each cluster’s center of mass, yielding the nucleus locations of the image and the cell count c.

Nucleus segmentation.

Random sample consensus estimates model parameters in the presence of outliers. While RANSAC requires the model to be predefined, it is robust to noise and is highly efficient. The underlying method operates by selecting a small, random subset of points from the set and fitting the model to them. The model’s fit is evaluated against the remaining set of points and the prior step is repeated until the model is determined to estimate the entire set well. This ellipse fitting RANSAC model operates as follows:

1. Identify the set of boundary points B(p) in a rectangular neighborhood around each centroid l_i from the gradient image G_n, generating binary image B.

2. Skeletonize each z-slice of B, S(p) = skel(B(p_z = z)).

3. From the set of non-zero coordinates in S(p_z = z), randomly select 5 points and perform a least-squares ellipse fitting to obtain ellipse coordinates (e_x, e_y), 5 points being the fewest necessary to perform this fitting. The number of inliers n between the ellipse and binary image S(p_z = z) is calculated as the number of points less than distance d to the ellipse,

$$n=\sum _{\left(p_x,p_y\right)\in L}1\left(min\left(\Vert \left(p_x,p_y\right)-\left(e_x,e_y\right)\Vert \right)<d\right)$$

1. For each z value, repeat step 3 until n > n_threshold for an ellipse whose major axis lies within a pre-defined lower and upper bound [M_min, M_max] or a certain number of iterations has been completed.

2. Define the set of ellipse points across all z planes, E ={ (e_x, e_y, z) ∀z ∈ S(p)}.

3. An ellipsoid is generated from the set of points E using least-squares estimation.

Data.

Post-conception week 17 (PCW17) human neocortex was treated with A1P4 CLARITY hydrogel, polymerized, sectioned into 1mm slices, cleared, and stained with DAPI to reveal nuclei [14]. Fetal tissue was obtained according to IRB guidelines following voluntary termination of pregnancy. Three stacks of 54 images were acquired on a Zeiss LSM 780 confocal laser scanning microscope using a 20X objective with 16-bit resolution and interslice spacing 0.8682μm. Image dimensions and resolutions were: 1) 2048×2048 pixels, pixel size 0.2076μm×0.2076μm; 2) 1024×1024 pixels, pixel size 0.4152μm × 0.4152μm; and 3) 512 × 512 pixels, pixel size 0.8303μm × 0.8303μm.

We performed manual cell counting on slice 27 of all three image stacks using the ImageJ plug-in Cell Counter [15]. We counted 1539 unique nuclei in the 512 × 512 image, 2171 nuclei in the 1024 × 1024 image, and 2015 nuclei in the 2048 × 2048 image. The 2048 × 2048 image stack was also labeled manually to identify all pixels corresponding to each cell intersecting a single slice of the stack (slice 27). Labeling followed a written protocol that specified how to identify nuclei and how to label overlapping nuclei. Each nucleus in the slice was assigned a unique integer label, which was also assigned to any pixels determined to belong to the same nucleus in other slices in the image stack. This process identified 1985 unique nuclei that intersected slice 27 and required approximately 100 hours of person effort. We note that the two cell counts for this image were performed by different raters using different methods, and thus differ slightly (1.51%).

Preprocessing.

We applied two preprocessing steps to normalize the intensity within and between the slices. Image intensity is dependent on the intensity of the laser illuminating the plane of section during imaging. Light travels through the sample to illuminate the plane of section, thus deeper sections are illuminated with less light, producing an image intensity drop-off with z. We corrected for this by: 1) computing the mean of each z-slice image; 2) fitting a 3rd-order curve to the mean values; and 3) dividing each z-slice by the corresponding value in the fitted curve. Within each slice, we normalized the intensity values by applying an anisotropic diffusion filter. This removes noise while maintaining nucleus boundaries, which smooths the image content within the nucleus. Optical section thickness creates a physical limitation to interslice resolution, making sampling sparser across the interslice axis relative to the in-plane axes. As a result, voxel dimensions are larger in the z-component than in x and y, increasing the gradient contribution along this axis. We used a cubic spline to interpolate the image along the z-axis and then subsampled the gradient of this interpolated image to create a set of image gradients with normalized components.

Validation.

For each image stack, we applied our segmentation method to a subset of 11 slices centered on slice z = 27, for which cell nuclei had been counted manually and, in the case of the 2048 × 2048 image, labeled manually. For each result, we then calculated the intersection of each identified ellipsoid model with that slice and labeled each corresponding voxel within the ellipsoid with an integer designating the ellipsoid. This produced a 2D image, S(x, y), of individually labeled nuclei (see Fig. 2).

Fig. 2: Segmentation results.

(a) Original microscopy image section. (b) Image section feature map, calculated as its gradient magnitude. (c) Voting landscape. (d) Manual segmentation. (e) Segmentation generated by our algorithm. Section was selected to show correct and incorrect segmentation, e.g., the algorithm failed to detect a background nucleus in Fig. 2d’s upper-left corner.

We also processed the data using three other methods: Watershed (implemented in MATLAB), FARSight[9], and CellProfiler[10]. We ran Watershed 3D after Laplacian filtering of the preprocessed image to improve its segmentation results. We ran FARSight on both the preprocessed image and on the original image; the original data produced better results, which we present in this work. In CellProfiler, Gaussian global thresholding and shape-based object splitting with a predefined diameter range were used to segment the original microscopy image. For each method, this produced an integer labeling of each identified nucleus in slice z = 27 for each image stack. Nuclei counts were evaluated using accuracy a = 1 − |n_seg − n_gt|/n_gt, where n_seg and n_gt are the number of nuclei in the automatically segmented and manually counted images, respectively.

For the high resolution image, we compared the label result, S(x, y), of each algorithm against the corresponding slice, T (x, y), extracted from slice z = 27 of the manually labeled data. In this case, we computed accuracy using the count from the manual labeling. We computed recall and precision to determine the number of correctly detected nuclei. Label identifiers in S(x, y) and T (x, y) were matched to minimize the total distance between segmented nuclei and their ground-truth counterparts, where each ground-truth label was matched to exactly one segmented nuclei. The number of true positives TP was calculated as the number of segmented nuclei that were matched to nuclei from the manually labeled data. This was done by searching a spherical neighborhood around each manually labeled nucleus for a corresponding segmented nucleus. If a nucleus was found in this neighborhood that had not already been matched to a different manually labeled nucleus, it was counted as a true positive. If multiple unmatched nuclei were found, the nearest was assigned as the corresponding match. The number of false positives was calculated as FP = n_seg − TP and the number of false negatives was calculated as FN = n_gt − TP, yielding:

$$\text{Recall }=\frac{TP}{TP+FN}\text{\hspace{1em} Precision }=\frac{TP}{TP+FP}$$

$$\text{F-score}=2\frac{\text{ Precision }\times \text{ Recall }}{\text{ Precision }+\text{ Recall }}$$

Because manual and automated labels were assigned independently, there is not a direct correspondence of label identifiers between T and S. For each labeled nucleus λ_i in T(x, y), we determined the corresponding nucleus in λ_j in S(x, y) by finding the label value for the nucleus in T that had the largest overlap with the nucleus in S, to produce a new labeled image T′. We then computed the Dice similarity coefficient [16], $d_{{\lambda }_i}=2\left|S_{{\lambda }_i}\cap T_{{\lambda }_j}^{\prime }\right|/\left(\left|S_{{\lambda }_i}\right|+\left|T_{{\lambda }_j}^{\prime }\right|\right)$where S_λ and $T_{\lambda }^{\prime }$ are the subsets of pixels in S and T^′ that are labeled as λ, respectively. If no nucleus in S was found to overlap the nucleus in T′, a Dice score of zero was assigned.

3. RESULTS

We implemented our method using MATLAB and applied it to the test data. Processing the 2048×2048×11 image stack required 8.2h on a 4.2GHz Intel i7–7700k with 64GB RAM. Fig. 2 shows an example of the segmentation output and manual labeling for a small subset of the 2048 × 2048 image, selected to demonstrate successful and unsuccessful detections.

Cell count accuracy relative to the manually counted data is shown in Table 1. All methods showed reduced accuracy at lower resolutions, likely due to poorly defined boundaries between overlapping nuclei. Our method achieved the highest count accuracy at each resolution. For the 2048×2048 image, our method identified 1965 cells, yielding a count accuracy of 97.52% compared to the 2015 cells counted manually.

Table 1: Nuclei count accuracy.

Nuclei count accuracy for each test image resolution comparing each algorithm’s results with the manual count obtained using ImageJ.

Algorithm	512×512	1024×1024	2048×2048
Voting	70.83%	90.23%	97.52%
CellProfiler	67.06%	64.67%	74.99%
FARSight	43.60%	61.95%	84.91%
Watershed	6.36%	35.00%	61.14%

Detailed comparisons with manually labeled data were performed on the 2048 × 2048 image, including accuracy, recall, precision, and F-scores (Tab. 2), and mean Dice similarity coefficients and Dice score percentiles (Tab. 3). Our method segmented 62.34% of the cells with a Dice measure greater than 0.7, compared to 43.68% for CellProfiler, 32.31% for FARSight and 0.51% for Watershed. At a Dice threshold of 0.85, our method labeled 44.22% of the cells at this level, while CellProfiler labeled 29.32% and FARSight labeled 2.11% of the cells at this level; Watershed 3D segmented no cells with Dice scores above or equal to 0.85.

Table 2: Nuclei classification performance.

Classification performance for each algorithm compared to the manually labeled 2048 × 2048 nuclei image.

Algorithm	Accuracy	Recall	Precision	F-score
Voting	98.99%	0.9768	0.9833	0.9800
CellProfiler	76.12%	0.6766	0.9414	0.7873
FARSight	86.20%	0.5864	0.7395	0.6541
Watershed	62.07%	0.5441	0.9532	0.6928

Table 3: Segmentation performance.

Average Dice similarity coefficients and Dice percentiles relative to a manually labeled set of data. Percentiles reflect the fraction of segmented nuclei with Dice scores greater than 0.7 and 0.85.

Algorithm	Mean Dice	% Dice≥0.7	% Dice≥0.85
Voting	0.6498	62.34%	44.22%
CellProfiler	0.5963	43.68%	29.32%
FARSight	0.5061	32.31%	2.11%
Watershed	0.2224	0.51%	0.00%

4. DISCUSSION

The main contribution of this work is a new method for segmenting cell nuclei in fluorescence microscopy images using a 3-D extension to the existing Iterative Radial Voting algorithm described in [11] combined with a RANSAC-based ellipsoid fitting model. Our approach produced strong agreement with manually labeled high-resolution data based on measures of total cell count (98.99% accuracy) and average DICE similarity coefficient (0.6498) computed over all labeled cell nuclei. For this CLARITY data set, our method outperformed CellProfiler, FARSight, and Watershed. The purpose of this comparison was to demonstrate the application of segmenting overlapping nuclei in high-resolution CLARITY images, a problem not specifically addressed by these existing cell segmentation toolkits. While we do not claim that our method is a superior general purpose segmentation algorithm, we have shown that it can segment nuclei in DAPI-stained CLARITY images with higher accuracy at a number of different resolutions.

These promising results suggest several future directions for further evaluation and improvements to the method. One key improvement would be to reduce the time required to apply the method, particularly in the radial-voting portion of our algorithm. This might be achieved by improving the efficiency of the voting scheme or by developing a GPU-based implementation. The method would also benefit from a more extensive evaluation, based on additional manually labeled datasets and comparison with additional competing algorithms. We expect that these further studies will help to make our method more practical as a tool for quantitative analysis of cellular microscopy imaging.