AUTOMATED CELL SEGMENTATION WITH 3D FLUORESCENCE MICROSCOPY IMAGES

Abstract

A large number of cell-oriented cancer investigations require an effective and reliable cell segmentation method on three dimensional (3D) fluorescence microscopic images for quantitative analysis of cell biological properties. In this paper, we present a fully automated cell segmentation method that can detect cells from 3D fluorescence microscopic images. Enlightened by fluorescence imaging techniques, we regulated the image gradient field by gradient vector flow (GVF) with interpolated and smoothed data volume, and grouped voxels based on gradient modes identified by tracking GVF field. Adaptive thresholding was then applied to voxels associated with the same gradient mode where voxel intensities were enhanced by a multiscale cell filter. We applied the method to a large volume of 3D fluorescence imaging data of human brain tumor cells with (1) small cell false detection and missing rates for individual cells; and (2) trivial over and under segmentation incidences for clustered cells. Additionally, the concordance of cell morphometry structure between automated and manual segmentation was encouraging. These results suggest a promising 3D cell segmentation method applicable to cancer studies.

1. INTRODUCTION

Fluorescence microscopy imaging is an essential tool for quantitative investigation of cell biological characteristic in cancer research, with its main advantage being that it is compatible with living cells [1, 2]. With the recent advance in imaging technologies, three dimensional (3D) fluorescence microscopic images are now available to support such studies. As a first step to characterize such common cell properties as migration, division, and apoptosis, an effective and robust cell segmentation method is required. Despite the fact that a large number of cell segmentation methods have been developed, this problem remains challenging. Thresholding methods are efficient and applicable to images where foreground objects have sharp contrast to background noise [3]. However, this class of methods can not accommodate cases in which contrast between objects of interest and background noise is either weak or spatially variant. Watershed segmentation methods are good at differentiating clumped cells [4], but are prone towards over-segmentation. Cell segmentation has also been approached by kernel-based dynamic clustering [5]. However, this method involves an iterative clustering process that can be computationally expensive. Another new idea for 3D cell segmentation is to naturally group voxels by tracking the image gradient flow [6]. This method works well given the assumption that gradient vectors within a cell generally point to its center. However, this assumption only holds when parameters required for elastic deformation transformation are precisely selected. The final result is also subject to the bimodal voxel intensity distribution within each gradient-driven sub-volume. Therefore, the choice of gradient field regulation methods and cell component identification within each sub-volume could pose a challenge in practice. In this paper, we present solutions to address these problems.

2. METHOD FOR 3D CELL SEGMENTATION

Our workflow for 3D cell segmentation with fluorescence microscopy images consists of several modules, including pre-processing (volume data interpolation and smoothing), gradient field regulation, gradient mode detection, multiscale cell enhancement, adaptive thresholding, and post-processing (cell smoothing, and small object removal).

2.1. Data pre-processing

As the physical sizes per pixel in x, y, and z direction are dx = 1.19 μm, dy = 1.19 μm, and dz = 2.00 μm, we interpolated the entire image volume V (x, y, z) ∈ ℝ³ with 3D cubic interpolation [8]. To mitigate noise coupled in image volume, we applied Gaussian filter to the interpolated data volume: $\widehat{V}(x,y,z)=\tilde{V}(x,y,z)\ast G_{{\sigma }_0}(x,y,z)$ where * is the convolution operator.

2.2. Gradient field regulation

Driven by the idea that boundaries of 3D cell sub-volumes can be naturally found by tracking the image gradient flow [6], we investigated distinct methods to make gradient vectors within a cell volume generally point to its central core. The Gradient Vector Flow (GVF) field, first proposed to replace the traditional potential force, is a non-irrotational external force field composed of the irrotational and solenoidal component [10]. It is diffused from the original edge map and presents many new merits for gradient field based analysis. It can be derived from a variational formulation where the following energy functional is minimized:

$$\mathcal{J}(\mathbf{f})={\displaystyle \int {\displaystyle \int {\displaystyle {\int }_{\mathrm{\Omega }}ℒ\left(u_1,u_2,u_3,v_1,v_2,v_3,w_1,w_2,w_3\right)d{x}_{1}d{x}_{2}d{x}_3}}}$$ (1)

where: $ℒ(u_1,u_2,u_3,v_1,v_2,v_3,w_1,w_2,w_3)=\mu {\displaystyle {\sum }_{i=1}^3(u_i^2}+v_i^2+w_i^2)+|\nabla \widehat{V}{|}^2[{(u-\widehat{V_1})}^2+{(v-\widehat{V_2})}^2+{(w-\widehat{V_3})}^2]=\mu (|\nabla u{|}^2+|\nabla v{|}^2+|\nabla w{|}^2)+|\nabla \widehat{V}{|}^2|{(u,v,w)}^T-\nabla \widehat{V}{|}^2$;u_i = ∂u/∂x_i, i = 1, 2, 3, and ∇ is the gradient operator. As ℒ involves first-order derivative of functions {u, v, w} in GVF field with respective to multiple variables {x_i, i = 1, 2, 3}, functions {u^*, v^*, w^*} minimizing functional in (1) can be found by solving the following system of Euler-Lagrange equations:

$$\begin{array}{r}\frac{\partial ℒ}{\partial u}-\frac{\partial }{\partial x_1}\frac{\partial ℒ}{\partial u_1}-\frac{\partial }{\partial x_2}\frac{\partial ℒ}{\partial u_2}-\frac{\partial }{\partial x_3}\frac{\partial ℒ}{\partial u_3}=0\\ \frac{\partial ℒ}{\partial v}-\frac{\partial }{\partial x_1}\frac{\partial ℒ}{\partial v_1}-\frac{\partial }{\partial x_2}\frac{\partial ℒ}{\partial v_2}-\frac{\partial }{\partial x_3}\frac{\partial ℒ}{\partial v_3}=0\\ \frac{\partial ℒ}{\partial w}-\frac{\partial }{\partial x_1}\frac{\partial ℒ}{\partial w_1}-\frac{\partial }{\partial x_2}\frac{\partial ℒ}{\partial w_2}-\frac{\partial }{\partial x_3}\frac{\partial ℒ}{\partial w_3}=0\end{array}$$ (2)

Given $\mathcal{L}$ in (1), system of equations in (2) becomes:

$$\begin{array}{r}\mu \mathrm{\Delta }u-|\nabla \widehat{V}{|}^2(u-\widehat{V_1})=0\\ \mu \mathrm{\Delta }v-|\nabla \widehat{V}{|}^2(v-\widehat{V_2})=0\\ \mu \mathrm{\Delta }w-|\nabla \widehat{V}{|}^2(w-\widehat{V_3})=0\end{array}$$ (3)

where Δ is the Laplacian operator. A typical gradient field and the resulting GVF field $\overrightarrow{\mathbf{f}}={(u^{\ast },v^{\ast },w^{\ast })}^T$ over a small image volume are shown in Figure 1.

Fig. 1.

Gradient field of a small image volume. (Top) original gradient field ${\mathrm{f}}_0=\nabla \widehat{V};$ (Bottom) GVF field f_gvf after 15 iterations with μ = 0.1; Yellow points indicate |θ(f₀) − θ(f_gvf)| > π/4.

2.3. Gradient convergence mode detection

After GVF field computation, voxels were next grouped into distinct sub-volumes based on gradient modes detected by tracking directions of vectors in GVF field. Given a voxel ${\mathbf{x}}^{(a)}={(x_1^{(a)},x_2^{(a)},x_3^{(a)})}^T$, the next tracked voxel x^(b) is:

$${\mathbf{x}}^{(b)}={\mathbf{x}}^{(a)}+\mathcal{U}\left(\overrightarrow{\mathbf{f}}\left({\mathbf{x}}^{(a)}\right)+\eta \right)+\mathcal{U}\left(\overrightarrow{\mathbf{f}}\left({\mathbf{x}}^{(a)}\right)-\eta \right)-1$$ (4)

where $\mathcal{U}\left(\overrightarrow{\mathbf{f}}(\mathbf{x})+\eta \right)$ is a vector of step functions:

$$\mathcal{U}\left(\overrightarrow{\mathbf{f}}(\mathbf{x})+\eta \right)=\left(\begin{array}{c}U\left(u(\mathbf{x})+\eta \right)\\ U\left(v(\mathbf{x})+\eta \right)\\ U\left(w(\mathbf{x})+\eta \right)\end{array}\right),U(x)=\{\begin{array}{ll}1,& \mathrm{if }x\ge 0\\ 0,& x<0\end{array}$$

The tracking process is continued until either it converges to a gradient mode or all voxels have been associated with some gradient mode. The location of a voxel is defined as the gradient mode m_ij, i.e. mode j in cell i from the set M_i = {m_ij, ∀j}, if the angle between GVF vector at this voxel and that of its previous voxel on the tracking path is greater than or equal to π/2. Due to artifacts, tracking paths P_i = {p_ik, ∀k} from distinct voxels of a cell can converge to a set of distinct gradient modes nearby $\widetilde{M_i}=\{m_{ij}|\exists m_{ik}\in \widetilde{M_i},s.t.\Vert m_{ij}-m_{ik}\Vert <\epsilon \}$. If the set cardinality $|\widetilde{M_i|}$ is not zero, adjacent gradient modes were combined [6].

2.4. Cell component identification

For a given cell i, we assigned a unique integer label to all voxels on tracking paths P_i = {p_ik, ∀k} leading to the same combined gradient mode. However, not all associated voxels represent foreground cell in most cases. Some exhibit too weak fluorescence signal strength to be a cell voxel. As a result, there are two classes of voxels to be identified based on intensity: cell and background voxels. This is a classic problem where adaptive Otsu thresholding can be used to find the optimal threshold maximizing the between-class variance [9, 6]. To enhance the bimodal intensity distribution required by Otsu method, we used a multiscale cell enhancement filter to improve intensity contrast between cell and background voxels.

With the smoothed and interpolated 3D volume $\widehat{V}(\mathbf{x})$, the intensity of voxel x in neighborhood of voxel x₀ can be represented by Taylor series expansion:

$$\begin{array}{l}\widehat{V}(\mathbf{x})=\widehat{V}({\mathbf{x}}_0)+{(\mathbf{x}-{\mathbf{x}}_0)}^T\nabla \widehat{V}({\mathbf{x}}_0)\\ +\frac{1}{2!}{(\mathbf{x}-{\mathbf{x}}_0)}^T\mathscr{H}\{\widehat{V}({\mathbf{x}}_0)\}(\mathbf{x}-{\mathbf{x}}_0)+\mathcal{O}({\mathrm{x}}^3)\end{array}$$ (5)

where $\mathscr{H}(\widehat{V}(\mathbf{x}))$ is a 3 × 3 Hessian matrix with its entry $H\left(i,j\right)={\partial }^2\widehat{V}(\mathbf{x})/(\partial x_i\partial x_j)$. As Hessian matrix is symmetric, it is always diagonalizable. We denote eigen values of $\mathscr{H}(\widehat{V}(\mathbf{x}))$ by λ_i, i = 1, 2, 3, subject to |λ₁| ≤ |λ₂| ≤ |λ₃|. If voxel x₀ is close to the brightest voxel at the center of a bright sphere, the resulting eigenvalues of $\mathscr{H}(\widehat{V}(\mathbf{x}))$ are constrained by [11]:

$$\{\begin{array}{l}{\mathrm{\lambda }}_3\le {\mathrm{\lambda }}_2\le {\mathrm{\lambda }}_1\ll 0\\ |{\mathrm{\lambda }}_1|\approx |{\mathrm{\lambda }}_2|\approx |{\mathrm{\lambda }}_3|\end{array}$$ (6)

With $\overrightarrow{\mathrm{\lambda }}={({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2,{\mathrm{\lambda }}_3)}^T$ the following cell similarity function is formulated to describe the intensity similarity between the local volume structure and a 3D sphere centered at x₀:

$$C(\mathbf{x},s)=\left[1-\mathit{exp}(-\frac{{\mathrm{\lambda }}_1^2}{2{\sigma }_1^2|{\mathrm{\lambda }}_2||{\mathrm{\lambda }}_3|})\right]\left[1-\mathit{exp}(-\frac{s^2|\overrightarrow{\mathrm{\lambda }}{|}^2}{2{\sigma }_2^2})\right]$$

when λ_i < 0 ∀i; C(x, s) = 0, otherwise.

As cells have variable sizes, we accommodated this variation by convolving $\widehat{V}$ with Gaussian filter with different scale G(x, s_i) and selected the maximum cell similarity function value of all scales. Assuming s^* is the scale associated with the maximum cell similarity, we enhanced voxel intensity as:

$$V^{\ast }(\mathbf{x})=\widehat{V}(\mathbf{x})\mathit{exp}(\kappa (C(\mathbf{x},s^{\ast }(\mathbf{x}))-\alpha ))$$ (7)

where α > 0 is close to zero, and κ > 0. For any voxel x with C(x, s^*(x)) > α, the voxel intensity is increased. Otherwise, it is depressed. In this way, we forced voxel intensity distribution to be more bimodal and constrained cell shapes deviated from a sphere substantially.

2.5. Data post-processing

In post-processing, we smoothed each 3D cell shape by performing morphological closing on the binary image volume associated with each cell component [12]. In addition, we removed all unduly small objects by a lower volume bound ${\mathcal{V}}_{min}$. In Figure 2, we present a 3D view of voxel association within a small image volume, its cell enhancement result at a middle plane, local adaptive thresholding result, and final segmentation result.

Fig. 2.

3D view of cell identification with a small image volume. (A) voxel association; (B) multiscale cell enhancement at a middle plane; (C) adaptive cell segmentation; and (D) final segmentation result.

3. EXPERIMENTS AND RESULTS

We developed our method with the driving use case of glioblastoma (GBM), a high-grade brain tumor. In this study, GBM that were resected neurosurgically were grown in-vitro as neurosphere cultures. Cultured cells were then implanted on biologically relevant 3D scaffolds that were 200 μm in thickness. This ex-vivo platform enables replication of a nearly authentic brain environment in which glioma cells interact with extracellular matrices within a 3D space [7]. Using confocal microscopy (Zeiss LSM 510), we acquired 3D fluorescence Z sliced images of tumor cells. In aggregation, this image volume consists of 73 interpolated images, with each image resolution of 512 × 512 pixels. Figure 3 shows the 3D fluorescence image volume and the Maximum Intensity Projection (MIP) of cell profiles.

Fig. 3.

(A) 3D data volume; (B) 2D MIP image.

We applied the proposed method to the 3D fluorescence image volume of human brain tumor cells with the following parameter setup: σ₀ = 0.65, μ = 0.1, N_gvf = 15, η = cos(π/3), ε = 2, κ = 5, α = 0.05 and ${\mathcal{V}}_{min}=10$. This set of parameter values is empirically chosen. Moderate variations to this set can lead to similar results. Final segmentation result over the entire data volume is presented in Figure 4 where cells are color coded. We quantitatively assessed segmentation results by two validation studies. With the interpolated image volume $\tilde{V}(512\times 512\times 73\mathrm{ by size})$, each image slice was manually reviewed. The resulting annotations were used as ground truth and compared with machine-generated 3D cell segmentation results. In the first validation study, four metrics were computed for each image slice: Miss, False, Over, and Under. We evaluated our approach with single and clustered cells separately. Miss and False represent missing and false recognition rates of machine-based method on individual cells with no touching neighbors, whereas Over, and Under show number of cell clusters with machined-segmented cells more and less than the true number by human, respectively. The resulting validation outputs (mean%±std) are presented in Table 1 for two sets of images from the volume, as there is a large variation in cell number per image across slices. Set S₁ includes all 73 image slices, whereas set S₂ contains 28 images with cell number greater than or equal to 20 each. We notice that our method presents satisfactory performance as measured by all measures. False recognition and miss segmentation rates for single cells are comparable, while over and under segmentation incidences are small for overlapped cells.

Fig. 4.

3D cell segmentation of the entire image volume with typical 3D cell mesh representations at corners.

Table 1.

Validation result (mean%±std) of cell segmentation on all image slices (set S₁) and 28 slices with N ≥ 20 each (set S₂); N and $\widehat{N}$ are the true and machine-identified cell number per image slice, respectively.

Set	N	$\widehat{N}$	Miss	False	Over	Under
S1 (73)	31.92±42.36	32.07±42.76	0.68%±0.02	0.59%±0.01	0.69%±0.01	0.46%±0.01
S2 (28)	77.36±35.52	77.82±36.14	1.09%±0.01	1.32%±0.01	1.80%±0.01	1.01%±0.01

To further characterize the concordance between human- and machine-generated cell morphometry structures, we randomly selected 20 cells and compared the cell structure similarity by volume- and distance-based measures. We denote A and B as 3D cell structures by human and machine, respectively. Volume-based measures include Jaccard coefficient (J = |A ∩ B|/|A ∪ B|), Precision (P = |A ∩ B|/B), Recall (R = |A∩B|/A) and F₁ score (F₁ = 2PR/(P+R)), while Hausdorff distance is used as a distance-based metric. In Figure 5, we present validation results on cell surface concordance between human and machine segmentation results. Comparisons with these randomly selected cells lead to the following results by the five metrics above: J = 0.96 ± 0.06, P = 0.99 ± 0.01, R = 0.97 ± 0.06, F₁ = 0.98 ± 0.03, and d_H = 1.81 ± 0.93 (voxel). Note that most cells present high agreement between human and machine segmentation results, with two cells perfectly matched. Cells with the worst concordance by different metrics still present satisfactory result: J_low = 0.75, P_low = 0.95, R_low = 0.75, F_1low = 0.86, and $d_{H_{\mathit{high}}}=4$. In future work, we will develop a tracking algorithm and apply proposed segmentation method to longitudinal 3D image volumes of brain tumor cells to quantitatively characterize cell invasion property in 3D space.

Fig. 5.

Evaluation of human-machine segmentation agreement on 20 randomly selected cells with different metrics. (A) 1 − J; (B) 1 − P; (C) 1 − R; (D) 1 − F₁; and (E) d_H

4. CONCLUSIONS

In this paper, we present a promising cell segmentation method applicable to 3D fluorescence microscopic images, with specific solutions to mitigate parameter sensitivity and to improve intensity bimodal distribution assumption required for gradient field regulation and adaptive thresholding proposed in [6]. Our validation results on a 3D imaging volume of human brain tumor cells present a good concordance between automated and manual segmentation, suggesting our method’s potential to be applied to a large set of cell-oriented biology and cancer research.