A Probabilistic Approach to Joint Cell Tracking and Segmentation in High-Throughput Microscopy Videos

Abstract

We present a novel computational framework for the analysis of high-throughput microscopy videos of living cells. The proposed framework is generally useful and can be applied to different datasets acquired in a variety of laboratory settings. This is accomplished by tying together two fundamental aspects of cell lineage construction, namely cell segmentation and tracking, via a Bayesian inference of dynamic models. In contrast to most existing approaches, which aim to be general, no assumption of cell shape is made. Spatial, temporal, and cross-sectional variation of the analysed data are accommodated by two key contributions. First, time series analysis is exploited to estimate the temporal cell shape uncertainty in addition to cell trajectory. Second, a fast marching (FM) algorithm is used to integrate the inferred cell properties with the observed image measurements in order to obtain image likelihood for cell segmentation, and association. The proposed approach has been tested on eight different time-lapse microscopy data sets, some of which are high-throughput, demonstrating promising results for the detection, segmentation and association of planar cells. Our results surpass the state of the art for the Fluo-C2DL-MSC data set of the Cell Tracking Challenge (Maška et al., 2014).

Graphical abstract

1. Introduction

Thanks to advances in automation, thousands of cell populations can be perturbed and recorded by an automated microscope, making live cell imaging a widespread and versatile platform for quantitative analysis of cellular processes. Nevertheless, the pace of modern imaging far outstrips the capability of biologists to manually analyse the resulting movies. Automated image processing can extract a richness of quantitative measures far beyond what a human can observe. Yet, to fully exploit the power of live-cell imaging, a spatiotemporal tracing of multiple cells in a dynamic environment is required. To address this challenge for large data sets, numerous cell tracking algorithms have been developed and have become a focus in the bioengineering community (Maška et al., 2014; Ulman et al., 2017). While many tracking and detection algorithms for specific experimental setups exist, the construction of a generally applicable tool for a variety of datasets, without exhaustive training, remains a challenge. We hereby present an unsupervised approach for joint cell segmentation and tracking that allows automatic and systematic extraction of quantitative measurements that can be applicable to different experimental situations.

Spatiotemporal tracing is often considered as two sequential problems: the spatial definition of the cells within the frame, i.e., segmentation, and the temporal frame to frame association, i.e., tracking. There are numerous approaches that address the segmentation problem in general and that of live cell imaging in particular. The majority of these methods rely on low level image features, such as pixel intensities and gradients, to separate cells from background and to distinguish between the different cells. This could be done by different algorithms ranging from basic methods such as adaptive thresholding (Padmanabhan et al., 2010; Otsu, 1975) to more sophisticated methods, such as deformable models and active contours (Chan and Vese, 2001; Osher and Sethian, 1988). For example, the Active Mesh Dufour et al. (2011) approach allows the users to utilize computer graphics models to find accurate segmentation with low computational power. Watershed transformation introduced by Beucher (Beucher and Meyer, 1992) is widely used for cell segmentation e.g., Wählby et al. (2004). The graph-cut method Boykov and Funka-Lea (2006), common in natural image processing has also been applied to microscopy images e.g., Bensch and Ronneberger (2015).

In most cases relying on low level image features alone is insufficient for the segmentation task. Prior knowledge adds orthogonal information and can improve the overall result. An interesting model-driven approach suggested by Kanade et al. (Kanade et al., 2011) and Su et al. (Su et al., 2013) incorporates the physical properties of the microscope within the segmentation. Machine learning methods that aim to be general and not specific to acquisition conditions or modality, require training data, usually obtained by manual annotations. For example, Ilastik (Sommer et al., 2011) - a commonly used interactive tool for cell segmentation, is based on the Random Forest classifier (Breiman, 2001), and is trained by labeled pixels provided by the user annotation. In Ronneberger et al. (2015) a deep artificial neural network (U-Net) is presented for segmentation. The classification deep neural network in Kraus et al. (2016) utilizes the Jacobian maps for cell segmentation. While these Deep Learning methods and Ilastik provide accurate segmentation results they require a comprehensive set of annotated examples.

Provided that the segmentation problem is well-posed, cell-to-cell association is commonly approached by finding correspondences between cell features in consecutive frames. Cell association becomes complicated when the feature similarity of a cell to its within-frame neighbors is comparable to the similarity of the same cell in consecutive frames. When cells cannot be easily distinguished, a more elaborate cell matching criterion is needed, for example, considering cell dynamics (Yang et al., 2006), solving sub-optimal frame-to-frame assignment problems, via linear and integer programming optimization (Kachouie and Fieguth, 2007) and its expansion to global cell association (Türetken et al., 2016, 2017; Bise et al., 2011), or by using multiple hypothesis testing (MHT) (Reid, 1979) and its relaxation (Jaqaman et al., 2008). Common recent approaches, in this spirit, to address the cell tracking problem are based on graphical models. The key idea is the construction of a graph, in the spatial and/or time domain, of possible hypotheses and using it to find the globally optimal solution (Jaiswal et al., 2016; Akram et al., 2016; Schiegg et al., 2013; Magnusson et al., 2015; Padfield et al., 2011).

Often an accurate delineation of cell boundaries is a challenging task. A high degree of fidelity is required for cell segmentation, even in instances where the cells are far apart and hence can be easily distinguished. Moreover, in many cases the extracted cellular features (e.g., shape or intensity profile) are also the intended subject of the biological experiment. Therefore, several recent methods attempt to support segmentation through solving the cell association problem and thus extract prior information from previous frames. For example, the initial cell boundaries in the active contour framework can be derived from the contours of the associated cells in the previous frame as long as cell position is relatively stable Bergeest and Rohr (2012); Dufour et al. (2005); Wang and Chung (2007); Zimmer et al. (2002); Dzyubachyk et al. (2010). An alternative active contour strategy is to segment a series of time-lapse images as a 3D volume (Padfield et al., 2008). Li et al. (2008) incorporate multiple modules including cell motion prediction and active contours to simultaneously perform segmentation and data association. More recent methods successfully deal with complex data sets using probabilistic frameworks. In the graphical model suggested by Schiegg et al. (2014) cell segments are merged by solving multiple hypothesis testing subject to inter-frame and intra-frame constraints. The Gaussian mixture model proposed by Amat et al. (2014) is based on the propagation of cell centroids and their approximated Gaussian shape to the following frame in order to combine super-voxels into complete cell regions.

In this paper, cell tracking and segmentation are jointly solved via two intertwined estimators. The first is the motion estimation filter, inspired by the Kalman Filter. The second is maximum a posteriori probability (MAP) of a pixel belonging to a cell. Consider Figure 1 showing an example of two consecutive frames at times t and t+1. The segmentation of the cell at time t (red) may not match the true segmentation at time t+1 due to the cell’s dynamics. Translation of the contour in frame t by using cell motion estimation (magenta) allows better alignment. The final cell segmentation (green) can be calculated based both on the estimated contour and frame t+1.

Figure 1. Frame to frame association and segmentation.

An example of two consecutive frames at times t and t+1. The segmentation of the cell at time t (red) may not match the true segmentation at time t+1 due to the cell’s dynamics. Translation of the contour, from frame t, by using cell motion estimation (magenta) allows better alignment. The final cell segmentation (green) can be calculated based both on the estimated contour and frame t+1.

One of the key ideas is the extension of the commonly-used Kalman state vector to account for shape fluctuations for dynamic shape modeling (DSM). Shape inference requires a probabilistic modeling of cell morphology, which is not mathematically trivial. We address this challenge by applying a sigmoid function to the signed distance function (SDF) of the cell boundaries such that the slope of the sigmoid models the shape uncertainty and defines a prior probability for each cell. Given the estimated cell poses, shape models and velocity maps, that are generated from the observed image measurements, we calculate the likelihood maps of each cell via a fast marching (FM) algorithm. Using the prior probabilities and the likelihood maps, we calculate the posterior probabilities of the image pixels. The partitioning of the image into individual cells and background is defined by the MAP estimates.

The proposed method is mathematically elegant and robust, with just a few parameters to tune. The algorithm has numerous advantages. A main contribution is the DSM, which serves as a prior for the consecutive frame segmentation without imposing any predetermined assumptions on cell shape. In contrast to existing approaches (Türetken et al., 2017; Amat et al., 2014; Türetken et al., 2016), which explicitly or implicitly assume ellipsoidal structure, the proposed algorithm can handle non-convex cell shapes. Consider for example the cells’ shapes in Figure 5, which exhibit significant irregularities. Furthermore, introducing the boundary uncertainty estimate to the shape model makes our algorithm robust against temporal, morphological fluctuations. In addition, estimating the cell temporal dynamics facilitates accurate frame-to-frame association, particularly in the presence of highly cluttered assays, rapid cell movements, or sequences with low frame rate. We note that mitotic events (i.e., cell divisions) significantly complicate cell tracking. We address this issue by initiating tracks for the daughter cells based on the MAP segmentation.

Figure 5. Visual Results.

Selected frames presenting full views (odd rows) and zoom-in (even rows) of the results obtained for three of Cell Tracking Challenge data sets: Fluo-C2DL-MSC (rows 1-2), Fluo-N2DH-SIM+ (rows 3-4), and Fluo-N2DH-GOWT (rows 5-6). For links to full videos refer to Table 7.

We demonstrate the proposed method both quantitatively and qualitatively on several data sets of different cell types acquired in a variety of laboratory and imaging settings including the Cell Tracking Challenge (Maška et al., 2014). The results show that the method is capable of robustly handling both segmentation of cells with irregular shapes and tracking of long sequences (hundreds of frames). We note that for the Cell Tracking Challenge - Fluo-C2DL-MSC data set, our method was ranked the first in all three categories (Tracking, Segmentation, and combined score) by the challenge organizers .

The code is freely available at https://github.com/arbellea/CellTrackingAndSegmentationPublic.git and a compiled version is available at https://github.com/arbellea/CellTrackingAndSegmentationCompiled.git. Access to the RPE data set will be given upon request. The current version of the code is implemented for 2D data sets.

This paper is an extension of our preliminary work Arbelle et al. (2015) with a more general mathematical formulation and a variety of challenging data sets.

The rest of the paper is organized as follows. In Section 2 we introduce the proposed cell tracking and segmentation approach, which consists of four main components: Section 2.3 discusses the time series analysis for motion and shape uncertainty estimation; Section 2.4 defines the probabilistic DSM based on previous frame segmentation and the estimated boundary uncertainty; Section 2.5 utilizes the FM algorithm for the calculation of the likelihood; Section 2.6 presents the MAP estimation of the final segmentation and association. Section 3 presents experimental results for eight different datasets. We conclude and outline future directions in Section 4.

2. Methods

2.1. Problem Formulation

In the following section, we aim to define the association and segmentation process by partitioning the image to a labeled set such that each cell has a unique label for the duration of the entire sequence. Let $\mathcal{C}=\left\{C^{(1)},\dots ,C^{(K)}\right\}$ denote K cells in a time lapse microscopy sequence, containing $\mathcal{T}$ frames. We define Ω ⊂ ℝ² as the image domain. Let $\mathrm{ℐ}:\mathrm{\Omega }\to \mathrm{ℝ}$ be a random variable with probability function $P_{\mathrm{ℐ}}$. Let I_t be the observed values of t’th frame in that sequence, t = 1, …, $\mathcal{T}$. We assume that each I_t is a gray-level image of ${\mathcal{K}}_t$ cells, that form a subset of $\mathcal{C}$. Our objective is twofold and consists of both cell segmentation and frame-to-frame cell association defined as follows:

Segmentation

For every frame I_t, find a function f_t : Ω → L_t, (where L_t is a subset of ${\mathcal{K}}_t+1$ integers in [0, …, K]) that assigns a label l_t ∈ L_t to each pixel x = [x, y] ∈ Ω. The function f_t partitions the t’th frame into ${\mathcal{K}}_t$ regions, where each segment corresponding to a cell ${\mathrm{\Gamma }}_t^{(k)}=\{\mathbf{x}\in \mathrm{\Omega }|f_t(\mathbf{x})=l_t=k\}$ forms a connected component of pixels, in frame t. The background, i.e., ${\mathrm{\Gamma }}_t^{(0)}$, includes all non-cell pixels and is not limited to a single connected component. Note that ${\cup }_{k=0}^{{\mathcal{K}}_t}{\mathrm{\Gamma }}_t^{(k)}=\mathrm{\Omega }$ and ${\mathrm{\Gamma }}_t^{(i)}\cap {\mathrm{\Gamma }}_t^{(j)}=0$, ∀i ≠ j.

Association

For every frame I_t, find an injective function h_t : L_t−1 → L_t that corresponds to cell segments in frame t − 1 and frame t. As we will show in the following, the segmentation and association steps are merged and ${\mathrm{\Gamma }}_t^{(k)}$, k ≥ 1 defines the segmentation of cell C^(k) in frame t. We assume that each cell is represented by a state vector ${\xi }_t^{(k)}$ (including the location, velocity, and shape uncertainty of the cell).

2.2. Probabilistic Model

The proposed method for joint cell segmentation and association is based on a graphical model presented in Figure 2. Shaded circles represent the observed variables I_t−1,I_t and I_t+1. The non-shaded circles represent the unknown variables: cell segmentations ${\mathbf{\Gamma }}_{\mathbf{t}}={\left\{{\mathrm{\Gamma }}_t^{(k)}\right\}}_{k=0}^{K_t}$ and cell state vectors ${\mathbf{\xi }}_{\mathbf{t}}={\left\{{\xi }_t^{(k)}\right\}}_{k=0}^{K_t}$. Given a sequence of microscopy videos as input, the proposed algorithm’s output is the segmented sequence, where each cell is assigned a unique label. Our goal is to find the partitioning Γ_t given I_t and information from previous frames. We do not require cell shape to be elliptical or otherwise convex. Our only assumption, related to cell topology, is that its segmentation, i.e., ${\mathrm{\Gamma }}_t^{(k)}$ is represented as a single connected component. Figure 3 presents the flow of the proposed algorithm, to be detailed below, using a single representative cell. For every cell C^(k) there exist a number of properties that describe its state at a given time t. Let ${\xi }_t^{(k)}\in {\mathrm{ℝ}}^F$, Eq. (1), denote the hidden state vector that holds the true, unknown, state of the cell comprised of F features. In our case the state vector holds the following features:

$${\xi }_t^{(k)}={\left[c_{x_t}^{(k)},c_{y_t}^{(k)},v_{x_t}^{(k)},v_{y_t}^{(k)},{\epsilon }_t^{(k)}\right]}^T={\left[{\mathbf{c}}_t^{(k)T},{\mathbf{v}}_t^{(k)T},{\mathbf{\epsilon }}_t^{(k)}\right]}^T$$ (1)

where, ${{\mathbf{c}}^{(\mathbf{k})}}_t={\left[c_{x_t}^{(k)},c_{y_t}^{(k)}\right]}^T$ denote the center of mass(COM) of the cell at time t and ${{\mathbf{v}}^{(\mathbf{k})}}_t={\left[v_{x_t}^{(k)},v_{y_t}^{(k)}\right]}^T$ denote the COM velocities. In addition, the commonly used state vector is extended to include a shape uncertainty variable, denoted by ${\epsilon }_t^{(k)}$, which will be explained in Section 2.4. We note that although we deal with planar cells, the extension to 3D is straightforward and only requires adaptation of the state-vector, e.g., using 3D coordinates for the calculation of cell COM and velocity.

Figure 2. Graphical Model.

associated with the proposed algorithm. Shaded circles represent the observed variables I_t−1,I_t and I_t+1. The non-shaded circles represent the unknown variables: cell segmentations Γ_t and cell state vectors ξ_t. The dashed line connecting Γ_t−1 and Γ_t (as well as Γ_t−1 and Γ_t+1) indicates that Γ_t−1 (Γ_t), which is estimated in the previous frame, is no longer considered an unknown random variable

Figure 3. Segmentation flow of a specific cell.

(a) Original image. The estimated COM of the specific cell k is marked by a red cross. (b) DSM (Spatial prior probability). (c) Intensity probability of the foreground $P_t^{(FG)}$. (d) Traversability image g (∇_xI_t). (e) Speed image ${\widehat{S}}_{t|t-1}^{(k)}$, the product of (c–d). (f) FM distance. (g) Likelihood. (h) Posterior. (i) MAP-based labeled segmentation.

In the following we present the probabilistic modelling for the assignment of a specific pixel to a specific cell. Although each pixel is treated independently, the spatial relations are indirectly introduced into the prior and likelihood as will be explained in Sections 2.4 and 2.5 respectively. Let ${\mathrm{\Theta }}_{t|t}^{(k)}(\mathbf{x})$ denote the probability of $x\in {\mathrm{\Gamma }}_t^{(k)}$ given the current frame and all relevant information from previous frames:

$$\begin{gathered} {\mathrm{\Theta }}_{t|t}^{(k)}(\mathbf{x})=P\left(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|I_t,{\mathbf{\Gamma }}_{\mathbf{0}},{\mathbf{\xi }}_{\mathbf{0}},\dots ,{\mathbf{\Gamma }}_{\mathbf{t}-1},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right) \\ =P\left(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|I_t,{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right) \end{gathered}$$ (2)

Note the subscript t|t implies that the current time step and history are taken into account. Using Bayes theorem we get:

$$P\left(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|I_t,{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)=\frac{P\left(I_t|\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)},{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)P\left(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)}{P\left(I_t|{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)}.$$ (3)

We denote the prior probability that $\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}$ given only history with the subscript t|t − 1:

$${\mathrm{\Phi }}_{t|t-1}^{(k)}(\mathbf{x})=P\left(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)$$ (4)

Here, we assume that Γ_t−1, that was estimated from the previous frame, is known and is no longer considered an unknown random variable. The image likelihood is denoted by:

$$L_{t|t}(\mathbf{x})=P\left(I_t|\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)},{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)=P\left(I_t|\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}\right)$$ (5)

Since P (I_t|Γ_t−1, ξ_t−1) is not a function of k or x we refer to it as a normalization constant:

$$\beta =\frac{1}{P(I_t|{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}})}$$ (6)

Substituting Eqs. (5,4,6) into Eq. (3) we get a compact notation:

$${\mathrm{\Theta }}_{t|t}^{(k)}(\mathbf{x})=\beta {\mathrm{\Phi }}_{t|t-1}^{(k)}(\mathbf{x})L_{t|t}(\mathbf{x})$$ (7)

The final segmentation is then defined by the maximum a posteriori estimator (MAP):

$${\mathrm{ℒ}}_t(\mathbf{x})=\arg \underset{k\in {\mathcal{K}}_t}{max}\left({\mathrm{\Theta }}_{t|t}^{(k)}(\mathbf{x})\right)=\arg \underset{k\in {\mathcal{K}}_t}{max}\left({\varphi }_{t|t-1}^{(k)}(\mathbf{x})L_{t|t}(\mathbf{x})\right)$$ (8)

and the partitioning is given by:

$${\widehat{\mathrm{\Gamma }}}_t^{(k)}=\left\{\mathbf{x}\right|{\mathrm{ℒ}}_t(\mathbf{x})=k\}$$ (9)

2.3. Time series analysis

In the following discussion the superscript (k) is removed for clarity. Let Q_t ∈ ℝ^F×F, R_t ∈ ℝ^O×F be known covariance matrices and w_t, r_t ∈ ℝ^F be random variables drawn from $\mathcal{N}(\mathbf{0},{\mathrm{Q}}_t)$ and $\mathcal{N}(\mathbf{0},{\mathrm{R}}_t)$respectively. We refer to w_t and r_t as the process noise and measurement noise, respectively. Let A ∈ ℝ^F×F denote the state transition model. We assume that the state vector approximately follows a linear time step evolution:

$${\xi }_t=\mathrm{A}{\xi }_{t-1}+w_t$$ (10)

In order to predict the state of a cell at time t we adopt the equations of the Kalman Filter (Kalman, 1960). The predicted (a priori) state vector estimation and error covariance matrix at time t given measurements up to time t − 1 are:

$$\begin{gathered} {\widehat{\xi }}_{t|t-1}=\mathrm{A}{\widehat{\xi }}_{t-1|t-1} \\ {\sum }_{t|t-1}=\mathrm{A}{\sum }_{t-1|t-1}{\mathrm{A}}^T+{\mathrm{Q}}_t \end{gathered}$$ (11)

Figure 3.a shows an estimation of the COM component of ${\widehat{\xi }}_{t|t-1}$ superimposed on I_t marked by a red cross. The estimated segmentation of a cell C^(k) in frame t, i.e., ${\widehat{\mathrm{\Gamma }}}_{t|t-1}^{(k)}$ is obtained by a translation of the cell segmentation in frame t − 1 : $t-1:{\widehat{\mathrm{\Gamma }}}_{t|t-1}^{(k)}=\left\{\mathbf{x}\right|(\mathbf{x}-{\widehat{\mathbf{v}}}_{t|t-1}^{(k)}dt)\in {\mathrm{\Gamma }}_{t-1}^{(k)}\}$, where ${\widehat{\mathbf{v}}}_{t|t-1}^{(k)}dt$ is the estimated cell displacement. The importance of the cell displacement estimation is illustrated in Figure 1. Since the true state is hidden, the observed state ζ_t ∈ R^O, where O is the number of observed variables and B ∈ R^O×F is the observation matrix, is modeled as:

$${\zeta }_t=\mathrm{B}{\xi }_t+r_t$$ (12)

The state of the cell, ζ_t, is calculated once the segmentation is complete (Section 2.6). This includes the measured COM and ε_t, which also requires the segmentation of the previous frame.

Let G_t = Σ_t|t−1B^T (BΣ_t|t−1B^T + R_t)⁻¹ define the Kalman Gain matrix. The a posteriori state estimate and error covariance matrix at time t given measurements up to and including time t are:

$$\begin{gathered} {\widehat{\xi }}_{t|t}=\mathrm{A}{\widehat{\xi }}_{t|t-1}+G_t({\zeta }_t-\mathrm{B}{\widehat{\xi }}_{t|t-1}) \\ {\sum }_{t|t}=(\mathrm{I}-G_t\mathrm{B}){\sum }_{t|t-1} \end{gathered}$$ (13)

2.4. Prior Probability - Dynamic Shape Model (DSM)

The main concepts of the DSM are illustrated in Figure 4. Let ${\widehat{\varphi }}_{t|t-1}^{(k)}|\mathrm{\Omega }\to \mathrm{ℝ}$ define the signed distance function (SDF) and is constructed as follows:

$${\widehat{\varphi }}_{t|t-1}^{(k)}(\mathbf{x})=\{\begin{array}{cc}{min}_{{\mathbf{x}}^{\prime }\in \partial {\widehat{\mathrm{\Gamma }}}_{t|t-1}^{(k)}}{d}_E(\mathbf{x},{\mathbf{x}}^{\prime })& \mathbf{x}\in {\widehat{\mathrm{\Gamma }}}_{t|t-1}^{(k)}\\ -{min}_{{\mathbf{x}}^{\prime }\in \partial {\widehat{\mathrm{\Gamma }}}_{t|t-1}^{(k)}}{d}_E(\mathbf{x},{\mathbf{x}}^{\prime })& \mathbf{x}\notin {\widehat{\mathrm{\Gamma }}}_{t|t-1}^{(k)}\end{array}$$ (14)

where d_E (·, ·) denotes the Euclidian distance and $\partial {\widehat{\mathrm{\Gamma }}}_{t|t-1}$ denotes the estimated segmentation boundary. Figure 4.a shows two pairs of contours with different shape variations. The top cell varies greatly while the bottom does not. Figure 4.b is an overlap of the two contours. Figure 4.c, visualizes the SDF relative to the contour at time t. In the spirit of (Riklin-Raviv et al., 2010; Pohl et al., 2006; Bishop, 2006) we define the probability that a pixel x belongs to the domain of cell k by the logistic regression function:

$${\widehat{\mathbf{\Phi }}}_{t|t-1}^{(k)}(\mathbf{x})=P(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}},{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}})\triangleq {\left(1+\exp \left(-\frac{{\widehat{\varphi }}_{t|t-1}^{(k)}(\mathbf{x})}{{\widehat{\epsilon }}_{t|t-1}^{(k)}}\right)\right)}^{-1}$$ (15)

where, ${\widehat{\epsilon }}_{t|t-1}^{(k)}$ is the estimation of ${\epsilon }_t^{(k)}$, which determines the slope of the logistic sigmoid. The slope should be indicative of the uncertainty of the k-th cell outline; however, as this value cannot be calculated, we set ${\epsilon }_t^{(k)}$ to be proportional to the difference between the boundaries of the cell in two consecutive frames, (refer to the pink region in Figure 4.b). We chose to measure this difference using the Modified Hausdorff Distance (MHD) (Dubuisson and Jain, 1994), denoted by d_MHD (·). Recall that the d_MHD () of two sets $\mathcal{X}$ and $\mathcal{Y}$ of cardinality $N_{\mathcal{X}}$ and $N_{\mathcal{Y}}$, respectively, is defined as:

$$d_{\mathit{MHD}}(\mathcal{X},\mathcal{Y})=\frac{1}{2}\left(\frac{1}{N_{\mathcal{X}}}{\displaystyle \sum _{\mathbf{x}\in \mathcal{X}}\underset{\mathbf{y}\in \mathcal{Y}}{min}({\Vert \mathbf{x}-\mathbf{y}\Vert }_2)}+\frac{1}{N_{\mathcal{Y}}}{\displaystyle \sum _{\mathbf{y}\in \mathcal{Y}}\underset{\mathbf{x}\in \mathcal{X}}{min}({\Vert \mathbf{x}-\mathbf{y}\Vert }_2)}\right),$$ (16)

where ‖·‖₂ denotes the L₂ norm. We can equivalently consider the expression in Eq. (15) as the zero-mean cumulative distribution function of the logistic distribution, where ${\epsilon }_t^{(k)}$ is proportional to the standard deviation. We therefore set ${\epsilon }_t^{(k)}$ as follows:

$${\epsilon }_t^{(k)}\triangleq d_{\mathit{MHD}}\left(\partial {\mathrm{\Gamma }}_{t-1}^{(k)},\partial {\mathrm{\Gamma }}_t^{(k)}\right)\frac{\sqrt{3}}{2\pi }$$ (17)

Note, as can be seen in Figure 4.d–e, large temporal shape fluctuations increase d_MHD, which in turn increases the slope of the logistic regression function and the uncertainty in cell boundaries. Further explanation of the choice of ${\epsilon }_t^{(k)}$ is available in Appendix A. Eq. 15 defines the proposed DSM, which is the prior probability that a pixel belongs to the cell. Figure 3.b shows an example of a prior probability, ${\widehat{\mathrm{\Phi }}}_{t|t-1}^{(k)}$, for a given cell.

Figure 4. DSM construction from two consecutive instances of a cell.

Each row refers to a different cell. (a) Raw data depicting two instances of the same cell in frames t − 1 and t. (b) The associated contours of the two instances (red and light blue) superimposed. The area of mismatch (pink) corresponds to ε. Note that ε is larger (upper row) in the presence of significant shape fluctuations. (c) The signed distance function (SDF) associated with the cell instance in frame t. The SDF’s zero level is defined by the cell’s contour (light blue). (d) A logistic regression function $1/\left(1+\mathit{exp}\left(-\frac{\mathit{SDF}}{\epsilon }\right)\right)$. (e) The DSM, which represents the cell’s probability map. The smoother function of the cell shape in the upper row reflects high uncertainty.

2.5. Likelihood

We now present the calculation of the Likelihood of the proposed segmentation algorithm given the state vector estimation ${\widehat{\xi }}_{t|t-1}$ and cell segmentation of the previous frame. The modeling of cell (foreground) and background intensity distributions, ${\widehat{f}}_{FG}(\cdot )$ and ${\widehat{f}}_{BG}(\cdot )$, respectively, can be done via several different methods. We suggest either using Gaussian Mixture Model (GMM) as long as the data fits the GMM assumptions (as proposed by Arbelle et al. (2015)) or in more complicated cases, when the underlying PDF is unknown, the Kernel Density Estimation (KDE) (Rosenblatt et al., 1956; Parzen, 1962) can be applied.

The intensity-based probability of being a cell or background (Figure 3.c) is defined using the estimated PDFs as follows:

$$P_t^{(BG)}(\mathbf{x})=\frac{\alpha {\widehat{f}}_{BG}(I_t(\mathbf{x}))}{\alpha {\widehat{f}}_{BG}(I_t(\mathbf{x}))+(1-\alpha ){\widehat{f}}_{FG}(I_t(\mathbf{x}))};P_t^{(FG)}(\mathbf{x})=1-P_t^{(BG)}(\mathbf{x})$$ (18)

where 0 < α < 1 is a predetermined weight. The robustness of the parameter α is examined in Appendix B

We use the FM algorithm (Hassouna and Farag, 2007) to find the shortest path from each pixel x to the estimated COM of a cell k s.t. a speed image ${\widehat{S}}_{t|t-1}^{(k)}:\mathrm{\Omega }\to [0,1]$ The FM distance, $d_{FM}\left(\mathbf{x},{\widehat{\mathbf{c}}}_{t|t-1}^{(k)}|{\widehat{S}}_{t|t-1}^{(k)}\right)$, is the minimal geodesic distance from x to ${\widehat{\mathbf{c}}}_{t|t-1}^{(k)}$. In other words, the value ${\widehat{S}}_{t|t-1}^{(k)}(\mathbf{x})$ is the speed of a pixel x along the shortest path to ${\widehat{\mathbf{c}}}_{t|t-1}^{(k)}$. For each pixel x in frame t we define its speed ${\widehat{S}}_{t|t-1}^{(k)}(\mathbf{x})$ as the product of two terms: 1. The intensity-based probability of belonging to the foreground (Eq. (18)). 2. The “traversability” (Figure 3.d), which is in inverse proportion to the image edges in frame I_t, is defined by $g(\left|{\nabla }_{\mathbf{x}}{I}_t\right|)={\left(1+\frac{\left|{\nabla }_{\mathbf{x}}{I}_t\right|}{{\sigma }_{\mathit{grad}}}\right)}^{-2}$ where $\left|{\nabla }_{\mathbf{x}}{I}_t\right|=\sqrt{{\left(\frac{\partial }{{\partial }_x}{I}_t\right)}^2+{\left(\frac{\partial }{{\partial }_x}{I}_t\right)}^2}$ defined and σ_grad is the standard deviation of all values in |∇_xI_t|. The speed as:

$${\widehat{S}}_{t|t-1}^{(k)}=\log \left(1-P_t^{(FG)}·g(\left|{\nabla }_{\mathbf{x}}{I}_t\right|)\right)$$ (19)

The absolute value of the spatial gradient, i.e., |∇_xI_t|, can be interpreted as “speed bumps” which make the “FM journey” more difficult across edges. An example of the speed image and the final FM distance can be seen in Figure 3.e and Figure 3.f, respectively.

Let $d_{FM}\left(\mathbf{x},{\widehat{\mathbf{c}}}_{t|t-1}^{(k)}|{\widehat{S}}_{t|t-1}^{(k)},\right)$ define the FM distance from every pixel to the estimated cell center. The likelihood of x given C_k (Figure 3.g) can be defined as:

$$L_{t|t}(C_k|\mathbf{x})={\left(d_{FM}\left(\mathbf{x},{\widehat{\mathbf{c}}}_{t|t-1}^{(k)}|{\widehat{S}}_{t|t-1}^{(k)},\right)+1\right)}^{-1}$$ (20)

Note that 0 ≤ L_t|t ≤ 1 and that L_t|t = 1 iff d_FM = 0.

2.6. MAP Segmentation and Association

The Posterior Probability for each pixel to belong to a cell is given as the normalized product of the Prior (Eq. 15) and the Likelihood (Eq. 20) as defined in Eq. (7) (see Figure 3.h). The final segmentation of the image is then given as the MAP estimation as defined in Eq. (8) (see Figure 3.i). In fact, we see that cell association is inherent to the defined segmentation problem, since each cell is segmented using its estimated properties from the previous frame. Given the final segmentation, we calculate the state vector ζ_t and apply the Kalman correction given in Eq. (13) to obtain the estimated state vector of the current frame. Note that since the measurements are dependent on the segmentation, the method is limited to a forward pass on the sequence.

2.7. New Track Detection

We look for new tracks once the MAP estimation is completed, and the image pixels are labelled. New cell tracks can be initiated either as a result of mitosis or entrance to the frame’s field of view We refer to a mitotic event when two or more connected components are associated with the same cell label. In which case the track of the mother cell is terminated and a new track is initiated for each connected component. In addition we look for pixels that were labelled as background but satisfy:

$${\mathrm{\Gamma }}_t^{(\mathit{New})}=\left\{\mathbf{x}|(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(0)})\mathrm{\Lambda }(P_t^{(FG)}(\mathbf{x})>0.5)\right\}$$ (21)

A new cell is detected for each connected component in the region extracted with size within the range [T_{min−cell−size}, T_{max−cell−size}].

3. Experimental Results

3.1. Experimental Setup

Initialization

The first two frames of each data set were manually annotated using the initialization tool described in Appendix C, which is made freely available. We, however, note that any other utility may be used to create the initial segmentations, e.g., ImageJ (Schindelin et al., 2015) or Ilastik (Sommer et al., 2011). Table 1 shows the percentage of manually annotated cells with respect to each of the full length sequences. Note that the method requires annotation of the first two frames regardless of sequence length, number of cells, type of cells, or number of mitotic events. This amounts to a very low percentage of the total number of cells within the sequence, especially in long and dense data sets such as the RPE set, where the manual annotations amount to 0.31% of the total number of detected. Furthermore, the initial segmentation allows us to accurately estimate the very few parameters used in the proposed framework. These include the minimum and maximum cell sizes, T_{min−cell−size} and T_{max−cell−size}, respectively; the initial parameters state vector for each cell, including the location, velocity, and shape uncertainty variables; and the initial foreground and background intensity distributions, ${\widehat{f}}_{FG}$ and ${\widehat{f}}_{BG}$, respectively. These manual annotations eliminate the need for parameter tuning, which requires technical expertise. We also note that the parameters extracted from the initialization are valid for an entire sequence, regardless of its length and therefore can be useful to other sequences acquired under the same conditions. Appendix D shows an example of such a scenario.

Table 1. Manual Annotation.

The percentage of manually annotated cells (within the first two frames) with respect to the total number of cells detected by the proposed method in the entire sequence.

Data	#Man. Annotated	#Total Detected	Percentage	#Frames
Fluo-C2DL-MSC	24	767	3.13%	48
Fluo-N2DH-SIM+	60	5167	1.16%	110
Fluo-N2DH-GOWT1	64	2987	2.14%	92
H1299	84	3623	2.31%	72
MCF-10A	84	8009	1.05%	141
RPE	56	180881	0.31%	400

Parameters

The following parameters were defined for all data sets: The covariance matrices Q_t, R_t (see Section 2.3) are set as the identity matrix. The transition matrices A and B (see section 2.3) are defined as follows: A_i,i = 1, for i = 1 … 5; A₁,₃ = A₂,₄ = 1. B_i,i = 1 for i ∈ [1, 2, 5] and otherwise 0. Likelihood: The parameter α (defined in Eq. (18)), is set to 0.5.

Pseudocode

For all experiments we followed Algorithm 1.

3.2. Evaluation Methods

We evaluated the method using the scheme proposed in the online version of the Cell Tracking Challenge. Specifically, SEG for segmentation, as defined in Maška et al. (2014); Ulman et al. (2017), and TRA for tracking. TRA uses the Acyclic Oriented Graph Matching (AOGM), as defined in Matula et al. (2015), which assesses how difficult it is to transform a computed graph into a given ground-truth graph. The TRA measure is defined as follows:

$$\mathit{TRA}=1-min\{\mathit{AOGM},{\mathit{AOGM}}_0\}/{\mathit{AOGM}}_0$$

where AOGM₀ is the AOGM value required for creating the reference graph from scratch (i.e., it is the AOGM value for empty tracking results). OP is defined as the mean of TRA and SEG. We submitted our results on three of the Cell Tracking Challenge competition sequences and received the TRA, SEG and OP scores from the challenge organizers. We also evaluated several additional data sets using the same measures. Furthermore, we evaluated the tracking results using two measures suggested by Kan et al. (2011), P_track, representing the percentage of correctly detected tracks, and P_links, representing the percentage of correct frame to frame associations.

Algorithm 1.

Pseudocode.

Require: Initial Segmentation of first two frames,
1:	for k ≤ Kt=1 do
2:	Calculate the observed state vector for cell k, ${\zeta }_{t=1}^{(k)}$
3:	end for
4:	for 2 ≤ t ≤ T do
5:	for k ≤ K do
6:	Estimate the Kalman state vector, ${\widehat{\xi }}_{t|t-1}^{(k)}$ given ${\xi }_{t-1|t-1}^{(k)}$ as defined in Eq. (11)
7:	Calculate prior pixel probability (DSM), ${\mathrm{\Phi }}_{t|t-1}(\mathbf{x}|C_k)$ given ${\widehat{\xi }}_t^{(k)}$ as defined in Eq. (15)
8:	Calculate the likelihood, Lt(Ck|x) given the frame It and ${\widehat{\xi }}_t^{(k)}$ as defined in Eq. (20)
9:	Calculate posterior probabilities Pt (Ck|x) as defined in Eq. (7)
10:	Apply MAP estimator to get cell segmentation, ${\mathrm{\Gamma }}_t^{(k)}$ as defined in Eq. (8)
11:	Detect new cells, ${\mathrm{\Gamma }}_t^{(\mathit{New})}$ as defined in Eq. (21)
12:	Calculate ${\zeta }_t^{(k)}$ given ${\mathrm{\Gamma }}_t^{(k)}$
13:	Calculate the corrected state estimation ${\widehat{\xi }}_{t|t}$ using Eq. (13)
14:	end for
15:	end for

Table 5 shows the results on our data for all measures.

Table 5.

Quantitative results for three additional data sets. Columns 2-4 show the OP, SEG, and TRA measures as explained in Maška et al. (2014). The Plinks measure (percentage of correct matches between two consecutive frames is shown in column 5. The 6th column shows the P_tracks measure (Section 3.2). It is evident that P_tracks is significantly lower than the TRA, this is due to its sensitivity to single mistakes along the track. The P_tracks measure, however, better answers the needs of the intended biological experiment.

Data Set	OP	SEG	TRA	Plinks	Ptracks
H1299	0.786	0.687	0.90	0.993	0.79
RPE	—	0.846	—	0.989	0.69
MCF-10A	0.888	0.796	0.981	0.999	0.93

3.3. Experiments

We examined eight different live cell microscopy sequences: three from the Cell Tracking Challenge, two from our own datasets, two datasets of our colleagues, and an additional publicly available dataset (Rapoport et al., 2011). All experiments were conducted on an Intel i7 3.40GHz CPU with 16GB RAM. Processing time of five sequences is detailed in Table 2. Run-time varies between data sets and is dependent on the cells’ size, the number of cells per frame, and sequence length.

Table 2. Processing Time.

Run time of most data sets that were tested. Run time varies depending on the length of the sequence, the size of the cells, and the number of cells in each frame. All evaluations were conducted on an Intel i7 3.40GHz CPU with 16GB RAM.

Data	#Frames	Total Time [min]	Avg. Time per frame [sec]
Fluo-C2DL-MSC	48	53	76.5
Fluo-N2DH-SIM+	110	38.5	25.6
Fluo-N2DH-GOWT1	92	58	31.8
H1299	72	8	6.6
MCF-10A	141	36	15.3
RPE	400	2264.5	340

3.3.1. Cell Tracking Challenge data sets

We tested our method on three of the Cell Tracking Challenge data sets, namely Fluo-C2DL-MSC, Fluo-N2DH-GOWT1, and Fluo-N2DH-SIM+. Reported scores were calculated by the challenge organizers and published on the website under the label BGU-IL: www.celltrackingchallenge.net/latest-results.html. We note that, in contrast to all other data sets, the Fluo-C2DL-MSC cells exhibit highly irregular shapes as can be seen in Figure 5. In addition, the cells’ motion is relatively fast. Nevertheless, as shown by the results in Table 3, our tracking and segmentation method obtained the best scores for all measures (TRA, SEG, OP) by a significant margin. These results highlight one of the advantages of the method, which does not assume convexity or any other shape model. On the other hand, when cell shapes are elliptical, such as the Fluo-N2DH-GOWT1 and Fluo-N2DH-SIM+, our method ranks in the middle.

Table 3.

Results on Challenge Data Sets. The results of the current first to third ranking algorithms were taken from the Cell Tracking Challenge website. The Fluo-C2DL-MSC shows the strength of our method.

	Fluo-C2DL-MSC			Fluo-N2DH-GOWT1			Fluo-N2DH-SIM+
	OP	SEG	TRA	OP	SEG	TRA	OP	SEG	TRA
BGU-IL (Ours)	0.759	0.645	0.873	0.760	0.656	0.864	0.855	0.807	0.902
1st	0.686	0.582	0.800	0.878	0.791	0.975	0.951	0.927	0.976
2nd	0.636	0.574	0.691	0.874	0.781	0.957	0.901	0.893	0.925
3rd	0.546	0.465	0.645	0.858	0.770	0.948	0.901	0.887	0.916

3.3.2. Additional data sets

We tested the algorithm on five additional, high-throughput data sets, consisting of one sequence each, as listed in Table 4: (1) H1299 cells, expressing eYFP-DDX5 in the background of an mCherry tagged nuclear protein, rate: 3fph, 72 frames (Cohen et al., 2008) (Alon Lab, Weizmann Institute of Science). (2) Two data sets RPE cells, expressing eYFP-DDX5 in the background of a p21-mVenus tagged nuclear protein, rate: 4fph, 400 frames (Lahav Lab, Harvard Medical School). One of the RPE data sets along with its manual annotations is available upon request. MCF-10A cells, expressing RFP-Geminin and NLS-mCerulean, rate: 3fph, 142 frames (Brugge Lab, Harvard Medical School). See also (Rapoport et al., 2011) for the PSC data set. These data sets exhibit difficult challenges such as unclear boundaries (H1299), very long sequence with considerable motion (RPE), numerous mitotic events (MCF-10A) and dense cell population (PSC). Figure 6 shows examples from these experiments. Further, qualitative evaluations are shown in Figure 7. Figure 8 shows a visualization of the tracking results for the MCF-10A and H1299 data sets. For quantitative evaluation of three of these data sets, refer to Table 5, which shows the OP, SEG, and TRA measures, as other tracking measures. Results can be better evaluated when considering the length of each sequence as well as the tracks’ and cells’ sizes. Refer to Table 6 for this information. We note that although the Plinks measure seems very high (greater than 98%), the P_tracks drops drastically. Note that the RPE data set is especially long, 400 frames, and thus prone to errors that greatly affect P_tracks, produced an accuracy of 69% and still kept a relatively high SEG measure of 0.846. Refer to Table 7 for the full visualization videos of all data sets.

Table 4. Additional data sets.

Examples of the results for each data set can be seen in Figures 6–7.

Data Owner	Cell Type	#of Frames	Frame Dimensions	#Cells First-Last Frames
Brugge Lab Harvard Medical School	MCF-10A	142	501×400	42-80
Alon Lab Weizmann Institute of Science	H1299	72	640×511	41-69
Lahav Lab Harvard Medical School	RPE	564	1024×1024	67-92
Lahav Lab Harvard Medical School	RPE	433	1024×1023	31-49
Rapoport et al.	PSC	209	1376×1038	283-2344

Figure 6. Visual Results.

Selected frames presenting full views (odd rows) and zoom-in (even rows) of the results obtained for the MCF-10A data set (rows 1-2, courtesy of Brugge Lab, Harvard Medical School); H1299 data set (rows 3-4, courtesy of Alon Lab, Weizmann Institute of Science), and the RPE data set (rows 5-6, Lahav Lab, Harvard Medical School), respectively. Cells’ instances initiating new tracks (right after mitosis) are outlined in magenta. For links to full videos refer to Table 7.

Figure 7. Visual Results.

Selected frames presenting full views (odd rows) and zoom in (even rows) of the results obtained for the RPE data set (rows 1-2, Lahav Lab, Harvard Medical School) and the PSC data set (Rapoport et al., 2011) (rows 3-4). Cells’ instances initiating new tracks (right after mitosis) are outlined in magenta. For links to full videos refer to Table 7

Figure 8.

XYT plots of cell tracks; the horizontal axes represent the image plane, the vertical axis represents time. Each colored line represents a cell track. Refer to 3D tracks: https://youtu.be/YS5COOY3jeA for live 3D demonstration for all the tracking results presented at Figures 6–7.

Table 6.

Properties of additional data sets. Some visual results obtained for these data sets can be seen in Figure 6. Data owners: 1. Alon Lab, Weizmann Institute of Science. 2. Lahav Lab Harvard, Medical School. 3. Brugge Lab, Harvard Medical School. Quantitative results are shown in Table 5.

Cell Type	Sequence Length	Track Length (frames)			Cell Size (pixels)
		Mean	Min	Max	Mean	Min	Max
H12991	72	34	3	72	345	12	1084
RPE2	400	54	2	185	516	138	1055
MCF-10A3	141	50	2	141	649	87	1938

Table 7.

Results for all available data sets. Each data set points to two links, the first is a link to the video with the contours and labels of the cells marked throughout the sequence. The second link is a visualization of the traces of the cells in 3D space (see Figure 8), where the z-axis indicates the frame number. Data Owners: 1. Cell Tracking Challenge Maška et al. (2014); 2. Brugge Lab, Harvard Medical School; 3. Alon Lab, Weizmann Institute of Science; 4. Lahav Lab, Harvard Medical School; 5. Rapoport et al. (Rapoport et al., 2011)

Cell Type	Results Link	Results Link - Traces
Fluo-C2DL-MSC1	https://youtu.be/u30jSoZj62k	https://youtu.be/-UjynjlYB88
Fluo-N2DH-SIM+1	https://youtu.be/i3uPOxDQ8KA	https://youtu.be/DJrHMviF4uM
Fluo-N2DH-GOWT11	https://youtu.be/3KzPWOR2kSg	https://youtu.be/xjpU_SS4zNY
MCF-10A2	https://www.youtube.com/watch?v=hWXhNe2G7EY	https://www.youtube.com/watch?v=2cRTHSRSfYc
H12993	https://www.youtube.com/watch?v=mdB77zKbIHU	https://www.youtube.com/watch?v=KTfw365zgf8
RPE4	https://www.youtube.com/watch?v=mlYAYIcl1yc	https://www.youtube.com/watch?v=W2540XJ9WCc
RPE4	https://www.youtube.com/watch?v=CBYSNkeIHG8	https://www.youtube.com/watch?v=7mk5AzX3V5Q
PSC5	https://www.youtube.be/K5D2YjR58ic	https://www.youtube.com/watch?v=3R8RBjOLviQ

4. Summary and Conclusions

Cell segmentation and tracking are intimately related. In this work we demonstrated the strength of jointly solving these problems, where each supports the other, and together yield more accurate results. This was accomplished by our novel probabilistic formulation of cell-to-cell association and segmentation. By expanding the Kalman state vector to include the shape uncertainty and intensity levels, we were able to accurately estimate the dynamics of cells’ shape as well as their location and motion. The proposed contribution, termed the dynamic shape model (DSM), also allows accommodating versatile cell shapes, and serves as a prior in our model. We also introduced a unique view of the FM distance and use it to construct the likelihood of the pixel segmentation. The prior, from the DSM, and the likelihood, from the FM, constructed the posterior probabilities. By applying the commonly used MAP estimator, we obtained the final segmentation and association of the cells at each frame.

Our method was tested on several different data sets. These include three Cell Tracking Challenge (Maška et al., 2014) data sets and five additional data sets acquired in a variety of ways and in a variety of laboratory settings. The results obtained demonstrate the ability of the proposed method to handle long sequences (hundreds of frames) in an elegant and robust manner. Qualitative results (see link to videos in Table 7) and quantitative comparisons demonstrate that our method outperforms the state of the art, where the cells follow no apparent shape assumption (Table 3). It should be noted that we tested our method using a very strict full track measure in addition to the commonly used evaluation metrics. This measure, which cannot tolerate even a single error within a cell’s track, is more suitable to cell lineage construction. Finally, recognizing the substantial challenges in the analysis of high-throughput microscopy imaging, we believe that the key concepts introduced here have great potential for a wide variety of biological experiments.

Future work will aim to complete the missing link for cell lineage reconstruction, and focus on mitosis detection in the spirit of Gilad et al. (2015).

A compiled version of the method and a utility for the annotation of the first two frames can be downloaded from: https://github.com/arbellea/CellTracking AndSegmentationCompiled.git and https://github.com/arbellea/ManualAnnotationTool.

Highlights.

• We propose an unsupervised, automatic tracking and segmentation framework for high-throughput microscopy image sequences.

• Cell segmentation and tracking are tied together via Bayesian inference of dynamic models.

• The Kalman inference problem is exploited to estimate the time-wise cell shape uncertainty in addition to cell trajectory. The inferred cell properties are integrated with the observed image, using a fast marching algorithm, to obtain the image likelihood for cell segmentation and association.

• We present highly accurate results, surpassing the state of the art, for a variety of microscopy data sets with high dynamics, including long sequences (hundreds of frames).

Appendices

Appendix A. Shape Uncertainty

The modified Hausdorff distance, d_MHD, calculates a mean measure of distances between the two contours from frames t and t − 1. The fluctuation around the midline between the contours is thus $\frac{1}{2}{d}_{\mathit{MDH}}$. We assume that the fluctuations in each direction, inside and outside of the contour, in frame t + 1 will also be in the magnitude of $\frac{1}{2}{d}_{\mathit{MHD}}$. We set ε such that at pixels, where the signed distance from the contour is ${\widehat{\varphi }}_{t|t-1}^{(k)}(\mathbf{x})=\pm \frac{1}{2}{d}_{\mathit{MHD}}$ the value of the prior would be $P\left(\mathbf{x}\in {\mathrm{\Gamma }}_t^{(k)}|{\mathbf{\Gamma }}_{\mathbf{t}-\mathbf{1}}{\mathbf{\xi }}_{\mathbf{t}-\mathbf{1}}\right)\triangleq {\left(1+\exp \left(\mp \frac{\frac{1}{2}{d}_{MH}}{\frac{\sqrt{3}}{2\pi }{d}_{MH}}\right)\right)}^{-1}={\left(1+\exp \left(\mp \frac{\pi }{\sqrt{3}}\right)\right)}^{-1}=0.5\pm 0.36$. This allows the prior to accommodate reasonable fluctuations with respect to d_MH. Note that the cumulative distribution function of a logistic distribution with zero mean and standard deviation σ is defined as: $P_{\mathit{log}}(x)={\left(1+\exp \left(\frac{\pi x}{\sqrt{3}\sigma }\right)\right)}^{-1}$. Thus, defining ${\epsilon }_t^{(k)}\triangleq \frac{\sqrt{3}}{2\pi }{d}_{MH}$ is equivalent to setting $\sigma =\frac{d_{MH}}{2}$.

Appendix B. Parameter Robustness

In order to evaluate the robustness of parameter α from Section 2.5 we examined the TRA and SEG measures of the Cell Tracking Challenge (Maška et al., 2014) training sequences. We changed the parameter in the range (0, 1) and ran the complete sequence. Figure B.9 shows a wide range of equally good selections of parameter α (defined in Sec.2.5 which denotes the weight of the foreground vs. background) demonstrating the method’s robustness to the choice of parameter.

Figure B.9.

SEG and TRA measures for the Cell Tracking Challenge data sets as a function of α. The wide plateau shows the robustness of the method to the selection of α.

Appendix C. Initialization Tool

As our method requires the manual annotation of the first two frames, we present a utility for easy annotation. The utility can be freely downloaded from https://github.com/arbellea/ManualAnnotationTool. The user can easily annotate two frames using the mouse and keyboard shortcuts to zoom in and out of the frame, move the frame to all sides, clear and correct annotations. The utility also supports the option to save intermediate results that can be later loaded for correction or continued annotation. Instruction for use and all the keyboard shortcuts can be found in the README.txt file along with the utility.

Appendix D. Sequence To Sequence Initialization

Our method currently requires the manual annotation of the first two frames of each sequence. The initial segmentation allows us to accurately estimate the very few parameters used in the proposed framework. However, we believe that this requirement can be relaxed when an initialization of a similar sequence, under similar conditions, is available. We conducted the following experiment using the Cell Tracking Challenge Fluo-N2DH-SIM+ data set consisting of both train (with available ground truth) and test sequences. We first ran the method on the training sequence and evaluated the SEG and TRA results as a baseline, 0.8431 and 0.9642 respectively. We then ran the method on the training sequence again, with initialization from the test sequence and compared the results. The results show a minor degradation with SEG and TRA values 0.8347 (−0.0084) and 0.9605 (−0.0037) respectively. The minor difference in the results shows the method’s robustness to initialization parameters and may indicate that a similar process will reduce the need for manual annotations for similar sequences.