DMNet: Dual-Stream Marker Guided Deep Network for Dense Cell Segmentation and Lineage Tracking

Abstract

Accurate segmentation and tracking of cells in microscopy image sequences is extremely beneficial in clinical diagnostic applications and biomedical research. A continuing challenge is the segmentation of dense touching cells and deforming cells with indistinct boundaries, in low signal-to-noise-ratio images. In this paper, we present a dual-stream marker-guided network (DMNet) for segmentation of touching cells in microscopy videos of many cell types. DMNet uses an explicit cell marker-detection stream, with a separate mask-prediction stream using a distance map penalty function, which enables supervised training to focus attention on touching and nearby cells. For multi-object cell tracking we use M2Track tracking-by-detection approach with multi-step data association. Our M2Track with mask overlap includes short term track-to-cell association followed by track-to-track association to re-link tracklets with missing segmentation masks over a short sequence of frames. Our combined detection, segmentation and tracking algorithm has proven its potential on the IEEE ISBI 2021 6th Cell Tracking Challenge (CTC-6) where we achieved multiple top three rankings for diverse cell types. Our team name is MU-Ba-US, and the implementation of DMNet is available at, http://celltrackingchallenge.net/participants/MU-Ba-US/.

1. Introduction

The capacity of cells to exert forces on their environment and alter their shape as they move [3] is essential to many biological processes including the cellular immune response to infections [25], embryonic development [48], wound healing [8] and tumor growth [16]. Detecting cell shape and their changes over time as cells navigate the microenvironment are essential for understanding the multiple mechanisms guiding and regulating cell motility [74]. We propose an end-to-end pipeline for accurate cell detection, segmentation and tracking as shown in Figure 1.

Figure 1:

Overall pipeline with two stream DMNet for cell segmentation and M2Track for tracking.

Manually segmenting and tracking cells is an expensive, labor intensive and subjective (difficult to reproduce) task due to the need for deep expert domain knowledge and large amounts of image data acquired during live-cell studies. Automated methods and pipelines are needed to perform microscopy video analysis, particularly to segment, track, and characterize cells to accelarate scientific discovery and clinical adoption.

Over several decades, many classical computer vision methods and pipelines have been developed for automated cell detection and segmentation [8, 54, 23, 47]. More recently, various models have been developed for cell boundary prediction to handle segmentation of touching cells [63, 57, 63, 57, 40, 39, 27, 64]. However, accurate cell analysis under different protocols, imaging modalities and cell types remains challenging due to experimental variability, low signal-to-noise ratios, touching or overlapping cells, indistinct deforming boundaries particularly in high cell density cases, agile, unpredictable motion of individual cells, and dynamic interactions between cells.

Recently, deep-learning methods have shown tremendous success in many applications of computer vision including natural object image classification [18], aerial scene classification [13], feature tracking in wide area motion imagery [26], 3D point cloud classification [5], and particularly in various biomedical image analysis e.g. vessel segmentation [33], and malaria diagnosis [34] etc. If adapted, these methods offer promising solutions for cell detection and segmentation.

Cell tracking and lineage is the process of locating cells of interest in images and maintaining their identity over time across cell divisinos to analyze their spatio-temporal behavior (i.e. proliferation, mitosis, and apoptosis). Cell tracking plays an important role in biomedical research for tasks such as cell lineage-tracing [60, 6, 19] and high-throughput motion or behavior analysis [24, 32, 4, 67]. Analyzing cell behavior on live-cell videos requires robust cell tracking approaches to overcome the challenges that the videos have, such as, frequent cell deformations, non-distinct appearance, low image quality in term of contrast, resolution, and imaging acquisition artifacts.

A two-stage segmentation and tracking pipeline is proposed in this work to localize and track different cell types in time-lapse video sequences, as shown in Figure 1. The pipeline consists of two main modules: cell segmentation and cell tracking modules. The cell segmentation module is designed to precisely localize and segment different cells, and the tracking module uses a multi-step data association approach to efficiently track cells across frames. Our pipeline participated in CTC-6 with results on eight 2D datasets with different characteristics in term of cell shape, density, motion patterns, and microscopy modalities. Our results either outperformed the other methods that participated to the challenge or produced comparable results as described in the experimental results section.

To summarize, our contributions are three-fold: (i) we developed DMNet a dual-stream marker guided network for accurate cell segmentation and detection, (ii) we designed M2Track a two level cell tracking module for associating detections and linking tracklets, and (iii) our proposed pipeline demonstrates state-of-the-art performance, scalability and robustness across cell types on the CTC-6 microscopy videos. The subsequent parts of this paper are organized as the following. Section 2 reviews the related work in cell segmentation and tracking. Section 3 describes our proposed approach and details in the design of cell segmentation and tracking. Section 4 presents quantitative results on CTC-6 followed by conclusions.

2. Related Work

2.1. Cell Segmentation

Early methods for cell segmentation include simple thresholding methods [37, 72], hysteresis thresholding [30], edge detection [70, 65], or shape matching [14, 68]. Some methods use sophisticated approaches based on region growing [44, 51, 15], machine learning [61, 10, 56] or energy minimization [52, 62, 73, 21, 45, 22, 20, 7]. For a more comprehensive review of earlier cell segmentation methods, please refer to [47, 53].

More recently, with the development of deep learning networks, many methods benefit from training neural networks with annotated data. Existing methods usually design models for cell boundary or border prediction to handle touching cells. [63] proposes to predict adapted thicker borders and smaller cells in the model to reduce the amount of merged cells. [64] designs a novel representation of cell borders, the neighbor distances, to segment touching cells of various types. [40] utilizes distance transforms with discrete boundaries for single cell nuclei, and [27] uses horizontal and vertical gradient maps. [39] tackles the label inconsistencies problem through encoding a center vector.

Different from these other methods working on various border prediction for handling clustered cells, we propose a dual-stream network to generate guided markers to help splitting cells for accurate cell detection and segmentation.

2.2. Cell Tracking

Cell tracking and behaviour understanding algorithms study individual cell movement, velocity, formation, mitosis, cell groups behavior and etc. Tracking methods can be categorized into two groups: tracking-by-detection [42, 45, 59] and tracking-by-model evolution. Tracking-by-detection, requires locating cells in advance on the entire sequences using segmentation [42, 49, 38], or detection algorithms [69, 12] followed by an association process to link detections in time to generate cell trajectories. Tracking-by-model evolution involves an initialization step to locate the cells of interest on the first frame followed by a per cell model evolution in time by using deformable models such as active contours to keep track of individual cell states (position, motion, shape, and orientation) in the following frames [28, 36, 29]. The most popular cell tracking methodology is tracking-by-detection [42, 45, 59]. Some methods use online mode [41], which implement track linking by associating detections between consecutive frames. In such cases, the information is gathered only from current and past frames. These methods tend to be sensitive to detection errors, and produce fragmented tracklets. While the offline models [43], exploit information from the whole time-lapse sequence (i.e. past, and future frames) and have longer and more reliable trajectories.

Our tracking-by-detection cell tracking module is adapted from our earlier works on multi-object tracking for video surveillance [1, 2]. Multi-cell M2Track module is used to track the cells detected by our DMNet segmentation module. The goal is to link the detected cells, recover from missed-detections by better data association using a fast intersection-over-union (IOU) mask matching, predict cell motion using Kalman filtering [31], and link tracklets by taking into account tracklet history such as velocity, motion, and spatial information. Our tracking module can explicitly handle cells entering and exiting the field of view, birth and death of cells, and mitosis.

3. Our Approach Using DMNet and M2Track

The overall pipeline is illustrated in Figure 1. There are two modules in our pipeline: cell segmentation module DMNet, and multi-cell tracking module, as described in the following parts.

3.1. DMNet: Detection and Segmentation

The cell detection and segmentation task is defined to find segmentation mask of each cell. There are two streams in the proposed DMNet, one stream is designed for cell marker detection, and the other is designed for cell mask prediction, as show in Figure 1.

Marker Detection Stream

The marker-based loss function L_marker(·) is computed pixelwise with respect to the labeled marker annotations using a soft Jaccard and weighted cross-entropy loss functions,

$$L_{\text{marker }}=\alpha L_{\text{Jaccard}}(\cdot )+\beta L_{\text{wce}}(\cdot )$$ (1)

where α and β are used to balance the Jaccard loss L_Jaccard and weighted cross-entropy loss L_wce. The Jaccard loss is,

$$L_{\text{Jaccard}}=\frac{1}{N}\sum _{k=1}^N\frac{-y_k{\widehat{y}}_k}{{\text{y}}_k+{\widehat{y}}_k-{\text{y}}_k{\widehat{y}}_k}$$ (2)

Since the distribution of marker and non-marker pixels is highly biased, we use a class balanced cross-entropy loss, which is defined as,

$$L_{\text{wce}}=-{\lambda }_{-}\sum _{y_k(i,j)\in Y_{-}}\text{log}\left(1-{\widehat{y}}_k\right)-{\lambda }_{+}\sum _{y_k(i,j)\in Y_{+}}\text{log}\left({\widehat{y}}_k\right)$$ (3)

where each prediction map in the mini-batch of marker detection stream is ${\widehat{y}}_k$, of size R × C, ${\widehat{y}}_k\in (0,1)$ denotes a predicted marker map (see Figure 2 (e)), y_k is the groundtruth mask (see Figure 2 (b), y_k is the binarized version of it). ${\lambda }_{+}=\frac{\left|Y_{+}\right|}{\left(\left|Y_{+}\right|+\left|Y_{-}\right|\right)},{\lambda }_{-}=\frac{\left|Y_{-}\right|}{\left(\left|Y_{+}\right|+\left|Y_{-}\right|\right)}$ balance the marker/non-marker pixels to control the weight of positive over negative samples.

Figure 2:

Illustration of intermediate results in the DMNet workflow for cell segmentation: (a) Normalized raw input image, (b) Marker Ground-Truth for the supervision of the marker detection stream, (c) Distance penalty map in L_dist, (d) Cell segmentation ground-truth showing cells with tracking ids (binarized version provides supervision for mask prediction stream), (e) Marker prediction output from marker detection stream, (f) Labeled predicted markers after thresholding and connected component labeling, (g) Mask prediction output from mask prediction stream, and (h) Cell segmentation prediction using mask and marker, after splitting cells using marker guided morphological watershed algorithm.

Mask Prediction Stream

For the mask prediction stream, the loss function L_mask is computed pixelwise with respect to the labeled mask segmentation annotations using a a soft Jaccard and distance penalized cross-entropy loss functions as,

$$L_{\text{mask }}=\alpha L_{\text{Jaccard }}(\cdot )+\beta L_{\text{dist }}(\cdot ).$$ (4)

The L_dist is defined as,

$$L_{\mathit{\text{dist}}}=-\frac{1}{N}\sum _{k=1}^N\sum _{i=1}^R\sum _{j=1}^C(1+\varphi (i,j))L_{ce}$$ (5)

where L_ce is the cross-entropy loss, and $L_{ce}=\left(m_k(i,j)\text{log}{\widehat{m}}_k(i,j)+\left(1-m_k(i,j)\right)\text{log}\left(1-{\widehat{m}}_k(i,j)\right)\right)$. Here each prediction map in the mini-batch of mask prediction stream is ${\widehat{m}}_k$ (see Figure 2 (g)), of size R × C. The cross-entropy loss is modified by a distance penalty map ϕ, which inverses and normalizes the distance transform map D. The Euclidean distance transform map is computed as,

$$D^2(i,j)=\sum _i^R\sum _j^C{(x(i,j)-b(i,j))}^2$$ (6)

where b(i, j) is the location of a background pixel (value 0) that is closest to corresponding input points x(i, j), where edge pixels of cells are 0, and remaining pixels are 1. Figure 2 (c) shows an example distance penalty map ϕ.

Cell Detection & Segmentation

During the inference, both markers and masks are generated, and then the morphological operation watershed [55] is applied to split cell mask guided by our generated markers.

Encoder-Decoder Backbone

For each stream, we use the same Convolutional Neural Network (CNN) structure HRNet [71, 66] as the CNN model to learn the marker localization and mask prediction map since the HRNet encodes rich representations of low-resolution and high-resolution information.

3.2. M2Track: Multi-Cell Tracking Module

Our multi-cell tracking module in Figure 3, tracks the detected cells estimated by the DMNet segmentation module. Tracking module is a multi-step cascade data association process. The cascade data association has two steps: first, short-term tracking which is frame-to-frame data association and matching using mask intersection over union (IOU) score. IOU computation speeds up for overcoming dense scenes with very large cell number. IOU mask score is used for matching current frame detections with previous frame trajectories using linear assignment optimization algorithm [17]. Followed by the second step, long-term tracking, which is called global data association step that connects cells at the track level using spatial and temporal clues to re-link fragmented tracklets. Several modules are used to improve the performance including gating strategy for reducing assignment complexity of ids by pruning improbable assignments; Kalman filter for recovering from miss-detections, and removing unreliable tracklets. For more details of the tracking algorithm, please refer to [1, 2].

Figure 3:

M2Track with intersection-over-union mask overlap matching for multi-cell tracking-by-detection. The two major modules are: Level 1 for managing frame-to-frame linear assignments between detected cells and handles entering and exiting cells, Level 2 for tracklet linking and missing or occluded cells.

Short-Term Tracking:

Short-term data association step, optimizes the associations of current detected cells D^t at frame t to the predicted track T^t−1 at frame t − 1, where the set of detecitons, D^t = {d₁, d₂, …., d_N} is assigned to the previously tracked objects T^t−1 = {T₁, T₂, …., T_M}, and T^t−1 is the set of predicted cell trajectories from previous cell motion history computed using Kalman filter with constant velocity model, N is the number of the detected cells at frame t, M is the number of tracked cells at frame t − 1. Mask IOU score is used to assign detection-to-track between following frames by minimizing a cost matrix using Munkres Hungarian algorithm [50] as:

$$\underset{b\in B}{\text{min}}\sum _{i=1}^m\sum _{i=1}^n{c}^{ij}{b}^{ij}$$ (7)

where $c_t^{ij}$ is an i row to j column entry on cost matrix representing the cost of assigning detection j to tracklet i at time t and its value represents the IOU between the area of i and j detect masks as:

$$c_t^{ij}=\frac{\left|A_i\cup A_j\right|-\left|A_i\cap A_j\right|}{\left|A_i\cup A_j\right|}$$ (8)

with constraints,

$$\begin{array}{l}\sum _{i=1}^m{b}^{ij}=1\hspace{1em}j=1,2,\dots .,n;\\ \sum _{i=1}^n{b}^{ij}=1\hspace{1em}i=1,2,\dots .,m.\end{array}$$

Circular gating regions around the predicted track positions are used to eliminate highly unlikely associations to reduce computational cost, and to reduce false matches. Pairs of detection and tracks represent the results of minimum optimization. For each individual cell, a (one out of four) status (new track, linked track, lost track, and dead track) is assigned according to the assignment process. Since this step considers only information from consecutive frames, having false detections, occlusions, and matching ambiguities causes track fragmentation. Further step is important to improve the performance.

Long-Term Tracking:

Problems during object detection or data association process result in implicit fragmentation of cells. Long-term tracking is used to re-link fragmented trajectories to produce longer tracks. Using information across long video segments can make this process expensive. Optimizing hypotheses at the track level rather than the object level reduces the computational cost of data assignment by gating uncertain hypotheses. Spatial distances and temporal information are used for filtering.

4. Experimental Results

4.1. CTC-6 Dataset

Cell segmentation and tracking benchmark [11] consists of 2D and 3D time-lapse video sequences of fluorescent counterstained nuclei or cells moving on top or immersed in a substrate. The benchmark consists 20 different datasets (10 for (2D) and 10 for (3D)). They can be either contrast enhancing, or fluorescence microscopy recordings of live cells and organisms. Each dataset consists of two training and two testing videos. The training videos were provided with annotations, gold annotation (containing human-made reference annotations but not for all cells), and silver annotation (containing computer-generated reference annotations). The benchmark has different challenges: 1) Different appearances between datasets; 2) Low contrast between foreground and background; 3) The benchmark were taken in different light condition and different image acquisition environment; And 4) The ground-truth annotations for training set are not fully provided for gold annotations and not accurate for silver annotations.

We participated in ISBI 2021 CTC-6, with over thirty teams reporting results on the CTC website which is updated monthly. Not every method reported results for all datasets in the benchmark. We evaluated our pipeline on eight 2D datasets for cell segmentation and tracking. OP_CSB is used for evaluating cell segmentation which is composed of the segmentation metric SEG, and the detection metric DET, as in:

$$O{P}_{CSB}=0.5\cdot (DET+SEG)$$ (9)

OP_CTB is used for cell tracking which is composed of segmentation metric SEG, and tracking metric TRA, as in:

$$O{P}_{CTB}=0.5\cdot (SEG+TRA)$$

For details of evaluation metrics, please refer to the CTC challenge website [11] and [46].

4.2. Implementation Details

Input images are pre-processed to enhance contrast using a z-score mapping. During training, the marker detection stream is trained with supervision using ground-truth of tracking markers, and segmentation mask is supervised by silver-truth of annotations. Both the marker localization and mask prediction streams were trained on eight 2D datasets (see Tables 1, 2, 3) and five 3D datasets that are not shown (Fluo-C2DL-MSC, Fluo-C3DH-H157, Fluo-C3DL-MDA231, Fluo-N3DH-CE, and Fluo-N3DH-CHO). When using 3D datasets, we used one frame or slice per volume with the most annotated labels for training. Input images are resized then cropped for training. Resize scale factor for each dataset are: Fluo-C2DL-MSC: 0.35, Fluo-C3DH-H157: 0.35, Fluo-C3DL-MDA231: 2, Fluo-N3DH-CE: 0.5, Fluo-N3DH-CHO: 0.6, PhC-C2DL-PSC: 3, BF-C2DL-MuSC: 0.75, BF-C2DL-HSC: 0.75. We crop patches with size of 256 × 256 from images in each dataset to train the networks, except BF-C2DL-HSC, BF-C2DL-MuSC which we crop patches of 512 × 512. Regular data augmentation strategies were used including rotation, flip, and scale from 0.8 to 1.5 for each sample. Hyperparameters are learning rate of 0.001 with Adam Optimizer for training both streams for 300 epochs with, α = 2.5 and β = 10.

Table 1:

DMNet cell segmentation performance (OP_CSB) on CTC-6 of March 2021. All reported results are from the CTC Challenge website. The first row is OP_CSB and the second row is ranking compared to other submitted algorithms. Not all methods reported results for all datasets which are shown as NA. Since CALT-US did not report results for Fluo-C2DL-MSC we provide two sets of Rankings – 8 datasets and 7 datasets for equivalent comparison. Rank Sum is the sum of all the ranks across cell types. DMNet consistently outperforms other methods on 2D cell segmentation.

Dataset	BF-C2DL-HSC	BF-C2DL-MuSC	DIC-C2DH-HeLa	Fluo-C2DL-MSC	Fluo-N2DH-GOWT1	Fluo-N2DL-HeLa	PhC-C2DH-U373	PhC-C2DL-PSC	Avg	Rank Sum(8,7)
KIT-Sch-GE [35]	0.905	0.878	0.850	0.686	0.895	0.938	0.927	0.859	0.893
	1/14	1/14	14/27	6/32	20/42	11/40	15/30	1/33		69,63
PURD-US [58]	0.745	0.678	0.703	0.478	0.915	0.943	0.940	0.790	0.816
	13/14	14/14	19/27	24/32	14/42	6/40	11/30	13/33		114,90
CALT-US [9]	0.901	0.852	0.925	-	0.948	0.915	0.959	0.703	0.886
	2/14	4/14	1/27	-	3/42	18/40	1/30	25/33		NA,54
DMNet (Ours)	0.835	0.860	0.864	0.602	0.939	0.954	0.949	0.826	0.890
	10/14	3/14	12/27	17/32	5/42	1/40	6/30	6/33		60,43

Table 2:

DMNet cell tracking performance (OP_CTB) on CTC-6 of March 2021. The first row is OP_CTB and the second row is ranking compared to other submitted algorithms. All reported results are from the CTC Challenge website. Unreported results are shown as NA. DMNet consistently outperforms other methods on 2D cell tracking.

Dataset	BF-C2DL-HSC	BF-C2DL-MuSC	DIC-C2DH-HeLa	Fluo-C2DL-MSC	Fluo-N2DH-GOWT1	Fluo-N2DL-HeLa	PhC-C2DH-U373	PhC-C2DL-PSC	Avg	Rank Sum
KIT-Sch-GE	0.901	0.872	0.848	0.683	0.894	0.938	0.925	0.855	0.865
	1/10	1/10	8/20	3/26	13/35	10/33	10/24	1/26		47
PURD-US	0.716	0.670	0.684	0.479	0.914	0.941	0.939	0.783	0.766
	9/10	10/10	13/20	18/26	10/35	6/33	8/24	9/26		83
CALT-US	NA	NA	NA	NA	NA	NA	NA	NA	NA
	NA	NA	NA	NA	NA	NA	NA	NA		NA
DMNet (Ours)	0.828	0.849	0.854	0.591	0.939	0.953	0.947	0.821	0.848
	6/10	2/10	6/20	12/26	2/35	1/33	3/24	4/26		36

Table 3:

Performance of DMNet segmentation and tracking pipeline on CTC-6 of March 2021. For each performance metric, the first row is accuracy and the second row is ranking compared to all other submitted algorithms (as of March 2021). Top three performance of DMNet by cell type are bolded.

Dataset	BF-C2DL-HSC	BF-C2DL-MuSC	DIC-C2DH-HeLa	Fluo-C2DL-MSC	Fluo-N2DH-GOWT1	Fluo-N2DL-HeLa	PhC-C2DH-U373	PhC-C2DL-PSC
OPCTB	0.828	0.849	0.854	0.591	0.939	0.953	0.947	0.821
	6/10	2/10	6/20	12/26	2/35	1/33	3/24	4/26
OPCSB	0.835	0.860	0.864	0.602	0.939	0.954	0.949	0.826
	10/14	3/14	12/27	17/32	5/42	1/40	6/30	6/33
DET	0.971	0.979	0.926	0.681	0.946	0.985	0.975	0.945
	8/14	2/14	13/27	17/32	10/42	10/40	16/30	9/33
SEG	0.699	0.742	0.802	0.522	0.931	0.923	0.923	0.708
	6/10	2/10	6/20	13/26	1/35	1/33	3/24	4/26
TRA	0.957	0.957	0.907	0.661	0.946	0.983	0.972	0.933
	4/10	4/10	9/20	12/26	7/35	10/33	11/24	8/26

4.3. Comparison on CTC-6 Benchmark

DMNet+M2Track performance is compared with stateof-the-art methods on Cell Segmentation and Cell Tracking Tasks. Since not every method reported results for all 2D datasets, we show the three most competitive methods KIT-Sch-GE [35], PURD-US [58] and CALT-US [9], which have results for almost all eight 2D cell microscopy videos.

DMNet is robust and achieves state-of-the-art cell detection and segmentation performance on all eight 2D CTC-6 datasets. In Table 1, we compare our DMNet with the state-of-the-art methods on CTC-6 challenge. We compute the rank sum of each method of OP_CSB on all the datasets. Because CALT-US didn’t report results on Fluo-C2DL-MSC, we put NA in that column, and compute the rank sum on eight datasets and seven datasets respectively as shown in the last column of Table 1. Our DMNet achieves best results on all the 2D datasets with rank sum 60 (eight datsets) and 43 (seven datasets), which demonstrates the robustness and effectiveness on 2D cell segmentation. DMNet+M2Track is robust and achieves the state-of-the-art cell tracking performance on all eight 2D CTC datasets. In Table 2, we compute the rank sum of each method of OP_CTB on all the datasets. Because CALT-US didn’t perform cell tracking, therefore it is empty in this table. Our DMNet + M2Track achieves best rank sum with 36, which demonstrates the robustness and effectiveness on 2D cell tracking task. Table 3 shows the ranks of our pipeline compared to the other participants on CTC-6. Our pipeline ranked in the top three on four out of the eight 2D cell type microscopy videos. Not every method provided results for all cell types, whereas DMNet+M2Track results are given for all videos.

Figure 4 shows the results of DMNet+M2Track segmentation and tracking pipeline for three cell types. The first column shows the Raw Input image for three cell types, which are typically low contrast, with dense, clustered cells and small object size. A z-score normalization is applied to the raw input image to remove outliers. The raw input is stretched to increase image contrast, as shown in the second column (Normalized Input). We show the groundtruth segmentation with tracking ids as Tracking GT in the third column. The fourth and fifth columns are the final marker detections and cell tracking predictions for all the cells in each frame of the video. We can clearly see in Figure 4 column (d) that DMNet accurately predicts and separates the cell markers. Hence, using labeled markers as guidance for the watershed algorithm to split the predicted cell masks results in consistently satisfactory cell segmentation results.

Figure 4:

Visualization of DMNet+M2Track segmentation and tracking results for three cell types including Fluo-N2DH-GOWT1, PhC-C2DL-PSC and BF-C2DL-HSC exhibiting a range of cell sizes and densities.

5. Conclusions

The proposed DMNet and M2Track cell segmentation and tracking pipelines provide a common framework across a variety of cell types for high accuracy lineage estimation under challenging sample conditions of high cell density, touching or overlapping cells, deforming cell shape, variable size and indistinct boundaries. For cell segmentation, DMNet uses a dual-stream marker guided deep networks for detection and separation of touching cells. For cell tracking, our M2Track multi-object tracking pipeline generates accurate cell trajectories under challenging conditions (e.g high density, irregular shapes, and cell mitosis activity). DMNet+M2Track is among the best performing methods on the CTC-6 cell microscopy videos across a range of cell types with segmentation and tracking accuracies of over 82 percent (excluding Fluo-C2DL-MSC which has thin elongated mesenchymal stem cells). For 2D cell types our proposed approach has the best rank of all submitted methods in both the cell segmentation and cell tracking subtasks.