A review for cell and particle tracking on microscopy images using algorithms and deep learning technologies

Abstract

Time-lapse microscopy images generated by biological experiments have been widely used for observing target activities, such as the motion trajectories and survival states. Based on these observations, biologists can conclude experimental results or present new hypotheses for several biological applications, i.e. virus research or drug design. Many methods or tools have been proposed in the past to observe cell and particle activities, which are defined as single cell tracking and single particle tracking problems, by using algorithms and deep learning technologies. In this article, a review for these works is presented in order to summarize the past methods and research topics at first, then points out the problems raised by these works, and finally proposes future research directions. The contributions of this article will help researchers to understand past development trends and further propose innovative technologies.

Nearly two decades, recording and storing biological images is already a commonly used and necessary job in most of biological experiments. Among them, the microscopes are usually used to display these images based on various scale requirements. Hence, time-lapse microscopy images generated by biological experiments have been widely used for observing the activities of targets, such as cells and particles, and these observations are useful for solving biological applications, such as the virus research or drug design works [1,2]. In order to observe the target activities, such as the motion trajectories and survival states, in general, the first step is to identify and segment the shape and location of target in each microscopy image, and then track the target from the first microscopy image to last one. Finally, several physical measurements of target, such as the size an instantaneous and speed, are calculated in order to observe the changes in the experiments.

Due to the characteristics of various microscopes, the captured images are different resulting in different sizes and colors of targets. Moreover, there are many types of cells and particles. Therefore, in the past, many methods and tools have been proposed to solve the single cell tracking (SCT) and single particle tracking (SPT) problems by using state-of-the-art algorithms and deep learning (DL) technologies. For example, Dhada et al. [3] demonstrated that stem cell viability can be tracked by using photoacoustic imaging based on a nanoparticle-based contrast agent; Cui et al. [4] showed that the partitioning and dynamics of AtPIP2;1 are cell type-specific by using SPT method; Holsteen et al. [5] presented a light-field-based approach to obtain the SPT trajectory by placing a standard microscope coverslip with a multifunctional and light-field meta-surface on a specimen. Therefore, in this article, a mini-review for these works is presented in order to summarize the past methods and research topics at first, then points out the problems raised by these works, and finally proposes future research directions. In order to avoid confusion and excessive descriptions, in the following, we ignore the works that only focus on the classification and counting problems of cells and particles. Besides, we also ignore the works that focus on the problems of non-microscopic images, such as the magnetic resonance imaging (MRI), computed tomography, x-ray CT, and 3D images. Many famous works solving the segmentation and tacking problems have been illustrated and divided into the following sections.

SCT by algorithms

Chen et al. [6] presented a system with several cellular image analysis methods to segment, classify, and track individual cells in microscopy images. Li et al. [7] proposed a multi-target tracking system for cells in phase contrast microscope (PCM). Their system integrated multiple modules, including cell detector, a topology-constrained contour tracker, a motion filter, and spatio-temporal trajectory optimization. Dzyubachyk et al. [8] further introduced several modifications and extensions to the coupled-active surfaces algorithm for multi-cell segmentation and tracking to overcome the shortcomings of multiple-level-set method. Padfield et al. [9] showed a tracking approach to analyze cell behaviors in a graph-theoretic framework. The split and merged cells can be processed by calculating a minimum-cost flow algorithm. Schindelin et al. [10] presented an open-source project Fiji to update the architecture of ImageJ, and Fiji allows researchers to develop the process of biological-image analysis. The Fiji project has powerful tools to process images via scripting languages and feature-rich libraries. Bergeest and Rohr [11] proposed a globally optimal approach for cell nuclei segmentation based on active contours and level sets with Chan-Vese functional and the Bayesian functional in fluorescence microscope (FM) images. Meijering et al. [12] presented a survey work with various approaches, tools, and quantitative measures in microscopy images. In general, these cells are segmented by the thresholding approach, which classifies pixels as the object and background. However, the thresholding approach fails when the image with severe noise and poor quality. Several methods are also proposed to overcome the above situations, such as the template matching and deformable models. However, new problems arise with these methods. Many tools under the different platforms and dimensions for SCT problem, such as Braincells, CellTrack, CellTracker, DcellIQ, DIAS, DYNAMIK, FARSIGHT, LevelSetTracker, LineageTracker, Oko-Vision, QuimP, StarryNite, TLA, have been proposed, and the survey of these tools can be seen in this article.

Su et al. [13] showed a semi-supervised learning-based algorithm for SCT problem in PCM images. In their algorithm, a PCM image is partitioned into phase-homogeneous atoms by clustering neighboring pixels’ feature vectors at first, and then cells are segmented by classifying the phase-homogeneous atoms. Chowdhury et al. [14] presented a matching and linking method for bipartite graphs to track human monocyte cells in a FM video. A cost function is used to track cells over a pair of frames, and the tracking results are refined by a rank-based filtering mechanism. Dimopoulos et al. [15] proposed a cell segmentation method to detect cell boundaries using the cell membrane information. Moeller et al. [16] showed a topology preserving variational segmentation approach for SCT problem in PCM videos. Magnusson et al. [17] presented a track linking algorithm Viterbi to link segmented cells into tracks by considering the information from the complete image sequence. They also used a way to alter previously created tracks when new tracks are added in order to mitigate the effects of error propagation. Schiegg et al. [18] proposed a probabilistic graphical model to select the best segmentation and tracking for multiple cells by using the intra-frame and inter-frame constraints between conflicting segmentation and tracking hypotheses. Paintdakhi et al. [19] developed an open-source package Oufti to measure microbial cells and fluorescence signals from microscopy images. Oufti can handle various cell morphologies and provide quantitative analysis of diffraction and non-diffraction limited fluorescence signals.

Hilsenbeck et al. [20] developed two software tools, tTt and qTfy, in FM images. In tTt tool, individual cells in each time frame were observed and tracked by manually evaluating, not a computer algorithm. Yang et al. [21] proposed two frameworks for SCT problem. In the first framework, each cell is detected, segmented and represented as a dot in PCM images. These dots are then linked between time frames to create cell trajectories. In second framework, an SCT algorithm is proposed by a mathematical model with the data evolution. Masuzzo et al. [22] reviewed research works for in vitro cell migration with image pre-processing, motion estimation and feature extraction at first, then several silico models of cell migration are summarized, and available software tools for cell migration are listed finally. In their article, these SCT algorithms can be classified into three categories: (1) tracking by detection, (2) tracking by model evolution, and (3) tracking by filtering. In tracking by detection category, cells must be segmented from the background image by using pixel labeled or edge detection, and then these cells need to be connected over time to form SCT trajectories. The commonly used way is to connect each segmented cell in a frame to the nearest cell in the subsequent frame by considering the cell centroids. When cells are moving slowly and their distribution is sparse, this way is efficient than others. Another way is the feature matching (as template matching), which is to locate similar cells using a list of features such as the morphology and area. In general, this way requires the users to specify the maximal distance that cells can move between two consecutive frames. In tracking by model evolution category, a deformable model is designed for SCT problem to match the images. The analysis result in one frame is then used as an initial condition for the next frame. In tracking by filtering category, SCT can be seen as the problem of estimating object's state (i.e. posterior density function), and it also called sequential Monte Carlo technique. By comparing with the article [12], more tools under the different platforms and dimensions for SCT problem are proposed, such as CELLMIA, Cell motility BioApplications, OpenLab, Adapt, AveMap, Cell Image Velocimetry, iTrack4U, Pathfinder, TScratch. The survey of these tools can be seen in this article.

Ulman et al. [23] presented a report on the results of SCT challenge in order to promote the development and objective evaluation of algorithms and machine learning (ML) models. In the discussion of this article, they pointed out several results for the comparison of these algorithms and models, which are briefly described as follows: (1) in most practical scenarios, SCT algorithms by detection outperformed those by model evolution. (2) ML models performed best in most segmentation scenarios. (3) In SCT algorithms by detection, it is better by considering the global and spatio-temporal contexts than that by only considering the nearest neighbors. (4) Long runtimes by complex algorithms can be reduced by running them on graphics hardware. (5) There is no simple way to point out the right algorithm for a given data set due to the complex factors for affecting the results of SCT problem. (6) They encouraged researchers to make their method available to biologists through simple installation and intuitive user interface. Arbelle et al. [24] presented a framework in microscopy videos to segment and track cells through the graphical and probabilistic model without the assumption of cell shape and a marching algorithm.

SPT by algorithms

Jaqaman et al. [25] presented a SPT algorithm to link segmented particles between consecutive frames at first, and then it links the resulting tracks into complete trajectories to capture split and merged particles. In their algorithm, a mathematical framework as a linear assignment problem is used to provide an accurate solution. Yang et al. [26] proposed a framework with an enhancement filter based on a probability model. The detection method is combined foreground and background markers. They also developed the multiple mode filter for particle motion modeling and data association. Meijering et al. [12] also presented a survey work for SPT problem. In general, these particles can be segmented by similar methods used in SCT problem. However, these particles are hardly visible in bright field microscopy (BFM) or PCM images since the size of particles is general two orders of magnitude smaller than cells, and they are usually imaging in the FM. Moreover, particles may disappear, (re)appear, split, and merge in FM images. The consistent results should be achieved by using global linking strategies rather than local one, such as the spatio-temporal tracing, graph-based optimization and Bayesian estimation approaches. Several tools under the different platforms and dimensions for SPT problem, such as ClusterTrack, ManualTracking, Mtrack2, MTrackJ, Octane, ParticleTracker, plusTipTracker, SpotTracker, TIKAL, u-track, have been proposed.

When the particle number is known and allowed particles to disappear and (re)appear, the aim is important to match as many particles as possible. Hence, a large number of SPT tasks (or scenarios) should be excluded. Vallotton and Olivier [27] devised a software Tri-track to reduce SPT tasks, which is formed as a max-flow min-cost problem, through a graph structure comprised from three consecutive image frames. Chenouard et al. [28] proposed a Bayesian model to solve the problem of SPT in microscopy images at first, and then described a multiple hypothesis tracking algorithm for extracting trajectories from the analysis results of the former. Shuang et al. [29] reviewed a range of available techniques used in SPT problem. They summarize some observations and briefly described as follows: (1) It is difficult to do the quantitative analysis on SPT data without an automatic program. (2) No general SPT program is existed to be suitable for all cases. (3) The future directions in the program development focused on higher speed and more reliability, i.e., using the GPU to speed up the calculation. (4) It will become an important technique to synchronous data analysis with SPT measurements.

Liang et al. [30] proposed a SPT method for analyzing an essential subcellular process to manage trajectories, solve data association problems, and handle pseudo-split/merged particles. Chenouard et al. [31] organized an open competition for many SPT algorithms and ML models. In the discussion of this article, they pointed out several results for the comparison of these algorithms and models, which are briefly described as follows: (1) SPT algorithms with multi-frame and/or multi-track optimization schemes in the linking stage perform better performance than those only using simple per-frame and per-particle nearest-neighbor approaches. (2) The parameter tuning and prior knowledge are important for computational image analysis, and suggest to develop the learning-based tracking methods. (3) None method performed perfectly on any data set, and real biological data may be more complex, it is necessary to develop new SPT methods. Jaiswal et al. [32] proposed a SPT approach based on the multi-scale detection and two-step multi-frame association. The proposed algorithm is to determine reliable associations for each particle by local neighborhoods at first, and then the global spatial information over multi-frames are used to determine optimal associations.

Smal and Meijering [33] presented the comparison results of data association techniques with multi-frame or two-frame for solving the linking problem in SPT problem. In the discussion of this article, they pointed out several results for the comparison of these techniques, which are briefly described as follows: (1) all linking techniques have good performance with the assumption of no detection errors; however, their performance decreases as the number of detection errors increases. (2) In most cases, the linking techniques with multi-frame outperformed those with two-frame. (3) For SPT problem, the performance by the linking technique may depend on the motion type and density level. Shen et al. [34] summarized the algorithms and applications of SPT problem, including the particle identification, localization, and trajectory reconstruction. They pointed out that the future direction for SPT method development is to integrate compressive imaging methods and ML techniques to achieve the goal of real-time analysis. Tinevez et al. [35] proposed an open source tool TrackMate for SPT to provide a simple and intuitive user interface that the developers can write their own algorithms; Zhang et al. [36] proposed a program UmUTracker to detect and track 3D particles in the microscopy video. UmUTracker detects the 2D lateral positions of particles based on the isosceles triangle transform at first, then reconstructs their 3D axial positions by the Rayleigh–Sommerfeld model with a radial intensity profile.

Duple work by algorithms

Although many works above have been proposed, these works are designed to solve the individual problem for SCT or SPT. A few of works and tools have been proposed with the ability to solve SCT and SPT at the same time. Carpenter et al. [37] presented the free and open-source system CellProfiler for SCT problem. In CellProfiler, the original images are processed at first, then cells in each image are identified, and the measurements of every cell in each image are calculated finally, such as the location, size, shape, intensity, texture. Except for the SCT problem, CellProfiler also provide the measurement for Coulter particle counter. Chaumont et al. [38] presented an open source software ICY for image analysis applications. In ICY, the multiplatform software framework is bundled with, and many API functions are provided to solve SCT and SPT problems. Vallotton et al. [39] developed a software package Diatrack for SPT problem. In Diatrack, an input image is preprocessed at first, then particles are produced and selected, finally all images are processed to do the particle tracking and calculate the measurements, such as the speed. The functions in Diatrack for SPT problem also were used to solve the SCT problem by other researchers. Several commercial tools, such as Image-Pro Plus, Imaris Trak, MetaMorph and Volocity, are also provided for image analysis applications. The survey of these tools can be seen in the articles [12,22].

SCT and SPT by DL

Recently, the DL technologies [40] have been used to solve more and more problems, especially for the image analysis applications. For example, Litjens et al. [41] reviewed the major DL concepts for medical image analysis and summarized the contributions to this field. Several works by adopting ML models have been proposed to solve the related problems of particles and cells. However, most of these works are focused on the classification, identification and segmentation. For example, Held et al. [42] presented a computational strategy, called CellCognition, to combine ML method (for classification by support vector machine, SVM) and hidden Markov modeling (for cell detection) to measure morphological classes in live-cell FM movies. Zaritsky et al. [43] proposed a segmentation algorithm, called MultiCellSeg, to separate between multi-cellular and background regions for BFM image, which is partition into local patches and then each patch is classified by SVM. Ronneberger et al. [44] presented a network, called U-Net, and training strategy with data augmentation in PCM images to win the ISBI cell tracking challenge 2015. Akram et al. [45] developed a CNN-based method to provide the functions for cell detection and segmentation, in three kinds of microscopy images. Valen et al. [46] showed that a deep Convolutional neural network (CNN) (called conv-net) can robustly segment bacterial and mammalian cells in PCM and FM images. Song et al. [47] presented a framework based on multi-scale deep CNN and a deformation model to segment overlapped cervical cells in Pap smear images. Different from cell segmentation works above, Xie et al. [48] proposed fully convolutional regression networks for cell density maps to do both cell counting and detection tasks in FM images.

A few works were proposed to solve the tracking problem by integrating DL technologies. For example, Tsai et al. [49] proposed a pipeline, called Usiigaci, implemented in Python, to do the cell segmentation, cell tracking, and visualization of cell movement and morphological changes in PCM images. They applied a mask regional convolutional neural network (Mask R–CNN) to do the cell segmentation, and then a graphical user interface is designed for cell tracking and verification. Hu et al. [50] presented an integrated graphical user interface software, called CellTracker implemented in Python, for cell segmentation and tracking of time-lapse microscopy images. CellTracker covers several steps, including project management, image pre-processing, cell segmentation (by U-Net), cell tracking, manually correction and statistical analysis, such as the cell size. Lou et al. [51] showed a cell tracking approach with ML technique (as a max-margin structured learning) to optimize the parameters based on user annotated tracks. He et al. [52] proposed a cell tracking method based on CNN with a filter motion model and an update strategy. In their tracking procedure, the cell position in the first frame is assigned, then, a filter model is used to generate a set of candidate bounding boxes in the subsequent frames. The filter motion model predict and produce the confidence probabilities for each candidate, and choice the candidate with the highest probability. Finally, an update strategy is applied for entire tracking procedure.

Yao et al. [53] presented DL-based methods, such as CNN and long short-term memory networks, to extract dynamics features and predict the movement of a particle in FM images from one time point to the next. In their method, the results of particle detection (segmentation) by other previous works are used as the input. Chamier et al. [54] presented an entry-level platform ZeroCostDL4Mic based on Google Colab to apply DL models to perform various tasks for microscopy images. In ZeroCostDL4Mic, U-Net and StarDist are used to do the segmentation for 2D and 3D electron microscopy images; YOLOv2 is used to do the object detection for cells; CARE and Noise2Void are used to do the image denoising and resolution improvement; Deep-STORM is used for the super-resolution microscopy; fnet, pix2pix and CycleGAN are used to do the image-to-image translation. For the tracking problem, in ZeroCostDL4Mic, the StarDist model is directly compatible with the TrackMate [35] to enable automated cell tracking.

Conclusion

From the survey above, we can see that many methods or tools have been proposed to solve SCT and/or SPT problems by using algorithms or DL technologies. Most of the methods and tools by algorithms were proposed in 2006–2018. After that, few innovative methods are proposed. On the contrary, most of the methods and tools by DL technologies were proposed in 2014–2021 and new innovative methods continue to be proposed. However, in practical applications, new methods or tools by algorithms and DL technologies are still needed by applying new technologies or doing the integration work.

For solving SCT and SPT problems by algorithms, the advantage is that the algorithms can be applied into different types of cells and particles. The reason is that the algorithms are usually designed based on universal physical/chemistry properties. In the past, different methods or tools were used to solve individual problems or constrains. However, there is a lack of integrated tools that can solve these problems or constraints at the same time. Moreover, with the advancement of microscope technology and the advancement of biotechnology, more and more new forms of cell and particle images have been published. It will be necessary to integrate these methods or tools. Besides, the disadvantage of them by algorithms is the time-consuming when doing this work. Although some papers have pointed out this problem, and then recommended to use GPU to accelerate the calculations. Unfortunately, few successful (famous) methods or tools are proposed at present. There are two possible reasons: (1) it is not easy and sustainable to obtain GPUs by researchers or users under the limited funding, (2) the threshold for programming skill is relatively high. Based on the above summary and observation, there are two suggestions for future research directions: (1) using the multi-core CPU and OpenMP may be another option to speed up the calculations; (2) developing an integrated tool that can solve these problems or constrains at the same time.

For solving SCT and SPT problems by using DL technologies, the advantage is that non-expert users (as biologists) can use the training models for their applications. The reason is that the parameters of algorithms should be carefully adjusted by expert users; the adjustment of the training model is relatively simple for non-expert users. However, the disadvantage of using training models is that it needs to build many individual models for different types of cells and particles. Due to the small scale and brightness of the particles, there is a lack of relatively mature technology. Moreover, its performance of them by DL technologies is easily affected by the image quality or noise. Based on the above summary and observation, there are two suggestions for future research directions: (1) developing related particle identification technology will help the development of related applications; (2) combining computer graphics algorithms with DL technologies may be helpful in order to label different types of cells and particles and correct the image quality.

Conflicts of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.