Comparing Deep Learning Performance for Chronic Lymphocytic Leukaemia Cell Segmentation in Brightfield Microscopy Images

Abstract

Objectives:

This article focuses on the detection of cells in low-contrast brightfield microscopy images; in our case, it is chronic lymphocytic leukaemia cells. The automatic detection of cells from brightfield time-lapse microscopic images brings new opportunities in cell morphology and migration studies; to achieve the desired results, it is advisable to use state-of-the-art image segmentation methods that not only detect the cell but also detect its boundaries with the highest possible accuracy, thus defining its shape and dimensions.

Methods:

We compared eight state-of-the-art neural network architectures with different backbone encoders for image data segmentation, namely U-net, U-net++, the Pyramid Attention Network, the Multi-Attention Network, LinkNet, the Feature Pyramid Network, DeepLabV3, and DeepLabV3+. The training process involved training each of these networks for 1000 epochs using the PyTorch and PyTorch Lightning libraries. For instance segmentation, the watershed algorithm and three-class image semantic segmentation were used. We also used StarDist, a deep learning-based tool for object detection with star-convex shapes.

Results:

The optimal combination for semantic segmentation was the U-net++ architecture with a ResNeSt-269 background with a data set intersection over a union score of 0.8902. For the cell characteristics examined (area, circularity, solidity, perimeter, radius, and shape index), the difference in mean value using different chronic lymphocytic leukaemia cell segmentation approaches appeared to be statistically significant (Mann–Whitney U test, P < .0001).

Conclusion:

We found that overall, the algorithms demonstrate equal agreement with ground truth, but with the comparison, it can be seen that the different approaches prefer different morphological features of the cells. Consequently, choosing the most suitable method for instance-based cell segmentation depends on the particular application, namely, the specific cellular traits being investigated.

Introduction

The combination of state-of-the-art instrumentation for the temporal surveillance of biological specimens, along with deep learning-based image processing techniques, offers promising opportunities to advance fundamental scientific research. We have used the latest advancements in image data processing using neural networks with different backbones and have proposed a method for segmenting chronic lymphocytic leukaemia (CLL) cells, which is an initial step towards further analyses of cell image data. Accurate image segmentation of cells is necessary to determine their morphological characteristics, which are important indicators of cell properties. Accurate cell segmentation is also a primary step towards successful cell tracking during in vitro migration analyses. Although most studies address the efficient detection of cancer cells from blood samples, our study addresses the need to segment cells in brightfield time-lapse microscopy images to study cell migration and changes in cell morphology using readily available computational technologies.

Our experiments answer the following questions:

• What are the most suitable new deep learning algorithms for cell detection in brightfield images?

• What is the best way to separate instances of individual objects? Comparison of three approaches. Which approach works best in our data?

• What parameters can be used to characterise cells? How do the values of these parameters vary depending on the different approaches?

To address this, we conducted a comprehensive comparative analysis using a variety of neural network architectures and tested their performance with different backbones.

The need for precision cell segmentation of CLL cells

CLL is a chronic disease with significant genetic, morphological, and phenotypical heterogeneity. The condition exhibits considerable clinical variability among patients with respect to prognosis and clinical course, ranging from slow and stable to extremely aggressive. ¹ CLL is characterised by the progressive accumulation of CD5⁺, CD23⁺, and CD19⁺ B cells in the peripheral blood, bone marrow, lymph nodes and spleen. ² Recent investigations have shown that variations in the size of CLL cells are indicative of their activation status, polarisation, and motility compared to the population of smaller CLL cells and indicate that the functional differences observed in vitro may reflect a different in vivo leukaemic potential, probably responsible for the heterogeneous characteristics of the disease and may contribute to different clinical manifestations in CLL patients. ³ In addition, a negative prognostic value was found to be associated with atypical CLL cell morphology. ⁴ Although CLL cells in peripheral blood are floating freely, their migration to and from the bone marrow is essential for their survival, ⁵ thus, a deeper understanding of the morphology and migratory behaviour of CLL cell subpopulations can be useful for both fundamental research and precision medicine.

Furthermore, state-of-the-art studies focus on automated detection and classification of cells from blood smear to perform accurate segmentation of tumour cells from normal cells,^6-8 classification of different types of blood cells,^9,10 or detection of specific properties of blood cells, such as deformability ¹¹ and velocity. ¹²

Cell segmentation methods

The investigation of cell segmentation has garnered growing interest in recent years due to the importance of cell morphology as a crucial phenotypic characteristic that provides insight into a cell’s physiological condition. Furthermore, the delineation of cell boundaries is frequently necessary for the subsequent analysis of intracellular processes at the nanoscale or for studying cellular interactions at the millimetre scale. Therefore, cell segmentation is a major challenge in image data analysis due to its relevance in understanding cellular behaviour and enabling comprehensive investigations into various biological phenomena. ¹³

Automated segmentation and detection of cells from optical microscopy and other imaging devices, particularly time-lapse microscopy, can be a valuable addition in addressing various biological questions that are heavily based on single-cell analysis. During the past decade, the field has evolved significantly, incorporating both conventional and machine learning-based methodologies. ¹⁴

Traditional (conventional) methods for image segmentation

Conventional techniques rely on predefined rules and heuristics to separate foreground objects from the background. Currently, the field of conventional cell segmentation algorithms encompasses a range of methodologies. These include techniques such as thresholding, ¹⁵ watershed segmentation,^16-18 and active contour segmentation,^19-21 which are known to have an over-segmentation bias when the images are contaminated with noise. However, the advent of machine learning has revolutionised cell segmentation, enabling the development of sophisticated algorithms capable of learning and adapting to complex image patterns. ²²

Traditional methods are often combined and integrated into automated workflows,^23,24 not only to accelerate the segmentation procedure but also to complement the pre- and post-processing stages of neural networks. ²⁵ They can also be effectively used in data annotation to streamline the generation of training data, which is essential for training neural networks.^26,27

Deep learning-based segmentation and object detection

The current trend in microscopic cell image segmentation is based on machine learning and deep learning methods, which have gained dominance in cell segmentation and other fields in the last decade. The increasing complexity of microscopy images and the limitations of conventional approaches in cell segmentation have prompted substantial advancements in these techniques, which can be broadly categorised into supervised and unsupervised methodologies.^28,29 Supervised methods involve the development of mathematical functions or models through training data to enable the accurate prediction of new samples. Conversely, unsupervised machine learning algorithms operate independently of pre-assigned labels or scores in the training data. In particular, clustering methods have emerged as the predominant unsupervised techniques for addressing these challenges.^30-33

In general, there are two main approaches to image segmentation, namely semantic and instance segmentation. In semantic segmentation, the objective is to assign a class label to each pixel in an image without explicitly distinguishing between individual instances of the same class. Conversely, the second category, instance segmentation, aims to differentiate between individual instances belonging to the same class. However, unlike semantic segmentation, instance segmentation does not provide predictions for all pixels in an image and focuses solely on predicting one particular type of object, namely, the foreground, while disregarding other objects present in the background.^34,35

Cell segmentation requires instance segmentation because in microscopic images, individual cells touch, and using semantic segmentation alone results in the touching cells acting as a single object. Thus, a major challenge in cell segmentation is to design reliable feature representations that can recognise cell boundaries with high accuracy. To address this, there are several possible approaches. The first involves classifying each pixel into a semantic class and grouping pixels of the same class into individual instances, ³⁴ which can be achieved, for example, by applying connected component analysis. ³⁶ This approach requires the most accurate semantic segmentation possible to achieve the precise categorisation of pixels in the cell-background transition region. Another approach involves localising each cell via rough shape representation using axis-aligned bounding boxes ³⁷ or convex polygons for each pixel. ³⁸ This approach is categorised as object detection rather than segmentation, in which object detection algorithms offer the capability to generate bounding boxes around individual cells. The process of drawing bounding boxes involves the labelling of two points in spatial coordinates and can exhibit greater efficiency compared with pixel-level annotation. In addition, the use of bounding boxes facilitates straightforward differentiation of overlapping objects. ³⁹ In scenarios where pixel-level segmentation is necessary, object detection can be followed by post-processing techniques that establish accurate boundaries for each object. ⁴⁰

Consequently, a transition from pure semantic segmentation to instance-aware semantic segmentation is imperative. A viable approach involves incorporating a narrow artificial background ridge, measuring one pixel in width between the touching elements within the actual segmentation mask. ⁴¹

For comparison, we selected different types of architectures developed for image segmentation and object detection. These networks differ in their structures on the desired results. The requirements of an optimal network vary with respect to the input data and the purpose of use. We defined the basic requirements as segmentation accuracy, ie, defining the boundary of an object as accurately as possible and distinguishing it from the background.

U-net and U-net++

U-net is a neural network architecture known for its encoder-decoder network structure, which enables end-to-end solving of semantic segmentation tasks, even when the training data set is limited. The U-net network, proposed by Ronneberger et al, ⁴² builds on the foundation established by Shelhamer et al ⁴³ and their use of fully convolutional networks.

An important common feature of encoder-decoder networks used in image segmentation tasks is the integration of skip connections. In the U-net architecture, these skip connections are achieved by concatenating up-sampled feature maps and incorporating convolutions and non-linearities between each up-sampling step. These skip connections improve gradient flow and enable the transfer of information between the down-sampling and the up-sampling paths by connecting each pair of encoder-decoder layers, thus proving its effectiveness in recovering full spatial resolution at the network output and making fully convolutional methods suitable for accurate semantic segmentation. ⁴⁴

Since its introduction in 2015, the U-net network has experienced a rapid rise in the field of medical and biological image data evaluation; naturally, various versions of it are emerging, namely 3D U-net, ⁴⁵ Residual U-net, ⁴⁶ Attention U-net, ⁴⁷ Inception U-net, ⁴⁸ Dense U-net, ⁴⁹ R2 U-net, ⁵⁰ U-net++, ⁵¹ SE U-net, ⁵² and many other networks that make use of an ensemble of U-nets.^53-56

In our study, we employed the U-net++ architecture, which incorporates nested and dense skip connections. The fundamental premise underlying this architecture is that the model can capture intricate details of foreground objects by progressively enhancing high-resolution feature maps from the encoder network before merging them with semantically enriched feature maps from the decoder network. U-net++ comprises an encoder and decoder linked by a sequence of nested dense convolutional blocks. The primary purpose behind U-net++ is to close the semantic gap between the feature maps of the encoder and decoder prior to fusion. ⁵¹

The Feature Pyramid Network (FPN)

Feature pyramids are a fundamental element within recognition systems designed to detect objects across various scales. The concept of feature pyramids was originally introduced by Lin et al, ⁵⁷ with the aim of effectively leveraging multiscale features for tasks involving object detection and recognition. FPN is constructed on top of a backbone network, typically a pre-trained convolutional neural network (CNN) like ResNet or VGGNet, and is responsible for extracting features from the input image. Feature pyramid networks enable these CNNs to capture features at distinct levels of abstraction and spatial resolution. The uppermost levels of the pyramid encompass high-level features that are semantically rich but possess a lower spatial resolution compared to the lower levels of the pyramid. On the contrary, the lower levels of the pyramid exhibit high spatial resolution but contain less semantic information in comparison with the higher levels. By integrating features from multiple scales, the FPN facilitates robust and precise object detection and recognition.

The Multiscale Attention Net (MA-Net)

Attention is an important mechanism that can be used for a variety of deep learning models in many different domains. ⁵⁸ Attention mechanisms have been introduced into computer vision to simulate the human ability to detect salient regions in complex scenes and have been successfully applied in many areas, including semantic segmentation.^59-62 MA-Net was first proposed by Fan et al ⁶³ for the segmentation of the liver and tumours. Apply self-attention modules in the bottleneck of the U-net architecture. The first module, the Position-wise Attention Block, is used to obtain spatial dependencies between pixels in feature maps, while the second module, the Multiscale Fusion Attention Block, considers the channel dependencies between any feature maps.

The Pyramid Attention Network (PAN)

The PAN is an architectural design that aims to enhance the feature representation capabilities of CNNs by incorporating attention mechanisms. It was inspired by the attention mechanism to provide precise pixel-level attention for high-level features extracted from CNNs and was designed to capture both local and global contextual information in an image. This is achieved by using two modules: The Feature Pyramid Attention module provides pixel-level attention information and increases the receptive field by utilising a pyramid structure, and the Global Attention Up-sample module exploits high-level feature maps to guide low-level features recovering pixel localisation. By effectively integrating attention mechanisms at different scales, the PAN architecture enables the network to capture meaningful context and enhance its power for image classification, object detection, and semantic segmentation. ⁶⁴

LinkNet

LinkNet was initially designed to address the requirements of real-time semantic segmentation of visual scenes. It follows an architectural paradigm to that of U-net and other segmentation networks, featuring an encoder on the left and a decoder on the right. The achievement of temporal efficiency in LinkNet is due to the specific interconnections established between the encoder and the decoder. This entails selectively directing the input of each encoder layer to its corresponding decoder output, thereby facilitating the segmentation process. Because of this strategy, the decoder can function with reduced parametrisation in each layer, contributing to improved temporal efficiency. ⁶⁵ LinkNet was employed in Araújo et al ⁶⁶ for the segmentation of melanoma skin cancer in combination with transfer learning and fine tuning. Although LinkNet makes efficient use of parameters and a number of operations in real-time applications, it lags behind U-net in accuracy.

DeepLabV3 and DeepLabV3+

DeepLabV3 is a state-of-the-art model in the domain of semantic segmentation; it utilises atrous convolution and post-processing via fully connected conditional random fields. ⁶⁷ Using atrous convolution, DeepLabV3 achieves an expansion of the receptive field of filters without increasing the number of parameters, facilitating the acquisition of local and global contextual information. This is accomplished through the introduction of voids within convolutional kernels, consequently regulating the spacing between the values utilised for convolutional computation. Dilated convolutions support exponential expansion of the receptive field without loss of resolution or coverage and provide explicit control over the resolution of features generated by CNNs. ⁶⁸

DeepLabV3+ expands on the foundations of DeepLabV3 by incorporating an encoder-decoder architecture. This architecture employs DeepLabV3 as an encoder module and a simple yet effective decoder module. The encoder module handles multiscale contextual information by employing atrous convolutions across various scales. In contrast, the decoder module further enhances the precision of segmentation outcomes in relation to object boundaries. ⁶⁹

Methods

For our experiments, we selected image sequences featuring CLL cells isolated from a single patient, which were captured using brightfield time-lapse microscopy. A detailed description of the conducted experiments and the resulting image data are presented below.

Sample preparation and image acquisition

Peripheral blood mononuclear cells were isolated by density gradient centrifugation from a patient with a very high percentage (> 95%) of lymphocytes. CLL cells were identified and then sorted by gating for CD19⁺, with > 99% purity. Small (s-CLL) and large (l-CLL) cells were sorted by gating by the size of the smallest and largest portions of CLL cells in the CLL population (20% each). From 2.5 million CLL cells, approximately 350 000 s-CLL and 250 000 l-CLL cells were sorted and cultured in complete foetal bovine serum, 2 mM L-glutamine, supplemented with 10% heat-inactivated foetal bovine serum, 2 mM L-glutamine, 100 U mL⁻¹ penicillin and 100 µg mL⁻¹ streptomycin. Cells s-CLL and l-CLL were treated separately with CpG (3 µg mL⁻¹), cultured in a climate-controlled chamber (37 °C and 5% CO₂) for 24 h and then placed on a heated microscopy stage (37 °C and 5% CO₂). Cell polarisation and motility were monitored by time-lapse video microscopy of cells for 60 min and recorded using Gen5 software and a BioTec Cytation 5 reader at a final magnification of 200× (Figure 1). The camera used was a Blackfly BFLY-U3-23S6M, with an objective Olympus lens (size: 20; numerical aperture: 0.45; objective PSF Sigma: 0.806) and an image size of 1224 × 904 pixels.

Figure 1.

Brief description of the input image data. The appearance of typical chronic lymphocytic leukaemia cells: (orange) rounded cell with a blurry dark edge, brighter inner ring part, and darker middle; (blue) nontypical: non-rounded shape (red).

Comparison of state-of-the-art neural network architectures for semantic segmentation of CLL cells

In this study, our primary aim was to detect CLL cells (rounded, touching low-contrast objects) from brightfield time-lapse microscopy images. To accomplish this, we conducted a comprehensive analysis utilising a variety of neural network architectures. Specifically, the neural networks evaluated in our study included the ones described in Section 3; namely U-net, U-net++, the Multiscale attention Net (MA-Net), LinkNet, the FPN, DeepLabV3, DeepLabV3+ and the Pyramid Attention Network (PAN). The purpose of utilising these architectures was to facilitate the accurate identification and localisation of CLL cells within the acquired BF images without distinguishing individual cells in the case of their contact.

The training process involved training each of these networks for 1000 epochs using PyTorch and PyTorch Lightning libraries. Using segmentation models, ⁷⁰ we incorporated various backbone encoders into neural networks, namely InceptionResNetV2, ⁷¹ ResNeSt-269, ⁷² ResNet-152, ⁴⁶ ResNeXt-101_32x32d, ⁷³ and VGG19. ⁷⁴

Compared variables

Loss function

The Dice coefficient, or the Dice-Sørensen coefficient, is a common metric for pixel segmentation that can also be modified to act as a loss function:

$$DCS=\frac{2\left|X\cap Y\right|}{\left|X\right|+\left|Y\right|}.$$

The Dice-Sørensen coefficient is calculated by taking twice the count of elements shared by both sets and dividing it by the sum of the elements in each set. In general, the purpose of loss functions is to compute a metric that the model should try to minimise throughout the training process.

Intersection over union metric

In image segmentation, intersection over union (IoU) is a primary metric for evaluating model accuracy. The IoU is the intersection ratio of the area between the ground truth (GT) and the segmentation mask (TP = true positive) to the union of TP, the predicted area outside the ground truth (FP = false positive), and the number of pixels in the GT area not predicted by the model (FN = false negative):

$$IoU=\frac{TP}{TP+FP+FN}.$$

We calculate IoU in two ways: the per-image IoU and the data set IoU. The per-image IoU means that we first calculate the IoU score for each image and then compute the mean of these scores. IoU of the data set means that we aggregate the intersection and union over the whole data set and then calculate the IoU score. The difference between IoU per data set and the IoU per image can be observed in data sets with ‘empty’ images (ie, images without a target class). Empty images have a large influence on the IoU per image and much less influence on the IoU of the data set.

Training

The graphics processing unit hardware used for this test was an NVIDIA V100 with 32 GB of VRAM. ⁷⁵ The original CLL images had a resolution of 1224 × 904 pixels with a colour depth of 16 bits; for labelling and training, we chose 224 × 224 pixel cutouts. Each network was run for 1000 epochs, and there was no restriction on the time it took to train.

Proposed workflow for instance segmentation of CLL cells and other lymphoblast-like cells

The preceding investigation aimed to identify the optimal neural network to effectively detect CLL cells in brightfield microscopy. The process involved semantic segmentation, where each pixel within the images was assigned specific values based on the probability of being part of the target object or the background. To achieve instance segmentation, which is particularly advantageous in this context (as it enables differentiation between touching and overlapping objects), requires another step in the post-processing stage. Since the methods for achieving instance-based cell segmentation depend on the annotation of the data itself, an overview of three types of workflows is given below (Figure 2).

Figure 2.

The workflow was tailored to processing and evaluating brightfield microscopy images of CLL cells and cells with similar morphology. The input comprises microscopy images or image sequences, the dimensions of which require adjustment depending on the semantic segmentation algorithm employed. Image cutouts with dimensions of 224 × 224 pixels and 256 × 256 pixels (StarDist) were utilised for neural network training. Preprocessing entails image normalisation and augmentation. Masks are prepared based on the segmentation type: 2-class, 3-class, instance. Post-processing involves employing traditional image processing methods such as thresholding and the watershed algorithm.

First step–data annotation

There are several procedures available to separate individual segmented cells, some of which require specific preprocessing steps. These steps influence cell segmentation during the annotation of image data, the initial phase of image segmentation. In this study, we used three types of data annotation, each tailored to the subsequent cell segmentation procedure. Basic annotation involves creating a binary mask, distinguishing between two classes: black representing the background and white representing cells. By slightly modifying this mask (adding a 1-pixel border between cells and then algorithmically outlining each cell), a mask of three classes was created (green – cell, red – background, blue – boundary). The output of a neural network trained on such data comprises predictions for these three classes, including the cell-background boundary. The last type is the annotation of each cell separately, that is, each cell corresponds to a different pixel value. To create these masks, it is possible to use freely available tools and plugins, eg, Labkit in ImageJ, ⁷⁶ AnnotatorJ, ⁷⁷ or QuPath. ⁷⁸

Instance segmentation methods

In this study, we used three types of approaches to perform instance-based cell image segmentation. The first is the application of traditional image analysis algorithms to the segmented image through 2-class segmentation (Section 4 Materials and methods), the second is the application of these algorithms to the data obtained through 3-class segmentation, and the third is the use of the StarDist detector, ³⁸ which uses a lightweight neural network based on U-net and works similarly to most detection algorithms; however, instead of axis-aligned bounding boxes, it predicts a star-convex polygon for each pixel to represent the shapes. StarDist is widely used to detect convex-shaped objects, such as cell nuclei. It is available as a library written in Python and as a tool in Fiji/ImageJ, ⁷⁹ where it is also part of the TrackMate ⁸⁰ cell tracking tool.

A typical algorithm used in post-processing to separate objects is the watershed algorithm, which interprets pixel values as local topography or elevation, facilitating the demarcation of object boundaries within images. Although the output of a 2-class segmentation framework can be post-processed using this algorithm, more precise results were obtained through 3-class segmentation (Figure 3).

Figure 3.

Proposed workflow containing post-processing steps leading to instance segmentation of CLL cells with utilising U-net++. The watershed algorithm was used on the output of the U-net++ model. The result revealed that certain cell boundaries were not detected with the necessary precision, potentially impacting subsequent analyses. Consequently, we decided to adopt an alternative approach: the three-class segmentation method. In this scenario, we found that the boundaries of touching cells were more effectively resolved.

The resulting segmented images were then processed using Fiji/ImageJ software. Since the data set contained time series content, ie, images taken through a device designed for time-lapse microscopy, the Fiji/ImageJ plugin TrackMate with integrated threshold object detector was used to obtain cellular characteristics (area, circularity, solidity, perimeter, radius, and shape index). The input was a sequence of 47 images in .png format of 1280 × 928 pixels, converted to 8-bit depth in binary format, and the output is a .csv table with the positions of detected cells and their characteristic. The post-processing pipeline of U-net++ 2-class and U-net++ 3-class segmentation outputs is depicted in Figure 4.

Figure 4.

Post-processing pipeline. Difference in procedure when using 2-class and 3-class prediction. In the case of 3-class prediction, the output is an RGB image in which the cell boundaries are already detected during prediction, which facilitates post-processing because it is not necessary to use cell separation algorithms, in this case the watershed algorithm, which creates a 1-pixel gap between cells.

Results

The following section summarises the results of both parts of the study, the first is the selection of the most appropriate architecture for semantic segmentation, followed by a comparison of approaches to creating instance-based cell segmentation and a comparison with the widely used StarDist tool.

Comparison of different neural network architectures for semantic segmentation of CLL cells

Although some of the network architectures converged to the solutions faster than others, as seen in Figure 5, where the FPN network has a lower loss function value faster than other networks, and the same can be observed in Figure 6.

Figure 5.

Training loss per epoch for each network.

Figure 6.

Validation loss for each epoch.

Finally, U-net++ had the best results in terms of the loss function and all IoU metrics. From the loss graphs, the IoU metric chart (Figure 7) and the results in Table 1, U-net++ provides the best results, followed by U-net and MA-Net. U-net, U-net++, and MA-Net were primarily designed for medical image segmentation. It was found that the attention mechanisms applied in the MA-Net do not improve the final score in the segmentation of CLL cells. However, MA-Net outperformed U-net in the Test Dataset-IoU metric with the ResNet-152 backbone, where it provided the second-best result. It also has the advantage of having shorter training times than U-net++, but this metric does not play a significant role in the case of this application. The optimal combination was U-net++ architecture with ResNeSt-269 backbone, with an IoU score of 0.8902.

Figure 7.

Best dataset-IoU score for architecture and backbone.

Table 1.

The results for the backbone resnet152.

Network	Parameters	Training time	Loss	Test data set
	(millions)	(mm: ss)		IoU (%)
U-net	67.2	21:41	0.0165	87.21
U-net++	83.6	30:10	0.0149	88.88
MA-Net	182	25:31	0.0242	87.37
LinkNet	65.8	21:02	0.0517	86.76
FPN	60.8	15:39	0.0272	84.01
DeepLabV3	74.3	18:49	0.0572	75.57
DeepLabV3+	61.3	48:33	0.0394	83.50
PAN	58.9	16:46	0.0581	80.35

An example of the input and neural network output can be seen in Figure 8, where outputs from both DeepLab architectures have blurry outputs, and LinkNet has some undesired artefacts, which can be corrected in post-processing. A single image slice of several cells was selected and then compared across all outputs. The highlighted values represent the best result in each measured category.

Figure 8.

Comparison of the detail of the image in the output from the neural networks used to detect CLL cells in the brightfield time-lapse microscopy image. In the magnified images, the different segmentations of this detail can be observed, as well as the cell boundaries; yellow was added to make the details more visible. It is desirable that the neural network detects pixels belonging to cells with the highest possible probability, that there is the most obvious transition between the cell and the background.

Comparison of different approaches for instance segmentation of CLL cells

For the characteristics examined (area, circularity, solidity, perimeter, radius, and shape index), the difference in mean value using different CLL cell segmentation approaches appeared to be statistically significant (Mann–Whitney U test, P < .0001). Figure 9 shows the distribution of the values of the different cell characteristics using three different approaches, 2-class U-net++, 3-class U-net++, StarDist (custom model). From these graphs, the differences are mainly in the distribution of outliers. Another finding is the characteristic ‘CIRCULARITY’, which acquires a small variance when using the StarDist detector, while its variance is larger when using other post-processing procedures.

Figure 9.

Boxplots showing the variability of a given cell parameter (area, circularity, solidity, perimeter, radius, shape index) when using a specific method, namely U-net++ 2-class segmentation, U-net++ 3-class segmentation and StarDist (custom model).

Subsequently, the effectiveness of each approach to fit the manually labelled data was compared (GT). Individual approaches were found to differ primarily in the nature of cell segmentation, not in the overall IoU score (Figure 10). Individual approaches were also compared with each other.

Figure 10.

IoU visualisation comparing predictions against manually labelled cells (ground truth). When compared, we found that overall the algorithms demonstrate equal agreement with GT, but with the comparison it can be seen that the different approaches prefer different morphological features of the cells, ie, StarDist due to its primary purpose of detecting round (convex) shapes preferred round cells, while 3-class segmentation better defined irregular shapes. The difference between 2-class segmentation with watershed and 3-class segmentation is largely due to the fact that watershed separates cells from each other using a 1-pixel gap, which however does not always reflect the actual cell shape, unlike 3-class segmentation, where the gaps are already present in the training data.

Discussion

Deep learning has exerted substantial influence across multiple domains in the realm of image processing, including computer vision, image classification, semantic segmentation, instance segmentation, and panoptic segmentation. In the domain of medical imaging, as well as cell segmentation and tracking, CNN and deep learning models have demonstrated their ability to achieve state-of-the-art accuracy, primarily by effectively harnessing both local and global image features. ⁷¹

Our methodology for segmenting time-lapse microscopy images of CLL cells involved a comprehensive evaluation of state-of-the-art neural network architectures, resulting in the selection of the optimal model for segmenting CLL cells. This chosen methodology also outlines procedures applicable to segmenting time-lapse images of other cell types in brightfield microscopy, with the primary focus being on accurately defining cell boundaries, a prerequisite for successfully determining their characteristics. Furthermore, we investigated different network backbones to identify the optimal configuration, ultimately finding that the combination of the U-net++ network with the ResNeSt-269 backbone was the most effective for our specific application. The choice of the appropriate method for cell segmentation depends on the quality and quantity of the input data but also on the experience of the evaluator. In this case, the cells have a relatively regular, circular to ellipsoidal shape but vary in size and contain irregularities. However, irregularities may have a significant effect on the evaluation of the experiments performed. For these reasons, accurate identification of object shapes in microscopic images is essential. The workflows mentioned demonstrate the possibilities to segment this type of image data, ie, to separate individual cells as accurately as possible, which is also important for subsequent monitoring of cell motility and migration.

The effectiveness of the proposed methods is based on the following points.

• (a) By comparing the combinations of backbone and advanced deep learning architectures for image semantic segmentation, the most efficient was selected and used in the two proposed methods.

• (b) Then 3-class segmentation was applied to the proposed method, where the pixels of the training data were segmented into object (cell), cell boundary and background classes. The image data labelled in this way was then used for training.

• (c) 3-class segmentation, in contrast to 2-class segmentation, shows a more precise separation of individual cells, ie, creating instances without the need to apply an additional post-processing algorithm, for example, watershed, which is not always efficient.

Conclusion

The result of the whole study is a comparison of methods that can assign characteristics to image data/cells, which can then be used in the study of cell migration. The main objective of the study is to show differences in cell segmentation using different approaches and how they can affect the resulting measurements. Our next objective is to develop an optimal algorithm for tracking CLL cells. Subsequent research will investigate the potential of CLL cellular characteristics to predict disease progression, which could be beneficial for precision medicine. The difference in the mean value of some cellular parameters using the different approaches to segmentation was found to be statistically significant; however, the different approaches differed in the nature of cell segmentation, not in the overall IoU score.