Split and Merge Watershed: a two-step method for cell segmentation in fluorescence microscopy images

Abstract

The development of advanced techniques in medical imaging has allowed scanning of the human body to microscopic levels, making research on cell behavior more complex and more in-depth. Recent studies have focused on cellular heterogeneity since cell-to-cell differences are always present in the cell population and this variability contains valuable information. However, identifying each cell is not an easy task because, in the images acquired from the microscope, there are clusters of cells that are touching one another. Therefore, the segmentation stage is a problem of considerable difficulty in cell image processing.

Although several methods for cell segmentation are described in the literature, they have drawbacks in terms of over-segmentation, under-segmentation or misidentification. Consequently, our main motivation in studying cell segmentation was to develop a new method to achieve a good tradeoff between accurately identifying all relevant elements and not inserting segmentation artifacts.

This article presents a new method for cell segmentation in fluorescence microscopy images. The proposed approach combines the well-known Marker-Controlled Watershed algorithm (MC-Watershed) with a new, two-step method based on Watershed, Split and Merge Watershed (SM-Watershed): in the first step, or split phase, the algorithm identifies the clusters using inherent characteristics of the cell, such as size and convexity, and separates them using watershed. In the second step, or the merge stage, it identifies the over-segmented regions using proper features of the cells and eliminates the divisions. Before applying our two-step method, the input image is first preprocessed, and the MC-Watershed algorithm is used to generate an initial segmented image. However, this initial result may not be suitable for subsequent tasks, such as cell count or feature extraction, because not all cells are separated, and some cells may be mistakenly confused with the background. Thus, our proposal corrects this issue with its two-step process, reaching a high performance, a suitable tradeoff between over-segmentation and under-segmentation and preserving the shape of the cell, without the need of any labeled data or relying on machine learning processes. The latter is advantageous over state-of-the-art techniques that in order to achieve similar results require labeled data, which may not be available for all of the domains. Two cell datasets were used to validate this approach, and the results were compared with other methods in the literature, using traditional metrics and quality visual assessment. We obtained 90% of average visual accuracy and an F-index higher than 80%. This proposal outperforms other techniques for cell separation, achieving an acceptable balance between over-segmentation and under-segmentation, which makes it suitable for several applications in cell identification, such as virus infection analysis, high-content cell screening, drug discovery, and morphometry.

Graphical abstract

1. Introduction

Progress on the development of health-related technologies has allowed insights to be obtained from the inside the human body (from entire systems to microscopic cells) that in the past were unimaginable. Different microscopy technologies allow biologists to collect large amounts of images each containing thousands of cells. Although many studies at the cellular level focus on the population as a whole, one challenge of biology is to understand how each cell responds to certain processes and disturbances.

Single-cell heterogeneity in cell populations is a relevant factor in diverse fields, including virus infection analysis. Several types of studies have demonstrated that components of observed cellular heterogeneity contain meaningful information. Researchers have associated features at the level of individual cells with virus infections [1] [2], and the study of phenotypic variability requires measurements from single cells and not just information on population averages [3]. Cellular heterogeneity studies require advanced imaging techniques and algorithms for image processing (including a suitable segmentation method), which allow significant data to be extracted from each cell.

High-content Screening (HCS) is a technique that offers solutions for the study of phenotypic variability. HCS is an approach that allows simultaneous multi-dimensional image analysis in microscopy. It optimizes image processing and reduces the analysis time, increasing the number of analyzed cells and images [4], and, consequently, has been used to industrialize cell biology. Optimized image processing increases the number of cells analyzed and, as a result, generates more significant and multifaceted cell identification data. HCS is a platform that allows the morphological features of cells to be extracted, for example in studies of virus-cell interactions, the behavior of cellular subpopulations, and drug discovery [5]. This tool is widely used in cell biology, medical research, and morphometry. Automated microscopy in high-content screening uses several detection methods. One approach is to stain cells with fluorescent probes, such as those that recognize DNA or actin, and then it is possible to detect fluorescent structures against a less fluorescent background. Other studies [6] have evaluated phenotypic parameters using HCS, and have analyzed images to measure different features of cells in cancer samples, including cell count, area, and intensity.

Even though these microscopic technologies offer high-resolution images, the availability of large datasets and the lack of easily-observable patterns for human experts demand the implementation of digital image analysis techniques for automated cell identification [7]. Algorithms for analyzing cell images continue to improve, and software for detecting and characterizing cells is becoming even more sophisticated. However, the field of cell image analysis poses particular challenges with regards to the segmentation process. Images obtained from microscopy show not only isolated cells but hundreds of cells touching one another, causing standard image processing algorithms to fail in segmenting cells correctly [8][9]. Cell segmentation has important applications, such as quantification of cells and for identification and characterization, for example in virus infection analysis of cells. A real cell separation yields accurate results and reliable analysis, but a poor segmentation can lead to misinterpretation of the images. Consequently, it is necessary that this first stage in the cell image analysis is performed accurately, with correct cell identification, individual cell separations, a low noise index, and a low over-segmentation rate.

Many techniques have been proposed for cell segmentation and each method is fitted to the type of cell. The most common techniques are those based on thresholds and on the Watershed transform, which has drawbacks such as over-segmentation. Other algorithms described in the literature have been proposed for improving the accuracy of the segmentation process, including improved Watershed algorithms [10], active contours and level sets [11] [12], clustering [13] and even deep neural networks [14]. Commonly, the evaluations accompanying such methods report satisfactory performance. Nevertheless, most authors use indicators that do not necessarily reflect the desired behavior/performance required for later applications of the technique. There are different quality features in medical image segmentation that define the categories of segmentation errors [15] In the case of cell image segmentation, the quality of the processed image with respect to cell-by-cell separation, and not just background-cell segmentation (as for most existing methods) is of great importance, since this result could affect statistical studies or subsequent classification processes.

The segmentation method described in this paper was developed to be used by biologists in practice for the research project described in [16]. This study aimed to determine whether the population context influences the cell-to-cell variability in Respiratory Syncytial Virus (RSV) infection using fluorescence microscopy images. In order to support such study, it was essential to have accurate enough information from each cell, as the research required data about the area, shape, and location of individual cells. Consequently, the algorithms implemented in the cellular segmentation processes, despite being efficient for tasks such as counting, exhibited limitations in the correct cell by cell separation and affected the original shape of the cell. For this reason, a new method being able to overcome the limitations existing techniques was required.

In this paper, we propose a new method for cell segmentation in fluorescence microscopy images. Compared to previous works, our method comprises five main contributions. First, the method combines the MC-Watershed, which identifies the cells from the background, and SM-Watershed, which separates the clumped cells, providing better performance in the cell segmentation task. Second, the SM-Watershed method performs a selective separation of clumped cells, using the information on their inherent characteristics, which is a simple way to identify touching cells. Next, the Watershed transform is fitted and applied only to the clumped cells, reducing computational operations. The issue of over-segmentation in Watershed is addressed using information about cell morphology to eliminate incorrect divisions. Third, the two stages of SM-Watershed (split and merge) generate a suitable tradeoff between over-segmentation and under-segmentation. Four, our method does not use an ellipse or geometric fitting, adapting to the shape of the cell. Finally, the method does not require training processes whereby the amount of training data is not an important factor in the performance of the algorithms and it saves training time.

The new method was assessed with some commonly used performance indicators, such as accuracy, precision, recall, and F-index, but additionally, the measurements that aim to evaluate segmentation methods using relevant aspects for the final users of the type of technique were also considered. Specifically, we obtained the ratio of over-segmentation, under-segmentation, noise, and non-detected cells, by cell-to-cell visual inspection with two public datasets. The results were compared with CellProfiler [8] segmentation and the Marker-Controlled Watershed (MC-Watershed) algorithm [10][17]. Additionally, we compared this new method with other methods proposed in the literature. Experimental results revealed that the new SM-Watershed algorithm compares favorably when considering classical techniques, and it performs slightly worse than deep-learning based methodologies. Although this may seem discouraging, it is the other way around: we have developed a methodology that does not rely on training (data fitting/modeling) processes, does not need of data labeled by experts, can be fully understood by the final users and obtains acceptable performance in cell segmentation. The proposed methodology is being used in practice by biologists in the aforementioned project [16].

The remainder of this paper is organized as follows. The main steps of the new cell segmentation method are described in detail in section §2. Materials and Methods. After the description of the method, we expose the main results in section §3. Results and Discussion. Finally, the conclusions of the study are given.

1.1. Related Work

Contributions on cell segmentation can be divided into conventional and deep learning based approaches. On the one hand, traditional methods implement classical image processing techniques, such as intensity thresholding, morphological operations, Watershed transform, active contours, graph-based methods, supervised classification of pixels and clustering techniques. These methods have achieved high performance in the cell segmentation task, but the main challenge is to deal with over-segmentation or under-segmentation. A desired feature of this sort of methods is that, in general, they do not rely on models learned from data; therefore, they can work similarly for domains/datasets formed by a handful of images. On the other hand, deep learning approaches learn a model to segment cell images starting from large amounts of data. These methods often outperform traditional techniques in benchmark datasets in terms of cells correctly separated and a balance between over-segmentation and under-segmentation. However, these methods are computationally demanding, require large amounts of data, and rely on data fitting processes. Whenever enough data is available, deep learning based methods could be a good option. However, in general, a method not requiring large amounts of data is preferred. Moreover, such methods are black boxes that cannot be interpreted; this is a critical aspect in cellular biology, as experts will not rely on a method they cannot understand.

Watershed has been widely used for cell segmentation, but watershed-based algorithms tend to under-segmentation when cells have high overlapping or over-segmentation when cells have different shapes and sizes. The study presented in [18] suggested a technique based on the Watershed transform and elliptic adjustment, to detect automatically HEp-2 cells with diverse patterns. It does not need to segment cells with great accuracy, but it must help to extract features to support cell classification. The proposed method demonstrated a better segmentation performance when compared to the CellProfiler software in an evaluation regarding some volumetric metrics, with a volume overlap of 89%. Also, this method gave a lower number of miss-segmented cells than CellProfiler. However, a major limitation of this work is that it fails to fit in cells with irregular shapes.

Similarly, other authors have proposed the segmentation of HEp-2 cells using an adaptive Marker-Controlled Watershed algorithm [10]. The automatic marker selection involved several steps: an adaptive fuzzy c-means clustering was used to roughly separate the foreground from the background; then, the cell clusters were located, and a Randomized Hough Transform was applied to separate the objects. After this procedure, internal markers defined the positions of the individual cells, which allowed extraction of the external labels fixing the location of the background. Overall, the results had a precision of 89.0%, a recall of 63.9% and an F-index value of 74.4%. About the qualitative visual assessment, the algorithm was able to correctly identify about 80% of the cells and clustered cells were successfully separated in 83% of the cases. One advantage of this proposal is the adaptation of the parameters to the characteristics of the input image in the preprocessing and in the MC- Watershed stages. However, the geometrical adaptation not always is suitable for cell datasets with irregular shapes and the MC-Watershed algorithm has limited performance regard to cell separation.

Gradient computing and contour extraction for cell segmentation have also been studied [19]. Overlapping cells were dealt with by applying Watershed and distance transforms. A genetic algorithm-based parameter detector (GAPD) determined the most suitable values of the parameters. This approach obtained on average a precision of 96%, an accuracy of 85%, and a sensitivity of 87% in uniform and fleck pattern images used as test data. The over-segmentation was minimized by using only information about the cellular area, which may have caused errors in cell merging.

Although methods using Watershed are popular in cell image analysis, other works have opted for alternative solutions. One approach consists of the core phases based on polygon approximation, bottleneck detection, and ellipse fitting [20]. Initially, a binary image with the contours of the cells is obtained and a polygon approximation is carried out to obtain characteristic points. For each connected region, the algorithm determines if there are one or more cells grouped by an elliptical adjustment and detection of the bottleneck. Finally, the cells are separated at the dividing points, and the edge is modified to close the contour. This method separated overlapping cells, and the accuracies for blood and fluorescence cell segmentations were 92% and 90%, respectively. Additionally, other indicators obtained visually, regarding the number of cells correctly segmented, were shown in this paper and comparisons showed that this method outperformed other reported methods. Nevertheless, an elliptic adjustment is not always suitable for representing the shape of a cell because of its uneven appearance.

An alternative approach was a segmentation method based on active contours, which detects objects with boundaries not defined by the gradient [21]. The authors obtained a recall of 80.8%, a precision of 89.6%, and an F-index of 84.4%. However, the segmentation pipeline described in this study was unable to split all overlapping cells as the main goal of such work was cell classification and the presence of cell clusters in the segmentation was not a limitation for classification.

Another technique that has been used for cell segmentation is the multiphase level set method, combined with the iterative voting method. The drawbacks of this approach were the time complexity, space complexity, possible local minima and poor initialization [22]. When the number of cells increased, the operation time increased and the requirement for memory also grew. Therefore, this method is not appropriate for a real-time application. The Chan-Vese model was developed to deal with these limitations. However, if the gray-level of cells is similar to the background, it is difficult to identify them. Moreover, the Chan-Vese Model had problems segmenting cells located too close together or overlapping [23].

New techniques based on deep learning have also been applied to cell image segmentation, for example, U-Net [24] and DeepCell [25], which are based on convolutional neural networks. These architectures have demonstrated high performance in the cell identification task. In the current paper, we compare our new method with both of these deep learning strategies. Traditional cell segmentation approaches often require parameter tuning, while deep convolutional neural networks (DCNN) require training data [26] and these models generally have a high computational cost. DCNN training requires large amounts of high-quality labeled samples, which could be a tedious and expensive task.

Despite the amount of research in the cell segmentation field, the accurate segmentation of cells is still challenging, and most of the proposed techniques have drawbacks, with either over-segmentation or under-segmentation problems or some others require minimum levels of training data. These weaknesses motivated the current research. In this work, we focus on fluorescence microscopy images introducing a new cell segmentation method based on a split and merge process and Watershed transform. Our algorithm reached a suitable tradeoff between over-segmentation and under-segmentation as none of the aforementioned works does, without relying on the training process.

2. Materials and Methods

2.1. Dataset

To validate our proposed method, two public databases were used: the SNP HEp-2 Cell Dataset [27] and the fluorescence cell database shared by Coelho et al. [28]. Examples from these two databases are shown in Fig. 1.

Figure 1.

Cell images. First row: examples from the SNP dataset. Second row: examples from the U2OS dataset.

The SNP HEp-2 Cell Dataset (SNPHEp-2) [27] was obtained between January and February 2012 at Sullivan Nicolaides Pathology Laboratory, Australia. The SNPHEp-2 specimen images were captured using a monochrome high dynamic range microscopy camera, with a plan-Apochromat 20×/0.8 objective lens and an LED illumination source [27]. The dataset contains images of five cell classes: centromere, coarse speckled, fine speckled, homogeneous, and nucleolar; and consists of 1,884 cell images extracted from 40 specimen images. A DAPI image channel was used to obtain the cell image masks automatically. With the aim of validating our method, 40 cell images from the homogeneous class were chosen randomly, as this was the cell line used in reference [16], from which our method was developed.

The dataset collected by Coelho et al. consists of 48 U2OS cell images with a size of 1340×1030 pixels. They were obtained using a 40×/0.9 numerical aperture (NA) water immersion objective. High throughput microscopy was used to image living cells; details about tissue culture and high-content imaging are given in reference [29]. In our study, 40 fluorescence cell images were chosen randomly and the results compared with those from previous studies that used this dataset. The human osteosarcoma U2OS cell line was one of the first cell lines generated, and it is used in many different areas of research [30].

In this work, we developed algorithms in Matlab® 2014-B and carried out the statistical validation with IBM SPSS Statistic. The source code and the results obtained during the current research are available from the corresponding author upon request.

2.2. Watershed Transform

Our method for cell segmentation in fluorescence microscopy images was based on a Watershed transform approach, which is generally used in image segmentation processes. The Watershed transform is the most popular region-based method used for the segmentation task. The term watershed comes from geography: it refers to a ridge line that divides areas drained by different river systems [31]. The frontiers of the dams are called watershed lines [32] [33]. The Watershed transform can be applied to gradient magnitude images, gray intensity images or distance transform maps. The definition of Watershed transform for the continuous case is based on distance functions and has been described in detail [33].

The Marker-Controlled version of Watershed extracts a set of markers for the positions of the objects in the image, to reduce over-segmentation (a drawback of Watershed) [34]. Morphological operations are necessary to obtain good markers. Nevertheless, the automatic extraction of markers is still a challenge [10] [35].

2.3. Mathematical Morphology

Basic morphological operations are used in the MC-Watershed process: erosion, dilatation, opening, and closing. We followed the definition for these processes given in reference [36]. These operations were used to obtain good foreground markers and to segment them from the background in the MC-Watershed algorithm.

The morphological opening is erosion followed by dilation, and morphological closing is dilation followed by erosion. In both cases, the accuracy of the image restoration depends on the connection between the figures and the structuring element. Reconstruction is a morphological transformation including a structuring element (defines the connectivity), the marker image (starting point), and the mask image (constrains the transformation). If we combine opening and closing with reconstruction, these procedures restore precisely the forms of the objects that remain after erosion/dilation using the mask image. Reconstruction-based opening and closing operations are better than the regular opening and closing operations in eliminating small imperfections without affecting the shapes of the objects.

2.4. MC-Watershed

The first stage of our proposed method aims at obtaining an initial (approximate) segmentation: distinguishing between the background and the cells. The goal of this block is to recognize as many cells as possible. A flowchart of this process is shown in Fig. 2. Four stages integrate the MC-Watershed: preprocessing, marking the foreground objects, computing background markers and computing the Watershed transform [36].

Figure 2.

Flowchart of the MC-Watershed algorithm.

The preprocessing stage is necessary to adapt the input image I_input and obtain the markers of foreground and background. The algorithm cleans the borders as the cells there are not complete and it would result in incorrect information for subsequent processes, such as characterization. Next, the algorithm adjusts the intensity, which helps to improve the contrast between the cell and background, giving the adjusted image (I_adj). Morphological operations like opening and closing by reconstruction are performed to enhance the image and obtain the markers (see subsection §2.3. Mathematical Morphology). The output of the preprocessing block is I_o, which is the input to obtain the foreground and background markers.

The next block calculates the regional maxima of I_o to acquire foreground markers. The regional maxima are connected pixels with a constant intensity value and whose external boundary pixels have a lower value. It is necessary to clean the edges and remove the isolated pixels after this operation is performed, giving the image I_F.

A threshold based on Otsu’s method is used to obtain the background markers. Background markers are not suitable if too close to the edges of the cells. Next, the background is thinned by applying the Watershed transform to the distance transform of the internal marker. The resulting image is I_B.

The last stage computes the Watershed transform. This block needs the inputs I_F, I_B and I_adj. The gradient magnitude image is adjusted so that its regional minima occur at foreground and background marker pixels. Then the Watershed transform is computed for this modified gradient. The output of this first step is a segmented image with cells identified in the background (see Fig. 4). The resulting image is a label matrix I_label. However, many cells are not separated and the level of noise is high. Therefore, the second processing with SM-Watershed is necessary.

Figure. 4.

MC-Watershed process. a) Input image I_input. b) Adjusted image I_adj. c) Results of the opening and closing by reconstruction, I_o. d) Regional maxima superimposed on the adjusted image, I_F. e) Thinned background I_B. f) Output image I_label with boundaries on the original image.

2.5. SM-Watershed

The second stage is based on the morphologic characteristics of the cell, in order to achieve the separation of the merged elements. This process is performed once since only one application is necessary to achieve a suitable separation. Additional iterations showed no improvement in the results. A flowchart of this process is shown in Fig. 3. The input I_seg is the image resulting from the MC-Watershed step. This stage has two core processes: Split and Merge.

Figure 3.

Flowchart of the SM-Watershed algorithm.

Step 1: Split.

The split process identifies unseparated cells using information about the area and convexity.

The solidity value of each closed region is defined as:

$$Solidity=\frac{Area}{ConvexArea}$$ (1)

where Area means the area of the closed region and ConvexArea is the smallest convex polygon that contains the closed region. A new binary image (I_s1) is produced that is composed only of elements with an area and solidity bigger than those for typical cells (the threshold for Area, Area_T1, and solidity S1 are fitted from average values for each dataset). A hole-filling operation is implemented to enhance the image since some kinds of cells have little nucleolus, which could cause over-segmentation or misidentification. Next, the Watershed transform is applied to obtain the image I_w. The city block distance transform was selected for this stage. This distance between two points (m₁,n₁) and (m₂,n₂) is obtained as follow:

$$d_{cityblock}\left(\left[m_1,n_1\right],\left[m_2,n_2\right]\right)=\left|m_1-m_2\right|+\left|n_1-n_2\right|$$ (2)

Alternatives to the city block distance were less suited to our needs. The Euclidean distance has a high possibility of over-segmentation. The reason is that Euclidean distance propagates to the neighboring pixels in a shape close to a circle, and it forms a small island made of a few pixels [37]. The chessboard propagation has similar behavior, and in the cell segmentation case, too generates over-segmentation. Thus, the city block distance was the most appropriate for the split stage. Even so, in some cases the split process generates small parts due to the over-segmentation caused by Watershed; it is necessary to remove them to avoid misidentification. The resulting image (I_s) is then used as the input to the second step, the Merge process.

Step 2: Merge.

The objective of this step is to identify the over-segmented regions, using information about area and eccentricity, and to eliminate these divisions with morphological closing.

Eccentricity is the eccentricity of the ellipse that has the same second-moments as the closed region. The eccentricity of the ellipse is the ratio of the distance between the foci of the ellipse and its major axis length and has a value between 0 and 1.

The over-segmented regions are smaller and more eccentric than the typical elements (we fitted the threshold for Area, Area_T1, and eccentricity (Ecc) from average values for each dataset). The final output of the SM-Watershed step is image I_output (See Fig. 5).

Figure. 5.

SM-Watershed process. a) Unseparated cells I_s1 from I_label. b) Watershed transform. Zoom to I_w. c) over-segmented regions. d) Output image I_output.

2.6. Evaluation.

The Hit Rate and the Confusion Matrix are commonly used metrics for assessing the quality of segmentation methods. In the case of image segmentation, several metrics are used to obtain performance indicators [15]. Some measures are based on the number of corrected segmented pixels compared with an image reference or ground truth. These metrics are based on the spatial overlap and result from the four basic cardinalities of the confusion matrix: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). We obtained these cardinalities following the definition in reference [15].

Based on the cardinalities of the confusion matrix, we obtained the following performance indicators: sensitivity or recall, specificity, precision, negative predictive value (NPV), F-Index and accuracy.

Recall specifies the portion of foreground pixels or cell pixels in the ground truth that matches with cells identified by the segmentation method. Similarly, specificity measures the percentage of negative pixels, or background in the ground truth, which are also identified by the segmentation method. However, these two indicators are sensitive to segment size, and they penalize more mistakes in small objects than in large ones [15].

Other measures obtained from the confusion matrix are the precision and NPV, which give an idea about the correct value, positive or negative, as a proportion. However, they are not commonly used in the validation of medical images [15]. However, precision and recall are meaningful when considered jointly. In general, under-segmentation generates high values of precision and low recall, while over-segmentation generates the opposite [10]. An appropriate balance between precision and recall is revealed in higher F-index values. Two other metrics used to assess segmentation are the Dice coefficient and Jaccard Index, defined as:

$$DICE=\frac{2\left|S_g\cap S_t\right|}{\left|S_g\right|+\left|S_t\right|}=\frac{2TP}{2TP+FP+FN}$$ (3)

$$JAC=\frac{\left|S_g\cap S_t\right|}{\left|S_g\cup S_t\right|}=\frac{TP}{TP+FP+FN}$$ (4)

Where S_g is the ground truth segmentation, and S_t is the segmentation being evaluated.

Other authors prefer to use metrics based on the volume of contours [18]. The Volumetric Similarity (VS) is 1 − VD where VD is the volumetric distance. That is:

$$VS=1-\frac{\left|\right|s_t|-|s_g\left|\right|}{\left|s_g\right|+\left|s_t\right|}=1-\frac{|FN-FP|}{2TP+FP+FN}$$ (5)

In the PVD metric, the absolute volume of the segmented region is compared with the corresponding volume in the ground truth. Thus, the overlap between the segments is not considered. The volumetric similarity has its maximum value even when the overlap is zero [15].

Although these indicators have been widely used, these metrics do not offer a measure of the correct separation of clustered cells. As they are based on comparisons with the ground truth, they only give an indication of the separation between background and foreground.

Therefore we used a qualitative visual assessment using the definition of visual accuracy (VAC) [20]:

$$VAC=\frac{N_{segment}}{N_{segment}+N_{segment}+N_{split}+N_{merge}+N_{add}+N_{missing}}$$ (6)

where N_split represents the number of single cells that are divided into several cells by mistake, N_merge the number of overlapping cells assigned to a single cell by mistake, N_add the number of cells added by mistake from the background, N_missing the number of cells assigned to the background by mistake, and N_segment the number of cells correctly segmented.

2.7. Parameters set-up.

In this paper, we present the results for our algorithm using parameters selected by a heuristic methodology. This selection was based on the inherent characteristics of the cells as average area and convexity, and knowledge derived from previous related works.

For the parameters set-up, we selected randomly one image from each dataset. Since images have been acquired with the same magnification, the variations of size and convexity are minimal between images. Therefore, the thresholds for area, solidity, and eccentricity (parameters used in SM-Watershed) are estimated by calculating the average values of all the foreground elements in the image. This is an advantage versus other methods since the parameters can be fixed only once for each dataset acquired with the same magnification. The robustness of our parameter selection was verified by preliminary experiments, where other samples of each dataset were used to obtain average values. We found a non-significant difference in the estimated parameters between each image (variations less than 2%)

Additionally, in a preliminary study [38], the crucial parameters of the MC-Watershed and SM-Watershed algorithms have been tuned by a GA (genetic algorithm) with the aim to optimize its performance. The F-index was the output of the fitness function (the value to optimize by the GA) and the input vector contained the options to be fitted. The test was done with the SNP HEp-2 cell dataset. The entire process of optimization is described in detail in [38]. The results regarding F-index showed that the auto-tuning algorithm reached similar performance to the values generated with the input parameters selected by the human expert: F-index with the heuristic method was 0.85863 and F-index with GA auto-tuning was 0.86309. Taking into account this minimal difference regarding the parameter selection, we can state that the results in this paper are suitable for comparison with related works in the literature.

3. Results and Discussion

In this section, experimental results are presented using cell images from the two datasets described in subsection §2.1. Dataset. This section is divided to show the results for the MC-Watershed and SM-Watershed blocks and the indicators obtained. These results are compared with the standard software for cell processing (CellProfiler) and with other methods from the literature.

3.1. MC-Watershed results.

An example of the results of each operation in the MC-Watershed process, using a sample of the U20S dataset, is presented in Fig. 4. Fig. 4b shows the adjusted image after conversion to grayscale: the elements touching borders were cleaned (see top-right and left borders) and the conversion to grayscale image increased the contrast in the image I_adj. The image I_o (Fig. 4c) is the result of morphological operations and reconstruction. The pixels obtained from regional maxima I_F were superimposed on the adjusted image I_adj (see Fig. 4d). Only the pixels with high intensity surrounded by pixels with low intensity were extracted with regional maxima. Fig. 4e shows division lines obtained from the application of Watershed to the binary background. Fig. 4f shows the output image I_label; cells that have not been separated are highlighted by red ellipses.

3.2. SM-Watershed results.

In Fig. 5, the results of each operation in the SM-Watershed process are presented. The input to this process is the image obtained after the MC-Watershed step (I_label). Fig. 5a is the binary image, containing only the clustered elements. They were detected using information about Area and Solidity. After applying the Watershed transform to these elements, division lines were drawn as shown in Fig. 5b. Over-segmentation caused by Watershed is evident. Each of these regions was identified (Fig. 5c) using information about Area and Eccentricity. Then division lines of over-segmented areas were eliminated.

In Fig. 5d, two samples of correctly separated cells are highlighted. Although the most significant elements were separated, the division line between regions 1 and 2 in Fig. 5c was incorrectly drawn: this division was eliminated, and finally, these two cells were not separated.

3.3. Performance Indicators.

This section is separated into two parts to present the indicators for the two databases. The SNPHep-2 database has been used for classification purposes in the literature, and also to validate the results in the segmentation process. The performance of our proposal was compared with CellProfiler and MC-Watershed. The U2OS dataset was used to compare our new approach with other methods [20][39][40][41][42]. For the qualitative visual assessment, each output image was reviewed and classified manually by PhD-level expert biologists. Classifications were made using the labels from equation (6): N_segment, N_split, N_merge, N_add, N_missing. An inter-rater reliability analysis using the Kappa statistic [43] was performed to determine the level of agreement between observers.

The primary goal of our new method is to achieve a balance between over-segmentation and under-segmentation rates and between misidentified and unidentified cells, with an acceptable accuracy preserving the original shape of the cell. The shapes identified corresponded to the visible part of the cluster since the method did not include any geometrical recuperation. This was adequate for the type of data used, with cells of irregular shape and low overlap.

3.3.1. SNPHEp-2 Indicators.

For this dataset the ground truth was available. Hence, some commonly used metrics presented in section §2.5. Evaluation were first evaluated. Additionally, a qualitative visual assessment was performed. In Fig. 6, two examples of a processed image are presented. The cells indicated with the continuous red circle are examples of the comparison with the ground truth, where the three results gave the same indicators about identification, but cells were not correctly separated. The accurate segmentation of the SM-watershed can only be evaluated with a qualitative visual assessment. Dashed circles show examples of over-segmentation or under-segmentation. Continuous arrows show cells that have been mistakenly added and dashed arrows show cells that were not identified.

Figure 6.

Example of segmentation for the SNPHep-2 dataset. First column: ground truth, second column: CellProfiler, third column: MC-Watershed and fourth column: SM-Watershed.

3.3.1.1. Quantitative assessment for SNPHep-2.

In Table 1, several performance indicators that compare the new SM-Watershed method with CellProfiler and MC Watershed are shown. These indicators do not provide a useful metric for segmentation performance. For example, a method that labels all the pixels as cells and none of the background has a recall of 100% [10]. CellProfiler obtained the highest recall, but in the cells marked with the dashed red circle in Fig. 6, second column, some background pixels were marked as foreground pixels. The F-Index, accuracy, Jaccard, and VS indicators were very similar for MC-Watershed and SM-Watershed. Both of these methods had good performance when compared with the ground truth and they preserved the original shape of the cell. Nevertheless, these indicators do not provide an evaluation of the correct separation of cells.

Table 1.

Performance indicators for SNPHEP-2 dataset

Indicator	CellProfiler	MC-Watershed	SM-Watershed
Recall	0.96648	0.83771	0.74063
Specificity	0.89695	0.98078	0.99158
Precision	0.65309	0.89756	0.94574
NPV	0.99354	0.96672	0.94945
F-Index	0.77158	0.86303	0.82732
Accuracy	0.90849	0.95594	0.94907
DICE	0.78124	0.86542	0.83104
Jaccard	0.64102	0.76276	0.71093
VS	0.77857	0.86197	0.82701

3.3.1.2. Qualitative assessment for SNPHep-2.

In Table 2, we present the results of the qualitative visual assessment. The values presented in Table 2 were obtained from a visual inspection of the output images of each method by PhD-level expert biologists. The new SM-watershed method outperforms the other two methods with regard to percentage accuracy (VAC%); the number of overlapping cells (N_merge) and the noise (or cells obtained from the background) (N_add) were the lowest of the three methods. The SM-watershed method only takes clumped objects, selected by an area or solidity out of average range. Nevertheless, this stage could cause over-segmentation in some well-segmented cells obtained in the MC-watershed stage. Because of this, the indicator for over-segmentation (N_split) was not the lowest of the three methods, but it still had a low overall value. The number of cells not identified (N_missing) was similar for all three methods.

Table 2.

Qualitative Visual Assessment for SNPHEP-2 dataset

Indicator	CellProfiler	MC-Watershed	SM-Watershed
VAC%	83.77%	73.78%	88.97%
Nsegment	92.40%	77.64%	92.19%
Nsplit	2.23%	0.06%	3.28%
Nmerge	2.60%	19.56%	1.67%
Nadd	10.30%	5.23%	3.61%
Nmissing	2.76%	2.74%	2.86%

An ANOVA test was completed to compare the performance of the three methods in the qualitative visual assessment [44]. The Analysis of Variance (ANOVA) gives a statistically significant result indicating that one method differs from the other two [45]. In Table 3, F is the F-statistic, while the p-value, or significance, is the probability of obtaining the F values, or higher, under the null hypothesis of the equality of means. If the p-value associated with the statistical F is less than 0.05, we reject the null hypothesis, and the alternative is accepted, which means that the average values of the three methods are different. Only the indicator N_missing had p>0.05, so for this indicator, the three methods gave similar results.

Table 3.

ANOVA Test

Indicator	F	p-value
VAC	36.857	0.001
Nsplit	34.667	0.001
Nmerge	152.764	0.001
Nadd	19.230	0.001
Nmissing	0.185	0.831

To analyze the pattern of differences between the means of the methods, it was necessary to execute tests of specific hypotheses comparing pairs of means (pairwise comparisons or post hoc test). The Games-Howell post hoc test is a nonparametric method to compare combinations of groups. The Games-Howell test does not presume equal variances and sample sizes. Therefore it is often recommended over other approaches [46]. The results of the Games-Howell test are shown in Table 4.

Table 4.

Games-Howell Test

				95% confidence interval
Indicator	Method(i)	Method(j)	p	Lower bound	Upper bound
VAC	SM	MC CP	0.001 0.001	0.107 0.020	0.197 0.091
	MC	CP	0.001	−0.143	−0.048
Nsplit	SM	MC CP	0.001 0.027	0.025 0.001	0.042 0.025
	MC	CP	0.001	−0.029	−0.012
Nmerge	SM	MC CP	0.001 0.061	−0.214 −0.020	−0.147 0.000
	MC	CP	0.001	0.137	0.204
Nadd	SM	MC CP	0.182 0.001	−0.042 −0.107	0.006 −0.043
	MC	CP	0.001	−0.091	−0.023
Nmissing	SM	MC CP	0.975 0.920	−0.011 −0.015	0.013 0.010
	MC	CP	0.826	−0.016	0.009

If the confidence interval of the difference in means is determined, it is possible to verify if the null hypothesis is rejected, as the confidence interval does not include the value of zero for the difference of means. The null hypothesis states that the difference between the means is zero, therefore if zero is in the range, this hypothesis cannot be rejected. For the indicator N_merge, there was no difference between the results for our SM-Watershed method and CellProfiler. The same behavior was found for N_add if the SM-Watershed method was compared with MC-Watershed. However, it can be seen in Table 2 that SM-Watershed had the lowest values for these two indicators. Although the SM-Watershed method had the same performance as CellProfiler with regards to the N_merge indicator, CellProfiler had a higher indicator of noise (N_add). So, the SM-Watershed method had the best balance between real separation, identification, and noise in the cell segmentation process.

3.3.2. U2OS Indicators.

This dataset contains manually segmented images, with a red boundary for cell separation. From these images, we created a binary ground truth, with the background in black and foreground in white. Next, some commonly used metrics presented in section §2.5. Evaluation were evaluated. Additionally, a qualitative visual assessment was performed.

In Fig. 7, examples of processed images with SM-Watershed are presented. These images indicate that an elliptical or geometric approximation is not suitable since it does not fit the morphology of the cell. Furthermore, it has been demonstrated that the root mean square error of an ellipse fitting increases with the eccentricity value of the point set, which can lead to having wrong information for the size and shape of some cells [47].

Figure 7.

Images show segmentation for the U2OS dataset. These cells are irregular and a geometric fit was not a suitable option.

3.3.2.1. Quantitative assessment for U2OS.

Table 5 shows a quantitative analysis with several performance indicators comparing the new SM-Watershed method with two classical algorithms (the automatic graphcut segmentation with λ adaptive [48] and MC-Watershed) and two state-of-the-art methodologies (deep learning architectures: U-net and DeepCell [14]). The results of the graphcut method were taken from Oyebode et.al [48], the results of the deep learning configurations were taken from the reported data in [14] and the results of MC-Watershed and SM-Watershed were obtained by running the algorithms in Matlab® 2014-B. This quantitative analysis for the U2OS cell line was performed only with these five methods since the other proposal shown in Table 6 do not have quantitative metrics reported.

Table 5.

Performance indicators for U2os dataset

Indicator	DeepCell	U-net	Graphcut	MC-Watershed	SM-Watershed
Recall	0.78641	0.87405	0.80778	0.80567	0.79208
Precision	0.94870	0.92455	0.97646	0.90576	0.93094
F-Index	0.85996	0.89859	0.88415	0.84554	0.85427
Jaccard	0.75433	0.81586	0.79235	0.73866	0.74714

Table 6.

Quality Visual Assesment U2OS dataset

Reference	VAC%	Nsegment	Nsplit	Nmerge	Nadd	Nmissing
Liao et. al [20]	90.22%	1310	1.37%	6.18%	2.06%	1.22%
Huang et. al [39]	82.93%	1258	0.00%	10.65%	0.00%	9.94%
Lim et. al [40]	82.61%	1406	6.33%	11.74%	2.77%	0.21%
Farhan et. al [41]	81.70%	1308	15.60%	3.44%	2.29%	1.07%
Wang et. al [42]	82.65%	1305	14.56%	2.15%	2.76%	1.53%
Graphcut [48]	82.01%	930	0.00%	11.83%	8.71%	1.40%
U-net [14]	91.52%	5185	0.35%	0.87%	2.47%	2.43%
DeepCell [14]	88.02%	4829	0.18%	1.40%	1.52%	4.20%
This study	90.29%	930	2.15%	3.55%	2.8%	2.26%

In this quantitative analysis, the two deep learning based techniques outperformed our SM-watershed method in terms of F-index and Jaccard indicators. F-index detects performance differences better than the Jaccard index [14]. Higher F-index values reveal an appropriate balance between precision and recall. The quantitative results for the classical methods are very similar to SM-watershed.

We can confirm that deep learning architectures had good performance compared with the ground truth and it preserved the original shapes of the cells. Although the deep learning based solutions obtain better performance, our proposed method has additional advantages that make it competitive with such techniques. Namely, it does not require labeled data, it can be applied to datasets for which only a handful of images are available, it does not need training process, and it can be interpreted by domain experts. The latter makes our method a relevant and useful state of the art methodology for cell segmentation in realistic environments.

3.3.2.2. Qualitative assessment for U2OS.

According to authors in [14], from a cell biology imaging perspective, pixel-wise overlap alone is not appropriate to diagnose the errors in the segmentation process, such as over-segmented and merged objects. Thus, we additionally used metrics that explicitly count correctly segmented objects.

Consequently, a qualitative visual assessment was performed as it is proposed in [20] and explained in (6) and the results are shown in Table 6. N_split, N_merge, N_add, and N_missing were used to obtain the percentage of the number of cells correctly segmented (N_segment) with the objective of comparing the findings of this study with alternative methods reported in the literature. The values presented in Table 6 were taken directly from each referenced paper, and the results from our study and the graphcut approach were obtained from a visual inspection of the output images by PhD-level expert biologists. The inter-rater reliability was calculated using the kappa coefficient with the aim of validating the indicators obtained from the visual inspection. As suggested [49], we obtained the kappa coefficient for all coder pairs, then we used the arithmetic mean to provide an overall index of reliability [50]. There was a substantial agreement between the observers’ judgments, with kappa=0.783 (p<0.001), 95% CI (0.757, 0.809).

The method of Liao et. al [20] separated the overlapping cells by implementing bottleneck detection with ellipse fitting. This method had a similar VAC to our SM-Watershed method, but the N_merge value was higher than the SM-Watershed method. Additionally, the ellipse fitting approach is not always suitable, for example in the cases shown in Fig. 7. In the method of Huang et. al [39], the operations erosion and reconstruction were used to separate cells but generated under-segmentation; this can be seen by the high values of N_merge and N_missing. The method of Lim et. al [40] used a watershed algorithm applied to the gradient, which can avoid over-segmentation, but the process fails when the cells are overlapped, and this method had a high N_merge value.

The method of Farhan et. al [41] used concave point detection to separate the overlapping cells. However, this was limited to objects of particular sizes and shapes and it was sensitive to noise. This method had a low N_merge value but a high N_split value, indicating problems with over-segmentation. The method of Wang et. al [42] used a shape classifier to decide whether a connected component should be split. Due to the irregular shapes of the fluorescent cells, this method separated one cell into several elements by mistake, which lead to a higher N_split value.

The graphcut approach used neighbor information to segment foreground and background, allowing identification of all cells and, consequently giving a very low N_missing indicator and 0% over-segmentation. However, the process failed when the cells were overlapped and it added objects to the background, which was reflected in high N_merge and N_add values.

We find that deep learning architectures exhibit higher performance than classical segmentation algorithms in terms of the number of correctly identified objects. The two configurations of deep learning, U-net and DeepCell, had better indicators of N_merge, N_split and N_add than our method and U-net had the highest VAC value of all. Both deep learning methods identify boundaries to separate touching cells, but small cells are not easy to segment using the border detection process because most of the pixels are in the boundary, and few pixels are in the interior [14]. This explains the high values for N_missing in the deep learning configurations: small nuclei were not detected. These results were expected since deep learning implementation has demonstrated high performance in several image detection tasks. Although deep learning models are slightly superior to our SM-watershed method regarding VAC indicator, they generally run at higher computational cost, and more importantly require of large amounts of training data, which is difficult and costly to obtain, and they must be trained, usually with special hardware.

Analyzing both quantitative and qualitative approaches, we can state that in the configuration of parameters in classical algorithms, they have to accept more splits in order to fix mergers or vice-versa. On the contrary, our method tries to compensate these two behaviors in a balanced way.

These results demonstrated that, compared with other classic methods, SM-Watershed had the best balance between separation, identification, and noise in the cell segmentation process, with an acceptable accuracy preserving the original shape of the cell. Also, unlike deep learning architecture, this method does not require training data and the parameters are fitted with the dataset to process.

4. CONCLUSIONS

A novel SM-Watershed method for cell segmentation in fluorescence microscopy images was developed. Satisfactory results were generated with this approach. This method is suitable for cell separation, which allows an appropriate cell-by-cell characterization for complex studies, such as virus infection analysis. First, a Marker-Controlled Watershed algorithm was used to extract the cells from the background. This initial segmented image was the input for the two-stage algorithm of the SMWatershed method. It applies the Split and Merge processes based on the Watershed transform to separate the cells correctly. The split process identifies the clustered cells using fitted features of the cells like area and solidity, and then the distance transform is calculated to apply Watershed. The merge process uses the area and eccentricity to identify the over-segmented regions and employs morphological operations to eliminate the divisions.

The SM-Watershed method was tested on two different databases of fluorescence cell images, which was a good indicator of the reproducibility and quality of the method. A quantitative and qualitative visual assessment was performed to provide appropriate validation of the SM-Watershed method. The results obtained from the indicators showed that this method had high accuracy and a good balance between over-segmentation and under-segmentation. Furthermore, this method follows the irregular shape of the cells since it is not based on a geometric adjustment. Compared with the state-of-the-art methodologies as deep learning architectures, they outperformed our method, but SM-Watershed does not require a training process, which is suitable for databases with reduced data. Even though the SM-Watershed cell segmentation method performed well in identifying cells in a fluorescence image, additional research is needed to improve the indicators for missing cells and the added noise.

Highlights SM-Watershed.

• A new cell segmentation algorithm using split and merge techniques was proposed.

• Morphological features of the cells are used to separate clustered cells.

• The algorithm preserves the shape of the cells and does not need training images.

• SM-Watershed reached a balance between over-segmentation and under-segmentation.