Extraction of target specimens from bioholographic images using interactive graph cuts

Abstract.

It is necessary to extract target specimens from bioholographic images for high-level analysis such as object identification, recognition, and tracking with the advent of application of digital holographic microscopy to transparent or semi-transparent biological specimens. We present an interactive graph cuts approach to segment the needed target specimens in the reconstructed bioholographic images. This method combines both regional and boundary information and is robust to extract targets with weak boundaries. Moreover, this technique can achieve globally optimal results while minimizing an energy function. We provide a convenient user interface, which can easily differentiate the foreground/background for various types of holographic images, as well as a dynamically modified coefficient, which specifies the importance of the regional and boundary information. The extracted results from our scheme have been compared with those from an advanced level-set-based segmentation method using an unbiased comparison algorithm. Experimental results show that this interactive graph cut technique can not only extract different kinds of target specimens in bioholographic images, but also yield good results when there are multiple similar objects in the holographic image or when the object boundaries are very weak.

1. Introduction

Since holography was first presented by Gabor in 1948,¹ three-dimensional (3-D) optical imaging systems based on digital holography^2–6 are widely viewed as a promising approach in life sciences,^7–10 defense and security,^11–13 medical diagnoses,^14–17 robotics,^18,19 and medicine.^20–22 These systems have the strengths of being able to acquire images rapidly, illustrate the amplitude and phase information conveniently, and apply advanced image-processing techniques to complex field data never possible before. The two-dimensional (2-D) bright-field images of transparent or semi-transparent cells resulting from conventional bright-field optical microscopes often have low-contrast and produce limited details of the cell structure for cell analysis.¹⁶ Even though some 2-D imaging systems, such as quantitative phase contrast and differential interference contrast microscopies,^23,24 have the feature to quantitatively investigate the biological microorganisms to some extent, they cannot provide optical thickness information about cells.¹⁶ On the contrary, the holographic images reconstructed numerically from a hologram obtained by digital holography-based imaging systems can provide rich 3-D information, such as the surface morphological and optical thickness data of these microorganisms, which make it possible for the 3-D measurement of objects.^25,26 However, to clearly explore targets in holographic images, one must eliminate unnecessary background information which affects the examination. Furthermore, for higher level image processing such as the kind needed for 3-D object identification, recognition, and tracking, segmentation as a preprocessing step is also inevitable.

Segmentation is a technique of partitioning an image into several regions, so that each region will have similar color, texture, or intensity value.²⁷ For a successful segmentation, all of the needed targets should be separated from the background. Conventionally, segmentation approaches can be broadly categorized into three groups. The first one is the region-based method.^27,28 The typical technique that belongs to this category is a threshold scheme.^27,29 It is the simplest and most efficient algorithm, since it only uses a threshold to divide the image into foreground and background. However, when the object is complex or the intensity values of the foreground and background are similar, this method will fail to get a practical result. Moreover, finding a reasonable threshold is also not easy. Most of the time, this method is used in conjunction with other approaches. Another famous region-based algorithm is region growing and region splitting-merging.²⁷ Nevertheless, these methods suffer from difficulties in finding appropriate stop criteria for region growing or splitting. They will also make the segmented boundary not smooth. The second type is the edge-based segmentation.³⁰ Those well-known algorithms include Sobel, Roberts, Canny, and Hough transformations.²⁷ All these methods require that the boundary between foreground and background should be distinct. They are also very sensitive to noise and easily get many false edges. When discontinuity exists at the edge, they will fail to get an isolated object. Another widely used edge-based method is a watershed transform algorithm.^27,31 It has the advantage in being able to find an isolated target which will satisfy most of the analyzers. On the other hand, the watershed transform algorithm easily produces over-segmentation and results in complex postprocessing steps. The last segmentation category is the energy-based algorithms. This method is needed to establish an energy function which can reach a minimal value when the image is segmented. Live wire,³² level sets,^33,34–37 and graph cut^38,39 are all grouped into energy-based approaches. The live wire method requires the user to identify the seed points which have to be located at the object boundary. Then, an image is segmented by minimizing a predefined energy function. The disadvantage of this method is that it is difficult to accurately choose those seed points. As the level-set algorithm, the segmentation result is related to the given initial curve and it only utilizes a boundary or regional information in the image. Moreover, the level set may converge at a local minimal value and cannot be counted on to get an optimal minimum result for the energy function. In contrast, the energy function of a graph cut is constructed by combining regional information with boundary information. Furthermore, a graph cut can obtain the globally optimal value for the energy function. Compared with a level set, a graph cut is also faster, which is a significant benefit.⁴⁰

In most of the bioholographic images, sole region-based or edge-based segmentation will not work appropriately, since a large number of amplitude or phase values within the 3-D object are very close to the values in the background and some of the object boundaries are very weak. In this article, in consideration of the property of bioholographic images, we apply the graph cut algorithm to the segmentation of these holographic images. However, totally automatic segmentation for various types of bioholographic images is impractical and is unable to perfectly extract 3-D target objects. Therefore, appropriate user interaction is necessary in order to create reasonable segmentation. Consequently, we offer three convenient input buttons which can identify the foreground and background by drawing lines, ellipses, or strokes according to the shape of the 3-D object for the implementation of an interactive graph cut. Moreover, the coefficient of importance factor between regional and boundary information in the graph cut can be dynamically modified by the user for the segmentation of different types of bioholographic images. Experimental results show that this interactive graph cut method can work well for the segmentation of a variety of bioholographic images, even when there are multiple 3-D objects or when the differences between the object and background are not distinct.

In a recent research,⁴¹ we have presented that it was possible to extract target specimens in the bioholographic image using the interactive graph cut method with a simple user interface. However, the proposed method in the work is short of strong persuasion, since only one kind of holographic image is tested, and lack of comparison among the segmentation results. In this present study, the procedure for extracting target biological specimens in holographic images with an interactive graph cut algorithm is the same as that in Ref. 41. However, various bioholographic images including simple and complex ones are used to evaluate the segmentation performance of the interactive graph cut algorithm. This article is focused on showing the robustness of the interactive graph cut-based holographic image segmentation scheme, whereas other segmentation approaches may only work well on a specific bioholographic image. Moreover, a very user-friendly interface difference from the previous study is provided in this article, which incorporates more input buttons and a much clearer button arrangement. The coefficient of importance factor between regional and boundary information is also provided as a slider in user interface, which can be dynamically changed by the user. It seems that more input buttons may make the operation more complicated. But, clear button arrangement can reduce this difficulty. For example, the three buttons, including Line, Ellipse, and Stroke under the Foreground button, can be arbitrarily selected to identify an object according to the various structures of target biological specimens. In addition to that, the segmentation results from the interactive graph cut in this study are compared with those from an advanced level-set-based segmentation technique in terms of segmentation accuracy, iteration times, and processing time with an unbiased comparison algorithm. In summary, the previous work is extended with detailed concepts, user-friendly interface, various tested holographic images for algorithm verification, as well as comparison of segmentation results using an unbiased technique in terms of segmentation accuracy, iteration times, and computational time.

This article is organized as follows. In Sec. 2, we briefly introduce digital holographic microscopy (DHM) with single exposure. The graph-cut approach is described in Sec. 3. In Sec. 4, we present a procedure for creating an interactive graph cut for bioholographic image segmentation. In Sec. 5, we illustrate the experimental results obtained by the interactive graph cut and compare them with other methods. Finally, we conclude this article in Sec. 6.

2. Digital Holographic Microscopy with a Single Exposure for Bioapplications

Among 3-D microscopy based on optical interferometry, DHM^2–6 with single exposure comes to the forefront as the most promising tool for real-time 3-D microbial imaging, pattern recognition, and the study of their dynamics. The single-exposure DHM records digital Fresnel on- or off-axes holograms for 3-D biological imaging (see Fig. 1). The DHM having off-axis configuration provides a quantitative phase-contrast image by separating original and twin images in the reconstructed image with a limited angle between the object and reference beams. Meanwhile, the crosstalk between the original and twin images in the DHM having on-axis Gabor configuration is bound to low-spatial frequencies, so that the on-axis Gabor DHM can be utilized for investigation of microbial identification, dynamics, and distribution in 3-D space.

Fig. 1.

Schematic of the off-axis digital holographic microscopy (DHM).

To acquire the phase information of biological specimens, they are illuminated with coherent or temporally quasi-coherent light. The light starts to be diffracted from a biological specimen, and then the microbial digital hologram is obtained by magnifying diffracted light through a microscope objective. At the same time, the charge-coupled detector (CCD) sensor array, which is interfaced with a computer, records the digital hologram. The four components recorded by the CCD sensor are

$$I(x,y)={|R+O(x,y)|}^2={|R|}^2+{|O(x,y)|}^2+R^{*}O(x,y)+R{O}^{*}(x,y),$$ (1)

where $R$ and $O$ denote the reference plane and the object waves, respectively. The magnified 3-D image or stack of 2-D images of the specimen is numerically reconstructed from their digital hologram at different depths with no mechanical scanning by using Fresnel transformation.^9,42,43 Thus, it facilitates the analysis of the electric field variation of the specimen in the longitudinal dimension as well as in the transverse plane. In the Gabor DHM, the reference beam is automatically provided by ballistic photons, which pass through the specimen and its surrounding medium without any scattering. Since partially temporal incoherent light such as light-emitting diodes (LEDs) in Gabor geometry can be utilized instead of a laser beam, a much higher quality and lower cost 3-D holographic image can be obtained.

3. Graph-Cut Segmentation Algorithm

Since Yuri and Jolly first presented the graph-cut segmentation approach in 2001,³⁸ graph cut-based algorithms have attracted great attention.^{39,40,44–48} It is an energy function-based method and can achieve a globally optimal result while minimizing the energy function.

3.1. Graph Cut

Let $G=\langle V,E\rangle$ denotes an undirected graph, where $V$ is a set of nodes and $E$ is a set of edges.⁴¹ Nodes $V$ consist of two types of vertices. One is called the terminal nodes including $s$ (source) and $t$ (sink) and the other is the neighborhood nodes, which correspond to the pixels in the image. There are also two kinds of edges. The edges between two neighboring nodes are named $n$-links, while the edges connecting terminal nodes with neighboring ones are described as $t$-links. This kind of graph is regarded as an $s-t$ graph. For an $s-t$ graph, each edge will be given a non-negative weight $w_e$. A cut $C$ is a subset of edges $E$, and the cost of a cut is the sum of the weight on the edges in $C$, which is expressed as $C_{\text{cost}}=\sum _{e\in \mathrm{C}}{w}_e$. A minimum cut is a cut that has the minimum cost among all the cuts. For image segmentation, a minimum cut should hopefully occur at the position that can appropriately separate the object from the background. Min-cut is a method to obtain a minimum cut in a graph, and it can be implemented by finding the graph’s maximum flow.^38,44 A min-cut/max-flow algorithm will disconnect the graph nodes into two clusters. However, nodes $s$ (source) and $t$ (sink) have to be grouped into different sets. Conventionally, a cluster with an $s$ node represents the foreground, while the other clusters denote the background. Figure 2 is the illustration of an $s-t$ graph and a min-cut.

Fig. 2.

Illustration of an $s-t$ graph and a min-cut. The thickness of the edge denotes the magnitude of the weight.

3.2. Energy Function for Graph Cut Segmentation Technique

Regarding graph-cut segmentation, the key step is to establish an energy function and assign weight to the two types of edges ($t$- and $n$-links).⁴¹ The energy function in a graph cut is defined by the following equation:^38,39

$$E(L)=\lambda R(L)+(1-\lambda )B(L),$$ (2)

where $L\in \{0,1\}$ is a label assigned to pixel (node) in the image, $R(L)$ is called the regional term, $B(L)$ is a boundary term, and $\lambda$ (between 0 to 1) is the coefficient which can specify the importance of the regional and boundary terms. The segmentation process can be explained as designating a different label for each pixel. When the pixel is labeled with 1, it can be classified as the foreground, and it can be viewed as the background when it is marked with 0. The graph-cut algorithm aims to find an assignment that can minimize the energy function. The regional term $R(L)$ in Eq. (2) will comprise all the weights in $t$-links, while the boundary term $B(L)$ includes all the weights in $n$-links. Thus, $R(L)$ can be described as $R(L)=\sum _{p\in P}{R}_p(L_p)$, where $p$ is one pixel in the image or one of the nodes in the neighborhood of $P$. $R_p(L_p)$ is the penalty for assigning node $p$ with label $L_p\in \{0,1\}$. Intuitively, when pixel $p$ is more likely to be the foreground, the penalty of $R_p(1)$ should be bigger than $R_p(0)$, so as not to separate pixel $p$ from node $s$ and to make this pixel or node belong to the foreground. Consequently, the weight of the $t$-links can be defined with the following equation:^38,39

$$\text{weight}=\{\begin{array}{ll}-\lambda ·\ln pr(I_p|\text{'}\mathrm{bkg}\text{'})& \mathrm{edge}\{p,s\}\\ -\lambda ·\ln pr(I_p|\text{'}\mathrm{obj}\text{'})& \mathrm{edge}\{p,t\}\end{array},$$ (3)

where $pr(I_p|\text{'}\mathrm{obj}\text{'})$ and $pr(I_p|\text{'}\mathrm{bkg}\text{'})$ are the intensity models of the foreground and background, respectively. Equation (3) shows that when a pixel $p$ is inclined to be the object, which means $pr(I_p|\text{'}\mathrm{obj}\text{'})>pr(I_p|\text{'}\mathrm{bck}\text{'})$, the weight of an edge between $p$ and $s$ is larger than that of nodes $p$ and $t$. On the other hand, boundary term $B(L)$ is expressed as $\sum _{\{p,q\}\in N}{B}_{<p,q>}·\delta (L_p,L_q)$, where nodes $p$ and $q$ are two neighboring pixels (neighborhood nodes) and $\delta (L_p,L_q)$ belongs to set {0,1}. When $L_p=L_q$, the value of $\delta (L_p,L_q)$ will be 1. Otherwise, it is set to be 0. Instinctively, a high weight should be given to the $n$-links edge when the two neighboring pixels are very similar, since they tend to belong to the same cluster. In this case, attempting to separate the two nodes will receive more penalties, which means that it will increase the energy. Accordingly, the weight of the $n$-links edge can be specified as follows:^38,39

$$\text{weight}=(1-\lambda )\exp [-\frac{{(I_p-I_q)}^2}{2{\sigma }^2}]\{p,q\}\text{are two neighboring pixels},$$ (4)

where $\sigma$ is regarded as camera noise.^38,39,44 From the above description, a summary can be briefly given. In order to use the graph cut to segment the image and to minimize the energy function, the graph should be established first by using the image (see Fig. 2). Then, the intensity model of the foreground and background should be obtained in advance or calculated by the analysis of the image. The most important step for graph cut is to assign a weight to the $t$- and $n$-links edges. When one pixel (node) is more likely to be the object, the corresponding weight between this point and node $s$ should be larger than that with node $t$. Otherwise, the weight of edge $\langle p,t\rangle$ should be bigger than that of $\langle p,s\rangle$. Similarly, when the intensity of two neighboring pixels is very similar, the weight of the edge between them should be relatively large. After graph establishment and weight assignment to its edges, the max-flow algorithm can be used to get the min-cut in the graph,^34,44,49 and the result will correspond to the image segmentation. At the same time, the minimum energy shown in Eq. (2) can be achieved in the segmented image.

4. Procedure of Interactive Graph-Cut Segmentation for Bioholographic Image

In this procedure, the bioholographic image obtained by DHM is first converted into a grayscale image, and the noise is removed by a median filter. Subsequently, the user has to interactively mark the foreground and background by drawing lines, strokes, or ellipses on the image according to the shape of the 3-D object in the bioholographic image. For a 3-D that is roughly linear or narrow, it is better to use a straight line to identify the object since it can accurately locate the foreground by two endpoints, whereas a stroke is little bit difficult to control, especially when the object is small or narrow. On the other hand, when the target appearance is inclined to be a circle or an ellipse, the user can also utilize the ellipse shape to identify it. The stroke is better to be used for identifying the incorrect segmented foreground and background obtained by previous segmentation results or some area without any specific characteristics. Anyway, we offer line, ellipse, and stroke options to conveniently mark the object and background for the interactive graph cut. When the segmentation result is not reasonable, more foreground and background points can be added interactively, so as to segment the image again by a graph cut based on the previous segmentation result. Furthermore, a user can also dynamically choose the coefficient of importance between regional and boundary information used in the graph-cut technique. When the regional information contributes more to segmentation compared with edge information, the coefficient can be set larger than 0.5. Otherwise, it is given a value of less than 0.5.

In summary, there are a total of six input buttons and one slider in this user interface [as shown in Fig. 4(a)]. The function of three buttons consisting of Line, Ellipse, and Stroke under Foreground is the same. These three buttons are used to identify the target biological specimens in the holographic image. To some extent, these buttons can be regarded as one button. Three different input buttons are adopted so as to make the identification convenient according to the structure of target biological specimens. Similarly, the three input buttons under Background play the same role in marking the background, which means that each of them can be equally used depending on the composition of the image. Theoretically, if more points are identified, a better segmentation will be achieved the first time, since more points are helpful to reconstruct the intensity model and set the weight in graph as described in Sec. 3.2. This step will not affect the final performance, since the uncorrected extraction results can be corrected in the revision process, although more computing time is needed. The slider in user interface decides the importance of regional and boundary terms [see Eq. (2)]. When the target region is much different from the background, it is better to set the slider value more than 0.5. On the other hand, the slider value should be less than 0.5 when the edge of the target biological specimens is more obvious. However, if it is difficult to judge which term, regional or boundary, is more important, the slider value can be set as 0.5. Once the slider value is set by above method, the segmentation results will not be varied much. The following subsections will describe these steps more clearly.

Fig. 4.

Foreground and background are marked. (a) Reconstructed Diatom alga amplitude image. (b) Reconstructed RBC phase image. Red color is for foreground, and blue color is for background.

4.1. Foreground and Background Identifications

In this procedure of interactive graph-cut segmentation, when most of the 3-D objects in the bioholographic images are very similar and very distinct from the background [see Fig. 3(c)], only some of the targets are needed to be marked as foreground by choosing one of the convenient input buttons. However, when the bioholographic images are extremely complex and the 3-D objects are very similar to the background [see Fig. 3(a)], it is better for the user to identify all the objects directly. In this process, when some of the targets in the holographic image are very small and narrow [see Fig. 3(b)], we can choose the line button to mark the needed object, since it is relatively easy and accurate to make the identification because we only need to click two endpoints for drawing a straight line. All the points passing the line will be regarded as the foreground or background. On the other hand, when the target is inclined to be a circle or an ellipse [see Fig. 3(c)], it is better to apply the ellipse button to mark these objects, since this is easy to be controlled. All the points within the ellipse will be taken as the foreground, and only the boundary points are viewed as a background when using the ellipse button for background identification. The stroke button can be used at any step, especially in the revision when the previous segmentation is not good, since it is easy to identify a small area using a stroke when some parts are mis-segmented. After finishing the marking, all the marked points will be expanded by morphological dilation operations with a small structuring element.^27,50 The aim of this step is to increase the number of foreground and background pixels that will be beneficial to establish the foreground/background intensity models, which can be represented by a histogram with the obtained points. These intensity models will be used as standards to decide the weight of the points without being marked in the graph. In order to avoid the case that some points will be marked as both foreground and background in this step or in the revision step, we make a decision that the current identified points will replace the previous ones. That is, when a point is first marked as foreground and then identified as background, this point will be only viewed as background and the previous mark ignored.

Fig. 3.

Different types of 3-D objects in bioholographic images. (a) Target is not distinct from the background. (b) Target is very small and narrow. (c) Multiple similar targets are different from the background.

4.2. Graph-Cut Segmentation

For the graph segmentation, the most important step is to assign weight to the $t$- and $n$-links for establishing an $s-t$ graph.⁴¹ After the foreground and background are approximately marked in the previous step, the foreground/background intensity models, expressed as $pr(I_p{|}^{\prime }{\mathrm{obj}}^{\prime })$ and $pr(I_p{|}^{\prime }{\mathrm{bkj}}^{\prime })$, respectively, are achieved using the corresponding histogram. Then, the weight of the edges in the graph can be assigned as in Table 1.^38,39,44 The marked points, which can be determined as belonging to either the foreground or background, are regarded as hard constraints. The weight between the marked foreground point and the $s$ node will be high, whereas it is low with the $t$ node in the graph. Similarly, the weight between the $s$ node and the marked background point will be small or large with the $t$ node. Thus, this can guarantee that the marked points will not be separated with a corresponding foreground or background.

Table 1.

Weight of the edge in the graph.

Edge	Weight	Condition
$\langle p,q\rangle$	$(1-\lambda )\exp [-\frac{{(I_p-I_q)}^2}{2{\sigma }^2}]$	$\{p,q\}\in \mathrm{N}$
$\{p,s\}$	$-\lambda ·\ln pr(I_p|\prime \mathrm{bkg}\prime )$	$p\in P$ (Unknown)
	$1+\underset{p\in P}{\max }{\sum }_{q\{p.q\}\in N}(1-\lambda )\exp (-\frac{I_p-I_q}{2{\sigma }^2})$	$p\in \text{Object}$
	0	$p\in \text{Background}$
$\{p,t\}$	$-\lambda ·\ln pr(I_p|\prime \mathrm{obj}\prime )$	$p\in P$ (Unknown)
	0	$p\in \text{Object}$
	$1+\underset{p\in P}{\max }{\sum }_{q\{p.q\}\in N}(1-\lambda )\exp (-\frac{I_p-I_q}{2{\sigma }^2})$	$p\in \text{Background}$

In Table 1, set $P$ includes all pixels in the reconstructed bioholographic image, nodes $p$ or $q$ are one of the pixels, $N$ denotes the neighborhood relationship, while $I_p$ and $I_q$ are the values of pixels $p$ and $q$, respectively. When the weight of the edge is successfully set, the graph cut can be executed with a max-flow algorithm, and the segmentation result can be obtained. In this article, the default $\lambda$ value is set to be 0.1, since edge information is more obvious compared with the regional information in our data. However, this value can be dynamically modified by the user at any time during segmenting.

4.3. Further Segmentation

When the segmentation result of the first time is not good, the user can revise the segmentation even further.⁴¹ The user can use any one of the input buttons to mark the foreground and background. Especially, the stroke button is convenient for the revision process. Newly marked points are added as hard constraints, and the weight of the edge has to be updated. When there is no overlap between previously marked foreground points and newly added background ones or previously marked background points and newly identified foreground ones, the max-flow algorithm can be conducted based on the previous result, which already found a maximum flow in the graph with the previous edge weight.^38–44 However, when some already marked points change their status in the revision step, which means that some points identified as foreground or background previously are varied to background or foreground, the max-flow algorithm has to be run first. During the revision step, only the weight of the newly added points, which are also viewed as hard constraints, are updated, whereas the weight of other points remain the same. The weights of the edges that are related to the newly added pixels are given in Table 2.^38,39

Table 2.

Edge weight for newly added points.

Edge	Weight	Condition
$\{p,s\}$	$1+\underset{p\in P}{\max }{\sum }_{q\{p.q\}\in N}(1-\lambda )\exp (-\frac{I_p-I_q}{2{\sigma }^2})-\lambda ·\ln pr(I_p|\prime \mathrm{obj}\prime )$	$p\in \text{Object}$
	$-\lambda ·\ln pr(I_p|\prime \mathrm{obj}\prime )$	$p\in \text{Background}$
$\{p,t\}$	$-\lambda ·\ln pr(I_p|\prime \mathrm{bkg}\prime )$	$p\in \text{Object}$
	$1+\underset{p\in P}{\max }{\sum }_{q\{p.q\}\in N}(1-\lambda )\exp (-\frac{I_p-I_q}{2{\sigma }^2})-\lambda ·\ln pr(I_p|\prime \mathrm{bkg}\prime )$	$p\in \text{Background}$

Similarly, when the segmentation result is still not perfect, the user should identify more foreground or background points with the input buttons in the user interface based on the previous extraction result. Then, the graph-cut segmentation method is executed again. If the user is satisfied with the segmentation result, the interactive process can be avoided, and the extracted target specimens can be used for high-level analysis directly. Otherwise, the revision step should be repeatedly conducted until the extraction results are visually satisfied by the user, which means that the target biological specimens are mostly segmented. In this experiment, the criteria for stopping the revision process can be quantitatively achieved by comparing the segmented target specimens with the reference image and computing the correlation between the two. For example, when the correlation value reaches 0.95, then the user may stop further revision. However, the criterion for stopping revision is not easy to be measured in the practical situation, since there is no reference image. This can only be done by the user. That is, when the segmented target specimen is satisfied, the user can decide not to revise the segmentation results anymore and thus use the extracted target specimens directly without further revision.

5. Experimental Results

In our experiment, the Gabor digital holograms of the Diatom alga and the Sunflower stem cell were recorded with an image sensor array of $2048\times 2048\text{\hspace{0.17em}\hspace{0.17em}pixels}$ with a pixel size of $9\times 9{\mu \text{m}}^2$, where the specimens were sandwiched between two transparent cover slips. Diatom alga and Sunflower stem cell holographic images (complex amplitude wavefronts in focus) were numerically reconstructed at a distance of 25 and 9 μm, respectively, from their own Gabor digital holograms by using the numerical algorithm described in Ref. 9. It is noted that the reconstructed image, which is the multiplication of the amplitude and the phase terms, means the wavefront of the original 3-D biological specimen. Also, the off-axis digital holograms of the red blood cells (RBCs) were recorded with an image sensor array of $1024\times 1024\text{\hspace{0.17em}\hspace{0.17em}pixels}$ with a pixel size of $10\times 10{\mu \text{m}}^2$. Quantitative phase images of RBCs in focus were reconstructed from the off-axis digital holograms obtained by DHM. It is noted that the reconstructed image means the phase term in the wavefront of the original 3-D biological specimen. The automated reconstruction and aberration compensation of the RBC wavefront are obtained by using the numerical algorithm described in Refs. ⁴² and ⁴³. The original RBCs were donated by healthy people and stored in a transfusion bag, which were obtained from the Service Régional Vaudois de Transfusion Sanguine, at 4°C during the storage period.⁵¹ These holographic images were used to test the segmentation performance by an interactive graph cut. The coefficient $\lambda$, which decides the importance of regional and boundary terms, is set to be 0.1, since the boundary takes more importance in our most reconstructed bioholographic images. Depending on the shape of the 3-D object in the holographic image, we can choose the appropriate input button in order to identify the foreground and background. In Fig. 4(a), the reconstructed image of Diatom alga tends to be long and narrow, so it is better to use a straight line to mark the objects. Conversely, when the shape of the reconstructed image of RBCs [Fig. 4(b)] is close to a circle, the ellipse button will be utilized. Figure 4 shows the holographic images with object and background marked.

One of the segmentation steps by the interactive graph cut is presented in Fig. 5. It is noted that some of the parts are misclassified. Thus, further revision is needed. At the revision process, the user should add these misclassified points to their correct group (foreground or background). In this article, it is assumed that there is no hole for each object, which means that we will fill in the foreground after segmentation. Also, we will remove some small regions. Without filling in the holes, some holes in Fig. 4(b) will appear. In Fig. 5, the green maker indicates the segmented boundary. In Fig. 5(b), we can find that even if only a few of objects are marked, the graph-cut method can automatically get the other similar objects. However, when all the objects are totally different, most of the objects have to be identified in order to obtain a relative good segmentation result.

Fig. 5.

First segmentation result for a graph cut. (a) Reconstructed Diatom alga amplitude image. (b) Reconstructed RBC phase image.

Due to the inaccuracy of the first segmentation result, more points have to be added to either the object or background. Thus, further segmentation needs to be executed again, based on a previous result. In Fig. 6, some further marks for the foreground/background and the final segmentation results are shown. This process will be conducted repeatedly until the interactive graph-cut segmentation results are visually satisfied by the user. In Fig. 6, three revision steps are performed to get reasonable targets.

Fig. 6.

Interactive revision steps. (a and b) The added points for the foreground/background. (c and d) A reasonable segmentation result after a two-step revision process.

In Fig. 7, more holographic images [Fig. 7(a): Diatom alga, Fig. 7(c): Sunflower stem cell, and Fig 7(e): RBC] were used to test the interactive graph-cut method. Most of the targets can be successfully segmented within four steps in our experiment. However, for images where the targets are very similar to the background and the boundaries are not obvious, more than five steps are needed. For example, about five steps are conducted for the segmentation of bioholographic images in Fig. 7(c).

Fig. 7.

Segmentation of bioholographic images. (a) Reconstructed Diatom alga holographic image. (c) Reconstructed Sunflower stem cell holographic image. (e) Reconstructed red blood cell (RBC) holographic image. (b, d, and f) The corresponding segmentation results by the interactive graph-cut method.

In addition, the experimental results obtained by an interactive graph cut are compared with those from a level-set approach presented in Ref. 35, which is a robust region-based active contour model achieving better performance than traditional geodesic active contours and Chan–Vese active contours^36,37 in terms of both efficiency and accuracy. Figure 8 shows the segmentation results from the level set in Ref. 33. It is noted that this level-set method generates many over-segmentation problems when the object region is not different from that in the background, such as the results in Figs. 8(a), 8(c), and 8(e), whereas the results are accepted when the object intensity is dissimilar to the background, such as the results shown in Figs. 8(g) and 8(i), even though the weak boundary cannot be appropriately detected.

Fig. 8.

Segmentation of bioholographic images with the method in Ref. 35. (a, c, e, g and i) The reconstructed holographic images. (b, d, f, h, and j) The corresponding segmentation results by the active contour method in Ref. 35. The scale bar in the $y$-axis is the same as that in the $x$-axis.

In order to avoid biased comparison results caused by varied parameters existing in different algorithms (e.g., the comparison between the best segmentation result in one algorithm and the worst segmentation result in the other algorithm), segmentation performance is evaluated by varying the parameter in both algorithms with the method in Ref. 52. The primary parameter in this interactive graph cut is the coefficient $\lambda$, which decides the importance of regional and boundary terms, whereas the primary parameter in Ref. 35 is the value of $\alpha$ that controls the propagation speed. The segmentation accuracy is defined as the correlation between the segmented image and reference that is manually obtained. When the correlation value tends to be 1, it means high-segmentation accuracy. Moreover, the iteration and revision times for the two methods are measured in this experiment as well as the computing time. Figure 9 quantitatively shows the performance models for the five bioholographic images (Sunflower stem cell, Diatom alga with single target, Diatom alga with multiple targets, RBCs with predominantly stomatocyte shape, and RBCs with predominantly discocyte shape, as in the order shown in Fig. 8) between the level-set method in Ref. 35 and the interactive graph-cut method in this article. Figure 10 presents a comparison of computation times of the two algorithms. Here, we only consider the total execution time of the automatic segmentation part in the interactive graph cut which is the time taken in the automatic graph cut, excluding the time for the user interaction process. It is noted from Fig. 9 that the best performance from the interactive graph cut is superior to that from the level set segmentation in Ref. 35 on average. Especially, the experimental result shows that for the segmentation of the holographic images with twin image noises reconstructed from the Gabor digital hologram [see Figs. 8(a), 8(c), and 8(e)], the interactive graph-cut method achieves much better performance in terms of segmentation accuracy. It is also noted that, in our experiment, most of the bioholographic images can be successfully segmented within five revision steps without considering the specific 3-D shape using the interactive graph-cut segmentation approach, whereas the level-set method requires a large number of iterations for the segmentation of the holographic images. Even though the two methods can achieve similar performance results with some images, the interactive graph-cut approach definitely outperforms the level-set method for complex bioholographic images. Undeniably, most of the bioholographic images are very complex with multiple objects. Thus, the interactive graph-cut method is a better method for the application of segmentation to all kinds of bioholographic images. As the computing time in Fig. 10 shows, the automatic segmentation part in the interactive graph cut is superior to the level set in Ref. 35. However, we have to point out that we do not count the time taken for the user interaction process. On average, 40 s is needed for the user interaction for each image, which will increase the total segmentation time. Interestingly, people are usually sensitive to the computing time spent in the automatic part and lack of sensitivity to the interaction process, since unoccupied waiting feels longer than occupied waiting in the view of psychology.⁵³ In this article, we concentrate on showing the robustness of the interactive graph scheme to various kinds of holographic images, while other segmentation approaches may only work well on a specific bioholographic image.

Fig. 9.

Performance models: (a, c, e, g and i) are the performance models for the five bioholographic images (sunflower stem cell, Diatom alga with single target, Diatom alga with multiple targets, RBCs with predominantly stomatocyte shape, and RBCs with predominantly discocyte shape, as in the order showed in Fig. 8) in the method of Ref. 35, where the $\alpha$ is varied from 5 to 230 with an interval of 25. (b, d, f, h and j) are the performance models for the five bioholographic images (sunflower stem cell, Diatom alga with single target, Diatom alga with multiple targets, RBCs with predominantly stomatocyte shape, and RBCs with predominantly discocyte shape, as in the order showed in Fig. 8) in an interactive graph-cut segmentation method, where $\lambda$ is varied from 0 to 1 with an interval of 0.1.

Fig. 10.

Computing time between level-set method in Ref. 35 and automatic graph-cut part in interactive graph-cut method. The first image is sunflower stem cell with a size of $600\times 600$, the second image is Diatom alga with a single target and a size of $950\times 700$, the third image is Diatom alga with multiple targets and an image size of $700\times 750$, the fourth image is RBCs with predominantly stomatocyte shape and an image size of $700\times 700$, and the fifth image is RBCs with predominantly discocyte shape and an image size of $700\times 700$, as in the order showed in Fig. 8. In this figure, the computing times are measured with the parameters including $\lambda$, $\alpha$, and iteration times selected from Fig. 9, where the best segmentation accuracy is achieved.

6. Conclusion

In this article, the interactive graph-cut method, which incorporates regional and boundary information, has been applied to the segmentation of bioholographic images reconstructed from a digital hologram. In this interactive graph-cut procedure, three convenient input buttons labeled line, ellipse, and stroke are used to identify the foreground and background that can easily handle various types of holographic images. Moreover, the user can dynamically modify the coefficient of importance between regional and boundary information. In our experiment, we have shown that most of the reconstructed bioholographic images can be successfully segmented within five revision steps without considering the specific 3-D shape using the interactive graph-cut segmentation approach. Moreover, the interactive graph-cut algorithm can simultaneously extract similar multiple microbes in bioholographic images while only needing to identify some of the objects. The interactive graph-cut segmentation technique will be beneficial to the analysis of target biological specimens in the reconstructed bioholographic images, which can provide rich 3-D information about the target biological specimens. Compared with the other segmentation methods, the interactive graph-cut method can be appropriate for any bioholographic images, even though they have complex objects and backgrounds.