Multi-focus Image Fusion Using Epifluorescence Microscopy for Robust Vascular Segmentation

Abstract

Automatic segmentation of three-dimensional microvascular structures is needed for quantifying morphological changes to blood vessels during development, disease and treatment processes. Single focus two-dimensional epifluorescent imagery lead to unsatisfactory segmentations due to multiple out of focus vessel regions that have blurred edge structures and lack of detail. Additional segmentation challenges include varying contrast levels due to diffusivity of the lectin stain, leakage out of vessels and fine morphological vessel structure. We propose an approach for vessel segmentation that combines multi-focus image fusion with robust adaptive filtering. The robust adaptive filtering scheme handles noise without destroying small structures, while multi-focus image fusion considerably improves segmentation quality by deblurring out-of-focus regions through incorporating 3D structure information from multiple focus steps. Experiments using epifluorescence images of mice dura mater show an average of 30.4% improvement compared to single focus microvasculature segmentation.

I. INTRODUCTION

Automated vessel extraction and quantification is an important processing step in characterizing vessel networks, angiogenis and normal vs. diseased tissue behavior. Many techniques for vessel extraction have been proposed in literature [1]–[5] mostly for neuro, cardiovascular or retinal imaging. The application of interest in this paper is vascular network reconstruction using epifluorescence imagery [6], [7]. In epifluorescence, imaging tissue is stained using a lectin SBA and imaged using fluorescence microscopy. Vascular segmentation in epifluorescence microscopy poses unique challenges such as varying contrast levels due to diffusivity of lectin stain, spatially varying focus, high background signal due to leakage of the stain from vessels, and high variance in vessel intensity. In these type of images, traditional global thresholding schemes [8] fail to pick up salient structures effectively.

Active contour models [7], [9]–[12] relying on edge or region based indicators may fail to capture all the salient structures. Another major challenge is vascular structures at variable depths of focus. Spatially varying out-of-focus blur not only hinders accurate segmentation but also results in quantification errors. Fusing various focus (z-stack) images thus becomes an efficient alternative to improve the segmentation of 3-D vessel structures without requiring volumetric data collection and computationally intensive 3-D reconstruction. Several multifocal image fusion techniques have been studied in the past such as pixel-level analysis [13], and sparse representation for wavelet based methods [14].

In this paper we propose a system that couples a robust adaptive filtering based segmentation with a multi-focus image fusion module. The robust adaptive filtering scheme handles noise without destroying small structures, and the multi focal image fusion considerably improves the overall segmentation quality by integrating information from multiple images. Denoising schemes are not able to resolve detailed image structures present in the blurred out-of-focus regions. We propose incorporating a multi-focus image fusion stage prior to denoising and show that this is key step in increasing true detections of vessel regions while minimizing false positives. We use a Hessian-based image fusion approach to combine information from multi-focus images. There are various methods based on the Hessian matrix such as [3], [15], [16]. In order to improve our current vessel segmentation scheme we fuse the images combining different areas of focus and in this article we assess the effectiveness of this step.

The major contributions of our work are: (a) a multiscale Hessian matrix based scale selection and composite multi-focus image generation, (b) an adaptive robust filtering scheme that handles noise without destroying small structures, and (c) experimental vessel segmentation results showing the effectiveness of the multifocus image fusion approach. The rest of the paper is organized as follows. Section II describes the fusion method for multifocus composite image generation. Section III describes our robust smoothing based segmentation. Section IV provides experimental results.

II. Multifocus Image Fusion Using Multiscale Hessian

There are various methods to fuse images such as naive pixel level averaging of the different focus images. Although image averaging is effective at reducing independent Gaussian noise, it leads to reduced contrast and increased blurring overall. As shown in [17] blur can adversely affect the detection rate of curvilinear features. Since the derivatives do not respond as strongly in areas that are out of focus, we can identify regions that are in-focus within each image by maximizing the second order derivative response to create a fused multi-focus pixel map. The Hessian matrix of second-order image derivatives for gray level image I is defined as:

$$\mathbf{H}(I)=\left[\begin{array}{cc}\hfill I_{xx}\hfill & \hfill I_{xy}\hfill \\ \hfill I_{xy}\hfill & \hfill I_{yy}\hfill \end{array}\right]$$ (1)

Let λ₁, λ₂ be the eigenvalues and e₁, e₂ the corresponding eigenvectors of the Hessian matrix with |λ₁| ≥ |λ₂|. Scale space images are computed by convolving the image with 2D isotropic Gaussians with varying standard deviations,

$$G(x,y,\sigma )=\frac{1}{\sqrt{2\pi }\sigma }\text{exp}\left(\frac{-(x^2+y^2)}{2{\sigma }^2}\right)$$ (2)

where σ is the standard deviation or scale of the Gaussian. Let I_p denote a single focus image with p ∈ {1, 2, …, N}, the set of N single focus images. We compute the Hessian matrix (1) over a range of suitable scales for each single focus image I_p. We define a multi-scale matrix norm that is based on the eigenvalues of the Hessian at the maximum response scale:

$${\Vert \mathbf{H}(x,y,p)\Vert }_F=\sqrt{{\lambda }_{1p}^2(x,y,{\sigma }^{*})+{\lambda }_{2p}^2(x,y,{\sigma }^{*})}$$ (3)

where σ* = arg max_σ|λ_1p(x, y, σ)|. Note that we normalize the Hessian matrix by the squared scale factor σ² before computing the eigenvalues and the Frobenius matrix norm in (3). The final focus selection map is given by,

$$p^{*}(x,y)=\underset{p\in \{1,2,\dots ,N\}}{\text{arg max}}{\Vert \mathbf{H}(x,y,p)\Vert }_F$$ (4)

where p*(x, y) is the index of the best focus source image at pixel location (x,y). The proposed fusion algorithm consists of the following steps:

• Step 1: Compute multi-scale Hessian values for each single focus image I_p, p = {1, …, N}.

• Step 2: Estimate max scale response at each pixel in I_p.

• Step 3: Compute multi-scale Frobenius norm as in Eqn. (3).

• Step 4: Compute focus selection image map as in Eqn. (4).

• Step 5: Generate composite image I_c using:

  $$I_c(x,y)=I_{p^{*}(x,y)}(x,y)$$ (5)

Figure 1 shows an example of the multi-focus fusion approach from a set of four single focus images. Figure 1(a)–(d) shows single focus images where the focus shifts from bottom left to top right in a diagonal fashion. Figure 1(e) shows the final fused image using our proposed multi-scale Hessian approach. In Figure 1(f) we show the focus selection map given in Eqn. (4) plotted as a colormap.

Fig. 1.

Images for mouse 1 (022106ERbKOOVX(28)) showing variable regions of focus for the same vessel (a)–(d) and the fused result (e). Colored pixels in (f) indicates the source images 1 to 4 after multi-focus fusion in blue, cyan, yellow and red respectively.

III. Vascular Segmentation Using Robust Smoothing

A robust image smoothing operator is applied to improve the segmentation of vascular regions from composite in-focus images produced by the multi-focus image fusion module described in Section II. Robust smoothing reduces noise and enhances object boundaries [18], [19]. The module consists of four main steps: 1) background estimation and removal using morphological filtering, 2) robust image filtering with convex optimization, 3) adaptive thresholding on filtered image to generate a binary vascular mask, 4) post-processing using morphological operations to remove spurious detections and to generate the final vasculature network. Robust image filtering based segmentation is briefly described.

Let Ω ⊂ ℝ² be the rectangular image domain, and the input image I : Ω → ℝ with I(x) represents the value at a pixel x ≔ (x, y) ∈ Ω. In a robust statistics framework finding a best fit of a smooth image u from a given noisy input image I can be posed as a minimization problem [18],

$$\underset{u}{\text{min}}\{\sum _{\mathbf{x}\in \mathrm{\Omega }}\sum _{\mathbf{y}\in {𝒩}_{\mathbf{x}}}\omega (\mathbf{x}-\mathbf{y})\rho (I(\mathbf{x})-u(\mathbf{x}),\sigma )\}$$ (6)

with a robust estimator function ρ, we use the Tukey’s biweight robust function due its strong edge preserving property. Here 𝒩_x represents the neighborhood of pixel of x, ω is a spatial weighting function (e.g. Gaussian kernel (2)) and σ is a scale (variance) parameter. To solve the minimization problem (6) we use the dilation convex approximation [20] of the Tukey function,

$${\rho }_{\gamma }(\xi ,\sigma )=\{\begin{array}{cc}\frac{{\gamma }^2{\sigma }^2}{6}(1-{[1-{(\xi /\gamma \sigma )}^2]}^3)\hfill & \hfill |\xi |\le \gamma \sigma ,\\ 1/3\hfill & \hfill \text{otherwise}.\end{array}$$ (7)

We use an iterative reweighted method to solve (6),

$$u_{\mathbf{y}}^{t+1}=\frac{{\sum }_{\mathbf{x}\in {𝒩}_{\mathbf{y}}}\omega (\mathbf{x}-\mathbf{y})c(I(\mathbf{x})-u^t(\mathbf{y}))I(\mathbf{x})}{{\sum }_{\mathbf{x}\in {𝒩}_{\mathbf{y}}}\omega (\mathbf{x}-\mathbf{y})c(I(\mathbf{x})-u^t(\mathbf{y}))}$$ (8)

where $c(\xi )={\rho }_{\gamma }^{\prime }(\xi )/\xi$ and t represent the number of iterations. This process results in enhanced smooth image where edges are preserved. Adaptive thresholding using the local mean and variance values [21] is applied to the enhanced smooth multi-focus fused images to produce a binary vascular mask.

IV. Experimental Results

For our experiments we have compared single focus versus multifocus segmentation results using robust smoothing based segmentation in both cases. The two methods are referred to as RIS and RIM for fluorescence intensity-based Robust Image segmentation using Single focus images (RIS), and Multi-focus (RIM) images respectively. The metric used for validation is the Dice Similarity Coefficient, DSC(P,Q) = 2(|P ∩ Q|)/(|P| + |Q|), where P and Q are automatically and manually segmented microvasculature regions and |·| denotes the total number of pixels. DSC ∈ [0, 1.0] with zero being no overlap and one being perfect overlap. We have manually segmented 10 epifluorescent microvasculature images to be used as ground truth vessel segmentation masks for evaluation.

Our results show that fusing multi-focus images on an average always resulted in better segmentations. As can be seen from Table I, the DSC overlap measure for segmentations using multi-focus (RIM) images shows marked improvement over any of the individual single focus image based segmentations (RIS). For RIS, we have reported the maximum Dice coefficient from a set of single focus image based segmentations. The highest improvement is seen in WT-07 with a 122.9% improvement in vasculature detection. Detailed segmentation results for WT-03 are shown in Figure 2 along with ground truth mask. Figure 2(c) shows the segmentation result obtained using RIM and Figure 2(d)–(h) show segmentation results obtained using five single focus RIS. As can be seen the RIM captures all the vessels which are not obtained by individual RIS results.

TABLE I.

DSC values for 7 wild type (022006-ERbWT-OVX(28)) and 3 knock out mice (022006-ERbKO-OVX(28)) comparing best single focus (RIS) and multi-focus (RIM) segmentations. The values shown for RIS DSC are the maximum and the improvement is computed between the mean RIS DSC and RIM DSC.

Image ID	RIS	RIM	Improvement (%)
WT-02	0.3902	0.5110	30.9
WT-03	0.3550	0.7916	122.9
WT-06	0.4868	0.3677 (0.6363)	−24.4 (30.7)
WT-07	0.2913	0.4992	71.3
WT-08	0.3477	0.5512	58.5
WT-09	0.3582	0.3682	02.7
WT-10	0.4144	0.4972	19.9
KO-01	0.7922	0.8449	06.6
KO-04	0.8138	0.8424	03.5
KO-06	0.7570	0.8497	12.2

Fig. 2.

Detailed segmentation results for 022006-ERbWT-OVX(28)-WT-3 (see row WT-03 in Table I). (a) Fused image (b) Ground truth vessel mask (c) Multi-focus RIM segmentation result. RIM segmentation result. Single focus images and their corresponding RIS segmentation results are shown in (d) to (g) and (h) to (k) respectively. For (c) and (h) to (k) white regions are correctly segmented pixels (TP), red are missing (FN) and blue are extra regions (FP) compared to the ground truth segmentation. Better viewed in color and online.

Figure 3 shows an example where RIS outperformed RIM (entry WT-06 in Table I). The single-focus RIS (Figure 3(g)–(f)) performs better with DSC values 0.4868, 0.4706, 0.4834 respectively. These segmentation results in Table I were obtained using a single adaptive thresholding parameter k = 1.5 for both single and multi-focus images. Using separate optimized threshold value (k = 2.0) for multi-focus images improves their result as shown in parenthesis in Table I and Figure 3(d). Parameter set should be analyzed and optimized for single and multi-focus images separately since they have different image and noise characteristics.

Fig. 3.

High background noise levels can reduce the performance of RIM. (a) Fused image for WT-06, (b) Ground truth vessel mask, (c) Multi-focus (RIM) result. (d) Multi-focus (RIM) result for k = 2.0 (e) Single focus image for WT-06 (best result) (f) Single-focus (RIS) best result Segmentation masks (c,d,f) show correctly segmented pixels as white pixel regions, red are missing regions and blue are extra regions compared to the ground truth segmentation. Better viewed in color and online.

V. Conclusions

Segmentation of complex three-dimensional microvascular structures is hampered by using only single focus epifluorescence images. Significant improvement in the accuracy of the vessel segmentation can be achieved by fusing multi-focus images coupled with robust adaptive filtering. Experimental results on high resolution epifluorescence of mice dura mater images showed an average of 30.4% improvement compared to single focus segmentations. Deblurring and denoising individual single focus images prior to fusion is expected to further improve performance. The proposed automatic segmentation and quantitative vessel morphology measurement system will be used to characterize ovary excised versus normal intact cases in different animal models. This will help in studying systemic influence of hormone therapy on angiogenesis. Further studies are being planned to determine if these microvascular morphological changes can be used to characterize disease progression and response to treatment.