Multi-focus Image Fusion for Confocal Microscopy Using U-Net Regression Map

Abstract

Characterizing the spatial relationship between blood vessel and lymphatic vascular structures, in the mice dura mater tissue, is useful for modeling fluid flows and changes in dynamics in various disease processes. We propose a new deep learning-based approach to fuse a set of multi-channel single-focus microscopy images within each volumetric z-stack into a single fused image that accurately captures as much of the vascular structures as possible. The red spectral channel captures small blood vessels and the green fluorescence channel images lymphatics structures in the intact dura mater attached to bone. The deep architecture Multi-Channel Fusion U-Net (MCFU-Net) combines multi-slice regression likelihood maps of thin linear structures using max pooling for each channel independently to estimate a slice-based focus selection map. We compare MCFU-Net with a widely used derivative-based multi-scale Hessian fusion method [8]. The multi-scale Hessian-based fusion produces dark-halos, non-homogeneous backgrounds and less detailed anatomical structures. Perception based no-reference image quality assessment metrics PIQUE, NIQE, and BRISQUE confirm the effectiveness of the proposed method.

I. Introduction

Confocal microscope is an imaging device that allows optical sectioning or depth discrimination by using a limiting pinhole that blocks the light emitted from out-of-focus planes. Each single focus image captures the details of the specimen regions that lie close to its focal plane, while the remaining regions are imaged with poor contrast [1]. Multi-focus image fusion aims to create a single sharp and detailed composite image that captures the essential 3D structure information in a set of single focus images. There are a number of multi-focus image fusion approaches designed for a variety of applications ranging from remote sensing to medical imaging [2]. However, multi-focus image fusion to capture the highly complex structures of the blood and lymphatic microvasculature remains to be a challenging due to contrast level variation caused by lectin stain diffusion, high variance in vessel intensity, fine microvascular structures, leakage of stain from vessels, variable depth of focus, and complex 3D structures. The existing multi-focus image fusion approaches can be broadly categorized as transform domain-based, spatial domain-based, and neural networks-based. Transform domain approaches first transform the source images into a desired domain, then fuse the obtained transform coefficients, and finally apply inverse transform to obtain the fused image [3]. Spatial domain approaches rely on focus measures based on first and second order image derivatives such as energy of gradients, sum of modified Laplacian, energy of Laplacian that are computed at image blocks [5]. These focus measures highly depend on block sizes and perform poorly at object boundaries [4]. The neural network-based fusion approaches exploit learned, data-driven features to produce fusion results, that often lead to reduced fusion artifacts [6]. Producing a homogeneous vessel region and a clear background is substantial to improve microvasculature segmentation approaches [23]–[26].

In this paper, we propose a spatial domain pixel-based multi-focus fusion approach that uses domain specific, learned, focus measures. The approach relies on a novel vesselness likelihood index computed using a U-Net convolutional neural network [15] trained to segment vascular networks in microscopy images. The pretrained U-Net model from [16] is applied independently to each single-focus source images in a confocal microscopy z-stack to produce a set of vesselness likelihood maps. The composite fused image is constructed by selecting for each pixel the focus layer with the maximum U-Net vesselness likelihood value. Figure 1 shows two multi-focus fused images for a sample confocal microscopy z-stack obtained using our proposed MCFU-Net fusion and another spatial domain, multi-scale Hessian based, multi-focus fusion approach [8]. Figure 2 shows single-focus layers and the corresponding multi-focus image for a sample dual stain microscopy z-stack of dura mater. In the past, it was believed that the dura mater structures do not contain any lymphatics. Very recently, lymphatics like structures were observed with dura mater vascular networks [7]. The red channel in these images captures the blood microvasculature, while the green channel captures the lymphatics like structures. These images, particularly the channel corresponding to lymphatics structures, are very complex. It is of critical importance to develop multi-focus fusion algorithms that can reliably reveal the details of these complex anatomical structures in order to advance our scientific knowledge and help guide medical decision making.

Fig. 1:

Two fused multi-focus images for a sample confocal microscopy z-stack. Fused images are obtained using a multi-scale Hessian based fusion approach [8] (a), and our proposed MCFU-Net fusion (b). The proposed approach demonstrates better contrast, significant reduction in background noise, higher signal-to-noise ratio, and lower fusion artifacts.

Fig. 2:

Dual stain confocal microscopy images of mice dura mater. Red channel captures the blood microvasculature, while the green channel captures the lymphatics like structures. (a-c) Sample single-focus images from a z-stack. (d) Max intensity projection, (e) multi-scale Hessian fusion, and (f) MCFU-Net fusion of 229 single-focus images for the same z-stack.

The major contributions of this paper include: (a) a multi-focus fusion system for dual stain, blood and lymphatic microvasculature images; (b) a novel, learned, domain specific focus measure based on U-Net convolutional neural networks trained for vessel segmentation; and (c) reduction of banding artifacts caused by derivative based fusion approaches. We performed our experiments and evaluation in MATLAB (R2019a, MathWorks ^®) environment in a Windows machine having Intel Core i5 CPU with 8GB RAM. The rest of the paper is organized as follows. Section II describes the details of the proposed multi-focus fusion approach. Section III presents our experimental results and performance evaluations. Section IV concludes the paper and presents some future works.

II. Deep Learning-Based Multi-Focus Image Fusion

A. Multi-focus image fusion using multi-scale Hessian [8]

Derivative operations respond differently to in-focus and out-of-focus regions of an image. Classical multi-focus fusion approaches rely on first or second derivative based measures to identify focused regions of an image. Because the structures of interest for our application domain are curvilinear microvascular structures, we have chosen to use second order derivative information. Hessian matrix of an image I is a square matrix of second-order partial derivatives [8]:

$$H(I)=\left[\begin{array}{ll}{I}_{xx}\hfill & I_{xy}\hfill \\ I_{xy}\hfill & I_{yy}\hfill \end{array}\right]$$ (1)

It describes the local curvature of a function. In order to perform multi-focus fusion, we have computed multi-scale Hessian response for each channel separately. We have used the standard deviations scales from 1 to 7 for Gaussian blurring to generate the multi-scale derivatives and Hessian response for each image slice [8] and selected the scale that produced the maximum response. At the maximum response scale, we calculated the Frobenius norm. Then we generated an image selection map that selects at every pixel which single focus image in the corresponding stack provides the maximum Frobenius norm result. Based on the image selection map, we produced the fused image for both red and green channels. Results show that the multi-scale Hessian-based approach produces banding effects with non-homogeneous background. Also, the fused results are non-smooth with some scattered components. Such structural inconsistency results in false positives in segmentation which in turn hinders the quantitative analysis.

B. Multi-focus image fusion using U-Net regression maps

Deep learning based-focus selection and multi-focus image fusion has recently gained significant importance in different type of images [11], [12], [13], [14]. However, studies focusing on microscopy image fusion using deep learning are few. Hence, we took the advantage of widely used and accepted deep learning Convolutional Neural Networks called U-Net [15] for the fusion of microscopy images. The detailed architecture of the proposed MCFU-Net pipeline is shown in Figure 3. U-Net is popular in the biomedical field since it preserves the fine details for the anatomical structure which is extremely important in our microscopy vessel analysis. Thin vessels could disappear through CNN max-pooling operations that could be a problematic issue in our vessel analysis work. U-Net preserves those tiny spatial details by concatenating intermediate layers with their corresponding feature maps along the opposite expansion path. As a result, larger structural context is captured along the contraction (feature) path without loss of spatial semantic details along the expansion (pixel labeling) path.

Fig. 3:

Block diagram for proposed MCFU-Net fusion algorithm. We took the single focus images for each z-stack and generate the regression likelihood map for each single focus image using U-Net from [16]. A pixel wise max pooling operation is used to get the maximum score map to generate the fusion along Z-axis. Finally, multi focus green and red channel fused images are combined to get the Red-Green dual channel composite fusion result.

We used the U-Net architecture that uses input image patches or tiles of size 256 × 256 pixels with filter sizes, padding, and strides as used in [16]. Our architecture has four stages along the contraction path as well as four stages in the expansion path. Each block consists of 2 convolutions, 2 ReLU’s and one max pooling in the contraction path. In the expansion path, transpose convolution replaces the max pooling. U-Net architecture is shown as part of Figure 3.

The input microscopy images having vascular networks and their corresponding masks is used to train the U-Net model with stochastic gradient descent optimization. The energy function of the U-Net is the pixel-wise softmax over the final feature map. The cross-entropy then calculates the difference at each pixel location from the true class. In our case we have only two classes, either vessel or background. The network was trained for 60 epochs.

We used the trained model to generate the regression likelihood map for each of the slices of z-stacks. U-Net regression map has pixel-wise probability map or scores. We utilize those maps because they provide rich information and robust way to estimate pixel-wise structure’s spatial appearance across frames in z-stacks. In other words, the probability map works as a decision map to decide which pixel should be selected to form the final fused multi-focus image. The intensity of each pixel across all frames corresponds to the maximum probability value across all single focus U-Net regression maps.

Let I_s denote the single focus image with s∈1,2,3,… ,n where n is the total number of single focus images in each stack. If P (x, y, s) is the U-Net regression likelihood map for single focus images then we define the focus selection map as

$$s^{\ast }(x,y)=\mathit{\text{argmax}}(P(x,y,s)),s\in \{1,2,3,\dots ,N\}$$ (2)

The final fused composite image is generated as

$$I_F\left(x,y\right)=I_{s*\left(x,y\right)}\left(x,y\right)$$ (3)

Multi-scale fusion using Hessian utilizes frobenious calculation as a decision tool for the final selection map. As a result, it could consider noise rather than the real structures or it may generate noisy background in homogeneous regions. While U-Net regression’s maps are acquired after complex non-linear operations with thousand of calculations including automated generations for rich feature maps. The net of all these operations are summarized in the final foreground regression map that make it a robust and confident tool for our fusion problem.

C. Quality Metrics for Performance Evaluation of Fusion

To understand how well the fusion algorithm is working, only visual interpretation is not enough. We need some quantitative analysis. There are lots of reference-based image quality assessment tools. But non-reference or blind image quality assessment tools are few. Among the non reference image quality assessment tools, specially for microscopy images, methods to analyze the vesselness are very limited [17], [18]. That’s why we performed quantitative analysis of our fused results based on four image quality metrics. Three widely accepted perception based non-reference image quality assessment tools namely Perception based Image QUality Evaluator (PIQUE) [19], Naturalness Image Quality Evaluator (NIQE) [20], and Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE) [21]. And the fourth one is called Structural Similarity Index (SSIM) [22] to analyze how much dissimilarity there is between the classical multi-scale Hessian based fusion method and our proposed one.

PIQUE is a local feature-based image quality assessment tool that calculates the Mean Subtracted Contrast Normalized (MSCN) coefficient. Then divide the image into non overlapping blocks to identify perceptually important spatially active blocks based on variance of MSCN. Then it classifies the distorted and non-distorted blocks using thresholding and PIQUE score is the mean of scores in distorted blocks.

$$\mathit{\text{PIQUE}}=\frac{\left({\sum }_{k=1}^{N}D\right)+C}{(N+C)}$$ (4)

Where N, is the number of spatially active blocks in any given image, C is a positive constant, and D is the distortion assignment procedure.

NIQE and BRISQUE are based on spatial features derived from natural scene statistics. In case of NIQE, the quality is expressed as the distance between a Multivariate Gaussian (MVG) fit of the Natural Scene Statistics (NSS) features extracted from the test image, and MVG model of the quality aware features extracted from the set of natural images.

$$\mathit{\text{NIQE}}=\sqrt{{\left({\mu }_1-{\mu }_2\right)}^T{\left(\frac{{\Sigma }_1+{\Sigma }_2}{2}\right)}^{-1}({\mu }_1-{\mu }_2)}$$ (5)

Where,μ₁ , μ₂ are the mean vectors and Σ₁ , Σ₂ are covariance matrix of the natural MVG model and the distorted image’s MVG model.

The BRISQUE scores are predicted using a Support Vector Regression (SVR) model trained on an image database containing compression artifacts, blurring, and noise. In all three non-reference quality measures, lower score indicates better perceptual quality of the fused image. We have used SSIM, another quality metric that requires reference. As we do not have any reference fusion image, we are using SSIM scores to identify how similar the two fusion results are. The SSIM relies on the computation of luminance, contrast, and structural terms.

III. Experimental Results

A. Data collection

All animal experimental procedures were approved by the University of Missouri Institutional Animal Care and Use Committee. Wild type C57BL/6J female mice were used in this study. The staining of cranial vasculature was previously described in [7]. In brief, immediately following sacrifice, the chest cavity was opened and the entire body was perfused through the heart with Kreb’s albumin solution containing Alexa Fluor 488-conjugated anti-Lyve-1 antibody (eBioscience, clone ALY7, 1:100) and 20 micro gram/ml AlexaFluor 594-conjugated soybean agglutinin (SBA) lectin to stain and identify lymphatic structures and blood vascular networks respectively. The cranial skull together with the dura mater was removed and fixed in 10 percent neutral buffered formalin solution. Flat mounts were prepared and mounted on a slide. Microvascular networks were imaged with 20x lens using Green Fluorescent Protein (GFP) and Texas Red channels on confocal FluoView FV1000 inverted microscope system (Olympus, Thronwood, NY, USA) with FV10-ASW (Ver 02.01.03.10) software. The 157 to 267 micrometer thick z-stacks were acquired with 1 micro meter step size. The red channel stain characterizes the blood vessels having small arteries, capillary networks, and small venules. The green channel characterizes lymphatics like structures sinusoids that could be either pure (does not have any blood vessels) or has one blood vessel or multiple blood vessels.

B. Results and Discussion

Figure 4 shows the MCFU-Net fusion results for the red channel (blood microvasculature) and for the green channel (lymphatic-like structures) for a sample z-stack. Zoomed view of some regions of interest are provided to better visualize improved results. The upper part of the figure shows multi-focus fusion results obtained using the multi-scale Hessian approach described in Section II-A. The lower part of the figure shows the multi-focus fusion result obtained using the proposed MCFU-Net fusion described in Section II-B. As we can see, the blood vessels (red channel) have more discontinuities and holes in the upper image compared to the lower image produced by our proposed MCFU-Net fusion approach. Similarly for the green channel (lymphatic like structures), Hessian-based fusion produces noisy background and spurious holes in the lymphatic structures, while the proposed MCFU-Net approach results in smoother background, more complete lymphatic structures, and better contrast with respect to background. Using the proposed fusion approach, visibility and shape of the microvascular structures have been significantly improved and revealed; noisy background problem and scattered disconnected component problem of multi-scale Hessian fusion have been addressed. Beyond visual perception, these improvements are crucial for the success of follow-up image analysis steps such as vessel segmentation and quantification.

Fig. 4:

Visualization of two zoomed-in areas for a multi-focus fused images, the upper part contains multi-focus fused images using the Hessian method [8], the lower part contains multi-focus fused images using our proposed method. The zoomed-in regions for both red and green channels show how our proposed method could produce a homogeneous region without any artifacts or holes compared to the Hessian method that leads to structures with holes and produces noisy background.

In order to better assess the fusion performance, we have computed the unsupervised image quality measures described in Section II-C. To demonstrate the structural differences between the multi-focus fusion results obtained by multi-scale Hessian versus our MCFU-Net approaches, we have computed structural similarity index SSIM. The results from the two approaches have an average SSIM score of 0.75, indicating structural differences between the two sets of multi-focus fused images. The quality metrics for fifteen z-stacks chosen from different parts of the whole dura mater are presented in Table I. The name of the each stack indicates that it is a control intact female mice with the corresponding staining. The last number is an identification of image site in the whole dura mater. For each of the three quality metrics PIQUE, NIQE, and BRISQUE, our MCFU-Net fusion resulted in lower scores, indicating better image quality, compared to multi-scale Hessian based fusion. For PIQUE score, if the score is within the range of 0 to 20, then the image perceptual quality is marked as ‘Excellent’ [13]. Proposed MCFU-Net fusion results for all the experiments fell in the ‘Excellent’ category whereas multi-scale Hessian fusion produced worse scores.

TABLE I:

Performance comparison of quality of multi-focus image fusion

Input Stacks	PIQUE		NIQE		BRISQUE		SSIM
	Multi-scale Hessian Fusion	Proposed MCFU-Net Fusion	Multi-scale Hessian Fusion	Proposed MCFU-Net Fusion	Multi-scale Hessian Fusion	Proposed MCFU-Net Fusion
Ctrl_Lyve1-(2) No. of slices: 157	21.97	14.64	8.20	3.31	39.96	13.16	0.75
Ctrl_Lyve1-(4) No. of slices: 221	26.00	16.06	8.64	4.13	39.60	29.99	0.69
Ctrl_Lyve1-(7) No. of slices: 234	22.04	14.92	8.06	4.55	34.70	29.82	0.74
Ctrl_Lyve1-(16) No. of slices: 140	25.13	10.45	8.63	3.38	41.06	20.55	0.75
Ctrl_Lyve1-(17) No. of slices: 164	22.29	6.94	8.49	3.25	33.56	14.89	0.77
Ctrl_Lyve1-(18) No. of slices: 143	23.07	10.41	9.88	3.49	31.46	19.54	0.77
Ctrl_Lyve1-(27) No. of slices: 130	26.89	7.61	9.58	3.34	41.39	24.73	0.70
Ctrl_Lyve1-(34) No. of slices: 264	18.92	7.58	8.01	3.15	30.06	11.21	0.80
Ctrl_Lyve1-(37) No. of slices: 267	26.83	9.33	7.96	3.4	29.19	29.51	0.73
Ctrl_Lyve1-(39) No. of slices: 185	22.68	6.58	9.02	3.03	33.55	31.53	0.74
Ctrl_Lyve1-(46) No. of slices: 138	26.77	10.83	8.55	3.18	43.11	12.87	0.73
Ctrl_Lyve1-(51) No. of slices: 257	20.04	5.42	7.98	3.59	22.27	25.11	0.80
Ctrl_Lyve1-q No. of slices: 241	22.96	14.96	8.40	3.72	31.04	17.18	0.75
Ctrl_Lyve1-u No. of slices: 264	25.90	6.47	8.48	3.41	35.26	29.22	0.73
Ctrl_Lyve1-v No. of slices: 229	18.30	7.35	8.41	3.62	28.51	18.32	0.79
Average Scores	23.32	9.97	8.55	3.50	34.31	21.84	0.75

Figure 5 presents the comparison of the Hessian based fusion results and the results of our proposed MCFU-Net fusion. MCFU-Net fusion for red and green channels, as well as the final combined dual channel RGB image along with the quality index beneath each image are shown. If we take a close look visually, it is evident that there is a huge improvement using deep learning. For both of the channels, multi-scale Hessian based fusion results in structural discontinuity, missing structures, scattered structures, and noisy background. These problems are more severe for the more complex green channel compared to the red channel. All these fusion problems are either solved or improved in the MCFU-Net fusion resulting in better structural and image quality. These qualitative and quantitative results demonstrate that the proposed MCFU-Net fusion approach leads to a promising tool for multi-focus image fusion of microscopy images.

Fig. 5:

Comparison results between Hessian fusion method [8] and our proposed MCFU-Net Fusion. Results demonstrates that for both independent single channel fusion and combined channel fusion, our proposed algorithm produces better results in terms of visual interpretation and quantitative scores. For each independent channel fusion, we presented the axial projections too. We presented results for four extremely complicated microscopy image stacks having structures that contains both blood vessels and lymphatic like structures with quantitative scores underneath.

IV. Conclusions

We presented a new approach for multi-focus image fusion of confocal microscopy images that includes extremely complex structures of blood vessels, and lymphatics like structure. Separating the images into two independent channels, reveals that red and green channels have different structural characteristics. Classical derivative-based methods, such as multi-scale Hessian based image fusion, fail to capture the structural complexities of these images, particularly for the green channel corresponding to lymphatics like structures. Our proposed solution, MCFU-Net, relies on a novel vesselness likelihood index computed using a U-Net convolutional neural network trained to segment vascular structures in microscopy images. The learned and data-driven nature of the MCFU-Net fusion approach allows us to better capture the complexities of microvascular structures, and other staining and imaging characteristics compared to hand-crafted fusion methods that rely on smaller number of filters corresponding to single or multi-scale image derivatives. The data-driven nature of this approach also makes it more expandable, flexible, and adaptable to other structures that may be of interest. Improvements in multi-focus fusion performance has implication beyond visual perception. Better fusion is important for follow-up image analysis steps such as segmentation and quantification. Reliable multi-focus fusion is also critical to efficiently summarize and overview the 3D structure information captured in a large number of slices. Our future goal is to use all the fused images to generate a high-resolution mosaic of the whole dura mater vascular networks that may reveal significant unknown anatomical structures and functionalities.