Image denoising in fluorescence microscopy using feature based gradient reconstruction

Abstract.

Purpose

The utility of fluorescence microscopy imaging comes with the challenge of low resolution acquisitions, which severely limits information extraction and quantitative analysis. Image denoising is a technique that aims to remove noise from microscopy acquisitions by taking into account prior statistics of the corrupting noise. In this work, we propose an image denoising technique for fluorescence microscopy imaging.

Approach

The proposed technique is based on the principle of multifractal feature extraction from a noisy sample followed by a reconstruction technique from these features. It is observed that by following a proper hierarchical classification procedure, meaningful features can be extracted from a noisy image. A denoised image is then estimated from this sparse feature set through proper formulation of an optimization problem.

Results

Experiments are performed on both synthetic image databases as well as on real fluorescence microscopy data. Superior denoising results, in comparison to multiple comparing techniques, validate the potential of the proposed approach.

Conclusion

The proposed method gives superior denoising results for low resolution fluorescence microscopy image acquisitions and can be used for post processing of data by biologists.

1. Introduction

Over a decade, fluorescence microscopy has become a tool of choice for biologists conducting in vivo analysis of cellular structures. However, due to various physical constraints, such as short exposure time, photo-toxicity, photo-bleaching, device imperfections, etc., the resulting image is a low resolution noisy acquisition. This seriously hampers further analysis like cell segmentation,¹ recognition,² reconstruction,³ etc of the acquired samples. Noise in fluorescent microscopy acquired samples is characterized by a mixed Poisson–Gaussian statistics.^4–8 Image denoising is a technique that specifically addresses noise removal in a sample through proper knowledge of the noise source and its distribution.

Multiple denoising strategies have been proposed in the literature for improving the quality of the low resolution fluorescent microscopy samples. These strategies range from Bayesian based formulations,^9,10 wavelet based techniques,^11,12 non-local patch based techniques^4,13 to variance stabilization (VST) based approaches,^14,15 to name a few. Bayesian based techniques rely on deriving an appropriate noise restoration model using the Bayes principle, taking into account the statistics of the nature of the noise. In this regard, the denoising model in Ref. 9 is particularly remarkable as it derives a convex optimization model using the Bayes formulation, for fluorescent microscopy samples, based on the Poisson–Gaussian likelihood. Wavelet based denoising schemes operate on manipulating the wavelet coefficients within the framework of forward and inverse wavelet transform. The SURE-LET¹² approach is one such technique. It is an orthonormal wavelet thresholding based denoising scheme that takes into account the interscale dependencies between the wavelet coefficients for removing noise. This scheme has been extended to a Poisson unbiased risk estimate-linear expansion of thresholds (PURE-LET)¹¹ scheme, where in the authors have generalized the concept of Haar wavelet transform to fit into a series of linear expansion of thresholds. Patch based non-local denoising techniques, like,⁴ work on the principle of exploiting patch-based redundancy within a given sample. The NDSAFIR technique⁴ is one such method, where the authors take into account the spatio-temporal image patches of observed fluorescent samples for minimizing an objective non-local energy functional. Stein’s unbiased risk estimator (SURE) is a similarity criteria that is often used in image denoising for predicting optimal solutions. In PG-URE,⁵ the authors extend the concept of SURE to address the case of Poisson–Gaussian noise removal in fluorescent samples. Another popular category of image denoising technique are the VST based techniques.^14,15 VST based methods make use of appropriate transformations to convert the signal dependent noisy component into signal independent noise. VST + BM3D¹⁵ and GAT + BM3D¹⁴ are two such popular denoising schemes for fluorescent microscopy samples that make use of the Anscombe transform as a stabilizier for the signal-dependent Poisson noise.

Deep learning based denoising techniques have also become popular in recent years.^16,17 A content-aware image restoration based on deep learning is proposed in Ref. 18 for denoising fluorescence microscopy samples. A three dimensional deep neural network based residual channel attention denoising network for fluorescence microscopy is proposed in Ref. 19. In addition to these, self-supervised denoising networks that work on the principle of learning only from the single noisy image is also widely used for fluorescence microscopy denoising (FMD). The Self2Self²⁰ model is one such network that uses Bernoulli sampling as a dropout strategy for denoising images. The neighbor2neighbor (Nb2Nb)²¹ is another approach that uses a random neighbor up-sampler to generate training data for effective denoising.

The denoising technique that we propose in this paper is based from ideas encountered in the analysis of multifractal systems.^22–25 These are scale-free complex systems whose characteristic quantities display a power-law behaviour and are characterized by values called singularity exponents (SEs).²⁶ Natural scale-free complex systems exhibit multifractal traits²⁷ implying thereby that signals acquired from such systems can be characterized by multifractals, an approach to complex systems, which has been validated over a large dataset.²² The denoising scheme that we propose is a two-step concept where in the first step we extract a sparse feature map through multifractal analysis on the acquired noisy image. In the second step, we build an optimization problem to estimate a denoised image from the gradients confined to the extracted feature map. The contribution of this paper are as follows.

• •We formulate a novel denoising algorithm that uses multifractal analysis to extract meaningful features from a noisy image and then reconstruct a denoised image from these features. Under the hypothesis of a multifractal system, the integrated signal’s gradient of the original image has a power-law behaviour whose exponent at each point $x$ of the signal’s domain, $h(x)$, known as an SE, can be computed using a local correlation measure.
• •We recall the computation of the local correlation measure and of the set of lowest exponents, ${\mathcal{L}}_{\infty }$, which is is of fundamental significance in signal processing as it allows the reconstruction of a new signal from the SEs.
• •We design a reconstruction algorithm to estimate a denoised image from the feature set ${\mathcal{L}}_{\infty }$.
• •Experimental results over three image databases, under different levels of noise, show that the proposed method outperforms both classical denoising techniques as well as recent deep learning techniques in the field.

The rest of the paper is organized as follows: in Sec. 2.1, we mathematically formulate the image acquisition problem and objective and briefly summarize the multifractal analysis of acquired samples. In Sec. 2.3, we propose a technique for computing the quantity $h(\mathbf{x})$ required for estimating the feature set ${\mathcal{L}}_{\infty }$, and in Sec. 2.4, we propose a method for estimating a denoised image from ${\mathcal{L}}_{\infty }$. In Sec. 3, we present the experiment results and discuss them and finally conclude in Sec. 4.

2. Proposed Method

2.1. Preliminaries and Objective

If $f:\mathrm{\Omega }\to \mathbb{R}$ is the acquired noisy image defined on the image domain $\mathrm{\Omega }\subset \mathbb{R}$ and $u:\mathrm{\Omega }\to \mathbb{R}$ its noise free version, the image formation process for fluoescence microscopy can be written as^5,11

$$f=\tau (u)+w,$$ (1)

where $\tau (·)\sim \mathcal{P}(·/\zeta )$ represents the signal dependent noisy component with $\zeta \ge 0$ being the gain of the overall image acquisition system. $w\sim \mathcal{N}(b,{\sigma }^2)$ represents the signal independent additive noise component with mean $b$ and variance ${\sigma }^2$. $\sigma ,\zeta \ge 0$ determines the intensity of Poisson–Gaussian noise in $f$.⁵ $(\sigma ,\zeta )$ were empirically determined keeping into account the fact that higher the $\sigma$ value, higher the Gaussian noise. Similarly higher the $\zeta$ value, higher is the Poisson noise. Noise is generated with different sets of values of $(\sigma ,\zeta )$, over the ground truth, following the above equation. We intend to recover an image $\widehat{u}\approx u$ from the noisy fluorescent microscopy sample $f$.

2.2. Multifractal Analysis and Feature Extraction

As discussed earlier, natural scale-free systems exhibit self-similarity and multifractal behavior implying therefore that signals (or images) acquired through such systems will exhibit similar traits. If $f(\mathbf{x})$ (where $\mathbf{x}$ is a vector representing the location of a pixel), which in our case is the acquired noisy image, is a signal with multifractal properties then it can be used to define a measure with multi-scale properties,^22–24,28 $\mu (\mathbf{x},r)$ ($\mathbf{x}$: pixel location, $r$ scale) such that we have the limiting behaviour

$$h(\mathbf{x})=\underset{r\to 0}{\lim }\frac{\log \mu (\mathbf{x},r)}{\log r},$$ (2)

which in turn implies that

$$\mu (\mathbf{x},r)\approx \alpha (\mathbf{x})r^{h(\mathbf{x})},$$ (3)

where $\alpha (\mathbf{x})$ is a signal dependent constant. $h(\mathbf{x})$ is known as the singularity exponents or SE’ s. The transitions within a signal are well-recorded in $h(\mathbf{x})$,²⁶ which are in turn independent of the measure $\mu$, as long as it is multiscale. There are different ways to compute the $h(\mathbf{x})$, once $\mu$ is defined properly. The most evident algorithm consists in noting that Eq. (3) implies that, by taking the logarithms on both sides

$$\log (\mu (\mathbf{x},r))=\log (\alpha (\mathbf{x}))+h(\mathbf{x})\log (r)(r\to 0),$$ (4)

and then perform log-log regression for many different scales $r$. However, the accuracy of such method depends on the number of scales $r$ chosen for the regression; it has been shown²⁸ (and the references herein) that a most advanced methodology consists in evaluating $h(\mathbf{x})$ by use of a local correlation measure. This is justified by the existence of a local reconstruction of the whole signal at the points where $h(\mathbf{x})$ at the lowest values.²⁸ We postpone this computation to Sec. 2.3 below. For the moment, supposing that $h(\mathbf{x})$ has been properly calculated, they are used to define a multifractal hierarchy of multiple fractals ${\mathcal{L}}_h$ as level sets: ${\mathcal{L}}_h=\{\mathbf{x}|h(\mathbf{x})=h\}$,²⁵ from where it is possible to extract a particular set ${\mathcal{L}}_{\infty }$ consisting of components associated to the smallest possible value $h_{\infty }$ defined as

$${\mathcal{L}}_{\infty }=\{\mathbf{x}|h(\mathbf{x})=h_{\infty }=\min (h(\mathbf{x}))\}.$$ (5)

The $h_{\infty }$’s only encode the pixels where it observes sharp transitions in the image i.e., the boundaries of objects inside the image, and maintains its stability under noise. This can be visualized in Fig. 1 (middle row). The procedure for computing the set ${\mathcal{L}}_{\infty }$ has been listed in Algorithm 1.

Fig. 1.

Top: noisy realizations. Middle: ${\mathcal{L}}_{\infty }$ computed over them and bottom: proposed denoising results.

Algorithm 1.

Input: $h(\mathbf{x})$ and output: ${\mathcal{L}}_{\infty }$.

Step 1: Compute the multifractal exponents $h(\mathbf{x})$ from Algorithm 2.

Step 2: Define density $${\delta }_{{\mathcal{F}}_{\infty }}(\mathbf{x})=\{\begin{array}{ll}1& \text{if}{h}_{\infty }-\mathrm{\Delta }h\le h(\mathbf{x})\le h_{\infty }+\mathrm{\Delta }h\\ 0& \text{otherwise}\end{array}$$

Step 3: $\mathrm{\Delta }h$ is conventionally fixed (0.2 is usually a good choice).

Step 4: ${\delta }_{{\mathcal{L}}_{\infty }}(\mathbf{x})$ is the binary map that locates the ${\mathcal{L}}_{\infty }$ pixels.

2.3. Local Correlation Measure for the Computation of $h(\mathbf{x})$

As explained in the last subsection, a local correlation measure can be used to evaluate the SEs $h(\mathbf{x})$, instead of log-regression across the scales (which necessitates to define various multiscale versions of the original signal). The key observation lies in the existence of the set ${\mathcal{L}}_{\infty }$, which allows a local reconstruction of the signal from the most singular exponents (see Secs. 2.4 and Ref. 28). In this study, we propose a new local correlation measure, well adapted the noise reduction problem we have in sight.

In this paper, we propose a method to compute $h(\mathbf{x})$ at every pixel location $\mathbf{x}=(i,j)$ in the image taking into consideration their four neighbouring pixels: denoting $f(i,j)$ the gray-level value at pixel $\mathbf{x}=(i,j)$, the four neighbouring pixel values form the set $\{f(i-1,j),f(i+1,j),f(i,j-1),f(i,j+1)\}$. In an image of size $N\times M$ ($N,M$: numbers of pixels in the $x$ and $y$ directions, respectively) and assuming the whole image of surface 1, the minimum resolution is $r_0=\frac{1}{\sqrt{N\times M}}$. The complete procedure to compute $h(\mathbf{x})$ at resolution $r_0$ is given in Algorithm 2 (where $*$ denotes convolution and $T$ stands for transposition). Note that at minimum fixed resolution $r_0$, $\mu (x,r_0)$ depends only on $x$.

Algorithm 2.

Input: image $f$ and output: SE $h(\mathbf{x})$ at $\mathbf{x}=(i,j)$.

Step 1: ${ϵ}_x=f\star [10-1])$; ${ϵ}_y=f\star -{[10-1]}^T$, ${ϵ}_x,{ϵ}_y$ correspond to unit pixel displacements at maximum scale $r_0$.

Step 2: $e(\mathbf{x})={ϵ}_x^2+{ϵ}_y^2$.

Step 3: $\mu (\mathbf{x},r_0)=\sqrt{e(\mathbf{x})}$.

Step 4: $h(\mathbf{x})=\frac{\log \mu (\mathbf{x},r_0)}{\log r_0}$.

2.4. Reconstructing the Denoised Image

In this section, we propose a novel formulation to reconstruct a denoised image $\widehat{u}\approx u$ from the sparse feature set ${\mathcal{L}}_{\infty }$. Such a reconstruction comes from the possibility on integrating gradients over the set ${\mathcal{L}}_{\infty }$ (see Refs. 25 and 28). However in this section we will introduce a new reconstruction algorithm that makes use of a L2-L1 minimization with sparse gradients instead of reconstruction in Fourier space as in Refs. 25 and 28. This choice is motivated by our primary goal of noise reduction, for which sparse methods have been shown to be particularly effective.^29–31

Let $f_x,f_y$ be the gradients of the noisy image $f$ and ${\delta }_{{\mathcal{L}}_{\infty }}$ be the binary map of the set ${\mathcal{L}}_{\infty }$. We intend to recover $u$ from the sparse gradient set $\mathbf{v}=\{{\widehat{f}}_x,{\widehat{f}}_y\}$, where ${\widehat{f}}_x=f_x·{\delta }_{{\mathcal{L}}_{\infty }}$ and ${\widehat{f}}_y=f_y·{\delta }_{{\mathcal{L}}_{\infty }}$. Since ${\mathcal{L}}_{\infty }$ encodes the most meaningful pixels of the image, the gradients corresponding to ${\mathcal{L}}_{\infty }$ will be the most informative. A reconstruction from the set $\mathbf{v}$ will therefore lead to a proper estimation of the whole image and noise propagation will be automatically minimal since a lot of noisy pixels are dropped. Keeping this in mind, we therefore propose to minimize the following optimization problem:

$$\widehat{u}=\underset{u}{\arg \min }\frac{1}{2}{\Vert \nabla u-\mathbf{v}\Vert }_2^2+\lambda {\Vert \nabla u\Vert }_1,$$ (6)

where $\lambda$ is a positive regularization parameter. With half-quadratic splitting, using an intermediate variable $z$ and an additional positive regularization term $\beta$, we get

$$\widehat{u}=\underset{u,z}{\arg \min }\frac{1}{2}{\Vert \nabla u-\mathbf{v}\Vert }_2^2+\lambda \{{\Vert z\Vert }_1+\frac{\beta }{2}{\Vert \nabla u-z\Vert }_2^2\},$$ (7)

which is further organized into independent sub-problems $p_1$ and $p_2$

$$\begin{gathered} p_1:z^{(k+1)}\leftarrow \underset{z}{\arg \min }\Vert z\Vert +\frac{\beta }{2}{\Vert \nabla u^{(k)}-z\Vert }_2^2 \\ {p}_2:u^{(k+1)}\leftarrow \underset{u}{\arg \min }{\Vert \nabla u-\mathbf{v}\Vert }_2^2+\lambda {\Vert \nabla u-z^{(k+1)}\Vert }_2^2, \end{gathered}$$

Problem $p_1$ is in the proximal form $\underset{z}{\arg \min }{\Vert z\Vert }_1+\frac{\beta }{2}{\Vert x-z\Vert }_2^2$ whose solution correspond to a soft thersholding shrinakge operator³⁰

$$z={\text{shrink}}_{l_p}(x,\beta )=\max \{0,|x|-\frac{{|x|}^{p-1}}{\beta }\}\frac{|x|}{x}.$$ (8)

Considering $z$ as a two-component vector, the solution to sub-problem $p_1$ is given as

$$z^{(k+1)}=\{\begin{array}{l}{z}_x^{(k+1)}={\text{shrink}}_{l_1}({\nabla }_x{u}^{(k)},\beta )\\ z_y^{(k+1)}={\text{shrink}}_{l_1}({\nabla }_y{u}^{(k)},\beta ).\end{array}$$ (9)

Problem $p_2$ is in the regular quadratic form and can be solved using Euler–Lagrange. The final solution using FFT will be

$$u^{(k+1)}={\mathbf{F}}^{-1}\left\{\frac{\mathbf{F}(\mathrm{div}(\mathbf{v})+\mathrm{div}(z^{(k+1)}))}{(1+\lambda )\mathrm{\Delta }}\right\}.$$ (10)

where $\mathbf{F}$ denotes Fourier transform, $\mathrm{\Delta }={\nabla }^2$ the Laplace operator. The results of reconstruction, under different levels of noise, are shown in Fig. 1 (bottom row) and Fig. 2.

Fig. 2.

Result over synthetic data corrupted with MPG noise. Top row: $\sigma ,\zeta ={10}^{-2.5}$, ${10}^{-3}$; middle row: $\sigma ,\zeta ={10}^{-1}$, ${10}^{-2}$; and bottom row: $\sigma ,\zeta ={10}^{-2}$, ${10}^{-1.5}$.

3. Experiment and Results

We have used two types of data: synthetic data and fluorescence microscopy acquisition data. For comparison purpose, we have considered the four popular classical denoising techniques designed for removing MPG noise: SURE-LET,¹¹ PG-URE,⁵ VST+BM3D,¹⁵ and GAT+BM3D.¹⁴ We have also used two recent deep learning based self-supervised denoising methods that learns only from the input noisy image, thereby making them ideal for comparison. They are: Self2Self²⁰ and Nb2Nb.²¹ We have also used denoising convolutional neural network (DnCNN)³² and noise2noise (N2N)³³ models for comparison on FMD dataset.⁸ The parameter settings for all these methods have been kept the same as suggested by their authors in the papers and in the codes. For the purpose of quantitative evaluation, we have considered the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) metrics, defined as follows:

$$\mathrm{PSNR}=10{\log }_{10}\frac{u_{\max }^2}{({\Vert \widehat{u}-u\Vert }^2/(m\times n))},$$ (11)

where $u_{\max }$ is the maximum intensity of the noise-free image $u$ and $m\times n$ is the size of the image $u$

$$\mathrm{SSIM}(u,\widehat{u})=\frac{(2{\mu }_u{\mu }_{\widehat{u}}+c_1)(2{\sigma }_{u\widehat{u}}+c_2)}{({\mu }_u^2+{\mu }_{\widehat{u}}^2+c_1)({\sigma }_u^2+{\sigma }_{\widehat{u}}^2+c_2)},$$ (12)

where ${\mu }_u,{\mu }_{\widehat{u}}$ are the average of $u$ and $\widehat{u}$; ${\sigma }_u^2,{\sigma }_{\widehat{u}}^2$ are the variance of $u$ and $\widehat{u}$; ${\sigma }_{u\widehat{u}}$ the covariance of $u$ and $\widehat{u}$ respectively; and $c_1$ and $c_2$ are two normalizing parameters.

3.1. Synthetic Data

We have considered three image datasets: the Classic5, Set12, and BSD68, which consist of 5, 12, and 68 synthetic images, respectively. We have generated multiple noisy versions of these datasets, by adding MPG noise of level $\{\sigma ,\zeta \}=\{{10}^{-2.5},{10}^{-3}\}$, $\{{10}^{-1},{10}^{-2}\}$, and $\{{10}^{-2},{10}^{-1.5}\}$, over which we perform denoising. In Fig. 2, we present the visual comparison results of our technique with the competing methods. As can be seen from the excerpts (highlighted in green bounding boxes), SURE-LET is capable of retaining the details to a certain extent, but introduces artefacts in the denoised output. PG-URE, VST + BM3D, and GAT+BM3D does better denoising than SURE-LET but are mostly accompanied by over smoothing, which leads to loss in details. For the cameraman image, one can see that the background building behind the tripod, in the case of boat image the paint linings corresponding to the boat name and in the case of foreman image the black spots in the building behind are best reconstructed by the proposed method. The proposed technique is therefore able to retain the fine details of the image while rendering better visual quality than the others in terms of noise removal. This is also justified quantitatively by the results of the mean PSNR and mean SSIM, calculated over all the images of the datasets, under different levels of noise, as is shown in Table 1.

Table 1.

Quantitative evaluation using PSNR [refer Eq. (11)] and SSIM [refer Eq. (12)] metric over standard image dataset. Best cases are highlighted in bold.

Param			Mean PSNR
	$\sigma$	$\zeta$	Input	PG-URE	SURE-LET	VST + BM3D	GAT + BM3D	Self2Self	Nb2Nb	Proposed
Classic5	${10}^{-2.5}$	${10}^{-3}$	24.56	33.33	32.90	32.19	32.01	30.20	30.02	34.91
	${10}^{-1}$	${10}^{-2}$	18.48	25.04	23.84	23.90	25.32	24.78	24.14	26.84
	${10}^{-2}$	${10}^{-1.5}$	13.88	22.824	21.96	20.73	21.53	20.19	20.87	24.90
Set12	${10}^{-2.5}$	${10}^{-3}$	25.99	33.92	31.09	30.79	30.42	29.85	29.64	35.46
	${10}^{-1}$	${10}^{-2}$	18.80	25.16	24.14	25.18	25.38	24.64	24.48	26.45
	${10}^{-2}$	${10}^{-1.5}$	13.45	22.41	20.48	22.82	22.57	21.48	21.98	24.67
BSD68	${10}^{-2.5}$	${10}^{-3}$	25.37	31.23	28.25	30.22	29.77	28.35	28.85	33.89
	${10}^{-1}$	${10}^{-2}$	17.39	23.27	22.61	23.64	23.78	22.37	22.13	24.87
	${10}^{-2}$	${10}^{-1.5}$	13.52	21.53	19.50	21.84	21.53	20.63	20.23	23.07
Param			Mean SSIM
Classic5	${10}^{-2.5}$	${10}^{-3}$	0.751	0.929	0.916	0.870	0.865	0.854	0.856	0.942
	${10}^{-1}$	${10}^{-2}$	0.597	0.792	0.711	0.756	0.772	0.714	0.710	0.817
	${10}^{-2}$	${10}^{-1.5}$	0.417	0.657	0.622	0.644	0.653	0.620	0.624	0.688
Set12	${10}^{-2.5}$	${10}^{-3}$	0.768	0.934	0.924	0.918	0.905	0.893	0.896	0.954
	${10}^{-1}$	${10}^{-2}$	0.588	0.784	0.695	0.771	0.770	0.707	0.701	0.811
	${10}^{-2}$	${10}^{-1.5}$	0.437	0.662	0.547	0.654	0.645	0.631	0.635	0.694
BSD68	${10}^{-2.5}$	${10}^{-3}$	0.748	0.929	0.891	0.905	0.911	0.898	0.904	0.943
	${10}^{-1}$	${10}^{-2}$	0.568	0.765	0.700	0.755	0.748	0.702	0.701	0.798
	${10}^{-2}$	${10}^{-1.5}$	0.444	0.654	0.588	0.648	0.632	0.611	0.621	0.688

3.2. Fluorescence Microscopy Data

The dataset that we have used is the FMD dataset.⁸ The visual results of denoising are shown in Figs. 3–5, respectively. The ground truth images shown in these figures are obtained (as described by the authors) image averaging over 50 realizations of the noisy image. In Fig. 3, the denoising results of a single channel (gray) image of zebrafish embryo captured under confocal microscopy is shown. The image is of size $512\times 512\text{pixels}$. It can be observed from the zoomed section that the details of the embryo are best preserved in the output of the proposed method. N2N and VST + BM3D also tries to retain the details to a certain extent but are not that accurate in comparison to the output of the proposed method. PG-URE and GAT + BM3D gives a blurred impression whereas SURE-LET and DnCNN gives oversmoothed results. Figure 4 shows the denoising results of a single channel (gray) image of mice brain captured under two photon microscopy. The image is of size $512\times 512\text{pixels}$. Careful observation shows that the proposed method better reconstructs the details and sharpness of the white spots as well as the blob like structures inside the cells, than the N2N model, which is slightly smoothed in its output. DnCNN, VST + BM3D, and PG-URE also gives better denoising but has smoothed outputs compared to the proposed model and N2N. The rest gives over moothened outputs. Quantitative evaluation of Figs. 3 and 4, as shown in Table 2, also justifies the superior denoising results of the proposed method.

Fig. 3.

Result over FMD dataset: single-channel (gray) image of zebrafish embryo under confocal microscopy.

Fig. 4.

Result over FMD dataset: single-channel (gray) image of mice brain under two-photon microscopy.

Fig. 5.

Result over FMD dataset: multi-channel (color) image of BPAE cells under widefield microscopy.

Table 2.

Quantitative evaluation using PSNR [refer Eq. (11)] and SSIM [refer Eq. (12)] metric on FMD datasets. Best cases are highlighted in bold.

Sample	Mean PSNR
	Raw	PG-URE	SURE-LET	VST + BM3D	GAT + BM3D	DnCNN	N2N	Proposed
Confocal zebrafish	22.76	30.63	27.75	31.46	30.86	32.24	33.04	35.66
Two-photon mice brain	22.37	33.46	28.38	34.25	33.22	34.58	34.91	36.06
Widefield BPAE	20.06	27.05	28.05	27.73	27.78	28.76	29.69	29.11
Sample	Mean SSIM
Confocal zebrafish	0.427	0.834	0.856	0.883	0.888	0.898	0.911	0.947
Two-photon mice brain	0.278	0.907	0.798	0.915	0.908	0.922	0.924	0.949
Widefield BPAE	0.759	0.885	0.927	0.892	0.911	0.914	0.968	0.944

Figure 5 shows the denoising results over a multi channel (color) image of bovine pulmonary artery endothelial (BPAE) cells captured under widefield microscopy. The image is of size $512\times 512\times 3\text{pixels}$. It is clear from the results that the proposed method gives the best denoising while retaining the finer details. The zoomed section shows that the filament like structures are well retained as well as the circular background patch. The rest of the denoising method fails to retain such detail and sharpness. The color contrast is also improved in the denoising results. It is to be noted that the FMD dataset ground truth images are obtained by image averaging over 50 realizations of the noisy image. The ground truth is therefore also contaminated with noise, albeit less than the noisy acquisition. For a high noise acquisition like BPAE (which has multiple noisy channels), this is more pronounced than compared to single channel acquisitions of zebrafish and mice brain. PSNR and SSIM of N2N is highest for BPAE because N2N has been trained on the FMD dataset (where it has considered the less noisy ground truth as the clean image during training) and has therefore recovered the low noise also in its reconstruction. The proposed method (as well as PG-URE, SURE-LET, VST + BM3D, GAT + BM3D, and DnCNN) cleans the background noise completely. However its PSNR and SSIM values will be lower because it mismatches with the low noisy ground truth. The ground truth of BPAE is therefore not suitable for quantitative analysis using PSNR and SSIM.

3.3. Run Time

In Table 3, we provide the running time for all the comparing methods for real data all of whose image size are $512\times 512\text{pixels}$. It is to be noted that PG-URE, SURE-LET, VST+BM3D, GAT+BM3D, and the proposed methods are Matlab based methods and utilize the CPU. We tested it on a standard core i5 8th Gen laptop with 8GB RAM. DnCNN and N2N are deep learning models and ultilize the GPU, for which they are faster but computationally expensive in terms of resource utilization.

Table 3.

Running time comparison.

Sample	Running time (in seconds)
	CPU running time					GPU running time
	PG-URE	SURE-LET	VST + BM3D	GAT + BM3D	Proposed	DnCNN	N2N
Widefield BPAE	16.56	4.54	8.23	4.15	3.9	0.19	0.08

Algorithm 1 always achieves real-time. Half quadratic splitting depends of the size of the input image. For a $10000\times 10000\text{pixel}$ image, GPU implentation of half quadratic splitting can be reduced down to 1 s CPU time on a high level desktop (we tested it on Nvidia Quadro NVS 290 with Cuda).

4. Conclusion

This paper presents a denoising technique for recovering low resolution fluorescence microscopy samples. The denoising technique is motivated from the concept of multifractal feature extraction from an observed noisy sample and subsequent reconstruction from the gradient information limited to those features. The feature extraction method records only the most informative pixels and drops the remaining pixels. A denoised image is then reconstructed from the gradients of this feature set by solving a half-quadratic based optimization problem. Experiments were performed on both synthetic image databases and real fluorescence microscopy data. Results, both qualitative and quantitative, have proved the superior denoising potential of the proposed technique over multiple competing techniques in the field.

Biographies

Suman Kumar Maji received his BTech degree in electronics and communication engineering from West Bengal University of Technology, India, in 2006; postgraduate degree in telecommunication networks from the Indian Institute of Technology Kharagpur, India, in 2008; and PhD in computer science from INRIA Bordeaux France in 2013. From 2014 to 2015, he worked as a research engineer at the Institute of Hematology, University Paris 7 and INSERM. He is currently an assistant professor with the Department of Computer Science and Engineering at the Indian Institute of Technology Patna, India. His research interests are in the area of medical imaging, bioinformatics, machine learning, and image processing. He has authored several conferences and journal papers and is the recipient of various research fellowships and awards, such as the European CORDIS Doctoral Fellowship (2010), Region Aquitaine OPTAD Research Fellowship (2010), FRM Research Fellowship (2014), SERB Early Career Research Award from DST, Govt of India (2017), etc.

Hussein Yahia (M’87) received his doctorat de troisième cycle from the University of Paris-Sud, Orsay, France, in 1987, and the habilitation à diriger des recherches from Paris 13 University, Villetaneuse, France, in 2003. He is currently the head of the Geostat (Geometry and Statistics in Acquisition Data) Research Team with the French National Public Research Institute in Computer Science and Applied Mathematics, Bordeaux, France. He specializes in the analysis of complex signals and systems using approaches from statistical physics; he is also studying sparse signal representations and filtering with specific application to signals in astrophysics in collaboration with the Astrophysics Laboratory of Bordeaux (analysis of turbulent properties in the interstellar medium). He is involved in many national, European, and international contracts and has been supervising over 15 PhD students. He has authored or coauthored more than 100 publications in international peer-reviewed top journals and conferences.

Disclosures

The authors have no relevant financial interests in the manuscript and no other potential conflicts of interest to disclose.

Code and Data Availability

The datasets analyzed during the current study are available in the CBSD68 repository, https://paperswithcode.com/dataset/cbsd68, and FMD repository.⁸