Perceptually Motivated Generative Model for Magnetic Resonance Image Denoising

Abstract

Image denoising is an important preprocessing step in low-level vision problems involving biomedical images. Noise removal techniques can greatly benefit raw corrupted magnetic resonance images (MRI). It has been discovered that the MR data is corrupted by a mixture of Gaussian-impulse noise caused by detector flaws and transmission errors. This paper proposes a deep generative model (GenMRIDenoiser) for dealing with this mixed noise scenario. This work makes four contributions. To begin, Wasserstein generative adversarial network (WGAN) is used in model training to mitigate the problem of vanishing gradient, mode collapse, and convergence issues encountered while training a vanilla GAN. Second, a perceptually motivated loss function is used to guide the training process in order to preserve the low-level details in the form of high-frequency components in the image. Third, batch renormalization is used between the convolutional and activation layers to prevent performance degradation under the assumption of non-independent and identically distributed (non-iid) data. Fourth, global feature attention module (GFAM) is appended at the beginning and end of the parallel ensemble blocks to capture the long-range dependencies that are often lost due to the small receptive field of convolutional filters. The experimental results over synthetic data and MRI stack obtained from real MR scanners indicate the potential utility of the proposed technique across a wide range of degradation scenarios.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10278-022-00744-2.

Introduction

Magnetic resonance imaging (MRI) is a popular and effective medical imaging technique used to visualize anatomical details and analyse physiological processes in the human body’s soft biological tissues. This aids in making clinical decisions at the appropriate time. MRI differs from CT scan in that it does not use ionizing radiation. It takes advantage of the magnetic properties of hydrogen protons, which are abundant in the human body. Strong primary static magnetic field and low variable secondary field in the presence of radio-frequency (RF) coils help in the spatial encoding of the received signal. An acquired MRI sequence is limited by its low temporal resolution. Further, images acquired in temporary k-space are composed of real and imaginary parts in the frequency domain [1]. Image reconstruction using the inverse Fourier transform (IFT) yields magnitude and phase images [2]. However, the resulting image is found to have Gaussian noise due to multiple factors. These include thermal agitation caused by the human body in the receiver’s coil elements, tissue inhomogeneity, static field intensity, and receiver element bandwidth [3]. Impulse noise is caused by factors such as analog-to-digital conversion, bit error transmission, and improper RF coil alignment. MR image restoration [4] is a longstanding problem in the scientific literature and different methods appear to tackle this problem. Clinically accurate restoration results enable successful downstream analysis such as disease detection, tissue localization, and quantitative MRI analysis [5].

Images in MRI are obtained in the complex domain using the method proposed by Kumar et al. [6] and Henkelman [1]. This complex domain representation includes both real and imaginary parts. Images obtained from the RF coils are in the frequency domain and are represented in a temporary space known as k-space. The resulting data contains both signal and noise components along both real and imaginary planes. Due to thermal noise in the patient [2], the noise is complex white Gaussian in nature. The signal must be converted to imaging domain before any useful information can be extracted from it. Inverse discrete Fourier transform (IDFT) is the most common tool available to accomplish this. The resultant image obtained from IDFT retains the Gaussian noise in the data due to the orthogonality of the Fourier transform [2]. We present an extensive literature review on recent developments in MR image denoising works in the following paragraphs. Filtering-based and model-driven approaches for noise removal are presented in the following first and second paragraphs respectively. Learning-based techniques are discussed in the third paragraph while the final paragraph explains MR image denoising schemes devoted exclusively to the removal of impulse and mixed Gaussian-impulse noise.

In an attempt to investigate previous works on MR image denoising, it was discovered that many of them used the non-local means (NLM) filter in various forms [7]. The authors of [8] used NLM for 3D denoising of MR stacks through block-wise and parallel implementation with automatic tuning of the smoothing parameter. In [9], authors propose a locally adaptive noise estimation method to accommodate parallel imaging environments such as GeneRalized Autocalibrating Partial Parallel Acquisition (GRAPPA) [10] where noise is spatially variant. The authors of [11] used an adaptive soft wavelet coefficient mixing to perform adaptive MR denoising in a multi-resolution framework. In another work, the properties of self-similarity and sparsity in MR data are explored [12]. This is accomplished through hard thresholding of the discrete cosine transform (DCT) and a rotationally invariant version of the NLM filter. Other attempts at NLM-based denoising in natural images in general, and MR data in particular, include [13] and [14]. Authors in [7] provide a comprehensive overview of denoising in MRI using NLM. Here, NLM methods are classified into four categories: fast NLM, adaptive NLM, multi-resolution NLM, and statistical methods. However, because the weighted average of the neighbouring voxels must be estimated to find the target pixel, NLM-based methods are mostly computationally intensive.

Authors in [15] investigate a model-free approach for denoising diffusion-weighted (DW) MR data using principal component analysis (PCA). Similarly, authors in [16] attempted to denoise DW-MRI using random matrix theory and PCA of noise by exploiting the fact that eigenvalues of noise-only data are corrupted by the Marchenko-Pastur distribution. Prior to obtaining magnitude data, Wirestam et al. [17] using Weiner filtering in the wavelet domain, performed complex-valued DW-MRI denoising. A linear minimum mean square error (LMMSE) approach is used to denoise 3D MR data by leveraging data redundancy and estimating local SNR [18]. In another work [19], authors employ an adaptive combination of fuzzy logic and bilateral filtering to denoise MR data. Total Variation (TV) [20] has also been used in several methods for removing noise from MR data. Contributions by Liu et al. [21] include a two-step wavelet filtering method to estimate noise maps and a TV-based hyper-Laplacian parameter to adaptively model spatially varying noise parameters. Similar approach is explored in [22]. Zhang et al. [23] discuss an alternating minimization approach that uses a Gaussian mixture model (GMM) to cluster non-local similar patches of MR data and a hyper-Laplacian prior to reduce ringing artefacts. Authors in [24] investigate a comparison of Gaussian denoising and wavelet-based approaches for denoising functional MRI (fMRI) data. However, TV-based priors are limited in their ability to characterize different levels of data and noise information [25], resulting in smoothed results in highly textured regions. In general, model-driven methods are restricted by their prior assumptions on noisy/clean data. These assumptions, however, are frequently violated in data acquired with diverse commercially available MRI scanners.

Since the last few years, there has been a surge of machine learning approaches in general and deep learning in particular for a wide class of computer vision tasks in medical imaging; including magnetic resonance imaging [26]. In [27], spatial-temporal denoising of dynamic contrast enhanced (DCE) MR data is performed using a deep learning framework to study the blood-brain barrier using pharmacokinetic parameters. Authors in [28] incorporate a 3D channel residual learning strategy for denoising MR images. The authors of [29] and [30] used a wide residual learning strategy to remove noise from MR data in order to increase the receptive field size and speed up training. A 3D MR image denoising is performed in [31] using a Wasserstein generative adversarial network (WGAN) in a residual setting. In [32], the authors propose a two-phase denoising model in which they estimate a signal-dependent and spatially variant noise level map using a maximum-likelihood based unbiased estimator. This estimated noise level map is appended as prior to the corrupted input data into the denoiser model. Bermudez et al. [33] and Moreno López et al. [34] are two other data-driven MRI denoising schemes. Most deep learning methods, however, necessitate a large amount of training data. In practice, generating labelled groundtruth of this size is not feasible. This lack of labelled groundtruth limits the model’s ability to accommodate the intricate noise patterns observed in real MR data. This is further exacerbated by computationally demanding training of 3D MR patches.

It was discovered that the combination of Gaussian and impulse noise significantly contributes to the corruption of MR data. Most of the works in the previous paragraphs emphasized the presence of Gaussian noise in complex frequency domain images obtained from RF coils. There is a wealth of literature available that denoises MR data under the assumption of impulse noise. The authors of [35] evaluated the denoising results for the removal of Gaussian and impulse from T2-weighted MR images using three different filters: adaptive, averaging, and median filters. A dual fidelity elastic net regularization is proposed in [36] to collectively handle the effect of Gaussian and impulse noise. In [37], a joint approach using linear and median filters for the elimination of impulse, fluctuation, and geometric noise from heart MRI was proposed. Two successful adaptive median filtering (AMF) strategies for the removal of impulse noise from MRI data are [38] and [39]. In [38], authors employed a switch-based AMF where a bell-shaped membership function instead of triangular membership function was used for obtaining better results. The same set of authors extended their work in [39] where neuro-fuzzy logic was used for both decision-making and filtering sub-operations in addition to a multilayer perceptron (MLP) architecture for detecting and removing impulse noise. Another AMF approach was proposed in [40] which judges a noisy pixel in a sub-image based on standard deviation of that sub-image and average value of the filtering window. In [41], a fixed weighted mean filter and an adaptive median filter were designed to remove high-intensity noise from MR images. More realistic random-valued impulse noise was considered in [42] to design a hardware device which emphasized on several different noisy scenarios in corrupted data. In [43], a local similarity with overlapped block partitioning approach was proposed for the detection of noisy pixels and their replacement with appropriate values. In [44], a quaternion median filtering approach was proposed, which selects a noisy pixel based on the sum of pixel differences with other pixels in the filtering window. Recently, authors in [45] developed an adaptive switching modified decision based unsymmetric trimmed median filter (ASMDBUTMF) for noise reduction in grayscale MRI data. However, all of these methods have two major flaws. To begin, most works cannot handle both Gaussian and impulse noise mixtures in MRI data at the same time. Second, in designing denoising techniques, mostly fixed valued impulse noise (FVIN) in the form of salt-and-pepper noise is considered, with little consideration given to more realistic random-valued impulse noise (RVIN) [46, 47].

This work attempts to overcome the limitations in present approaches. The paper’s main contributions are summarized below:

• The Wasserstein generative adversarial network (WGAN) is used for training to reduce the effect of diminishing gradient, which is more conspicuous in adversarial training.

• To achieve visually appealing denoising results, a perceptually motivated loss function is used to guide the training process for data corrupted by mixed Gaussian-impulse noise.

• Batch renormalization facilitates the use of non-iid data for training under variable noise levels.

• 3D depth-wise separable convolution enables computationally intensive training processes to be performed on high-dimensional data.

• A global feature attention module (GFAM) is plugged at the beginning and end of the proposed architecture to capture the long-range dependencies that are oftenly lost in vanilla deep convolutional layers.

The remainder of the paper is organized as follows. We explain the image formation model under the Gaussian-impulse noise assumption in “Image Formation Model and Objective’’. The proposed methodology is explained in the “Adversarial Training’’ to “Training Details’’, while the experimental results and discussions are presented in “Results’’ and “Discussion’’ respectively. Finally, the paper concludes in “Conclusion’’.

Methods

Image Formation Model and Objective

The image formation model for data corrupted by mixed Gaussian-impulse (G-I) noise is given by:

$$\begin{array}{c}\hfill v=u+g+s\end{array}$$ 1

where $u\in R^{w\times h\times c}$ is the latent clean data. w, h and c are the data dimensions along horizontal, vertical and depth axes respectively. Clean data u is corrupted by the Gaussian noise component g and impulse noise component s. g is a random variable following Gaussian distribution with mean ${\mu }_g=0$ and variance ${\sigma }_g^2$; denoted by $g\sim \mathcal{N}({\mu }_g,{\sigma }_g^2)$. Similarly, the random variable s follows Laplace distribution [48, 49] with ${\mu }_s=0$ as the location parameter and ${\sigma }_s$ as the scaling parameter; denoted by $s\sim \mathcal{L}({\mu }_s,{\sigma }_s)$. Under impulse noise, the pixel value $u_i$ is replaced with a uniform random variable $s_i$ in the range $[s_{\text{min}},s_{\text{max}}]$. However, due to pixel value range clipping, the resultant distribution does not remain uniform or multimodal, but follows Laplacian distribution [49]. Hence, the resultant data is corrupted by mixed G-I noise to give the composite noisy signal v. The dimensions of other variables are the same as that of u. The noise level map for Gaussian as well as for impulse component can be spatially variant along the depth axis.

The objective of this work is to obtain clean estimate $\widehat{u}$ from the noisy observation v which is as close as possible to the clean data $(\widehat{u}\sim u)$. This is made possible in an adversarial setting under the generative model [50]. More precisely, using a large pair of noisy-clean image pairs ${\{v_i,u_i\}}_{j=1}^N$, we intend to achieve the following objective:

$$\begin{array}{c}\hfill \underset{G}{\text{min}}\underset{D}{\text{max}}\mathcal{L}(D,G)=E_{u\sim P_r}[logD(u)]+E_{v\sim P_n}[1-logD(G(v))]\end{array}$$ 2

by training the generator G and discriminator D model in succession. Once the model is trained, the discriminator is discarded and the generator is trained to the extent so as to produce perceptually appealing visual results. In Eq. (2), $P_r$ and $P_n$ are the real and noisy data distribution respectively and $\mathcal{L}(D,G)$ is the loss function in the min-max game [50]. However, to generate visually realistic results, Eq. (2) needs to be modified.

Adversarial Training

The vanilla generative adversarial network (GAN) model proposed by Goodfellow et al. [50] has several flaws. To begin, GAN is known to suffer from the vanishing gradient problem, which occurs when a discriminator network becomes overly successful in discarding samples, as observed by [51]. Second, mode collapse prevents the generator network from producing diverse samples, according to [51]. Thirdly, non-convergence is also a prevalent issue [51]. Lastly, according to Eq. (2), the loss function used in GAN training does not correlate to the visual quality of the generated images.

To address the aforementioned limitations, several changes are being made to the vanilla GAN model. Wasserstein GAN (WGAN) is one example of such a modification. Rather than determining whether a sample is real or fake, the discriminator in WGAN (here referred to as the critic network) assigns a score based on the quality of the image generated. This requires a different objective in the minimization of the distance between true and generated data distributions [51] (of Eq. (2)). As a result, for model training, an alternative distance measure known as Earth-mover distance or Wasserstein distance is used [51]. This distance measure is known to be continuous and differentiable and allows smooth back-propagation. Furthermore, unlike discriminators, the critic gradient in WGAN does not saturate and continues to provide useful gradient to the generator network. This avoids the vanishing gradient issue. In the min-max game, the loss function of WGAN is modified as follows:

$$\begin{array}{c}\hfill \underset{G}{\text{min}}\underset{D}{\text{max}}\mathcal{L}(D,G)=E_{u\sim P_r}[D(u)]-E_{v\sim P_n}[D(G(v))]\end{array}$$ 3

It is important to remember that the implementation of the critic network requires the former to be in the space of 1-Lipschitz functions. This is enforced by constraining the weight gradient to be in the range $[-c,c]$ [51]. This, however, leads to suboptimal critic performance because the critic network is unable to capture the complexity of data under the limitations of 1-Lipschitz functions. The problem is mitigated by providing an alternative way of enforcing the Lipschitz constraint. It is known that a function is 1-Lipschitz continuous if it has gradient norm at most 1 everywhere in the manifold [52]. Hence, Eq. (3) can be modified to provide the loss of the discriminator as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill L_{\text{WGAN}}(D)& =\underset{G}{\text{min}}\underset{D}{\text{max}}\mathcal{L}(D,G)=E_{u\sim P_r}[D(u)]\hfill \\ \hfill & -E_{v\sim P_n}[D(G(v))]+\underset{\text{gradient penalty factor}}{\underbrace{\lambda E_{\widehat{v}\sim P_{\widehat{v}}}[{(\Vert {\mathrm{\nabla }}_{\widehat{v}}D(\widehat{v}){\Vert }_2-1)}^2]}}\hfill \end{array}\end{array}$$ 4

Here, the gradient is calculated over the output of the critic network with respect to its input [52]. $\lambda$ denotes the penalty coefficient and $P_{\widehat{v}}$ is a probability distribution that is used to uniformly sample along straight lines between the real data distribution $P_r$ and the generating data distribution $P_g$. Similarly, loss of the generator is given by:

$$\begin{array}{c}\hfill L_{\text{WGAN}}(G)=-E_{v\sim P_n}[D(G(v))]\end{array}$$ 5

We conclude this section with the following exploitation of notations. D and G are the critic and generator models respectively. The aim of the generator is to generate a sample distribution $P_g$ which is as close as possible to the real data distribution $P_r$. During the course of training, the generator excels in generating samples that mimic real (clean) data distribution.

Perceptually Motivated Loss Function

It is observed that generative models are not inherently suited for image restoration tasks. The generated samples lack low-level features (high-frequency components in the form of edges and textures) in the restored data. Therefore, in this section, we intend to augment Eq. (5) with a perceptually motivated content loss $L_{\text{MSSIM}}(u,G(v))$. The final loss of the generator is given as:

$$\begin{array}{c}\hfill \mathrm{L}=\underset{\text{Perceptual}}{\underbrace{\underset{\text{content}}{\underbrace{L_{\text{MSSIM}}(u,G(v))}}\text{loss}+\underset{\text{adversarial}}{\underbrace{L_{\text{WGAN}}(G)}}\text{loss}}}\text{Loss}\end{array}$$ 6

From the next section, $\widehat{u}(=G(v))$ is used to denote the clean estimated data obtained from the generator network. Content loss will be discussed shortly in this section.

In contrast to the work of residual encoder-decoder WGAN (RED-WGAN) [31], which uses a combination of extracted VGG features and mean squared error (MSE) loss to design generator loss, we provide an alternative proposition here. We use a perceptually motivated loss function because MSE does not synchronize with the image visual quality [53] and VGG network is intrinsically designed for 2D data [54, 55]. Structural similarity index (SSIM) [56] has been around for nearly 15 years; used as perceptually motivated image quality assessment metric according to Human Visual System (HVS). This encourages us to use this metric to modify the generator loss in Eq. (5). In a traditional setting, any gradient-based minimization technique requires the loss function to be differentiable in nature; this is also true during back-propagation steps in neural network training. As a result, before it can be used for network training, the derivative of SSIM must be calculated. SSIM is written as:

$$\begin{array}{c}\hfill \text{SSIM}(u,\widehat{u})=l{(u,\widehat{u})}^{\alpha }·c{(u,\widehat{u})}^{\beta }·s{(u,\widehat{u})}^{\gamma }\end{array}$$ 7

The three terms used in the product above have the same usual meaning as defined in [56].

$$\begin{array}{c}\hfill l(u,\widehat{u})={\displaystyle \frac{2{\mu }_u{\mu }_{\widehat{u}}+C_1}{{\mu }_u^2+{\mu }_{\widehat{u}}^2+C_1}};c(u,\widehat{u})={\displaystyle \frac{2{\sigma }_u{\sigma }_{\widehat{u}}+C_2}{{\sigma }_u^2+{\sigma }_{\widehat{u}}^2+C_2}};s(u,\widehat{u})={\displaystyle \frac{2{\sigma }_{u\widehat{u}}{C}_3}{{\sigma }_u{\sigma }_{\widehat{u}}+C_3}}\end{array}$$ 8

Here, l, c and s are the luminosity, contrast and structural information. $C_1$, $C_2$ and $C_3$ are small constants used to avoid numerical instability where $C_3=C_2/2$ and $\alpha$, $\beta$ and $\gamma$ all are set to be 1. ${\mu }_u$, ${\sigma }_u$ and ${\sigma }_{u\widehat{u}}$ are the mean, variance and covariance respectively calculated over entire pixels of the image. ${\mu }_{\widehat{u}}$ and ${\sigma }_{\widehat{u}}$ are calculated similarly for $\widehat{u}$.

However, it is observed that SSIM calculated over entire image yields inaccurate results for l, c and s in Eq. (8) due to spatially variant values of these quantities. In order to circumvent this problem, we use image patch $P_u$ which is centred at pixel p for finding patch-wise SSIM for u(p) and $\widehat{u}(p)$. SSIM for all the patches are averaged together to obtain mean SSIM (MSSIM). We make use of Gaussian filter $G_{\sigma }$ with standard deviation $\sigma$ to obtain patch results such that ${\mu }_u(p)=G_{\sigma }\star P_u$, ${\sigma }_u(p)=G_{\sigma }\star P_u^2-{\mu }_u^2(p)$ and ${\sigma }_{u\widehat{u}}=G_{\sigma }(P_u·P_{\widehat{u}})-{\mu }_u(p)·{\mu }_{\widehat{u}}(p)$. ${\mu }_{\widehat{u}}(p)$ and ${\sigma }_{\widehat{u}}(p)$ are defined analogously. $\star$ and $·$ represent convolution and point-wise multiplication operations respectively. For SSIM to be used as a loss function, we make use of three different properties of it: (i) boundedness $(\le 1)$, (ii) convexity and (iii) symmetry (SSIM $(u,\widehat{u})=$ SSIM $(\widehat{u},u)$) [57]. As a result, MSSIM between u(p) and $\widehat{u}(p)$:

$$\begin{array}{c}\hfill L_{\text{MSSIM}}=1-{\displaystyle \frac{1}{M}}\sum _p(\text{SSIM}(u(p),\widehat{u}(p))\end{array}$$ 9a

where $M=$ total number of image patches. Upon manipulation of Eqs. (7), (8) and (9a), we obtain:

$$\begin{array}{c}\hfill \text{SSIM}(u(p),\widehat{u}(p))=\underset{l(p)}{\underbrace{({\displaystyle \frac{2{\mu }_u{\mu }_{\widehat{u}}+C_1}{{\mu }_u^2+{\mu }_{\widehat{u}}+2+C_1}})}}·\underset{cs(p)}{\underbrace{({\displaystyle \frac{2{\sigma }_{u\widehat{u}}+C_2}{{\sigma }_u^2+{\sigma }_{\widehat{u}}^2+C_2}})}}\end{array}$$ 9b

In order to apply back-propagation during model training, we need to find out the derivative of loss function (here, MSSIM).

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \text{SSIM}(u(p),\widehat{u}(p))}{\partial u(q)}}=cs(p){\displaystyle \frac{\partial }{\partial u(q)}}·l(p)+l(p){\displaystyle \frac{\partial }{\partial u(q)}}·cs(p)\end{array}$$ 10

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial l(p)}{\partial u(q)}}={\displaystyle \frac{\partial l(p}{\partial {\mu }_u(p)}}·{\displaystyle \frac{\partial {\mu }_u(p)}{\partial u(q)}}=2G_{\sigma }(p-q)[{\displaystyle \frac{{\mu }_{\widehat{y}}-{\mu }_{u}l(p)}{{\mu }_u^2+{\mu }_{\widehat{u}}^2+C_1}}]\end{array}$$ 11a

where $\partial {\mu }_{u}p/\partial u(q)=G_{\sigma }(p-q)$. Similarly, $\partial cs(p)/\partial u(q)$ can be evaluated as:

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial cs(p)}{\partial u(q)}}={\displaystyle \frac{\partial cs(p)}{\partial {\mu }_u(p)}}·{\displaystyle \frac{\partial {\mu }_u(p)}{\partial u(q)}}=2G_{\sigma }(p-q)[(\widehat{u}(q)-{\mu }_y)-cs(p)(u(q)-{\mu }_u)]\end{array}$$ 11b

Feeding the expressions of Eqs. (11a) and (11b) in Eq. (10), we obtain the final expression for MSSIM as:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\displaystyle \frac{\partial \text{MSSIM}(u,\widehat{u})}{\partial u(q)}}& ={\displaystyle \frac{2}{M}}\sum _p[cs(p)({\displaystyle \frac{{\mu }_{\widehat{u}}-{\mu }_{u}l(p)}{{\mu }_u^2+{\mu }_{\widehat{u}}^2+C_1}})\hfill \\ \hfill G_{\sigma }(p-q)& +l(p)({\displaystyle \frac{G_{\sigma }(p-q)}{{\sigma }_u^2+{\sigma }_{\widehat{u}}^2+C_2}})(\widehat{u}(q)-{\mu }_{\widehat{u}}-cs(p)(u(q)-{\mu }_u))]\hfill \end{array}\end{array}$$ 12

Batch Renormalization and 3D Depth-wise Separable Convolution

In its pristine form, batch normalization (BN) is used to reduce internal covariate shift in data [58]. However, they are constrained by the assumption of independent and identically distributed (iid) data. In any image restoration problem, model needs to be trained over a wide range of noise levels in data, thus exhibiting non-iid property. Further, the small batch size makes the use of BN suboptimal. This is primarily due to differences in the BN operation during training and inference [59]. Moving average statistics are calculated during model training but only used during inference. By incorporating moving average statistics into the training process, batch renormalization (BRN) helps to mitigate the aforementioned issues [59].

Model training suffers from latency when using 3D data. To address this issue, we employ 3D depth-wise separable convolution [60]. This aids in reducing learnable parameters by a factor of 1/10 without compromising noisy to clean image mapping during inference.

Network Design

The generator model’s architecture is depicted in Fig. 1. Images are permuted between $y\leftrightarrow z$ and $x\leftrightarrow z$ dimensions from their original xyz dimensions to exploit spatial information from different anatomical planes (coronal, sagittal, and transverse). The resultant dimensions are xzy, yxz and zyx. The results of these permutations are sent to global feature attention modules (GFAM, which will be discussed later). The GFAM output is fed into two sets of convolutional blocks. This is made up of two parallel blocks that use 3D normal and dilated convolutions. There are 11 blocks in sequence; each consisting of convolutional layer, BRN layer, and rectified linear units (CBR). The convolutional filter size is fixed at $3\times 3\times 3$, and the number of filters is increased from 4 to 128 from block 1 to 6 and symmetrically decreased to 1 from $7^{\text{th}}$ to ${11}^{\text{th}}$ block. Finally, the permutation layer is used once more to bring back images to the original xyz plane. The outputs are passed through GFAM once more to acquire global contextual features learned in the previous parallel ensemble blocks. The clean estimated image is obtained by subtracting the resultant residual image from the corrupted input data. The discriminator network architecture is the same as used in [54]. BN layers, on the other hand, are removed from the original implementation to avoid correlations between samples in the same batch. This affects the gradient penalty term used in [52]. Furthermore, a linear activation function is used at the end of the critic model rather than a sigmoid function.

Fig. 1.

The architecture of the proposed generator model. The 3D data patches are permuted along the three dimensions to obtain three data patches such that $xzy=\mathrm{permute}(y,z)$, $yxz=\mathrm{permute}(x,y)$ and $zyx=\mathrm{permute}(x,z)$. The global feature attention module (GFAM) blocks can accept arbitrary number of inputs (see Fig. 2). The $\to (green),\to (blue)$ and $\to (brown)$ arrows coming out from the three permuted 3D patches are fed as input to the three GFAM blocks. The top row shows the overall model while the bottom row shows the expanded views of the red and yellow convolutional blocks. The output features obtained from the convolutional blocks are again permuted to their original dimensions (xyz) and fused together into one 3D image using the final GFAM block

Adaptive Global Feature Attention Module

Conventional convolutional filters are limited in three different ways. Firstly, they are unable to map long-range dependencies existing in the 3D data blocks due to small receptive field size [61]. Secondly, all spatial locations in the image are treated similarly. This is more problematic in image restoration problems where low- and high-frequency details need to be properly separated into distinct regions [62]. A typical convolutional layer does not account for these low-level details. Thirdly, most of the traditional supervised learning solutions for image restoration like DnCNN [63] rely on the final feature maps obtained in the last layer. However, features obtained from initial and intermediate layers provide complementary information about the clean estimation. Therefore, we incorporate global feature attention module (GFAM) at the beginning and end of the set of parallel blocks. Since different features in the permuted input data and different feature maps along the parallel paths may provide hidden contextual information, we merge these features under a global setting. As shown in Fig. 2, the feature maps obtained from the previous layers are denoted as ${\mathcal{F}}_1,{\mathcal{F}}_2,\cdots ,{\mathcal{F}}_N$ such that ${\mathcal{F}}_i\in R^{w\times h\times l\times c}$; where $\{w,h,l\}$ are the dimensions of the 3D data and c is the number of channels. Global features from each feature map ${\mathcal{F}}_i$ are obtained by the application of average pooling $P_{\text{avg}}$ followed by the convolutional filter $W_i$ to obtain the features; denoted by $A_i=W_i{P}_{\text{avg}}{F}_i$; $A_i\in R^{1\times 1\times 1\times 1}$. All these features are then concatenated together to obtain $A\in R^{1\times 1\times 1\times N\times 1}$. Softmax function is applied over the concatenated features to obtain $\omega \in R^{1\times 1\times 1\times N\times 1}$:

$$\begin{array}{c}\hfill {\omega }_j={\displaystyle \frac{exp(A_j)}{{\sum }_{i=1}^{N}exp(A_i^h)}}\end{array}$$ 13

Fig. 2.

Adaptive Global Feature Attention Module (GFAM)

Further, the original ${\mathcal{F}}_i$’s are concatenated together to obtain $B\in R^{w\times h\times l\times N\times c}$. Global contextual modelling is performed by the matrix multiplication between $\omega$ and B.

$$\begin{array}{c}\hfill \upsilon =\sum _{i=1}^N{\omega }_i{B}_i\end{array}$$ 14

where $\upsilon \in R^{w\times h\times l\times c}$. Further, an adaptive learnable weight $\beta$ is point-wise multiplied with B. The final output of the GFAM is obtained by element-wise sum of previous output and the last feature input ${\mathcal{F}}_N$.

$$\begin{array}{c}\hfill {\text{GFAM}}_{\mathit{out}}={\mathcal{F}}_N+\beta \sum _{i=1}^N{\omega }_i{B}_i\end{array}$$ 15

Training Details

We have trained the model for MR denoising using simulated MR images obtained from Brainweb [64] database.¹It consists of normal and multiple sclerosis (MS) datasets containing T1, T2 and PD-weighted images in five different slice thicknesses: 1 mm, 3 mm, 5 mm, 7mm and 9 mm. This gives rise to 181, 60, 36, 26 and 20 slices respectively. The spatial dimension of each image is $181\times 217$. Images of size $181\times 217\times 181$ from each modality are used for testing while the rest are used for model training. Image non-uniformity is set to $0\%$ and initial noise levels are set to zero. The levels of noise used for training are: Gaussian noise in the signal to noise (SNR) range [2,  20] dB and impulse noise in the range $[1\%,30\%]$. Ten noise levels are uniformly and randomly sampled from the range of each noise type. This gives a combination of 100 noise levels $=(10\text{Gaussian noise levels}\times 10\text{impulse noise levels})$ that can be added to the training data. We have used patch-based training in our model [65]. Non-overlapping patches of size $40\times 40\times 6$ are extracted from each image. Total number of patches generated are 108, 000.

Batch size is selected to 16. Adaptive moment estimation (ADAM) is used as the optimization technique as the de facto standard due to its superiority over other stochastic gradient descent methods. Its parameters are set as $\alpha =0.01$, ${\beta }_1=0.9$ ${\beta }_2=0.99$ and $ϵ=1e-6$. We could find that 200 epochs were sufficient for the convergence of the model.

Results

In this section, we conduct experiments on synthetically corrupted and real MR data obtained from clinical level MRI scanners.

Experimental Setup

Image of size $181\times 217\times 181$ from each imaging modality (T1- and PD-weighted) was separated from the Brainweb [64] database during training time for evaluation of the trained model during test time. The selected 3D images correspond to the 1 mm slicing of simulated data. For comparison, seven different techniques are used. Four of them are filtering-based approaches² (optimized non-local means (ONLM) [8], adaptive optimized NLM (AONLM) [9], prefiltered rotationally invariant NLM (PRINLM) [12] and multi-resolution NLM (MRNLM) [11]) while the other three competing methods are learning-based (DnCNN) [63], Deep Image Prior (DIP) [66] and residual encoder-decoder-Wasserstein generative adversarial network (RED-WGAN) [31]). Under ONLM, a computationally efficient and parallelized version of NLM is proposed with features of automatic parameter tuning and relevant voxels selection. An adaptive version of NLM is used in AONLM where the strength of filter is decided based on the local noise level in the image. Authors in PRINLM exploit the sparsity and self-similarity properties by using moving window discrete cosine transform. Based on the spatial and frequency information in noisy data, a multi-resolution analysis is presented in MRNLM. For the NLM-based methods, optimal filtering parameters are chosen so as to generate best visual and metrics results [7, 67]; tabulated in Table 1. DnCNN is an implementation of neural network based denoiser where the network predicts the residual data rather than the clean estimation. Since DnCNN is inherently designed for grayscale/RGB data, we retrained the model using our own 3D MRI training data. DIP uses a novel proposition that assumes that an untrained or randomly trained neural network acts as a natural prior over the image to be estimated. RED-WGAN uses a generative model based on residual encoder-decoder architecture.

Table 1.

Optimal filtering parameters for comparing techniques. S is the radius of search window and N is the radius of the patch, h is the smoothing parameter and $\sigma$ is the standard deviation of Gaussian distribution

Comparing Techniques	Filtering Parameters
ONLM	$S=5$, $h=\sigma$, $N=2$
AONLM	$S=3$, $N1=2$, $N2=2$, $h=\sqrt{2\beta {\sigma }^2|V|}$ where $\beta$ is a constant and |V| is the number of voxels
PRINLM	$S=5$, $N=2$, h = 0.4$\sigma$
MRNLM	$S=11$, h = $\sigma$, $N=2$
UKR	Radius of the second order regression = 4

Two full-reference image quality metrics are used for the quantitative evaluation of restoration results: peak signal to noise ratio (PSNR) and structural similarity index (SSIM) [56], which are defined as follows:

$$\begin{array}{c}\hfill \text{PSNR}=10·{log}_{10}({\displaystyle \frac{{\text{MAX}}_u}{{\text{MSE}}^2}})\end{array}$$ 16a

$$\begin{array}{c}\hfill \text{SSIM}(u,\widehat{u})={\displaystyle \frac{(2{\mu }_u{\mu }_{\widehat{u}}+C_1)(2{\sigma }_{u\widehat{u}}+C_2)}{({\mu }_u^2+{\mu }_{\widehat{u}}^2+C_1)({\sigma }_u^2+{\sigma }_{\widehat{u}}^2+C_2)}}\end{array}$$ 16b

Here, u and $\widehat{u}$ are the patches of clean and recovered images from the local window respectively, ${\mu }_u$, ${\sigma }_u^2$ and ${\sigma }_{u\widehat{u}}$ are the mean, variance and covariance matrix, respectively.

Experiments on real data are conducted on IXI datasets³ obtained from Institute of Psychiatry (IoP). The full dataset consists of brain MRI obtained from five different imaging formats: T1, T2, PD-weighted, MRA and diffusion-weighted images from 600 healthy subjects. There are 74 images each from T1, T2 and PD modalities. Dimension of each image is $256\times 256\times 136$. A PD-weighted image was used for evaluation purpose.

Discussion

Simulated MR Data

Each simulated MR image (corresponding to T1 and PD-weighted images) is corrupted with six different levels of Gaussian-impulse noise: $(15\text{dB,}3\%)$, $(12\text{dB,}6\%)$, $(10\text{dB,}10\%)$, $(6\text{dB,}15\%)$, $(3\text{dB,}20\%)$ and $(1\text{dB,}25\%)$. PSNR and SSIM corresponding to these noise levels are shown in Tables 2 and 3 respectively. It can be observed that our proposed technique can maintain a considerable gap with comparing techniques. Visual inspection is necessary to confirm the validity of the metric results. For this, we have presented the visual results for T1 and PD-weighted images in Figs. 3 and 4 respectively. To present a fair comparison on 3D reconstruction ability of different methods, we have presented the results on all three anatomical planes (the first image in each tabular block represents the transverse/axial section which when divided into cross-sectional planes gives coronal (shown by the red line) and sagittal (shown by the yellow line) plane results). For T1-weighted images, results are presented for the noise level $(10\text{dB,}10\%)$ and for PD-weighted images, results are displayed for the noise level $(6\text{dB,}15\%)$. It can be observed that results produced by DnCNN are not able to reduce noise considerably. Also, visuals produced in sagittal and coronal planes in DIP result in post-processing artefacts. However, images generated by the proposed neural network model can maintain fine details in recovered images. This is more prominent for PD-weighted images.

Table 2.

Peak Signal to Noise Ratio (PSNR) for three different imaging modes (T1 and PD) at different levels of noise for different methods

Methods/Noise (dB,%)	(15, 3)	(12, 6)	(10, 10)	(6, 15)	(3, 20)	(1 ,25)	(15, 3)	(12, 6)	(10, 10)	(6, 15)	(3, 20)	(1 ,25)
	T1-Weighted						PD-Weighted
Noisy	38.47	28.22	22.24	18.90	17.25	16.53	38.21	28.74	22.60	20.27	18.22	16.92
ONLM	44.98	37.17	35.53	33.31	30.16	24.99	42.70	37.16	33.16	34.10	30.55	28.26
AONLM	43.23	35.54	30.13	25.85	23.12	20.06	41.79	35.94	30.10	27.64	23.09	20.37
PRINLM	45.88	39.86	35.99	33.35	29.60	23.88	45.83	38.86	35.86	30.49	31.66	31.08
MRNLM	44.85	37.32	34.95	33.11	30.36	25.04	42.10	36.69	32.62	34.00	29.46	26.84
DnCNN	41.02	33.38	29.90	27.19	25.00	22.66	35.70	33.27	28.40	27.33	25.25	23.71
DIP	45.24	37.59	32.23	28.45	25.98	23.55	45.19	37.77	30.35	29.14	26.52	24.55
RED-WGAN	44.60	38.70	33.68	29.19	25.48	23.56	38.92	37.74	34.66	30.07	26.97	23.87
GenMRIDenoiser	47.09	41.21	37.12	34.09	32.28	30.03	46.61	39.50	35.72	33.66	31.13	28.80

Table 3.

Structural Similarity (SSIM) for three different imaging modes (T1 and PD) at different levels of noise for different methods

Methods/Noise (dB,%)	(15, 3)	(12, 6)	(10, 10)	(6, 15)	(3, 20)	(1 ,25)	(15, 3)	(12, 6)	(10, 10)	(6, 15)	(3, 20)	(1 ,25)
	T1-Weighted						PD-Weighted
Noisy	0.704	0.361	0.216	0.144	0.106	0.084	0.671	0.349	0.208	0.155	0.115	0.090
ONLM	0.966	0.941	0.899	0.847	0.796	0.704	0.958	0.921	0.890	0.841	0.773	0.681
AONLM	0.921	0.657	0.449	0.336	0.268	0.201	0.904	0.638	0.622	0.537	0.457	0.408
PRINLM	0.963	0.936	0.801	0.863	0.803	0.670	0.953	0.918	0.891	0.854	0.846	0.829
MRNLM	0.966	0.942	0.889	0.866	0.818	0.715	0.959	0.921	0.896	0.857	0.800	0.713
DnCNN	0.806	0.746	0.673	0.614	0.564	0.516	0.770	0.611	0.528	0.493	0.451	0.417
DIP	0.965	0.777	0.526	0.456	0.356	0.280	0.955	0.762	0.573	0.564	0.468	0.418
RED-WGAN	0.958	0.927	0.873	0.805	0.721	0.665	0.945	0.916	0.890	0.851	0.810	0.742
GenMRIDenoiser	0.986	0.946	0.897	0.878	0.832	0.742	0.988	0.935	0.901	0.875	0.860	0.831

Fig. 3.

Results on simulated Brainweb-T1 dataset for slice thickness 1mm. The results of the comparing methods are shown for the noise level $(G,I)=(10\text{dB,}10\%)$. Each figure in a group shows the axial (top left), sagittal (right) and coronal (down) planes. Upon careful visualization, it can be observed that the proposed methodology removes noise along with the preservation of details. This is more evident in the axial and coronal sections

Fig. 4.

Results on simulated Brainweb-PD dataset for slice thickness 1mm. The results of the comparing methods are shown for the noise level $(G,I)=(6\text{dB,}15\%)$. Each figure in a group shows the axial (top left), sagittal (right) and coronal (down) planes. Upon careful visualization, it can be observed that the proposed methodology removes noise along with the preservation of details. This is more evident in the axial and coronal sections

Results on running time are mandatory for testing the efficiency of different methods under the same hardware conditions. All experiments on conventional methods are conducted on MATLAB-2019 using 16GB RAM and an i7 processor. All deep learning models are trained and evaluated using Google Colab under Nvidia P100-PCIE-16GB GPU. Results on running time are tabulated in Table 4.

Table 4.

Results on running time (in seconds) using different methods

Methods	ONLM	AONLM	MRNLM	PRINLM	DnCNN	DIP	RED-WGAN	GenMRIDenoiser
Running Time (in seconds)	73.21	513.49	315.03	130.59	18.82	990.75	34.66	28.14

Real MR Data

Experiments on real data are performed on the IXI datasets. This database consists of real MR images from three different medical centres: Guys hospital (GH), Hammersmith hospital (HH) and Institute of Psychiatry (IoP). The results presented in Fig. 5 are obtained from IoP. Size of the image is $256\times 256\times 136$. Just like the synthetic data, we have visualized results on all three anatomical planes. The results produced by the proposed technique provide considerably better visuals with detail preserving capability. This is attributed to the perceptually motivated loss being used for model training. Results on running time (in seconds) are specified in the caption of the figure itself.

Fig. 5.

Real PD-weighted data obtained from IXI database Institute of Psychiatry. Each figure in a group shows the axial (top left), coronal (right) and sagittal (right) planes

Ablation Study

As discussed in the proposed section, Batch renormalization (BRN) [59] is explored in the network training as it is a feasible alternative when data is non-iid and batch size is small. In Fig. 6 (left), we have validated this proposition by plotting validation loss for two different batch sizes: 40 and 200. We can observe that models trained using both BN [58] and BRN for batch size 200 yields good results. However, we get more benefits over validation loss when batch size is chosen to be 40 and BRN is used as the normalization strategy. This is not true for model trained using BN. However, training is more stable when large batch size is used or when BN is used.

Fig. 6.

Left: Validation loss for large (200) and small (40) batch size using BN and BRN. Right: Validation loss using normal and 3D depth-wise separable convolution

Further, since training over 3D data is computationally costly; we used 3D depth-wise separable convolution to reduce the number of model parameters without significant drop in model performance. To validate this result, we plotted validation loss for over 200 epochs for models trained using normal and 3D depth-wise separable convolution. We can observe negligible difference in the two training paradigms. The plot is shown in Fig. 6 (right).

Conclusion

In this paper, we discussed the image denoising process for MR data under the assumption of Gaussian-impulse noise. A deep generative model was proposed with emphasis on perceptually motivated loss to guide the training process suited for low-level computer vision tasks. The problem of non-iid data during training process was handled using batch renormalization while the problem of computational burden during training was mitigated using 3d depth-wise separable convolution. The augmentation of global feature attention block further boosted the performance of the network by separately paying attention to the low- and high-frequency details in the image during training and inference. As an extension to this work, we intend to explore the effect of attention blocks in further improving the denoising accuracy of the network in the same adversarial setting. Furthermore, we intend to explore noise removal in the complex MR data and other imaging modalities like diffusion-weighted MRI (DW-MRI). Noise modelling in multi-coil acquisition systems and deep restoration frameworks under such settings can also be explored in subsequent works.

Supplementary Information

Below is the link to the electronic supplementary material.

Declarations

Ethics Approval

This is a numerical simulation study for which no ethical approval was required. The research did not involve any human/non-human subjects.

Conflict of Interest

The authors declare no competing interests.