Fidelity Imposed Network Edit (FINE) for Solving Ill-Posed Image Reconstruction

Abstract

Deep learning (DL) is increasingly used to solve ill-posed inverse problems in medical imaging, such as reconstruction from noisy and/or incomplete data, as DL offers advantages over conventional methods that rely on explicit image features and hand engineered priors. However, supervised DL-based methods may achieve poor performance when the test data deviates from the training data, for example, when it has pathologies not encountered in the training data. Furthermore, DL-based image reconstructions do not always incorporate the underlying forward physical model, which may improve performance. Therefore, in this work we introduce a novel approach, called fidelity imposed network edit (FINE), which modifies the weights of a pre-trained reconstruction network for each case in the testing dataset. This is achieved by minimizing an unsupervised fidelity loss function that is based on the forward physical model. FINE is applied to two important inverse problems in neuroimaging: quantitative susceptibility mapping (QSM) and under-sampled image reconstruction in MRI. Our experiments demonstrate that FINE can improve reconstruction accuracy.

1. INTRODUCTION

Image reconstruction from noisy and/or incomplete data is often solved with regularization of various forms, which can be formulated as maximum a posteriori (MAP) probability estimation (Gindi et al., 1993). Traditionally, these regularizations promote desired properties with explicitly extracted image features, such as image gradients or wavelet coefficients (Block et al., 2007; Fessler, 2010; Kressler et al., 2010; Lustig et al., 2007). Deep learning (DL) using a convolutional neural network (CNN) of many layers has demonstrated superior capability in capturing image features compared to explicit feature extraction and achieved tremendous success in a wide range of computer vision applications (Goodfellow et al., 2016; LeCun et al., 2015). Accordingly, CNNs have recently been used in image reconstruction (Bahadir et al., 2019; Hyun et al., 2018; Jin et al., 2017).

A fundamental concern on the performance of a trained DL model in predicting outcome for previously unseen test data is the generalization error (Shalev-Shwartz and Ben-David, 2014). While CNNs can perform well with low generalization errors, their robustness remain concerning, with open questions (Jakubovitz et al., 2018), including their susceptibilities to adversarial attacks (Szegedy et al., 2014). Considering image reconstruction as a very high dimensional regression task, generalization errors may inevitably occur at some voxels in CNN based image reconstruction. It is therefore important to investigate the performance of CNN-based image reconstructions when test data deviate from training dataset, due to for example abnormal morphology or the presence of pathology, or when test data contains adversarial noise.

To improve the robustness of network-based image reconstructions, the use of the underlying physical model of the forward problem has been explored. One approach is to use the network output as a regularization in the standard MAP optimization reconstruction of the test data (Wang et al., 2016), which maintains data fidelity of the test data but may suffer when the network output is not sufficiently close to the true solution. Another approach is to incorporate the physical model into the network architecture (Aggarwal et al., 2019; Schlemper et al., 2018), which can improve reconstruction accuracy and reduce generalization error.

In this paper, building on the U-Net architecture (Ronneberger et al., 2015) as the reconstruction network, we propose to update the weights of a pre-trained U-Net for each case in the test dataset by minimizing an unsupervised fidelity loss function. The fidelity loss is defined according to the forward physical model and data noise property. We refer to this method as fidelity imposed network edit (FINE). We report FINE results on two neuroimaging reconstruction problems using U-Net, quantitative susceptibility mapping (QSM) (de Rochefort et al., 2010) and MRI reconstruction from under-sampled k-space data.

2. THEORY

A major challenge in medical image reconstruction is to invert an ill-posed system matrix A of a known physical process in the presence of data noise n:

$$y=Ax+n,$$ (1)

where x is the desired image and y the measured data. For example, in QSM, the dipole kernel is zero on the cone surface defined by the magic angle, making the inversion from measured magnetic field to susceptibility source ill-posed (de Rochefort et al., 2010; Kee et al., 2017; Liu et al., 2012). In under-sampled image reconstruction, the sampling mask contains many zeroes, making reconstruction of under-sampled data ill-posed. Additional prior knowledge is required to obtain a solution. The MAP inference approach provides an optimal estimation according to the measured data noise property and prior knowledge. Gaussian noise is observed in MRI complex data and is a widely-used approximate model for various other data. This leads to the common Bayesian reconstruction under Gaussian noise:

$$\widehat{x}={\text{argmin}}_x\frac{1}{2}\Vert W(Ax-y){\Vert }_2^2+R(x),$$ (2)

where W is the square root of the inverse of the noise covariance matrix, R(x) is a regularization term that characterizes prior knowledge. The first term in Eq.2 is referred to as data fidelity. Eq.2 can be solved using numerical optimization procedures, such as the quasi-Newton method that iteratively linearizes the problem with each linear problem solved by the conjugate gradient method. Common choices for R(x) include sparsity enforcement, such as Total Variation (TV) (Osher et al., 2005) or the L1 norm in an appropriate wavelet domain (Donoho, 1995). These types of priors are critical for solving the ill-posed inverse problem. However, they can also limit the quality of the reconstruction, such as introducing artificial blockiness.

Fundamentally, regularization promotes desired image features, for which DL may be better suited than conventional explicit feature extraction. A data-to-image neural network model ϕ(·; Θ₀) with Θ₀ the network weights can be trained in a supervised fashion based on training data consisting of pairs {α_i, β_i}, with α_i the ground-truth image and β_i the input data. The weights at each convolutional layer, along with non-linear activation functions, may be regarded as a collection of feature extractors for the desired image reconstruction (Krizhevsky et al., 2012; LeCun et al., 2015). The large number of weights in DL may explain its advantage over explicit feature extraction that uses a single or few weights (Lee et al., 2009; Zeiler and Fergus, 2014). Given a case in the test dataset or a test data point y, one can use this model to compute a reconstruction by evaluating the model on the observed measurement vector:

$$\widehat{x}=\varphi (y;{\Theta }_0)$$ (3)

The supervised learning strategy described above may perform poorly if there is a structural change in the test data, such as a certain pathology that is not present in the training dataset. To improve the robustness of DL-based reconstruction, it has been proposed to treat the network output in Eq. 3 as a regularization in Eq. 2 penalizing the L2 difference between the network output and the final optimized solution (Wang et al., 2016):

$$\widehat{x}={\text{argmin}}_x\frac{1}{2}\Vert W(Ax-y){\Vert }_2^2+\lambda \Vert x-\varphi (y;{\Theta }_0){\Vert }_2^2.$$ (4)

We refer to this reconstruction as DL with L2 regularization (DLL2). In another recently proposed approach, the physical model is incorporated into the network architecture solving a quasi-newton optimization scheme (Aggarwal et al., 2019; Schlemper et al., 2018).

In this paper, we use a U-Net model as the reconstruction network and propose to exploit the data fidelity term directly in adapting the model through backpropagation without any explicit supervision. Specifically, the desired image x is reconstructed by editing the weights of a pre-trained U-Net under the guidance of data fidelity for a given test data point y. The network weights Θ are initialized with Θ₀ (obtained with supervised training) and are updated using the physical model of the imaging system as follows:

$$\widehat{\Theta }=\underset{\Theta }{\text{arg}}\text{min}L(y;\Theta )=\Vert W(A\varphi (y;\Theta )-y){\Vert }_2^2.$$ (5)

Then the output of the updated network is the reconstruction of x with both data fidelity and deep learning regularization:

$$\widehat{x}=\varphi (y;\widehat{\Theta })$$ (6)

We refer to this approach as “fidelity imposed network edit (FINE)” for solving an ill-posed inverse problem using deep learning and imaging physics.

3. METHOD

In this paper, we applied the proposed FINE to two inverse problems in MRI: QSM and under-sampled image reconstruction. Data were acquired following an IRB approved protocol. All images used in this work were de-identified to protect privacy of human participants. Data and code are available to interested researchers upon request.

3.1. QSM

First, we applied FINE to QSM (de Rochefort et al., 2010), which is ill-posed because of zeroes at the magic angle in the dipole kernel. Consequently, streaking artifacts appear in the image domain after un-regularized dipole inversion (Kee et al., 2017). The Bayesian approach has been widely used to address this issue. One example is the Morphology Enabled Dipole Inversion (MEDI) method (Liu et al., 2012), which employs the following cost function:

$$\widehat{x}={\text{argmin}}_{\chi }\frac{1}{2}\Vert W(d\ast \chi -f){\Vert }_2^2+\lambda \Vert M_G\nabla \chi {\Vert }_1$$ (7)

with χ the susceptibility distribution to solve, f the field measurement, d the dipole kernel. The regularization is a weighted total variation, with ∇ the gradient operator, M_G a binary edge mask determined from the magnitude image (Liu et al., 2012) which enforces morphological consistency between magnitude and susceptibility.

3.1.1. Data acquisition and pre-processing

MRI was performed on 6 healthy subjects using a 3T system (GE, Waukesha, WI) with a multi-echo 3D gradient echo (GRE) sequence. Detailed imaging parameters included FA = 15°, FOV = 25.6 cm, TE1 = 5.0 ms, TR = 39 ms, #TE = 6, ΔTE = 4.6 ms, acquisition matrix = 256×256×48, voxel size = 1×1×3 mm³, BW = ±62.5 kHz. The local tissue field was estimated using non-linear fitting across multi-echo phase data (Kressler et al., 2010) followed by graph-cut based phase unwrapping (Dong et al., 2015) and background field removal (Liu et al., 2011). GRE imaging was repeated at 5 different orientations per subject for COSMOS reconstruction (Liu et al., 2009), which was used as the gold standard for brain QSM.

A second data set was obtained by performing GRE MRI on 8 patients with intracerebral hemorrhage (ICH) and 8 patients with multiple sclerosis (MS) at the standard supine orientation. ICH patient data was acquired using the same scanner and imaging parameters as above. MS patient data was acquired using a 3T system (Magnetom Skyra, Siemens Healthcare, Erlangen, Germany) and imaging parameters included FA = 15°, FOV = 24.0 cm, TE1 = 6.69 ms, TR = 49 ms, #TE = 10, ΔTE = 4.06 ms, acquisition matrix = 256×256×48, voxel size = 1×1×3 mm³, BW = 260 Hz/Px.

A third data set was obtained by resampling the 2016 QSM challenge data (Langkammer et al., 2018) to 1×1×3 mm³ voxel size to be consistent with our experimental setting.

A final data set was obtaining by simulating 6 ICH brains. This was done by segmenting 6 hemorrhagic lesions from the patients of the second data set above (reconstructed using MEDI) and combining it with the 6 susceptibility maps from the multi-orientation data set described above (reconstructed using COSMOS). The corresponding local fields were generated using the dipole convolution forward model, followed by adding Gaussian noise.

3.1.2. Dipole inversion network

We implemented a 3D U-Net (Çiçek et al., 2016; Yoon et al., 2018), a fully convolutional network architecture, for mapping from the local tissue field f to susceptibility distribution. The convolutional kernel size was 3 × 3 × 3. 5 of the 6 healthy subjects with COSMOS data were used for training, with augmentation by in-plane rotation of ±15°. Each 3D volume data was divided into patches of size 64 × 64 × 32, giving a total number of 12025 patches for training. 20% of these patches were randomly chosen as a validation set during training. We employed the same combination of loss function as in (Yoon et al., 2018) in training the network with Adam optimizer (Kingma and Ba, 2014) (initial learning rate: 10⁻³, epochs: 40), resulting in a 3D U-Net ϕ(; Θ₀).

3.1.3. Fidelity Imposed Network Edit (FINE) for QSM

After pre-training the network using 3D patches described above, for a given test data, a whole local field volume f was fed into the network, and the network weights Θ₀ from pre-training were used to initialize the following minimization:

$$\widehat{\Theta }={\text{argmin}}_{\Theta }\Vert W(d\ast \varphi (f;\Theta )-f){\Vert }_2^2$$ (8)

This minimization essentially fine-tuned the pre-trained dipole inversion network ϕ(f; Θ) to produce an output adapted for a given test field data f that is consistent with the forward dipole model. Eq. 8 was minimized using Adam (Kingma and Ba, 2014) with initial learning rate 10⁻⁴. FINE was stopped when the relative change of the unsupervised fidelity loss between two consecutive iterations fell below 5 × 10⁻³. The final reconstruction of the fine-tuned network was $\widehat{\chi }=\varphi (f;\widehat{\Theta })$.

FINE was compared with MEDI (Eq. 7) with λ = 0.001 (Liu et al., 2012) and with DLL2 (Eq. 4):

$$\widehat{\chi }={\text{argmin}}_{\chi }\frac{1}{2}\Vert W(d\ast \chi -f){\Vert }_2^2+{\lambda }_2\Vert \chi -\varphi (f;{\Theta }_0){\Vert }_2^2$$ (9)

with λ₂ = 0.01.

3.1.4. Data analysis

Root mean square error (RMSE), which measures general reconstruction error, high-frequency error norm (HFEN), which measures the similarity at high spatial frequencies (Ravishankar and Bresler, 2010), and structural similarity index (SSIM), which quantifies image intensity, structural and contrast similarity between pairs of image patches (Wang et al., 2004), are used to quantify reconstruction accuracy on data with COSMOS or synthetic ground truth. For the healthy subjects, cross-validation was performed in which each COSMOS brain was selected as test data, leaving the other 5 brains as training dataset, resulting in 6 pre-trained U-Nets. U-Net trained by the first 5 healthy subjects was applied to the remaining experiments. For the MS patients, lesions were manually segmented by an experienced neuroradiologist (S.Z.) based on the corresponding T2FLAIR maps which were spatially registered to the magnitude of the GRE data. Deming regression (Martin, 2000) of all lesion mean values across patients were employed between MEDI and the other three methods to get each pair’s linear relationship. For ICH patients, hemorrhagic lesions were segmented manually and their mean susceptibilities for each reconstruction method were calculated and compared. A reference-free metric to measure the blurring in images (Crete et al., 2007) was used to quantify tissue susceptibility reconstruction quality surrounding each hemorrhage (scores between 0 and 1, the less the better in terms of blur perception). For simulated ICH brains, the reconstructed QSM was compared with synthetic ground truth in terms of RMSE, HFEN and SSIM.

3.2. Under-sampled reconstruction

Second, we applied FINE to MRI reconstruction with under-sampled data. T2 weighted (T2w) images were retrospectively under-sampled. A compressive sensing reconstruction using Total Variation (TV) regularization was used to reconstruct images from the under-sampled k-space data:

$$\widehat{u}=\underset{u}{\text{argmin}}\Vert UFu-b{\Vert }_2^2+\lambda \Vert \nabla u{\Vert }_1$$ (10)

where U is the binary k-space under-sampling pattern, F the Fourier Transform operator, b the measured under-sampled k-space data, ∇ the 2D gradient operator, λ a regularization parameter and u the image to be solved.

3.2.1. Data acquisition and pre-processing

We obtained real-valued T2w axial images of 237 MS patients and 5 glioma patients, with 256 × 184 matrix size and 1mm² resolution. For each MS patient, we extracted 50 axial 2D images from each volume, giving a total number of 11850 images. For 5 glioma patients, we extracted 44 images with glioma tumors. The intensity range of each image was normalized to range [0, 1]. We obtained complex-valued T2w sagittal images of fully-sampled subjects used in MoDL (Aggarwal et al., 2019) as another dataset for experiments, with 256 × 232 matrix size and 1mm² resolution. A natural image dataset (Oliva and Torralba, 2001) resized to 256 × 184, which contained coast, forest highway, city center, mountain, open country, street and tall building images, was also used in our experiments as another training data set.

3.2.2. Under-sampled reconstruction network

2D U-Net (Ronneberger et al., 2015) was used as the network architecture for mapping from A^Hb (where A = UF is the system matrix) to a fully sampled T2w image, where U was chosen as a variable-density Cartesian random sampling pattern (Uecker et al., 2015). Two 2D U-Nets were employed, one for real-valued image reconstruction and the other for complex-valued image reconstruction, with complex-valued images represented by two separate real and imaginary channels, similar to (Bahadir et al., 2019). The network was trained using a 3 × 3 convolutional kernel. We used the L₁ difference between the network output and target image as the loss function and the Adam optimizer (Kingma and Ba, 2014) (initial learning rate: 10⁻³, epochs: 100). For real-valued image reconstruction, 8800 images from 176 MS patients were used for training and 2200 images from 44 MS patients were used for validation. 850 images from the remaining 17 MS patients and 44 images with tumor from glioma patients formed two test datasets. A variable-density sampling pattern in real-valued dataset was generated with acceleration factor 3.24. For complex-valued image reconstruction, we used the same training/test dataset as in MoDL (Aggarwal et al., 2019), yielding training data dimensions in rows × columns × slices × coils as 256 × 232 × 360 × 12 and test data dimensions as 256 × 232 × 160 × 12 . The same variable-density sampling patterns used in MoDL (Aggarwal et al., 2019) were applied here, resulting in an acceleration factor of 6. We used the same symbol ϕ(·; Θ₀) to represent both trained 2D U-Nets for conciseness.

3.2.3. Fidelity Imposed Network Edit (FINE) for under-sampled reconstruction

Test data b for a test subject was obtained by under-sampling an axial T2w image of the subject using the same sampling pattern as in the pre-training step. Similar to Eq. 8, we initialized the network weights Θ using Θ₀ and updated them using the following minimization:

$$\widehat{\Theta }={\text{argmin}}_{\Theta }\Vert UF\varphi (A^{H}b;\Theta )-b{\Vert }_2^2,$$ (11)

which was solved using Adam with initial learning rate 10⁻⁴. FINE was stopped when the relative change of the data fidelity between two consecutive iterations fell below 10⁻⁴. The final FINE reconstruction for the T2w image was $\widehat{u}=\varphi (A^{H}b;\widehat{\Theta })$.

FINE reconstruction was compared with TV (Eq. 10) with λ = 0.001, U-Net, and DLL2 (Eq. 4):

$$\widehat{u}={\text{argmin}}_u\frac{1}{2}\Vert UFu-b{\Vert }_2^2+{\lambda }_2\Vert u-\varphi (A^{H}b;{\Theta }_0){\Vert }_2^2,$$ (12)

with λ₂ = 0.01, and MoDL with 10 repetitions of sub-blocks (Aggarwal et al., 2019).

3.2.4. Data analysis

Peak signal-to-noise ratio (PSNR) and SSIM are used to quantify reconstruction accuracy on all data. To test the stability of FINE with respect to the choice of network structure and optimizer details, we repeated FINE reconstructions with different initial learning rates (2 × 10⁻⁴ and 5 × 10⁻⁵), a second solver, RMSprop (Tieleman and Hinton, 2012), and a second network structure, consisting of 5 convolutional layers with 32 channels in layers 2-4 and ReLU activations. To test the dependency of FINE’s performance on the initial training dataset, we pre-trained multiple networks on either natural or MR images with a range of training sizes. To test the generalization ability of FINE compared to MoDL with respect to various test data noise levels, we applied FINE and MoDL to the MS test dataset with simulated Gaussian noise described above. To test the stability of FINE against adversarial attacks, we constructed various levels of adversarial noise following (Antun et al., 2019) and compared TV regularization reconstruction, U-Net, and FINE - the proposed method. Different levels of adversarial noise were selected from the intermediate solutions during the adversarial noise construction procedure as proposed in (Antun et al., 2019).

4. Results

4.1. QSM

4.2.1. Healthy subjects

QSMs of 2016 QSM challenge data reconstructed by MEDI, U-Net, DLL2 and FINE are displayed in Figure 1a, with COSMOS as ground truth. The RMSE was 69.08, 74.19, 53.34 and 53.44, for MEDI, U-Net, DLL2 and FINE respectively. The corresponding SSIM was 0.9060, 0.9321, 0.9503 and 0.9483, respectively. The corresponding HFEN was 70.25, 65.47, 50.21 and 51.00, respectively. Figure 2 shows the median relative change of the weights per layer in U-Net structure between Θ of FINE and Θ₀ of U-Net reconstruction in Figure 1a. In this case, FINE changed predominantly the weights in high-level layers of U-Net (layers 1 through 5 and layers 17 through 21). Quantitative metrics averaged among cross-validation experiments on COSMOS dataset are shown in Table 1, with FINE demonstrating consistently good performance.

Figure 1.

a) Comparison of QSMs (first row) and the corresponding difference maps (second row) of 2016 QSM challenge data reconstructed by MEDI, U-Net, DLL2 and FINE with COSMOS as ground truth. All methods showed similar performance. b) Comparison of QSMs (first row) and the corresponding difference maps (second row) of one simulated brain with ICH. Hemorrhage was underestimated in U-Net, but were recovered progressively from DLL2 to FINE. MEDI and FINE had minimal reconstruction errors among all four methods. All images were displayed using a [−0.3, 0.3] ppm window except in insets in the third row, which used [−0.6, 1.5] ppm.

Figure 2.

Median relative change of the weights per layer in U-Net structure between weights of U-Net reconstruction (initialization) and FINE reconstruction in Figure 1b. FINE changed predominantly the weights in high-level layers of U-Net (layers 1 through 5 and layers 17 through 21).

Table 1.

RMSE, SSIM and HFEN for various QSM reconstructions averaged among cross-validation experiments in healthy subjects, with COSMOS as the ground truth reference

	RMSE	SSIM	HFEN
MEDI	43.19 ± 2.59*	0.9645 ± 0.0036*	34.29 ± 3.29*
U-Net	41.32 ± 1.81*	0.9737 ± 0.0015*	40.90 ± 1.87*
DLL2	30.70 ± 2.34	0.9853 ± 0.0020	30.07 ± 2.51
FINE	31.36 ± 1.81	0.9861 ± 0.0017	29.38 ± 1.90

^*denotes statistical significance for the comparison between MEDI/U-Net/DLL2 and FINE; p < 0.05.

4.2.2. Simulated ICH brains

QSMs of one simulated brain with ICH reconstructed by MEDI, U-Net, DLL2 and FINE are displayed in Figure 1b, with the simulated brain as ground truth. Hemorrhage was underestimated in U-Net, but were recovered progressively from DLL2 to FINE. MEDI and FINE had minimal reconstruction errors among all four methods. Quantitative metrics of all 6 simulated brains are shown in Table 2, with FINE demonstrating one of the best performances. Mean susceptibility values (ppm) of hemorrhage lesions on 6 simulated ICH brains from each reconstruction method and ground truth were calculated, giving mean susceptibility values ± standard deviations: 0.57 ± 0.10, 0.27 ± 0.05, 0.41 ± 0.07, 0.60 ± 0.11 and 0.64 ± 0.10 for MEDI, U-Net, DLL2, FINE and ground truth, respectively.

Table 2.

RMSE, SSIM and HFEN for various QSM reconstructions of 6 simulated ICH brains

	RMSE	SSIM	HFEN
MEDI	27.13 ± 8.10	0.9256 ± 0.0312	20.18 ± 9.96
U-Net	64.04 ± 5.99*	0.9380 ± 0.0195	63.37 ± 8.19*
DLL2	49.06 ± 3.93*	0.9371 ± 0.0231	48.99 ± 7.05*
FINE	25.67 ± 2.79	0.9480 ± 0.0310	25.53 ± 4.15

^*denotes statistical significance for the comparison between MEDI/U-Net/DLL2 and FINE; p < 0.05.

4.2.3. MS patients

QSMs reconstructed by MEDI, U-Net, DLL2 and FINE for two representative MS patients are displayed in Figure 3a. MS lesions were depicted using four methods (solid arrows). Compared to MEDI, U-Net reconstruction showed reduced lesion contrast, which was improved using DLL2 and FINE. Deming regression of lesion susceptibilities between MEDI and each of the other three methods are shown in Figure 3b. Regression slopes for FINE (1.05, R²=0.8) and DLL2 (0.96, R²=0.79) were closer to unity than for U-Net (0.88, R²=0.74). In addition, the fine structure of periventricular veins was shown more clearly on FINE and DLL2, as compared to MEDI or U-Net (hollow arrows). Despite the merits above, some shadow artifacts were introduced at the boundary of CSF in FINE.

Figure 3.

a) representative axial images from two MS patients, showing that FINE improves lesion appearance in reference to MEDI. From left to right: QSMs reconstruction by MEDI, U-Net, DLL2 and FINE, respectively. Lesions (solid arrows) near the ventricle were underestimated in U-Net, but were recovered progressively from DLL2 to FINE. The fine structure of periventricular veins was shown more clearly on FINE and DLL2, as compared to MEDI or U-Net (hollow arrows). Despite the merits, some shadow artifacts near the CSF were introduced in FINE. b) Deming regressions of all patients’ lesion mean values between MEDI and the DL based three methods, showing FINE improves the lesion susceptibility accuracy in reference to MEDI. All images were displayed using a [−0.15, 0.15] ppm window.

4.2.4. ICH patients

For ICH patients, QSMs reconstructed by MEDI, U-Net, DLL2 and FINE for a representative ICH patient are displayed in Figure 4. Hemorrhage was underestimated in U-Net, but were recovered progressively from DLL2 to FINE. Mean susceptibility values (ppm) of hemorrhage lesions on 8 ICH patients from each reconstruction method were calculated, giving mean susceptibility values ± standard deviations: 0.63 ± 0.09, 0.33 ± 0.10, 0.36 ± 0.10 and 0.55 ± 0.09 for MEDI, U-Net, DLL2 and FINE, respectively. In contrast to MEDI, which showed the highest mean susceptibility values inside lesions, U-Net and DLL2 had considerable underestimation of lesion susceptibility, while FINE gave results closest to MEDI. The blurring scores of tissues surrounding hemorrhagic lesions were 0.18 ± 0.02, 0.22 ± 0.03, 0.18 ± 0.02 and 0.18 ± 0.01 for MEDI, U-Net, DLL2 and FINE, respectively. MEDI, DLL2 and FINE had comparable sharpness surrounding hemorrhages, while brain tissue of U-Net surrounding hemorrhages was blurrier compared to that of the other three methods.

Figure 4.

QSM shown in three orthogonal planes in a representative ICH patient. From left to right: QSMs reconstructed by MEDI, U-Net, DLL2 and FINE, respectively. Hemorrhage was underestimated in U-Net, but were recovered progressively from DLL2 to FINE. All images were displayed using a [−0.15, 0.15] ppm window except in insets, which used [−0.6, 1.5] ppm.

4.3. Under-sampled reconstruction

4.3.1. Single-channel real-valued image reconstruction

T2w axial images with glioma reconstructed by TV, U-Net, DLL2, MoDL and FINE are displayed in Figure 5. TV suffered from modest structural error. U-Net suffered from substantial structural error (failure in the large uniform region of the glioma). DLL2 substantially reduced structural error in U-Net but still suffered modest error. MoDL also suffered modest structural error (arrow). FINE provided the most accurate reconstruction. Quantitative metrics regarding PSNR and SSIM of MS and glioma test datasets are shown in Tables 3 and 4, with FINE demonstrating the best performance.

Figure 5.

Reconstruction results of one image with glioma. From left to right: fully sampled ground truth, under-sampled k-space reconstruction by TV, U-Net, DLL2, MoDL and FINE, respectively. a) reconstructed image. b) magnitude of reconstruction error with respect to truth. c) zoomed in regions. TV suffered from modest structural error. U-Net suffered from substantial structural error (failure in the large uniform region of the glioma). DLL2 substantially reduced structural error in U-Net but still suffered modest error. MoDL also suffered modest structural error (arrow). FINE provided the most accurate reconstruction.

Table 3.

PSNR and SSIM for real-valued T2w MS patient test dataset reconstruction.

	PSNR (dB)	SSIM
TV	38.11 ± 2.62*	0.9791 ± 0.0090*
U-Net	32.55 ± 1.57*	0.9493 ± 0.0144*
DLL2	37.17 ± 1.78*	0.9765 ± 0.0078*
MoDL	40.98 ± 2.94	0.9874 ± 0.0063
FINE	40.52 ± 1.86	0.9869 ± 0.0051

^*denotes statistical significance for the comparison between TV/U-Net/DLL2/MoDL and FINE; p < 0.05.

Table 4.

PSNR and SSIM for real-valued T2w Glioma patient test dataset reconstruction.

	PSNR (dB)	SSIM
TV	38.48 ± 2.16*	0.9756 ± 0.0098*
U-Net	31.79 ± 1.46*	0.9228 ± 0.0229*
DLL2	36.64 ± 1.57*	0.9653 ± 0.0115*
MoDL	40.57 ± 2.28	0.9838 ± 0.0076
FINE	40.88 ± 2.15	0.9831 ± 0.0080

^*denotes statistical significance for the comparison between TV/U-Net/DLL2/MoDL and FINE; p < 0.05.

Using the Adam solver with different initial learning rates (2 × 10⁻⁴ and 5 × 10⁻⁵), or using a different solver (RMSprop) with initial learning rate 1 × 10⁻⁴ yielded similar results on the MS test dataset, with average PSNRs: 40.30 ± 1.85, 39.97 ± 1.85 and 39.86 ± 1.58, respectively and average SSIMs: 0.9858 ± 0.0056, 0.9868 ± 0.0051 and 0.9863 ± 0.0048, respectively. Using a different structure for the pre-trained network (5 convolutional layers) resulted in higher error on the MS dataset, with average PSNR of 37.22 ± 2.19 and SSIM of 0.9710 ± 0.0117. Changes in PSNR and SSIM (Table 3) by changing optimizers and learning rates were not statistically significant (p > 0.05), while those induced by a network architecture change were significant (p < 0.05). This indicates that the encoder-decoder structure of U-Net with low dimensional latent features was more efficient in image-to-image generation task.

4.3.2. Multi-channel complex-valued image reconstruction

The PSNR and SSIM of the TV, U-Net, DLL2, MoDL and FINE reconstructions in the complex-valued multi-coil T2w sagittal test images are shown in Table 5, indicating FINE and MoDL had the best performance.

Table 5.

PSNR and SSIM for complex-valued T2w test dataset reconstruction.

	PSNR (dB)	SSIM
TV	39.52 ± 1.66*	0.9867 ± 0.0041*
U-Net	28.75 ± 1.95*	0.9206 ± 0.0259*
DLL2	38.95 ± 2.22*	0.9853 ± 0.0062*
MoDL	43.16 ± 1.59	0.9922 ± 0.0026*
FINE	42.93 ± 2.53	0.9898 ± 0.0051

^*denotes statistical significance for the comparison between TV/U-Net/DLL2/MoDL and FINE; p < 0.05.

4.3.3. Dependency of FINE performance on initial training dataset

Figure 6 shows the reconstruction performance on the MS test dataset in terms of PSNR and SSIM as a function of training dataset type and size. The performance of FINE was improved as the size of the training dataset increased, whether training was performed on MR or natural images. In addition, the performance of FINE trained on natural images was below that of FINE trained on MR images, but was slightly better than end-to-end mapping trained on natural images without using FINE. It’s worth noting that FINE trained on 2200 MR images had nearly identical performance as that trained on 8800 MR images (Table 2), which indicates that FINE could reach optimal performance with less than 2200 pre-trained MR images.

Figure 6.

a) PSNR metrics of two types of pre-training dataset with different number of images before and after FINE. b) SSIM metrics of two types of pre-training dataset with different number of images before and after FINE. Given the same size of pre-trained dataset, FINE trained on MR image dataset had better performance than trained on natural image dataset. FINE with 2200 MR images for pre-training had nearly identical performance to the one with 8800 MR images for pre-training shown in table 2.

4.3.4. Dependency of FINE performance on noise.

MoDL and FINE reconstructions on test data with added Gaussian noise (σ = 0.01 and 0.05) were deployed and reconstruction results of a representative T2w image with MS lesions are shown in Figure 7. MoDL reconstructions appeared noisier than FINE and MS lesions contrast appeared higher in FINE. Quantitative metrics regarding PSNR and SSIM are shown in Table 6. Compared to MoDL, FINE showed similar performance for σ = 0.01 , and improved performance for σ = 0.05.

Figure 7.

Reconstructions of one representative image with MS lesions at two noise levels (a: σ = 0.01, b: σ = 0.05). All the reconstructions became noisier as test data noise levels increase. Compared to MoDL, FINE reconstructions looked less noisy and had better depicted lesions.

Table 6.

PSNR and SSIM of MoDL and FINE reconstructions on MS test dataset with noise σ = 0.01 and 0.05.

	σ = 0.01		σ = 0.05
	PSNR (dB)	SSIM	PSNR (dB)	SSIM
TV	36.03 ± 1.79*	0.9713 ± 0.0095*	29.89 ± 1.09*	0.9092 ± 0.0214*
U-Net	32.19 ± 1.57*	0.9463 ± 0.0152*	28.27 ± 0.91*	0.8777 ± 0.0274*
DLL2	36.54 ± 1.66*	0.9718 ± 0.0090*	30.52 ± 0.94*	0.9092 ± 0.0226*
MoDL	37.64 ± 1.77	0.9757 ± 0.0085*	28.00 ± 0.54*	0.8558 ± 0.0319*
FINE	37.51 ± 1.39	0.9795 ± 0.0058	31.57 ± 1.74	0.9428 ± 0.0170

^*denotes statistical significance for the comparison between MoDL and FINE; p < 0.05.

4.3.5. Stability of FINE performance against adversarial attack

Two levels of adversarial noises (∣r₁∣ < ∣r₂∣) were generated on a specific T2w image of pre-trained U-Net, and FINE was applied after U-Net’s outputs. Reconstructions are shown in Figure 8 along with TV method as a comparison. Compared to U-Net outputs that suffered from adversarial attacks, FINE was able to effectively reduce those artifacts.

Figure 8.

Reconstructions of a representative T2w image without adversarial noise (a) and with adversarial noise (b-c). Artifacts in U-Net output increased as adversarial noise levels increased from (b) to (c). These artifacts were suppressed in both TV regularized and FINE reconstructions (arrows), which appeared more similar to their corresponding ground truth images (first column).

5. DISCUSSION

Our results indicate that the proposed approach of fidelity imposed network edit (FINE) can be effective in reducing errors when using neural network models to solve ill-posed inverse problems in medical image reconstruction. FINE improves the fidelity of test data by adapting the weights of a pre-trained CNN through backpropagation according to the physical model for each case in the test dataset. Therefore, FINE offers two distinct benefits in solving the ill-posed inverse problem of reconstruction: the implicit regularization achieved via a pre-trained neural network model, and the physical model defined data fidelity. Compared to a conventional reconstruction with explicit total variation regularization, a supervised CNN (U-Net) reconstruction, and a CNN (U-Net) reconstruction with further explicit regularization, FINE can provide more accurate reconstruction as exemplified in QSM and under-sampled reconstruction (Figures 1, 3, 4 and 5).

It is observed in FINE that updating network weights on test data improves the fidelity of the reconstruction. This FINE approach is closely related to prior work, deep image prior, which trains a DL network from scratch on a single data for inverse problems of denoising, super-resolution, and inpainting (Ulyanov et al., 2018). However, in FINE, the network is initialized to a pre-trained network, rather than trained from scratch and the iterative updating is stopped when the relative change in fidelity loss reaches a threshold. When to stop updating weights of a network remains an empirical question, which should be carefully evaluated for a given application and network structure; This work was done on U-Net networks for QSM and undersampled image reconstruction. This FINE approach is also related to transfer learning that first trains a DL network on a base dataset and then uses the trained weights to initialize training on a target dataset (Yosinski et al., 2014). However, in transfer learning, once the updated weights are found, they are fixed for the test datasets. Moreover, they require ground-truth/labels for the target dataset during training. In contrast, FINE updates the weights for each case in a test dataset using a loss function that is different from that in the initial training and is based on the forward physical model. Additionally, it does not require ground-truth/labels for the target datasets.

The fidelity term in FINE is used in a generalized sense that it includes the physical forward model and other data characteristics. The core objective of FINE is to reduce the generalization error in the neural network based reconstruction, which may stem from discrepancies, such as pathologies and abnormal morphology between training and test data. Increasing noise levels in the test data often results in decreasing PSNR and SSIM of the network reconstruction, which can be improved by FINE. The inadequacy of the commonly used U-Net can be improved by incorporating the physical model in a CNN as in MoDL (Aggarwal et al., 2019; Schlemper et al) or in iterative projections using many CNN sub-blocks (Mardani et al., 2019). However, MoDL still suffers from possible discrepancies between training and test data. As FINE updates network weights with respect to test data, FINE can outperform MoDL at high noise levels in the undersampled reconstruction application we considered. However, FINE may not address U-Net’s structural inferiority to MoDL in the low noise case in Figure 7.

There is substantial neuroimaging interest in QSM (Wang et al., 2017), including studies of the metabolic rate of oxygen consumption (Zhang et al., 2015), brain tumor (Deistung et al., 2013), deep brain stimulation (Dimov et al., 2019), multiple sclerosis (Chen et al., 2014), cerebral cavernous malformation (Tan et al., 2014), Alzheimer’s disease (Acosta-Cabronero et al., 2013), Parkinson’s disease (Murakami et al., 2015), Huntington’s disease (van Bergen et al., 2016), and magnetic nanocarrier biodistribution (Liu et al., 2010). As QSM needs prior information to execute the ill-posed dipole inversion, seeking a better image feature for regularizing reconstruction has continuously been a major development effort (Kee et al., 2017; Langkammer et al., 2018; Wang and Liu, 2015). Mathematically, regularization should project out or suppress the streaking artifacts associated with granular noise and shadow artifacts associated with smooth model errors (Kee et al., 2017). Streaking artifacts have been effectively reduced using L1-type regularizations, but these techniques suffer from staircase artifacts or blockiness. Shadow artifacts have yet to be effectively suppressed, partly due to white matter magnetic anisotropy (Liu et al., 2018; Wisnieff et al., 2013). These QSM reconstruction challenges may be addressed more effectively using sophisticated and complex image features (Langkammer et al., 2018). Deep neural networks promise to provide the desired but indescribable complex image features. U-Nets have been used to map the tissue field into QSM (Rasmussen et al., 2018; Yoon et al., 2018). However, limited to the training data, these networks may not properly reconstruct new patterns not encountered in the training dataset. This is exemplified in Figure 4, where the hemorrhage, which was not present in training datasets obtained in healthy subjects, cannot be properly reconstructed by U-Net. MoDL overcoming structural limitation in U-Net and FINE overcoming network’s lack of fidelity of test data should be explored in future QSM reconstruction.

Similar to QSM, image reconstruction from noisy under-sampled k-space data in MRI is also an ill-posed inverse problem and requires suitable regularizations to suppress artifacts associated with the under-sampling pattern. L1-type regularizations for MAP inference based reconstruction have been shown to be effective in suppressing noise-like artifacts, but image quality suffers from blockiness. Using data fidelity with CNN image reconstructions have shown improvement in highly under-sampled data (Aggarwal et al., 2019; Schlemper et al., 2018). FINE promises improvements in image reconstruction of test data that differs from the training set. The MRI under-sampling rate may be further increased in imaging situations of redundancy, including multiple contrasts (Huang et al., 2012), various magnetization preparations (Nguyen et al., 2008), and navigator motion tracking (Wang et al., 1996). Deep neural network reconstruction with FINE as demonstrated here may be very promising to accelerate these imaging tasks.

There are several limitations present in this work. First, FINE was shown to improve image reconstruction compared to neural network based reconstructions. However, the effectiveness of FINE has to be assessed for specific training regimes and network architectures. Network structures incorporating the physical forward model may be preferred for image reconstruction. FINE improvements on such network are yet to be demonstrated and may be limited to reducing effects of discrepancy between training and test data. Second, FINE updates iteratively the network weights by minimizing the fidelity loss of the test data. While FINE achieved high quality image reconstruction across data sets tested in this work, which included various pathologies and/or noise levels not seen in the training data, early stopping is necessary to avoid overfitting to noise. This was achieved using a stopping criterion based on the relative change between iterations and a fixed threshold. While empirical, this criterion lead to good performance of FINE reconstructions in the various data sets used in this work. Third, the computational cost of FINE is much higher than a single pass through a deep neural network, due to the additional network updating based on the iterative optimization. The computational cost may be reduced by updating a subset of layers instead of the entire network in the optimization, as in transfer learning (Shin et al., 2016).

In summary, for each test case, data fidelity is used to update the neural network weights to improve reconstruction quality. This fidelity imposed network edit (FINE) strategy promises to be useful for solving ill-posed inverse problems in medical imaging.