The 2016 QSM Challenge: Lessons learned and considerations for a future challenge design

Abstract

Purpose:

The 4^th International Workshop on MRI Phase Contrast & QSM (2016, Graz) hosted the first QSM challenge. A single-orientation GRE acquisition was provided, along with COSMOS and the χ₃₃ STI component as ground-truths. The submitted solutions differed more than expected depending on the error metric used for optimization and tended to over-regularization. This raised (unanswered) questions about the ground-truths and the metrics utilized.

Methods:

We investigated the influence of background field remnants by applying additional filters. We also estimated the anisotropic contributions from the STI tensor to the apparent susceptibility, to amend the χ₃₃ ground-truth and to investigate the impact on the reconstructions. Lastly, we used forward simulations from the COSMOS reconstruction to investigate the impact that has noise on the metrics.

Results:

Reconstructions compared against the amended STI ground-truth returned lower errors. We show that the background field remnants had a minor impact in the errors. In the absence of inconsistencies, all metrics converged to the same regularization weights, whereas SSIM was more insensitive to such inconsistencies.

Conclusion:

There was a mismatch between the provided data and the ground-truths due to the presence of unaccounted anisotropic susceptibility contributions and noise. Given the lack of reliable ground-truths when using in vivo acquisitions, simulations are suggested for future QSM challenges.

Introduction

Quantitative Susceptibility Mapping (QSM) is a relatively new MRI technique, where susceptibility changes in tissues are conventionally estimated from Gradient Recalled Echo (GRE) acquisitions^1,2. The phase of the GRE signal is proportional to changes in the local (macroscopic) magnetic field induced by the underlying distribution of magnetic susceptibility in the tissue^3,4. The calculation of the magnetic susceptibility distribution from a given magnetization field is an ill-conditioned and also ill-posed inverse problem^5,6. One of the reasons for the ill-conditioning of the problem is that the dipole convolution kernel, which describes the susceptibility-to-field problem, decays to zero around a double-shaped conical surface at the magic angle around the main field direction. A straightforward inversion implies a division by small numbers, resulting in severe noise-amplification^5,7. Reasons for the ill-posedness of the problem include the limited field of view coverage, signal dropout due to fast T₂* decay (or lack of signal in bone and air regions), and non-Gaussian noise behavior^8–10. The majority of the most recently published QSM techniques rely on solving the inverse problem by minimizing an objective function that includes a regularization term. Other QSM techniques rely on modifications to the dipole kernel, or to exploit approaches derived from the Compressed Sensing theory¹¹. All these techniques try to reduce streaking artifacts and to prevent noise amplification during the inversion process¹². Other methodological studies focus on preprocessing steps, such as the removal of background fields (magnetization fields originated from objects outside of the field of view)^13,14, phase unwrapping and multi-echo / multi-coil combination techniques^15,16. The integration of those techniques and the inversion problem into a single process is referred as the so-called single-step algorithms^17–23.

In addition to those methodological papers, several clinical studies using QSM have been conducted, some of them combining well-established relaxometry measures such as T2*/R2*^24–29. Generally, it has been observed that the quantified susceptibility values vary depending on the employed methodology³⁰.

Given such a variety of methods and associated results, a QSM Challenge was proposed in the context of the 4^th International Workshop on MRI Phase Contrast & QSM (Graz, Austria, September 2016)³¹. The challenge aimed at defining a faithful reconstruction of magnetic susceptibility. A single head orientation GRE acquisition (from a multi-orientation set) was released to the contestants to test their algorithms. The whole multi-orientation set was used to calculate the susceptibility tensor³² (STI) and a Calculation Of Susceptibility through Multiple Orientation Sampling (COSMOS)³³ reconstruction. While the χ₃₃ term of the susceptibility tensor was used as ground-truth, the COSMOS reconstruction was also provided. Anisotropic elements of the susceptibility tensor (χ₁₃ and χ₂₃) were discarded, assuming that their contributions were negligible in an acquisition with an orientation parallel to the main field. Since most of the algorithms depend on one or more parameters that must be fine-tuned, the ground truth was provided to allow for optimization of the parameters, ensuring that submitted susceptibility maps represent the best results that can be achieved with a certain algorithm. Totally 27 reconstructed QSM images were submitted by 15 teams, and 4 metrics were evaluated (Root Mean Square Error - RMSE, High Frequency Error Norm - HFEN, Structural Similarity Index Metric - SSIM, and error in anatomical ROIs) in separate categories. Henceforth, teams optimized their results to minimize (or maximize in the case of SSIM) a certain quality metric. It turned out that most of numerically well performing approaches produced (visually) over-regularized susceptibility maps either blurry or piece-wise smooth. Reconstructions (and the optimal parameter values) also varied upon the employed quality metric. Furthermore, many approaches performed well according to a particular metric, but poorly according to others. Subsequent discussions and analyses pointed at several possible sources of inconsistency that might have degraded the given phase data and therefore its correspondence to the provided ground truth susceptibility. These effects include incomplete background field removal, noise, anisotropic susceptibility and microstructure contributions, flow and motion artifacts, and co-registration (i.e. misalignment of the multiple orientation acquisitions) errors. In addition, questions were raised about the behavior of the selected quality metrics, mainly whether or not they could provide the same optimal parameters.

In this paper, we investigated the viability of the provided χ₃₃ susceptibility map as ground truth for single orientation acquisitions and compared it with the COSMOS reconstruction and an STI derived scalar projection. We also investigated the impact of noise and the influence of background field residuals in the reconstructions.

Methods

MRI data

The 2016 QSM Reconstruction Challenge dataset (available at http://qsm.neuroimaging.at) included rf-spoiled 3D-GRE scans acquired on a 3T system (Siemens Tim Trio) using a 32-channel head-coil. Data were acquired with 1.06-mm isotropic voxels, 15-fold Wave-CAIPI³⁴ acceleration, 240×196×120 matrix size, echo time (TE)/repetition time (TR)=25/35ms, and flip angle=15°. Twelve head orientations were used for COSMOS and STI calculations. Challenge participants were provided with a single transverse-oriented acquisition (from the multi-orientation dataset), the raw phase and a background filtered phase, additionally containing the anatomical MPRAGE³⁵ data³¹. Unwrapping was performed using a Laplacian algorithm³⁶, while background removal was performed using the Laplacian Boundary Value (LBV)³⁷ approach. Since this method is not able to completely remove transmit/receiver coil contributions, a polynomial fit was also used to remove these components^31,38. In the analyses presented here, we worked on the same data set as the one provided by the challenge organizers. We also used the same error metrics (RMSE, HFEN and SSIM) and parameters, as those available in³¹ (http://www.neuroimaging.at/pages/qsm.php). Please note that RMSE values are normalized by the L2-norm of the given ground truth, as percentages. The HFEN metric is also normalized, by the L2-norm of the high frequency coefficients of the ground truth. The SSIM is not normalized.

Experiment design

1. Background field remnants.

Background field removal is one part of the QSM pipeline where results may vary substantially depending on the algorithm used. The QSM challenge data was processed using LBV and polynomial fit steps. To evaluate the quality of the provided local phase, and how background field remnants could impact in the error metrics, we included two additional methods to reduce the remaining background fields. The first method is the recently proposed Weak-harmonics QSM (WH-QSM) reconstruction¹⁹, where an estimate of the background field is jointly reconstructed with the susceptibility map, by imposing a weak-harmonics regularization to decouple terms. The second method is the popular Projection onto Dipole Fields³⁹ (PDF) algorithm, followed by a TV-FANSI reconstruction¹⁰ (300 iterations, and μ = 100*α, μ is an internal Lagrangian weight and α is the regularization weight). Both methods were chosen to avoid further ROI reduction and to minimize over-filtration. We optimized the metric scores by an exhaustive search, for different regularization weights (α), with multiplicative steps of 10^0.2. We compared these results with reconstructions performed using TV-FANSI and no additional background removal step. Error metrics were evaluated using χ₃₃ and COSMOS as ground truth.

2. Incorporation of susceptibility anisotropy and microstructure to QSM.

2.1. Forward phase estimates with anisotropic components.

One way to compare the validity of the provided ground-truths is to estimate the differences between the calculated magnetization field generated by such susceptibility maps with the single-orientation acquired phase. To account for anisotropic components, we calculated the whole susceptibility tensor from the in vivo dataset using all 12 orientations, provided by the challenge organizers, and then applied the full forward STI model to synthesize the phase of a single acquisition in the same orientation as the main field³²:

$$\frac{F{\Phi }_{STI}}{\gamma H_{0}TE}=\frac{1}{3}F{\chi }_{33}-\frac{k_z^2}{k^2}F{\chi }_{33}-\frac{k_z{k}_x}{k^2}F{\chi }_{13}-\frac{k_z{k}_y}{k^2}F{\chi }_{23},$$ Eq. 1

This simulated phase was compared with the acquired single orientation phase, as well as the simulated fields from the χ₃₃ and COSMOS ground-truths (obtained by convolution with the dipole kernel). RMSE calculation was repeated for each case.

2.2. Solving the inverse-problem with anisotropic components.

The susceptibility distribution calculated by QSM methods is an apparent susceptibility that incorporates both isotropic and anisotropic components. The latter is the result of micro-structure effects and the tensor nature of the susceptibilities of certain tissues. Whereas some studies^40–42 have described how microstructure affects both QSM and STI, we are interested in providing a scalar apparent susceptibility ground-truth from STI. For this effect, a projection of the STI data onto a scalar value is required. Hence, to assess the impact of the anisotropic components on the QSM estimations, and how it compares with the χ₃₃ component, a new ground-truth must be derived A direct division of the synthetic STI-derived phase (Eq. 1) by the dipole kernel yielded:

$${\chi }_{STI}={\chi }_{33}+F^{-1}\left(\frac{-k_z}{k^2-3k_z^2}\right)\left(k_{x}F{\chi }_{13}+k_{y}F{\chi }_{23}\right),$$ Eq. 2

This may be interpreted as:

$${\chi }_{STI}={\chi }_{33}+{\chi }_{\mathit{\text{corr}}},$$ Eq. 3

where χ_corr is a correction term that incorporates projections of the anisotropic components. However, χ_corr cannot be analytically derived since it involves division by zero-valued coefficients within the “magic cone”. Approximations to this correction term were computed using TKD⁵ (threshold = 0.17) and Tikhonov⁶ (λ = 6E-2) regularization methods, to obtain two new proposed ground-truths: χ_STI-TKD and χ_STI-L2. We chose the L2 and TKD algorithms for these correction terms to avoid giving TV-FANSI an unfair advantage if variational penalties were also applied as regularization in this step. Algorithmś parameters were chosen for the apparent well-behavior when working with the single acquisition data and were directly applied for this correction factor estimation. QSM reconstructions were performed using TV-FANSI¹⁰ (maximum 50 iterations), with an exhaustive search of the optimal regularization weight using multiplicative 10^0.1 steps from 10⁻⁶ to 10⁻². Metric scores were evaluated using χ_STI-TKD, χ_STI-L2, χ₃₃ and COSMOS as ground truth, respectively.

Alternatively, from Eq. 1, we may instead use the anisotropic χ₁₃ and χ₂₃ terms to forward simulate a phase correction term for the acquired data. This amended phase corresponds to Φ₃₃, the scalar magnetization of a susceptibility distribution χ₃₃ (i.e. convolving by the dipole kernel). Then, it becomes possible to compare QSM reconstructions using χ₃₃ as ground truth by using either the acquired phase or the amended phase as input data for the algorithm (Eq. 4).

$$F{\Phi }_{33}=\gamma H_{0}TE\left(\frac{1}{3}-\frac{k_z^2}{k^2}\right)F{\chi }_{33}={\Phi }_{acq}+\gamma H_{0}TE\left(\frac{k_z{k}_x}{k^2}F{\chi }_{13}+\frac{k_z{k}_y}{k^2}F{\chi }_{23}\right),$$ Eq. 4

To see if background field remnants had still an effect in the reconstructions when anisotropic components are taken into consideration (the cumulative effect), we repeated the background removal analysis described earlier (section 1) for the best scoring (lowest RMSE/HFEN and larger SSIM) phase/ground-truth set.

3. Impact of SNR to metrics and ground-truth solutions.

Data provided for the challenge corresponded to a single-orientation acquired using Wave-CAIPI with a high acceleration factor, thus enabling relatively short acquisition times (all 12 head orientations were acquired within one hour). In turn, this high acceleration factor produced low SNR data. Statistical analysis of the real and imaginary components in the signal-void background gave a peak SNR between 40 to 60. In order to assess the influence of noise in the reconstruction quality (measured by the metrics used in the challenge) and the correlation between the predicted optimal parameters by each metric, we designed an analytic simulation experiment. We used COSMOS-based forward simulations, where the phase data were simulated by convolving with the dipole kernel. In these experiments, susceptibility values outside the brain were set to zero (the mean susceptibility value). We used the calculated phase and the provided magnitude data (normalized between 0 and 1) to create real and imaginary images. Gaussian noise was added to each channel in the complex image domain, with SNR settings from 2 to 128 in a dyadic sequence. Noise-corrupted phase images were used directly as input for TV-FANSI reconstructions. Because only local phases were simulated, objects outside the region of interest were not included and consequently, unwrapping or background field removal methods were not required. Individual optimal reconstruction parameters were found for each error metric and SNR level.

The influence of background remnants and anisotropic components was also studied, on two phases forward simulated at SNR = 32 and SNR = 64, in concordance to our estimated SNR range for the in vivo acquisition. We included the remnants found by the PDF method, and the phase derived from the anisotropic tensor elements, χ₁₃ and χ₂₃ (Eq. 1) in this analysis. Metric scores were evaluated using TV-FANSI reconstructions and compared with the in vivo metric scores (obtained from the experiments described in previous sections).

In all the presented experiments, QSM reconstructions were evaluated with RMSE, HFEN and SSIM (with K = [0.01, 0.03] and L = 255 parameters⁴³, as used by the organizers of the challenge), with respect to each ground-truth. To provide further insights about quality metrics, we also include the analysis of metrics not used in the challenge, such as the correlation coefficient (CC), mutual information⁴⁴ (MI), root mean squared error of the gradient of susceptibilities (GXE), mean absolute deviation (MAD) and the mean absolute deviation of the gradient of susceptibilities (MADGX). GXE, MAD and MADGX errors are normalized by the L2-norm and L1-norm of the gradients of the ground truth, and the L1-norm of the ground truth, as correspond for each metric. Source code to evaluate all the described metrics is available at the FANSI-Toolbox site: http://gitlab.com/cmilovic/FANSI-Toolbox

Results

1. Background field remnants.

Estimations of the background field remnants using PDF and WH-QSM are presented in Figure 1. Visually, both methods yield similar remnants near the boundaries of the ROI, especially at lower and upper regions. Smooth remnants seem to be also present in deep brain areas. In both cases, remnant background phase information seems to be smaller in magnitude than the local phase data, revealing a minor contribution.

Figure 1.

Single orientation acquisition (A) and background remnant models using (B) weak-harmonics (WH-QSM, with β=5, μ_h=0.1, the weak harmonic regularization weights) and (C) the Projection onto Dipole Fields (PDF) methods.

Susceptibility calculations were performed using the provided local phase, and the local phase further filtered by PDF using TV-FANSI. The supplied phase was also used as input for WH-QSM, which jointly estimated the background and performed the susceptibility calculation. Error scores for these results are presented in Table 1.

Table 1:

Quality metric scores of QSM reconstructions with additional background filtering

	COSMOS
	RMSE	HFEN	SSIM
TV-FANSI	64.49	59.85	0.876
WH-QSM	60.05	58.05	0.885

Both background filtering methods show small improvements (lower than 5,5%) for all three metrics (except for HFEN, using PDF+TV-FANSI and COSMOS as ground truth which scored a 1,4% higher error). Optimal regularization weights (Table 2) were similar between methods (with WH-QSM obtaining slightly smaller optimal regularization weights for most metrics), and results were still visually over-regularized (Supplementary Figure S1). RMSE seems to be more sensible to background remnant errors than HFEN and SSIM.

Table 2:

Optimal Regularization weight found for each QSM reconstruction

	COSMOS
	RMSE	HFEN	SSIM
TV-FANSI	6.3E–05	4.0E–05	2.5E–05
WH-QSM	4.0E–05	4.0E–05	1.6E–05

2. Incorporation of susceptibility anisotropy and microstructure to QSM

2.1. Forward phase estimates with anisotropic components.

The simulated field (i.e. the phase data) estimated from the STI equation (Eq. 1) returned lower RMSE values (relative to the phase from the single acquisition) than forward simulations using χ₃₃ (−20.8%) and COSMOS (−9.4%), as seen in Figure 2. Phase differences are also reported in Supplementary Figure S2 with higher contrast. Notably, the STI-based phase error map (Figure 2D) contains no medium or large-scale anatomical features. Differences are more prominent in the venous system, which seem to be independent on the orientation of the vessels.

Figure 2.

Differences between forward simulated phases and the provided single orientation acquisition (A) for: (B and E) the χ₃₃ component, (C and F) COSMOS, and (D and G) the STI equation, including the χ₁₃ and χ₂₃ anisotropic components. The right column shows the same difference maps as the left column, but with a higher contrast to reveal subtle differences.

2.2. Solving the inverse-problem with anisotropic components.

Estimated χ_corr-L2 and χ_corr-STI correction images are shown in Figure 3. These corrections are incorporated into the χ₃₃ ground-truth (χ_STI maps), for optimization of the reconstructions with the single orientation acquisition. The χ_STI maps returned better quality metric scores than χ₃₃, and at least similar to COSMOS (Table 3). This indicates that the proposed χ_STI solutions constitute more consistent ground truths for single orientation QSM. Even so, the amended Φ₃₃ phase with anisotropic components turned χ₃₃ into the best ground truth in this comparison. Notably, optimal regularization parameters for QSM reconstructions (for a given target metric) were smaller using the STI-based ground-truths and the amended Φ₃₃ phase, thus achieving less over-regularized results (Figure 4 and Supplementary Information Figure S4). In addition, regularization weights were more similar across different metrics than parameter optimizations performed with the provided ground-truths. This greater convergence of the metrics indicates that incompatibilities between the input data and the proposed ground-truth are being minimized.

Figure 3.

The χ₃₃ (isotropic) component and correction factors, χ_corr (B and C, derived from χ₁₃ and χ₂₃ anisotropic components) used to estimate χ_STI-TKD and χ_STI-L2 respectively (D and E).

Table 3:

Quality metric scores and optimal QSM regularization parameters of ground-truth solutions obtained with TV-FANSI.

	RMSE	α	HFEN
χ33	76.0	1.0E–04	72.3
COSMOS	62.6	5.0E–05	59.6
χSTI-TKD	65.2	5.0E–05	60.4

Figure 4.

Selected optimal reconstructions of the challenge data, based on the RMSE (left panel) and SSIM (right panel) metrics, using χ₃₃ (A,D) and χ_STI-L2 (B, E) as ground truth. (C ,F) show optimal reconstructions of the amended phase (to remove anisotropic components) and using χ₃₃ as ground truth (χ₃₃|Φ₃₃).

Repeating the background removal analysis on this amended phase yielded small improvements (less than 2%) for susceptibility maps using WH-QSM and no improvement for PDF (see Supplementary Information Figure S5 and Table S1 for details). This is a minor improvement, compared to the ~15% gains shown in Table 3. We found a 4.8% RMSE improvement using the remnants calculated from Φ₃₃ to correct the forward simulation of the STI phase. In contrast, the background remnant calculated from the acquired phase returned a 3.6% RMSE improvement when comparing to the forward simulations using χ₃₃ and COSMOS. Both remnant maps are presented in Supplementary Figure S3.

3. Impact of SNR to metrics and ground-truth solutions.

TV-FANSI reconstructions and optimal metric scores using forward simulated phases with COSMOS as ground truth for SNR=2 to SNR=128 are displayed in Figure 5A–C. Simulations at higher SNR levels (10³ and 10⁶, not shown in Figure 5) reveal a minimal error level at approximately 17.6% RMSE, 15.7% HFEN and 0.99 SSIM. Scores are almost perfect for SNR=10⁶ if no magnitude information is incorporated in the simulation of the complex normal noise (0.02% RMSE, 0.01 HFEN and 1.0 SSIM), revealing that this plateau is caused by the truncation of the magnetization fields due to existence of regions without MRI signal. Simulations done at SNR = 20 yield errors comparable to those found in the best in-vivo scores, which is not the case of the supplied data (with an SNR at least in the 40–60 range, as shown in Supplementary Information Figure S6). In a realistic SNR range, RMSE and HFEN scores reveal that the noise level of the single acquisition cannot explain the scores obtained by reconstructions using χ₃₃ as ground-truth. Both our χ_STI proposed ground-truth and working with the Φ₃₃ amended phase reach a score level closer to the baseline.

Figure 5.

Metric scores of the reconstruction using COSMOS-based forward simulations. Optimal (A) RMSE, (B) HFEN and (C) SSIM scores are presented as function of SNR. Metric scores for in vivo reconstructions using χ₃₃ and χ_STI-L2 as ground truth, along with reconstructions of the amended phase with TV-FANSI (χ₃₃|Φ₃₃) and WH-QSM, are also included. (D) presents the optimal regularization weight for each metric and SNR setting. Ranges for the optimal reconstructions using χ₃₃, χ_STI-L2 and χ₃₃|Φ₃₃ also included.

In terms of the regularization weight (Figure 5D), optimal values for the HFEN metric tended to be the lowest in simulations, whereas SSIM yielded the lower values working with the in vivo data. Maximum differences in the optimal weights between metrics (at a given SNR) grows exponentially with noise, at a similar rate, with a high correlation between the metrics. Additional metrics (Supplementary Information Figure S7) seem also to be correlated. MI seemed to be the least reliable.

When background remnants and anisotropic components are taken into consideration in our simulations (Figure 6 and Supplementary Information Figure S8) the mismatch is closer in line with the behavior of the challenge data. Anisotropic components were the largest source of mismatch, with background remnants having a smaller contribution when both terms are considered together.

Figure 6.

(A) RMSE, (B) HFEN and (C) SSIM optimal scores for reconstructions at SNR=64 and SNR=32 and their regularization weight. Simulations included the phase forward calculated using COSMOS Φ with the addition of the background remnants obtained using PDF (Φ_PDF) and the anisotropic phase contributions (Φ_corr) calculated from the χ₁₃ and χ₂₃ components of the STI tensor. (D) Shows the range span of optimal regularization weight values by the different simulations and in vivo reconstructions, for a given metric.

Exhaustive metric scores for reconstructions of the provided in vivo phase, using both χ_STI maps, χ₃₃ and COSMOS as ground truth, are presented in Supplementary Information Figure S9. This figure includes all the analyzed metrics.

Discussion

The analysis of the submitted results to the 2016 QSM Reconstruction Challenge³¹ raised a few issues about the challenge design (input data and ground truth) and the metrics. Errors (measured by the metrics) using COSMOS as ground-truth were significantly smaller to those using χ₃₃, in opposition to the assumption that χ₃₃ was a better representation of the apparent susceptibility in the z-axis (by discarding any anisotropic contribution). Furthermore, most QSM submitted results were over-regularized. And finally, the regularization weights depended significantly on the metric chosen to be optimized for. Because of this, some algorithms high-scoring in one category performed worse in the others. To explain these observations, we conducted a series of experiments to determine the role of background remnants, anisotropic components and noise in the metric scores, and visual appeal of the results.

Background filtered reconstructions show small improvements for error metrics, independent of the ground truth. This indicates that although background remnants were present in the provided local phase, their impact on the reconstructions is secondary to other effects. Optimal regularization weights were consistent between methods (with WH-QSM obtaining slightly smaller optimal weights).

Additionally, arguing against the relevance of background remnants is that all the multi-orientation data were filtered using the same pipeline (LBV + polynomial fit). Although this may introduce errors in the STI estimation, background ghosting or boundary errors should be mostly contained in the isotropic χ₃₃ component, with a biased estimation similar to the one in the single-orientation reconstruction.

With background remnants discarded as main contributors to degradation of the metric scores, we analyzed the validity of assuming χ₃₃ as the ground truth for a single orientation acquisition. The assumption that anisotropic magnetizations and microstructure effects were negligible was invalidated by comparing the forward simulated phases using χ₃₃, COSMOS, and the complete STI equation. RMSE of the phase calculated from χ₃₃ was the highest, whereas the phase from the STI equation was the lowest (by 9.4% with respect to COSMOS and 20.8% to χ₃₃). The remnant phase map using the full STI model (Fig. 2D) has no large-scale features, and major discrepancies that are found in the vasculature may be associated with the blurring effect created by spatial interpolation in the co-registration step (in the process of calculating the STI tensor), or flow effects in this non-compensated acquisition. An additional source of uncertainty is the error generated at the STI estimation by the use of a limited set of orientations. As shown by Li et al⁴⁵, RMSE could rise up to 30% for acquisitions similar to those in the 2016 Challenge dataset (in terms of SNR and maximum angulation). Nevertheless, this error should account only for a fraction of the estimated anisotropy, given the structures visible in the remnant map, and could be related more to microstructure effects. Unfortunately, this source of error cannot be estimated due to the lack of a real ground truth in this in vivo setting. The remnant map also discards large co-registration errors as a potential source of mismatch, due to the lack of contiguous sharp edge features with a different sign that are created by the misalignment of structures. Some of the medium-scale features, especially near the para-nasal cavities and cerebellum, have similarities with the background remnants shown in Figure Supplementary S2. This explains why the metric scores overall improve when these fields are removed. The COSMOS forward simulated phase partially incorporated anisotropic components, and thus, reconstruction metrics using COSMOS as reference achieved overall better scores than using χ₃₃.

QSM reconstructions that incorporated the anisotropic tensor components (either by comparing to the χ_STI ground truths or using the amended Φ₃₃ phase) resulted in significantly lower errors than when using χ₃₃ or COSMOS as ground truth. In practice, these tensor elements incorporate contributions from susceptibility anisotropic effects as micro-structure effects and chemical shielding. Since calculating the χ_corr term involves a dipole inversion, we used two algorithms that do not impose piece-wise constant solutions to avoid an unfair advantage to TV-FANSI in the metric scores. This is partially reflected in the reconstructions using Φ₃₃ as input, where the anisotropic components were incorporated in the acquired phase and were jointly reconstructed, leading to the lowest error scores. Further background field subtraction improved the scores marginally. Optimal regularization weights associated with these reconstructions also were smaller (however, yielding more visually appealing results) and more consistent between metrics.

The analysis of the forward simulated reconstructions at different SNR scenarios gave more insights to explain the metric scores and the differences in their optimal values. Given that the actual SNR of the single orientation data is in the 40–60 range, noise does not sufficiently explain the measured (worse) metric scores in in vivo acquisitions. Optimal regularization weights were also smaller for our simulations in this noise range, yielding less over-regularization. While SSIM tended to return the lower regularization weights with in vivo data, in the simulations HFEN consistently yielded lower regularization weights, while SSIM tended to return similar values to those derived from using the RMSE. The discrepancy between the optimal regularization weights given by different metrics was also smaller for simulations. These tests revealed that SSIM is less sensitive to local discrepancies than HFEN and RMSE. HFEN promotes sharper (less regularization) results than SSIM and RMSE in the absence of local mismatches. L2 norm-based error metrics (HFEN and RMSE) tend to minimize the total energy of the mismatches, distributing it among neighboring pixels, thus promoting over-regularized results. Consequently, this is an indication that other sources for the mismatch between the acquisitions and the ground-truth are present, beyond noise.

By conducting forward simulations at SNR=32 and SNR=64, which is the expected noise level range of the in vivo acquisition, we explored the influence of background remnants and anisotropic components to the metric scores. This experiment revealed that anisotropic tensor components had a larger influence on the errors than background remnants. Both effects combined increased the errors to a level similar to those found in the challenge, confirming that these two were the major sources of mismatch. Optimal regularization weights also were larger when these effects were included, showing a similar behavior to the in vivo reconstructions for the RMSE and HFEN metrics. The SSIM metric, on the other hand, revealed a larger invariance to these effects. In addition, other error sources might be responsible for minor mismatches of the in vivo reconstruction and ground truth. For example, blurring of the multiple-orientation data due to interpolation might also play a role in this discrepancy. Please note that we are ignoring noise and other effects in the STI or COSMOS estimations as sources of errors. By comparing our reconstruction to noisy or flawed ground-truths (due to the limited available range of angulation values) we are under-estimating the influence of noise and anisotropic contributions. Unfortunately, these errors are hard to estimate without an appropiate ground-truth.

Other global metrics may also be considered, as proposed in Supplementary Information Figure S7–S9, which presented similar trends to RMSE, HFEN, and SSIM. Such metrics may provide additional information about the source of the errors. For image analysis purposes, other alternatives to consider in the future may follow multiscale or vectorized metrics⁴⁶, or the inspection of the SSIM map, which reveal more information regarding the nature of the errors. Unfortunately, such metrics or methods are not directly applicable to rank results for a challenge contest as compared to scalar scores.

Finally, while other potential sources of errors such as vascular flux, motion, etc. might be also considered, it remains an open question on how to properly numerically compare single orientation with multiple orientation reconstructions. Methods to project the susceptibility tensor onto the apparent scalar susceptibility should be explored and validated in future investigations. These should considerate projections onto different head orientations to validate the methods. This may be performed by simply rotating all the images to set a new reference “central” acquisition, thus creating a new, rotated, susceptibility tensor to account for the anisotropic contributions to the new χ₃₃ component. Experimental validation of these methods should also consider the errors incurred in the susceptibility tensor estimation, and estimate how noise impacts the tensor-to-scalar projection. Due to the limitations of in vivo STI acquisitions, this may be complemented with ex vivo acquisitions.

Conclusions

The 2016 QSM Reconstruction Challenge used a highly accelerated Wave-CAIPI GRE sequence to provide single and multiple orientation reconstructions. Despite the low SNR of the provided single orientation phase, noise alone cannot explain the poor metrics scores found in the reconstructions using χ₃₃ as ground truth, and why using COSMOS as ground truth significantly improved these scores. As demonstrated in the experiments in this paper, a non-negligible mismatch was found between the provided local phase data and using χ₃₃ component of the STI tensor as ground truth. Anisotropic magnetization contributions and microstructural effects incorporated in the χ₁₃ and χ₂₃ elements proved to be the major source for this mismatch. A more adequate ground truth may be obtained by projecting these anisotropic contributions into the apparent scalar susceptibility. However, because of the necessity to perform a dipole inversion, this estimation is not trivial and requires further investigation. COSMOS reconstructions can capture some of these effects and may be better suited as ground-truth targets, although differences might be relevant in some structures (white matter and the vascular system). A secondary contribution to the mismatch is the presence of minor background field remnants.

Considerations and Outlook for future QSM challenges

The majority of biomedical imaging challenges (such as from MICCAI) conducted in the past years aimed segmentation and classification⁴⁷. However, owing to the underlying physical problem without a ground truth, comparing the performance of QSM reconstruction algorithms is more complex than what can be reflected by a single numeric error measure, and the question on how to measure the quality of the reconstructions still remains open. Therefore, for future challenges it could be advisable to consider the overall performance instead of relying on a single metric. Multiple metric categories is also an option. To avoid controversy, categories should be announced beforehand, and the scoring source code made available with the release of the ground truth dataset⁴⁷.

In the case of working with in vivo data, thorough background removal steps should be carried out to provide clean local phases. This may be performed by including an additional background field removal step, such as PDF or V-SHARP. Major possible sources of errors could be also masked out by using smaller regions of interest. Given all the challenges involved in using multiple orientation reconstructions as ground-truth, and the unfeasibility of generating a single orientation ground truth, in vivo data remains a challenging ground for QSM reconstruction challenges.

Instead, analytic simulations of the acquired signal should be encouraged. Whole head susceptibility distributions may be derived from in vivo data to provide ground-truth with a realistic brain geometry, from which scalar (isotropic) magnetization fields may be calculated. Piece-wise analytic phantoms should be avoided to prevent unfair advantages to variational approaches (e.g. TV or TGV regularizations). In addition, analytic simulations may be even further extended in the future to incorporate the background field removal step and unwrapping and could provide a complete GRE signal simulation platform on which all different steps of the QSM reconstruction pipeline can be tested and evaluated⁴⁸.

Another consideration for a future challenge design is to include an evaluation of a priori optimization of the results. Parameter fine-tuning without knowing the ground-truth is a non-trivial task. Having the ground-truth hidden from the participants should allow the community to evaluate how good the analytic predictors of optimal parameters are, and how reliable or reproducible the results may be.

In conclusion, the presence of unaccounted for anisotropic susceptibility contributions, noise, and the lack of reliable ground-truths using in vivo acquisitions, strongly suggests synthetic data and forward simulations for future QSM challenge designs.

Supplementary Material

supplementary

Supporting Information Figure S1. Selected optimal reconstructions of the challenge data, based on RMSE (left column) and SSIM (right column), using χ₃₃ as ground truth (A). Panels show reconstructions of the provided local phase using (B, E) TV-FANSI, (C, F) WH-QSM (with joint background removal process), and (D, G) TV-FANSI of the PDF filtered local phase.

Supporting Information Figure S2. Difference between the forward simulated phases using χ₃₃ and COSMOS. Note the windowing.

Supporting Information Figure S3. (A) and (B) reveal the background extracted from the acquired phase and the amended phase, respectively, using WH-QSM (β=500, μ_h=10, the weak harmonic regularization weights).

Supporting Information Figure S4. Difference maps between optimal results using RMSE and SSIM, for the same ground-truth.

Supporting Information Figure S5. Selected optimal reconstructions of the amended phase (without anisotropic contributions) based on the RMSE and SSIM metrics, using χ₃₃ as ground truth. Panels show reconstructions using (A, D) TV-FANSI, (B, E) WH-QSM (with joint background removal process), and (C, F) TV-FANSI of the background filtered local phase, using PDF.

Supporting Information Figure S6. Effect of noise level on forward simulations. (A) Single orientation phase data and forward simulations of Φ_STI at SNR levels: (B) 20, (C) 32, (D) 40, (E) 60, (F) 100 and (G) noiseless.

Supporting Information Figure S7. Metric scores of the reconstruction using COSMOS-based forward simulations. Optimal (A) CC, (B) MI, (C) GXE, (D) MAD and (E) MADGX scores are presented as function of SNR. Metric scores for in vivo reconstructions using χ₃₃ and χ_STI-L2 as ground truth, along with reconstructions of the amended phase with TV-FANSI (χ₃₃|Φ₃₃) and WH-QSM, are also included. (F) presents the optimal regularization weight for all metrics and SNR setting. Ranges for the optimal reconstructions using χ₃₃, χ_STI-L2 and χ_STI-L2|Φ₃₃ also included.

Supporting Information Figure S8. (A) RMSE, (B) HFEN, (C) SSIM, (D) CC, (E) MI, (F) GXE, (G) MAD and (H) MADGX scores for reconstructions at SNR=64 and SNR=32 as function of the reconstruction weight. Simulations included the phase forward simulatedcalculated using COSMOS ? with the addition of the background remnants obtained using PDF (Φ_PDF) and the anisotropic phase contributions (Φ_corr) calculated from the χ₁₃ and χ₂₃ components of the STI tensor. (I) Shows the range span of optimal regularization weight values by the different simulations and in vivo reconstructions, for all given metrics.

Supporting Information Figure S9. Comparison of all proposed quality metrics as function of the regularization weigh for in vivo reconstructions. (A) RMSE, (B) HFEN, (C) SSIM, (D)CC, (E) MI, (F) GXE, (G) MAD and (H) MADGX. Reconstructions times are also provided (I).

Supporting Information Table S1. Quality metric scores and optimal regularization weights of QSM reconstructions of the amended phase Φ₃₃ with additional background filtering.