Sparse Reconstruction Challenge for diffusion MRI: Validation on a physical phantom to determine which acquisition scheme and analysis method to use?

Abstract

Diffusion magnetic resonance imaging (dMRI) is the modality of choice for investigating in-vivo white matter connectivity and neural tissue architecture of the brain. The diffusion-weighted signal in dMRI reflects the diffusivity of water molecules in brain tissue and can be utilized to produce image-based biomarkers for clinical research. Due to the constraints on scanning time, a limited number of measurements can be acquired within a clinically feasible scan time. In order to reconstruct the dMRI signal from a discrete set of measurements, a large number of algorithms have been proposed in recent years in conjunction with varying sampling schemes, i.e., with varying b-values and gradient directions. Thus, it is imperative to compare the performance of these reconstruction methods on a single data set to provide appropriate guidelines to neuroscientists on making an informed decision while designing their acquisition protocols. For this purpose, the SParse Reconstruction Challenge (SPARC) was held along with the workshop on Computational Diffusion MRI (at MICCAI 2014) to validate the performance of multiple reconstruction methods using data acquired from a physical phantom. A total of 16 reconstruction algorithms (9 teams) participated in this community challenge. The goal was to reconstruct single b-value and/or multiple b-value data from a sparse set of measurements. In particular, the aim was to determine an appropriate acquisition protocol (in terms of the number of measurements, b-values) and the analysis method to use for a neuroimaging study. The challenge did not delve on the accuracy of these methods in estimating model specific measures such as fractional anisotropy (FA) or mean diffusivity, but on the accuracy of these methods to fit the data. This paper presents several quantitative results pertaining to each reconstruction algorithm. The conclusions in this paper provide a valuable guideline for choosing a suitable algorithm and the corresponding data-sampling scheme for clinical neuroscience applications.

1. Introduction

Diffusion magnetic resonance imaging (dMRI) allows one to measure the diffusivity of water molecules in biological tissue. The thermal motion of water molecules is constrained by the surrounding tissue. Hence, dMRI provides an indirect measure of the micro-structural properties of the tissue. It has been used to study the connectivity of brain, see e.g. (Corouge et al., 2006; Hagmann et al., 2007; Malcolm et al., 2010; Sporns et al., 2005; Wang et al., 2014), and to investigate many brain disorders such as Alzheimer’s disease, schizophrenia, and mild traumatic brain injury, see e.g. (Shenton et al., 2012; Shi et al., 2013; Thomason and Thompson, 2011).

In practice, dMRI requires a collection of image volumes that are acquired using different diffusion sensitized gradient directions. The diffusion-weighted signal in each voxel reflects the displacement of spins during the sampling period. The probability distribution of the displacements is usually described by the ensemble average propagator (EAP) which captures important structural information of the underlying tissue. A reconstruction algorithm is usually used to estimate the EAP from the diffusion-weighted signal at each voxel of the image. A classical reconstruction method, popularly known as Diffusion Tensor Imaging (DTI), see (Basser et al., 1994; Stejskal and Tanner, 1965), models the EAP as a 3D Gaussian probability density function. This model is too simplistic and has limitations in regions of fiber crossing or branching, which forms a significant portion of the white matter tissue in the brain, see (Behrens et al., 2007). Moreover, the single tensor model in DTI does not provide correct information about signal decay along radial directions (as a function of b-value) which is related to subtle anomalies in white matter, see (Cohen and Assaf, 2002; Assaf et al., 2004). To resolve these problems, advanced models along with a large number of diffusion encoding directions are usually required. This, however, can make the acquisition time too long for several types of patients, rendering it impractical to use in many clinical research studies.

Based on the type of data sampling scheme used, the reconstruction methods can be broadly classified into two groups. The first group of methods only need diffusion data that is acquired on a single shell in q-space (single b-value), e.g., using a fixed diffusion time and gradient strength but with varying gradient directions, see (Campbell et al., 2005; Cohen-Adad et al., 2011; Descoteaux et al., 2008; Gur et al., 2012; Michailovich and Rathi, 2010; Tournier et al., 2004; Tuch et al., 2002; Tuch, 2004). These methods are typically used to estimate the orientation of the underlying fiber bundles. The second group of methods use data acquired at multiple shells in q-space (several b-values), see e.g. (Cheng et al., 2010; Merlet and Deriche, 2013; Ning et al., 2014; Özarslan et al., 2009; Rathi et al., 2011). By reconstructing the continuous diffusion-weighted signal in q-space, these methods are usually capable of estimating the three dimensional EAP, thus providing more detailed anatomical information of the underlying tissue, see (Özarslan et al., 2013).

Given the large number of methods that have been proposed, it would be quite informative to know which reconstruction algorithm to use in a given scenario. It would also be important to know the b-values and the number of gradient directions that are most suitable for a reconstruction algorithm. For this purpose, a Monte Carlo simulation method was used in (Jones, 2004) to determine the optimum number of measurements for DTI. Recently, a HARDI challenge was held as part of the International Symposium in Biomedical Imaging (ISBI) conference¹, to determine the best analysis method to use in the case of single b-value data. However, the data used for this challenge was generated synthetically and realistic MR acquisition uncertainties and noise originating from the scanner can only be imperfectly incorporated using certain statistical models. A more reasonable strategy is to use MRI data scanned directly from a phantom so that both the signal and noise are similar to the in-vivo case while the underlying structure is also known, see (Fillard et al., 2011).

We realize that creating a tissue-based phantom as done in (Campbell et al., 2005) would be more ideal. But creating such phantoms for arbitrary angles is quite difficult, see (Campbell et al., 2005), as it requires significant expertise in proper handling of the tissue as well as access to ex-vivo tissue samples (which can be extremely challenging). Devising such a phantom with a specified fiber crossing itself is a challenging research problem. An alternative, but a less challenging way is to design a physical phantom using filaments of known size arranged in a desired manner as done in (Moussavi-Biugui et al., 2011) and (Fillard et al., 2011). In this work, we use this strategy to design a phantom and report some results from the challenge organized.

SPARC dMRI Challenge

Consequently, the SPArse Reconstruction Challenge (SPARC)² was organized to test the performance of various reconstruction algorithms where the data was acquired from a physical phantom that was placed in a standard clinical MR scanner. This paper elaborates the results of SPARC, which was part of the Computational Diffusion MRI workshop (CDMRI ’14) held at the Medical Image Computing and Computer Assisted Intervention Conference in 2014 (MICCAI 2014). This work presents several quantitative and qualitative results from 16 different algorithms that model the diffusion data from noisy single-shell or multi-shell measurements obtained from a physical phantom. Note that, this is a compilation of all results and discussions held as part of the challenge.

In the rest of the paper, we first provide some background on diffusion MRI, followed by the details about SPARC, the construction of the physical phantom and the different types of data sets acquired. Next, we provide a brief overview of all the reconstruction algorithms (as provided by the participating teams). In the experiments section, we provide a detailed comparison of all the methods on several quantitative measures, which is the most important contribution of the paper. In order to keep the evaluation procedure independent of any type of model, we used four quantitative measures, namely, 1) signal reconstruction error, 2) error in estimation of the discrete approximation to the return-to-origin probability, 3) angular error in the estimation of fiber crossing, and 4) percentage of voxels at which any particular method failed to detect the correct number of fibers. A high error in any of these measures would indicate high error either in signal decay along radial directions or in fiber orientation estimation. Thus, low error on all of these metrics is imperative for accurate estimation of the underlying tissue properties and brain connectivity. The measures derived from DTI, such as fractional anisotropy (FA), mean diffusivity (MD), were not used to compare the algorithms, since these measures are specific to a particular model. Thus, this challenge is not designed for clinicians who use these measures in clinical practice, but more for neuroscience studies. The paper concludes with discussions that provide very useful insights into the algorithms to use for a given data type. Further, we provide valuable advice to neuroscientists on the acquisition protocol to use, keeping in mind the appropriate analysis technique.

2. Diffusion MRI

In the narrow pulse setting (for the single-pulsed field gradient experiment), the ensemble average diffusion propagator (EAP) of water molecules, denoted by P(r), is related to the normalized diffusion-weighted signal E(q) ≜ S(q)/S(0) via the Fourier transform:

$$P(\mathit{r})={\int }_{\mathit{q}\in {\mathrm{ℝ}}^3}E(\mathit{q})\text{exp}(-i2\pi \mathit{q}·\mathit{r})d\mathit{q}$$ (1)

as given in (Stejskal and Tanner, 1965), where r is the displacement vector, q ∈ ℝ³ is an experimentally controlled parameter and S(q) is the diffusion-weighted signal measured at q. The vector q is defined as q = (2π)⁻¹γδG, where γ is the gyromagnetic ratio and δ is the duration of the diffusion sensitizing gradients whose magnitude and orientation are determined by the vector G. Alternatively, E can be written as a function of b-value and a unit vector u ∈ 𝕊², such that E(b, u) : ℝ⁺ × 𝕊² → [0, 1], where the b-value in single-pulsed field gradient experiments is defined as b = γ²δ²(Δ − δ/3) ‖G‖² s/mm² with Δ being the time between the diffusion-encoding pulses.

In practice, the normalized signal E(q) is measured at a discrete set of locations q₁, …, q_N in q-space. The continuous diffusion-weighted signal E(q) or other features of the diffusion process are usually estimated from the measurement vector s ≔ [E(q₁), …, E(q_N)]^T. If the points q₁, …, q_N are uniformly scattered on a single shell, i.e. with a fixed b value, then the measurements do not provide enough information about the decay of E(q) with q-value (radial decay). In this case, E(q) is either estimated using a 3-dimensional parametric Gaussian model or is only estimated on the sphere to provide angular features of the underlying fiber bundles. In a more general sampling scheme, the measurements are acquired on multiple shells in q-space (or multiple q-values as in diffusion spectrum imaging or Hybrid diffusion imaging in (Wu and Alexander, 2007)). In this case, many methods have been proposed to reconstruct the three-dimensional continuous function E(q). Then the fundamental relation given in Eq. (1) provides an estimate of the EAP P(r). The fiber orientations in this case are usually estimated via the diffusion orientation distribution function (ODF) which is defined as:

$$\mathrm{O}\mathrm{D}\mathrm{F}(\mathit{u})={\int }_0^{\infty }{r}^{2}P(r\mathit{u})dr,$$

where u is a vector on the unit sphere. The directions corresponding to the peaks of the ODF are the principal diffusion directions of the underlying fiber bundles.

3. The SPArse Reconstruction Challenge (SPARC)

A very open problem in diffusion MRI is, how many gradient directions and b-values are required to faithfully reconstruct or represent the diffusion data. For a specific model such as single tensor, Jones (2004) had determined that not more than 30 gradient directions are required. The field has moved-on to more advanced schemes such as high angular resolution diffusion imagin (HARDI) and multi-shell data. Consequently, it is imperative to know the number of measurements required and the methods to use for analysis that provide the lowest fitting error as well as recover the fiber orientation accurately. The scope of the SPARC dMRI challenge was to determine the best method that can, a). accurately represent the single-shell or multi-shell data (in terms of the lowest normalized mean square error), b). recover the fiber orientation accurately, with minimum number of measurements.

The goal was to keep the evaluation of the algorithms as general as possible, without using model-specific measures (obtained from single tensor, multi-tensor, NODDI, CHARMED, etc). Since there are a large number of models, it would not have been possible to do a fair comparison of these methods. Hence, we used the most general criteria for evaluating the algorithms, such as error in data fitting and fiber orientation.

SPARC was organized as part of the MICCAI 2014, as an open challenge for various researchers to participate and compare the results obtained on a single data set without any bias. Each participant was provided with three data sets that were acquired from a physical phantom using different acquisition parameters. The task for the participants was to use their preferred methods on any part of the provided data set to reconstruct the diffusion-weighted signal on a dense set of points in q-space and to estimate the number of fiber bundles at each voxel. If more than one fiber bundle was detected, each team was asked to report the angle between the crossing fibers.

The reconstructed signal for each method was quantitatively and qualitatively compared with a gold-standard data set which was acquired on a dense grid with multiple repetitions (the average of all acquisitions was considered the gold standard). By computing the error in the reconstructed signal, we could compare how well different algorithms represent the data. Lower error implied that the reconstructed signal was close to the gold standard. The estimated angle was also compared with the ground truth to evaluate the capability of each of these methods in resolving crossing fibers. Note that the gold-standard data set and the actual crossing angle were not released to the participants during the first phase of the challenge.

The comparison results were revealed at the time of the SPARC challenge during MICCAI 2014. Following the declaration of the results, the gold standard data set and ground-truth were released and participants were given one more opportunity to use this information to further fine-tune their algorithms. However, none of the teams updated any of the existing results, although two teams did send in additional results on data sets that they had not used for the challenge. Additionally, while the organizers’ team did not participate in the challenge, we have included the results of their algorithms obtained from the same data set as was released for the challenge. More details about the challenge can be found on the SPARC dMRI website (http://projects.iq.harvard.edu/sparcdmri/challenge). In the following sections, we provide a more detailed information about the physical phantom and the data sets used in the challenge.

3.1. Physical phantom

The physical phantom used in the challenge consisted of a spherical phantom as shown in Figure 1a with a crossing angle of 45°. The phantom was developed along the same lines as described in (Moussavi-Biugui et al., 2011). In particular, poly fibers (50 dtex, consisting of several 15 µm fibers, Filamentgarn TYPE 611, Trevira GmbH, Bobingen, Germany) were wound on a spherical polyamide spindle. The groove size of the phantom was 1 × 0.7 cm². Phantom winding was performed using a semiautomatic in-house constructed winding machine driven by electric motors, and the guidance of the thread was performed manually. The fibers were dipped in NaCl solution during the winding process to remove potential air bubbles between and within fibers. After the winding, the phantom was immediately placed into a bin and its position was fixed by casting it into agarose gel.

Figure 1.

(a) The spherical phantom used for data acquisition. (b) The colored FA image of the data set used in SPARC. (c) The baseline image of the data set.

After data acquisition, eddy-current distortion correction was performed using the FSL toolbox with sinc interpolation kernel, see (Smith et al., 2004). The FA value computed using all measurements in the single fiber region was estimated to be 0.8. To investigate the long term stability of the phantom, the scans were repeated after 3 months and the deviation in FA was less than 2% between the two acquisitions.

3.2. Data

The challenge data sets were acquired on a 3T Trio Siemens scanner with an 8-channel knee coil using GRAPPA (Griswold et al., 2002) with an acceleration factor of 2 and using the following acquisition parameters: in-plane resolution of 2 × 2 mm² with slice thickness of 7 mm, field of view of 256 × 128 mm². While the acquired data set consisted of a much larger field-of-view, we selected a relevant small sub-region of the central slice to be released for the challenge. A thick slice was used to ensure that the slice covered the entire thick groove of 7 mm. The slice (which contained the crossing fiber region) consisted of a total of 168 voxels including 93 single-fiber voxels and 75 two-fiber voxels. Other acquisition parameters were TE/TR = 141/3400 ms, δ ≈ Δ = 62 ms. For b = 5000 s/mm², the gradient strength was G = 20.79 mT/m. The data was acquired at b = {1000, 2000, 3000, 4000, 5000} s/mm². For each b-value shell, we obtained a separate acquisition for each of the following set of gradient directions K = {20, 30, 60, 81} with the directions uniformly spread on the unit sphere (Caruyer et al., 2013). Each data set also included one b = 0 image volume that was acquired using the same TE/TR. A total of 10 repetitions were acquired for K = 81 gradient directions. The average of these 10 scans was used as the gold standard data. Color coded FA and baseline images of the data set are shown in Figure 1b and 1c, respectively.

Figure 2a and 2b show the normalized signal in the gold-standard set in the single-fiber voxels and a two-fiber voxels, respectively. Note that this is the raw data, with no smoothing. As is evident, there is very little noise in this gold standard data set. On each b-shell, the energy of the signal is distributed around the great circle perpendicular to the fiber orientation, making the “shape” of the single fiber similar to a plate while the thickness of the plate reduces with increasing b-values. For two-fiber crossing, the shape of the signal on each b-shell is similar to the crossing of two plates while the plates are better separated at high b-values. Figure 2c and 2d show the single-fiber and two-fiber signal displayed along the radial direction (signal as a function of b-value), respectively, with each curve corresponding to the signal along one gradient direction. We note that the signal decreases monotonically along each gradient direction in the b-value range that is used in this experiment.

Figure 2.

(a) and (b) show the diffusion-weighted signal of the gold-standard data set in a single-fiber voxel and a two-fiber voxel, respectively. (c) and (d) is the corresponding signal plotted as a function of b-value with each curve being the signal along one gradient direction with a total of 81 directions.

3.3. The challenge data set

In SPARC, single-slice data sets with b = {1000, 2000, 3000} s/mm² and K = {20, 30, 60} gradient directions per shell together with one b = 0 image volume were provided to the participants to test their reconstruction algorithms. The average SNR of the provided data sets is given in Table 1. The SNR was computed as the ratio between mean signal and the standard deviation on each b-value shell. These SNR values are very similar to those obtained on a standard clinical scanner for in-vivo human acquisitions. Further, the average FA value for single fiber voxels was 0.8, which is close to that measured in the corpus-callosum of human brains (Stieltjes et al., 2006). Thus, the quality of the data set was very similar to that of in-vivo human brain data sets.

Table 1.

The average SNR of the phantom data sets with b-value in s/mm²

	b = 0	b = 1000	b = 2000	b = 3000
single-fiber voxels	51	21	13	10
two-fiber voxels	26	15	11	9

The participants were allowed to use any of the above b-value shells or a combination of them to reconstruct the data on a dense grid of q-space points. These points were provided to all participants and these were the same set of points on which the gold standard data was acquired, allowing quantitative comparison for each method. Depending on the type of data sampling scheme used, different tasks were assigned to the participants.

Multi-shell challenge

If the data from multiple b-shells was used in the reconstruction algorithm, the following were required as the output of the reconstruction algorithms: (1) The estimated diffusion-weighted signal over the same set of q-values as in the gold-standard data set, i.e. the reconstructed signal along 81 directions on the b-shells with b = 1000, 2000, 3000, 4000, 5000 s/mm², or a total of 405 q-space points; (2) The estimated number of fiber bundles at each voxel; (3) If multiple fiber bundles were detected, then the crossing angle between each of the fiber bundles was to be reported.

Single-shell challenge

If single-shell measurements were used, the following results were required: (1) The estimated diffusion-weighted signal on the same b-shell along the 81 directions as in the gold standard; (2) The number of fiber bundles at each voxel; (3) If multiple fiber bundles were detected in a voxel, then the crossing angle between the fiber bundles were to be reported.

4. Methods

In this section, we provide a brief overview of all the algorithms that the participants used for the challenge. For convenience, Table 2 tabulates the name of the algorithms, their abbreviations, the type of data sets they used and figures that contain their results. A detailed mathematical derivation of each method can be found from the references given in each section. Further, details about the specific parameter settings used by each method are also provided. These parameters and estimation methods were selected by the participants. Hence, similar performance can be expected from these algorithms if these parameter settings are used in other data sets with similar acquisition parameters.

Table 2.

List of the algorithms that participated in SPARC

Name	Abbreviation	Data type	Result
Constrained Spherical Deconvolution	CSD	multi-shell	Figure 3–7
		single-shell	Figure 8
Constrained Spherical Deconvolution with non-local means	CSDn	multi-shell	Figure 3–5
		single-shell	Figure 8
Constrained Spherical Deconvolution with non-local spatial and angular matching	CSDm	multi-shell	Figure 3–5
		single-shell	Figure 8
Sharpening Deconvolution Transform	SDT	multi-shell	Figure 3–7
		single-shell	Figure 8
Sharpening Deconvolution Transform with non-local means	SDTn	multi-shell	Figure 3–5
		single-shell	Figure 8
Sharpening Deconvolution Transform with non-local spatial and angular matching	SDTm	multi-shell	Figure 3–5
		single-shell	Figure 8
Spherical Finite Rate of innovation	SFR	multi-shell	Figure 3–7
		single-shell	Figure 9
Fiber Orientation Distribution	FOD	single-shell	Figure 8, 9
Spherical Fourier-Bessel expansion	SFB	multi-shell	Figure 5, 7
Simple Harmonic Oscillator	SHO	multi-shell	Figure 5, 7
Mean Apparent Propagator	MAP	multi-shell	Figure 3–7
Mean Apparent Propagator with non-local means	MAPn	multi-shell	Figure 3–6
Self-Adjusted Basis functions	SAB	multi-shell	Figure 3–7
		single-shell	Figure 8, 9
Directional Radial Basis functions	DRB	multi-shell	Figure 3–7
Spherical Ridgelets	SR	single-shell	Figure 8, 9
Spherical Ridgelets with Radial decay	SRR	multi-shell	Figure 3–7

4.1. Reconstruction algorithms

4.1.1. Spherical deconvolution based methods

Spherical deconvolution is a class of methods for reconstruction of multi-compartment diffusion-weighted signal using parametric or nonparametric models. It has been widely used to resolve fiber crossings and to estimate the orientation of fiber bundles, see e.g. (Jian and Vemuri, 2007; Tournier et al., 2004; Alexander, 2005; Descoteaux et al., 2009). In particular, the diffusion-weighted signal, denoted by E_b(u) ≔ E(b, u), is modeled as the convolution of a fiber orientation distribution function (fODF) F(u) with a response function R such that E_b(u) = ∫_‖v‖=1 R(u, v) F (v)dv. The response function kernel R(u, v) describes the diffusion-weighted signal for a single fiber bundle aligned to direction v. This kernel can be estimated using the signal from voxels that are believed to have only one fiber bundle, e.g. voxels with high fractional anisotropy (FA). By representing the fODF and the response function in the basis of spherical harmonics, the measurement vector s is represented as s = Y K f̂ where Y is the basis of spherical harmonics evaluated at the set of acquisition directions, K denotes the spherical harmonic representation of response function R, f̂ is the spherical harmonic decomposition of the fODF. Then f̂ is estimated so that the representation error s − Y K f̂ is small. The peaks of the estimated fODF are used to estimate the fiber-bundle orientations.

In an analogous manner, spherical deconvolution can also be applied to multi-shell acquisitions as E(q) = ∫_‖v‖=1 R(q, v) F (v)dv where R(q, v) denotes a 3D response function with orientation indicated by v, see e.g. (Gur et al., 2012; Jiao et al., 2011). In this case, Y is block diagonal and K is a vertical concatenation of multiple response functions corresponding to different shells. A tensor model is usually used for R(q, v) to estimate the continuous function E(q). Next, we describe the different variations of spherical-deconvolution methods that participated in the challenge.

Constrained Spherical Deconvolution (CSD)

In (Tournier et al., 2007), CSD was introduced for more robust performance by requiring the fODF to be non-negative. In SPARC, CSD was performed in conjunction with two denoising algorithms for data preprocessing: the non-local means, see (Descoteaux et al., 2008) and the non-local spatial and angular matching algorithms introduced in (St-Jean et al., 2014). We denote the original version as CSD and the two versions with the denoising procedures as CSD_n and CSD_m. Comparison of CSD with CSD_n and CSD_m is useful in understanding the contribution of each denoising technique.

In this challenge, spherical harmonics of order 4, 6 and 8 were used for the data sets with 20, 30 and 60 gradient directions (per b-shell), respectively. These experimental parameters were chosen based on the fact that a larger number of measurements may provide better accuracy. The eigenvalues for the tensorial response function R were set to λ₁ = 30 × 10⁻⁴, λ₂ = 2 × 10⁻⁴. For the CSD_n and CSD_m methods, the standard deviation of noise was estimated from the provided data using a patch of size 3 × 3 × 3. Other parameters settings were as given in (Descoteaux et al., 2008; St-Jean et al., 2014).

Sharpening Deconvolution Transform (SDT)

SDT was introduced in (Descoteaux et al., 2009) as a simple linear transformation on the spherical harmonic coefficients to provide a sharper fODF. In this challenge, SDT was also implemented with non-local means and spatial-and-angular matching denoising techniques. The three versions are denoted as SDT, SDT_n and SDT_m, respectively. The parameters needed for the algorithms were set to the same values as in CSD.

4.1.2. Spherical Finite Rate (SFR) of innovation and sparse regularization

In the SFR method introduced in (Deslauriers-Gauthier and Marziliano, 2013a,b), the spherical harmonic coefficient vector is approximated as $\hat{\mathit{f}}=Y_M^{*}\mathit{f}$, where Y_M denotes the basis matrix of spherical harmonics evaluated at M points on the unit sphere, * denotes the complex conjugate and f denotes a vector obtained by discretizing the fODF at the same set of M points on the unit sphere. Assuming f is nonnegative and sparse, the solution is obtained by minimizing the following:

$$\underset{\mathit{f}}{\text{min}}\{{\Vert \mathit{s}-YR{Y}_M^{*}\mathit{f}\Vert }_2+{\lambda }_1{\Vert \mathit{f}\Vert }_1+{\lambda }_2{\Vert Y_M^{*}\mathit{f}\Vert }_2\text{subject to}\mathit{f}\ge 0\}$$

with λ₁, λ₂ > 0. In this challenge, the parameters were set as λ₁ = λ₂ = 0.005 and the response function was constructed using a tensor model with the three eigenvalues set to {1700, 300, 300} × 10⁻⁶.

4.1.3. Fiber Orientation Distribution (FOD) from non-negative sparse recovery

In (Ghosh et al., 2013), the fODF corresponding to a single fiber bundle is modeled by a positive-semidefinite rank-1 high-order tensor C = c^⊗2n and is given by the polynomial C(u) ≔ ∑_i+j+k=2n (c · u)²ⁿ where the vector c ∈ ℝ³ is aligned with the direction of the underlying fiber bundle. For r fiber bundles, the fODF is given by a linear mixture: $F(\mathit{u})={\sum }_{i=1}^r{w}_i{C}_i(\mathit{u})$ with w_i ≥ 0. The response function is assumed to be a bipolar Watson function R(q, u) = exp(−bd_‖ (q · u)²) which is estimated from voxels with high FA (FA > 0.8) to ensure “single-fiber” configuration. The fODF is estimated by first discretizing the convolution integral with a selected set of points ${\{{\mathit{c}}_i\}}_{i=1}^{321}$ on the unit sphere and computing a least squares fit with non-negative constraints on the fODF coefficients, see (Ghosh et al., 2014). For this challenge, the order of the high order tensor used to represent the fODF was set to 24, i.e. 2n = 24.

4.1.4. Spherical Fourier-Bessel (SFB) expansion for multi-shell reconstruction

The SFB expansion of the diffusion-weighted signal in (Hosseinbor et al., 2013) is given by

$$E(q\mathit{u})=\sum _{n=0}^N\sum _{\ell =0}^L\sum _{m=-\ell }^{\ell }{a}_{n\ell m}{\varphi }_{n\ell m}(q\mathit{u}),$$

where ${\varphi }_{n\ell m}(q\mathit{u})=\sqrt{\frac{2}{\pi }}{j}_{\ell }(nq)Y_{\ell m}(\mathit{u})$, and j_ℓ is the Bessel function of the first kind and Y_ℓm(u) is the order ℓ degree m spherical harmonics. Then, the vector of measurements s is given by

$$\mathit{s}=\mathrm{\Psi }\mathrm{\Phi }\mathit{x}+\epsilon ,$$

where Φ is the basis matrix with each column being a function ϕ evaluated at a dense set of points in q-space, Ψ is a sub-sampling matrix and ε denotes the measurement noise. The coefficient x is obtained by solving

$$\underset{\mathit{x}}{\text{min}}{\Vert \mathit{s}-\mathrm{\Psi }\mathrm{\Psi }\mathit{x}\Vert }^2+\lambda {\Vert \mathit{x}\Vert }_1$$

with λ > 0. For this challenge, the participants set N = 4 and L = 16, which led to a total number of 1445 coefficients to be estimated.

Though SFB provides a continuous expression for the diffusion-weighted signal, in this challenge, the corresponding diffusion propagator was not computed for estimating the orientation of fiber bundles. Instead, the spherical deconvolution method was used to compute the fODF. In particular, the estimated diffusion-weighted signal was first evaluated along 81 directions on b-shells with b = 1000, 2000, …, 5000 s/mm². Then, spherical deconvolution was used to obtain the fODF from $f(\mathit{u})={\sum }_{k=1}^K{({\mathit{\alpha }}_k·\mathit{u})}^d$ with α_k ∈ ℝ³ being the unknown parameters corresponding to the fiber-bundle orientations. The parameters were estimated by solving a non-linear optimization problem that minimizes the squared-error between the estimated signal and that given by the convolution model. The solutions were obtained using the same method proposed in (Gur et al., 2012).

4.1.5. Simple Harmonic Oscillator (SHO) based reconstruction and estimation

The SHO method was introduced in (Özarslan et al., 2009) and was then modified in (Merlet and Deriche, 2013). Similar to SFB, it also represents the signal using a basis of three-dimensional functions that extends the spherical harmonic representation to a continuous 3D function along the radial directions. In particular, the signal is represented as

$$E(q\mathit{u})=\sum _{\ell =0,\text{even}}^N\sum _{n=\ell }^{(N+l)/2}\sum _{m=-\ell }^{\ell }{c}_{n\ell m}{\mathrm{\Phi }}_{n\ell m}(q\mathit{u})$$

with

$${\mathrm{\Phi }}_{n\ell m}(q\mathit{u})={\left[\frac{2(n-\ell )!}{{\zeta }^{\frac{3}{2}}\mathrm{\Gamma }(n+\frac{3}{2})}\right]}^{\frac{1}{2}}{\left(\frac{q^2}{\zeta }\right)}^{\frac{\ell }{2}}\text{exp}\left(\frac{-q^2}{2\zeta }\right)L_{n-\ell }^{\ell +\frac{1}{2}}\left(\frac{q^2}{\zeta }\right)Y_{\ell }^m(\mathit{u})$$

where $L_{\alpha }^{\beta }(·)$ denotes a generalized Laguerre polynomial and Γ(·) is the Gamma function. The parameter ζ is estimated from the signal mean diffusivity (MD) as ζ = 1/(8π²τMD), see (Merlet and Deriche, 2013). Then the ODF is computed analytically from the estimated coefficients.

In SPARC, the coefficients c_nℓm’s were estimated by solving a quadratic programming problem that minimized the fitting error together with a Laplace-Beltrami regularization term. The maximal basis order was set to be N = 8 which led to 95 basis functions to fit the signal. The EAP was enforced to be positive on a Cartesian grid of size 11 × 11 × 5 with maximal radial distance set to 30 × MD. This implementation is available as part of the Dipy (Diffusion Image in Python) package introduced in (Garyfallidis et al., 2014).

4.1.6. Mean Apparent Propagator (MAP) MRI

The MAP method introduced in (Özarslan et al., 2013) represents the diffusion-weighted signal by an anisotropic Gaussian modulated by a series of Hermite polynomials. Consider a tensor A′ and a rotation matrix R that diagonalizes A′ such that $A=RA\prime R^T=\text{Diag}(u_x^2,u_y^2,u_z^2)$. Then the diffusion-weighted signal is represented in the rotated frame defined by the rotation matrix R^T as

$$E(\mathit{q})=\sum _{n=0}^N\sum _{\begin{array}{c}\hfill n_1,n_2,n_3\hfill \\ \hfill n_1+n_2+n_3=n\hfill \end{array}}{c}_{n_1{n}_2{n}_3}{\mathrm{\Phi }}_{n_1{n}_2{n}_3}(A,\mathit{q})$$

with Φ(A, q) = ϕ_n1 (u_x, q_x)ϕ_n2(u_y, q_y)ϕ_n3 (u_z, q_z), q = [q_x, q_y, q_z]^T and

$${\varphi }_n(u,q)=\frac{i^{-n}}{\sqrt{2^{n}n!}}\text{exp}(-2{\pi }^2{q}^2{u}^2)H_n(2\pi uq)$$

where H_n(x) is the nth order Hermite polynomial. If A = 2Dt_d where D is the diffusion tensor as in DTI and t_d is the diffusion time, then Φ₀₀₀(q) corresponds to a Gaussian EAP. Similar to the 3D-SHO method, the EAP and ODF can be obtained analytically in terms of the representation coefficients, see (Özarslan et al., 2013). Moreover, this method can be used to obtain several important scalar statistics for studying the tissue microstructure.

In SPARC, two versions of MAP were implemented and submitted by two different research groups. In the first algorithm, the representation coefficients were estimated by solving a quadratic programming problem that minimized the fitting error of the signal together with a Laplacian regularization term similar to the method in (Fick, 2014). The signal at each voxel was fitted using a set of Hermite polynomials of maximum order 4, resulting in 22 coefficients to be estimated. The second algorithm also used the Laplacian regularization but with data pre-processed by a 3D non-local mean denoising method, see (Descoteaux et al., 2008). Hermite polynomials of order 6 were used for the data sets with 20 gradients per shell and order 8 for 30 and 60 gradients per shell. In the remainder of this paper, the two algorithms are denoted by MAP and MAP_n, respectively.

4.1.7. Self-Adjusted Basis functions (SAB)

The self-adjusted basis functions were devised based on the method introduced in (Ramirez-Manzanares et al., 2007), which represents the diffusion-weighted signal using a basis of anisotropic Gaussian functions centered at the origin. In SAB method, the diffusion-weighted signal is represented as

$$E(\mathit{q})=\sum _{i=1}^N{w}_i\text{exp}(-{\mathit{q}}^{T}R({\mathit{\theta }}_k){\overline{T}}_k({\mathit{\delta }}_k)R{({\mathit{\theta }}_k)}^T\mathit{q})$$

where R(θ_k) denotes a rotation matrix parameterized by θ_k = [θ_x, θ_y, θ_z]^T and T̄_k (δ_k)’s are anisotropic tensors with orientations uniformly distributed on the unit sphere with eigenvalues [λ_ℓ + δ_k,ℓ, λ_r + δ_k,r, λ_r + δ_k,r]. The parameters λ_ℓ and λ_r are the longitudinal and radial diffusivities that are fixed for all tensors. The representation coefficients and the parameters Θ ≔ {θ_k, k = 1, …, N}, Δ ≔ {δ₁, …, δ_N} are the unknowns that need to be estimated. If Θ and Δ are all set to zero, then the method reduces to the one used in (Ramirez-Manzanares et al., 2007). If only Δ = 0, it corresponds to the method in (Aranda et al., 2013).

For this challenge data set, the solution was obtained by solving the following non-convex optimization problem

$$\underset{\mathrm{\Delta }\mathrm{\Theta },\mathit{w}}{\text{min}}\{{\Vert \mathit{s}-\mathrm{\Phi }(\mathrm{\Theta },\mathrm{\Delta })\mathit{w}\Vert }^2\text{subject to}\mathit{w}\ge 0\}$$

where Φ(Θ, Δ) denotes the basis matrix that is parameterized by Θ and Δ. An alternate minimization method (alternately minimizing Δ, Θ and w) was used to solve the non-convex optimization problem. The initial diffusivity profile (λ₁, λ₂, λ₃) was designed such that λ₂ = λ₃ = λ₁/10 with λ₁ estimated from the diffusion-weighted signal, see (Ramirez-Manzanares et al., 2007).

4.1.8. Directional Radial Basis functions (DRB)

In (Ning et al., 2014, 2015), the diffusion-weighted signal is represented using anisotropic Gaussians with centers scattered on a dense grid in q-space. In particular, given a dense set of points {q̂₁, …, q̂_N}, the DRB method represents the diffusion-weighted signal as

$$E(\mathit{q})=\sum _{n=0}^N{w}_i\text{exp}(-{(\mathit{q}-{\hat{\mathit{q}}}_n)}^T{D}_k(\mathit{q}-{\hat{\mathit{q}}}_n))$$

where q̂₀ = 0, D₀ is the diffusion tensor as in DTI, D_k for k ≥ 1 have the same eigenvectors as D₀ with eigenvalues [λ₀, λ₁, λ₂] such that λ₀ ≥ λ₁ = λ₂. According to the theory of radial basis functions, the off-centered Gaussian functions can be used for approximating any function if the set of center points is dense enough and the number of measurements is above some critical number, see (Buhmann, 2003). Moreover, the corresponding EAP, ODF and several important scalar statistics all have analytical expressions in terms of the basis coefficients.

In SPARC, the centers {q̂₁, …, q̂_N} were pre-selected on the b-shells with b = 2000, 4000 s/mm² with N = 162. The eigenvalues of the tensors D_k with k ≥ 1 were set to λ₀ = 0.0011 mm²/s and λ₁ = λ₂ = 0.0006 mm² /s. The representation coefficients were estimated by solving a quadratic program with numerical constraints to enforce the signal to be positive and monotonically decreasing along 81 gradient directions on the b-value shells b = {1000, 2000, …, 8000} s/mm².

4.1.9. Spherical Ridgelets (SR) for single-shell reconstruction

SR was introduced in (Michailovich and Rathi, 2010) as a frame for representing L₂ functions on the unit sphere. This frame is devised based on the fundamental theory of wavelets. In particular, the diffusion-weighted signal on a b-shell E_b(u) = E(b, u) is represented as

$$E_b(\mathit{u})=\sum _{m=-1}^M\sum _{n=1}^{N_m}{c}_{mn}{\varphi }_{m,{\mathit{v}}_n}(\mathit{u})$$

where ϕ_m,v (u) denotes a spherical ridgelet function at the mth resolution and the unit vector v corresponds to the orientation of the basis function. The expression for ϕ_m,v (u) is given by

$${\varphi }_{m,\mathit{v}}(\mathit{u})=\frac{1}{2\pi }\sum _{n=0}^{\infty }\frac{2n+1}{4\pi }{\lambda }_n({\kappa }_{j+1}(n)-{\kappa }_j(n))P_n(\mathit{u}·\mathit{v})$$

where ${\kappa }_j(x)=\text{exp}\{-\rho \frac{x}{2^j}(\frac{x}{2^j}+1\left)\right\}$ for j ≥ 0, κ₋₁ = 0, P_n denotes the Legendre polynomial of order n, ${\lambda }_n=2\pi {(-1)}^{\frac{n}{2}}\frac{1.3\dots (n-1)}{2.4\dots n}$ if n is even and λ_n = 0 otherwise.

In this challenge, the representation coefficients were estimated by solving an ℓ₁-norm regularization problem as in (Michailovich and Rathi, 2010). The algorithm introduced in (Asif and Romberg, 2010) was used for obtaining the coefficients with the weighting coefficient for the ℓ₁-norm set to 0.03.

4.1.10. Spherical Ridgelets with Radial decay (SRR)

The method in (Rathi et al., 2014) represents the multi-shell diffusion-weighted signal using spherical ridgelets introduced in (Michailovich and Rathi, 2010) on each shell and enforces the signal to be monotonically decreasing along gradient directions. In particular, the diffusion-weighted signal is represented as

$$E(b_i,{\mathit{u}}_j)=\sum _{m=-1}^M\sum _{n=1}^{N_m}{c}_{mn}^i{\varphi }_{m,{\mathit{v}}_n}({\mathit{u}}_j){(1+b_i^{{\alpha }_j})}^{-{\beta }_j}$$

where ϕ_m,v (u) denotes a spherical ridgelet function at the mth resolution with orientation along v. Along each gradient direction u, the signal is modeled as (1 + b^α)^−β with α, β ≥ 0 being the unknown parameters to be estimated from measurements. In this challenge, the parameters $c_{mn}^i$, α_j and β_j were estimated by solving a convex optimization problem that regularizes the sparsity of spherical-ridgelet coefficients, monotonicity and the total-variation norm of signal using the method proposed in (Rathi et al., 2014). The weighting coefficients for the three regularization terms were all set to 1.

4.2. Comparison metrics

We used the following quantitative metrics to evaluate the performance of the reconstruction algorithms. Note that, we tried to use the most general metrics for evaluation and avoided using any model specific derived measure (such as FA or mean diffusivity), as it would naturally create a bias.

Normalized mean square error (NMSE) in signal reconstruction

Let ŝ_x denote the reconstructed signal vector at location x and let s_x,gold denote the corresponding gold-standard signal. The normalized mean square error (NMSE) in signal reconstruction is defined as

$$\mathrm{N}\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{|\mathrm{\Omega }|}\sum _{\mathit{x}\in \mathrm{\Omega }}\frac{{\Vert {\mathit{ŝ}}_{\mathit{x}}-{\mathit{s}}_{\mathit{x},\text{gold}}\Vert }^2}{{\Vert {\mathit{s}}_{\mathit{x},\text{gold}}\Vert }^2},$$

where Ω denotes the set of all voxel locations and |Ω| is the total number of voxels. Lower NMSE implies that the reconstructed signal is close to the “gold-standard” signal. For a more detailed comparison, we evaluated the NMSE in both the single-fiber region and multiple-fiber regions. For the multi-shell challenge, we separately computed the NMSE at low b-values of b = 1000, 2000, 3000 s/mm² (which was used in data fitting) and also for the extrapolated signal at b = 4000, 5000 s/mm². For computing several subtle measures such as return-to-origin probability and mean-squared-displacement, the signal over the entire q-space is used. Thus, accurate extrapolation of the signal beyond a b-value of 3000 is required for accurate estimation of these diffusion measures. Thus, a good reconstruction method will not only have lower NMSE in the region of the fit (b-values up-to 3000), but also in the extrapolated region of higher b-values.

NMSE of the return-to-the-origin probability (RTOP)

The RTOP is the value of the EAP at the origin, which is proportional to the inverse of the average pore volume as shown in (Özarslan et al., 2013). From Eq. 1, it is also equal to the integral of all the signals in q-space. Since not all the methods that participated the multi-shell challenge provided analytical expressions for the estimated diffusion-weighted signal, we took the sum of the estimated signal along 81 directions on the 5 b-shells as a numerical approximation of RTOP. Then the NMSE of RTOP is defined as

$$\mathrm{N}\mathrm{M}\mathrm{S}\mathrm{E}\text{of}\mathrm{R}\mathrm{T}\mathrm{O}\mathrm{P}=\frac{1}{|\mathrm{\Omega }|}\sum _{\mathit{x}\in \mathrm{\Omega }}\frac{|\mathrm{R}\mathrm{T}\mathrm{O}{\mathrm{P}}_{\mathit{x},\text{est}}-\mathrm{R}\mathrm{T}\mathrm{O}{\mathrm{P}}_{\mathit{x},\text{gold}}|}{\mathrm{R}\mathrm{T}\mathrm{O}{\mathrm{P}}_{\mathit{x},\text{gold}}},$$

where RTOP_x,est and RTOP_x,gold denote the RTOP at location x of the estimated data set and the gold-standard data set, respectively. Similar to NMSE, we also evaluated the NMSE of RTOP in both, the single-fiber region and the multi-fiber region. Hence, a good reconstruction method that provides correct tissue microstructure should have small error in the estimation of RTOP.

Estimated angle (EA)

In general, the reconstruction methods estimate the number of fiber bundles by counting the number of peaks in the estimated ODF (or fODF) that are higher than a threshold selected by the participants. If multiple fiber bundles were detected, the reconstruction methods also reported the crossing angle. If exactly two fiber-bundles were detected, the estimated angle was compared with the ground truth value of 45°. In particular, let Ω̂₂ denote the set of voxels where two fibers were detected and let Ω₂ denote the gold-standard set of two-fiber voxels. For each voxel x ∈ Ω̂₂, let a_x denote the estimated angle. The average and standard deviation for the set of angles {a_x s.t. x ∈ Ω̂₂ ∩ Ω₂} was computed to evaluate the reconstruction algorithms.

Percentage of false peaks (PFP)

The crossing angle was computed only at those locations where two peaks were detected. However, in many cases, the recovered signal may underestimate or overestimate the number of peaks. Thus, the percentage of false peaks (PFP) was used to evaluate the accuracy of peak detection with respect to the gold standard as follows:

$$\mathrm{P}\mathrm{F}\mathrm{P}=\frac{1}{|\mathrm{\Omega }|}\sum _{\mathit{x}\in \mathrm{\Omega }}ℬ(n_{\mathit{x}}-{\hat{n}}_{\mathit{x}})$$

where n_x and n̂_x denote the number of peaks in the gold standard and the estimated result at voxel x, respectively.

5. Results

5.1. Multi-shell results

In this section, we present the multi-shell reconstruction results using data sets with 20, 30 and 60 gradients on each of the three b-shells with b = 1000, 2000, 3000 s/mm². Hence, the three data sets had a total of 60, 90 and 180 measurements, respectively. We also note that some of the participants did not provide results for all the three data sets.

5.1.1. 60 measurements: 20 directions on 3 b-shells

Figure 3 shows a comparison of the results for 12 reconstruction algorithms using a total of 60 measurements scattered on three b-shells along 20 directions. The first 7 algorithms are different versions of the spherical-deconvolution method and the last 5 algorithms use different basis functions for representing the 3D diffusion-weighted signal. Figure 3a represents the estimated angle with mean and standard deviation shown by the blue bar and red line, respectively. We observe that the three versions of CSD and three versions of SDT have similar but very biased estimates of the crossing angle with the average error being in the range of [30, 45]°. On the other hand, angular error for MAP_n, DRB and SRR was only 2°, 4° and 2°, respectively. Though the MAP method used the same basis functions as MAP_n, its estimated angle is about 14° lower than the gold standard, indicating that either the larger number of basis functions or the pre-processing step of denoising the data has helped in better estimation of the crossing angle. SFR and SAB have similar performances with the angular error being about 7°.

Figure 3.

Method-wise comparison for multi-shell reconstruction algorithms using input data with a total of 60 measurements on 3 b-shells with b = 1000, 2000, 3000 s/mm² along 20 directions. All errors are relative to the gold standard data (5 b-shells with b = 1000, 2000, 3000, 4000, 5000 s/mm², 81 directions, 10 repetitions). The names and abbreviations of the methods are listed in Table 2.

Since the estimated angle was computed only in those voxels where two crossing fibers were detected by each method, the angular error does not provide a complete picture of the accuracy of each method. As such, it is imperative to determine if the correct number of peaks are detected in all voxels of data. Figure 3b shows the percentage of false positives (PFP) for each of the algorithms. The large values for the different versions of CSD and SDT implies that these methods failed to detect the crossing fibers in many voxels. Though MAP_n has smaller angular error than MAP, it has much higher PFP, implying that the denoising part of the algorithm might have introduced some bias to create artificial peaks. Thus the denoising method used to pre-process the data in MAP_n should be used with caution (with full knowledge of the pitfalls). The PFP for MAP, SAB, DRB and SRR are all about 10% or lower. The lowest value was for the SRR method with only 7% false positives.

Figure 3c shows the NMSE in signal reconstruction and Figure 3d and Figure 3e are the corresponding NMSE for signals in single-fiber voxels and multi-fiber voxels, respectively. The blue bars correspond to the NMSE obtained using the reconstructed signal on all the 5 b-shells from b = 1000 s/mm² to b = 5000 s/mm². The green and red bars correspond to the NMSE using signal interpolated on the b-shells with b = 1000, 2000, 3000 s/mm² and signal extrapolated on b-shells with b = 4000, 5000 s/mm², respectively. A general observation is that the NMSE for the extrapolated signal is higher than that at low b-values. The different versions of CSD and SDT algorithms along with SFR and MAP_n have high NMSE, especially in the extrapolated region. On the contrary, the NMSE for MAP at high b-values was only about 5%, and the lowest error was for the SRR method. In particular, the small NMSE at high b-values indicates that it has correctly modeled the decay of signal along radial directions. We also observe that MAP, DRB and SRR have lower NMSE in the two-fiber voxels than in single-fiber voxels.

Figure 3f shows the NMSE in the estimation of RTOP and Figure 3g and Figure 3h are the corresponding errors in the estimation of RTOP in single-fiber voxels and multi-fiber voxels, respectively. We note that SFR and the different versions of CSD and SDT algorithms still have very high error especially in multiple-fiber voxels. MAP, DRB and SRR are still the three best methods whose NMSE is lower than 1% in both single-fiber and multiple-fiber voxels. In general, the error in the estimation of RTOP follows a similar pattern as the NMSE in the estimation of the signal.

5.1.2. 90 samples: 30 directions on 3 b-shells

Figure 4 shows the comparison results for the same set of methods with a total number of 90 measurements scattered on 3 b-shells along 30 directions. In general, all the methods show better performance as more measurements are used in the estimation. In particular, in Figure 4a, the three versions of CSD have much lower angular error in the range of [0, 5]°. On the other hand, the three versions of SDT still show a large angular error of about 30°. SFR, MAP_n, SAB, DRB and SRR still show excellent performance, but the estimated angle for MAP has an error of 13°. In Figure 4b, the PFP for the three versions of CSD is about 15%, indicating that in 15% of the voxels, the number of peaks were incorrectly estimated even with 90 measurements. DRB and SRR have the lowest values of PFP of about 3% and 4%, respectively. The PFP for MAP_n is now significantly lower from 26% in Figure 3b to 8% in Figure 4b. SRR still gives the lowest NMSE among all the methods, while signal fitting error for MAP and DRB are also very low and comparable to the SRR method. MAP, DRB and SRR also have the lowest NMSE of RTOP among all the methods.

Figure 4.

Method-wise comparison for multi-shell reconstruction algorithms using input data with a total of 90 measurements on 3 b-shells with b = 1000, 2000, 3000 s/mm² along 30 directions. All errors are relative to the gold standard data (5 b-shells with b = 1000, 2000, 3000, 4000, 5000 s/mm², 81 directions, 10 repetitions). The names and abbreviations of the methods are listed in Table 2.

5.1.3. 180 samples: 60 directions on 3 b-shells

Figure 5 shows the comparison results for 14 different algorithms with a total of 180 measurements scattered on 3 b-shells along 60 directions. Using this large set of measurements, the PFP for all the three versions of CSD is lower than 10%. Though the average estimated angle for SDT is close to the ground truth, the large variation in the estimated angle and the high false positive rate indicates that it requires even more measurements for proper estimation. The largest angular error is for the SFB method, with the SDT method having the highest percentage of false positives. Note that the results for the SHO method are only available for this data set, and it seems to have low angular error but quite high PFP (about 18%). The estimated angle for MAP is still about 10° lower, though it has the lowest PFP and very low NMSE of signal and NMSE of RTOP. The performance of SAB in general is similar to the results shown in Figure 4e, but the angular error is larger despite an increase in the number of samples, indicating the effect of noise on the performance of this algorithm. DRB and SRR consistently have extremely low angular error (less than 1°), small PFP, low NMSE in the estimation of signal and low NMSE of RTOP. Thus, the performance of these two methods seem to be quite robust to the number of measurements, as will become more clear in the next section.

Figure 5.

Method-wise comparison for multi-shell reconstruction algorithms using input data with a total of 180 measurements on 3 b-shells with b = 1000, 2000, 3000 s/mm² along 60 directions. All errors are relative to the gold standard data (5 b-shells with b = 1000, 2000, 3000, 4000, 5000 s/mm², 81 directions, 10 repetitions). The names and abbreviations of the methods are listed in Table 2.

5.1.4. Within-method comparison

For a better understanding of the performance of each method with different numbers of measurements, we plot within-method error metrics for all different input data sets in Figure 6. As was shown in Figures 3 to 5, the three versions of CSD have very similar performance, and in this section we only present the result for one CSD method as a representative to avoid redundancy. Similarly, SDT is shown as a representative for the three different versions. Since SFB and SHO only have results using the data set with 180 measurements, their within-method comparison is not presented. In each sub-figure in Figure 6, the upper, middle and lower panels are the estimated angle, PFP, and NMSE respectively, with the x-axis being the total number of measurements. We observe that, overall, the PFP and angular error decreases as the number of measurements increase. In particular, increasing the total number of measurements from 60 to 90 usually leads to significant improvements in PFP for all methods. There is no significant improvement by increasing the total number of measurements from 90 to 180, indicating that a robust multi-shell acquisition scheme is to use measurements along 30 directions on three b-shells. However, DRB and SRR methods have low errors for all metrics even with 60 measurements, while MAP works quite well in terms of signal fitting and PFP. We also note that a more economical scheme may be possible for some reconstruction methods, e.g. using measurements along 30 directions on two b-shells. However, it is beyond the scope of this work to explore the smallest number of acquisitions that can be robustly used for each method.

Figure 6.

Comparison metrics (estimated angle, percentage of false peaks, and NMSE in estimation of signal) for each multi-shell reconstruction algorithms using input data with a total number of 60, 90, 180 measures shown as the x axis. All errors are relative to the gold standard data (5 b-shells with b = 1000, 2000, 3000, 4000, 5000 s/mm², 81 directions, 10 repetitions). The names and abbreviations of the methods are listed in Table 2.

5.1.5. Radial profile of the reconstructed signals

To better understand why MAP performs poorly in terms of angular error while having very low NMSE, we look at the signal estimation error as a function of the b-value. Figure 7 presents the estimated signal and the corresponding estimation error for 9 methods using 180 samples scattered on three b-values along 60 directions. Each 2 × 2 sub-figure of Figure 7 corresponds to one method. The left column of each sub-figure shows the average signal in single-fiber and multi-fiber voxels, while the signal reconstruction error is shown in the second column (Figure 2c and 2d). We note that the y-axis of the error plots are scaled differently for better visualization.

Figure 7.

The estimated signal for single-fiber and multi-fiber voxels for 9 methods using input data with a total number of 180 samples from 3 b-shells with b = 1000, 2000, 3000 s/mm², and the corresponding estimation error compared with the gold-standard signal shown in Figure 2. The color-coded curves are the signal along 81 gradient directions. The names and abbreviations of the methods are listed in Table 2.

We observe that the reconstructed signal for all methods (except SFB) decreases monotonically with b-value (as expected). Moreover, the reconstructed signal for CSD, SDT and SFR have similar radial profiles. For MAP though, (see Figure 7f) the error in signal estimation for the multi-fiber voxels is quite different for different gradient directions causing higher angular error despite the overall NMSE being low. The estimation errors for DRB and SRR are quite low (note the range on the y-axis), but share similar feature in the extrapolation region. Though the error is slightly large at high b-values, the magnitude of error is still quite small, about 0.05, which is much lower than other methods.

5.2. Single-shell results

For the single-shell challenge, most of the participants used the measurements with b = 2000 s/mm² or b = 3000 s/mm². On each b-shell, there were three data sets available that had 20, 30 and 60 gradient directions, respectively. Hence there were a total of 6 data sets used for comparison. In the following, we divide results into two groups based on the b-values of the data. We note that not all the teams provided results for all the 6 scenarios.

5.2.1. Single-shell reconstruction with b = 2000 s/mm²

Figure 8 shows the results for 9 reconstruction methods at b = 2000 s/mm². The three versions of CSD have very similar performances as in the multi-shell case. Using 60 measurements, these three methods have low angular error and false positives (PFP) but high NMSE, indicating the independence between angular performance and NMSE for single-shell reconstructions. Again, the three versions of SDT show poor performance compared to other methods. FOD has low angular error, but higher NMSE, indicating higher error in estimation of mean diffusivity (which depends on the signal strength). FOD, SAB and SR all have low PFP and small angular error, but FOD has higher NMSE than SAB and SR.

Figure 8.

Comparison results of 8 reconstruction algorithms using input data measurements on the b-shell with b = 2000 s/mm², and varying numbers of samples. All errors are relative to the gold standard data (with b = 2000 s/mm², 81 directions, 10 repetitions). The names and abbreviations of the methods are listed in Table 2.

To summarize, with 60 measurements, the CSD methods have the best angular accuracy but lower quality of signal fit. This implies that, these methods could perform quite well for tractography applications, but computing diffusion measures such as mean-diffusivity would incur more error compared to other methods such as SAB, FOD or SR. Overall, the SAB, FOD and SR methods seem to perform quite well with 30 measurements.

5.2.2. Single-shell reconstruction with b = 3000 s/mm²

Figure 9 shows the results for 3 reconstruction methods at b = 3000 s/mm². Though SFR has high NMSE, its has low angular error and low PFP. The angular error for FOD is reduced to 3° with 20 samples, but the PFP and NMSE are both higher from the case at b = 2000 s/mm². This implies that FOD is perhaps more sensitive to noise. SR seems to be the best method in this category with lower PFP for 20 measurements compared with the result at b = 2000 s/mm². We should note that, none of the methods in the single shell category used an isotropic compartment to model the signal, which could be a significant contributor to the signal at low b-values. Thus, most of the single shell methods analyzed above perform well only at higher b-values.

Figure 9.

Comparison results of 3 single-shell reconstruction algorithms using input data measurements on the b-shell with b = 3000 s/mm². All errors are relative to the gold standard data (with b = 3000 s/mm², 81 directions, 10 repetitions). The names and abbreviations of the methods are listed in Table 2.

6. Conclusion and Recommendations

In this paper, we compared the performance of 16 different algorithms for reconstructing diffusion MRI data based on data sets acquired from a physical phantom with known crossing angle. Based on these results, the following conclusions can be drawn for multi-shell reconstruction methods:

1. The pre-processing methods such NLM (non-local means) and SAM (non-local Spatial and Angular Matching) did not show any significant improvement in the performance of CSD and SDT based methods.

2. While MAP performed very well in terms of the NMSE (fitting error) and PFP (percentage of false positive peaks), it failed to estimate the crossing angle correctly (in all cases). Thus, MAP can be used to accurately estimate diffusion properties such as return-to-origin probability, and mean-squared displacement etc., but should not be used for tractography (in its current form). Using the NLM pre-processing and a larger number of basis functions helps the method MAP_n to better estimate the crossing angle, but it is at the cost of increased false positives and higher NMSE (in signal estimation and RTOP computation).

3. There was significant improvement in the performance of most methods by increasing the number of measurements from 60 to 90, although going to 180 did not seem to have much added advantage.

4. While the SAB method performed very well for 60 measurements, its performance did not improve significantly with increasing number of measurements.

5. The DRB and SRR methods seemed to perform quite well even with 60 measurements, with very low errors on all metrics. In particular, the angular error, NMSE and percentage of false peaks for DRB were 3°, 1.8% and 9%, respectively. The corresponding values for SRR method were 2°, 1.2% and 7%. While the SRR method performed slightly better than DRB, yet the latter could be a more preferable method given that all diffusion measures can be analytically computed, see (Ning et al., 2014, 2015).

Overall, with about 60 to 90 measurements (which takes about 6–20 minutes of scan time depending on the acquisition method), the DRB or SRR method should be used to estimate the ensemble average propagator (EAP) and orientation of the fiber bundles (from the ODF). Alternatively, the MAP method could be used to compute diffusion measures, but not the ODF (for tract tracing).

The following are the conclusions for single-shell reconstruction methods:

1. As in the multi-shell case, the pre-processing methods such NLM and SAM did not substantially improve the performance of the CSD and SDT methods.

2. The angle between fiber bundles can be correctly estimated even if the signal estimation error (NMSE) is high. Thus, we recommend to quantitatively compare both angular error and NMSE of the signal.

3. A small number of measurements at high b-values could provide good estimate of fiber crossings. Using only 20 measurements at b = 3000 s/mm², the FOD, SFR and SR algorithms were quite accurate in their estimate of the number of peaks and the crossing angles of 45°, although the FOD method seemed more sensitive to noise as the angular error increased with more measurements.

4. At b = 2000 s/mm², the SAB and SR methods had the best performance, although the FOD method could also be used with slightly higher error in signal estimation.

If connectivity analysis is the main aim of a clinical neuroimaging study with very little time to scan the patients, then we recommend to acquire the data at a b-value of 3000 s/mm² with at-least 20 measurements. This data can then be analyzed using the FOD, SFR or SR methods. Note that, several other methods could also perform well in this case, yet from the results presented in this paper, it is difficult to recommend any of those methods given that only 3 methods reported the results for this scenario (b = 3000 s/mm²).

If, however, one would like to compute subtle measures of diffusion (such as, pore volume or pore cross-sectional area), then, a multi-shell method (such as DRB or SRR) with at-least 60 measurements spread over 3 b-shells (1000, 2000, 3000 s/mm²) should be preferred. While none of the methods in this challenge incorporated an isotropic diffusion fraction (at low b-values), such methods may provide better performances, see (Dell’Acqua et al., 2010; Schultz et al., 2010; Zhou et al., 2014). However, we recommend the readers to test their algorithms on this phantom and perform a similar analysis to ensure lower reconstruction errors.

We should note that numerous methods have been proposed to model diffusion MRI data, and a comparison of all such methods presents several practical difficulties (e.g., non-availability of publicly available code bases, the specific parameter settings used in each algorithm, etc.). Nevertheless, the data sets released as part of this challenge are publicly available at http://projects.iq.harvard.edu/sparcdmri/Challenge_Data, which will allow for testing of new algorithms. We hope this paper can serve as a benchmark for testing of future algorithms, which will be of great help to the neuroimaging community.

Highlights.

• The outcome of the SParse Reconstruction Challenge in MICCAI 2014 is presented.

• A total of 16 diffusion-MRI reconstruction algorithms from 9 teams were compared.

• Several quantitative results pertaining to each algorithm were presented.

• The conclusions provide guidelines for choosing algorithms in clinical applications.