A contrast-adaptive method for simultaneous whole-brain and lesion segmentation in multiple sclerosis

Abstract

Here we present a method for the simultaneous segmentation of white matter lesions and normal-appearing neuroanatomical structures from multi-contrast brain MRI scans of multiple sclerosis patients. The method integrates a novel model for white matter lesions into a previously validated generative model for whole-brain segmentation. By using separate models for the shape of anatomical structures and their appearance in MRI, the algorithm can adapt to data acquired with different scanners and imaging protocols without retraining. We validate the method using four disparate datasets, showing robust performance in white matter lesion segmentation while simultaneously segmenting dozens of other brain structures. We further demonstrate that the contrast-adaptive method can also be safely applied to MRI scans of healthy controls, and replicate previously documented atrophy patterns in deep gray matter structures in MS. The algorithm is publicly available as part of the open-source neuroimaging package FreeSurfer.

1. Introduction

Multiple sclerosis (MS) is the most frequent chronic inflammatory autoimmune disorder of the central nervous system, causing progressive damage and disability. The disease affects nearly half a million Americans and 2.5 million individuals world-wide (Goldenberg, 2012; Rosati, 2001), generating more than $10 billion in annual healthcare spending in the United States alone (Adelman et al., 2013).

The ability to diagnose MS and track its progression has been greatly enhanced by magnetic resonance imaging (MRI), which can detect characteristic brain lesions in white and gray matter (Bakshi et al., 2008; Blystad et al., 2015; García-Lorenzo et al., 2013; Lövblad et al., 2010). Lesions visualized by MRI are up to an order of magnitude more sensitive in detecting disease activity compared to clinical assessment (Filippi et al., 2006). The prevalence and dynamics of white matter lesions are thus used clinically to diagnose MS (Thompson et al., 2018), define disease stages and to determine the efficacy of a therapeutic regimen (Sormani, 2013). MRI is also an unparalleled tool for characterizing brain atrophy, which occurs at a faster rate in patients with MS compared to healthy controls (Azevedo et al., 2018; Barkhof et al., 2009) and, especially in deep gray matter structures and the cerebral cortex, has been shown to correlate with measures of disability (Geurts et al., 2012).

Although manual labeling remains the most accurate way¹ of delineating white matter lesions in MS (Commowick et al., 2018), this approach is very cumbersome and in itself prone to considerable intra- and inter-rater disagreement (Zijdenbos et al., 1998). Furthermore, manually labeling various normal-appearing brain structures to assess atrophy is simply too time consuming to be practically feasible. Therefore, there is a clear need for automated tools that can reliably and efficiently characterize the morphometry of white matter lesions, various neuroanatomical structures, and their changes over time directly from in vivo MRI. Such tools are of great potential value for diagnosing disease, tracking progression, and evaluating treatment. They can also help in obtaining a better understanding of underlying disease mechanisms, and to facilitate more efficient testing in clinical trials. Ultimately, automated software tools may help clinicians to prospectively identify which patients are at highest risk of future disability accrual, leading to better counseling of patients and better overall clinical outcomes.

Despite decades of methodological development (cf. García-Lorenzo et al., 2013 or Danelakis et al., 2018), currently available computational tools for analyzing MRI scans of MS patients remain limited in a number of important ways:

• Poor generalizability: Existing tools are often developed and tested on very specific imaging protocols, and may not be able to work on data that is acquired differently. Especially with the strong surge of supervised learning in recent years, where the relationship between image appearance and segmentation labels in training scans is directly and statically encoded, the segmentation performance of many state-of-the-art algorithms will degrade substantially when applied to data from different scanners and acquisition protocols (García-Lorenzo et al., 2013; Valverde et al., 2019), severely limiting their usefulness in practice.

• Dearth of available software: Despite the very large number of proposed methods, most algorithms are only developed and tested inhouse, and very few tools are made publicly available (Griffanti et al., 2016; Schmidt et al., 2012; Shiee et al., 2010; Valverde et al., 2017). In order to secure that computational methods will make a real practical impact, they must be accompanied by software implementations that work robustly across a wide array of image acquisitions; that are made publicly available; and that are open-sourced, rigorously tested and comprehensively documented.

• Limitations in assessing atrophy: There is a lack of dedicated tools for characterizing brain atrophy patterns in MS: many existing methods characterize only aggregate measures such as global brain or gray matter volume (Smeets et al., 2016; Smith et al., 2002) rather than individual brain structures, or require that lesions are pre-segmented so that their MRI intensities can be replaced with placeholder values to avoid biased atrophy measures (Azevedo et al., 2018; Battaglini et al., 2012; Ceccarelli et al., 2012; Chard et al., 2010; Gelineau-Morel et al., 2012; Sdika and Pelletier, 2009) (so-called lesion filling).

In order to address these limitations, we describe a new open-source software tool for simultaneously segmenting white matter lesions and 41 neuroanatomical structures from MRI scans of MS patients. An example segmentation produced by this tool is shown in Fig. 1. By performing lesion segmentation in the full context of whole-brain modeling, the method obviates the need to segment lesions and assess atrophy in two separate processing phases, as currently required in lesion filling approaches. The method works robustly across a wide range of imaging hardware and protocols by completely decoupling computational models of anatomy from models of the imaging process, thereby sidestepping the intrinsic generalization difficulties of supervised methods such as convolutional neural networks. Our software implementation is freely available as part of the FreeSurfer neuroimaging analysis package (Fischl, 2012).

Fig. 1.

Segmentation of white matter lesions and 41 different brain structures from the proposed method on T1w-FLAIR input. From left to right: sagittal, coronal, axial view. From top to bottom: T1w, FLAIR, automatic segmentation.

To the best of our knowledge, only two other methods have been developed for joint whole-brain and white matter lesion segmentation in MS. Shiee et al. (2010) model lesions as an extra tissue class in an unsupervised whole-brain segmentation method (Bazin and Pham, 2008), removing false positive detections of lesions using a combination of topological constraints and hand-crafted rules implementing various intensity- and distance-based heuristics. However, the method segments only a small set of neuroanatomical structures (10), and validation of this aspect was limited to a simulated MRI scan of a single subject. McKinley et al. (2019) use a cascade of two convolutional neural networks, with the first one skull-stripping individual image modalities and the second one generating the actual segmentation. However, the whole-brain segmentation performance of this method was only evaluated on a few structures (7). Furthermore, as a supervised method its applicability on data that differs substantially from its training data will necessarily be limited.

A preliminary version of this work was presented in Puonti and Van Leemput (2016). Compared to this earlier work, the current article employs more advanced models for the shape and appearance of white matter lesions, and includes a more thorough validation of the segmentation performance of the proposed method, including an evaluation of the whole-brain segmentation component and comparisons with human inter-rater variability.

2. Contrast-adaptive whole-brain segmentation

We build upon a method for whole-brain segmentation called Sequence Adaptive Multimodal SEGmentation (SAMSEG) that we previously developed (Puonti et al., 2016), and that we propose to extend with the capability to handle white matter lesions. SAMSEG robustly segments 41 structures from head MRI scans without any form of preprocessing or prior assumptions on the scanning platform or the number and type of pulse sequences used. Since we build heavily on this method for the remainder of the paper, we briefly outline its main characteristics here.

SAMSEG is based on a generative approach, in which a forward probabilistic model is inverted to obtain automated segmentations. Let D = (d₁ , …, d_I) denote a matrix collecting the intensities in a multi-contrast brain MR scan with I voxels, where the vector ${\mathbf{d}}_i={(d_{\mathrm{i}}^1,\dots ,d_N^i)}^T$ contains the intensities in voxel i for each of the available N contrasts. Furthermore, let l = (l₁ , …, l_I)^T be the corresponding labels, where l_i ∈ {1 , … K} denotes one of the K possible segmentation labels assigned to voxel i. SAMSEG estimates a segmentation l from MRI data D by using a generative model, illustrated in black in Fig. 2. According to this model, l is sampled from a segmentation prior p(l|θ_l), after which D is obtained by sampling from a likelihood function p(D|l, θ_d), where θ_l and θ_d are model parameters with priors p(θ_l) and p(θ_d). Segmentation then consists of inferring the unknown l from the observed D under this model. In the following, we summarize the segmentation prior and the likelihood used in SAMSEG, as well as the way the resulting model is used to obtain automated segmentations.

Fig. 2.

Graphical model of the proposed method. In black the existing contrast-adaptive whole-brain segmentation method SAMSEG (without lesion modeling), in blue the proposed additional components to also model white matter lesions. Shading indicates observed variables. The plate indicates I repetitions of the included variables, where I is the number of voxels.

2.1. Segmentation prior

To model the spatial configuration of various neuroanatomical structures, we use a deformable probabilistic atlas as detailed in Puonti et al. (2016). In short, the atlas is based on a tetrahedral mesh, where the parameters θ_l are the spatial positions of the mesh’s vertices, and p(θ_l) is a topology-preserving deformation prior that prevents the mesh from tearing or folding (Ashburner et al., 2000). The model assumes conditional independence of the labels between voxels for a given deformation:

$$p\left(\mathbf{l}|{\mathit{\theta }}_1\right)=\prod _{i=1}^{I}p\left(l_i|{\mathit{\theta }}_{\mathbf{l}}\right),$$

and computes the probability of observing label k at voxel i as

$$p\left(l_i=k|{\mathit{\theta }}_{\mathbf{l}}\right)=\sum _{j=1}^J{\alpha }_j^k{\psi }_j^i\left({\mathit{\theta }}_{\mathbf{l}}\right),$$ (1)

where $a_k^j$ are label probabilities defined at the J vertices of the mesh, and ${\psi }_j^i({\mathit{\theta }}_{\mathbf{l}})$ denotes a spatially compact, piecewise-linear interpolation basis function attached to the j^th vertex and evaluated at the i^th voxel (Van Leemput, 2009).

The topology of the mesh, the mode of the deformation prior p(θ_l), and the label probabilities ${\alpha }_k^j$ can be learned automatically from a set of segmentations provided as training data (Van Leemput, 2009). This involves an iterative process that combines a mesh simplification operation with a group-wise nonrigid registration step to warp the atlas to each of the training subjects, and an Expectation Maximization (EM) algorithm (Dempster et al., 1977) to estimate the label probabilities ${\alpha }_k^j$ in the mesh vertices. The result is a sparse mesh that encodes high-dimensional atlas deformations through a compact set of vertex displacements. As described in Puonti et al. (2016), the atlas used in SAMSEG was derived from manual whole-brain segmentations of 20 subjects, representing a mix of healthy individuals and subjects with questionable or probable Alzheimer’s disease.

2.2. Likelihood function

For the likelihood function we use a Gaussian model for each of the K different structures. We assume that the bias field artifact can be modelled as a multiplicative and spatially smooth effect (Wells et al., 1996). For computational reasons, we use log-transformed image intensities in D, and model the bias field as a linear combination of spatially smooth basis functions that is added to the local voxel intensities (Van Leemput et al., 1999). Letting θ_d collect all bias field parameters and Gaussian means and variances, the likelihood is defined as

$$p(\mathbf{D}|\mathbf{l},{\mathit{\theta }}_{\mathbf{d}})=\prod _{i=1}^{I}p({\mathbf{d}}_i|l_i,{\mathit{\theta }}_{\mathbf{d}}),$$

$$p({\mathbf{d}}_i|l_i=k,{\mathit{\theta }}_{\mathbf{d}})=\mathcal{N}({\mathbf{d}}_i|{\mathit{\mu }}_k+\mathbf{C}{\mathit{\varphi }}_i,{\mathbf{\Sigma }}_k),$$

$$\begin{array}{l}\mathbf{C}=\left(\begin{array}{c}{\mathbf{c}}_1^T\\ ⋮\\ {\mathbf{c}}_N^T\end{array}\right),{\mathbf{c}}_n=\left(\begin{array}{c}{c}_{n,1}\\ ⋮\\ c_{n,P}\end{array}\right),{\mathit{\varphi }}_i=\left(\begin{array}{c}{\varphi }_1^i\\ ⋮\\ {\varphi }_P^i\end{array}\right),\end{array}$$

where P denotes the number of bias field basis functions, ${\varphi }_p^i$ is the basis function p evaluated at voxel i, and c_n holds the bias field coefficients for MRI contrast n. We use a flat prior for the parameters of the likelihood: p(θ_d) ∝ 1.

2.3. Segmentation

For a given MRI scan D, segmentation proceeds by computing a point estimate of the unknown model parameters θ = {θ_d, θ_l}:

$$\widehat{\mathit{\theta }}=\arg \underset{\mathit{\theta }}{\max }p(\mathit{\theta }|\mathbf{D}),$$

which effectively fits the model to the data. Details of this procedure are given in Appendix A. Once $\widehat{\mathit{\theta }}$ is found, the corresponding maximum a posteriori (MAP) segmentation

$$\widehat{\mathbf{l}}=\arg \underset{\mathit{I}}{\max }p(\mathbf{l}|\mathbf{D},\widehat{\mathit{\theta }})$$

is obtained by assigning each voxel to the label with the highest probability, i.e., ${\widehat{l}}_i=\arg {\max }_k{\widehat{w}}_{i,k}$, where $0\le {\widehat{w}}_{i,k}\le 1$ are probabilistic label assignments

$$w_{i,k}=\frac{\mathcal{N}({\mathbf{d}}_i|{\mathit{\mu }}_k+{\mathbf{C}}_{{\mathit{\varphi }}_i},{\mathbf{\Sigma }}_k)p(l_i=k|{\mathit{\theta }}_{\mathbf{l}})}{{\mathbf{\Sigma }}_{k\prime =1}^K\mathcal{N}({\mathbf{d}}_i|{\mathit{\mu }}_{k\prime }+\mathbf{C}{\mathit{\varphi }}_i,{\mathbf{\Sigma }}_{k\prime })p(l_i=k\prime |{\mathit{\theta }}_{\mathbf{l}})}$$ (2)

evaluated at the estimated parameters $\widehat{\mathit{\theta }}$. It is worth emphasizing that, since the class means and variances {μ_k, Σ_k} are estimated from each target scan individually, the model automatically adapts to each scan’s specific intensity characteristics – a property that we demonstrated experimentally on several data sets acquired with different imaging protocols, scanners and field strengths in Puonti et al. (2016).

Our implementation of this method, written in Python with the exception of C++ parts for the computationally demanding optimization of the atlas mesh deformation, is available as part of the open-source package FreeSurfer². It segments MRI brain scans without any form of preprocessing such as skull stripping or bias field correction, taking around 10 minutes to process one subject on a state-of-the-art computer (measured on a machine with an Intel 12-core i7-8700K processor). As explained in Puonti et al. (2016), in our implementation we make use of the fact that many neuroanatomical structures share the same intensity characteristics in MRI to reduce the number of free parameters in the model (e.g., all white matter structures share the same Gaussian mean μ_k and variance Σ_k, as do most gray matter structures). Furthermore, for some structures (e.g., non-brain tissue) we use Gaussian mixture models instead of a single Gaussian. In addition to using full covariance matrices Σ_k, our implementation also supports diagonal covariances, which is currently selected as the default behavior.

3. Modeling lesions

In order to make SAMSEG capable of additionally segmenting white matter lesions, we augment its generative model by introducing a binary lesion map z = (z₁ , … , z_I)^T, where z_i ∈ {0, 1} indicates the presence of a lesion in voxel i. The augmented model is depicted in Fig. 2, where the blue parts indicate the additional components compared to the original SAMSEG method. The complete model consists of a joint (i.e., over both l and z simultaneously) segmentation prior p(l, z|h, θ_l), where h is a new latent variable that helps constrain the shape of lesions, as well as a joint likelihood p(D|l, z, θ_d, θ_les), where θ_les are new parameters that govern their appearance. In the following, we summarize the segmentation prior and the likelihood used in the augmented model, as well as the way the resulting model is used to obtain automated segmentations.

3.1. Segmentation prior

We use a joint segmentation prior of the form

$$p\left(l,z|h,{\mathit{\theta }}_{\mathbf{l}}\right)=p\left(\mathbf{z}|\mathbf{h},{\mathit{\theta }}_{\mathbf{l}}\right)p\left(l|{\mathit{\theta }}_{\mathbf{l}}\right),$$

where p(l|θ_l) is the deformable atlas model defined in Section 2.1, and

$$p\left(\mathbf{z}|\mathbf{h},{\mathit{\theta }}_{\mathbf{l}}\right)=\prod _{i=1}^{I}p\left(z_i|h,{\mathit{\theta }}_{\mathbf{l}}\right)$$

is a factorized model where the probability that a voxel is part of a lesion is given by:

$$p\left(z_i=1|{|h,\mathit{\theta }}_{\mathbf{l}}\right)=f_i\left(h\right){\rho }_i\left({\mathit{\theta }}_{\mathbf{l}}\right).$$

Here 0 ≤ f_i(h) ≤ 1 aims to enforce shape constraints on lesions, whereas 0 ≤ ρ_i(θ_l) ≤ 1 takes into account a voxel’s spatial location within its neuroanatomical context. Below we provide more details on both these components of the model.

3.1.1. Modeling lesion shapes

In order to model lesion shapes, we use a variational autoencoder (Kingma and Welling, 2013; Rezende et al., 2014) according to which lesion segmentation maps z are generated in a two-step process: An unobserved, low-dimensional code h is first sampled from a spherical Gaussian distribution $p\left(\mathbf{h}\right)=\mathcal{N}\left(\mathbf{h}\right|\mathbf{0},\mathbf{I})$, and subsequently “decoded” into z by sampling from a factorized Bernoulli model:

$$p_{\omega }\left(z|\mathbf{h}\right)=\prod _{i=1}^I{f}_i{\left(h\right)}^{z_i}{\left(1-f_i\left(h\right)\right)}^{\left(1-z_i\right)}.$$

Here f_i(h) are the outputs of a “decoder” convolutional neural network (CNN) with filter weights ω, which parameterize the model.

Given a training data set in the form of N binary segmentation maps $\mathcal{D}={\left\{{\mathbf{z}}^{(n)}\right\}}_{n=1}^N,$, suitable network parameters ω can in principle be estimated by maximizing the log-probability assigned to the data by the model :

$$\log p_{\omega }\left(\mathcal{D}\right)=\sum _{z\in \mathcal{D}}\log p_{\omega }\left(z\right),\mathrm{where}{p}_{\omega }\left(z\right)={\int }_h{p}_{\omega }\left(z_n|h\right)p\left(h\right)\text{d}h.$$

However, because the integral over the latent codes makes this intractable, we use amortized variational inference in the form of stochastic gradient variational Bayes (Kingma and Welling, 2013; Rezende et al., 2014). In particular, we introduce an approximate posterior

$$q_{\upsilon }\left(h|z\right)=\mathcal{N}\left(h|{\mu }_{\upsilon }\left(z\right),\mathrm{diag}\left({\mathbf{\sigma }}_{\upsilon }^2\left(z\right)\right)\right),$$

where the functions μ_υ(z) and ρ_υ(z) are implemented as an “encoder” CNN parameterized by υ. The variational parameters υ are then learned jointly with the model parameters ω by maximizing a variational lower bound ${\mathbf{\sum }}_{\mathbf{z}\in \mathcal{D}}{\mathcal{L}}_{\omega ,\upsilon }\left(z\right)\le \log p_{\omega }\left(\mathcal{D}\right)$ using stochastic gradient descent, where

$${\mathcal{L}}_{\omega ,\upsilon }\left(z\right)=-D_{KL}\left(q_{\upsilon }\left(h|z\right)||p\left(h\right)\right)+{\mathbb{E}}_{q_{\upsilon }\left(h|z\right)}\left[\log p_{\omega }\left(\mathbf{z}|h\right)\right].$$ (3)

The first term is the Kullback–Leibler divergence between the approximate posterior and the prior, which can be evaluated analytically. The expectation in the last term is approximated using Monte Carlo sampling, using a change of variables (known as the “reparameterization trick”) to reduce the variance in the computation of the gradient with respect to υ (Kingma and Welling, 2013; Rezende et al., 2014).

Our training data set $\mathcal{D}$ was derived from manual lesion segmentations in 212 MS subjects, obtained from the University Hospital of Basel, Switzerland. The segmentations were all affinely registered and resampled to a 1 mm isotropic grid of size 197×233×189. In order to reduce the risk of overfitting to the training data, we augmented each segmentation in the training data set by applying a rotation of 10 degrees around each axis, obtaining a total of 1484 segmentations. The architecture for our encoder and decoder networks is detailed in Fig. 3. We trained the model for 1000 epochs with mini-batch size of 10 using Adam optimizer (Kingma and Ba, 2014) with a learning rate of 1e-4. We approximated the expectation in the variational lower bound of Eq. (3) by using a single Monte Carlo sample in each step.

Fig. 3.

Lesion shape model architecture consisting of two symmetrical convolutional neural networks: (a) decoder network and (b) encoder network. The decoder network generates lesion segmentations from a low-dimensional code. Its architecture has ReLU activation functions (f(x) = max(0, x)) and batch normalization (Ioffe and Szegedy, 2015) between each deconvolution layer, with the last layer having a sigmoid activation function, ensuring 0 ≤ f_i (h) ≤ 1. The encoder network encodes lesion segmentations into a latent code. The main differences compared to the decoder network are the use of convolutional layers instead of deconvolutional layers and, to encode the mean and variance parameters, the last layer has been split in two, with no activation function for the mean and a softplus activation function (f(x) = ln(1 + e^x)) for the variance.

3.1.2. Modeling the spatial location of lesions

In order to encode the spatially varying frequency of occurrence of lesions across the brain, we model the probability of finding a lesion in voxel i, based on its location alone, as

$${\rho }_i\left({\mathit{\theta }}_{\mathbf{l}}\right)=\sum _{j=1}^J{\beta }_j{\psi }_j^i\left({\mathit{\theta }}_{\mathbf{l}}\right),$$

where lesion probabilities 0 ≤ β_j ≤ 1 defined in the vertices of the SAM-SEG atlas mesh are interpolated at the voxel location. This effectively defines a lesion probability map that deforms in conjunction with the SAMSEG atlas to match the neuroanatomy in each image being segmented, allowing the model to impose contextual constraints on where lesions are expected to be found.

We estimated the parameters β_j by running SAMSEG on MRI scans (T1-weighted (T1w) and FLAIR) of 54 MS subjects in whom lesions had been manually annotated (data from the University Hospital of Basel, Switzerland), and recording the estimated atlas deformations. The parameters β_j were then computed from the manual lesion segmentations by applying the same technique we used to estimate the ${\alpha }_j^k$ parameters in the SAMSEG atlas training phase (cf. Section 2.1).

3.2. Likelihood function

For the likelihood, which links joint segmentations {l, z} to intensities D, we use the same model as SAMSEG in voxels that do not contain lesion (z_i = 0), but draw intensities in lesions (z_i = 1) from a separate Gaussian with parameters θ_les = {μ_les, Σ_les}:

$$p\left(D|l,z,{\mathit{\theta }}_{\mathbf{d}},{\mathit{\theta }}_{les}\right)=\prod _{i=1}^{I}p\left(d_i|l_i,z_i,{\mathit{\theta }}_d,{\mathit{\theta }}_{les}\right),$$

where

$$p\left(d_i|l_i=k,z_i,{\mathit{\theta }}_d,{\mathit{\theta }}_{les}\right)=\{\begin{array}{cc}\mathcal{N}\left(d_i|{\mathit{\mu }}_{les}+C{\mathit{\varphi }}_i,{\mathbf{\Sigma }}_{les}\right)& \mathrm{if}{z}_i=1,\\ \mathcal{N}\left(d_i|{\mathit{\mu }}_k+C{\mathit{\varphi }}_i,{\mathbf{\Sigma }}_k\right)& \mathrm{otherwise}.\end{array}$$

In order to constrain the values that the lesion intensity parameters θ_les can take, we make them conditional on the remaining intensity parameters using a normal-inverse-Wishart distribution :

$$p\left({\mathit{\theta }}_{les}|{\mathit{\theta }}_{\mathbf{d}}\right)=\mathcal{N}\left({\mathit{\mu }}_{les}|{\mathit{\mu }}_{WM},{\nu }^{-1}{\mathbf{\Sigma }}_{les}\right)\mathrm{IW}\left({\mathbf{\Sigma }}_{les}|\kappa \nu {\mathbf{\Sigma }}_{WM},\nu -N-2\right).$$ (4)

Here the subscript “WM” denotes the white matter Gaussian and κ > 1 and ν ≥ 0 are hyperparameters in the model.

This choice of model is motivated by the fact that the normal-inverse-Wishart distribution is a conjugate prior for the parameters of a Gaussian distribution: Eq. (4) can be interpreted as providing ν “pseudo-voxels” with empirical mean μ_{W M} and variance κΣ_{W M} in scenarios where the lesion intensity parameters μ_les and Σ_les need to be estimated from data. In the absence of any such pseudo-voxels (ν = 0), Eq. (4) reduces to a flat prior on θ_les and lesions are modeled as a completely independent class. Although such models have been used in the literature (Guttmann et al., 1999; Kikinis et al., 1999; Shiee et al., 2010; Sudre et al., 2015) their robustness may suffer when applied to subjects with no or very few lesions, such as controls or patients with early disease, since there is essentially no data to estimate the lesion intensity parameters from. In the other extreme case, the number of pseudo-voxels can be set to such a high value (ν → ∞) that the intensity parameters of the lesions are fully determined by those of WM. This effectively replaces the Gaussian intensity model for WM in SAMSEG by a distribution with longer tails, in the form of a mixture of two Gaussians with identical means (μ_les ≡ μ_{W M}) but variances that differ by a constant factor (Σ_les ≡ κΣ_{W M} vs. Σ_{W M}). In this scenario, MS lesions are detected as model outliers in a method using robust model parameter estimation (Huber, 1981), another technique that has also frequently been used in the literature (Aït-Ali et al., 2005; Bricq et al., 2008; García-Lorenzo et al., 2011; Liu et al., 2009; Prastawa and Gerig, 2008; Rousseau et al., 2008; Van Leemput et al., 2001).

Based on pilot experiments on a variety of datasets (distinct from the ones used in the results section), we found that good results are obtained by using an intermediate value of ν = 500 pseudo-voxels for 1 mm³ isotropic scans, together with a scaling factor κ = 50. In order to adapt to different image resolutions, ν is scaled inversely proportionally with the voxel size in our implementation. We will visually demonstrate the role of these hyperparameters in constraining the lesion intensity parameters in Section 5.1.

3.3. Segmentation

As in the original SAMSEG method, segmentation proceeds by first obtaining point estimates $\widehat{\mathit{\theta }}$ that fit the model to the data, and then inferring the corresponding segmentation posterior:

$$p\left(l,z|D,\widehat{\mathit{\theta }}\right),$$

which is now jointly over l and z simultaneously. Unlike in SAMSEG, however, both steps are made intractable by the presence of the new variables θ_les and h in the model. In order to side-step this difficulty, we obtain $\widehat{\mathit{\theta }}$ through a joint optimization over both θ and θ_les:

$$\{\widehat{\mathit{\theta }},{\widehat{\mathit{\theta }}}_{les}\}=\arg \underset{\left\{\mathit{\theta },{\mathit{\theta }}_{les}\right\}}{\max }p\left(\mathit{\theta },{\mathit{\theta }}_{les}|D\right)$$

in a simplified model in which the constraints on lesion shape have been removed, by clamping all decoder network outputs f_i(h) to value 1. This simplification is defensible since the aim here is merely to find appropriate model parameters, rather than highly accurate lesion segmentations. By doing so, the latent code h is effectively removed from the model and the optimization simplifies into the one used in the original SAM-SEG method, with only minor modifications due to the prior p(θ_les|θ_d). Details are provided in Appendix B.

Once parameter estimates $\widehat{\mathit{\theta }}$ are available, we compute segmentations using the factorization

$$p(l,z|D,\widehat{\mathit{\theta }})=p(z|D,\widehat{\mathit{\theta }})p(l|,z,D,\widehat{\mathit{\theta }}),$$

first estimating z from $p(z|D,\widehat{\mathit{\theta }})$ (Step 1 below), and then plugging this into $p(l|,z,D,\widehat{\mathit{\theta }})$ to estimate l (Step 2):

Step 1: Evaluating $p(z|D,\widehat{\mathit{\theta }})$ involves marginalizing over both h and θ_les, which we approximate by drawing S Monte Carlo samples ${\{h^{\left(s\right)},{\mathit{\theta }}_{les}^{\left(s\right)}\}}_{s=1}^S$ from $p(h,{\mathit{\theta }}_{les}|D,\widehat{\mathit{\theta }})$:

$$\begin{gathered} p(z|D,\widehat{\mathit{\theta }})={\int }_{h,{\mathit{\theta }}_{les}}p(z|D,\widehat{\mathit{\theta }},h,{\mathit{\theta }}_{les})p(h,{\mathit{\theta }}_{les}|D,\widehat{\mathit{\theta }})dh,{\mathit{\theta }}_{les} \\ \simeq \frac{1}{S}\sum _{s=1}^{S}p(z|D,\widehat{\mathit{\theta }},h^{(s)},{\mathit{\theta }}_{les}^{(s)}). \end{gathered}$$

This allows us to estimate the probability of lesion occurrence in each voxel, which we then compare with a user-specified threshold value γ

$$p(z_i=1|d_i,\widehat{\mathit{\theta }})\gtrless \gamma$$

to obtain the final lesion segmentation ${\widehat{z}}_i$. Details on how we approximate $p(z_i=1|d_i,\widehat{\mathit{\theta }})$ using Monte Carlo sampling are provided in Appendix C.

Step 2: Voxels that are not assigned to lesion (${\widehat{z}}_i=0$) in the previous step are finally assigned to the neuroanatomical structure with the highest probability $p(l_i=k|z_i=0,d_i,\widehat{\mathit{\theta }}),$, which simply involves computing ${\widehat{l}}_i={{\displaystyle \arg \max }}_k{\widehat{w}}_{i,k}$ with ${\widehat{w}}_{i,k}$ defined in Eq. (2).

In agreement with other work (Aït-Ali et al., 2005; García-Lorenzo et al., 2011; Jain et al., 2015; Prastawa and Gerig, 2008; Shiee et al., 2010; Van Leemput et al., 2001), we have found that using known prior information regarding the expected intensity profile of MS lesions in various MRI contrasts can help reduce the number of false positive detections. Therefore, we prevent some voxels from being assigned to lesion (i.e., forcing ${\widehat{z}}_i=0$) based on their intensities in relation to the estimated intensity parameters ${\{{\widehat{\mathit{\mu }}}_k,{\widehat{\mathbf{\Sigma }}}_k\}}_{k=1}^K:$: In our current implementation only voxels with an intensity higher than the mean of the gray matter Gaussian in FLAIR and/or T2 (if these modalities are present) are considered candidate lesions.

Since estimating $p(z_i=1|d_i,\widehat{\mathit{\theta }})$ involves repeatedly invoking the decoder and encoder networks of the lesion shape model, as detailed in Appendix C, we implemented the proposed method as an add-on to SAMSEG in Python using the Tensorflow library (Abadi et al., 2015). Estimating $\widehat{\mathit{\theta }}$ has the same computational complexity as running SAM-SEG (i.e., taking approximately 10 minutes on a state-of-the-art machine with an Intel 12-core i7-8700K CPU), while the Monte Carlo sampling takes an additional 5 minutes on a GeForce GTX 1060 graphics card, bringing the total computation time to around 15 minutes per subject.

4. Evaluation datasets and benchmark methods

In this section, we describe four datasets that we will use for the experiments in this paper, including two taken from public challenges. We also outline two relevant methods for MS lesion segmentation that the proposed method is compared to in detail, as well as the metrics and measures used in our experiments.

4.1. Datasets

In order to test the proposed method and demonstrate its contrast-adaptiveness, we conducted experiments on four datasets acquired with different scanner platforms, field strengths, acquisition protocols and image resolution:

• MSSeg: This dataset is the publicly available training set of the MS lesion segmentation challenge that was held in conjunction with the MICCAI 2016 conference (Commowick et al., 2018). It consists of 15 MS cases from three different scanners, all acquired using a harmonized imaging protocol (Cotton et al., 2015). For each patient a 3D T1w sequence, a contrast-enhanced (T1c) sequence, an axial dual PD-T2-weighted (T2w) sequence and a 3D fluid attenuation inversion recovery (FLAIR) sequence were acquired. Each subject’s lesions were delineated by seven different raters on the FLAIR scan and, if necessary, corrected using the T2w scan. These delineated images were then fused to create a consensus lesion segmentation for each subject. Both raw images and pre-processed images (pre-processing steps: denoising, rigid registration, brain extraction and bias field correction – see Commowick et al. (2018) for details) were made available by the challenge organizers. In our experiments we used the pre-processed data, which required only minor modifications in our software to remove non-brain tissues from the model. We note that the original challenge also included a separate set of 38 test subjects, but at the time of writing this data is no longer available.

• Trio: This dataset consists of 40 MS cases acquired on a Siemens Trio 3T scanner at the Danish Research Center of Magnetic Resonance (DRCMR). For each patient, a 3D T1w sequence, a T2w sequence and a FLAIR sequence were acquired. Ground truth lesion segmentations were automatically delineated on the FLAIR images using Jim software³, and then checked and, if necessary, corrected by and expert rater at DRCMR using the T2w and MPRAGE images.

• Achieva: This dataset consists of 50 MS cases and 25 healthy controls acquired on a Philips Achieva 3T scanner at DRCMR. After a visual inspection of the images, we decided to remove 2 healthy controls from the dataset as they present marked gray matter atrophy and white matter hyperintensities. For each patient, a 3D T1w sequence, a T2w sequence and a FLAIR sequence were acquired. Ground truth lesion segmentations were delineated using the same protocol as the one used for the Trio dataset.

• ISBI: This dataset is the publicly available test set of the MS lesion segmentation challenge that was held at the 2015 International Symposium on Biomedical Imaging (Carass et al., 2017). It consists of 14 longitudinal MS cases, with 4 to 6 time points each, separated by approximately one year. Images were acquired on a Philips 3T scanner. For each patient, a 3D T1w sequence, a T2w sequence, a PDw sequence and a FLAIR sequence were acquired. Images were first preprocessed (inhomogeneity correction, skull stripping, dura stripping, again inhomogeneity correction – see Carass et al. (2017) for details), and then registered to a 1 mm MNI template. Each subject’s lesions were delineated by two different raters on the FLAIR scan, and, if necessary, corrected using the other contrasts. As part of the challenge, a training dataset of 5 additional longitudinal MS cases is also available, with the same scanner, imaging protocols and delineation procedure as the test dataset.

A summary of the datasets, with scanner type, image modalities and voxel resolution details, can be found in Table 1. For each subject all the contrasts were co-registered and resampled to the FLAIR scan for MSSeg, and to the T1w scan for Trio, Achieva and ISBI. This is the only preprocessing step required by the proposed method.

Table 1.

Summary of the datasets used in our experiments.

Dataset	Scanner	Modality	Voxel resolution [mm]	Subjects
MSSeg	Philips Ingenia 3T	3D FLAIR	0.74×0.74×0.7	5
		3D T1w	0.74×0.74×0.85
		3D T1c	0.74×0.74×0.85
		2D T2w	0.45×0.45×3
		2D PD	0.45×0.45×3
	Siemens Aera 1.5T	3D FLAIR	1.03×1.03×1.25	5
		3D T1w	1.08×1.08×0.9
		3D T1c	1.08×1.08×0.9
		2D T2w	0.72×0.72×4 (Gap: 1.2)
		2D PD	0.72×0.72×4 (Gap: 1.2)
	Siemens Verio 3T	3D FLAIR	0.5×0.5×1.1	5
		3D T1w	1×1×1
		3D T1c	1×1×1
		2D T2w	0.69×0.69×3
		2D PD	0.69×0.69×3
Trio	Siemens Trio 3T	2D FLAIR	0.7×0.7×4	40
		3D T1w	1×1×1
		2D T2w	0.7×0.7×4
Achieve	Philips Achieva 3T	3D FLAIR	1×1×1	73
		3D T1w	0.85×0.85×0.85
		3D T2w	0.85×0.85×0.85
ISBI	Philips 3T	2D FLAIR	0.82×0.82×2.2	14
		3D T1w	0.82×0.82×1.17
		2D T2w	0.82×0.82×2.2
		2D PDw	0.82×0.82×2.2

4.2. Benchmark methods for lesion segmentation

In order to evaluate the lesion segmentation component of the proposed method in detail, we compared it to two publicly available and widely used algorithms for MS lesion segmentation:

• LST-lga⁴ (Schmidt et al., 2012): This lesion growth algorithm starts by segmenting a T1w image into three main tissue classes (CSF, GM and WM) using SPM12⁵, and combines the resulting segmentation with co-registered FLAIR intensities to calculate a lesion belief map. A pre-chosen initial threshold κ is then used to create an initial binary lesion map, which is subsequently grown along voxels that appear hyperintense in the FLAIR image. We set κ to its recommended default value of 0.3, which was also used in previous studies (Mühlau et al., 2013; Rissanen et al., 2014).

• NicMsLesions⁶ (Valverde et al., 2017, 2019): This deep learning method is based on a cascade of two 3D convolutional neural networks, where the first one reveals possible candidate lesion voxels, and the second one reduces the number of false positive outcomes. Both networks were trained by the authors of the method on T1w and FLAIR scans coming from a publicly available training dataset of the MS lesion segmentation challenge held in conjunction with the MICCAI 2008 conference (Styner et al., 2008) (20 cases) and the MSSeg dataset (15 cases). This method was one of the top performers on the test dataset of the MICCAI 2016 challenge (Commowick et al., 2018), and one of the few methods for which an implementation is publicly available.

We note that both these benchmark methods are specifically targeting T1w-FLAIR input, whereas the proposed method is not tuned to any particular combination of input modalities.

Although we only compared our method in detail to these two benchmarks, many more good methods for MS lesion segmentation exist. We refer the reader to the MSSeg paper (Commowick et al., 2018), the ISBI challenge paper (Carass et al., 2017) and the ISBI challenge website⁷ to compare the reported performance further with other ones.

4.3. Metrics and measures

In order to evaluate the influence of varying the input modalities on the segmentation performance of the proposed method, and to assess segmentation accuracy with respect to that of other methods and human raters, we used a combination of segmentation volume estimates, Pearson correlation coefficients between such estimates and reference values, and Dice scores. Volumes were computed by counting the number of voxels assigned to a specific structure and converting into mm³ , whereas Dice coefficients were computed as

$${\mathrm{Dice}}_{X,Y}=\frac{2\cdot \left|X\cap Y\right|}{\left|X\right|+\left|Y\right|},$$

where X and Y denote segmentation masks, and | · | counts the number of voxels in a mask.

The proposed method and both benchmark algorithms produce a probabilistic lesion map that needs to be thresholded to obtain a final lesion segmentation. This requires an appropriate threshold value to be set for this purpose (variable γ in the proposed method). In order to ensure an objective comparison between the methods, we used a leave-one-out cross-validation strategy in which the threshold for each test image was set to the value that maximizes the average Dice overlap with manual segmentations in all the other images of the same dataset. For the reported performance of the methods on the ISBI dataset, the thresholds were tuned on the 5 training subjects that are part of the challenge instead.

5. Results

In this section, we first illustrate the effect of the various components of our model. We then evaluate how the proposed model adapts to different input modalities and acquisition platforms. Subsequently we compare the lesion segmentation performance of our model against that of the two benchmark methods, relate it to human inter-rater variability, and analyze its performance on the ISBI challenge data. Finally, we perform an indirect validation of the whole-brain segmentation component of the method.

Throughout the section we use boxplots to show some of the results. In these plots, the median is indicated by a horizontal line, plotted inside boxes that extend from the first to the third quartile values of the data. The range of the data is indicated by whiskers extending from the boxes, with outliers represented by circles.

5.1. Illustration of the method

In order to illustrate the effect of the various components of the method, here we analyze its behaviour when segmenting T1w-FLAIR scans of two MS subjects – one with a low and one with a high lesion load. Fig. 4 shows, in addition to the input data and the final lesion probability estimate $p(z_i=1|d_i,\widehat{\mathit{\theta }})$, also an intermediate lesion probability obtained with the simplified model used to estimate $\widehat{\mathit{\theta }}$, i.e., before the FLAIR-based intensity constraints and the lesion shape constraints are applied. From these images we can see that the lesion shape model and the intensity constraints help remove false positive detections and enforce more realistic shapes of lesions, especially for the case with low lesion load.

Fig. 4.

Illustration of how intensity constraints and the lesion shape model help reduce false positive lesion detections in the method. Top row: a subject with a low lesion load; Bottom row: a subject with a high lesion load. From left to right: T1w and FLAIR input; intermediate lesion probability obtained with the simplified model used to estimate $\widehat{\mathit{\theta }}$; mask of candidate voxels based on intensity alone (intensity higher than the mean gray matter intensity in FLAIR); and final lesion probability estimate $p(z_i=1|d_i,\widehat{\mathit{\theta }})$ produced by the method.

Fig. 5 analyzes the effect of the prior p(θ_les|θ_d) on the lesion intensity parameters θ_les for the two subjects shown in Fig. 4. When the lesion load is high, the prior does not have a strong influence, leaving the lesion Gaussian “free” to fit the data. However, when the lesion load is low, the lesion Gaussian is constrained to retain a wide variance and a mean close to the mean of WM, effectively turning the model into an outlier detection method for WM lesions. This behavior is important in cases when few lesions are present in the images, ensuring the method works robustly even when only limited data is available to estimate the lesion Gaussian parameters.

Fig. 5.

Illustration of the effect of the prior p(θ_les|θ_d) on the lesion intensity parameters, both in the case of a lesion load that is low (left, corresponding to the subject in the top row of Fig. 4) and high (right, corresponding to the subject in the bottom row of Fig. 4). The illustration is from the Monte Carlo sampling phase of the method: In each case, the value of the parameters of the lesion Gaussian is taken as the average over the Monte Carlo samples ${\left\{{\mathit{\theta }}_{les}^{(s)}\right\}}_{s=1}^S,$ and the points represent the resulting lesion posterior estimate $p(z_i=1|d_i,\widehat{\mathit{\theta }})$ in each voxel.

In order to analyze the effect of the lesion shape prior, we compared the lesion segmentation performance of the proposed method with that obtained when the shape prior was intentionally removed from the model (i.e., all the decoder network outputs f_i(h) clamped to value 1). For a fair comparison, the lesion threshold value γ was re-tuned to maximize performance for the method without shape prior, in the way described in Section 4.3. Table 2 summarizes the results across the MSSeg, Trio and Achieva datasets, for different ranges of lesion load. In addition to Dice scores, the table also reports results for precision and recall, defined as

$$precision=\frac{TP}{TP+FP}recall=\frac{TP}{TP+FN},$$

where TP, FP and FN count the true positive, false positive and false negative voxels compared to the manual segmentation. The results indicate that, although performance is unchanged for high lesion loads, for which segmentation is generally easier (Commowick et al., 2018), the lesion shape prior clearly improves segmentations in subjects with small and medium lesion loads.

Table 2.

Comparison in terms of lesion segmentation performance between the proposed method and a method where the lesion shape model was intentionally removed. Results are expressed in terms of mean ±standard deviation of Dice overlap, precision and recall for different ranges of lesion load. Lesion segmentations were computed across three different datasets (MSSeg, Trio and Achieva) on T1w-FLAIR input.

Lesion load	Dice		Precision		Recall
	Shape model	No shape model	Shape model	No shape model	Shape model	No shape model
(0, 2] [ml]	0.42 (±0.10)	0.38 (±0.10)	0.32 ( ±0.12)	0.24 (±0.07)	0.28 (±0.09)	0.24 (±0.07)
(2, 10] [ml]	0.50 (±0.13)	0.47 (±0.13)	0.37 (±0.13)	0.33 (±0.11)	0.34 (±0.12)	0.32 (±0.12)
(10, −) [ml]	0.70 (±0.11)	0.70 (±0.11)	0.62 (±0.20)	0.62 (±0.20)	0.55 (±0.12)	0.55 (±0.13)
(0, −) [ml]	0.57 (±0.16)	0.55 (±0.17)	0.46 (±0.20)	0.43 (±0.20)	0.42 (±0.16)	0.40 (±0.16)

In order to demonstrate that the model also works robustly in control subjects (with no lesions at all), and can therefore be safely applied in studies comparing MS subjects with controls, we further segmented T1w-FLAIR scans of the Achieva dataset, and computed the total volume of the lesions in each subject. The results are shown in Fig. 6; the volumes were 8.95±9.18 ml for MS subjects vs. 0.98±0.77 ml for controls. Although the average lesion volume for controls was not exactly zero, a visual inspection revealed that this was due to some controls having WM hyperintensities that were segmented by the method as MS lesions, which we find acceptable.

Fig. 6.

Difference between healthy controls (HC) and MS subjects in lesion volume, as detected by the proposed method on the Achieva dataset (23 HC subjects, 50 MS subjects, T1w-FLAIR input). Lines indicate means across subjects.

5.2. Scanner and contrast adaptive segmentations

In order to demonstrate the ability of our method to adapt to different types and combinations of MRI sequences acquired with different scanners, we show the method’s segmentation results along with the manual segmentations for a representative subset of combinations for one subject in the MSSeg (consensus as manual segmentation), the Trio and the Achieva datasets in Fig. 7. It is not feasible to show all possible combinations. For instance, mixing the 5 contrasts in the MSSeg dataset alone already yields 31 possible multi-contrast combinations. Nonetheless, it is clear that the model is indeed able to adapt to the specific contrast properties of its input scans. A visual inspection of its whole-brain segmentation component seems to indicate that the method benefits from having access to the T1w contrast for best performance. This is especially clear when only the FLAIR contrast is provided, as this visually degrades the segmentation of the white-gray boundaries in the cortical regions due to the low contrast between white and gray matter in FLAIR.

Fig. 7.

Contrast-adaptiveness of the proposed method to different combinations of input modalities. Segmentations are shown for one subject of the MSSeg (top row), the Trio (middle row) and the Achieva MS (bottom row) dataset. For each subject the top row shows slices of the data and the manual lesion annotation; the middle row shows the lesion probability map and Dice score computed by the proposed method for specific input combinations; and the bottom row shows the corresponding complete segmentations produced by the method. Enlarged figures for each subject are available in the Supplementary Material Figs. 1–3.

When comparing the lesion probability maps produced by the method visually with the corresponding manual lesion segmentations, it seems that the method benefits from having access to the FLAIR contrast for the best lesion segmentation performance. This is confirmed by a quantitative analysis shown in Fig. 8, which plots the Dice overlap scores for each of the seven input combinations that all our three datasets have in common, namely T1w, T2w, FLAIR, T1w-T2w, T1w-FLAIR, T2w-FLAIR, and T1w-T2w-FLAIR. Although the inclusion of additional contrasts does not hurt lesion segmentation performance, across all three datasets the best results are obtained whenever the FLAIR contrast is included as input to the model. This finding is perhaps not surprising, given that the manual delineations were all primarily based on the FLAIR image.

Fig. 8.

Lesion segmentation performance of the proposed method in terms of Dice overlap with manual raters on three different datasets when different input contrasts are used (T1w, T2w, FLAIR, T1w-T2w, T1w-FLAIR, T2w-FLAIR, T1w-T2w-FLAIR). From left to right: Dice scores on MSSeg, Trio and Achieva MS data.

Considering both the whole-brain and lesion segmentation performance together, we conclude that the combination T1w-FLAIR is well-suited for obtaining good results with the proposed method, although it will also accept other and/or additional contrasts beyond T1w and FLAIR.

5.3. Lesion segmentation

In order to compare the lesion segmentation performance of our model against that of the two benchmark methods, and relate it to human inter-rater variability, we here present a number of results based on the T1w-FLAIR input combination (which is the combination required by the benchmark methods). We also analyze the lesion segmentation performance of our method on the public ISBI challenge.

5.3.1. Comparison with benchmark lesion segmentation methods

Fig. 9 shows automatic segmentations of two randomly selected subjects from the MSSeg, the Trio and the Achieva datasets, both for our method and for the two benchmark methods LST-lga and NicMSLesions, along with the corresponding manual segmentations (consensus manual segmentations for MSSeg). Visually, all three methods perform similarly on the Achieva MS data, but some of the results for NicMSLesions appear to be inferior to those obtained with the other two methods on MSSeg and Trio data. This qualitative observation is confirmed by the quantitative analysis shown in Fig. 10, where the three methods’ Dice overlap scores are compared on each dataset: similar performances are obtained for all methods on the Achieva data, but NicMSLesions trails the other two methods on MSSeg and Trio data. Especially for MSSeg data this is a surprising result, since NicMSLesions was trained on this specific dataset, i.e., the subjects used for testing were part of the training data of this method, potentially biasing the results in favor of NicMSLesions. Based on Dice scores, the proposed method outperforms LST-lga on MSSeg data, although there are no statistically significant differences between the two methods on the other datasets.

Fig. 9.

Visual comparison of lesion probability maps on three different datasets for the proposed method and two state-of-the-art lesion segmentation methods (LST-lga and NicMsLesions) on T1w-FLAIR input. (Top) Two subjects from the MSSeg dataset; (Middle) Two subjects from the Trio dataset; (Bottom) Two subjects from the Achieva dataset. For each subject the top row shows slices of the data and the manual annotation while the bottom row shows the lesion probability maps for our model, LST-lga and NicMsLesions.

Fig. 10.

Lesion segmentation performance in terms of Dice overlap with manual raters for the proposed method and two benchmark methods (LST-lga and NicMsLesions) on T1w-FLAIR input. Statistically significant differences between two methods, computed with a two-tailed paired t-test, are indicated by asterisks (“***” for p-value < 0.001, “**” for p-value < 0.01 and “*” for p-value < 0.05). From left to right: results on the MSSeg, the Trio and the Achieva dataset.

5.3.2. Results on the ISBI data

We also evaluated the performance of the proposed method on the ISBI challenge data, obtaining a mean Dice score of 0.58 when T1w-FLAIR input is used. This score is comparable to the ones we obtained on the other three datasets analyzed in this paper (cf. Fig. 10) – MSSeg: 0.65, Trio: 0.58 and Achieva: 0.54. A few example segmentation results on the ISBI data are available in the Supplementary Material, Fig. 4.

The ISBI challenge website⁸ ranks submissions according to an overall lesion segmentation performance score that takes into account Dice overlap, volume correlation, surface distance, and a few other metrics (see Carass et al., 2017 for details). A score of 100 indicates perfect correspondence, while 90 is meant to correspond to human inter-rater performance (Carass et al., 2017; Styner et al., 2008). We obtained a score of 87.87, which places us around half-way in the ranking of the original challenge (Carass et al., 2017), although we note that the website currently lists methods with a much higher score.

In order to relate the performance of our method to the one obtained with the two benchmark methods, we also attempted to run LST-lga and NicMSLesions on this dataset. However, the preprocessing applied to the ISBI challenge data proved problematic for LST-lga, and we were not able to get any results with this method. Results for NicMSLesions in terms of Dice overlap are shown in Fig. 11, together with those obtained with the proposed method. It is clear that NicMSLesions suffers strongly from the domain shift between its training data and the ISBI data, a fact that was already reported in Valverde et al. (2019). For completeness, Fig. 11 also includes results for NicMSLesions when its network was updated on the ISBI training data as described in Valverde et al. (2019): different subsets of network parameters were retrained on the baseline scan of each of the five ISBI training subjects, and the combination that performed best on all 21 training images was retained. From the figure it can be seen that this partially retrained network has comparable performance to the proposed model, although the latter attains this performance without any retraining.

Fig. 11.

Lesion segmentation performance in terms of Dice overlap with manual raters on the ISBI dataset for the proposed method, NicMsLesions, and NicMsLesions with partial retraining (see text for details). Statistically significant differences between two methods, computed with a two-tailed paired t-test, are indicated by asterisks (“***” indicates p-value < 0.001).

5.3.3. Inter-rater variability

To evaluate the proposed method’s lesion segmentation performance in the context of human inter-rater variability, we took advantage of the availability of lesion segmentations by seven different raters in the MSSeg dataset. Table 3 shows the lesion segmentation performance in terms of average Dice overlap between each pair of the seven raters, and between each rater and the proposed method. On average, our method achieves a Dice overlap score of 0.57, which is slightly below the mean human raters’ range of [0.59, 0.69]. We note that this result is in line with those obtained in the MSSeg challenge (Commowick et al., 2018).

Table 3.

Comparison of lesion segmentation performance in terms of average Dice score between each pair of the seven raters of the MSSeg dataset, and between each rater and the proposed method (T1w-FLAIR input).

	R1	R2	R3	R4	R5	R6	R7	Ours
R1	–	0.68	0.59	0.70	0.75	0.59	0.59	0.54
R2	0.68	–	0.59	0.71	0.72	0.60	0.57	0.56
R3	0.59	0.59	–	0.57	0.59	0.60	0.63	0.60
R4	0.70	0.71	0.57	–	0.90	0.57	0.54	0.53
R5	0.75	0.72	0.59	0.90	–	0.59	0.57	0.55
R6	0.59	0.60	0.60	0.57	0.59	–	0.61	0.57
R7	0.59	0.57	0.63	0.54	0.57	0.61	–	0.60
Avg	0.65	0.64	0.60	0.66	0.69	0.60	0.59	0.57

5.4. Whole-brain segmentation

Since no ground truth segmentations are available for a direct evaluation of the whole-brain segmentation component of our method, we performed an indirect validation, evaluating its potential for replacing lesion filling approaches that rely on manually annotated lesions, as well as its ability to replicate known atrophy patterns in MS. The results concentrate on the following 25 main neuroanatomical regions, segmented from T1w-FLAIR scans: left and right cerebral white matter, cerebellum white matter, cerebral cortex, cerebellum cortex, lateral ventricle, hippocampus, thalamus, putamen, pallidum, caudate, amygdala, nucleus accumbens and brain stem. To avoid cluttering, the quantitative results for left and right structures are averaged. We note that lesion segmentations are not merged into any of these brain structures (i.e., leaving “holes” in white matter), so that the results reflect performance only for the normal-appearing parts of structures.

5.4.1. Comparison with lesion filling

It is well-known that white matter lesions can severely interfere with the quantification of normal-appearing structures when standard brain MRI segmentation techniques are used (Battaglini et al., 2012; Ceccarelli et al., 2012; Chard et al., 2010; Gelineau-Morel et al., 2012; Nakamura and Fisher, 2009; Vrenken et al., 2013). A common strategy is therefore to use a lesion-filling (Chard et al., 2010; Sdika and Pelletier, 2009) procedure, in which lesions are first manually segmented, their original voxel intensities are replaced with normal-appearing white matter intensities, and standard tools are then used to segment the resulting, preprocessed images. Using such a procedure with SAMSEG would yield whole-brain segmentations that can serve as “silver standard” benchmarks against which the results of the proposed method (which works directly on the original scans) can be compared. In practice, however, we have noticed that replacing lesion intensities, which is typically done in T1w only, did not work well in FLAIR in our experiments. Therefore, rather than explicitly replacing intensities, we obtained silver standard segmentations by simply masking out lesions during the SAMSEG processing, effectively ignoring lesion voxels during the model fitting.

We wished to interpret segmentation vs. silver standard discrepancies within the context of the human inter-rater variability associated with manually segmenting lesions. Therefore, we performed experiments on the MSSeg dataset, repeatedly re-computing the silver standard using each of the seven raters’ manual lesion annotations in turn. The results are shown in Tables 4 and 5 for Pearson correlation coefficients between estimated volumes and Dice segmentation overlap scores, respectively. Each line in these tables corresponds to one structure, showing the average consistency between the silver standard of each rater compared to that of the six other raters, as well as the average consistency between the proposed method’s segmentation and the silver standards of all raters. The results indicate that, in terms of Pearson correlation coefficient, the performance of our method falls within the range of inter-rater variability, albeit narrowly (average value 0.988 vs. inter-rater range [0.988, 0.992]). In terms of Dice scores, however, the method slightly underperforms compared to the inter-rater variability (average value 0.971 vs. inter-rater range [0.978, 0.980]).

Table 4.

Average Pearson correlation coefficients of brain structure volume estimates between the silver standard of each rater compared to that of the six other raters in the MSSeg dataset, as well as the average consistency between the proposed method's segmentation and the silver standards of all raters (T1w-FLAIR input). Each line shows an average across raters for a specific brain structure.

	R1	R2	R3	R4	R5	R6	R7	Ours
Cerebral White Matter	0.992	0.992	0.991	0.993	0.993	0.993	0.987	0.989
Cerebellum White Matter	0.994	0.997	0.997	0.996	0.997	0.997	0.997	0.989
Cerebral Cortex	0.997	0.999	0.999	0.999	0.999	0.999	0.999	0.997
Cerebellum Cortex	0.999	0.999	0.997	0.999	0.997	0.999	0.999	0.999
Lateral Ventricles	0.996	0.995	0.996	0.997	0.998	0.994	0.996	0.992
Hippocampus	0.982	0.989	0.987	0.979	0.977	0.979	0.984	0.981
Thalamus	0.998	0.997	0.998	0.998	0.998	0.997	0.997	0.996
Putamen	0.999	0.999	0.999	0.999	0.999	0.999	0.999	0.996
Pallidum	0.988	0.993	0.993	0.994	0.993	0.994	0.990	0.989
Caudate	0.994	0.993	0.987	0.990	0.995	0.989	0.993	0.985
Amygdala	0.953	0.967	0.970	0.973	0.941	0.957	0.972	0.963
Accumbens	0.985	0.987	0.987	0.966	0.989	0.953	0.988	0.971
Brain Stem	0.991	0.994	0.990	0.992	0.992	0.988	0.992	0.989
Average	0.990	0.992	0.992	0.990	0.990	0.988	0.992	0.988

Table 5.

Same as Table 4, but with Dice segmentation overlap scores. Each line shows an average across raters – similar to the last row of Table 3 – for a specific brain structure.

	R1	R2	R3	R4	R5	R6	R7	Ours
Cerebral White Matter	0.982	0.982	0.982	0.983	0.983	0.982	0.981	0.978
Cerebellum White Matter	0.987	0.987	0.987	0.987	0.988	0.987	0.987	0.983
Cerebral Cortex	0.989	0.990	0.989	0.990	0.989	0.989	0.989	0.986
Cerebellum Cortex	0.996	0.996	0.995	0.996	0.996	0.995	0.995	0.994
Lateral Ventricles	0.972	0.970	0.972	0.974	0.976	0.971	0.971	0.954
Hippocampus	0.975	0.972	0.971	0.973	0.974	0.972	0.972	0.965
Thalamus	0.980	0.981	0.981	0.982	0.981	0.982	0.981	0.975
Putamen	0.987	0.987	0.988	0.988	0.988	0.988	0.987	0.980
Pallidum	0.985	0.985	0.986	0.986	0.986	0.986	0.985	0.978
Caudate	0.961	0.957	0.956	0.961	0.964	0.957	0.954	0.937
Amygdala	0.973	0.972	0.972	0.973	0.972	0.972	0.972	0.967
Accumbens	0.957	0.958	0.960	0.960	0.960	0.943	0.960	0.945
Brain Stem	0.987	0.986	0.984	0.986	0.986	0.986	0.986	0.983
Average	0.979	0.979	0.979	0.980	0.980	0.978	0.978	0.971

5.4.2. Detecting atrophy patterns in MS

In a final analysis, we assessed whether previously reported volume reductions in specific brain structures in MS can automatically be detected with the proposed method. Towards this end, we segmented the 23 controls and the 50 MS subjects of the Achieva dataset, and compared the volumes of various structures between the two groups. Volumes were normalized for age, gender and total intracranial volume by regressing them out with a general linear model. The intracranial volume used for the normalization was computed by summing the volumes of all the structures, as segmented by the method, within the intracranial vault. The results are shown in Fig. 12. Although not all volumes showed significant difference between groups, well established differences were replicated. In particular, we demonstrated decreased volumes of cerebral white matter, cerebral cortex, thalamus and caudate (Azevedo et al., 2018; Chard et al., 2002; Houtchens et al., 2007) as well as an increased volume of the lateral ventricles (Zivadinov et al., 2016).

Fig. 12.

Differences between healthy controls (HC) and MS subjects in normalized volume estimates of various neuroanatomical structures, as detected by the proposed method on the Achieva dataset (23 HC subjects, 50 MS subjects, T1w-FLAIR input). Statistically significant differences between the two groups, computed with a Welch’s t-test, are indicated by asterisks (“**” for p-value < 0.01 and “*” for p-value < 0.05).

6. Discussion and conclusion

In this paper, we have proposed a method for the simultaneous segmentation of white matter lesions and normal-appearing neuroanatomical structures from multi-contrast brain MRI scans of MS patients. The method integrates a novel model for white matter lesions into a previously validated generative model for whole-brain segmentation. By using separate models for the shape of anatomical structures and their appearance in MRI, the algorithm is able to adapt to data acquired with different scanners and imaging protocols without needing to be retrained. We validated the method using four disparate datasets, showing robust performance in white matter lesion segmentation while simultaneously segmenting dozens of other brain structures. We further demonstrated that it can also be safely applied to MRI scans of healthy controls, and replicate previously documented atrophy patterns in deep gray matter structures in MS. The proposed algorithm is publicly available as part of the open-source neuroimaging package FreeSurfer.

By performing both whole-brain and white matter lesion segmentation at the same time, the method we propose aims to supplant the two-stage “lesion filling” procedure that is commonly used in morphometric studies in MS, in which lesions segmented in a first step are used to avoid biasing a subsequent analysis of normal-appearing structures with software tools developed for healthy brain scans. In order to evaluate whether our method is successful in this regard, we compared its whole-brain segmentation performance against the results obtained when lesions are segmented a priori by seven different human raters instead of automatically by the method itself. Our results show that the volumes of various neuroanatomical structures obtained when lesions are segmented automatically fall within the range of inter-rater variability, indicating that the proposed method may be used instead of lesion filling with manual lesion segmentations in large volumetric studies of brain atrophy in MS. When detailed spatial overlap is analyzed, however, we found that the automatic segmentation does not fully reach the performance obtained with human lesion annotation as measured by Dice overlap.

Like many other methods for MS lesion segmentation, the method proposed here produces a spatial map indicating in each voxel its probability of belonging to a lesion, which can then be thresholded to obtain a final lesion segmentation. Although in our experience good results can be obtained by using the same threshold value across datasets (e.g., γ = 0.5), changing this value allows one to adjust the trade-off between false positive and false negative lesion detections. Since some MRI sequences and scanners will depict lesions with a higher contrast than others, and because there is often considerable disagreement between human experts regarding the exact extent of lesions (Zijdenbos et al., 1998), in our implementation we therefore expose this threshold value as an optional, tunable parameter to the end-user. Suitable threshold values can be found by visually inspecting the lesion segmentations of a few cases or, in large-scale studies, using cross-validation as we did in our experiments.

By providing the ability to robustly and efficiently segment multicontrasts scans of MS patients across a wide range of imaging equipment and protocols, the software tool presented here may help facilitate large cohort studies aiming to elucidate the morphological and temporal dynamics underlying disease progression and accumulation of disability in MS. Furthermore, in current clinical practice, high-resolution multicontrast images, which can be used to increase the accuracy of lesion segmentation, represent a significantly increased burden for the neuroradiologist to read, and are hence frequently not acquired. The emergence of robust, multi-contrast segmentation tools such as ours may help break the link between the resolution and number of contrasts of the acquired data and the human time needed to evaluate it, thus potentially increasing the accuracy of the resulting measures.

The ability of the proposed method to automatically tailor its appearance models for specific datasets makes it very flexible, allowing it to seamlessly take advantage of novel, potentially more sensitive and specific MRI acquisitions as they are developed. Although not extensively tested, the proposed method should make it possible to, with minimal adjustments, segment data acquired with advanced research sequences such as MP2RAGE (Marques et al., 2010), DIR (Redpath and Smith, 1994), FLAIR² (Wiggermann et al., 2016) or T2* (Anderson et al., 2001), both at conventional and at ultra-high magnetic field strengths. We are currently pursuing several extensions of the proposed method, including the ability to go on and create cortical surfaces and parcellations in FreeSurfer, as well as a dedicated version for longitudinal data (Cerri et al., 2020).

Appendix A. Parameter optimization in SAMSEG

We here describe how we perform the optimization of p(θ|D) with respect to θ and in the original SAMSEG model. We follow a coordinate ascent approach, in which a limited-memory BFGS optimization of θ_l is interleaved with a generalized EM (GEM) optimization of the remaining parameters θ_d. The GEM algorithm was derived in (Van Leemput et al., 1999) based on (Wells et al., 1996), and is repeated here for the sake of completeness. It iteratively constructs a tight lower bound to the objective function by computing the soft label assignments ω_i,k based on the current estimate of θ_d (Eq. (2)), and subsequently improves the lower bound (and therefore the objective function) using the following set of analytical update equations for these parameters :

$$\begin{gathered} {\mathbf{\mu }}_k\leftarrow {\mathbf{m}}_k\text{and}{\mathbf{\Sigma }}_k\leftarrow {\mathbf{V}}_k,{\mathbf{\forall }}_k \\ \left(\begin{array}{c}{\mathbf{c}}_1\\ ⋮\\ {\mathbf{c}}_N\end{array}\right)\leftarrow {\left(\begin{array}{ccc}{A}^T{S}_{1,1}A& \cdots & A^T{S}_{1,N}A\\ ⋮& \ddots & ⋮\\ A^T{S}_{N,1}A& \cdots & A^T{S}_{N,N}A\end{array}\right)}^{-1}\left(\begin{array}{c}{A}^T\left({\sum }_{n=1}^N{S}_{1,n}{r}_{1,n}\right)\\ ⋮\\ A^T\left({\sum }_{n=1}^N{S}_{N,n}{r}_{N,n}\right)\end{array}\right), \end{gathered}$$

where

$$m_k=\frac{{\sum }_{i=1}^I{\omega }_{i,k}\left(d_i-C{\mathit{\varphi }}_i\right)}{N_k}\mathrm{with}{N}_k={\sum }_{i=1}^I{\omega }_{i,k},$$

$$V_k=\frac{{\sum }_{i=1}^I{\omega }_{i,k}\left(d_i-C{\mathit{\varphi }}_i-m_k\right){\left(d_i-C{\mathit{\varphi }}_i-m_k\right)}^T}{N_k},$$

$$A=\left(\begin{array}{ccc}{\varphi }_1^1& \cdots & {\varphi }_p^1\\ ⋮& \ddots & ⋮\\ {\varphi }_1^I& \cdots & {\varphi }_P^I\end{array}\right),S_{m,n}=\mathrm{diag}\left(s_i^{m,n}\right),r_{m,n}=\left(\begin{array}{c}{r}_1^{m,n}\\ ⋮\\ r_I^{m,n}\end{array}\right)$$

and

$$s_i^{m,n}={\sum }_{k=1}^K{s}_{i,k}^{m,n},s_{i,k}^{m,n}={\omega }_{i,k}{\left({\sum }_k^{-1}\right)}_{m,n},r_i^{m,n}=d_i^n-\frac{{\sum }_{l=1}^K{s}_{i,k}^{m,n}{\left({\mathit{\mu }}_k\right)}_n}{{\sum }_{l=1}^K{s}_{i,k}^{m,n}}.$$

Appendix B. Parameter optimization

Here we describe how we perform the optimization of p(θ, θ_les|D) with respect to θ and θ_les in the augmented model of Sec. 3 with the decoder outputs f_i(h) all clamped to value 1. In that case, the model can be reformulated in the same form as the original SAMSEG model, so that the same optimization strategy can be used. In particular, lesions can be considered to form an extra class (with index K + 1) in a SAMSEG model with K + 1 labels, provided that the mesh vertex label probabilities

$${\tilde{\alpha }}_j^k=\{\begin{array}{cc}{\beta }_j& \mathrm{if}k=K+1\left(\mathrm{lesion}\right),\\ {\alpha }_j^k\left({\beta }_j-1\right)& \mathrm{otherwise}.\end{array}$$

are used instead of the original ${\alpha }_j^k$’s in the atlas interpolation model of Eq. (1).

The optimization described in Appendix A does require one modification because of the prior p(θ_les|θ_d) binding the means and variances of the WM and lesion classes together. The following altered update equations for these parameters guarantee that the EM lower bound, and therefore the objective function, is improved in each iteration of the GEM algorithm:

$$\begin{gathered} {\mathit{\mu }}_{WM}\leftarrow {\left(N_{WM}I+\frac{v{N}_{WM}}{v+N_{WM}}{\Sigma }_{WM}{\Sigma }_{les}^{-1}\right)}^{-1} \\ \left(N_{WM}{m}_{WM}+\frac{v{N}_{WM}}{v+N_{WM}}{\Sigma }_{WM}{\Sigma }_{les}^{-1}{m}_{les}\right), \end{gathered}$$

$${\Sigma }_{WM}\leftarrow \frac{N_{WM}{V}_{WM}+{\Sigma }_{les}{\Sigma }_{WM}^{-1}{\Psi }_{les}}{N_{WM}+N_{les}+N+2},$$

$${\mathit{\mu }}_{les}\leftarrow \frac{N_{les}{m}_{les}+v{\mathit{\mu }}_{WM}}{N_{les}+v},$$

$${\Sigma }_{les}\leftarrow \frac{{\Psi }_{les}+v\kappa {\Sigma }_{WM}}{N_{les}+v},$$

where ${\mathbf{\Psi }}_{les}=\frac{N_{les}v}{N_{les}+v}\left(m_{les}-{\mathit{\mu }}_{WM}\right){\left(m_{les}-{\mathit{\mu }}_{WM}\right)}^T+N_{les}{V}_{les}.$

Appendix C. Estimating lesion probabilities

We here describe how we we approximate $p(z_i=1|d_i,\widehat{\theta })$ using Monte Carlo sampling. We use a Markov chain Monte Carlo (MCMC) approach to sample triplets {${\mathit{\theta }}_{les}^{(s)},{\mathbf{z}}^{(s)},{\mathbf{h}}^{(s)}$} from the distribution $p({\mathit{\theta }}_{les},\mathbf{z},\mathbf{h}|D,\widehat{\mathit{\theta }}):$ Starting from an initial lesion segmentation z⁽⁰⁾ obtained from the parameter estimation procedure described in Appendix B, we use a blocked Gibbs sampler in which each variable is updated conditioned on the other ones:

$$\begin{gathered} {\mathbf{\Sigma }}_{les}^{(s+1)}\sim p({\mathbf{\Sigma }}_{les}|D,\widehat{\mathit{\theta }},z^{(s)})= \\ \mathrm{IW}({\mathbf{\Sigma }}_{les}|{\psi }_{les}^{(s)}+v\kappa {\widehat{\mathbf{\Sigma }}}_{W,M},N_{les}^{\left(s\right)}+v-N-2) \end{gathered}$$

$$\begin{gathered} {\mu }_{les}^{\left(s+1\right)}\sim p({\mathit{\mu }}_{les}|D,\widehat{\mathit{\theta }},z^{\left(s\right)},{\mathbf{\Sigma }}_{les}^{\left(s+1\right)})= \\ \mathcal{N}\left({\mathit{\mu }}_{les}|\frac{N_{les}^{\left(s\right)}{m}_{les}^{\left(s\right)}+v{\widehat{\mathit{\mu }}}_{WM}}{N_{les}^{\left(s\right)}+v},\frac{{\Sigma }_{les}^{\left(s+1\right)}}{N_{les}^{\left(s\right)}+v}\right) \end{gathered}$$

$$h^{\left(s+1\right)}\sim p(h|z^{(s)})\simeq \mathcal{N}\left(h|{\mu }_{\upsilon }(z^{\left(s\right)}),\mathrm{diag}({\mathbf{\sigma }}_{\upsilon }^2(z^{(s)}))\right)$$

$$z^{\left(s+1\right)}~p(z|D,\widehat{\mathit{\theta }},h^{\left(s+1\right)},{\mathit{\theta }}_{les}^{\left(s+1\right)})=\prod _{i=1}^{I}p(z_i|d_i,\widehat{\mathit{\theta }},h^{\left(s+1\right)},{\mathit{\theta }}_{les}^{\left(s+1\right)}),$$

where we use the encoder variational approximation obtained during the training of the lesion shape model (see Sec. 3.1.2) to sample from h in the next-to-last step, and

$$p(z_i=1{|d}_i,\widehat{\mathit{\theta }},h,{\mathit{\theta }}_{les})=\frac{\mathcal{N}\left(d_i{|\mathit{\mu }}_{les}+\mathbf{C}{\mathit{\varphi }}_i,{\mathbf{\Sigma }}_{les}\right)f_i\left(h\right){\rho }_i({\widehat{\mathit{\theta }}}_{\mathbf{l}})}{{\sum }_{l_i=1}^K{\sum }_{z_i^{\prime }=0}^{1}p(d_i|l_i,z_i^{\prime },{\widehat{\mathit{\theta }}}_{\mathbf{l}},{\mathit{\theta }}_{les})p(z_i^{\prime }|{\widehat{\mathit{\theta }}}_{\mathbf{l}},h)p(l_i|{\widehat{\mathit{\theta }}}_{\mathbf{l}})}$$

in the last step. In these equations, the variables $N_{les}^{\left(s\right)}$, ${\mathbf{m}}_{les}^{\left(s\right)}$, ${\mathbf{V}}_{les}^{\left(s\right)}$ and ${\mathbf{\Psi }}_{les}^{\left(s\right)}$ are as defined before, but using voxel assignments $w_{i,\mathit{les}}=z_i^{\left(s\right)}$. Once S samples are obtained, we approximate $p(z_i=1|d_i,\widehat{\mathit{\theta }})$ as

$$p(z_i=1|d_i,\widehat{\mathit{\theta }})\simeq \frac{1}{S}\sum _{s=1}^{S}p(z_i=1{|d}_i,\widehat{\mathit{\theta }},h^{\left(s\right)},{\mathit{\theta }}_{les}^{\left(s\right)}).$$

In our implementation, we use S = 50 samples, obtained after discarding the first 50 sweeps of the sampler (so-called “burn-in” phase). The algorithm repeatedly invokes the decoder and encoder networks of the lesion shape model described in Sec. 3.1.2. Since this shape model was trained in a specific isotropic space, the algorithm requires transitioning between this training space and subject space using an affine transformation. This is accomplished by resampling the input and output of the encoder and decoder, respectively, using linear interpolation.