The effect of region of interest size on the repeatability of quantitative brain imaging biomarkers

Abstract

In the repeatability analysis, when the measurement is the mean value of a parametric map within a region of interest (ROI), the ROI size becomes important as by increasing the size, the measurement will have a smaller variance. This is important in decision-making in prospective clinical studies of brain when the ROI size is variable, e.g., in monitoring the effect of treatment on lesions by quantitative MRI, and in particular when the ROI is small, e.g., in the case of brain lesions in multiple sclerosis. Thus, methods to estimate repeatability measures for arbitrary sizes of ROI are desired. We propose a statistical model of the values of parametric map within the ROI and a method to approximate the model parameters, based on which we estimate a number of repeatability measures including repeatability coefficient, coefficient of variation, and intra-class correlation coefficient for an ROI with an arbitrary size. We also show how this gives an insight into related problems such as spatial smoothing in voxel-wise analysis. Experiments are conducted on simulated data as well as on scan-rescan brain MRI of healthy subjects. The main application of this study is the adjustment of the decision threshold based on the lesion size in treatment monitoring.

I. Introduction

Knowledge about the accuracy of quantitative brain imaging biomarkers is important in studying the brain tissue function in healthy subjects as well as diagnosis and treatment planning in patients. Repeatability (test-retest reliability) analysis may help us interpret a level of change in a tissue biological parameter as a real biological change or as a measurement error. Conventionally, repeated scans are acquired on a group of subjects and the desired parametric maps are generated. The mean value of the parametric map is then calculated within the desired ROIs. Repeatability may then be evaluated by measures such as repeatability coefficient (RC), coefficient of variation (COV), and intra-class correlation coefficient (ICC). RC provides an estimate for the between-measurement variance so that the difference between two measurements is expected to be between −RC and RC for 95% of subjects. COV represents the extent of variability in relation to the mean of measurements where smaller COV implies higher repeatability. ICC ranges from 0 to 1 and describes how strongly two within-subject measurements resemble each other relative to the variation between subjects.

The repeatability measures will depend on the ROI size as with increasing the ROI size, the mean value within the ROI tends to have a smaller variance. Thus, it is important to compute and report the repeatability measures for the desired ROI sizes. For example, to determine if a real change has occurred in a tissue after therapy, the level of change is compared with a threshold, conventionally the RC value calculated from a repeatability study [1]. As the RC value depends on the ROI size, the threshold needs to be adjusted based on the ROI size.

Evaluation of the effect of ROI size on repeatability becomes particularly important when there is variability in ROI, e.g. in studying tumors [2] [3] [4] [5], and when the ROI is small, e.g., in studying brain lesions in multiple sclerosis [6] [7] [8] [9] [10]. Although it is not the focus of this study, the effect of ROI size also relates to several other problems such as spatial smoothing in voxel-wise analysis as well as applications of arterial input function in dynamic MRI. In voxel-wise analysis, the corresponding voxels of coregistered images are compared [11] [12]. This means the size of ROI under study is essentially one voxel. Spatial smoothing is applied to reduce the effects of mis-coregistration, noise, signal variations, etc. [13]. This also has the effect of increasing the effective ROI size. However, spatial smoothing also results in the loss of spatial information. Therefore, the filter size becomes important [14], which may be optimized experimentally [15] [16] [17] and may be set to the smallest that gives the desired repeatability measures. This will be later discussed for the particular case of voxel-wise analysis for functional diffusion maps when applying average filtering.

Previous studies have experimentally reported improved repeatability for larger ROIs of brain apparent diffusion coefficient (ADC) [18] and brain volume measurements [19]. In this paper, we propose a statistical model of the voxel intensities to approximate three frequently used measures of repeatability, namely RC, COV, and ICC, for a desired ROI size. While the previous works experimentally studied the effect of ROI size on the repeatability measures, here we provide an analytical model to study the same. We conduct experiments on simulated data as well as on parametric maps generated from repeated brain MRI scans of 21 healthy subjects [20]. The main application of this work is treatment monitoring in prospective studies. Although we show how this problem is related to a number of other problems including smoothing in voxel-wise analysis, we do not provide a direct guideline for problems such as optimizing the filter size, as those are out of the scope of this work and require other comprehensive studies. This model, however, could potentially help future efforts of filter design in voxel-wise analysis.

II. MATERIALS AND METHODS

A. Statistical model

Assume the image of the brain is acquired K times for N subjects, the same ROIs are generated, and the mean value of a parametric map across all voxels within the ROI is calculated as a measurement. We propose the following model to represent the measurement for subject i, within voxel j in repetition k.

$$\begin{gathered} x_{ijk}=\mu +{\tau }_i+{\lambda }_k+{\gamma }_{ik}+{\beta }_{ij}+{\epsilon }_{ijk}{}_{;} \\ i=1,\dots ,N;j=1,\dots ,M;k=1,\dots ,K, \end{gathered}$$ (1)

where μ is the overall voxels mean for the population; τ_i is the effect of subject i on the measurement; λ_k is the deviation from the overall mean due to the kth scan; γ_ik the interaction effect between subject i and scan k; β_ij is the effect of voxel j within the ROI of subject i; and ε_ijk is a random error. Let us assume that the all main and interaction effects except for the voxel effect are random, and normally independently distributed. That is, τ_i~N(0, ${\sigma }_{\tau }^2$), λ_k~N(0, ${\sigma }_{\lambda }^2$), and γ_ik~N(0, ${\sigma }_{\gamma }^2$). Also let us assume that voxel effects β_ij and random errors ε_ijk are normally distributed with mean of zero and spatially correlated. We also assume that random errors are independent of the effects. As we will explain later, the assumption of normality is not necessary but under its assumption we will be able to obtain more accurate estimation of the repeatability measures.

To take into account the spatial dependence between intensities of the voxels within an ROI, knowing that the spatial dependence decreases with increasing the distance between voxels, we propose to use exponential covariogram model [21]:

$$\text{cov}\left({\beta }_{i{j}_{{}_1}},{\beta }_{i{j}_{{}_2}}\right)=\{\begin{array}{cc}{\sigma }_{\beta }^2,\hfill & j_1=j_2\hfill \\ {\rho }_{\beta }\text{exp}\left(-r∕{\eta }_{\beta }\right),\hfill & j_1\ne j_2\hfill \end{array}\phantom{\}}$$ (2)

where ${\sigma }_{\beta }^2$, ρ_β, and η_β are model parameters, and r is the distance between voxels j₁ and j₂. The choice of exponential model was based on experiments we conducted on the MRI data used in this paper (experiments not presented but described in the Supplementary Material, Section I). Other models may also be used depending on the data, in which case our derived formulas will be still valid (the formulas are independent of the covariogram model, but the parameter estimation method may need minor update depending on the model). Similarly, we use exponential covariogram model with parameters ${\sigma }_{\epsilon }^2$, ρ_ε, and η_ε for cov(ε_ij1_k, _εij2k). The spatial dependence between ε_ijk variables within an ROI is partially caused by coregistration errors.

Suppose that all subjects have the ROI of size M. Then the measurement within an ROI is obtained by

$$x_{iok}=\frac{1}{M}\sum _{j=1}^M{x}_{ijk}=\mu +{\tau }_i+{\gamma }_{ik}+{\lambda }_k+\frac{1}{M}\sum _{j=1}^M{\beta }_{ij}+\frac{1}{M}\sum _{j=1}^M{\epsilon }_{ijk}$$ (3)

B. Repeatability coefficient

Provided that ROI measurements follow model (3), RC is estimated by $\text{RC}=2.77\sqrt{\text{WMS}}$, where WMS is the within-subject mean square [1] and is calculated as follows.

$$\text{WMS}=\frac{1}{N(K-1)}\sum _{i=1}^N\sum _{k=1}^K{\left(x_{iok}-\frac{1}{K}\sum _{k=1}^K{x}_{iok}\right)}^2$$ (4)

We can show that (see Supplementary Material, Section II):

$$\mathrm{E}\left\{\text{WMS}\right\}={\sigma }_{\gamma }^2+{\sigma }_{\lambda }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\epsilon }$$ (5)

where E{⋅} is the expected value and

$${\tilde{\sigma }}_{\epsilon }=\sum _{j_1=1}^M\sum _{j_2=1}^M\mathrm{cov}\left({\epsilon }_{i{j}_{{}_1}k},{\epsilon }_{i{j}_{{}_2}k}\right).$$ (6)

Thus,

$$\mathrm{E}\left\{{\text{RC}}^2\right\}={2.77}^2\left({\sigma }_{\gamma }^2+{\sigma }_{\lambda }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\epsilon }\right)$$ (7)

As the exact equation for the expected value of RC cannot be derived, we may use the first order Taylor series to approximate it by:

$$\mathrm{E}\{\text{RC}\}\approx 2.77\sqrt{{\sigma }_{\gamma }^2+{\sigma }_{\lambda }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\epsilon }}.$$ (8)

As will be discussed later, the main applications of this study are related to the usage of RC in monitoring of treatment.

C. Coefficient of variation

Under model (3), the coefficient of variation is estimated by

$$\mathrm{COV}=\frac{\sqrt{\text{WMS}}}{\text{AVG}},$$ (9)

where $\text{AVG}=\frac{1}{NK}\sum _{i=1}^N\sum _{k=1}^K{x}_{iok}$. Using equation (5) and the Taylor expansion, the expected value of COV is written as

$$\mathrm{E}\{\mathrm{COV}\}\approx \frac{1}{\mu }\sqrt{{\sigma }_{\gamma }^2+{\sigma }_{\lambda }^2+\frac{1}{M^{{}^2}}{\tilde{\sigma }}_{\epsilon }}.$$ (10)

D. Intra-class correlation coefficient

Under model (3), ICC is calculated by [22]:

$$\text{ICC}=\frac{\text{BMS}-\text{EMS}}{\text{BMS}+(K-1)\text{EMS}+K(\text{RMS - EMS})∕N},$$ (11)

where BMS, EMS, RMS are between target mean squares, residual mean squares, and between rater mean squares, respectively. It can be shown that (see Supplementary Material, Section II):

$$\mathrm{E}\{\text{BMS}\}=K{\sigma }_{\tau }^2+{\sigma }_{\gamma }^2+\frac{1}{M^2}\left(K{\tilde{\sigma }}_{\beta }+{\widehat{\sigma }}_{\epsilon }\right),$$ (12)

$$\mathrm{E}\{\text{EMS}\}={\sigma }_{\gamma }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\epsilon },$$ (13)

$$\mathrm{E}\{\text{RMS}\}={\sigma }_{\gamma }^2+N{\sigma }_{\lambda }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\epsilon },$$ (14)

Where

$${\tilde{\sigma }}_{\beta }=\sum _{j_1=1}^M\sum _{j_2=1}^M\mathrm{cov}\left({\beta }_{i{j}_{{}_1}},{\beta }_{i{j}_{{}_2}}\right)$$ (15)

Consequently, the expected values of numerator and denominator in equation (11) respectively are

$$\mathrm{E}\{{\text{ICC}}_{\text{Numerator}}\}=K\left({\sigma }_{\tau }^2+{\tilde{\sigma }}_{\beta }\frac{1}{M^2}\right)$$ (16)

$$\mathrm{E}\left\{{\text{ICC}}_{\text{Denominator}}\right\}=K\left({\sigma }_{\tau }^2+{\sigma }_{\gamma }^2+{\sigma }_{\lambda }^2+\left({\tilde{\sigma }}_{\beta }+{\tilde{\sigma }}_{\epsilon }\right)\frac{1}{M^2}\right)$$ (17)

and using the first order Taylor series,

$$\mathrm{E}\{\text{ICC}\}\approx \mathrm{E}\left\{{\text{ICC}}_{\text{Numerator}}\right\}∕\mathrm{E}\left\{{\text{ICC}}_{\text{Denominator}}\right\}$$ (18)

and thus,

$$\mathrm{E}\{\text{ICC}\}\approx \frac{{\sigma }_{\tau }^2+{\tilde{\sigma }}_{\beta }∕M^2}{{\sigma }_{\tau }^2+{\sigma }_{\gamma }^2+{\sigma }_{\lambda }^2+\left({\tilde{\sigma }}_{\beta }+{\tilde{\sigma }}_{\epsilon }\right)∕M^2}$$ (19)

Note that the approximations in equations (8), (10), and (19) do not use the normality assumption of equation (1). Under normality assumption, we have provided second-order Taylor series approximations (thus more accurate) to RC, COV, and ICC in the Supplementary Material.

Note that ${\tilde{\sigma }}_{\epsilon }$ and ${\tilde{\sigma }}_{\beta }$ are functions of ROI size M and will also depend on the shape of ROI as the spatial covariance is a function of distance between voxels. Thus, RC, COV, and ICC are functions of M as well as the ROI shape. This also means that given the model parameters, we are able to estimate RC, COV, and ICC for arbitrary ROI size and shape.

Fig. 1 plots expected values of repeatability measures versus ROI size for RC, COV, and ICC using the model parameters μ = 0.7, σ_τ = 0.08, σ_λ = 10⁻⁴, σ_γ = 0.015, σ_β = 0.15, ρ_β = 0.02, η_β = 1.2, σ_ε = 0.05, ρ_ε = 0.002, & η_ε = 1.5. These parameters were selected to be close to the parameters that we estimated for fractional anisotropy (FA) of corpus callosum (CC) in the MRI dataset used in the experiments of Section III-B (we rounded the parameters for simple reporting). For simplicity, the ROIs were selected to be cubes with size L³ voxels, for L = 1,2, … ,10, but ROIs with an arbitrary shape (e.g., spherical) could also be evaluated using the derived equations. We have plotted the approximate measures from the second-order Taylor expansion as given in Supplementary Material, Section IV. As shown, for the given model parameters and ROI shape, with increasing the ROI size, RC and COV decrease whereas ICC increases.

Fig. 1.

RC, COV, and ICC curves versus ROI size for the model parameters μ = 0.7, σ_τ = 0.08, σ_λ = 10⁻⁴, σ_γ = 0.015, σ_β = 0.15, ρ_β = 0.02, η_β = 1.2, σ_ε 0.05, ρ_ε = 0.002, and η_ε = 1.5 for cubic ROIs calculated using second-order Taylor expansion.

E. Repeatability measures for a desired ROI

In order to estimate the repeatability measures for a desired ROI, we first need to estimate the parameters of the measurement model in (1) (i.e., μ, ${\sigma }_{\tau }^2$, ${\sigma }_{\lambda }^2$, ${\sigma }_{\gamma }^2$, ${\sigma }_{\beta }^2$, ρ_β, η_β, ${\sigma }_{\epsilon }^2$, ρ_ε, and η_ε). For this purpose, we propose a hybrid least-square (LS) and method-of-moment (MOM) approach based on the measurements of N available subjects. We begin with finding the population moments. We note that

$$\mathrm{E}\{\text{BMS-EMS}\}=K\left({\sigma }_{\tau }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\beta }\right)$$ (20)

$$\mathrm{E}\{\text{EMS}\}={\sigma }_{\gamma }^2+\frac{1}{M^2}{\tilde{\sigma }}_{\epsilon },$$ (21)

and

$$\mathrm{E}\{\text{WMS-EMS}\}={\sigma }_{\lambda }^2.$$ (22)

Following MOM, we need to substitute the population moments with their point estimates calculated from the available sample, and then, solve the above system of equations to estimate the parameters. However, since the number of parameters is larger than the number of equations, using MOM, we cannot distinctly estimate any of variance components but ${\sigma }_{\lambda }^2$. To overcome this issue, we suggest an ROI subsampling approach as follows. For each subject, a sample of M ≤ M_T. connected voxels is randomly chosen from the ROI with the size of M_T. Then these subsamples are used to calculate BMS, EMS, and WMS using equations (S-1), (S-6) (provided in Supplementary Material, Section II), and (4). This procedure is repeated N_repeat times for a fixed M, and the average of the calculated BMS, EMS, and WMS across the repetitions is found. Next, the whole procedure is repeated for a new M. Following this approach will lead to a number of estimations for E{EMS}, E{BMS}, and E{WMS} for different values of M, the ROI size.

Finally, using this information, the LS method can be utilized to estimate ${\sigma }_{\tau }^2$, ${\sigma }_{\beta }^2$, ρ_β, η_β, ${\sigma }_{\gamma }^2$, ${\sigma }_{\epsilon }^2$, ρ^ε, and η_ε based on equations (20) and (21). Here, we have two non-linear LS problems. The LS problem related to equation (20) is convex with respect to ${\sigma }_{\tau }^2$, ${\sigma }_{\beta }^2$, and Ρ_β, given η_β. The LS problem related to equation (21) is convex with respect to ${\sigma }_{\gamma }^2$, ${\sigma }_{\epsilon }^2$, and ρ_ε, given η_ε. Thus, given η_ε and η_β, the rest of parameters can be explicitly obtained by setting the corresponding derivatives to zero. Thus, the problem boils down to finding tuning parameters η_β and η_ε that minimize the estimation error. This is done by a linear search approach over a grid parameter in pre-specified interval. As the equations (20) and (21) have four parameters each to estimate, we will need average values of BMS-EMS, EMS, and WMS-EMS for at least four ROI sizes. The parameter μ may be estimated by finding the grand average of all measurements across the subjects. These parameter estimates along with equations (8), (10), and (19) can further be used to predict RC, COV, and ICC for ROIs with arbitrary sizes and shapes.

It is also interesting to compare the variance of RC, COV, and ICC estimates obtained by the proposed procedure described above with the variance of these measured when they are directly calculated from M‣ available voxels. For the latter, we have derived the variances of RC, COV, and ICC in the Supplementary Material under normality assumption in model (1). For the estimated RC, COV, and ICC using our proposed method, the equations for finding their variances would be more involved. Therefore, we evaluate their variances by simulation and calculate the average of variances.

If the ROI contains multiple tissue types, as long as the intensities of parametric maps obey model (1), the proposed method is valid. As shown later, the main changes in the repeatability measures occur for small ROIs, which have smaller chance of containing multiple tissue types.

We test the proposed model using simulation and brain MRI data as follows.

F. Simulation

Fig. 2 shows the block diagram of our simulation. We use the model introduced in (1) to simulate data points with the model parameters provided in Section II-D for a cube ROI of size 10×10×10 (i.e., M_T = 1000) and for N = 21 and K = 2. The choice of cube ROI is just for simplicity. ROIs with any shape other than cube can also be considered. Also, the choice of voxel volume does not affect our analysis. The above ROI is assumed to be the available ROI for the estimation of model parameters. We then estimate the parameters using the proposed hybrid method. We generate connected voxels by considering all possible cubic sub-ROIs of size L×L×L, for L = 1, 2, 5, and 10. We use fixed-shaped ROIs to eliminate the effect of ROI shape in the simulation. To test the performance of the proposed method on ROIs with arbitrary size, we generate repeatability curves versus ROI size for cubic ROIs of size L×L×L voxels for L = 1,2, … ,10, using the estimated parameters and compare them with the ground truth calculated from true model parameters and generate the mean square error (MSE) between them. We generate the ground truth repeatability measures using the second-order Taylor expansion equations given in Supplementary Material, Section IV. We repeat the simulations N_repeat =5000 times and calculate the mean, variance, and MSE of the generated curves. Furthermore, to compare with numeric estimation we separately simulate data points using the same model parameters for cube ROIs of size L×L×L voxels for L = 1,2, … ,10 and directly calculate the repeatability measures (numerical calculation) using equations (4), (9), and (11). We repeat this N_repeat =5000 times and calculate the mean, variance, and MSE of the generated curves. The number of repetitions N_repeat =5000 was chosen to be sufficiently large to generate accurate (smooth) variance curves.

Fig. 2.

Block diagram of the simulation. We repeat the whole process 5000 times with the same model parameters and generate average curves of RC, COV, and ICC versus ROI size.

G. Brain MRI dataset

We use a dataset of 21 healthy subjects with repeated brain MRI scans [20] to evaluate the effect of the ROI size. The subjects had no history of neurological conditions (11 M/10 F, 22-61 years old). Each subject had two MRI scans with a short break between them. A number of structural and functional images were acquired in each scan. We use MPRAGE (1.2 mm × 1.0 mm × 1.0 mm resolution), T2-weighted (1.0 mm × 0.94 mm × 0.94 mm resolution), DTI (32 directions, b-value = 700 s/mm², 0.83 mm × 0.83 mm × 2.2 mm), and ASL images (1.87 mm × 1.87 mm × 7.95 mm resolution) of this dataset. The image acquisition parameters are given in [20]. DTI is a promising method for in vivo characterization of the microstructural properties of WM [23]. ASL is used for measuring tissue perfusion using a freely diffusible intrinsic tracer [24]. We process the images of each subject as follows.

We automatically segment the whole brain of scan 1 into several regions by non-rigid registration of the Harvard-Oxford atlas [25], distributed with FMRIB Software Library (FSL, http://www.fmrib.ox.ac.uk/fsl/), to the MPRAGE image using ANTS [26]. Since this atlas does not have a label for the CC, we manually outlined the CC and added it to the atlas. We also use a segmentation algorithm [27] in FSL to segment the MPRAGE image into WM, GM, and CSF. We then map both generated label volumes to the T2W image of scan 1 (see Fig. 3) using rigid coregistration between MPRAGE and T2W images in SPM (http://www.fil.ion.ucl.ac.uk/spm/). We generate FA and mean diffusivity (MD) maps from DTI data of both scans using FSL and coregister them to the T2W image of scan 1 using rigid coregistration between B0 and T2W images. We also generate cerebral blood flow (CBF) maps from ASL images of both scans using the method described in [20] and map them to the T2W image of scan 1 by rigid coregistration between ASL and T2W images in SPM. Therefore, FA, MD, and CBF maps of both scans as well as two label maps will be available in the space of T2W image of scan 1 (Fig. 3). In this way, although the original resolutions of the parametric maps were different, we analyze them all in the same space, i.e., space of T2W image. We normalize CBF maps to the CBF mean value within the WM mask generated by FSL in order to obtain relative CBF values. The accuracy of atlas-based segmentation is not a concern here since the same segmentation outcome is used for both scans. We conduct three studies:

Fig. 3.

Sample parametric maps and label volumes in the space of T2W image of scan 1. Row one: Sample T2W image along with FA, MD, and CBF maps. Row two: Label maps of WM (red), GM (white), and CSF (blue); cross-section of a selected region in the parietal lobe; segmentation of the selected region into WM, GM, and CSF; and label map of the CC.

Study 1

Here we are interested in studying the effect of ROI size on region-based analyses of FA, MD, and CBF, as well as in verifying our statistical model and the estimation technique. We select three anatomical regions as follows. We select a segmented region in the parietal lobe and use tissue segmentation labels to divide the region into WM and GM sub-regions (see Fig. 3). FA, MD, and CBF are analyzed within these two regions as well as within the CC. All three anatomical regions are analyzed as follows. We randomly generate ROIs with S_T voxels from each region and then apply the proposed algorithm to generate curves of RC, COV, and ICC versus ROI size and average over 100 repetitions. We then calculate the MSE between the estimated values and their ground truth. Implementation details including the method to generate near-spherical shaped ROIs and the ground truth are presented in the Supplementary Material, Section V.

Study 2

The aim of this study is just to show how the problem of smoothing in voxel-wise analysis is related to the effect of ROI size. In the analysis of functional maps, such as functional diffusion maps [28], smoothing can change RC and the required threshold for determining the changed regions as it increases the effective ROI size by combining the voxels through weighted averaging the original voxels. We apply a Gaussian filter with full-width half maximum (FWHM) of 3.5 mm and repeat study 2 to show the effect of smoothing on the repeatability of voxel-wise analysis.

III. Results

A. Simulation

The results are presented in Fig. 4. In all figures, the volumes of ROIs are given in terms of the number of voxels in the space of T2W image (i.e., voxel sizes of 1.0 mm × 0.94 mm × 0.94 mm). As shown, the mean of estimated values obtained from our proposed hybrid estimation method are close to the approximated true mean. The MSE values are also small compared to the repeatability measures. The variances of the estimated values are smaller or comparable to the corresponding variances when estimated conventionally using the sample variance function. This is a desirable property of the proposed method. Variance and MSE values of the estimations using the proposed method are smaller for smaller ROIs since all voxels within the 10×10×10 ROI are used for the estimation of the repeatability measures (and more samples means lower variance). For ICC, the estimated value is slightly lower than its ground truth when the sub-sample ROI size is large. This is because the randomness between subsamples decreases as the ROI size increases and consequently the information obtained from sub-samples become quite similar. For the variance of ICC, there is a difference between the values generated by the model (ground truth calculated by formulas provided in the Supplementary Material) and those calculated conventionally, which is caused by the limitation of the approximation of var(ICC) provided in equation (S-15) in the Supplementary Material.

Fig. 4.

Mean values (row 1), variances (row 2), and MSE values (row 3) of the repeatability measures (RC, COV, and ICC) estimated using our proposed hybrid method from simulated data points denoted by “Estimated”, standard method using their definition denoted by “Standard Calculation”, and using the true model parameters denoted by “Model”. MSE curves (row 3) were generated by comparison with the true model. The data points were generated by 5000 simulations using the model proposed in equation (1) with normal distribution.

B. Brain MRI dataset

Study 1

Fig. 5 shows the estimated values of repeatability measures for FA, MD, and CBF maps with S_T = 2000, S_T = 1000, and S_T = 2500, for WM, GM, and CC, respectively. To choose four ROI sizes for curve fitting, we divided the interval [1, S_T] into four equi-distant intervals in the logarithmic scale. Specifically, we chose indices 1, 6, 12, and 22. We picked point 22 instead of 23 (i.e., S_T voxels) to avoid uncertainly of a single sample point. As shown, there is a good agreement between the mean curves of the estimated measurements and their ground truth. The MSE averaged over different ROI sizes had maximum values of 7.1e-4, 1.6e-4, and 8.7e-4 for RC, COV, and ICC, respectively.

Fig. 5.

Average RC, COV, and ICC of mean FA, mean MD, and mean CBF within WM and GM of a segmented region in the parietal lobe as well as CC versus ROI size, estimated using our proposed method, denoted by “Estimated”, along with their ground truth values denoted by “Standard Calculation”. We assumed S_T = 2000, S_T = 1000, and S_T = 2500 voxels were available for WM, GM, and CC, respectively, and obtained the averages over 100 repetitions.

We observe that the rate of change in the repeatability measures is higher for smaller ROI sizes. In a voxel-wise analysis where the effective ROI size is one voxel, RC is at its maximum and changes dramatically by increasing the ROI size. When the image is smoothed or interpolated into a different resolution or space, the effective ROI size may change as the voxels are combined. This affects the repeatability coefficients and is studied next.

Study 2

The results of studying the effect of smoothing on the repeatability of voxel-wise analysis are presented in Fig. 6. By comparing Figs. 5 and 6, we observe that smoothing substantially changes the repeatability measures of FA and MD, in particular in the case of ROI with size of one voxel or equivalently voxel-wise analysis. The effect of smoothing on the repeatability measures of CBF was smaller compared to FA and MD due to the large voxel size of the original CBF maps and thus smoother voxels in the space of T2W images making them less affected by smoothing.

Fig. 6.

Average RC, COV, and ICC of mean FA, mean MD, and mean CBF within WM and GM of a segmented region as well as CC versus ROI size, after smoothing the images and estimated using our proposed method, denoted by “Estimated”, along with their ground truth values denoted by “Standard Calculation”. Comparing with Fig. 5, we can observe the effect of smoothing on the repeatability measures.

IV. DISCUSSION

We presented a statistical model along with a parameter estimation method to estimate repeatability measures for brain ROIs with arbitrary sizes. This is important in treatment planning when the knowledge about whether a change has occurred in a tissue is desirable, e.g., in monitoring changes in MRI-based parametric maps within brain tumors following treatment [2] [3] [4] [5]. The threshold based on which a decision is made (e.g., RC of measurement) can be adjusted according to each individual tumor size using our proposed method. However, note that the threshold also depends on the imaging protocol and vendor. Thus, a relation between threshold and ROI size needs to be derived for each imaging protocol and vendor.

We observed that the major changes in repeatability measures occur for small ROIs. Thus, adjusting the threshold becomes more important for small lesions, such as brain lesions in multiple sclerosis with reported average size of 36 mm² in 2D MRI scans [29]. Our proposed method may be used for monitoring treatment of multiple sclerosis using various imaging biomarkers [7] [8] [9] [10] and in particular myelin water fraction using FAST-T2 [6]. Other applications that may involve small lesions include monitoring brain metastases by diffusion-weighted MRI [30] [31] and MRI perfusion [32], and brain focal lesions by MRI in mild traumatic brain injury [33]. Obviously, if a particular structure (e.g., hippocampus) is studied by ROI-based method, then the ROI size has smaller range of variations and our method may not be applicable.

Model parameters depend on factors such as tissue type, the extent of partial volume effect, and the signal-to-noise ratio (SNR) of the image within the ROI. For example, regions closer to the brain center usually have lower SNR due to the lower coil sensitivity in those regions. In our method, we had assumed that the ROIs under study had similar model parameters, with the ROI size or shape the only difference between them. We tested the proposed method for ROIs containing one tissue type. To test whether the proposed method practically works for ROIs containing multiple tissue types, we repeated Study 1 in Section II for WM+GM. The method accurately estimated the repeatability measures (results presented in Fig. S1 of the Supplementary Material).

Although we assumed all the random effects in model (1) followed a normal distribution, this assumption was not used in the derivation of first-order Taylor series approximation of the mean of the repeatability measures. It was only used to approximate means and variances using the second-order Taylor approximation (see the Supplementary Material). This assumption may not be valid in practice especially when the measurements have some intrinsic limited range of variations, for example in the case of FA values that range from 0 to 1. Despite this, the curves of the repeatability measures generated for FA, MD, and CBF using the second-order Taylor approximation for the dataset of repeated MRI scans had good agreement with their defined ground truth. This indicates that the proposed method is robust to the violation of the normality assumption.

Since the repeatability measures depend on the number of voxels and that in turn depends on the resolution, the question arises as to what resolution we should select given a fixed acquisition time. In MRI, the tradeoff between SNR and resolution is nonrecoverable [34] [35] and thus we need to find the best compromise. We note that the random error ε_ijk of the parametric map in model (1) depends on both SNR and the resolution of MRI. This is because ε_ijk models the overall effect of sources of variation such as segmentation and registration errors, partial volume effect, and noise. Thus, the best repeatability measures are not obtained by maximizing SNR. Unfortunately, deriving an analytical solution is not trivial and experiments need to be conducted to get the best compromise.

The problem of ROI size is also related to some other problems that involve ROIs with small size. Here we review several of such cases in the neuroimaging studies. Functional diffusion maps have recently become popular as an early and quantitative biomarker of clinical brain tumor treatment outcome [28]. In this technique, longitudinal maps of MD are generated from DTI data and mapped to a common space, typically a T2W image of the same subject. Voxel-wise differences of MD maps, before and after therapy, are then generated. Using a predetermined thresholded Δ, the tumor is classified into three regions of increased MD, decreased MD, and no change if the difference is above Δ, below Δ, and between −Δ and Δ, respectively. These regions are then studied to evaluate response to therapy. A threshold of 4×10⁻⁴ mm²/sec has been suggested in the literature [28] although more specific thresholds ranging from 2.5×10⁻⁴ mm²/sec to 7.5×l0⁻⁴ mm²/sec have been later proposed for WM, GM, their combination, and combination of WM, GM, and CSF [36]. These reported thresholds correspond to RC values and agree with our RC values for ROI size of one voxel (Fig. 5).

Spatial smoothing may be carried out (typically Gaussian filtering with FWHM of 4 mm to 12 mm) before running standard voxel-wise statistics and thresholding [12]. We applied a Gaussian filter with FWHM of 3.5 mm and showed that it decreased the RC values of FA and MD approximately by half (compare Figs. 5 and 6). Since smoothing can be regarded as a weighted averaging, it has the effect of increasing the effective ROI size if it is done within one tissue type to avoid partial volume effect. As β_ij and ε_ijk in model (1) are the only terms that have spatial information (i.e., change from voxel to voxel), smoothing will affect both, whereas the goal is to reduce the effect of error term ε_ijk while keeping β_ij intact. The right amount of smoothing may be decided based on a compromise between improving the repeatability and preserving the spatial information. Knowing the desired repeatability measures depending on the application, the smallest filter size can be estimated using the curves of the repeatability measures versus filter size. In the case of average filtering, our proposed method can generate the repeatability curves. Determining the desired repeatability measure will be based on the particular application and the data, and is out of scope of this paper.

In some image acquisitions, parametric maps are generated and standardized (normalized) using a reference region, which may be small. Examples include dynamic contrast enhanced (DCE) MRI, dynamic susceptibility contract (DSC) MRI, ASL, and positron emission tomography (PET). In DCE- and DSC-MRI, the parametric maps are generated by deconvolution of the dynamic image with a reference signal called arterial input function (AIF). AIF is the MRI signal of a feeding artery and may be determined from a few automatically or manually detected voxels. Reliable estimation of AIF is essential for a reliable generation of perfusion maps. AIF voxels need to be small enough in order to avoid the problem of partial volume effect. But as repeatability of measurements is relatively poor for smaller ROIs, measurements based on AIF determined from a small number of voxels may not be reliable. The size of AIF ROI could be set to the smallest one that gives the desired repeatability measures. In ASL, CBF is dependent on the equilibrium signal intensity within the CSF calculated from the inversion recovery sequence by segmentation of the CSF region [37]. As ASL images have relatively large voxels, the CSF may contain small number of voxels, in particular when the CSF is small. This may affect the repeatability of CBF measurements. Similar issues exist in other imaging techniques that use reference tissue, such as PET imaging [38] [39].