Brain Extraction in Pediatric ADC Maps, toward Characterizing Neuro-Development in Multi-Platform and Multi-Institution Clinical Images

Abstract

Apparent Diffusion Coefficient (ADC) maps can be used to characterize myelination and to detect abnormalities in the developing brain. However, given the normal variation in regional ADC with myelination, detection of abnormalities is difficult when based singularly on visual assessment. Quantitative and automated analysis of pediatric ADC maps is thus desired but requires accurate brain extraction as the first step. Currently, most existing brain extraction methods are optimized for structural T1-weighted MR images of fully myelinated brains. Due to differences in age and image contrast, these approaches do not translate well to pediatric ADC maps. To address this problem, we present a multi-atlas brain extraction framework that has 1) specificity: designed and optimized specifically for pediatric ADC maps; 2) generality: applicable to multi-platform and multi-institution data, and to subjects at various neuro-developmental stages across the first 6 years of life; 3) accuracy: highly accurate compared to expert annotations; and 4) consistency: consistently accurate regardless of sources of data and ages of subjects. We show how we achieve these goals, via optimizing major components in a multi-atlas brain extraction framework, and via developing and evaluating new criteria for its atlas ranking component. Moreover, we demonstrate that these goals can be achieved with a fixed set of atlases and a fixed set of parameters, which opens doors for our optimized framework to be used in large-scale and multi-institution neuro-developmental and clinical studies. In a pilot study, we use this framework in a dataset containing scanner-generated ADC maps from 308 pediatric patients collected during the course of routine clinical care. Our framework leads to successful quantifications of the changes in whole-brain volumes and mean ADC values across the first 6 years of life.

1. Introduction

ADC maps measure the magnitude of (intra- and extra-cellular) water diffusion [LeBihan86, Neil98]. They provide valuable information about diffusivity and myelination in the central nervous system [Grant01, Woodhams05, Verma08, Gerstner10, Menze14]. Pediatric ADC maps have been found to be very useful to characterize myelination-related early neuro-development [Snook05, Twomey10, McKinstry11, Yoshida13, Erus14, Ou14HBM, Retzepis14], and to identify neuro-developmental abnormalities [Barkovich00, Hüppi06, Dubois14, Guleria14]. For both purposes, automatic analysis of pediatric ADC maps is preferred. This, however, requires brain extraction as one of the first preprocessing steps, which is a challenging problem with a lack of algorithms specific to pediatric ADC maps. Here, brain extraction, or skull stripping, refers to an image processing step to remove the extra-meningial tissues and non-brain structures (eyes, nose, neck, etc.) from whole-head images (e.g., the red contours in Figure 1).

Figure 1.

Brain extraction in a pediatric ADC map (top row) and an adult structural (T1w) image (bottom row). Red contours are manually annotated brain boundaries. The differences in age and imaging modality are clearly visible.

Existing brain extraction approaches have been primarily developed for structural T1-weighted MR images of fully myelinated brains of teens and adults. They use morphological operators (e.g., [Dogdas05, Chiverton07], deformable surface evolution (e.g., [Smith02]), edge detection (e.g., [Shattuck02]), region growing (e.g., [Park09]), level sets (e.g., [Zhuang06]), watershed (e.g., [Hahn00, Segonne04]), graph cut (e.g., [Sadananthan10, Mahapatra12]), atlas priors (e.g., [Leung11, Doshi13a]), and hybrid techniques (e.g., [Iglesias11, Eskildsen12, Galdames12]). These approaches are built on age- and imaging-modality as well as contrast-specific assumptions that do not completely hold true for pediatric ADC maps. In terms of the difference in age, adults have a more mature anatomy; also, due to less motion and the completed myelination in adults, their images tend to have higher tissue contrast, a higher signal-to-noise ratio (SNR) and higher voxel resolution than images of pediatric subjects; and, there is a larger distance between the outer surface of the brain and the inner surface of the skull in adults as well [Hoeksma05, Shi12, Mahapatra12]. In terms of the difference in MR imaging sequences, structural T1-weighted images show non-brain structures (e.g., skull, eyes, nose, mouth and neck) more clearly than diffusion ADC maps, and have different inherent distortions [Dubois14b, Counsell06, Giménez08]. Figure 1 displays an adult structural (T1-weighted) image and a pediatric ADC map, where the differences in age and imaging sequence are clearly visible. Because of these fundamental differences, brain extraction approaches for adult structural images usually do not directly apply to pediatric ADC maps. The inapplicability can be seen in our experiment (Table III) and further experiments in recent literature [Shi11, Mahapatra12, Geng12b].

Table III.

Brain extraction accuracies (Dice coefficients) by BET, BSE and our proposed framework. The default parameters of BET and BSE are primarily for adult brain images and structural MR images, so they are not necessarily directly applicable to pediatric ADC maps. Tuning the parameters of BET and BSE to the MGH_3T pediatric ADC maps shows improvement but does not perform as well as the proposed work. For how we optimized BET and BSE for this task, please refer to Section 3.7.

	MGH_3T	MGH_1.5T	BCH_3T
BET (default parameters)	0.795 ± 0.101	0.748 ± 0.029	0.853 ± 0.057
BSE (default parameters)	0.334 ± 0.160	0.162 ± 0.110	0.154 ± 0.115
BET (optimized in MGH_3T)	0.813 ± 0.117	0.835 ± 0.044	0.855 ± 0.059
BSE (optimized in MGH_3T)	0.881 ± 0.035	0.867 ± 0.039	0.861 ± 0.047
Ours (optimized in MGH_3T)	0.958 ± 0.011	0.929 ± 0.023	0.936 ± 0.020

A few brain extraction methods do exist for pediatric T1 weighted images [Chiverton07, Mahapatra12, Shi12]. However, directly applying them to pediatric ADC maps may also encounter problems, due to differences in contrast properties and different types of distortions of the two image sequences. Other approaches exist for other image modalities of infant or pediatric subjects (e.g., for T2-weighted MRI of infants [Leroy11], for post mortem MRI of newborns and fetus [Orasanu14], for infant structural MRI with expert post-correction of brain extraction results [Wang12], for fetal brain MRI [Kuklisova-Murgasova12]) or for ADC maps of adults (e.g., [Liu07]). However, there is a lack of a validated algorithm for neonatal/pediatric ADC maps. Consequently, one either has to (semi-) manually annotate the diffusion images (e.g., [Hasan07, Irimia11]), or transfer the brain mask obtained from the structural image to the ADC map of the same pediatric subject. The former is a subjective and not fully reproducible process. The latter requires an additional structural-to-diffusion multi-modal registration that is non-trivial even in the case of adult images [Geng12b, Malinsky13].

Two other very important factors need to be considered when designing brain extraction algorithms for pediatric ADC maps. First, MRI scans of normative pediatric subjects are rare. Clinicians order MRI scans for infants and pediatric subjects with great caution, even though sedation is not always required [Kulikova14]. Additionally, even fewer datasets exist for research purposes, due to the difficulty in recruiting volunteers and the safety concerns [Barkovich12, Arthurs13, Edwards11, Rozovsky13]. As a result, many studies of neuro-development have to pool together clinical images collected from multiple institutions to assure statistically significant outcomes (e.g., [Evans06]). Therefore, it is preferable for an algorithm designed for pediatric images to be applied and validated in images from different sources (institutions, scanner platforms and scanner vendors). This is a very challenging requirement, given that images from different data sources may differ greatly in resolution, contrast properties, sequences, fields-of-views (FOVs) and distortions [Kirişli10, Hameeteman11, Guggenberger13, Malyarenko13, Kivrak13, Ou14TMI]. The second factor to consider is the inter-subject structural variations among pediatric subjects even at the same age, which are greater than among adolescents and adults. This is due to the rapid neuro-development in neonatal and early pediatric stages. Therefore, an ideal algorithm should also be robust to the large structural variations in pediatric subjects of the same or different ages.

Considering the above, our goal is to develop a brain extraction framework with 4 desirable properties: 1) specificity: designed specifically for pediatric ADC maps; 2) generality: applicable to data from different sources (institutions, platforms), and to subjects of different ages across the first 6, and in particular the first 2, years of life (during which the neuro-development is most dynamic [Kazemi07, Knickmeyer08, Geng12, Mori13]); 3) accuracy: highly accurate compared to expert annotations; and 4) consistency: consistently accurate in different data sources and subject ages.

This paper presents a multi-atlas framework for the brain extraction of pediatric ADC maps. An atlas in our framework is comprised of a pediatric ADC map and a corresponding expert-annotated brain mask (see Figure 2). We used multiple atlases to collectively infer the brain mask in the target images. We optimized each component in the framework and proposed new criteria to improve the atlas ranking component. The optimization of the framework was done in one dataset and was then tested in 2 other unseen datasets acquired at a different institution or on a different scanner.

Figure 2.

The proposed multi-atlas brain extraction framework. Given a set of atlas ADC maps {I_i}^N_i=1 and their corresponding expert-annotated brain masks {L_i}^N_i=1, and the ADC map I_T to be processed, this framework aims to compute the brain mask L_T in the space of I_T.

Having a multi-atlas framework is not our contribution. Such frameworks have been frequently used to segment brain (e.g., [Heckemann06, Lötjönen10, Sabuncu10, Wang11, Cabezas11, Ou12b, Iglesias13, Wu13, Wang14, Doshi13b]), prostate (e.g., [Klein08, Ou12a]), heart (e.g., [Isgum09, vanRikxoort10]) and other organs. Two multi-atlas frameworks even exist for skull stripping adult structural images [Leung11, Doshi13a]. Our contributions are: (i) our framework is the first algorithm specifically designed to skull strip clinical pediatric ADC maps; it is fully-automated after having obtained a set of expert-annotated atlases; (ii) unlike many other multi-atlas frameworks, ours does not require adoption or annotations of additional, often dataset-specific, atlases when new images/datasets are to be processed, nor does our framework require parameter tuning; and (iii) we have shown that an optimized framework can reach the desired specificity, generality, accuracy and consistency, which have been rarely demonstrated in the literature even for adult images. We further used the proposed framework in a pilot neuro-developmental study containing clinical ADC maps from a cohort of 308 normative pediatric subjects. As a result, we quantified changes in the whole-brain volume and in mean ADC values across the first 6 years of life.

2. Data

The Institutional Review Boards (IRBs) at both Partners Healthcare and Boston Children’s Hospital approved our study. We queried and retrieved clinical pediatric ADC maps for subjects 1) younger than 6 years of age at the time of the scan, and also 2) with clinical radiology reports and clinical records documenting the absence of major neurological disease. This data query and retrieval was made possible by the Research Patient Data Registry (RPDR)¹ data warehouse, and the recently released Medical Imaging Informatics Bench to Bedside (mi2b2²) workbench [Murphy14]. The clinical data obtained from the RPDR query, including the Radiology Reports, were analyzed using database queries to identify potential normative cases that were then confirmed by an expert review of the medical records. The mi2b2 workbench was then used to access the brain MRI scans of these potential cases directly from our institutions’ clinical Picture Archiving and Communication System (PACS), in compliance with the Health Insurance Portability and Accountability Act (HIPAA) Omnibus Rule.

Three datasets with expert annotations (Section 2.1) were used for algorithm development and validation. Another, relatively large dataset (Section 2.2), was used for applying the optimized framework in a whole-brain volumetric and ADC neuro-development study.

2.1. Three Datasets for Algorithm Development and Validation

As detailed in Table I, we used 3 datasets acquired on different scanner platforms from different institutions for development and validation of our framework. These datasets are: MGH_3T (N=15), MGH_1.5T (N=10) and BCH_3T (N=14). Each dataset contains ADC maps of multiple pediatric subjects, which were retrieved as described in Section 2. A trained expert (KR) annotated brain masks in all datasets, which we used as references in our experiments. The annotations took 0.5–1 hour per subject, using the FreeView³ visualization and annotation tool from the FreeSurfer software package [Fischl12].

Table I.

The 3 datasets that we used for developing and validating our framework.

Datasets	MGH_3T	MGH_1.5T	BCH_3T
Imaging institution	Massachusetts General Hospital (MGH)	Massachusetts General Hospital (MGH)	Boston Children’s Hospital (BCH)
MR scanner	3T Siemens Trio	1.5T Siemens Avanto	3T Siemens Trio
Repetition Time (TR)	7500–9500 ms	6500–8000 ms	7000–8000 ms
Echo Time (TE)	80–115 ms	80–100 ms	80–100 ms
b value	1000 s/mm2	1000 s/mm2	1000 s/mm2
Sequence	Echo Planar, segmented k-space \ spoiled, partial Fourier-Phase, fat saturation.	Echo Planar, Research Mode, Partial Fourier-Frequency, Spatial Presaturation, Fat saturation.	Echo Planar, segmented k-space, Spatial Presaturation, partial Fourier-Phase.
Typical image size	256×256×60 voxels	256×256×50 voxels	128×128×70 voxels
Typical voxel size	0.86×0.86×2.0 mm3	0.86×0.86×2.2mm3	1.7×1.7×2.0mm3
Manually-annotated?	Yes, brain masks	Yes, brain masks	Yes, brain masks
Number of subjects	15	10	14
Age distribution	Month 1 (4 subjects) Month 3 (1 subject) Month 4 (1 subject) Month 6–12 (4 subjects) Year 1–2 (1 subject) Year 3–4 (2 subjects) Year 4–5 (1 subject) Year 5–6 (1 subject)	Month 1 (1 subject) Month 4 (1 subject) Month 6–12 (2 subjects) Year 1–2 (2 subjects) Year 2–3 (1 subject) Year 3–4 (1 subject) Year 4–5 (1 subject) Year 5–6 (1 subject)	Month 1 (4 subjects) Month 2 (1 subject) Month 4 (1 subject) Month 6–12 (2 subjects) Year 1–2 (1 subject) Year 2–3 (1 subject) Year 3–4 (1 subject) Year 4–5 (1 subject) Year 5–6 (1 subject)

2.2. One Dataset for Whole-Brain Neuro-Development Study

We also applied the proposed framework to a relatively large dataset named Pediatric Dataset 308, or PD308 for quantitatively characterizing volumetric and diffusion aspects of neuro-development from birth to 6 years old. This PD308 dataset contains clinical ADC maps from N=308 pediatric subjects younger than 6 years of age at the time of the scan. The ADC maps were all acquired by the same protocol as in MGH_3T. A pediatric intensive care nurse (KM) confirmed that they were normative at the time of the scan, by reviewing their radiological reports and clinical records. Table II lists the age distribution of this dataset.

Table II.

Age distribution in the PD308 dataset (W-Week, Q-Quarter and Y-Year).

Age	Y0–1					Y1–2	Y2–3	Y3–4	Y4–5	Y5–6
	W1–2	Rest of Q1	Q2	Q3	Q4
# Subjects	20	22	13	17	25	47	45	38	31	50

3. Methods

3.1. Overview

Formulation

We denote an atlas A_n=(I_n, L_n), which comprises of an ADC map I_n, and the associated manually-annotated brain mask L_n. Similarly, we denote a target as T=(I_T, L_T), where I_T is the input ADC map (i.e., the target image), and L_T is the brain mask to be calculated in the target image space.

Framework

Similarly to other multi-atlas consensus frameworks (e.g., [Heckemann06, Klein08]), our multi-atlas brain extraction has the following components (also illustrated in Figure 2):

• a)a fixed library of N=15 atlases (Section 3.2);
• b)deformable registration and label propagation (Section 3.3). We used the deformation field h_n: I_n->I_T, calculated from ADC maps, to transfer the atlas brain mask L_n into the target space, i.e., h_n(L_n).
• c)atlas ranking (Section 3.4) and d) atlas selection (Section 3.5) for choosing the most relevant atlases;
• e)label fusion to generate final target brain mask L_T (Section 3.6).

In the following, we describe each component in the framework. The emphasis is on how we selected the optimal methods for each component, in order to establish a framework with the desirable generality, accuracy and consistency.

3.2. Atlas Library

Our atlas library contains the 15 ADC maps from the MGH_3T dataset. To represent the anatomical variations due to rapid neuro-development, we chose a bigger number of subjects that were scanned during the first year (especially during the first month) of their lives, and also some representative ones up until 6 years of age. Figure 3 shows the central sagittal slices of the chosen atlases. They had different contrasts and signal-to-noise ratios. Please note that some atlases (namely 2, 5, 6, 12, 14, 15) have different fields of views (FOVs), where some slices in the top of the brain are missing. This alone highlights a huge variability within the same dataset. Also, we point out that the chosen atlases are not spatially aligned in Figure 3. Thus, the central sagittal slices in different subjects may not exactly correspond across atlases.

Figure 3.

The library of 15 ADC atlases with manual annotations of brain masks shown by red contours. ADC atlases are 3D images, but for display purpose, this figure shows only a single sagittal slice. Ages are also shown.

3.3. Optimizing the Registration and Label Propagation Component

One consequence of having a fixed atlas library is that the atlases and the target may come from datasets of different sources (i.e., institutions, platforms). To this end, we need a deformable registration algorithm that is robust to inter-subject structural variability, and also robust to challenges in data acquired from different sources.

We tested four registration algorithms to see which one can satisfy this need. If an algorithm could demonstrate generality, accuracy and consistency regarding single- and multi-institution/platform images, then selecting it would optimize the registration and label propagation components in our framework. The four candidate algorithms that we tested are frequently used in the literature with demonstrated high accuracy [Klein09, Ou14TMI], are publicly available and are general-purpose. Three of the algorithms are intensity-based: ANTs⁴ (the “SyN” symmetric diffeomorphism option) [Avants08], Diffeomorphic Demons⁵ [Thirion98, Vercauteren09], and FSL’s FNIRT⁶ [Andersson08] and the fourth one is a texture-attribute-based one called DRAMMS⁷ [Ou11]. They represent a variety of transformation models, image similarity metrics, and optimization strategies (please see [Klein09, Ou14TMI] for detailed reviews of their algorithmic differences). In the literature, the performance of these algorithms has been extensively evaluated with respect to adult brain images [West97, Hellier01, Christensen06, Yassa09, Klein09, Klein10, Avants11, Diez13, Ou14TMI], and occasionally in pediatric structural images (4–11 years of age) [Ghosh10]. However, no comparable evaluation was completed for how they would perform in pediatric brain ADC maps.

We performed a total of 2,280 inter-subject registrations to evaluate these 4 algorithms in 3 different scenarios. In the first scenario, the two pediatric ADC maps to be registered were both from the MGH_3T dataset (within-dataset registrations). We performed all possible pair-wise registrations among the 15 ADC maps in this collection, resulting in 210 (=15×14) registrations for each algorithm. In the second scenario, the two ADC maps to be registered were from datasets acquired in different scanner platforms (across-scanner-platform registrations). We registered each of the 15 ADC maps in our atlas library (all from the MGH_3T dataset) to each of the 10 ADC maps in the MGH_1.5T_dataset. This resulted in 150 pair-wise registrations for each algorithm. In the third scenario, the two ADC maps to be registered were from datasets acquired at different institutions (across-institution registrations). We registered each of the 15 ADC maps in our atlas library to each of the 14 ADC maps in the BCH_3T dataset. This resulted in 210 pair-wise registrations for each algorithm. Therefore, there are 210+150+140 registrations per each of the 4 registration algorithms, totaling at 2,280 registrations.

The deformation fields that were computed when registering two pediatric ADC maps were used to resample/deform the corresponding manually-annotated brain masks into a common image space. Using these, we evaluated registration accuracy by computing the Dice coefficient [Dice45] between the deformed brain mask from the atlas to the target space and the brain mask of the target image. The registration algorithm with a higher Dice coefficient score was regarded as being better suited for the brain extraction task in our framework. Additionally, we also computed the Hausdorff Distance [Huttenlocher93] between the deformed manually-annotated brain masks. Since the Dice coefficients and Hausdorff Distances agreed in their accuracy rankings, we only report the results of the Dice coefficients in this paper. Let P be the manually-annotated atlas mask deformed into the target space, and Q the manually-annotated brain mask in the target space. The Dice coefficient between them is defined as

$$\mathit{DICE}(P,Q)=\frac{2\ast V(P\cap Q)}{V(P)+V(Q)},$$ (1)

where V(·) is the volume of a region.

3.4. Optimizing the Atlas Ranking Component

Atlases with inferior registration will cause poor inference regarding the target mask. To optimize the atlas ranking component in our framework, we propose a set of novel atlas ranking criteria, and compare them with respect to several existing ones.

Atlas ranking is a much studied problem. It is not feasible to directly measure and rank how accurately an atlas can infer the segmentation of the target image, since the ground truth for target segmentation is unknown (and to be computed). Existing studies instead rely on atlas ranking criteria that consist of demographic information (e.g., age, gender, race, etc.), and/or atlas-to-target image similarity measures [Klein08, Aljabar09, Lötjönen10, Leung11, Langerak13, Bai13]. The assumption is that those atlases that can better infer the target segmentation originate from subjects having similar demographic information to the target subject, or the ones having higher atlas-to-target image similarities. This assumption, however, was recently found to be biased [Wolz10, Yang13, Duc13, Sanroma14]. The fundamental problem is that intensity-based similarity metrics do not faithfully reflect registration accuracy [Klein09, Rohlfing11, Avants11, Ou12c, Ou14TMI, Ou14MRM], and hence do not necessarily imply the accuracy of the inferred segmentation. To address this issue, new ranking criteria have been proposed. For example, [Sanroma14] suggested a supervised machine learning approach, assuming that the utility of an atlas can be implied and learnt by image appearances and a more comprehensive set of registration properties other than purely image similarity. This supervised training process may have to be reestablished for each new dataset to be analyzed, especially when the new datasets are from different institutions or scanners of different platforms. Thus, although it improves the atlas ranking accuracy, it is not an optimal choice in our framework.

For the generality of our framework, we instead proposed several simple and unsupervised atlas ranking criteria. Whether an atlas is useful in deriving the brain mask of a target subject is determined by how accurate the atlas-to-target registration is. So, ranking atlases translates to measuring and ranking the accuracy of registering different atlases onto the same target. We thus measure registration accuracy by: (a) the Dice overlap coefficient between the deformed atlas brain mask in the target space and the tentative target brain mask (TTM). Here TTM is obtained by fusing the atlas-implied masks from all atlases, which we assume to be a good approximation of the unknown brain mask in the target image space; (b) intensity similarity (MI, CC) between the deformed atlas and the target ADC maps. Since intensity similarity alone is not necessarily an accurate indicator for registration performance [Klein09, Rohlfing11, Avants11, Ou12c, Ou14TMI, Ou14MRM], we used the overlap measurement alone, or the overlap measurement with the intensity similarity as in the three proposed ranking criteria below:

$$\mathbf{EstDice}=\text{DICE}(h_n(L_n),{L_T}^{\mathit{TTM}})$$ (2)

$$\mathbf{EstDice}\mathbf{\ast }\mathbf{CCdef}=\text{DICE}(h_n(L_n),{L_T}^{\mathit{TTM}})\ast \text{CC}(h_n(I_n),I_T)$$ (3)

$$\mathbf{EstDice}\mathbf{\ast }\mathbf{MIdef}=\text{DICE}(h_n(L_n),{L_T}^{\mathit{TTM}})\ast \text{MI}(h_n(I_n),I_T)$$ (4)

CCdef and MIdef refer to the correlation coefficient CC(.,.) and mutual information MI(.,.) image similarity metrics between the registered ADC maps h_n(I_n) and I_T. EstDice is the Dice coefficient between the atlas-implied brain mask h_n(L_n) and TTM L_T^TTM = FUSE({h_n(L_n)}^N_n=1), where we used the STAPLE (Simultaneous Truth And Performance Level Estimation) label fusion tool [Warfield04] to implement the FUSE() function (more in Section 3.6 and 4.1.3). Please note that, the novelty in Equation 2 is the use of tentative template mask (L_T^TTM), which we hypothesize and will later show is a good approximation of the true yet unknown brain mask.

We evaluated the performance of the above three atlas ranking criteria by comparing them with four conventional image-similarity-based criteria listed below [Klein08, Aljabar09, Lötjönen10, Leung11, Langerak13, Bai13].

$$\mathbf{CCaff}=\text{CC}({\mathrm{a}}_n(I_n),I_T);$$ (5)

$$\mathbf{MIaff}=\text{MI}({\mathrm{a}}_n(I_n),I_T);$$ (6)

$$\mathbf{CCdef}=\text{CC}(h_n(I_n),I_T);$$ (7)

$$\mathbf{MIdef}=\text{MI}(h_n(I_n),I_T).$$ (8)

In the above, a_n(I_n) and h_n(I_n) are the transformed atlas ADC maps, by affine (a(.)) and deformable (h(.)) registration, respectively. For affine registration, we used FLIRT [Jenkinson01]; for deformable registration, we used the one that empirically performed best in our study (see Section 3.3 and 4.1.1).

To quantify the atlas ranking accuracy, we measured the errors between the true and estimated ranks. We determined the true rank (R^True_n,T) of an atlas (A_n) with regard to target T based on the Dice coefficient between the atlas-implied brain mask and the expert-annotated target brain mask. We determined the estimated rank (R^Est_n,T) based on the criteria in Eqs. 2–8. The average ranking error (ARE_T) with respect to target T is defined as:

$${\text{ARE}}_T=\frac{1}{\mathrm{N}}\sum _{\mathrm{n}=1}^{\mathrm{N}}|{\mathrm{R}}_{\mathrm{n},\mathrm{T}}^{\text{Est}}-{\mathrm{R}}_{\mathrm{n},\mathrm{T}}^{\text{True}}|.$$ (9)

3.5. Optimizing the Atlas Selection Component

Selecting multiple atlases (e.g., [Rohlfing04, Han08]) usually leads to higher segmentation accuracy than selecting merely a single top ranked atlas (e.g., [Park03, Martin10]). We evaluated two strategies with various parameter settings. One strategy was to keep K (a user-specified number) top-ranked atlases [Aljabar09, Leung11]:

$$\text{keep}\text{atlas}{A}_n\text{if}{\mathrm{R}}_{\mathrm{n},\mathrm{T}}^{\text{Est}}\le \mathrm{K}.$$ (10)

In our experiments, we investigated how the final brain extraction accuracy changes as K varies.

The other strategy was to keep those atlases whose values, as measured by the objective function in a certain atlas ranking criteria, were above a user-specified percentage λ of the maximum value among all atlases [Klein08]. We call this λ a threshold ratio. The value of an atlas A_i with regard to a target T as measured by the cost function C in a certain atlas ranking criterion (Eqs. 2–8) is denoted by ${\mathrm{V}}_{\mathrm{T}}^{\mathrm{C}}(A_{\mathrm{i}})$. Then this second set of atlas selection is defined as

$$\text{keep}\text{atlas}{A}_n\text{if}{\mathrm{V}}_{\mathrm{T}}^{\mathrm{C}}(A_{\mathrm{n}})\ge \lambda \underset{\mathrm{i}=1,\dots ,\mathrm{N}}{max}{\mathrm{V}}_{\mathrm{T}}^{\mathrm{C}}({\mathrm{A}}_{\mathrm{i}})$$ (11)

In our experiments, we studied how the final brain extraction accuracy changes as λ varies.

3.6. Optimizing the Label Fusion Component

We evaluated 3 different label fusion methods: majority voting (MV), STAPLE [Warfield04], and Shape-Based Averaging (SBA) [Rohlfing07]. We used their implementations as provided in the NiftySeg⁸ software package [Cardoso11]. Including them for comparison was based on their popularity and their public availability. Other label fusion algorithms to be considered in the future, including, but are not limited to, SIMPLE [Langerak10], COLLATE [Asman11], improved STAPLE [Commowick12], STEPS [Cardoso13], etc.

3.7. Comparison with Other Algorithms

There are no brain extraction methods in the literature that are specifically designed for pediatric ADC maps. Consequently, we compared our approach to methods such as BET [Smith02] and BSE [Shattuck02], which are widely used methods, primarily designed for adult structural T1-weighted MR images. Below we present how we optimized BET and BSE for pediatric ADC maps.

BET⁹

The brain extraction tool (BET) is part of the FSL toolbox. It evolves a deformable model to the brain’s surface, based on local intensity profiles and subject to surface smoothness constraints. It is widely used because of its public availability, its simplicity of use, its speed (seconds) and the overall good quality of the results it generates. Unlike other brain extraction methods that only apply to T1-weighted structural images, BET can be also used with T2-weighted images, B0 images [Parker05, Xu07, Focke08, Acosta-Cabronero10], and even on FA images [Zhang10] with certain parameter adjustments.

We adjusted BET parameters for pediatric ADC maps based exclusively on the MGH_3T dataset, in the same way as we did for BSE and our own method. In particular, we ran BET multiple times by varying two key parameters --- the fractional intensity threshold and the vertical gradient. The former implicitly determines the brain size and where the boundary should be located. It is controlled by the “-f” option (default: 0.5). We varied this parameter from 0.2 to 0.8, with an increment of 0.05. The latter parameter applies a vertical gradient change of the intensity threshold as specified by the “-g” option (default: 0). We varied this parameter from −0.2 to 0.2, with an increment of 0.05. Figure 4(a) shows how the brain extraction accuracy changes as these two parameter values change. The highest accuracy occurs with the two parameters set at f = 0.45 and g = 0, which we will use in this paper.

Figure 4.

Visualization of brain extraction accuracy changes as parameters change. This set of results is based on the MGH_3T dataset. In each panel, an orange arrow points out the location of the maximum accuracy.

BSE¹⁰

The brain surface extractor (BSE) is a part of the BrainSuite software package. It conducts a series of anisotropic smoothing, edge detection and morphological operations. It is easy to run and finishes fast (typically less than 1 minute). It is designed for T1-weighted structural images only. [Shi11] reported that BSE with carefully adjusted parameter values could be used for pediatric structural images. Therefore we included BSE to test the hypothesis that BSE with adjusted parameters may improve its accuracy in pediatric ADC maps.

We ran BSE on the MGH_3T dataset multiple times by varying three key parameters --- the diffusion constant, the edge detection constant and the erosion size. The first parameter determines the strength of the edges to be kept. It is controlled by the “-d” option (default: 25). We varied it from 10 to 40, with an increment of 5. The second parameter influences how wide the edge must be in order to be identified. It is controlled by the “-s” option (default: 0.62). We varied it from 0.42 to 0.82, with an increment of 0.04. The third one modulates how far to extend the edge voxels. It is controlled by the “-r” option (default: 1). We tried to set it to values 1 and 2, and found that 2 usually gives better results. Figure 4(b) shows how the brain extraction accuracy changes as these parameters change. The highest accuracy occurs with d = 40, s = 0.54 and r = 2, which we will use in this paper.

4. Results

In this section, we first present performance evaluation results for each component in our framework. Then we demonstrate the overall performance, both qualitatively and quantitatively. The third set of results corresponds to the application of our framework to the PD308 population of a normative cohort, to characterize volumetric and diffusion changes across the first 6 years of life.

4.1. Results for Each Component

It should be emphasized that the optimization of each component of our framework was based exclusively on the MGH_3T dataset.

4.1.1. Results for Optimizing the Registration Component

Figure 5 shows the comparison results among the 4 deformable registration algorithms. Within-dataset (MGH_3T) registration results in Fig 4(a) show that DRAMMS had the highest mean Dice coefficient (0.92) and the smallest standard deviation (0.06). ANTs followed closely (mean Dice 0.89) with a similar standard deviation (0.06). Demons had a good mean Dice (0.84), but showed larger standard deviation (0.16). Perhaps this outcome can be best explained by the fact that this algorithm uses intensity difference as image similarity, which had been known to be more vulnerable to variations in image contrasts and anatomies [Klein09, Ou12c, Diez13, Ou14MRM, Ou14TMI, Sotiras13, Sotiras14].

Figure 5.

The Box-and-Whisker plot of brain mask Dice coefficients for four registration algorithms: FNIRT, Demons, ANTs and DRAMMS. P-values are also provided to show whether the performances of registration algorithms are significantly different.

Next we investigate how the registration algorithm of our choice, DRAMMS, performed in more challenging across-scanner-platform and across-institution registration tasks. ANTs and FNIRT showed robustness registering multi-platform data (Fig 4(b)). Conversely, they obtained lower Dice coefficients when registering multi-institution data (mean Dice down from 0.89 to 0.82 for ANTs, and from 0.66 to 0.56 for FNIRT, Fig 4(c)). Demons shows robustness registering multi-institution data (Fig 4(c)). Nevertheless, it had decreased Dice coefficients when registering multi-platform data (mean Dice down from 0.84 to 0.73, Fig 4(b)). In contrast, DRAMMS performed consistently well with a high mean Dice (0.91–0.92) and a low standard deviation (0.04–0.06), regardless of differences in platform or institution (Fig 4(a–c)). A summary of these results is provided in Fig. 4(d), and the difference between DRAMMS and others in our datasets are statistically significant (p<0.001). The performance from DRAMMS provided a foundation for our framework’s generality, accuracy and consistency when facing data from different sources and subjects of different ages.

4.1.2. Results for Optimizing the Atlas Ranking Component

Table A in Appendix A shows an example of how to calculate the average ranking errors (AREs, defined in Eq. 9) for different atlas ranking criteria, with regard to a randomly selected target. Figure 6(a) shows the distribution of AREs using all possible targets in the MGH_3T dataset. The results indicate that the criterion best indicative of an atlas’ inference ability was the estimated Dice coefficient with the TTM (Eq. 2, EstDice). It led to an ARE less than 2. In contrast, the conventional image similarity based atlas ranking criterion led to relatively larger AREs at around 3 and larger variations. The same finding was also true in two other datasets. For example, Figure 6(b) shows the same trend in the MGH_1.5T dataset. The findings here support our hypothesis from Section 3.4, that overlap-based metrics are more faithful indicators for the atlas-based segmentation accuracy than image similarity metrics.

Figure 6.

The distribution of AREs for various atlas ranking criteria in (a) MGH_3T and (b) MGH_1.5T datasets. In each figure, the columns from left to right correspond to criteria in Eqs. 2–8, respectively.

4.1.3. Results for Optimizing the Atlas Selection and Label Fusion Components

We aimed to select a combination of atlas selection and label fusion strategies balancing the following conditions: (1) high accuracy, in terms of the final brain extraction; (2) consistent accuracy in a dataset, (i.e., small standard deviation); (3) consistent accuracy when the values of the atlas selection strategy slightly changed; and (4) computational efficiency (i.e., using as few atlases as possible, or a threshold ratio as big as possible).

Figure 7(a) demonstrates results of leave-one-out experiments using the MGH_3T dataset. For label fusion, STAPLE and SBA performed almost equally well, and significantly better than MV. For atlas selection, parameter values had a big impact on the final brain extraction accuracy. Overall, two combinations of atlas selection and label fusion strategies performed better than others. One was to keep the top 5 ranked atlases, followed by the STAPLE label fusion (Dice coefficients at 0.958 ± 0.012, the highest in Figure 7(a) left panel); the other was to keep those atlases with which the estimated Dice coefficients (EstDice) were above 98% of the maximum EstDice over all atlases, followed by the STAPLE label fusion (Dice coefficients at 0.958 ± 0.011, the highest in Figure 7(b) right panel). The accuracies of the two combinations were statistically equivalent (p=0.97 in a two-sample t-test). Since their difference was statistically insignificant, for simplicity and efficiency in communication we chose to present only one, in this case, STAPLE, just because its accuracy is slightly (only slightly) higher than SBA, although statistically not significantly. By this atlas selection strategy, we recorded that 7~9 atlases were selected on average, in the leave-one-out tests in the MGH_3T dataset.

Figure 7.

Brain extraction accuracy when atlas selection and label fusion parameters change, in (a) MGH_3T and (b) MGH_1.5T datasets. Panels in the left and right columns are based on atlas selection strategies in Eqs. 10 and 11.

The chosen atlas selection and label fusion strategies also performed well in two other datasets. For example, Figure 7(b) shows the results from the MGH_1.5T dataset. The chosen strategies led to a high mean Dice coefficient (0.929), a stable performance in multiple subjects in this dataset (small standard deviation 0.023), and a stable performance when the threshold ratio for atlas selection slightly changed (mean and standard deviation of the Dice coefficients remained almost the same).

4.2. Results of the Overall Framework

The previous section presented quantitative comparison results aiming to optimize each component in our framework. To summarize, our optimized framework started with a fixed set of 15 pediatric atlases (MGH_3T), transferred the brain mask annotations into the target image space by the DRAMMS deformable registration, then ranked and selected atlases by the proposed Dice-coefficient-based ranking criterion and a thresholding-based atlas selection strategy, and finally fused the selected atlases by STAPLE. The output is the final brain mask for the target image.

Below, we tested how the optimized framework performed qualitatively (Section 4.2.1) and quantitatively (Section 4.2.2), recorded computation times (Section 4.2.3) and examined whether the brain extraction accuracy was consistent for subjects at different neuro-developmental stages (Section 4.2.4).

4.2.1. Qualitative Results

Figure 8 displays the brain extraction results in all three datasets as compared with expert annotations. In each dataset, we show results for subjects having the highest, medium and lowest brain extraction accuracies. Except for one subject in the MGH_1.5T dataset having a brain extraction accuracy at 0.88, all subjects in all three datasets had accuracies between 0.9 and 0.972. Qualitatively, in Figure 8, the results are consistently close to expert annotations, even though ADC maps clearly demonstrate differences in contrast, signal-to-noise ratio, and anatomy.

Figure 8.

Examples of algorithm-computed (green) and expert-annotated (red) brain masks in different datasets.

4.2.2. Quantitative Results and Comparison with Other Approaches

Table III compares the mean and standard deviation of accuracies by BET, BSE, and our proposed method, as measured by the Dice coefficient between the computed and the manually-annotated brain masks. The proposed method outperformed the other two approaches by a statistically significant margin (p<0.001). Its performance was accurate (high mean Dice coefficient values above 0.92 and even 0.95), consistent in subjects with different ages (very narrow standard deviation, 0.01—0.02), and consistent in datasets from different scanner platforms and institutions.

We also note that, compared to the single-atlas accuracy in Figure 5(d), the optimized multi-atlas pipeline not only improved the mean Dice coefficient (from 0.91–0.92 to 0.93–0.96 in different datasets), but more importantly, significantly reduced the standard deviation (from 0.04–0.06 to 0.01–0.02). This meant that our optimized multi-atlas framework significantly increased the desirable consistency with respect to data sources and subject ages. Since brain extraction is often times the first step in automated image analysis pipelines, the increased accuracy and consistency of our framework should have a significant impact and provide an improved initialization for subsequent steps (e.g., volume calculation, ADC calculation, tissue and structural segmentation, atlas construction, longitudinal registration).

4.2.3. Computational Time

Our method took about 10–12 minutes or 15–20 minutes per pediatric ADC map of image size 128×128×70 or 256×256×60. These times were recorded on a high-performance computer cluster that has 127 computer nodes, each node having 8 CPUs and 56 GB of shared virtual memory. The majority of time was spent in the atlas-to-target registration step. BET and BSE took about 5–10 seconds per case.

4.2.4. Consistency of Accuracy

Besides accuracy, we also investigated the consistency of the accuracy, with regard to data sources and subject ages. We plot the brain extraction of all subjects in all 3 data sources in Figure 9. For BET and BSE (Figure 9(a, b)), the brain extraction accuracies had larger variations for subjects under the age of 2 years than for older subjects. Thus, age and the associated structural maturation seem to be two dominant factors of accuracy for BET and BSE.

Figure 9.

Plots of brain extraction accuracies in different data sources and for subjects of different ages, to test the consistency of brain extraction algorithms. a) BET, b) BSE and c) our method.

For our framework (Figure 9(c)), in contrast, brain extraction accuracy remained very consistent, regardless of the data sources (different panels in Figure 9) and the ages of the subjects (x axes in each panel). This might be due to three reasons: 1) our atlas library had included atlases from many subjects in the first year (see Figure 3); 2) we had used a deformable registration algorithm that was relatively robust to variations in age and anatomy; 3) we optimized each component in the framework.

4.3. Applications to Neuro-Development Study of 308 Pediatric ADC Maps

We applied the proposed framework to the PD308 cohort. With the extracted ADC maps, we were able to characterize whole-brain volumetric and diffusion changes across the first 6 years of life.

We first looked at the brain volume changes. We computed the mean and standard deviation of brain volumes in the 10 age groups described in Table II. Results are shown in Figure 10 (a). At birth, the brain has an average volume of around 500 cc. The brain expands quickly in the first year, especially the first 6 months (steeply ascending curve in this period). At the end of the first year, the brain volume almost doubles to 900–1000 cc. Then the brain continues to expand, but at a much slower pace. At about 4 years of age, the brain volume reaches a plateau, and ranges between 1150 and 1450 cc, with the average at around 1300 cc. This is almost 2.5 times as big as the volume at birth and is also very close to the reported brain volume of young adults. All these findings agreed with those reported in the literature. For example, [Lenroot06] reported a mean brain volume of close to 1000 cc at around 1 year of age and about 1200 cc at around 2 years of age (N=511). [Reiss96] reported a mean brain volume of around 1200 cc for children 5–17 years of age (N=21 male and 64 female). [Ge02] reported brain volume of 1367 ± 147 cc for adults (N=32, ages 29–40).

Figure 10.

Characterization of neuro-development in the PD308 dataset. (a) mean and standard deviation of whole-brain volume in various age groups. (b) mean and standard deviation of whole-brain ADC values (CSF excluded) in various age groups.

We also report a preliminary estimate of how the average ADC values change across the first 6 years of life. When brain grows, neuronal myelination should restrict water diffusion, and hence decrease the ADC values in white matter (WM) and gray matter (GM) [Hüppi06, Dubois14, Guleria14]. To focus on WM and GM regions, we removed CSF by thresholding the ADC maps at a manually selected value for each image, usually around 1800 μm²/s. This threshold is based on observations of the histograms and prior knowledge from clinical literature [Morriss99, Helenius02, Naganawa03, Hüppi06, Provenzale10]. Figure 10(b) plots how the average ADC changes over time, after excluding the CSF tissue. The average ADC drops from 1300 to 1100 μm²/s, or almost 15% in the first 6 months; and from 1300 to 1000 μm²/s, or almost 25% in 1 year. From 1–5 years of age, the decrease in the mean ADC values happens at a much slower pace, at less than 10% rate over the whole 4 years. After 5 years of age, the whole-brain ADC value plateaus at 900–950 μm²/s, which is about the level that has been found in young adults [Helenius02, Engelbrecht02, Naganawa03, Counsell03, Hüppi06, Bonekamp07, Provenzale10].

We want to emphasize that we are not showing new information about neuro-development that was previously unknown. Rather, the results herein highlight the fact that our optimized framework has reached a high level of accuracy and consistency even when applied to medical images collected during the delivery of routine clinical care. These clinically collected images yield comparable valuable benchmark information as expected from the research literature.

5. Discussion

We presented and optimized a multi-atlas brain extraction framework specifically built for pediatric ADC maps beginning in infancy. We demonstrated its generality, accuracy and consistency in datasets from different platforms/institutions and for subjects of various ages. We also applied the proposed framework to study the volumetric and ADC changes in a larger population across the first 6 years of life. We showed that the optimized framework could achieve the desirable specificity, generality, accuracy and consistency using a fixed set of atlases and a fixed set of parameters. Because of these compelling properties, our framework was successfully used in challenging clinically-acquired data and still obtained information consistent with the literature (Figure 10).

Diffusion imaging for infants and pediatric subjects often have distortions arising from factors such as head size/shape, during-scan motion and scanner eddy currents [Dubois14b, Counsell06, Giménez08]. Distortion correction is a non-trivial analysis task additionally requiring the use of all diffusion weighted images. We purposely chose not to correct for distortions and show that our framework achieved high accuracy, consistency and robustness in our experiments in multi-institutional datasets. We believe that this is mainly because the DRAMMS registration method uses attribute-based matching which better deals with distortions than intensity-based registration algorithms. This advantage has been shown in Figure 5, and has also been reported in other registration comparative studies [Ou14TMI, Ou14MRM]. The independence from distortion correction highlights the generality and robustness of our proposed framework.

Similarly, our final results in Table III and especially Figure 9(c) showed that, even though we have used a fixed set of atlases from subjects whose ages were not necessarily matched to that of the target, our final brain extraction accuracies remain consistently high in a very narrow range (0.9–0.95 Dice with expert annotation), regardless of the age of the subject (Figure 9(c)), whereas BSE/BET’s brain extraction accuracies were in a more variable, wider range, and their performance dropped when the subjects were under 2 years old, and even more when they were 1 year old. Having a consistently high accuracy, independent of a subject’s age, is a strength in our framework.

We believe that the resulting generality and consistency can be largely attributed to the use of the DRAMMS registration algorithm that achieves consistently high accuracy in atlas-to-target alignment. This was shown in Figure 5 and also documented in [Ou14TMI]. Several studies have reported that attribute-based registration algorithms such as DRAMMS may perform more stable than intensity-based registration algorithms when anatomy varies greatly or when the image contrast is low, such in our pediatric ADC maps. Nevertheless, conclusively demonstrating this, or understanding the exact role of various components (transformation/similarity/optimization) in various registration algorithms is another large-scale study (such as [Klein09, Ou14TMI]) that we believe is outside the scope of this paper. Another reason for the generality and consistency in our framework is in comparatively choosing proper measurements in many components of the framework. Especially, the atlas ranking and atlas selection components can effectively reject atlases that fail to indicate a good brain mask.

There are several technical issues to be addressed in future studies. One issue is the optimization of the atlas library. We used 15 atlases, since existing studies (e.g., [Isgum09, Leung11]) showed that 9 to 15 atlases are usually sufficient to saturate the segmentation accuracy. However, with a more advanced registration algorithm as used in this paper, our brain extraction accuracy seems to saturate at around 7–9 atlases. We would like to better understand whether fewer atlases could be used to maintain the same level of accuracy and consistency, and how to choose the optimal atlas library to begin with. Another issue is the optimization of parameters in our framework. We optimized the choice of methods in each component of our framework. For the chosen methods, we used their default parameters. Even though they performed at a high level, we would like to examine whether another set of parameters could further improve the overall accuracy and consistency. A third issue is that, although our visual inspection led us to believe that the margin between our algorithm (average Dice at 0.93~0.96 with expert annotations) and the parameter-tuned BET/BSE (average Dice at 0.81~0.86 with expert annotations) should far exceed the intra-/inter-rater variability, our future work would include 2 or more experts annotating the brain masks in order to rigorously quantify intra-/inter-rater variability, or to employ unsupervised segmentation evaluation when the absolute gold-standard is absent [Hoppin02, Lebenberg12]. Last, due to limited accessibility of intermediate results in BET and BSE, we compared only the final results with BET and BSE.

For neuro-development studies, the access to large-scale multi-institution/platform/vendor data is quickly becoming widespread, due to advancements in data query tools such as the mi2b2 workbench [Murphy14] and the availability of large-scale databases such as the NIH Pediatric MRI Data Repository (NIH-PD) [Evans06]. Large-scale data help to better model the normal range of variation, which is important for accurate identification of structural brain abnormalities [Volkher02, Liauw05]. While many brain extraction algorithms that were developed for adult structural T1-weighted images have difficulties in large-scale multi-site structural data [Lee03, Fennema-Notestine06], our framework shows consistently high accuracy in different datasets. In the future, we will further test our framework’s performance in data from different vendors (GE, Philips). We also plan to repeat the experiments in Section 4.3 (whole-brain ADC and volumetric changes) for images acquired from 1.5T scanners, and compare the results with the ones from 3T scanners.

Neuro-development studies based on automated MR image analysis, while more often based on T1-weighted structural images [Lenroot06, Shi11, Fonov11], are increasingly using pediatric diffusion weighted images, either independently or as an additional sequence [Geng12, Melbourne13]. Therefore, the development of our framework is timely. In this paper, the automated analysis of N=308 pediatric subjects has enabled us to plot volumetric and ADC changes in the first 6 years of life. In the future, we plan to construct age-specific atlases to better quantify the mean shape and mean ADC values at the regional level. This will allow the development of abnormality detection tools. These are essential first steps toward the application of medical informatics approaches to data mining of clinical images to assist in computerized diagnosis for pediatric neurological and neuropsychiatric diseases.

Highlights.

• We present auto- brain extraction in clinical ADC maps of infants and pediatrics.

• We show accuracy, consistency and suitability in multi-site, multi-platform data.

• We apply it to quantify volumetric and ADC changes in the first 6 years of life.

Appendix A. Measuring Atlas Ranking Errors

This Appendix provides an example to explain how to calculate the average ranking error (ARE), with regard to a randomly chosen target ADC map. In this example, a randomly chosen ADC map A₁₃ from the MGH_3T dataset was used as the target image. The remaining 14 ADC maps were used as atlases.

Table A.

An example of how to calculate the average ranking error (ARE) of an atlas ranking criterion, with regard to a given target. We randomly chose the 13^th atlas (A₁₃) as the target, and used the remaining 14 as the atlases.

To A13	Truth		Proposed 1 EstDice (Eq. 2)		Proposed 2 EstDice*CCdef (Eq. 3)		Proposed 3 EstDice*MIdef (Eq. 4)		Conventional 1 CCaff (Eq. 5)		Conventional 2 MIaff (Eq. 6)		Conventional 3 CCdef (Eq. 7)		Conventional 4 MIdef (Eq. 8)
	True Dice	True Rank	Value	Rank	Value	Rank	Value	Rank	Value	Rank	Value	Rank	Value	Rank	Value	Rank
A2	.949	1	.953	3	.534	13	.261	13	.639	13	.318	2	.609	5	.302	2
A3	.943	2	.959	1	.609	11	.276	3	.639	12	.301	9	.613	2	.289	3
A10	.934	3	.941	4	.606	12	.263	12	.641	11	.300	11	.604	10	.282	9
A9	.932	4	.938	8	.613	7	.266	9	.643	9	.297	13	.603	11	.279	12
A4	.930	5	.930	10	.616	5	.270	6	.649	5	.303	7	.604	9	.281	8
A7	.929	6	.939	6	.614	6	.270	7	.645	7	.300	10	.606	6	.282	10
A8	.928	7	.938	7	.631	1	.286	1	.652	2	.306	3	.611	3	.287	4
A6	.928	8	.922	13	.617	4	.272	5	.649	6	.305	5	.598	12	.281	11
A1	.927	9	.958	2	.475	14	.201	14	.661	1	.325	1	.633	1	.311	1
A5	.917	10	.930	11	.613	9	.264	11	.651	4	.306	4	.605	7	.284	6
A14	.913	11	.939	5	.621	3	.274	4	.644	8	.301	8	.605	8	.283	7
A12	.912	12	.929	12	.613	8	.265	10	.642	10	.297	12	.597	13	.276	13
A11	.910	13	.935	9	.628	2	.279	2	.652	3	.305	6	.610	4	.285	5
A15	.866	14	.915	14	.611	10	.268	8	.637	14	.294	14	.583	14	.269	14
ARE	0		2.57		3.86		3.86		5.43		5		5.14		4.43