Abstract
Automated segmentation and labeling of individual brain anatomical regions is challenging due to individual structural variability. Although, atlas-based segmentation has shown its potential for both tissue and structure segmentation, the inherent natural variability as well as disease-related changes in MR appearance is often inappropriately represented by a single atlas image. In order to have a more accurate representation, several atlases may be used for the segmentation task in a given neuroimaging study. In this paper, we present the MultisegPipeline, it uses multiple atlases that have been visually inspected and capture the expected variability in a neonatal population. The MultisegPipeline transfers the labeled regions from each atlas to the target image using deformable registration (ANTs1 or QuickSilver2 is available for this task). Additionally, the set of labels are merged using a label fusion technique that reduces the errors produced by the registration. The final output is a single label map that combines the results produced by all atlases into a consensus solution. In our study, the MultisegPipeline is used to segment brain MR images from 31 infants, a leave-one-out strategy was used to test our framework. The average dice score coefficient was 0.89.
Keywords: segmentation, subject-specific, population, neonate, MRI, automatic, atlas, tissue
1. INTRODUCTION
Segmentation of brain structures in magnetic resonance imaging (MRI) is essential for quantitative studies of the brain. Manual anatomical labeling (identification of anatomical brain structures and assignment of a unique label to each structure) would provide the highest possible accuracy. Unfortunately, manual labeling is a time consuming operation and is biased depending on the rater. These issues makes manual labeling implausible for studies involving several subjects. Moreover, neonatal brain image segmentation is particularly challenging, due to the fact that there is low contrast between white and gray matter regions[3] [4]. Recent advancements5 using deep learning are able to accurately classify the white matter (WM), gray matter (GM) and cerebral spinal fluid (CSF); sub-cortical segmentation of brain structures6 using deep learning has also been explored. These studies achieved high segmentation accuracy, nevertheless, the age of the subjects that participated in these studies range from infants 7 years of age to adults. Therefore, a solution for the segmentation of sub-cortical structures in neonatal images remains to be created. In this paper we present the MultiSegPipeline, it is based on the NeoSegPipeline.7 The major contributions to this framework are the integration of the Quicksilver (QS)2 registration algorithm and the integration of a label fusion algorithm method available in ANTst[8] [9]. The following section explains the data used to test our framework.
2. MATERIALS
Images from 37 infant brains were taken from a larger study designated to investigate early brain development. Each dataset contains T1 weighted (T1w) and T2 weighted (T2w) MRI of newborns. Both images were acquired in the same scanning session. The newborns were scanned between the age of 40 weeks corrected gestational age (two premature infants) and 47 days. The Children were scanned unsedated while asleep, fitted with ear protection and with their heads secured in a vacuum-fixation device. These images have been previously segmented and the resulting label maps have been visually inspected and manually corrected. The following sections explains in detail the methods available in the MultisegPipeline.
The images are divided into 2 groups, 6 images to re-train the QuickSilver registration algorithm and 31 images to test the MultiSegPipeline.
3. METHODS
Figure 1 shows an overview of the framework. The NeoSegPipeline7 is shown in blue and the additions presented in this paper are shown in green. The three major steps involved are image pre-processing, registration by QS and label fusion algorithm. The inputs are T1 weighted (T1w) and T2 weighted (T2w) images. A leave-one-out strategy is used to test our framework.
Figure 1.
Overview of the framework. The new additions are shown in green.
3.1. Pre-processing
The first step in this framework is to skull-strip the images, a brain mask may be provided as input, otherwise FSL-bet10 is used to compute it. After the skull is removed, an in-homogeneity correction11 is applied. The resulting image is then registered to each individual atlas. The following section briefly explains the QS2 registration method.
3.2. Image registration
Quicksilver (QS) is a fast deformable image registration method. It uses a deep encoder-decoder network to predict deformation parameters. The input to the neural network are patches extracted at the same location and the output is a tensor that holds the deformation for the given patch. The ground truth data during training is the Large Deformation Dieomorphic Metric Mapping (LDDMM). Additionally, QS uses a correction network that refines the output produced by the prediction network.
For the work presented in this paper, we use ANTs to compute deformation fields between a pair of images, these deformation fields are used to train the prediction network in QS. The correction network was not used to refine the registrations produced by QS. By integrating this software into the MultiSegPipeline, we expect to achieve faster results and preserver the quality in the final output segmentation. Registration using ANTs is the most time consuming operation in our framework.
3.3. Label fusion
The label fusion strategy used in this software is available in ANTs. It is based on the assumption that atlases may produce correlated segmentation errors. Based on this assumption, the label fusion seeks to minimize this error without compromising the properties of ‘voting’. It formulates the weighted voting problem optimization over unknown voting weights, i.e., it requires the joint distribution of label errors produced by any pair of atlases in the neighborhood of each voxel to be known. The unknowns are estimated using image intensity similarity between a pair of atlases. The problem is formulated as follows,
| (1) |
Where is the estimated probability of label l for the target image at location x. is the probability that Ai votes for label l at x.
4. RESULTS
We tested the segmentation approach using both registration algorithms (ANTs and QS). We obtain a segmentations for the WM, GM, CSF and sub-cortical regions. Our goal was to compare the performance of ANTs v.s. QS. Figure 2 shows registration results for 3 subjects. There are differences between the registrations but we consider that the gain in speed using QS is significant when compared to ANTs. Figure 3 shows the 3D surface rendering of the sub-cortical structures. 3 subjects were chosen randomly to visualize the result at 3 different orientations. Figure 4 shows the segmentation result for 12 randomly chosen subjects. Figure 5 shows box plots of the dice coefficient obtained using the leave-one-out strategy. The structures shown are the 12 sub-cortical structures (left and right amygdala, putamen, caudate, thalamus, hippocampus and pallidum). The average dice was 0.92 using ANTs v.s. 0.89 using QS. Using the QS algorithm showed less accuracy, however, we expect the dice coefficient to increase when the correction network for the QS registration is used.
Figure 2.
From top to bottom, moving image, ANTs registration result, Quicksilver registration result, the target image.
Figure 3.
3D surfaces for sub-cortical structures, Amygdala, Caudate, Hippocampus Pallidum, Putamen, and Thalamus.
Figure 4.
Segmentation results for randomly chosen subjects.
Figure 5.
Dice coefficient comparing the output using registration by ANTs and Quicksilver for 10 sub-cortical structures. The following notation is used in the x-axis. Q=Quicksilver, A=ANTS, AL=Amygdala L, AR=Amygdala R, CL=Caudate L, CR=Caudate R, HR=Hippocampus R, HL=Hippocampus L, PaL=Pallidum L PaR=Pallidum R, PuL=Putamen L, PuR=Putamen R, TL=Thalamus L, TR=Thalamus R, L=Left, R=Right
5. CONCLUSION
In conclusion, we have presented a multi-atlas segmentation scheme implemented in the MultisegPipeline framework. The tool is available at our github repository https://github.com/NIRALUser/NeosegPipeline.git. This tools works on the major operating systems. We have binary packages that are ready for download.
The results show that the proposed method achieved comparable segmentation results using either ANTs or Quicksilver registration method for the sub-cortical structures. However, in future work we intent to improve the QS registration algorithm by including the correction network. We expect the output registration from QS to be of equal quality than those produced by ANTs. The gain in speed by QS is significant and is a desirable addition to be included in this framework.
ACKNOWLEDGMENTS
I would like to thank Celia Rousseau for visually inspecting the results of the segmentations produced by the MultiSegpipeline.
REFERENCES
- 1.Avants BB, Tustison NJ, Stauffer M, Song G, Wu B, and Gee JC, “The insight toolkit image registration framework,” Frontiers in neuroinformatics 8, p. 44, 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Yang X, Kwitt R, Styner M, and Niethammer M, “Quicksilver: Fast predictive image registration-a deep learning approach,” NeuroImage 158, pp. 378–396, 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Zhang W, Li R, Deng H, Wang L, Lin W, Ji S, and Shen D, “Deep convolutional neural networks for multi-modality isointense infant brain image segmentation,” NeuroImage 108, pp. 214–224, 2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Shi F, Fan Y, Tang S, Gilmore JH, Lin W, and Shen D, “Neonatal brain image segmentation in longitudinal mri studies,” NeuroImage 49(1), pp. 391–400, 2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Dolz J, Ayed IB, Yuan J, and Desrosiers C, “Isointense infant brain segmentation with a hyper-dense connected convolutional neural network,” in Biomedical Imaging (ISBI2018), 2018 IEEE 15th International Symposium on, pp. 616–620, IEEE, 2018. [Google Scholar]
- 6.Dolz J, Desrosiers C, and Ayed IB, “3d fully convolutional networks for subcortical segmentation in mri: A large-scale study,” NeuroImage , 2017. [DOI] [PubMed] [Google Scholar]
- 7.Cherel M, Budin F, Prastawa M, Gerig G, Lee K, Buss C, Lyall A, Consing KZ, and Styner M, “Automatic tissue segmentation of neonate brain mr images with subject-specific atlases,” in Medical Imaging 2015: Image Processing, 9413, p. 941311, International Society for Optics and Photonics, 2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wang H, Das SR, Suh JW, Altinay M, Pluta J, Craige C, Avants B, Yushkevich PA, Initiative ADN, et al. , “A learning-based wrapper method to correct systematic errors in automatic image segmentation: consistently improved performance in hippocampus, cortex and brain segmentation,” NeuroImage 55(3), pp. 968–985, 2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Wang H and Yushkevich P, “Multi-atlas segmentation with joint label fusion and corrective learningan open source implementation,” Frontiers in neuroinformatics 7, p. 27, 2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Smith SM, “Fast robust automated brain extraction,” Human brain mapping 17(3), pp. 143–155, 2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Tustison NJ, Avants BB, Cook PA, Zheng Y, Egan A, Yushkevich PA, and Gee JC, “N4itk: improved n3 bias correction,” IEEE transactions on medical imaging 29(6), pp. 1310–1320, 2010. [DOI] [PMC free article] [PubMed] [Google Scholar]