Multimodal neuroimaging computing: the workflows, methods, and platforms

Abstract

The last two decades have witnessed the explosive growth in the development and use of noninvasive neuroimaging technologies that advance the research on human brain under normal and pathological conditions. Multimodal neuroimaging has become a major driver of current neuroimaging research due to the recognition of the clinical benefits of multimodal data, and the better access to hybrid devices. Multimodal neuroimaging computing is very challenging, and requires sophisticated computing to address the variations in spatiotemporal resolution and merge the biophysical/biochemical information. We review the current workflows and methods for multimodal neuroimaging computing, and also demonstrate how to conduct research using the established neuroimaging computing packages and platforms.

Introduction

Neuroimaging has profoundly advanced neuroscience research and clinical care rapidly in the past two decades, prominently by magnetic resonance imaging (MRI), complemented positron emission tomography (PET), and electroencephalography (EEG)/magnetoencephalography (MEG). The art of neuroimaging today is shaped by three concurrent, interlinked technological developments [1]:

Data Acquisition The advances of imaging instrumentation have enabled digital image acquisition, as well as electronic data storage and communication systems, such as the picture archiving and communication system (PACS). These imaging systems, CT, MRI and PET showed obvious clinical benefits by providing high contrast tissue differentiation. The previous film-based reading was replaced by the electronic displays (axial, coronal and sagittal planes of the volume) without losing diagnostic quality.

Medical Image Computing The growth of neuroimaging has spurred a parallel development of neuroimaging computing methods and workflows, including bias correction, registration, segmentation, information extraction and visualization. We should note the difference between neuroimaging and neuroimaging computing. Neuroimaging focuses on the image acquisition, capturing the snapshot of the brain; whereas neuroimaging computing focuses on the computational analysis of the brain images, extracting and enhancing the information of relevance to best describe the brain anatomy and function.

Package and Platform Development To fit into research and clinical timelines and facilitate translational medicine, the neuroimaging computing methods and workflows are often integrated into software packages. Many such packages were added to imaging systems by the major vendors of medical imaging equipment and many specialized companies. However, a greater number of neuroimaging computing packages and platforms are free and open-source, designed and supported by the medical imaging research groups and communities.

Multimodal neuroimaging, i.e., the simultaneous imaging measurement (EEG/fMRI [2], PET/CT [3]) or summation of separate measurement (PET and sMRI [4], sMRI and dMRI [5], fMRI and dMRI [6]), has become an emerging research area due to better access to imaging devices, especially the hybrid systems, such as PET/CT [7, 8] and PET/MR [9]. The recent advances in neuroimaging computing methods also enabled joint analysis of the multimodal data. The free and open-source software (FOSS) packages and platforms for neuroimaging computing further facilitate the translation of the multimodal neuroimaging research from the lab to better clinical care.

Multimodal neuroimaging advances the neuroscience research by overcoming the limits of individual imaging modalities and by identifying the associations of findings from different imaging sources. Multimodal neuroimaging has been used to investigate a multitude of populations and disorders, such as Alzheimer’s disease (AD) [4, 10–12], schizophrenia [13–16], epilepsy [3, 17–19], obsessive-compulsive disorder (OCD) [20–22], bipolar disorder [23, 24], attention-deficit hyperactivity disorder (ADHD) [25–27], Autism spectrum disorder (ASD) [28–30], traumatic brain injury (TBI) [31–34], stroke [35, 36], multiple sclerosis (MS) [37–39], and brain tumors [9, 40–42]. We have recently reviewed advances in neuroimaging technologies and the applications of multimodal neuroimaging in these neuropsychiatric disorders [43]. Multimodal neuroimaging has also been used in many non-clinical applications, such as building brain machine interface (BMI) [44], tracing neural activity pathways [45] and mapping mind and behavior to brain anatomy [46–48].

Multimodal neuroimaging computing is a very challenging task due to large inter-modality variations in spatiotemporal resolution, and biophysical/biochemical mechanism. Compared to single imaging modality computing, it requires more sophisticated bias correction, co-registration, segmentation, feature extraction, pattern analysis, and visualization. Various methods for neuroimaging analysis have been proposed, and many have been integrated into the task-oriented packages or integrated platforms.

In this paper, we review the state-of-the-art methods and workflows for both modality-specific neuroimaging computing and multimodal neuroimaging computing, and demonstrate how to conduct multimodal neuroimaging research using the established packages and platforms. Fig. 1 provides an overview of the current status and illustrates the major components of neuroimaging computing, including neuroimaging modalities, modality-specific computing workflows (a series of tasks), multimodal computing methods, algorithms, packages, platforms and communities. MRI, PET, EEG/MEG and their computing workflows and methods are discussed in this review. A neuroimaging computing task in an analysis workflow may be fulfilled by multiple algorithms, and the most widely used algorithms, e.g., voxel-based morphometry (VBM) [49], are often integrated into software packages, e.g., Statistical Parametric Mapping (SPM)1, FMRIB Software Library (FSL)2, and Neurostat3. The new imaging tasks also demand the refinement of existing algorithms and development of new algorithms. Similar algorithms are often developed independently in different labs, sometimes with little awareness of existing packages/platforms.

Fig. 1.

Overview of the current status and major components of multimodal neuroimaging computing, including neuroimaging modalities, modality-specific computing workflows, multimodal computing methods, algorithms, task-oriented packages, all-integrated platforms, and neuroimaging research communities

This paper is organized as follows. In Sect. 2, we elaborate the computing workflows, which consist of a number of specific tasks, for individual modalities. In Sect. 3, we review the major multimodal neuroimaging computing methods, i.e., registration, segmentation, feature integration, pattern analysis and visualization. In Sect. 4, we introduce the task-oriented packages and platforms for the tasks mentioned in previous sections. We focus on the free and open source software (FOSS) in this review, since they could help to better realize the quickly evolved methods and workflows than their commercial counterparts, and thus accelerate translational medicine. For the sake of clarity and precision, the algorithms, packages and platforms are not described in detail, but we refer the interested readers to more specific papers instead. In Sect. 5, we give one example of brain tumor surgical planning using the established packages and platforms. Lastly, we outline the future directions of multimodal computing in Sect. 6.

Modality-specific neuroimaging computing workflows

Bias and artifacts correction

Different neuroimaging techniques have different spatiotemporal resolutions, and biophysical/biochemical mechanisms, and thereby require different computing workflows; yet a common step of all workflows is the correction of bias and artifacts in neuroimaging data. The main goal of this task is to remove the data components that contaminate the signals. Tustison recently provided a set of guidelines for managing the instrumental bias when designing and reporting neuroimaging studies [50].

Bias and artifacts in neuroimaging signals may result from imaging systems, environment, and body motion. Many biases and artifacts are induced by the imaging systems, e.g. inhomogeneous radio frequency (RF) coils in MRI, contrast agents in PET/CT, broken or saturated sensors in EEG/MEG system. Environment-related artifacts, arising from generators of magnetic fields outside the human body such as magnetic noise from power lines and other environmental noise sources, such as elevators, air conditioners, nearby traffic, mechanical vibrations transmitted to shielded room, bed vibration, and pulsation [51]. Motion-related artifacts are caused by eye movements, head movements, cardiac and muscular activity, and respiratory motion. The motion of magnetic implements, such as pacemakers and implantable defibrillators [52] may also give rise to artifacts, and may cause danger to patients in strong magnetic field, although there are new MRI compatible pacemakers/defibrillators that have been introduced [53].

The bias and artifacts in MRI are mainly system-related, e.g., RF inhomogeneity causing slice and volume intensity inconsistency. The nonparametric nonuniformity normalization (N3) algorithm and its variant based on Insight Toolkit [54, 55] (N4ITK) [56] are the de facto standard in this area. The acquisition protocols for dMRI are inherently complex, which require fast gradient switching in Echo-Planar Imaging (EPI) and longer scanning time. dMRI is prone to many other types of artifacts, such as eddy current, motion artifacts and gradient-wise inconsistencies [57]. Tortoise [58] and FSL diffusion toolbox (FDT) [59] are popular choices for eddy current correction and motion correction in dMRI data, and the recently proposed DTIPrep [60] offers a thorough solution for all known data quality problems of dMRI. Motion is a serious issue in fMRI, and may lead to voxel displacements in serial fMRI volumes and between slices. Therefore, serial realignment and slice timing correction is required to eliminate the effects of head motion during the scanning session. Linear transformation is usually sufficient for serial alignment, whereas a non-linear auto-regression model is often used for slice timing correction [61]. These two types of correction are commonly performed using SPM and FSL. Dedicated PET scanners have been replaced by the hybrid PET/CT systems [62]. The most commonly seen artifacts on PET/CT are mismatches between CT and PET images caused by body motion due to the long acquisition time of the scan. Metallic implants and contrast agents may also give rise to artifacts on PET/CT, usually leading to overestimation of PET attenuation coefficients and false-positive findings [63]. Knowledge and experience are needed to minimize these artifacts and, in that way, produce better-quality PET/CT images. EEG and MEG signals are often contaminated by all of the three types of artifacts, such as the system-related superconducting quantum interference device (SQUID) jumps, and the noise from the environment or body motion [51]. Visual checks and manual removal are usually required to exclude the artifacts. Another strategy uses signal-processing methods to reduce artifacts while preserving the signal. Linear transformation, e.g., principal component analysis (PCA) and independent component analysis (ICA) [64, 65], and regression, e.g., signal space projection (SSP) and signal space separation (SSS) [66, 67], are frequently applied to the raw EEG/MEG data.

Structural MRI computing

The sMRI computing workflows usually involve skull striping, tissue and region of interest (ROI) segmentation, surface reconstruction [68], and can include brain morphometry analysis, such as the voxel-based morphometry (VBM)/tensor-based morphometry (TBM)/deformation-based morphometry (DBM) [49], and surface-based morphometry (SBM) [69] by comparing one group of subjects to another or tracking the changes over a sequence of observations for the same subject. FreeSurfer [70] is a well-established tool for brain tissue segmentation and surface reconstruction. When registered into a standard brain space, e.g., the Talariach coordinates [71] and MNI coordinates [72], and labeled with different regions of interest (ROIs) using brain templates, e.g., ICBM template [73] and the AAL template [74], the sMRI datasets can further be analyzed at the ROI level. Various techniques have been investigated to quantitatively analyze the morphological changes in cortex, e.g., grey matter density [49], cortical folding [75], curvedness and shape index [76, 77], cortical thickness [69], and surface area [78, 79], local gyrification index (LGI) [75], and many other shape [78, 80] or texture features [81–83]. Mangin et al. [84] provided an extensive review on the popular morphological features, and Winkler et al. [85] demonstrated how to use these features in imaging genetics.

Diffusion MRI computing

The dMRI computing workflow consists of four major steps. The first step is to estimate the principle directions of the tensor or the orientation distribution function (ODF) in each voxel, which are used to quantitatively analyze the local white matter morphometry and probe the white matter fiber tracts in the following steps. Advanced fiber orientation estimation methods include the ball and stick mixture models [59], the constrained spherical deconvolution (CSD) [86], the q-ball imaging (QBI) [87], diffusion spectral imaging (DSI) [88], the generalized q-sampling imaging (GQI) [89], and the QBI with Funk Radon and Cosine Transform (FRACT) [90]. Wilkins et al. have provided a detailed comparison of these models [91]. In the second step, various parametric maps based on the tensors/ODFs, i.e., fractional anisotropy (FA), mean diffusivity (MD), radial diffusivity (RD), and axial diffusivity (AXD) maps [92], reveal the focal morphometry of the white matter [93]. The third step is to apply the fiber tracking algorithms [94] to construct 3D models of the white matter tracts, referred to as tractography. Tractography further enables the quantitative analysis of fiber tract morphometry i.e., orientation and dispersion [95], and the analysis of connectome, i.e., connectivity networks of populations of neurons [96]. Brain parcellation and fiber clustering are two major approaches that can separate the neurons into different groups/ROIs, and construct the connectome [97]. Jones et al. [98] recently provided a set of guidelines which define the good practice in dMRI computing.

Functional MRI computing

After bias and artifacts correction in fMRI, a mean image of the series, or a co-registered anatomical image, e.g., sMRI, is used to estimate some registration coefficients that map it onto a template, followed by spatial smoothing and parameter estimation. Friston [99] gave an introduction to these procedures. When the brain is performing a task, cerebral blood flow (CBF) usually changes as neurons work to complete the task. The primary use of task-evoked fMRI is to identify the correlation between brain activation pattern and brain functions, such as perception, language, memory, emotion and thought [100, 101]. Many models and methods have been suggested to detect patterns of brain activation, and some of them have been integrated into the software packages, such as the general linear model (GLM) in the SPM and FSL packages, and independent component analysis (ICA)/canonical correlation analysis (CCA) in AFNI package4. When brain is at resting state, fMRI is used to detect the spontaneous activation pattern in the absence of an explicit task or stimuli [102]. Resting-state fMRI enables us to deduce the functional connectivity between dispersed brain regions, which form functional brain networks, or resting state networks (RSNs). The Default Mode Network (DMN) is a functional network of several brain regions that show increased activity at rest and decreased activity when performing a task [103]. DMN has been widely used as a measure to compare individual differences in behavior, genetics and neuropathologies, although its use as a biomarker is controversial [104, 105]. Rubinov [106] provided a review of the connectivity measures.

PET computing

The computing of PET also requires spatial normalization and smoothing, and parameter estimation, similar to fMRI. SPM and Neurostat packages are available for voxel-by-voxel PET analysis. PET functional features are generally pertaining to the radioactive tracers, reflecting particular biochemical process. 2-[¹⁸F fluoro-2-deoxy-D-glucose (FDG) is the most widely used tracer to depict glucose metabolism. Several amyloid-binding compounds, ${}^{18}$F-BAY94-9172, ${}^{11}$C-SB-13, ${}^{11}$C-BF-227, ${}^{18}$F-AV-45 and ${}^{11}$C-Pittsburgh compound B (${}^{11}$C-PiB), have been reported as tracers for imaging amyloid plaques in AD. A number of extensive surveys have been conducted on these amyloid radioactive tracers [107–110]. A variety of static and kinetic parameters can be extracted from the PET data, i.e. the standard uptake value (SUV) [111, 112], cerebral metabolic rate of glucose consumption (CMRGlc) [81, 113], mean index [114], z-scores [115], hypo-metabolic convergence index (HCI) and amyloid convergence index (ACI) [116], tissue time activity curve (TTAC) [117], and difference-of-Gaussian (DoG) parametric maps [118].

EEG and MEG computing

In EEG and MEG there are usually four components after removing the artifacts or unwanted data components that contaminate the signals. The analysis of event-related potentials (ERP) in EEG or event-related fields (ERF) in MEG aims to analyze brain responses that are evoked by a stimulus or an action, followed by spectral analysis, which transforms the signals into time-frequency domain. The aim of source reconstruction is to localize the neural sources underlying the signals measured at the sensor level. MRI is usually used to provide anatomical reference for source reconstruction. The aim of connectome analysis is to investigate the causality of brain activities and connectivity of brain networks by exploring information flow and interaction between brain regions. Gross et al. provided basic guidelines on EEG and MEG in research [51]. MNE5, EEGLAB6 and eConnectome7 are the most widely used software packages specifically designed for EEG and MEG computing.

Multimodal neuroimaging computing methods

Registration

Registration is the most commonly used technique in a neuroimaging study, and it finds the spatial relationship between two or more images, e.g., multimodal neuroimaging data alignment, serial alignment, and atlas mapping. A registration method can be defined in five aspects, i.e., a cost function for evaluating the similarity between images, a transformation model to determine the degree-of-freedom (DOF) of the deformation, an optimization method for minimizing the cost function, a sampling and interpolation strategy for computation of the cost function, and a multi-resolution scheme for controlling the coarseness of the deformation [119].

Registration methods can be roughly classified into three categories according to the DOF of their transformation models. Rigid registration has a DOF of 6 and allows for global translations and rotations. Affine registration, i.e., linear registration, allows for translation, rotation, scaling and skew of the images. Rigid and affine registration methods are usually sufficient for registering the multimodal datasets of same subject. However, deformable registration, which supports local deformations, is frequently needed to register images with large differences, e.g., registering an image to a template, or registering pre- and post-contrast images of the same subject. Deformable registration always requires rigid or affine registration to obtain a rough initial alignment. In many multimodal studies, a combination of these registration methods were used. For example, we recently jointly analyzed the ADNI FDG-PET and T1-weighted MRI datasets to classify AD and mild cognitive impairment (MCI) patients [79]. The PET images were aligned to MRI using an affine registration method (FSL FLIRT) [120]. The MRI datasets were registered to the MNI template using a deformable method (IRTK) [121], and the output registration coefficients by IRTK were applied to register the PET images to the same template. There are many other widely used registration algorithms, such as B-Spline registration [119, 122], Demons [123], and SyN [124], and ITK [54] registration framework is a standard-bearer for all of these popular registration methods.

Segmentation

Segmentation is also referred to as brain parcellation or labeling. The brain can be segmented at different levels, i.e., tissues (grey matter, white matter, cerebrospinal fluid), cortical regions, and sub-cortical regions. The segmentation methods can be classified into three categories [125]. The first category is manual and semi-automatic methods, which require manually outlining the brain regions according to a protocol [126, 127] or labeling the landmarks or seed points [128, 129]. These methods are labour-intensive and prone to intra- and inter-operator variation.

The second category is the atlas inverse mapping methods, which can inversely map a labeled atlas, e.g., the standard ICBM and AAL template, or user-defined image, to the original image space. Yao et al. recently provided a review of popular brain atlases [130]. Atlas inverse mapping is simple, but its performance heavily depends on the selected atlas and mapping method.

A more robust but complex solution is multi-atlas labeling, including the multi-atlas propagation with enhanced registration (MAPER) [131] and its variants [132, 133]. These methods carry out whole-brain segmentation in the original image space by fusing multiple labeling results derived from the multiple atlases. Multi-atlas labeling is computationally expensive, but the performance is comparable to manual labeling [125]. FSL FAST8 and NifSeg9 are widely used for brain tissue segmentation. IRTK, Advanced Normalization Tools (ANTs)10 and NifReg11 are commonly used in multi-atlas labeling as the normalization tools.

Feature fusion

Various features can be extracted from the neuroimaging data, as described in Sect. 2. Feature fusion is needed to jointly analyze the features from multimodal data. A straightforward solution is to concatenate input multi-view features into a high-dimensional vector, and then apply feature selection methods, such as t-test [134], ANOVA [118], Elastic Net [10, 135], lasso [136] or a combination of these methods [137, 138], to reduce the ’curse of dimensionality’.

These methods show promising results. However, the inter-subject variations cannot be eliminated using the concatenation methods because the inter-subject distances measured by different features may have different scales and variations. With a focus on the subjects, the feature embedding methods, such as multi-view spectral embedding (MSE) [139] and multi-view local linear embedding (MLLE) [140], have been used to explore the geometric structures of local patches in multiple feature spaces and align the local patches in a unified feature space with maximum preservation of the geometric relationships.

In addition, machine learning, especially deep learning, is increasingly used to extract high-level features from neuroimaging data. The advantage of learning-based features is they do not depend on prior knowledge of the disorder or imaging characteristics as the hand-engineered features. They are also essentially suitable for multimodal feature learning, and could expect better performance with larger datasets. However, learning-based features heavily depend on the training datasets [141]. Recently, Suk et al. [142] proposed a feature representation learning framework for multimodal neuroimaging data. One stacked auto-encoder (SAE) was trained for each modality, then the learnt high-level features were further fused with a multi-kernel support vector machine (MK-SVM). They further proposed another deep learning framework based on the deep Boltzmann machine (DBM) and trained it using the 3D patches extracted from the multimodal data [143].

Pattern analysis

Pattern analysis aims to deduce the patterns of disease pathologies, sensorimotor or cognitive functions in the brain and identify the associated regionally specific effects. A substantial proportion of pattern analysis methods focused on classification of different groups of subjects, e.g., distinguishing AD patients from normal controls [10, 138]. Hinrichs et al. [144, 145] and Zhang et al. [4] recently proposed the multi-kernel support vector machine (MK-SVM) algorithm, which is based on multi-kernel learning and extends the kernel tricks in SVM to the multiple feature spaces. We previously proposed a multifold Bayesian kernelization (MBK) model [79] to transfer the features into diagnosis probability distribution functions (PDFs), and then merge the PDFs instead of the feature spaces.

Regression-based pattern analysis is often used to identify the biomarkers of a group of subjects and probe the boundaries between different groups. The multimodal biomarkers can be based on the voxel features, ROI features and other features, as described in Sect. 2. Regression, such as Softmax regression [10], Elastic Net [135], and lasso [136] can be combined with feature learning in a unified framework.

Recently, the pattern analysis methods have been extended to simulation of future brain development based on the previous states of the brain and comparison to other brains. The basic assumption is that brains with similar cross-sectional and longitudinal deformations would have similar follow-up development [146, 147]. When the population is sufficiently large to include a majority of neurodegenerative changes, the simulated results are more accurate.

Visualization

The neuroimaging data are mainly 2D and 3D, thus can be visualized in multi-dimensional spaces with 2D and 3D viewers. Multimodal data in 2D space are usually displayed with three layers, including background, foreground and label maps. The 3D viewer enables visualization of volume data, such as volume renderings, triangulated surface models and fiber tracts. Basic image visualization functions, such as look up tables, zoom, window / level, pan, multi-planar reformat, crosshairs, and synchronous pan / scroll for linked viewers, have been implemented in most visualization platforms, such as Slicer12 and BioImage Suite13. These platforms also can accommodate visualization of high-dimensional data, e.g., tensors and vector fields.

Image markup refers to the graphical elements overlay, such as fiducials (points), rulers, bounding boxes, and label maps. Image annotation refers to the text-based information [148]. Both image markups and annotations are used to describe the meta information of the images, and annotations can be associated with markup elements as free text.

Another important use of the image markups is interactive visualization. The aforementioned platforms also provide a graphical user interface to interact with the data. For example, the volume rendering module of Slicer allows the users to define a bounding box and visualize the content in the bounding box only. Another module, tractography interactive seeding, is designed for interactive seeding of DTI fiber tracts passing through a list of fiducials or vertices of a 3D model. Slicer also allows the configuration of the layouts and manipulation of content in the viewers to suit a specific use case.

FOSS packages and platforms

Task-oriented packages

The FOSS packages for neuroimaging computing are usually initially designed for a single task, such as registration and segmentation, and some of them then are extended to related tasks and become multifunctional packages. A number of the most widely used FOSS packages are listed in Fig. 1—packages and platforms.

Popular multifunctional packages include FreeSurfer, FSL, SPM, ANTs and NifTK. They cover similar aspects of functionality, but all have particular strengths. FreeSurfer and FSL provide a comprehensive solution of analysis tools for fMRI, sMRI and dMRI data. SPM is designed for the analysis of fMRI, PET, SPECT, EEG and MEG. The recently developed ANTs and NifTK are useful for managing, interpreting and visualizing multimodal data, and represent the state-of-the-art in medical image registration and segmentation. Tustison et al. [149] recently compared ANTs and FreeSurfer in a large-scale evaluation of cortical thickness measurements. Other packages may focus on a specific task or a set of related tasks. IRTK14, BRAINs [150], BrainVisa15, ITK-SNAP16 and MindBoggle17 are popular choices for registration and segmentation. In dMRI analysis, Camino18, DTI-TK19, DSI Studio20, TrackVis21 and MRTrix22 are most widely used packages. Soares et al. [151] recently conducted a thorough evaluation of these packages and the other dMRI computing packages in published studies. In functional neuroimaging computing, AFNI, PyMVPA23 and REST24 are widely used for fMRI analysis, whereas MNE, EEGLAB, eConnectome for EEG/MEG analysis.

All integrated platforms

For clinical applications, the medical image computing and visualization functions are part of the operation system and must meet the same standards of reliability, robustness, and simplicity of operation as the core imaging equipment. This is usually accomplished using software platforms added onto imaging system by the major vendors of medical image equipment and many specialized companies. Examples include Advantage Windows (General Electric), Syngo Via (Siemens), Vital Image Vitrea (Toshiba), Visage Amira (Visage Imaging), PMOD (PMOD Technologies Ltd.), Definiens (Definiens Inc.), and MimVista (MIM Software Inc.). These packages provide users with a set of analysis tools, compatibility with PACS and customer support. Such clinically oriented systems are not always affordable for academic researchers. Commercial solutions are typically not extensible by the end user, nor oriented towards prototyping of new tools, and may require specialized hardware, thereby limiting their applicability in projects that involve the development of new image computing methods.

As opposed to the commercial platforms, FOSS platforms are meant to provide a research platform that is freely available and does not require specialized hardware. A key step in the evolution of today’s flexible and sophisticated capabilities for image-data-based research medicine was the creation of the 3D Slicer, which is based on a modular architecture [1, 152]. 3D Slicer has become a successful and long-lived platform for the effective use of volumetric images in clinical research and procedure development. There are a number of platforms which aim to cover similar aspects of functionality, e.g., BioImage Suite, BrainSuite25, MIPAV26 and MITK27.

Some of the libraries contributing to the foundation of Slicer were designed in close collaboration and often share the same developer community. These libraries, including CMake, ITK, VTK and CTK, are distributed as part of the National Alliance for Medical Image Computing (NA-MIC) Kit [153], which are actively supported by the NA-MIC research community28. Many popular packages, e.g., ANTs, MindBoggle, ITK-SNAP, DTIPrep, and MITK are also based on the NA-MIC Kit. NIPY29 and NeuroDebian30 are another two major research communities for neuroimaging research and platform development. To promote open science, neuroimaging tools and resources are always shared to other community members, usually through the INCF31 and NITRC32 forums.

Example: surgical planning for brain tumor resection

Tractography derived from dMRI has great potential to help neurosurgeons determine tumor resection boundaries in functional areas involving eloquent white matter fibers. The MICCAI DTI Challenge33 is dedicated to comparing different fiber-tracking algorithms in reconstruction of white matter tracts, such as peritumoral tracts and cerebrospinal tract (CST). In this section, we present an example of pre-operative planning for brain tumor resection using the sMRI and dMRI data. The original data consist of a DWI volume and two structure scans of a patient with meningioma. The DWI scan was acquired with a spin-echo EPI sequence with the following parameters: voxel size 2.2 $\times$ 2.2 $\times$ 2.2 mm, FOV 220 mm, 58 slices, b-value 1000 s/mm${}^2$, 30 diffusion-weighted volumes and 1 baseline volume. The T1 original was acquired using a Ax 3D T1 MPRAGE sequence. The T2 original was acquired using a Ax 3D SPACE sequence.

The original data were computed in four steps, as illustrated in Fig. 2. For dMRI-specific computing, the tensors were estimated using a weighted least square (WLS) algorithm, and the output is a DTI volume. We then registered the T1 and T2 MRI volumes to baseline volume using the affine registration algorithm. The registered T1 and T2 volumes were used as the anatomical references. For sMRI-specific computing, tumor, ventricle and motor cortex were manually seeded and semi-automatically labeled in the baseline volume. The label map of the tumor and ventricle were than used to generate the 3D surface model using the Model Maker module in Slicer. The head surface, pial surface and white matter surfaces for both hemisphere were reconstructed using the Morphologist 2013 pipeline in BrainVisa [68]. For multimodal computing, the white matter tracts were visualized using the Slicer Tractography Interactive Seeding module, which allows users to mark the image with fiducials, and then move it around the tumor to visualize the peritumoral fiber tracts.

Fig. 2.

Experimental visualization of brain tumor case of DTI Challenge 2015 using 3D Slicer. The panel on the left shows the GUI of the Slicer Mosaic Viewer module previously developed by us. The right side shows the four data viewers, each visualizing a specific step in the surgical planning workflows. The up left viewer shows the registered T1 that overlaid on the DTI volume. The up right viewer shows the segmented tumor (green), ventricle (blue), and motor cortex (red) surfaces. The bottom left viewer shows the reconstructed pial surface of the right hemisphere and white matter surface of the left hemisphere. The bottom right viewer interactively visualizes the peritumoral fiber tracts as the user moves the fiducial. (Color figure online)

Future directions

The neuroimaging techniques will keep advancing rapidly, towards higher spatial/temporal/angular resolutions, shorter scanning time, and greater image contrast. In particular, the advances in the hybrid imaging scanners, e.g., PET/CT and PET/MRI, will enter more clinics and laboratories, lowering the cost for data acquisition and enabling more interesting discoveries in a greater multitude of populations and disorders.

The continued growth in the complexity and dimensionality of neuroimaging data will spur the parallel advances of computational models and methods to accommodate such complex data. Such models and methods need to keep increasing the grade of automation, accuracy, reproducibility and robustness, and eventually need to be integrated into the clinical workflows to facilitate clinical testing of the new neuroimaging biomarkers.

The multidisciplinary nature of neuroimaging computing will keep bringing together clinicians, biologists, computer scientists, engineers, physicists, and other researchers who are contributing to, and need to keep abreast of, advances in the neurotechnologies and applications. New methods and models will be developed by the collaboration of different groups or individuals, with rapid iterations. Therefore, future packages and platforms need to respond more quickly to the updates, without compromising the functionality, extensibility and portability. This might cause difficulties in the maintenance of large packages and platforms, but will encourage the researchers to provide smarter solutions, e.g., providing an online version to make the whole process of developing, sharing and updating much quicker for both developers and users.

Biographies

Sidong Liu

received his Bachelor of Applied Science in Bioinformatics from Harbin Institute of Technology (HIT) in 2007, Master of Applied Science and Master of Information Technology from The University of Sydney in 2008 and 2009, respectively. He is currently working toward the Ph.D. degree in Neuroimaging Computing in the School of Information Technologies, Faculty of Engineering & IT, The University of Sydney. His research interests include brain image computing, biomedical and health informatics, and large-scale data analytics.

Weidong Cai

received his Ph.D. degree in Computer Science from the Basser Department of Computer Science, The University of Sydney, in 2001. He is currently working as an Associate Professor in the School of Information Technologies, Director of the Multimedia Laboratory in The University of Sydney. He has been a Lead Investigator/Visiting Professor on medical image analysis and medical computer vision at Surgical Planning Laboratory (SPL), Harvard Medical School during his 2014 SSP. His research interests include medical image analysis, image/video processing and retrieval, bioimaging informatics, computational neuroscience, computer vision and pattern recognition, and multimedia computing.

Siqi Liu

received his Bachelor of Computer Science and Technologies and Master of Information Technology, both major in computer science from The University of Sydney in 2012 and 2013, respectively. He is currently working toward the Ph.D. degree in Computational Neuroscience in the School of Information Technologies, Faculty of Engineering & IT, The University of Sydney. His research interests include computational neuroscience, longitudinal neurodegeneration prediction, diffusion magnetic resonance imaging and tractography, large-scale neuroimaging data analysis and retrieval, and single-neuron modeling and tracing.

Fan Zhang

received his Bachelor of Software Engineering and Master of Engineering in computer applied technology from School of Software, Dalian University of Technology in 2010 and 2013, respectively. He is currently working toward the Ph.D. degree in large-scale biomedical image computing in the School of Information Technologies, Faculty of Engineering & IT, The University of Sydney. His research interests include big data analytics, content-based bioimage retrieval, computer vision, and pattern recognition.

Michael Fulham

graduated in 1979 (MBBS Hons) from The University of New South Wales. He trained in Internal Medicine and Neurology in Australia and then spent 5 years at the National Institutes of Health from 1988–1993 in the Neuroimaging Branch of NINDS. He returned to Australia in 1993. He is currently a full-time senior clinician and Director of the Department of PET and Nuclear Medicine at Royal Prince Alfred (RPA) Hospital; he is also the Clinical Director for Medical Imaging throughout the Sydney Health District. He has directed the first Australian PET program at RPA since 1993 and Medical Imaging Stream in the various iterations of Area Health Services (AHSs) and Local Health Districts (LHDs) since 2000. He is also Clinical Professor in the Sydney Medical School and Adjunct Professor in the Faculty of Engineering and Information Technologies at the University of Sydney. Throughout his career, the major theme of his work has been to improve patient care through leadership, translational research and technological sciences and engineering solutions in cancer and the neurosciences.

Dagan Feng

received his M.E. in Electrical Engineering & Computer Science (EECS) from Shanghai Jiao Tong University in 1982, MSc in Biocybernetics and Ph.D. in Computer Science from the University of California, Los Angeles (UCLA) in 1985 and 1988, respectively, where he received the Crump Prize for Excellence in Medical Engineering. Professor Feng is currently Head of School of Information Technologies and Director of the Institute of Biomedical Engineering and Technology at the University of Sydney. He has published over 700 scholarly research papers, pioneered several new research directions, and made a number of landmark contributions in his field. Professor Feng’s research in the areas of biomedical and multimedia information technology seeks to address the major challenges in “big data science” and provide innovative solutions for stochastic data acquisition, compression, storage, management, modeling, fusion, visualization, and communication. Professor Feng is Fellow of ACS, HKIE, IET, IEEE, and Australian Academy of Technological Sciences and Engineering.

Sonia Pujol

is an Instructor of Radiology at Harvard Medical School, and a research scientist at the Surgical Planning Laboratory at the Brigham and Women’s Hospital in Boston. Her research focuses on the development and validation of brain mapping technology for neurosurgical intervention which includes 3D exploration of white matter architecture using Diffusion MRI for brain tumor surgery planning and 3D computer-assisted neurovascular navigation for brain aneurysm intervention. Dr. Pujol is graduated with a Ph.D. in Biomedical Engineering and a post-graduate degree in Functional Materials and Nanophysics from the University Joseph Fourier, Grenoble, France. During her Ph.D. thesis work, she developed a prototype navigation system for computer-assisted endovascular surgery of abdominal aortic aneurysm. Dr. Pujol holds a M.Sc. in Physics and a M.S.Eng in Computer Science and Applied Mathematics from the Ecole Nationale Supérieure de Physique de Grenoble and the Ecole Nationale Supérieure d’Informatique et de Mathématiques Appliquées de Grenoble, which she integrated after completing the French preparatory classes to national competitive exams at the lycee Pierre de Fermat in Toulouse, France.

Ron Kikinis

is the founding Director of the Surgical Planning Laboratory, Department of Radiology, Brigham and Women’s Hospital, Harvard Medical School, Boston, MA, and a Professor of Radiology at Harvard Medical School. This laboratory was founded in 1990. In 2004, he was appointed as a Professor of Radiology at Harvard Medical School. In 2009, he was the inaugural recipient of the MICCAI Society “Enduring Impact Award.” On February 24, 2010, he was appointed as the Robert Greenes Distinguished Director of Biomedical Informatics in the Department of Radiology at Brigham and Women’s Hospital. On January 1, 2014, he was appointed as “Institutsleiter” of Fraunhofer MEVIS and Professor of Medical Image Computing at the University of Bremen. Dr. Kikinis is the Principal Investigator of the National Alliance for Medical Image Computing (NA-MIC), a National Center for Biomedical Computing, an effort which is part of the NIH Roadmap Initiative, and of the Neuroimage Analysis Center (NAC), a Biomedical Technology Resource Center funded by (NIBIB). He is also the Director of Collaborations for the National Center for Image Guided Therapy (NCIGT), which is jointly sponsored by NIBIB. He has served and is serving as member of external advisory boards for a variety of centers and research efforts. He is the Principal Investigator of 3D Slicer, a software platform for single subject image analysis and visualization. During the mid-80’s, Dr. Kikinis developed a scientific interest in image processing algorithms and their use for extracting relevant information from medical imaging data. Since then, this topic has matured from a fairly exotic topic to a field of science. This is due to the explosive increase of both the quantity and complexity of imaging data. Dr. Kikinis has led and has participated in research in different areas of science. His activities include technological research (segmentation, registration, visualization, high performance computing), software system development, and biomedical research in a variety of biomedical specialties. The majority of his research is interdisciplinary in nature and is conducted by multidisciplinary teams. The results of his research have been reported in a variety of peer-reviewed journal articles. He is the author and co-author of more than 310 peer-reviewed articles. Before joining Brigham & Women’s Hospital in 1988, he trained as a resident in radiology at the University Hospital in Zurich, and as a researcher in computer vision at the ETH in Zurich, Switzerland. He received his M.D. degree from the University of Zurich, Switzerland, in 1982.