Fusing DTI and FMRI Data: A Survey of Methods and Applications

Abstract

The relationship between brain structure and function has been one of the centers of research in neuroimaging for decades. In recent years, diffusion tensor imaging (DTI) and functional magnetic resonance imaging (fMRI) techniques have been widely available and popular in cognitive and clinical neurosciences for examining the brain’s white matter (WM) micro-structures and gray matter (GM) functions, respectively. Given the intrinsic integration of WM/GM and the complementary information embedded in DTI/fMRI data, it is natural and well-justified to combine these two neuroimaging modalities together to investigate brain structure and function and their relationships simultaneously. In the past decade, there have been remarkable achievements of DTI/fMRI fusion methods and applications in neuroimaging and human brain mapping community. This survey paper aims to review recent advancements on methodologies and applications in incorporating multimodal DTI and fMRI data, and offer our perspectives on future research directions. We envision that effective fusion of DTI/fMRI techniques will play increasingly important roles in neuroimaging and brain sciences in the years to come.

1. INTRODUCTION

Since their inceptions in early 90s’, diffusion magnetic resonance imaging (dMRI) (Chenevert et al., 1990; Le Bihan et al., 1986; Moseley et al., 1990; Turner and Le Bihan, 1990) and functional magnetic resonance imaging (fMRI) (Biswal et al., 1995; Ogawa et al., 1990a, 1990b) have evolved into two major neuroimaging techniques to examine white matter (WM) microstructures and gray matter (GM) functions due to their effectiveness, noninvasiveness and convenience. In healthy mature brains, the major components of WM are myelinated fiber tracts that provide a nature pathway for water molecules to move along. That is, water molecules tend to move along the axon fibers rather than the perpendicular directions. Based on this basic principle, in the diffusion tensor model (Basser and Pierpaoli, 1998; Basser et al., 1994; Pierpaoli and Basser, 1996; Pierpaoli et al., 1996), a 3×3 matrix was proposed to describe the diffusion property in dMRI, also referred as diffusion tensor imaging (DTI). A compact representation of the tensor model is an ellipsoid with three principal axes, the length of which reflects the diffusion tendency along that direction. To better characterize the multidirectional fiber architecture (e.g. fiber crossing (Parker and Alexander, 2003)) within single voxels, many new techniques were proposed based on their capabilities to detect multiple diffusion directions, such as high angular resolution diffusion imaging (HARDI) (Tuch, 2004; Tuch et al., 2002) and diffusion spectrum imaging (DSI) (Wedeen et al., 2008, 2005). However, obtaining large-amount of diffusion information needs longer scanning time, which might limit the practical feasibility in clinical applications. Hence, other methods such as the two-tensor model (Peled et al., 2006; Qazi et al., 2009) were developed to computationally deal with the crossing fibers. This approach used two cylindrical tensors to model two tracts (assumption) within a single voxel. As DTI and other multi-tensor model can reveal the diffusion properties in individual voxel, a natural extension is to estimate the diffusion relationship among neighboring voxels and eventually infer the whole WM fiber pathways within the whole brain, which is typically named tractography.

Neuroimaging researchers have developed numerous algorithms and methods (Basser et al., 2000; Conturo et al., 1999; Mori et al., 1999; Parker et al., 2003, 2002a, 2002b) of DTI tractography to explore the potential and/or possible fiber tracts and exciting achievements have been made since then. Streamline tractography (Basser et al., 2000; Conturo et al., 1999; Mori et al., 1999) has been widely used due to its simplicity and efficiency. Its basic idea is to gradually depict the tracts by stepping along the direction of the fastest diffusion (principal eigenvector). Another popular method is probabilistic tractography (Behrens et al., 2003a, 2003b) which tends to generate a probability distribution of the fiber orientations from a seed voxel. Because of easy acquisition and wide availability, most multimodal fusion studies we reviewed in this paper adopted DTI to examine the WM micro-structure properties. Therefore, in this review we mainly focus on DTI and leave other dMRI based techniques such as HARDI (Tuch et al., 2002) and DSI (Wedeen et al., 2012) techniques to other surveys.

Meanwhile, fMRI aims to capture the brain’s functional activities by leveraging the relationship between neural activity and hemodynamics in the human brain (Logothetis, 2008; Friston, 2009). Since its inception in early 90’s, modern fMRI techniques have been applied widely and have revolutionized the study of human brain functions (Friston, 2009; Logothetis, 2008) in the past decade. In particular, task-based fMRI has been commonly used as a benchmark approach to localizing and mapping functionally-specialized brain regions under specific task stimulus (Friston, 2009; Logothetis, 2008). Recently, resting state fMRI has been increasingly used to map resting networks based on the premise that correlated activity patterns in the brain are reported to have similar low-frequency oscillations in resting state fMRI time series (Cohen et al., 2008; Fox and Raichle, 2007; Van Den Heuvel et al., 2008b). Notably, natural stimulus fMRI (Bartels and Zeki, 2005, 2004; Golland et al., 2007; Hasson et al., 2010; Sun et al., 2012) has also gained growing interest in recent years from the brain imaging field to study brain functions in that the human subjects are more naturally engaged in the perception and cognition of natural stimulus multimedia streams. However, this survey paper will mainly focus on task-based fMRI and resting-state fMRI data analysis methods that are combined with DTI data.

Because of the intrinsic integration of WM/GM and the complementary information embedded in DTI/fMRI techniques, combining DTI and fMRI data becomes an increasingly important methodology in both cognitive and clinical neuroscience fields (Baird et al., 2005; Calamante et al., 2013; Conturo et al., 1999; Greicius et al., 2009; Guye et al., 2003; Koch et al., 2002; Madden et al., 2007; O’Donnell et al., 2012; Olesen et al., 2003; Schonberg et al., 2006; Supekar et al., 2010; Toosy et al., 2004; Van Den Heuvel et al., 2008b; Werring et al., 1999, 1998; Ystad et al., 2011). One of the earliest studies in which both DTI and fMRI data were collected and discussed was in Werring et al., 1998. In that study, the recovery of brain structure and function of a patient who had a penetrating brain injury 18 month before was examined. One of the earliest combined study on healthy subjects (Werring et al., 1999) was from the same authors one year later, in which fractional anisotropy (FA) values (from DTI) of the activated regions (from fMRI) were studied. Even though in both of these two early attempts DTI and fMRI data were simply overlaid without registration and were discussed in a qualitative way, they successfully demonstrated the feasibility and great potential of combining DTI and fMRI together. Since then, there has been an accelerated pace of combining DTI and fMRI in the neuroimaging literatures. To the best of our knowledge, until March 2013, there are already hundreds of papers published in the literature that are related to DTI/fMRI data fusion. Previous review papers (Rykhlevskaia et al., 2008) categorized the rationale for combining DTI and fMRI data into three major reasons: 1) facilitate structural connectivity studies, 2) examine the relationship between structure and function and 3) guide neurosurgical interventions. These three motivations are still driving forces of many recent DTI/fMRI fusion studies. Besides, there have been some novel integration needs and methods that have emerged. For example, exploration of relationships between large-scale structural and functional brain networks and attempt at construction of functional correspondence using structural consistence have become important new streams in the neuroscience and brain imaging fields. Therefore, in this survey paper, we plan to divide the large amount of relevant literature works into different categories according to different roles of DTI/fMRI data in the multimodal fusion process. In the rest of this paper, we will first review methods of DTI/fMRI data fusion (Section 2), which is then followed by cognitive and clinical applications using these fusion methods (Section 3). At the end, conclusions and perspectives on DTI/fMRI fusion methods will be provided (Section 4).

2. METHODS OF DTI/FMRI FUSION

2.1. Overview

In this section, we will review different methods of how DTI and fMRI data can be combined. In general, the methods can be categorized into three classes: fMRI assists DTI (fMRI -> DTI), DTI assists fMRI (DTI -> fMRI), and joint DTI and fMRI fusion (DTI <-> fMRI). Different roles of DTI and fMRI data in the multimodal fusion process reflect how they are combined and integrated. Table 1 summarizes the abovementioned categories and selected representative works. It should be noted that we have no intention to prioritize any surveyed articles here and we apologize in advance that we have no means to include all relevant important literature works in this paper.

Table 1. Three types of DTI/fMRI data fusion methods. Selected representative publications for each category are listed in the right column.

		Representative publications
fMRI -> DTI (fMRI assists DTI)	FMRI guided fiber tracking or filtering	Conturo et al. (1999), Dougherty et al. (2005), Upadhyay et al. (2007), Guye et al. (2003), Yang et al. (2009), Ethofer et al. (2011), Shimono et al. (2012), Broser et al. (2012), Greicius et al. (2009), Van den Heuvel et al. (2008b)
	FMRI based validation	Johansen-Berg et al. (2004), Schubotz et al. (2010)
DTI -> fMRI (DTI assists fMRI)	Functional analysis based on DTI-derived network	Douaud et al. (2011), Iyer et al. (2013), Pinotsis et al. (2013)
	Inferring functional roles from structural connectivity	Zhu et al. (2012b)
DTI <-> fMRI (Joint DTI/fMRI fusion)	Joint analysis	Andrews-Hanna et al. (2007), Goble et al. (2012), Propper et al. (2010), Teipel et al. (2010), Hagmann et al. (2008), Honey et al. (2009)
	Joint modeling	Bowman et al. (2012), Calamante et al. (2013), O’Donnell et al. (2012)

2.2. FMRI assists DTI

Currently, a majority of literature works that combined DTI and fMRI reviewed in this paper fall into this category (e.g., Broser et al., 2012; Dougherty et al., 2005; Ethofer et al., 2011; Greicius et al., 2009; Guye et al., 2003; Kim et al., 2006; Kleiser et al., 2010; Lowe et al., 2008; Schonberg et al., 2006; Shimono et al., 2012; Supekar et al., 2010; Upadhyay et al., 2007; van den Heuvel et al., 2008b; Yang et al., 2009). These literature works can be further divided into two sub-streams: the first one uses task-based fMRI to guide DTI tractography or isolate fiber tracts for later analysis; the other one uses fMRI-derived activations map to validate DTI-related results, e.g., cortical parcellation based on structural connectivity.

2.2.1 FMRI guided fiber tracking or filtering

Almost half of the publications surveyed in this paper employed a similar procedure as follows. First, a specific task-based fMRI was applied on the subjects and some activated regions that have the most prominent responses to the task were identified as Region of Interests (ROIs). Then, these ROIs serve as seed points for DTI tractography, in which probabilistic fiber tracking algorithms are often used. Sometimes deterministic algorithms (e.g., streamline) are also adopted to filter those fiber tracts which entered or on the verge of the acquired ROI regions.

The rationale behind this framework can be generally characterized as three folds. First of all, using task-based fMRI derived ROIs can effectively reduce the number of fiber tracts to be considered which makes the followed analyses more efficient and focused. Second, fMRI-guided fiber tracking or filtering enables selective isolation and study of the distinct functionally-related fiber bundles. This is achieved because task-based fMRI activation detection is considered as a reliable way to locate the brain regions corresponding to a specific function. The last consideration is that in group analysis, fMRI is generally regarded more reliable than cortical folding based anatomical landmarks (e.g., Brodmann atlas) to target the group-wise corresponding ROIs. In reality, the factors that have prompted neuroimaging researches to use fMRI to assist DTI tractography could be either one or a combination of the abovementioned reasons.

In the category of fMRI-guided fiber tracking, Conturo et al. (1999) is one of the earliest studies that introduced task-based fMRI to the traditional fiber tracking process. They used visual stimulation to locate activated lateral geniculate nucleus (LGN) and visual cortex and then these activated regions were treated as the seeds for filtering the whole brain fiber tracts. With this approach, they successfully identified the geniculo-calcarine tracts which connect the visual cortex to the thalamus. Dougherty et al. (2005) also used visual stimuli to achieve a visual field map and associated it with the fibers derived from streamline fiber tracking algorithm by identifying those fibers whose endpoints are close (in the range of 2 mm) to the fMRI-derived ROIs. Thus the occipital-callosal fiber tracts were grouped and subdivided according to their projections to different areas in the visual field maps. One early attempt to probe the WM connectivity patterns of the primary auditory cortex (PAC) was also achieved with the help of fMRI. Using a particular auditory stimulus which included six pure tones varying in frequency, Upadhyay et al. (2007) successfully identified high/low frequency activation clusters (regions) as the seed points for probabilistic fiber tracking. Guye et al. (2003) is one of the earliest researchers who adopted motor tasks (hand tapping) and placed three seed points in the WM (FA maps) which are adjacent to the regions with maximum fMRI activations. Because even the neighboring voxels might result in dramatically different connectivity profiles as the seeds for tracking, placing multiple seed points here could reduce the risk of missing the important potential pathways (Guye et al., 2003). Similarly, Yang et al. (2009) used hand related motor task to identify the corresponding functional regions in the primary sensorimotor cortex (SM1) as seeds areas. The target ROI was manually placed at the known region in the pons which had already been confirmed to be associated with the tracts of medial lemniscus (ML). Then, streamline based fiber tracking approach was applied to find the connections between these two regions. Saur et al. (2010) calculated directed partial correlations (dPC) of the activated regions detected from an auditory sentence comprehension task. Only those showed direct interactions were used as seeds to guide the anatomical fiber pathways tracking. One study from Ethofer et al. (2011) found regions that showed significant effects regarding the gaze shifts by using a series of facial stimuli and made them as seed regions in which the probabilistic fiber tracking was performed. Other recent literatures include Shimono et al. (2012) and Broser et al. (2012). The former one used a psychophysical task derived ROIs, and the latter one applied two tasks (Wilke et al., 2006, 2005) to generate language-related functional masks to guide the WM structure selection. One important issue is that most of the effective fMRI BOLD signals are detected in GM, which has a small distance to the corresponding WM that are interested in. So the ROIs defined in GM typically need to be enlarged or extended by a few voxels (Supekar et al., 2010; Upadhyay et al., 2007). To successfully map the BOLD signals to the cortex surface, Li et al. (2012a) proposed a few strategies including extending those fibers which do not end in GM along their orientations and finding the corresponding WM using the normal direction of the local surface.

Besides task-based fMRI, resting state fMRI (rsfMRI) (Biswal et al., 1995; Fox and Raichle, 2007; Smith et al., 2009) also plays an important role in fiber selection. Regions of resting state network (RSNs) including default mode network (DMN) (Greicius et al., 2003; Raichle et al., 2001) can be identified as ROIs to help filter the whole brain fiber tracts. Greicius et al. (2009) applied independent component analysis (ICA), which can be used to spatially identify distinct RSNs, to define DMN including the medial prefrontal cortex (MPFC), posterior cingulate cortex/retrosplenial cortex (PCC/RSC) and bilateral medial temporal lobe (MTL). These four DMN regions were used as seeds and the fiber tracts that did not pass through or end in them were discarded. The results revealed that there exist robust structural connections between MPFC and PCC. A similar ICA method (Greicius et al., 2004) was also used by Supekar et al. (2010) who identified DMN in both children and young adult populations. The derived DMN ROIs (PCC, MPFC and MTL) were then treated as masks to filter the fiber tracts. Another study from Van den Heuvel et al. (2008b) who create the DMN map using their previously proposed voxel-based “normalized cut group clustering” approach (Van Den Heuvel et al., 2008a). After selecting those fiber tracts that touched the DMN ROIs simultaneously, the structure connections between PCC and MFC were observed and verified.

2.2.2 FMRI based validation

Though not many, there are several studies that use fMRI data to validate, confirm or examine the results derived from DTI alone. Johansen-Berg et al. (2004) achieved functionally defined supplementary motor areas (SMA) and pre-SMA regions by performing two fMRI tasks (finger tapping and serial subtraction). Then, the structural connectivity profiles of activated voxels were computed. Only based on the homogeneity of the structural connectivity patterns, these voxels were clustered into two principal components which showed strong correlations within the cluster and weak correlations with the rest. After mapping these two clusters back to the volume space, they appeared to closely correspond to the activated SMA and pre-SMA regions identified in fMRI. In this study, fMRI was employed both to select seeds for further DTI connectivity computation and to validate/examine the structural parcellation results. In a similar study, Schubotz et al. (2010) divided the lateral premotor cortex (PM) into four sub-regions and used different task-based fMRI datasets to validate if these anatomically defined regions can predict functions. Zhu et al. (2012a) also used the improved functional connectivity consistency to validate the effectiveness of the proposed structure based optimization procedure. Other related studies that used fMRI to validate structure based parcellation or examined the associated functional connectivity include (Wang et al., 2012; Zhang et al., 2012).

2.3. DTI assists fMRI

The applications of DTI aided fMRI can be classified into two classes. The first class is to construct local or whole brain structural network through DTI data, based on which people can assess the associated functional connectivity. For example, Douaud et al. (2011) first identified those WM tracts which showed significant differences between the amyotrophic lateral sclerosis (ALS) patients and healthy controls. Then, a group-level structural network (GM) related to the identified abnormal WM was constructed. Lastly, the functional connectivity derived from resting-state fMRI data was examined based on the structural network. In one recent study by Iyer et al. (2013), the authors applied a community detection algorithm to identify three networks based on the modularity of the structural connectivity. Using these three structure derived networks as platforms, a comparison between traditional Pearson correlation coefficient and Bayesian network was employed. In this study, the structural connectivity was achieved from HARDI and assisted functional examination in two aspects. First, similar to Douaud et al. (2011), the structural connectivity was treated as a foundation to explore the associated functional interaction; Second, the structural connectivity also served as the baseline to evaluate which functional modeling methods have the most agreement with the underlying structural connectivity. Other examples belonging to this class include: Pinotsis et al. (2013) who constructed whole brain structural network and based on which they analyzed the relations between the theoretical graph properties and the simulated functional dynamics, and Wang et al. (2012), Mars et al. (2012) and Zhang et al. (2012) who did parcellations of parts of the brain (the left inferior parietal lobule, temporoparietal junction, posteromedial cortex) based on structural connectivity profiles and studied the functional connectivity between these sub regions.

The second class is to use structural connectivity patterns to infer or predict the corresponding functions. For instance, the recently developed DICCCOL (Zhu et al., 2012b) (Dense Individualized and Common Connectivity-Based Cortical Landmarks) system can be viewed as an example of using DTI structural information to construct functional correspondence across populations. In brief, 358 consistent and corresponding DICCCOL landmarks were successfully identified and predicted in over 240 brains, and each landmark was optimized to possess maximal group-wise consistency of fiber shape patterns. Then, nine functional brain networks derived from the corresponding tasks or ICA analysis (working memory, auditory, semantic decision making, emotion, empathy, fear, attention, visual and default mode networks) were examined based on DICCCOLs. The results demonstrated that 95 out of the 358 DICCCOL landmarks were consistently co-localized in one or more functional brain networks and they potentially represented reliable and specific functional roles. To further investigate the underlying functional roles of the DTI derived DICCCOL system, Yuan et al. (2013) performed a large-scale meta-analysis and functionally labeled 339 DICCCOL landmarks with 55 functional networks. Moreover, Li et al. (2012b) and Hu et al. (2013) predicted and applied DICCCOLs to different brain disease populations (prenatal cocaine exposure (PCE), mild cognitive impairment (MCI) and schizophrenia (SZ)) and applied different strategies to construct functional signatures for differentiating brain conditions against controls.

Notably, the DICCCOL system is constructed solely by optimizing DTI derived WM fiber shape patterns. However, it can successfully represent and predict functional correspondence across individuals. The neuroscience basis behind DICCCOL is that each brain’s cytoarchitectonic area has a unique set of extrinsic inputs and outputs, classed as “connectional fingerprint” (Passingham et al., 2002), which largely determine the function that each brain area performs. This close relationship between structural connection and brain functions has been discussed in recent literature studies (Honey et al., 2009; Passingham et al., 2002). In general, DICCCOL utilizes the deep-rooted regularity in the structural architecture of the brain organization to infer and predict the underlying brain functions.

2.4. Joint DTI/fMRI fusion

Because more and more literature studies have revealed a close relationship between the WM structures and GM functions, a variety of studies tried to combine DTI and fMRI for joint modeling and analysis. In the fusion procedure, two modalities are combined with equally important roles and each one is facilitating or validated by the other. Even though this integration must be carried out very carefully regarding the extent and the ways applied during the combination, these joint analysis and modeling methods have shown their superiority and promise. According to the fusion depth, here we categorize the reviewed publications into the following two levels: joint analysis and joint modeling.

Joint analysis

When DTI and fMRI data are used for joint analysis, the results of analyzing different modalities will be derived separately and then are combined together to perform statistical analysis such as t-test, Pearson correlation coefficient, linear regression or more complicated models. For example, Andrews-Hanna et al. (2007) used correlation analysis to reveal the relation of mean anisotropy of WM and the functional connectivity strength between the MPFC and PCC/RSC. Goble et al. (2012) identified clusters (voxels) which showed significant age-related difference in functional activation and computed the mean percent signal change (PSC) as well as the mean FA of the corresponding WM regions. Then both two-sample t-test and Pearson correlation between PSC and mean FA were conducted. To examine the relationship between structural measure of arcuate fasciculus (AF) and language lateralization, Propper et al. (2010) calculated the correlation of the laterality indices (LI) between fMRI activations and DTI measurements including arcuate fasciculus volume (AFV) and arcuate fasciculus length (AFL).

A more complicated approach includes using sophisticated models to study the relation between structure and function or compare them in a high-order manner, e.g. network-level analysis. Teipel et al. (2010) applied two methods to examine the relation between the default mode network and its underlying white matter microstructure. The first method is a multivariate analysis of variance (MANOVA) based on principal component analysis (PCA) to study the interactions between functional connectivity and FA/MD. In the second method, the data from fMRI and DTI were combined together and a joint ICA was employed on the mixed data matrix from two modalities. Hagmann et al. (2008) and Honey et al. (2009) adopted a similar method to conduct a network-scale joint analysis of function and structure data. They parcellated the cortex into multiple sub-regions (Fischl et al., 2004) which were treated as nodes, based on which the structural and functional connectivity networks were constructed. Then theoretical network characteristics and dependency between structural and functional networks were studied. These studies are among the first ones that demonstrate the close correlation between brain structure and function from a network perspective.

Joint modeling

Different from the above-mentioned joint analysis, in which the data from DTI and fMRI are processed separately and then integrated together for statistical analysis, joint modeling means that the data from DTI and fMRI are initially integrated, formulated and reorganized as a hybrid model that is superior than using either one of these two modalities alone.

Bowman et al. (2012) proposed a new approach, named anatomically weighted functional connectivity (awFC), to redefine the functional connectivity (FC) by introducing a new distance measure to characterize the FC similarity. The new awFC descriptor is the product of the traditional FC (e.g. correlation) and an extra weight factor which reflects the strength of DTI-derived structure connectivity (SC). This structural weight includes two parameters: Π_ij and λ. Π_ij represents the structure based connectivity strength which is a value from zero to one. λ is penalty item to attenuate the effect resulted from structure according to different situations and datasets. Overall, as mentioned by the authors, the awFC framework tends to emphasize the regions which exhibit highly correlated activity when strong SC exists. A similar but different study from Calamante et al. (2013) fused DTI and fMRI in another novel way: the fiber-tracking data and FC data were combined to generate an entirely new track-weighted (TW) FC map. The basic idea is to associate the FC information to the fiber tracts and further propagate to all the voxels which traverse. For example, given SC and FC matrix at the same time, for any pair of ROIs its FC value was associated with all the connecting tracts; each element in the target TW-FC map was computed as the mean total FC values associated with the fiber tracts in the corresponding grid element. Therefore, the achieved TW-FC map contains a different image contrast which highlights the structural connections of the FC network. Indeed, both Calamante et al. (2013) and Bowman et al. (2012) deeply fused structure and function information derived from DTI and fMRI at either the algorithm or framework level. The difference is that the latter one introduced the structural constrain to create a new FC measure, while TW-FC map combined two modality data into a single high-resolution image which reflects the structural and functional properties simultaneously. Another interesting example of DTI/fMRI joint modeling is the landmark distance (LD) model (O’Donnell et al., 2012). In essence, the LD model is a novel fiber tract descriptor which is based on the fact that the spatial relationships between WM fibers, anatomical ROIs and functional activations are relatively stable. The model consists of two steps: first, each fiber is represented by a fixed number of points which ensures the equality of length of feature vectors computed later; then the distances between each point along the fiber and the functional/anatomical landmarks are measured. In this way, the LD model of a single fiber tract can be formulated as a vector with the length of f × l, where f and l are the number of points on the fiber and the number of landmarks, respectively. The advantages of the LD model include rotation-invariance, translation-invariance, efficiency and effectiveness in fiber detection/prediction, and promising results were obtained when applied to a number of patients.

3. APPLICATIONS

In this survey, we mainly focus on the applications of DTI/fMRI fusion in cognitive neuroscience and clinical neuroscience which are summarized in Table 2, while leaving other important applications such as in neurosurgery to other future review articles.

Table 2. Summary of DTI/fMRI combined applications (cognitive neuroscience and clinical neuroscience).

	Specific topics	Representative publications
Cognitive neuroscience	aging-related studies	Ystad et al. (2011), Fling et al. (2012)
	working memory	Schulze et al. (2011), Schlösser et al. (2007), Sugranyes et al. (2012)
	perception	Saur et al. (2010), Shimono et al. (2012)
Clinical neuroscience	fMRI based localization	Schonberg et al. (2006)
	examining structure- function relationship	Seghier et al. (2004), Whitwell et al. (2011)

3.1. Cognitive neuroscience

In the cognitive neuroscience field, integration of DTI and fMRI has played important roles in a variety of studies such as aging-related cognition studies (Andrews-Hanna et al., 2007; Goble et al., 2012; Olesen et al., 2003; Persson et al., 2006; Supekar et al., 2010). For example, Ystad et al. (2011) successfully fused DTI, rsfMRI and neuropsychological tests together to examine the relations between changes of the cortical-subcortical fiber tracts and the cognitive declines in aging. The ROIs derived from ICA include cortical and subcortical components, which were used to extract the associated fiber tracts. The results suggested that the integrity (FA) of some fiber bundles is significantly correlated with the cognitive measures. Fling et al. (2012) examined the structure-function relation of the motor area in older and young adults. The author constructed the functional connectivity and structural measures (fractional anisotropy, mean diffusivity, radial diffusivity and longitudinal diffusivity) of the primary motor cortices (M1 regions) using a motor area template. The statistical analysis showed that stronger functional connectivity between M1 regions is accompanied with lower structural connectivity and motor performance in older but not in young adults. Koch et al. (2010) conducted a learning-and-prediction task to check the underlying structural and functional difference between more successful learners and less successful learners. The structural measures from DTI are jointly analyzed with the activation changes. Ethofer et al. (2012) identified the location of the emotion-sensitive auditory areas through a series of voice stimulus and examined the structural/functional connectivity of this region with other brain areas. An early working memory study from Olesen et al. (2003) tried to reveal whether the developments of structure (white matter) and function (brain activity) follow the similar trend. The author utilized the correlation between an outside-scanner working memory test and the FA/functional activation changes to pre-select some regions showing developmental trend. Then the correlation was applied between the structure and functional measures. A similar working memory study which aimed to investigate the relation between the white matter and brain functional changes was conduct by Schulze et al. (2011) recently. In this study the authors performed correlational analyses among GM/WM volume, WM integrity (FA) and functional activation changes. Other examples of cognitive neuroscience studies which combined DTI and fMRI include working memory (Schlösser et al., 2007; Sugranyes et al., 2012), speech perception/recognition (Saur et al., 2010), motion perception (Shimono et al., 2012), and language lateralization (Vernooij et al., 2007), among others.

3.2. Clinical neuroscience

In general, there are at least two reasons for clinical applications to combine DTI and fMRI data together. First of all, to those brain disease patients who have lesions like tumor or traumatic injury, the architectures of the white matter are significantly changed or deformed. This makes it difficult and sometimes impossible to select a desired ROI from which fiber tractography begins. Given the absence of feasible anatomical landmarks, fMRI becomes a complementary choice for anchoring the seed points (Kleiser et al., 2010; Lowe et al., 2008). For example, Schonberg et al. (2006) examined the clinical feasibility of using fMRI derived ROIs as seeds for DTI fiber tracking. The experiment included nine patients with lesion and five healthy controls. The results demonstrated that on healthy subjects, fMRI based seed selection can achieve similar tracking results compared with anatomical based seed selection methods. On patient, however, fMRI based landmarks have evident superiority than anatomical ones in serving seeds for tracking. One of the results showed that the use of anatomical seeds was able to reconstruct only part of the superior longitudinal fascicles (SLF) while the use of fMRI based seeds enabled identification of full SLF.

The second reason is to examine the structure-function relation under a specific clinical condition, e.g., disease or traumatic injury. Seghier et al. (2004) was among the earliest studies to apply the combined event-related fMRI and DTI on a three month old infant with perinatal stroke. The fMRI data showed a robust activation in the right hemisphere, but not in left side. DTI result proved the absence of the optic radiation in the left hemisphere. After projecting the activation map to DTI data, it was found that the activated region in the right hemisphere is close to the end of optic radiation, the absence of which in the opposite hemisphere might account for the abnormal function response previously observed. A typical clinical study on eighteen progressive supranuclear palsy (PSP) patients was reported in Whitwell et al. (2011), which aimed to explore whether the altered WM pathways are companied with possibly disrupted functional connectivity. The ROIs were defined by anatomical atlas (e.g., thalamus) and ICA analysis (e.g. DMN). Correlation was performed between the regional functional connectivity and the corresponding structure measurements including GM volumes and mean FA. Upadhyay et al. (2010) employed a combined DTI and fMRI study to evaluate the potential effects of long-term opioid use on brain structure and function. One interesting finding is that in opioid-dependent patients, there exists a significant correlation between the amygdala volume and the functional connectivity of amygdala to other two brain regions, that is, the inferior orbital frontal cortex and nucleus accumbens. These two functional paths are closely related to addictive behaviors and mediating reward. Other related clinical studies include Mild Cognitive Impairment (MCI)/Alzheimer’s disease (AD) (Wee et al., 2012), Schizophrenia (SZ) (Skudlarski et al., 2010; Sugranyes et al., 2012; Sui et al., 2011; Venkataraman et al., 2012), Autism (Rudie et al., 2013; Sahyoun et al., 2010), Mild traumatic brain injury (MTBI) (Mayer et al., 2011; Zhang et al., 2010), Temporal lobe epilepsy (TLE) (Voets et al., 2009) and Posttraumatic stress disorder (PTSD) (Admon et al., 2012).

4. CONCLUSION

It is clear that there is a growing interest and motivation in combining DTI and fMRI in the neuroimaging field, e.g., the majority of the literature reviewed in this paper were published within recent years. The development of the fusion method, from simply overlaid images for visual inspection, sophisticated joint-analysis methodologies to more advanced deeply fused structure-function model, is a sign for the coming new era of multimodal brain image analysis. When we enjoy the advantages of fusion of these two modalities, however, we still have a basic question to answer: what is the fundamental gain to combine these two different image modalities of DTI and fMRI, especially for those hybrid models. To answer this question, we must try to understand the essential relationships between structural brain architecture and the underlying dynamic functions. Even though many studies have demonstrated that they are closely related, e.g., strong structural connectivity is accompanied by strong functional connectivity (Honey et al., 2009; Koch et al., 2002; Li et al., 2012a). But it is far from full understanding of the intrinsic connections behind the WM pathways mapped by DTI and functional responses we observed by fMRI. Better understanding of this question entails the advancements in both neuroimaging and brain mapping areas. For example, high angular resolution diffusion imaging (HARDI) (Tuch et al., 2002) and DSI (Wedeen et al., 2012) allow us to have a more precise fiber reconstruction; functional dynamics modeling (Li et al., 2013; Iyer et al., 2013)) can effectively bridge the gaps between the traditional fMRI analysis and the dynamic and hierarchical cognitive processes of the brain. There is no doubt that in the future, DTI and fMRI can be incorporated more deeply and comprehensively with the advancements in revealing the essential structure-function relationships of the brain. We envision that better fusion of DTI/fMRI data will significantly advance our understanding of the functions and dysfunctions of the human brain in the near future.

Highlights.

1. Review of DTI/fMRI data fusion methodologies.

2. Review of applications in cognitive and clinical neuroscience.

3. Perspectives on future research directions.