Multi-scale and Multimodal Fusion of Tract-tracing, Myelin Stain and DTI-derived Fibers in Macaque Brains

Abstract

Assessment of structural connectivity patterns of brains can be an important avenue for better understanding mechanisms of structural and functional brain architectures. Therefore, many efforts have been made to estimate and validate axonal pathways via a number of techniques, such as myelin stain, tract-tracing and diffusion MRI (dMRI). The three modalities have their own advantages and are complimentary to each other. From myelin stain data, we can infer rich in-plane information of axonal orientation at micro-scale. Tract-tracing data is considered as ‘gold standard’ to estimate trustworthy meso-scale pathways. dMRI currently is the only way to estimate global macro-scale pathways given further validation. We propose a framework to take advantage of these three modalities. Information of the three modalities is integrated to determine the optimal tractography parameters for dMRI fibers and identify cross-validated fiber bundles that are finally used to construct atlas. We demonstrate the effectiveness of the framework by a collection of experimental results.

1 Introduction

Accurate assessment of structural connectivity patterns on primate brains provides an effective vehicle to understand structural architectures, mechanisms of cortical convolution and brain function [1]. Many advanced techniques have been developed for and applied to structural pathway investigations [2–4]. Among them, diffusion MRI (dMRI) and tractography approaches have been widely applied to estimate major macro-scale white matter pathways in primate brains in vivo [2]. A large number of reports on dMRI performance evaluation can be found where it is compared to other data modalities (e.g., [5,7]), such as tract-tracing [7] and myelin stain [5] which are deemed as trustworthy proof for either the existence of 3D inter-regional connections [6] or local in-plane axonal orientations [5]. As for tract-tracing approaches, it is difficult to conduct this method on primate brains due to ethical concerns [8]. Also, available reports are based on a variety of partition schemes [11]. Thanks to the “Collation of Connectivity Data for the Macaque” (CoCoMac) database [9], researchers can now estimate and quantify a meso-scale whole-brain connectivity diagram on a user-selected brain map based on hundreds of reports. However, the absence of inter-plane information in myelin stain data and absence of pathways in CoCoMac data make it impossible to construct a whole brain wiring diagram in 3D space.

In this paper, we take the complementary advantages of the abovementioned three modalities. Multi-scale information from them is integrated to determine the optimal tractography parameters for dMRI fibers (step 1 & 2 in Fig. 1) and identify cross-validated fiber bundles that are finally used to construct a ‘hybrid’ fiber atlas (step 3), which provides trustworthy pathways rarely found in available macaque tract-tracing databases and integrates the myelin-validated coherent score to them. Several evaluation experiments demonstrate the effectiveness of this framework and the derived atlases.

Fig. 1.

Flowchart of the framework.

2 Materials and Preprocessing

dMRI dataset

MRI scans were conducted on twenty macaques under IACUC approval. T1-weighted MRI: repetition time/inversion time/echo time of 2500/950/3.49 msec, a matrix of 256×256×192, resolution of 0.5×0.5×0.5 mm³. DTI: diffusion-weighting gradients applied in 60 directions with a b value of 1000 sec/mm², repetition time/echo time of 7000/108 msec, matrix size of 128×120×43, resolution of 1.1×1.1×1.1 mm³. Caret dataset: It includes the macaque ‘F99’ atlas with both surface and volume (T1-weighted, 0.5 mm resolution) templates, to which LVE00 brain map have been mapped [10]. CoCoMac database: It includes 40,000 reports on macaque anatomical connections [9]. We can retrieve wiring information from those collated reports and construct meso-scale tract-tracing connectivity matrix based on the LVE00 brain map. Myelin stain database: 36 coronal Weil’s stain slices covering whole brain of one adult macaque brain are available in http://brainmaps.org. The myelin structures are stained dark blue and red cells are stained brown. The in-plane resolution is 0.46 mircon/pixel. The slice thickness is 40 microns.

Preprocessing: dMRI data

We conduct skull removal and eddy current correction in FSL-FDT with DTI data. We adopt deterministic tractography via DTI Studio to reconstruct streamline fibers. T1-weighted MRI is used as the intra-subject standard space. We perform co-registration by aligning FA map to T1-weighted MRI via FSL-FLIRT. Myelin stain data: To align myelin staining to the same space, we use the 8-fold down sampled images (sampling factor is 2) and conduct rigid body 2D image co-registration with maximization of mutual information [12]. By selecting slice #22 as a reference, the slices after it are consecutively registered onto the previous one. Those slices before it are consecutively registered onto the latter one. Original images are aligned accordingly. Caret data: The ‘F99’ atlas is used as the cross-subject and cross modality common space. Individual T1-weighted MRI is registered to the atlas via FSL-FLIRT. All data in individual space is linearly transformed accordingly. The reconstructed 3D myelin staining images are linearly aligned to ‘F99’ atlas via FSL-FLIRT. As LVE00 brain map has been mapped to the atlas surface [10], connectivity derived from CoCoMac database is in the common space. It should be noted that macaque brains are less convoluted, making it feasible to register all four aforementioned datasets into the same macaque atlas space with acceptable misalignment error. It is noted that linear registration was used across subjects and modalities because nonlinear registration method may impose interpolation on myelin stain data and tensors derived from dMRI. The axonal orientations and pathways estimated may not be trustworthy.

3 Methods

3.1 Micro-scale Myelin Stain Pathway Orientations

Micro-scale myelin stain pathway orientations were estimated on 256×256 size image blocks. On each block, we estimated one myelin direction.

Step 1

The blood cells, myelin structures and others have different colors (brown, dark blue and white). We used support vector machine (SVM) method [13] to decompose each image block into three probabilistic color maps. We randomly selected 500 training voxels and manually labeled the three tissues. RGB colors were used as features to train the classification model. The training model was applied to all image blocks. Fig. 2(b) illustrates the ‘myelin’ probabilistic color maps. We use m(x) to denote the myelin probability at pixel x in image I(x).

Fig. 2.

Myelin orientation estimation method. Circled numbers represent processing steps (see text for details). (a) Original 256×256 myelin stain image block. Black arrows highlight sample stained myelin represented by white dashed curves; white circles highlight sample red cells; (b) The probabilistic map of dark blue color channel; (c) The probabilistic map of myelin texture shape; (d) Binary joint probabilistic map of myelin based on (b) and (c); (e) Orientations distribution of disconnected components in (d). Probabilistic color bars are shown beneath (b)&(c).

Step 2

We analyzed the texture to estimate the probability of myelin after converting the color image to the gray-scale. We applied Hessian matrix to filter the image:

$$H(x)=\left[\begin{array}{cc}{L}_{xx}(x)& L_{xy}(x)\\ L_{xy}(x)& L_{yy}(x)\end{array}\right]$$ (1)

The Hessian matrices were computed with a series of smoothed images. The derivatives L_i (x) in Eq. (1) associated with the i^th smoothing scale were obtained by convolving image with a Gaussian kernel of scale σ_i. On H(x), we computed the eigen values l₁(x) and l₂(x) (|l₁(x)|>|l₂(x)|). Line shape descriptor is defined as follows:

$$f(x)=e^{-{(l_2(x)/l_1(x))}^2/{\sigma }_r}\cdot (1-e^{-\left(l_1^2(x)+l_2^2(x)\right)/{\sigma }_s})$$ (2)

Because each pixel x has a series of f_i(x) values at different smoothing scales, the extrema f_max(x) was used to represent the line shape probability (Fig. 2(c)).

Step 3

Final probabilistic map p(x) is the product of f_max(x) and m(x).

Step 4

we estimated the myelin orientations in a binary image resulting from thresholding the map p (x) by 0.05 (Fig. 2(d)). Connected components were extracted from the binary image. The orientation α measures the angle (in degrees ranging from −90° to 90°) between the x-axis and the major axis of the ellipse that has the same second-moments as the connected component. The connected components less than 20 pixels were eliminated. We used unimodal Gaussian model to fit the distribution of connected component orientation in Fig. 2(e). The expectation (i.e., 50 degrees in Fig. 2(e)) was used as myelin orientation feature for the image block in Fig. 2(a). One direction was estimated because we jointly analyzed the data with DTI fibers, on which only one direction was estimated from diffusion tensors. The orientation distribution is found to be robust to errors introduced by myelin identification.

3.2 Meso-scale Tract-tracing and Macro-scale DTI Connectivity Matrices

We used E to denote the tract-tracing connectivity matrix, and A to denote the DTI connectivity matrix. 91 BrainSites on each hemisphere under LEV00 parcellation scheme were used as nodes for the matrices. e_ij of E was set to be ‘1’ if BrainSite i is an injected BrainSite and BrainSite j is a labeled region in the CoCoMac database. The matrix is not guaranteed to be symmetric. More technical details can be found in [14]. As for A, Element a_ij is the number of fibers which pass through BrainSite i and j simultaneously. Within each subject, A was divided by the total fiber number for normalization, so that we obtained an average matrix $\overline{\mathit{A}}$ of all subjects to represent group-wise information. As the meso-scale connectivity matrix E is binary, threshold α was applied to $\overline{\mathit{A}}$ to binarize it by setting all values below it to zero.

3.3 Data Fusion and Atlas Construction

3.3.1 Macro- and meso-scale data fusion

The fusion of macro-scale and meso-scale data was conducted on connectivity matrix base. $\overline{\mathit{A}}$ is regulated by tractography parameters such as angular value, FA value and the threshold α. An optimal set of parameters is the one that makes the corresponding $\overline{\mathit{A}}$ maximally matched with E. The matrix matching problem is equivalent to evaluation of a binary classifier performance. Using E as ground truth, we adopted receiver operating characteristic (ROC) curves to evaluate the performance of $\overline{\mathit{A}}$s under different sets of parameters (see Fig. 3(a)). For each ROC, we had a Youden’s index, denoted as Y. The maximal Y represents the maximal matching between $\overline{\mathit{A}}$ and E, and also determines the optimal parameter set. Details can be found in [14].

Fig. 3.

(a) ROC curves of matching $\overline{\mathit{A}}$ and E; (b) DTI fibers (white curves) and myelin stain slices; (c) and (d): How to measure the similarity between fiber local orientation and myelin orientation. Green boxes highlight the local region where the fiber tract penetrates the slice at dashed black circle location. Short lines in (d) represent myelin orientations, black arrow illustrates the local orientation of the fiber tract and the blue arrow illustrates the myelin orientation.

3.3.2 Macro- and micro-scale data fusion

We attempted to score the points on a DTI fiber tract by measuring if their local orientations are coherent with counterparts on myelin data. We used t(x) to denote a fiber tract, where x is the point on it. In Fig. 3(b)&(c), we illustrate a fiber tract highlighted by yellow arrow, which penetrate four myelin slices. At a penetrating location, as illustrated in Fig. 3(d), we used the angle β (0° ≤ β ≤90°) between myelin orientation (blue arrow) and fiber orientation (black arrow) to measure if they are coherent.

For each x on t(x), we found the nearest voxel on orientation slices to compute β. If the nearest distance is greater than 1mm, we marked the point as ‘unlabeled’. Also, as the myelin stain slices only contain in-plane information (x-y plane in Fig. 3(b)–(d)), we also measured the angle between local fiber orientation and the myelin plane for each point x, denoted by γ (0° ≤ γ ≤ 90°) in Fig 3.(c). If γ = 90°, no accurate information can be projected onto the slice. Finally, an overall coherent coefficient C for all fiber tracts on a subject can be obtained by:

(3)

where M is the number of fiber tracts and N_m is point number on fiber tract #m. The product term is local coherent coefficient. Similar to Y in section 3.3.1, we measured the average C across subjects for every combination of FA and angular values.

3.3.3 Three modality data fusion and DTI fiber atlas construction

We fused the three data modalities using the following equation:

$$\underset{fa,\mathit{\text{angle}}}{\mathit{arg}\mathit{max}}(\lambda \mathit{Y}(fa,\mathit{\text{angle}})+(1-\lambda )\mathit{C}(fa,\mathit{\text{angle}}))$$ (4)

Based on Eq. (4), we identified a set of parameters so that the linear combination of Y and C can reach the maximal value. That is, with this set of tractography parameters, the DTI fibers derived macro-scale global connectivity matrix can maximally match the tract-tracing derived meso-scale connectivity matrix, while the DTI fibers can be locally coherent to myelin derived micro-scale orientations.

Finally, a fiber atlas was constructed on the optimal set of parameters. We obtained the corresponding binary DTI connectivity matrix and overlaid it with the tract-tracing one (Fig. 4(b)). ‘Both’ connections (detected by both DTI and tract-tracing) were identified and the DTI fiber bundles are extracted from individuals. The points on those atlas fiber bundles also have coherent coefficients validated by myelin data.

Fig. 4.

(a) Y and C measurements based on different parameter combinations and the final fusion result on the right-most matrix; (b) Overlapped matrices E and $\overline{\mathit{A}}$ based on the optimal parameters; (c) The optimal parameters derived DTI fibers, onto which coherent coefficients between myelin and orientations are mapped; (d) The average coherent coefficients of the ‘both’ connections in (b); (e) DTI fiber bundles extracted from the ‘both’ connections in (b); (f) ‘both’ connection DTI fiber bundles, onto which myelin coherent coefficients are mapped; (g) Left panel: ‘both’ connection DTI fiber bundles on the 43rd row of matrix in Fig. 4(b). Right panel: ‘DTI-only’ connection fiber bundles. Arrows in indicate missing fiber bundles. Corresponding fiber bundles across subjects are of the same color in (e)&(g).

4 Experimental Results

The framework was conducted on ten DTI training subjects. The parameters used for myelin orientation estimation are experimentally determined (σ_r =0.5, σ_s =450). 11 σ_i are sampled from [1, 3] with equal spacing. Ys and Cs based on different FA and angular values are shown in Fig. 4(a). Values in the two matrices are normalized to [0, 1], respectively. Optimal parameters are determined (white box, FA=0.3 and angular value=70°) where the two measurements equally contribute (λ =0.5). The optimal threshold α is also determined on the ROC curve of FA=0.3 and angular value =70°, by searching for the maximal Y. We use this parameter set in the following analysis.

We overlay the binary DTI connectivity matrix $\overline{\mathit{A}}$ obtained via the optimal parameter set with the E (Fig. 4(b)). We also map the local myelin coherent coefficients onto the DTI fibers (Fig. 4(c)). Finally, for each ‘both’ connection, we extract myelin coherent coefficients mapped fiber bundles from subjects individually. The average local myelin coherent coefficients are then computed and assigned to ‘both’ connections. The final matrix in Fig. 4(d) shows the ‘both’ connections identified by DTI and tract-tracing data as well as the associated myelin coherent coefficients. This matrix integrates group-wise, multi-scale and multi-modal data information.

Fig.4(e) shows the DTI fiber bundles extracted from ‘both’ connections in Fig.4(b). Six randomly selected training subjects are shown in the left panel. The myelin coherent coefficients are mapped onto those fiber bundles, shown in Fig. 4(f). We also apply the optimal tractography parameters to two new testing subjects and extracted the corresponding fibers as well. Generally, we can observe the consistency of those fiber bundles across subjects within and across groups. Those fiber bundles in Fig. 4(e)&(f) contain rich information and defined as a ‘hybrid’ atlas.

We take the injected BrainSite VIPm (ventral intraparietal cortex, medial part on the 43^rd row of matrix in Fig. 4(b)&(d)) as an example to quantitatively validate the consistency of those atlas fiber bundles across subjects. We extract DTI fiber bundles emanating from VIPm and connecting to other regions via either ‘both’ or ‘DTI-only’ connections. The results from four training subjects are shown Fig. 4(g). Missing fiber bundles highlighted by arrows in ‘DTI-only’ panel suggest worse fiber consistency as compared with those in ‘both’ panel. Therefore, we define a missing ratio for each connection to describe portions of training subjects with no fibers on such connection. In the whole connectivity matrix, the missing ratio on ‘DTI-only’ connections is significantly greater than the one on ‘both’ connections with p = 0.02 via right-tail (‘DTI-only’ > ‘both’, α=0.05) t-test. Finally, we compute the myelin coherent coefficients on ‘DTI-only’ connections, similar to those on ‘both’ connections shown in Fig. 4(d). The average coherent coefficient on ‘both’ connections is 0.41. It is significantly greater than the one on ‘DTI-only’ connections (0.36) with p = 5.72×10⁻⁵ via right-tail (‘both’ > ‘DTI-only’, α=0.05) t-test. Those results suggest the effectiveness and robustness of atlas constructed on the cross-validated DTI fiber bundles.

5 Discussion and conclusion

This work provides a novel framework to identify cross-validated white matter pathways by fusing multi-scale and multi-modal data. The identified DTI fibers are used to construct atlases. The merits of the framework are summarized as follows: 1) the framework suggests a set of optimal parameters for deterministic streamline tractography; 2) the ‘hybrid’ fiber atlas is not only cross-validated by multi-scale and multi-modal data, but also contains rich information. The fiber atlases have the potential to be used as trust-worthy ‘ground truth’ pathways not available in macaque tract-tracing or myelin stain databases. We will take further studies on other dMRI modalities, such as HARDI and tractography models in the future.