A Geometric Framework for Feature Mappings in Multimodal Fusion of Brain Image Data

Abstract

Fusing multimodal brain image features to empower statistical analysis has attracted considerable research interest. Generally, a feature mapping is learned in the fusion process so the cross-modality relationship in the multimodal data can be more effectively extracted in a common feature space. Most of the prior work achieve this goal by data-driven approaches without considering the geometry properties of the feature spaces where the data are embedded. It results in a huge sacrifice of untapped information. Here, we propose to fuse the multimodal brain images through a novel geometric approach. The key idea is to encode various brain image features with the local metric change on brain shapes, such that the feature mapping can be efficiently solved by some geometric mapping functions, i.e., quasiconformal and harmonic mappings. We approach our multimodal fusion framework (MFRM) in two steps: surface feature mapping and volumetric feature mapping. For each of them, we design an informative Riemannian metric based on distinct brain anatomical features and achieve image fusion via diffeomorphic maps. We evaluate our proposed method on two brain image cohorts. The experimental results reveal the effectiveness of our proposed framework which yields better statistical performances than state-of-the-art data-driven methods.

1. Introduction

The proliferation of multimodal brain image data helps researchers better understand the neurobiology of psychiatric and neurological disorders than learning from a single modality. Generally, multimodal neuroimaging data may contain images with different resolutions and dimensions, thus it is natural to design the feature mappings from different modality data to a common feature space. The cross-modality relationship is thus extracted and projected to that latent space during the feature mapping and later can be fed to the statistical analysis. The widely applied multimodal fusion methods, such as mutual information [17], independent component analysis (ICA) [4], multiple kernel learning [19], etc, use either information theory or machine learning techniques in the data-driven fashion to build feature mappings. However, these methods ignore the intrinsic structural relationship between brain image data sources, e.g. geometric properties of brain shapes. It results in a huge sacrifice of untapped information. Hence, developing effective computational frameworks to transform multimodal multi-dimensional neuroimaging data into integrated, informative biological knowledge remains an open problem.

Fusing multimodal imaging data to improve statistical analysis calls for a feature mapping to a common latent feature space. For example, a typical strategy is to find a canonical parameter space, e.g. a chosen shape template, and compute the intermediate mappings to that parameter space [1]. However, due to the diversity of multimodal features, e.g., structural features such as cortical thickness or white matter integrity, it is non-trivial to search for such a common feature space. Inspired by research in [9], we integrate imaging features from different modalities by using a concept from geometry research, i.e. Riemannian metrics, and construct a common feature space. In Riemannian geometry, the Riemannian metric is a family of positive definite inner products defined on a differentiable manifold. A manifold could have an infinite number of Riemannian metrics representing specific geometric functions. For example, Fig. 1 (A) shows how different Riemannian metrics determine different geodesic curves on a human facial surface. Our main idea is as follows. First, we adjust the Riemannian metrics defined on the original mesh to obtain the feature embeddings. In the end, we pursue physically natural and geometrically intrinsic harmonic map solutions to register the feature-encoded metrics to the same canonical space (i.e., a unit ball domain). The multimodal fusion problem thereby can be solved by geometry registration approaches.

Fig. 1:

A. The black curve is the geodesic curve between two points(left). After we update its Riemannian metric to one that induces a fattened surface, the geodesic curve is the green line between these two points(right). B. Quasiconformal mapping. C. Volumetric string constant.

In this paper, we are interested in fusing two magnetic resonance imaging (MRI) modalities, i.e., structural MRI (sMRI) and diffusion MRI (dMRI). The sMRI provides the morphometric information on grey matter (GM) on the cortical surface whereas dMRI provides additional information to model white matter (WM) integrity in the interior of the brain. The GM geometry and WM integrity play vital roles during brain developmental and pathological processes [5,2] and they are the anatomical foundation of brain cognitive functions. There are a few attempts to model the high-level relationship between these two modalities. Tozer et al. [15] developed a methodology to find correspondences between WM fiber tracts and the GM regions linked by these fiber tracts. Savadjiev et al. [12] integrated multi-scale geometric properties of WM and cortical surface by using mutual information. However, the aforementioned studies are more interested in GM regions which are anatomically connected by WM fibers but did not aim to deal with the whole brain anatomical fusion, in general.

Here, we propose to adopt the Riemannian metric to model the intrinsic relationship between GM and WM anatomical features. We develop a practical algorithm to compute quasiconformal and harmonic maps under designed metrics carrying information from sMRI and dMRI data. In the experiments, we apply our algorithm to classify patients of Alzheimer’s disease (AD), mild cognitive impairment (MCI) and Schizophrenia (SCZ) from normal control subjects and the results demonstrate considerable promise. Our main contributions are as follows. 1) Our work shows that the multimodal fusion problem for volumetric neuroimaging data can be tackled using geometry methods with defined Riemannian metrics. 2) We design a unified multimodal fusion framework that maps both surface and volume features to a shared parameterization domain. 3) We validate the effectiveness of our framework on different datasets and achieve significantly better performance than some state-of-the-art methods.

2. Theoretical Background

2.1. Surface Quasiconformal Mapping

A diffeomorphism ϕ : (M₁, g₁) → (M₂, g₂) is a conformal mapping between two Riemannian manifolds if it preserves the first fundamental form of M₁, up to a scaling factor. Every conformal mapping is an angle-preserving mapping that preserves the local geometry of the surface. One generalization of a conformal mapping is a quasiconformal map, which is an orientation-preserving diffeomorphism between Riemann surfaces with bounded conformality distortion [10]. Its first order approximations takes small circles to small ellipses of bounded eccentricity. Mathematically, given a mapping ϕ, and z and w to be the local conformal parameters of manifolds (M₁, g₁) and (M₂, g₂), respectively, such that ${\text{g}}_1=e^{2u_1}dzd\overline{z}$, ${\text{g}}_2=e^{2u_2}dwd\overline{w}$, then we say ϕ is quasi-conformal if it satisfies the Beltrami equation:

$$\frac{\partial \varphi }{\partial \overline{z}}=\mu (z)\frac{\partial \varphi }{\partial z},$$ (1)

for complex valued functions μ(z) satisfying ∥ μ ∥_∞ < 1. μ is called Beltrami coefficient (BC). The ratio between two axes of the ellipse is called dilation given by $K=\frac{1+|\mu (z)|}{1-|\mu (z)|}$ and the orientation of the axis is arg(μ(z)). Thus, the BC μ gives us important information about the properties of the map. Fig. 1 (B) shows a quasiconformal map from a circle to an ellipse. Given a $\text{BC}\mu :ℂ\to ℂ$ with ∥ μ ∥_∞ < 1, there is always a quasiconformal mapping from a complex plane C onto itself which satisfied Eq. 1 [7]. The following theory provides computation of quasiconformal mapping with designed Riemannian metric.

Theorem 1 ([18]). Suppose f : (M, g_m) → (N, g_n) is a quasiconformal mapping associated with the BC μ. $g_m=\sigma (z)dzd\overline{z}$ and $g_n=\rho (w)dwd\overline{w}$. There is a well defined auxiliary Riemannian metric on $M:{\tilde{g}}_m=e^{\sigma (z)}|dz+\mu d\overline{z}{|}^2$ such that the mapping $\tilde{f}:\left(M,{\tilde{g}}_m\right)\to \left(N,g_n\right)$ is a conformal mapping.

2.2. Volumetric Harmonic Map

Suppose M is a simplicial complex, and $g:|M|\to {ℝ}^3$ a function that embeds |M| in ${ℝ}^3$, then (M, g) is called a mesh. For a 3-simplex, it is a tetrahedral mesh, Te, and for a 2-simplex, it is a triangular mesh, Tr. The boundary of a tetrahedral mesh is a triangular mesh, Tr = ∂Te.

Definition 1 (Discrete Harmonic Energy). Suppose a piecewise linear function f ∈ C^PL(M), the discrete harmonic energy is defined as :

$$E(f)=<f,f>=\sum _{[u,v]\in K}k(u,v)\Vert f(u)-f(v){\Vert }^2.$$ (2)

Suppose that edge e_{u, v} = [u, v] is shared by n tetrahedrons, the string constant [16] is formulated as:

$$k(u,v)=\frac{1}{12}\sum _{i=1}^n{l}_i\text{cot}\left({\theta }_i\right),$$ (3)

where l_i is the length of edge e_i to which edge e_{u, v} is against and θ_i is the dihedral angles associate with the edge e_i ( Fig. 1 (C)). By changing k(u, v) in Eq. 2, we can define different volumetric harmonic maps.

3. Algorithms

Fig. 2 shows our multimodal brain image fusion framework (MFRM). Mathematically, given a brain volumetric mesh, M, where its surface is ∂M and interior is M\∂M, we want to compute a feature mapping $f_M=\left\{f_{\partial M},f_{M\backslash \partial M}\right\}\to \mathbb{D}$ to a common feature domain $\mathbb{D}$, i.e., a unit ball. It can be divided into two steps: 1) surface feature mapping f_∂M for the cortical morphometric features, and 2) the volumetric feature mapping f_M\∂M for WM anatomical features.

Fig. 2:

Multimodal brain image fusion framewrok (MFRM). The left panel depicts surface feature mapping and the right panel depicts volumetric feature mapping. Multimodal imaging features are encoded in designed Riemannian metric. The framework produces feature mappings to a common space (a unit ball) where the vertex-wise correspondence is found through the geometric resampling.

In the first part, we have a spherical surface feature mapping

$$f_{\partial M}(v):{\tau }_{surf}\left(g_{\partial M},F_{\partial M}(v)\right)\to {\mathbb{S}}^2.$$ (4)

We encode the vertex-wise surface features F_∂M(v), e.g. Gaussian curvature and cortical thickness, to the initial Riemannian metric g_∂M by using an feature embedding function τ_surf (.) and obtain a new metric $\tilde{g}\partial M$. Then, a conformal mapping is computed to get the spherical (${\mathbb{S}}^2$) parameterization of $\tilde{g}\partial M$. Here, τ_surf (.) is learned with the optimized BC. Therefore, the variations of BC make the conformal mapping, i.e. spherical conformal mapping, to be the quasiconformal mapping. Details about BC variations in τ_surf (.) can be found in Alg. 1.

The second part of the proposed framework is a volumetric feature mapping,

$$f_{M\backslash \partial M}(v):{\tau }_{vol}\left(g_{M\backslash \partial M},F_{M\backslash \partial M}(v)\right)\to {\mathbb{S}}^3,s.t.f_{\partial M}.$$ (5)

In this mapping, the spherical parameterization in the surface feature mapping, i.e. f_∂M is the boundary constraint. Thus, the two steps of feature mapping in our framework are geometrically associated together. By changing the edge weights in the string constant k(u, v) in Eq. 3, we are able to embed the volumetric feature, e.g. white matter integrity, to the initial metric on tetrahedron meshes. We then learn the volumetric parameterization through the harmonic mapping to a unit ball domain (${\mathbb{S}}^3$). Details are shown in Alg. 2.

3.1. Surface Feature Mapping

As outlined in Theorem 1, we model the surface feature fusion with quasiconformal mappings. Commonly, in geometry registration, with spherical conformal mapping, the source and target brain cortical surfaces are both mapped to a sphere and then we search for the point-wise correspondence on the sphere domain. This process relies on Riemannian metrics on the source and target surface. Here, we extend the idea of geometric mapping to the feature mapping where we design the new metrics accounting for the brain imaging features. We deform the initial metrics of the brain shapes to obtain the new metrics such that the conformal mapping between the new metrics minimizes the vertex-wise difference of features. We notice that the local distortion or shrinkage can be controlled by the BC and thus the feature mapping can be solved with the quasiconformal mapping. The conformal mapping between the new Riemannian metrics is equivalent to the quasiconformal mapping between the initial Riemannian metrics.

The relation between BC and quasiconformal mapping is unique. Let $f:{\mathbb{S}}^2\to {\mathbb{S}}^2$ be any diffeomorphism of the sphere ${\mathbb{S}}^2$. Picking any 3-point correspondence $\left\{a,b,c\in {\mathbb{S}}^2\right\}\leftrightarrow \left\{f(a),f(b),f(c)\in {\mathbb{S}}^2\right\}$, there exist two unique Möbius transformations ϕ₁ and ϕ₂ that map {a, b, c} and {f(a), f(b), f(c)} to 0,1, ∞ respectively. Then, the composition map $\tilde{f}:={\varphi }_2\circ f\circ {\varphi }_1^{-1}$ is a diffeomorphism of ${\mathbb{S}}^2$ that fix 0, 1, ∞. There is a one-to-one correspondence between the set of diffeomorphisms {f^(t)} of ${\mathbb{S}}^2$ fixing 0, 1, ∞ and the set of BCs {μ^(t)} on ${\mathbb{S}}^2$ with ∥ μ ∥_∞ < 1. Given a diffeomorphism f of ${\mathbb{S}}^2$ with the fixed point correspondence, we can represent f uniquely by a BC.

Theorem 2 ([3]). Let {μ^(t)(z)} be the set of BCs at point $z\in ℂ$ depending on a real or complex parameter t. The variation of μ^(t)(z) can be written as:

$${\mu }^{(t)}(z)=\mu (z)+t\nu (z)+t{\sigma }^{(t)}(z).$$ (6)

When t → 0, ∥ σ^(t) ∥_∞ → 0. Then for all $w\in ℂ$, we can formulate the variation of the diffeomorphism f as:

$$f^{{\mu }^{(t)}}(w)=f^{\mu }(w)+tV\left(f^{\mu },\nu \right)(w)+o(|t|),$$ (7)

locally uniformly on $ℂ$ as t → 0, where

$$V\left(f^{\mu },\nu \right)(w)=-\frac{f^{\mu }(w)\left(f^{\mu }(w)-1\right)}{\pi }\times {\int }_{ℂ}\frac{\nu (z){\left(f_z^{\mu }(z)\right)}^2}{f^{\mu }(z)\left(f^{\mu }(z)-1\right)\left(f^{\mu }(z)-f^{\mu }(w)\right)}dz.$$ (8)

We further reformulate Eq. 8 as:

$$V\left(f^{\mu },\nu \right)(w)=A(\mu (\omega )){\int }_{ℂ}\left(\begin{array}{c}{G}_1{\nu }_1+G_2{\nu }_2\\ G_3{\nu }_1+G_4{\nu }_2\end{array}\right)dz,$$ (9)

where v = v₁ + iv₂ and G₁, G₂, G₃, G₄ are real-valued functions defined on $ℂ$. In this paper, we use $\left(\begin{array}{l}a\hfill \\ b\hfill \end{array}\right)$ to indicate complex value a + ib. Eq. 7 links the variation of the BC, μ^(t), with the variation of the diffeomorphism $f^{{\mu }^{(t)}}$. In other words, the quasiconformal mapping can be conveniently solved by adjusting the BC values with a variational formula.

Given any energy function E(f) defined on the space of surface diffeomorphisms f, we re-define it as E(μ) on the parameter domain of a sphere, which is an extended complex plane obtained by using the stereographic projection. We first formulate an energy function defined on the space of BC over the conformal parameter domain C, as:

$$E(\mu )={\int }_C{\left(F(\omega )-\tilde{F}\left(f^{\mu }(\omega )\right)\right)}^2+|\mu (\omega ){|}^{2}d\omega .$$ (10)

The optimization of minimizing E(μ) can be approximated by the gradient descent approach. We derived the Euler-Lagrange equation based on Eq. 7 and Eq. 9, as follow:

$$\begin{gathered} \frac{d}{dt}|{}_{t=0}E(\mu +t\nu )=-{\int }_{C}2\left(F-\tilde{F}\left(f^{\mu }\right)\right)\nabla \tilde{F}\left(f^{\mu }\right)\frac{d}{dt}|{}_{t=0}{f}^{\mu +t\nu }-2\mu \cdot \nu d\omega \\ =-{\int }_C\left[A\cdot {\int }_{D}G(z,\nu ,\omega ,\mu )\cdot \nu (z)dz-2\mu (\omega )\cdot \nu (\omega )\right]d\omega \\ =-{\int }_C\left[{\int }_D\left(\begin{array}{c}{G}_1{a}_1+G_3{a}_2\\ G_2{a}_1+G_4{a}_2\end{array}\right)d\omega -2\mu (z)\right]\cdot \nu (z)dz, \end{gathered}$$ (11)

where $A=a_1+i{a}_2=2\left(F-\tilde{F}\left(f^{\mu }\right)\right)\nabla \tilde{F}\left(f^{\mu }\right)$. Therefore, we could derive the descent direction for μ as follow:

$$\frac{d\mu (\omega )}{dt}={\int }_C\left(\begin{array}{c}{G}_1{a}_1+G_3{a}_2\\ G_2{a}_1+G_4{a}_2\end{array}\right)dz-2\mu .$$ (12)

We note that this is a general quasiconformal mapping framework. In this study, we design our energy function by considering two important geometric features on the cortex of the human brain, i.e., curvature F₁ (ω) and cortical thickness F₂(ω). The final energy function is:

$$E(\mu )=\alpha {\int }_C{\left(F_1-{\tilde{F}}_1\left(f^{\mu }\right)\right)}^2+\beta {\int }_C{\left(F_2-{\tilde{F}}_2\left(f^{\mu }\right)\right)}^2+|\mu (\omega ){|}^{2}d\omega$$ (13)

and update μ to optimize min E(μ) as:

$${\mu }^{n+1}-{\mu }^n=dt{\int }_C\left(\left[\begin{array}{c}{G}_1{G}_3\\ G_2{G}_4\end{array}\right]\cdot \left[\begin{array}{c}\alpha a_1+\beta b_1\\ \alpha a_2+\beta b_2\end{array}\right]\right)dz-2(\alpha +\beta )\mu ,$$ (14)

where $a_1+i{a}_2=2\left(F_1-{\tilde{F}}_1\left(f^{\mu }\right)\right)\nabla {\tilde{F}}_1\left(f^{\mu }\right)$ and $b_1+i{b}_2=2\left(F_2-{\tilde{F}}_2\left(f^{\mu }\right)\right)\nabla {\tilde{F}}_2\left(f^{\mu }\right)$. The detailed computational algorithm is summarized in Alg. 1

3.2. Volumetric Feature Mapping

With the surface quasiconformal mapping in the previous part as the boundary condition, we further design a volumetric feature mapping by using volumetric harmonic map. We first create a tetrahedral mesh based on the cortical surface to model geometry of the brain interior. Then the voxel-wise geometric features computed from dMRI are projected onto the mesh vertexes. In this study, we use three important features, i.e., FA (fractional anisotropy), MD (mean diffusivity) and B0 (raw T2 signal with no diffusion weighting), which are widely applied in dMRI-based neuroimaging analyses [8]. Given a vertex v in the tetrahedral mesh M, we define a feature vector F(v) = [FA(v), MD(v), B0(v)] and each element is scaled independently by the largest value of this feature among voxels in a brain. Recall the harmonic energy defined on a tetrahedral mesh:

$$E(f)=\sum _{[u,v]\in K}\tilde{k}(u,v)\Vert f(u)-f(v){\Vert }^2.$$ (15)

By changing the string constants k(u, v) in Eq. 3, we can define different string energies. For the proposed vertex-wise DTI feature, we design a new metric based on the cosine similarity to evaluate feature difference and adopt it into the original string constant as:

$$\widehat{k}(u,v)=e^{D_{Ang}(F(u),F(v))}k(u,v),$$ (16)

where $D_{Ang}(F(u),F(v))=\frac{1}{\pi }co{s}^{-1}\left(\frac{F(u)\cdot F(v)}{\Vert F(u){\Vert }_2\Vert F(v){\Vert }_2}\right)$. If features of two vertexes are similar, the new string parameter $\widehat{k}(u,v)=k(u,v)$. If their features are totally distinct, $\widehat{k}(u,v)=ek(u,v)$.

Lemma 1. A volumetric harmonic map with the new harmonic energy defined with Eq. 15 and Eq. 16 induces diffeomorphism.

Proof. As the newly defined harmonic energy has a string constant bounded by $k(u,v)<\tilde{k}(u,v)<ek(u,v)$ where k(u, v) is strictly positive due to acute dihedral angles in every tetrahedron, the energy defined with Eq. 15 is a harmonic energy. Since the spherical boundary induces a convex boundary, a harmonic map using Eq. 15 and 16 has a global minimum which induces diffeomorphism between source and target volumes [13]. □

Suppose a mapping $f:M\to {ℝ}^3$ minimize the given string energy E(f), it can be solved with the steepest descent method by iteratively updating along the direction

$$\frac{df(t)}{dt}=-\Delta f(t),t\in M.$$ (17)

Δf(t) is the tangential component of the piecewise Laplacian of f, Δ_PL(f).

Definition 2. The piecewise Laplacian is the linear operator Δ_PL : C^PL → C^PL on the space of piecewise linear functions f on M, defined by the formula

$${△}_{PL}(f)=\sum _{\{u,v\}\in K}k(u,v)(f(v)-f(u)).$$ (18)

For a map $f:M\to {ℝ}^3$, the piecewise Laplacian of f = (f₁, f₂, f₃) is Δ_PL(f) = (Δ_PL(f₁), Δ_PL(f₂), Δ_PL(f₃)).

The detailed volumetric feature mapping algorithm is summarized in Alg. 2.

4. Experiments

4.1. Datasets and Experimental Setting

To evaluate the effectiveness of our MFRM framework, we conduct disease classifications on two independent datasets which contain subjects from AD study and SCZ study, respectively. Dataset 1 is a subset of Alzheimers Disease Neuroimaging Initiative (ADNI), the second stage of the Northern American ADNI (http://adni.loni.usc.edu). There are 120 subjects, including 42 normal controls (NCs), 46 MCIs and 32 AD patients. Dataset 2 contains the multimodal data for SCZ studies collected by COBRE (http://cobre.mrn.org/). It contains 100 subjects, 50 of them are SCZ sufferers and the rest are matched NCs. In both datasets, brain images with sMRI and dMRI modalities are provided. We extract cortical surfaces from sMRI images by using FreeSurfer toolbox (https://surfer.nmr.mgh.harvard.edu) and then compute the vertex-wise structural features such as the cortical thickness and curvatures on the extracted GM/WM boundary. The dMRI images were processed using the FSL toolbox (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki) and WM integrity features, e.g. FA, MD and B0, are measured. The cortical and WM structural features are then mapped to the FreeSurfer space for consistency. The hyper-parameters α and β in Eq. 13 are both empirically chosen to be 10. We apply linear SVM in Statistics and Machine Learning Toolbox of MATLAB (http://www.mathworks.com) as the classifier to perform disease classifications. Experiment results of accuracy, F1 scores and ROC curves with 5-fold cross validation are reported.

We compare the performance of our proposed framework with some state-of-the-art multimodal fusion methods as well as the variant of our model: RawFA: It is the non-fusion model. Raw individual FA features have been aligned to the template. Then PCA works on the new FA maps for feature dimension reduction. PCA+jICA [4]: It is the state-of-the-art method for data-driven fusion of multimodal data. FA, MD, B0 and T1 maps are fused with this method after being registered to the same space. mCCA+jICA [14]: Another state-of-the-art method uses canonical correlation analysis and ICA to extract both shared and distinct properties across modalities. pFused: A variant version of our method that fuses cortical features without considering the feature mapping between WM structures. It is the partial fusion model which is merely based on the mapping from cortical surface fusion. HPvol: Measures of the hippocampal volume which is an ROI feature typically used in AD prediction. In our MFRM method, we map FA volume images to the template based on the feature mapping and recompute the distribution of FA values. Thus each subject eventually obtains a point-wise FA feature map as the new representation.

4.2. Results

On Dataset 1, we conduct 2 kinds of classifications, i.e. binary classification and multilabel classification, to distinguish AD, MCI and NC. Statistical validation results are given in Table 1 and Fig. 3. MFRM reaches 85% accuracy in AD vs. NC, 79% accuracy in AD vs. MCI, 66% accuracy in MCI vs. NC and 62% accuracy in AD vs. MCI vs. NC. Compared to the state-of-the-art methods, our method has the relatively better performance. For example, in AD vs. NC classification, MFRM boosts the accuracy of performance by around 16% compared to PCA+jICA and 29% compared to mCCA+jICA. The similar trends are observed in other binary classification tasks. Moreover, MFRM yields significant improvements over the non-fused model (RawFA) by raising the accuracy nearly 34% in classifying AD and NC. It is also better than the partial fusion model (pFused) with an accuracy increase of around 8%. It is consistent with discoveries of previous research that disease-related brain structural alterations are partially represented by cortical or WM geometry properties [2]. Generally, performance on MCI classification tasks is worse than that on AD classification but our method still achieves superior accuracy compared with other methods. On Dataset 2, we carry out a binary classification task for Schizophrenia disease and shows the performance of all the compared methods on Table. 2. Similar to AD results, MFRM significantly outperforms other competing methods. Features learned from MFRM increase the accuracy as opposed to the partial fusion model by 7% and to the state-of-the-art methods by 15%. Together with the results on dataset 1, the proposed multimodal fusion framework beats other methods with significant improvements in performance on brain disease classification.

Table 1:

Classification performance comparison on Dataset 1 (AD).

Methods	AD vs NC		AD vs MCI		MCI vs NC		AD vs MCI vs NC
	Acc	F1	Acc	F1	Ace	F1	Acc	F1
RawFA	51.35%	0.4375	58.97%	0.7419	52.27%	0.6866	38.33%	0.5542
PCA+jICA	68.92%	0.6102	61.54%	0.7059	53.28%	0.5714	35.0%	0.3276
mCCA+jICA	58.11%	0.5373	56.41%	0.6600	48.86%	0.5545	39.17%	0.4146
pFused	77.03%	0.7119	73.08%	0.6316	62.50%	0.6292	56.67%	0.5439
HPvol	78.40%	0.7241	60.30%	0.2791	64.80%	0.6353	51.70%	0.4660
MFRM	85.14%	0.8070	79.49%	0.7037	65.91%	0.6739	62.5%	0.5946

Fig. 3:

Visualization of MFRM results and ROC curves for classification tasks ((5) for AD, (6) for SCZ). (1) is the raw volumetric data and (2) is the corresponding spherical volumetric harmonic map without feature encoding. (3) and (4) is the result of feature mapping (to a unit ball domain) of NC and AD, respectively. As we can see, compared to (3), (4) has the reduced anisotropy (more uniformly diffusive) which is consistent to the clinical discoveries [11].

Table 2:

Classification performance comparison on Dataset 2 (SCZ).

Methods	SCZ vs NC
	Accuracy	F1
RawFA	52%	0.4894
PCA+jICA	61%	0.6061
mCCA+jICA	62%	0.6346
pFused	70%	0.6591
MFRM	77%	0.7294

The above results suggest that, after the feature mapping, raw features from the same group become closer to each other and those from different groups are driven away from each other. We further confirm this observation by comparing the pair-wise similarities among the ADNI cohort. We compute the earth mover’s distances (EMD) [6] between the fused features of each two subjects and compare the result with those from unfused features. Fig. 4 shows the distance matrices. After the MFRM fusion, intra-class features indeed become relatively closer to each other and inter-class features become far away from each other.

Fig. 4:

Similarity matrix for Dataset 1 before (left) and after (right) feature fusion. The orange box marks the similarity between NC and AD, where is significantly brighter than the diagonal blocks (has the larger EMD values).

5. Conclusion and Future Work

This paper describes a geometric framework for solving a multimodal brain images fusion problem. By varying Riemannian metrics to encode multimodal brain imaging features, we build the feature mapping efficiently with geometric registration methods, i.e., quasiconformal mapping and harmonic mapping. There are several interesting directions for the future work. For example, we can apply the idea of changing Riemannian metrics for feature mapping to the analysis in brain ROIs. Besides, the variational framework designed on top of the Riemannian metrics in the Euclidean space can be extended to metrics in other geometric spaces, e.g., the hyperbolic space. Lastly, some other modalities, such as functional MR images or electroencephalography, can be the additional information sources to brain structural data and fused features derived from these modalities may contribute to the exploration of sensitive disease-related biomarkers.