Model driven EEG/fMRI fusion of brain oscillations

Abstract

This article reviews progress and challenges in model driven EEG/fMRI fusion with a focus on brain oscillations. Fusion is the combination of both imaging modalities based on a cascade of forward models from ensemble of post‐synaptic potentials (ePSP) to net primary current densities (nPCD) to EEG; and from ePSP to vasomotor feed forward signal (VFFSS) to BOLD. In absence of a model, data driven fusion creates maps of correlations between EEG and BOLD or between estimates of nPCD and VFFS. A consistent finding has been that of positive correlations between EEG alpha power and BOLD in both frontal cortices and thalamus and of negative ones for the occipital region. For model driven fusion we formulate a neural mass EEG/fMRI model coupled to a metabolic hemodynamic model. For exploratory simulations we show that the Local Linearization (LL) method for integrating stochastic differential equations is appropriate for highly nonlinear dynamics. It has been successfully applied to small and medium sized networks, reproducing the described EEG/BOLD correlations. A new LL‐algebraic method allows simulations with hundreds of thousands of neural populations, with connectivities and conduction delays estimated from diffusion weighted MRI. For parameter and state estimation, Kalman filtering combined with the LL method estimates the innovations or prediction errors. From these the likelihood of models given data are obtained. The LL‐innovation estimation method has been already applied to small and medium scale models. With improved Bayesian computations the practical estimation of very large scale EEG/fMRI models shall soon be possible. Hum Brain Mapp, 2009. © 2008 Wiley‐Liss, Inc.

INTRODUCTION

The principled combination of information from both modalities to achieve images with simultaneously high spatial and temporal resolution is what we shall term EEG/fMRI fusion [Ritter and Villringer, 2006]. It can be either data driven or model driven (see Fig. 1). Although data driven fusion provides empirical constraints for modeling, it is model driven fusion that will provide deeper understanding of neural mechanisms. This article shall review progress and challenges in analyzing brain oscillations with model driven EEG/fMRI fusion. Some recent methodological advances will also be highlighted. At the onset, we state that we limit our analysis to resting state oscillatory brain activity due, not only to space limitations, but also because there is a consistent body of work in this area that can also provide insight into the analysis of evoked and induced activity. When relevant we will include information on models with steady state stimulation. Another advantage of limiting our attention to resting state activity is that here we sidestep the mismatch in temporal resolution between EEG and fMRI, only occupying our attention with slow variations in the parameters of the resting state.

Figure 1.

Strategies for EEG/fMRI data analysis. A: Simplified underlying forward models (FMs) for fusion. In a given voxel neural activity generates an ensemble of postsynaptic potentials (ePSP). Along the left branch of the diagram the temporally and spatially synchronized summated PSPs of neurons with open fields produce the primary current density (PCD). This is the PCD FM. The volume conductor properties of the head transform the PCD into EEG/MEG which is the FM for this type of signal. Along the right branch of this diagram, the ePSP generates a vasomotor feed forward signal (VFFS) via its own FM, which in turn is transformed via the hemodynamic FM into the observed BOLD signal. Note that any one of the model constructs enumerated here (ePSP, PCD, VFFS, EEG/MEG, BOLD) is time dependent and, according to the type of modeling, can be defined as either a single variable or a vector of variables. B: Fusion by measuring covariation of the EEG and BOLD. In this data driven approach to EEG/fMRI fusion the EEG is considered to have the same time evolutin as the PCD which is considered as a driver for the BOLD signal. The time varying power in an EEG band is convolved with a hemodynamic response function, h(t), and then correlated with the BOLD signal (correlation denoted by a thick arrow). Since the temporal dynamics of the EEG are taken as a surrogate for the VFFS this is an asymmetrical type of fusion. C: Fusion by measuring co‐variation of the PCD and VFFS. This is also a data driven approach in which the FMs for the EEG/MEG and BOLD are inverted, by solving respectively a spatial and temporal inverse problem to yield estimates of the PCD and VFFS. These estimates are then correlated (thick arrow) to accomplish a fusion that is symmetrical in that both modalities are given equal a priori weight. D: Model Driven Fusion by estimating the ePSP from EEG and BOLD. This is a model driven approach in which simultaneous Bayesian inversion is carried out with all FMs. In practice this involves repeated simulations with tentative values of ePSPs and other parameters and then modifying them to maximize an statistical measure of fit. One possible method for this estimation is shown in Figure 4.

Nevertheless the theoretical interpretation of this type of activity is of great importance. Oscillatory brain activity seems to be the weft that holds together the fabric of neural computations [Varela et al., 2001]. Several types of resting state and evoked rhythmic activity have been observed in local field potential (LFP)[Engel et al., 2001], electro and magneto‐encephalographic activity (EEG/MEG)[Buzsaki and Draguhn, 2004] as well as, more recently, in recordings of blood‐oxygen‐level dependent signals (fMRI)[Fox and Raichle, 2007]. These rhythmic activities have been found to be signatures of different behavioral states. The concurrent measurement of both the EEG and fMRI [Ives et al., 1993; Laufs et al., 2008] and the emergence of EEG/fMRI fusion methods promises improved identification of the neural ensembles—and the connections between them—that generate these different brain rhythms [Goldman et al., 2002].

Regarding EEG/fMRI fusion methods, are all based on the conceptual framework shown in Figure 1A. It is assumed that neural activity is transformed into recorded EEG or BOLD signal by corresponding forward models. We simplify current knowledge by assuming that the ensemble of postsynaptic potentials (ePSP) of neurons at a given voxel is the main contributor to both types of recordings [Attwell Iadecola, 2002; Logothetis, 2002; Riera et al., 2008]. To arrive at the EEG, a first forward model summarizes ePSPs from neurons with the appropriate geometry, spatial arrangement and temporal synchronization resulting in a net primary current density (PCD) distribution. This, in turn, is subject to a second transformation, a linear spatial convolution with the EEG Lead Field [Nunez and Silberstein, 2000]. The pathway to the BOLD signal also comprises two transformations, one which transforms the ePSPs into a local Vasomotor feed forward signal (VFFS) that then undergoes a (possibly nonlinear) temporal convolution with the hemodynamic response function (hrf). Usual inference from the data to the unobserved quantities proceeds in isolation along each branch solving the following inverse problems:

• Estimating the PCD from EEG/MEG–the EEG (spatial) inverse problem [Trujillo‐Barreto et al., 2004]

• Estimating the VFFS from the BOLD‐fMRI deconvolution or (temporal) inverse problem [Glover, 1999]

In contrast, EEG/fMRI fusion involves combining information (either observed data or estimated constructs) from both of the two cascades of forward models shown in Figure 1A. Fusion methods may be classified according to two criteria:

• 1Asymmetrical versus symmetrical fusion: If one modality is given privileged status as a prior for the other modality we shall call this “asymmetrical fusion.” For example if BOLD activation is used as a spatial constraint for EEG sources [Liu et al., 1998]. By contrast “symmetrical approaches” do not assign an a priori inferential preference to any given modality [Trujillo‐Barreto et al., 2001]. In symmetrical fusion the best of each modality will be exploited and their relative importance determined from the data [Daunizeau et al., 2007].
• 2Data versus model driven fusion: A further distinction is that of data driven fusion which is based on measuring mutual dependence between the two modalities in contrast to model driven fusion which exploits models of the chain of events leading to observed measurements. More specifically, the aim is to estimate the ePSP from both either the PCD or the VFFS. In terms of a widely used distinction [Friston, 1994] data driven approaches establish functional connectivities between observables/constructs while model driven approaches establish effective connectivity between them.

We now describe in more detail data driven EEG/fMRI fusion of resting state oscillatory activity that serves as a constraint for model driven efforts. For convenience of the reader a list of terms used in this article and their abbreviations are presented in Table I.

Table 1.

List of the abbreviations used in the paper

Abbreviation	Meaning
ACP	Anatomical Connection Probability
BOLD	Blood Oxygenation Level Dependent
CBF	Cerebral Blood Flow
DCM	Dynamic causal model
DWMRI	Diffusion weighted magnetic resonance imaging
EEG	Electroencephalogram
ePSP	Ensemble of post‐synaptic potentials
EPSP	Excitatory post synaptic potential
fMRI	Functional magnetic resonance imaging
fdr	False discovery rate
hrf	Hemodynamic Response Function
Inh	Inhibitory interneurons
IPSP	Inhibitory post synaptic potential
LL	Local linearization
LRC	Long range connections
MHM	Metabolic/hemodynamic model
ODE	Ordinary Differential Equation
PCD	Primary current density
PSP	Post‐synaptic potentials
Pyr	Pyramidal cells
RDE	Random differential equations
RE	thalamic inhibitory reticular neurons
SDE	Stochastic differential equations
SRC	Short range connections
SSM	State‐space models
St	Stellate cells
TC	thalamocortical excitatory relay neurons
VFSS	Vasomotor feed‐fodward signal

CONSTRAINTS PROVIDED BY DATA DRIVEN FUSION

Most work on data driven EEG/fMRI fusion of resting state rhythms has mapped measures of association or correlation of the EEG signal and BOLD as schematized in Figure 1C. According to our classification it has been of the asymmetrical fusion type, the EEG serving as a surrogate for the VFFS (Fig. 1A). Towards this end, estimates of power in specific spectral bands are summarized over certain EEG channels, convolved with a standard hemodynamic response function, and then correlated with the BOLD time course at each voxel. The resulting image is then thresholded to produce a SPM map of EEG‐BOLD correlation. Goldman et al. [2002] found positive alpha band/BOLD correlations in the thalamus and negative ones in the occipital cortex, somatosensory areas and the insula. Such findings have been replicated and extended by several other authors as nicely reviewed by Laufs [Laufs, 2008]. The initial findings were obtained with single EEG frequency band, alpha, averaged over posterior leads. This univariate approach is open to the criticism that the observed correlations with BOLD might actually be due to other, unobserved EEG frequency bands. This problem was remedied by multiple regressions of the BOLD on all EEG frequency bands [Laufs et al., 2003; Mantini et al., 2007]. This procedure highlighted the relation of the β2 band with the “default” fMRI resting state mode. An even more comprehensive approach is that of [Martinez‐Montes et al., 2004] who carried out EEG/fMRI fusion of the original Goldman et al. data set by a structured combination of spatial, temporal and frequency information of the EEG via a multilinear version of partial least squares. This method recognizes that a multichannel EEG time varying spectrum is a 3 dimensional array indexed by channel, frequency and time that can be decomposed into a sum of EEG “atoms” which each have a given spatial, spectral and temporal signature. The way these atoms are extracted ensures maximal covariance of their temporal signatures with those of BOLD atoms (with time and spatial signatures). Figure 2A shows the inverse solution of the EEG alpha atom spatial signature [Bosch‐Bayard et al., 2001] and the fMRI alpha spatial signature with a significant correlation that is positive for the thalamus and negative for cortical areas. It also shows that the EEG sources that mainly contribute to these correlations are concentrated in the occipital areas. The pattern has been speculated to be due to desynchronization of EEG generators with fluctuations to higher levels of vigilance and lower alpha power a conclusion reinforced by the experimental manipulation of these atoms by switching the subject from a resting state to mental arithmetic [Miwakeichi et al., 2004].

Figure 2.

(Color) A: Fusion by correlation of EEG time varying spectra over all channels and BOLD. Data driven asymmetrical fusion as in Figure 1b where EEG spectra convolved with hemodynamic response function is correlated with BOLD signal using multi‐linear partial least squares which detects common atoms in both modalities. Top: Spatial signatures of PCD for the EEG alpha atom estimated by f inverse solution. Bottom: spatial signature of BOLD alpha atom. B: Fusion by correlation of log PCD alpha power and log VFFS. Symmetrical data driven EEEG/fMRI fusion as in Figure 1c obtained by estimating the time course of EEG source power in the alpha band using the VARETA inverse solution, estimating the VFFS by smooth deconvolution of the BOLD signal with a standard hemodynamic response, and correlating the log of these quantities at each voxel. C: Correlation of source EEG and BOLD in neural mass network of moderate size. Correlation of estimated PCDs and VFFS obtained from a medium sized simulation of realistically connected neural masses. Connectivity measures for neural masses were estimated by means of DTI images. Correlation values threshold using the local false discovery rate (fdr). Red corresponds to positive correlations, blue to negative correlations. Simulations produced by the Local Linearization (LL) integration scheme for stochastic differential equations. D: Correlation of source EEG and BOLD in large scale neural mass network (surface of cortex and thalamus. Correlation of estimated PCDs and VFFS obtained from a very large sized simulation of realistically connected neural masses. Correlations shown for the cortical surface (above) and the thalamus (below not shown to scale) with the rostral part shown to the left. The cortical surface comprised 8203 Jansen modules were placed) and the left portion of the thalamus 438 TC/RT modules. The simulation produced (not shown) similar distributions for the right brain. All modules interconnected using the connectivity matrix shown in Figure 5. Note positive (red) PCD/BOLD correlation for caudal thalamus and negative (blue) for part of the occipital cortex marked by an arrow. Simulations produced by the approximae Local Linearization (aLL) integration scheme for stochastic differential equations.

Data driven methods for EEG/fMRI fusion of brain oscillations can be further improved. We now give a example. As mentioned these methods are asymmetrical in the sense that the EEG is taken as a surrogate for the VFFS but this involves degrading the temporal resolution of the EEG signal by filtering with a low pass signal (the hrf). A higher resolution, symmetrical, data driven fusion can be gained by measuring the correlation between estimates of the PCD and VFFS instead of using the usual correlation between the hrf filtered EEG and BOLD. Results from a 96 channel concurrent EEG/fMRI recording of the resting state are shown in Figure 2B. The estimate of power at the alpha peak of the nPCD was obtained by means of the VARETA inverse solution [Bosch‐Bayard et al., 2001]. The VFFS at each voxel was obtained by a spline variant of BOLD deconvolution (Appendix A). We found more widespread correlations with nPCD/VFFS fusion than with EEG/BOLD fusion, the pattern here being thalamic and anterior cortical areas directly related and posterior cortical areas inversely related to alpha power. It should also be pointed out that much higher correlations (range −0.73 to 0.52) were found when comparing the logarithms of PCD and VFFS than when comparing EEG and BOLD (range −0.54 to 0.41).

Thus a consistent pattern for resting state EEG/fMRI relations has been described by a number of authors. We now turn to model driven EEG/fMRI fusion methods to see if these patterns can be explained.

MODEL DRIVEN EEG/fmri FUSION: STATE SPACE MODELS

Model driven EEG/fMRI fusion is predicated on the formulation of explicit biophysical model for the two different chains of forward events (Fig. 1A) that lead from the ePSP to the EEG, on the one hand, and to BOLD on the other. Once these models are formulated it is possible to:

• Simulate EEG and fMRI signals originated by neural activity and study their interrelation

• Given data, estimate neural activity—the ePSP—as well as other model parameters (Figs. 1D and 4).

Figure 4.

Local linearization (LL)‐innovation approach to estimating states and parameters of nonlinear random systems. A state space model of a dynamical system is the combination of: (a) A continuous time stochastic differential equation describing the evolution of system states x, and (b) A discrete time observation equation that explains how the observations y _t are obtained from the states. Discretization of the state equation by means of the Local Linearization scheme allows the application of sequential Bayesian inference of the unobserved states via the Kalman Filter. This in turn allows the estimation of the innovations or prediction error which in turn can be used to estimate the likelihood of the model. Once the likelihood has been obtained estimation of model parameters and comparison of models is possible.

EEG/fMRI models are particular cases of State Space Models (SSM)[Kalman, 1960]1

(1)

The first line expresses the state equation, a set of stochastic differential equations (SDE) that describe how the state vector x(t) of the system evolves in continuous time. The vector v(t) describes external inputs, controls or causes that influence the system. The set of parameters specifying the model is Θ. The vector contains the dynamic noise, random inputs to the system that are modeled as a Gaussian white noise process2. μ is the mean of the random input and Σ is a square root of a covariance matrix. The second equation, the observation equation, describes how the observations y _t are determined by the states x _t = x(t) at discrete time instants t, and corrupted by measurement noise e _t. SSM are well known in control theory since the 1960s [Frost and Kailath, 1971; Kalman, 1960].

In the neuroimaging literature SSM have become popular under the name Dynamic Causal Models (DCM) [Friston et al., 2003]. The initial formulation of DCM [Friston et al., 2003] did not consider noise inputs (Σ = 0) and was therefore a deterministic SSM stated in terms of ordinary differential equations (ODE). More recent versions have been in terms of SDEs [Chen et al., 2008; Friston et al., 2008; Stephan et al., 2008]. We note that since the focus of this article is EEG/fMRI fusion of resting state oscillations, we shall not consider in the remainder of this article the external inputs v(t) in Eq. (1). SSM (DCM) with external inputs v(t) are of course necessary for studying event related [(David et al., 2006b, c; Friston, 2006]) or induced activity [Chen et al., 2008].

Mapping the EEG/fMRI generative model of Figure 1A onto a SSM we define:

• x ^ePSP are the state variables that define the evolution of ePSP according to some neural model.

• P is a projection matrix that selects and sums thosecomponents of the ePSP that contribute to the PCD.

• K is the lead field matrix that projects the PCD to an observed EEG/MEG measurement.

• R is the projection matrix that transforms x ^ePSP into the VFFS x ^VFFS; are the state variables that together with x ^VFFS define the evolution of a metabolic hemodynamic model (MHM) [Sotero and Trujillo‐Barreto, 2007, 2008].

• g ^BOLD is the model that relates the MHM model to the observed BOLD signal.

• y ${}_{\mathit{\text{t}}}^{\text{EEG}}$ and y ${}_{\mathit{\text{t}}}^{\text{BOLD}}$ are the discretized EEG and BOLD measurements respectively.

The particular form of SSM that underlies model driven EEG/fMRI fusion (Fig. 1D) is then

(2)

In these equations state variables, functions, dynamic noises, measurement noises and parameters all are labeled with corresponding superscripts that describe the part of Figure 1A they refer to. Note that we have assumed a deterministic model for the MHM though this can be generalized to a stochastic model with dynamic noise [Sotero et al., 2008].

Models f ^ePSP for describing the evolution of ePSPs (at the root of Fig. 1A) can be constructed at various levels of detail. Recent examples [Izhikevich and Edelman, 2008] have incorporated comprehensive information about the microcircuitry [Riera et al., 2008] of the brain. We shall focus rather on a mesoscopic level more suited to the coarse grained nature of both EEG and fMRI measurements. These are the class of neural mass models for EEG oscillators [David and Friston, 2003; Jansen and Rit, 1995; Lopes da Silva et al., 1974; Moran et al., 2007; Valdes et al, 1999a, b ; Wilson and Cowan, 1972] obtained by mean field approximations of subpopulations of excitatory and inhibitory neurons each governed by well known dynamics. See [David et al., 2006a; Harrison et al., 2006] for a review on how to obtain these equations. We shall now specify these mesoscopic models in a format slightly different from the cited references.

Consider i = 1,…,N _m neural mass models that give rise to the state vector x ^ePSP, these are chosen to correspond to a discretization of the brain that can be as coarse as the grid for estimating nPCD, BOLD or even finer. Each neural mass x ${}_{\mathit{\text{i}}}^{\text{ePSP}}$ is cast as a noisy oscillator (Fig. 3A)

(3)

Figure 3.

Neural Mass model for EEG/fMRI fusion. This model is obtained by linking the parameters of a interconnected set of N _m neural masses to the forward models for EEG and fMRI outlined in figure 1A where averaged population values of postsynaptic potentials (PSP) serve as state variables. A: State Space Model (SSM) for a Neural Mass. Each Neural Mass (numbered as i) is shown on the right part of the figure and depicted as a component which receives two external inputs: the net input PSP z _i(t) (continuous arrow), and Gaussian white noise (dashed arrow). The neural mass produces as an output the ePSPs x ${}_{\text{2}\mathit{\text{i}}\text{−1}}^{\text{ePSP}}$ which are also the state variables for this SSM. The ePSPs will feed into other neural masses. Note that z _i(t) is the sum of the ePSPs that are emitted from other connected neural masses, amplified by the synaptic contact coefficients z _i(t) = ∑${}_{\mathit{\text{j}}\text{=1}}^{\text{N}{}_{\mathit{\text{m}}}}$ c _i,j x _2j−1(t), where c _i,j indicates connections from population j to i. By convention these synaptic connectivities will be negative for when j is an inhibitory cell. On the left is shown in more detail the sequence of operations which take input to output: 1 − z _i(t) is converted into an average pulse density of action potentials p _i(t) = S(z _i(t)) by a static nonlinear sigmoid function S(v). 2‐The noise input, with mean μ_i and standard deviation σ_i, is added to the pulse density. 3 ‐ is converted into the output x ${}_{\text{2}\mathit{\text{i}}}^{\text{ePSP}}$(t) by linear convolutions with the neural PSP impulse responses h _i(t). The complete neural mass model creates two additional signals (not shown) that will act as the two components of the VFSS signal originating the BOLD signal. These are the sum of all EPSP u _E(t) = ∑${}_{\mathit{\text{i}}\text{=1,}\mathit{\text{N}}{}_{\mathit{\text{m}}}\text{;j excitatory}}$ c _i,j x2_j−1(t) and all IPSP u _I(t) = −∑${}_{\mathit{\text{i}}\text{=1,}\mathit{\text{N}}{}_{\mathit{\text{m}}}\text{;}\mathit{\text{j}}\text{inhibitory}}$ c _i,j x2_j−1(t). B: Model for a Jansen‐Rit Cortical Module. Here N _m = 3. Pyramidal cell (Pyr), stellate cell (St), and inhibitory interneuron (Inh) populations generate ePSPs, denoted respectively by {x ${}_{\mathit{\text{1}}}^{\text{ePSP}}$ (t), x ${}_{\mathit{\text{3}}}^{\text{ePSP}}$(t), x ${}_{\mathit{\text{5}}}^{\text{ePSP}}$(t)}. The output of Pyr cells, x ${}_{\mathit{\text{1}}}^{\text{ePSP}}$(t), multiplied by c _1,2, c _3,2 drives the St and Inh populations. Pyr receives feedback from St and Inh with coefficients c _2,1, c _2,3 respectively. The only dynamic noise input is to the St Population. On the one hand, the nPCD is the transmembrane PSP of the Pyr population which is equal to the net input PSP z ₂(t) generated by the St and Inh populations z ₂(t) = c _2,1 x ${}_{\text{1}}^{\text{ePSP}}$(t) + c _2,3 x ${}_{\text{3}}^{\text{ePSP}}$(t) (EPSP‐IPSP). This is also equal to the EEG with the trivial lead field K = 1. Loosely speaking, this is as though we were actually measuring a Local Field Potential (LFP). On the other hand, The VFFS has two components , the sum of excitatory PSP and u _I(t) = c _2,3 x ${}_{\text{5}}^{\text{ePSP}}$(t) the inhibitory PSP. The VFSS components are fed into the Metabolic Hemodynamic Model (MHM) which we shall not detail here (see Appendix C) a system of ODE which depends on additional state variables x ₇(t),…,x ₁₄(t). The observed BOLD is generated by the Balloon model then is transformed to BOLD by the equation y ₂(t) = g ^BOLD (x ₁₃(t),x ₁₄(t)).

The variable x ${}_{\text{2}\mathit{\text{i}}\text{−1}}^{\text{ePSP}}$(t) is the average output ePSP produced by each neural mass which can be either excitatory (EPSP) or inhibitory (IPSP) while is its time derivative. The variables x ${}_{\text{2}\mathit{\text{i}}\text{−1}}^{\text{ePSP}}$(t) are the transformation of the net input to the mass z _i(t) by the application of first a nonlinear sigmoid function S(z _i(t)) and then a linear convolution with the neural PSP impulse response functions h _i(t). The Laplace transform of h _i(t) originates the second order ODE. Note that dynamic noise may be added to S(z _i(t)) transforming (3) into a SDE.

The sigmoid function is defined [David et al., 2006a] as , where 2e ₀ is the maximum firing rate, v ₀ is the postsynaptic potential (PSP) corresponding to a firing rate e ₀, and parameter r controls the steepness of the sigmoid function S(z _i(t)). These sigmoid function parameters are usually fixed. The impulse responses are defined as h _i(t) = A _i a _i te ${}^{\text{−}\mathit{\text{a}}{}_{\mathit{\text{i}}}\mathit{\text{t}}}$ where the parameter A _i represents the maximum amplitude of the EPSP or IPSP, while the lumped parameters a _i depends on passive membrane time constants and other distributed delays in the dendritic network. Note that these constants differ for excitatory and the inhibitory neural masses respectively, though they are usually considered constant for whole groups of neural masses (see Appendix B for typical parameter values).

A Neural Mass model is the interconnection of several neural masses. The ePSPs x ${}_{\text{2}\mathit{\text{i}}\text{−1}}^{\text{ePSP}}$(t) produced by each component population will feed into other neural masses. Note that z _i(t) is the sum of the ePSPs that are emitted from other connected neural masses, amplified by the synaptic contact coefficients , where C = {c _i,j} indicates synaptic strengths of connections from neural mass j to neural mass i. By convention these synaptic connectivities will be negative when j is an inhibitory cell. These relations are summarized by:

(4)

In correspondence with the general EEG/fMRI SMM (2) the recorded EEG is

(5)

in which expression several things have happened

• x ^ePSP(t), and therefore z(t) has been discretized (see the next section on correct ways of doing this).

• Of all z _t only those corresponding to neural populations with open fields [Nunez et al., 2000, 2001] will contribute to the nPCD, as selected by the matrix P _open which also determines the level of spatial averaging we shall select for the primary current.

• The EEG is obtained by projection to the lead field and addition of sensor noise.

Both EPSP and IPSP may contribute to the BOLD signal [Sotero and Trujillo‐Barreto, 2007, 2008; Sotero et al., 2008; Riera et al., 2006; Babajani and Soltanian‐Zadeh, 2006; Babajani et al., 2005]. We will therefore consider additional variables that form part of the VFSS vector. These are the sum of all EPSP u _E(t) and all IPSP u _I(t) in a volume generating the BOLD signal

(6)

Generation of the BOLD signal proceeds then according to Appendix C

With this formalism in place we can discuss the different models, of every increasing complexity, have been dealt with in the literature. We shall classify these models into small scale, medium scale, and large scale models according to the ability to deal with hundreds, thousands and hundreds of thousands of neural masses.

SIMULATION OF STATE SPACE MODELS

We shall now review methods for simulations of SSM. These simulations are useful for two reasons. In the first place inspection of the simulated time series and comparison with actual data can provide face validity for the models being proposed. Moreover bifurcations of the modeled nonlinear systems with changes in parameter values suggest neural mechanisms of normal and abnormal brain activity [Lopes da Silva et al., 2003; Breakspear et al., 2006; Coombes et al., 2007]. In the second place, as will be argued in the next section, repeated simulations are the basis for the estimation of states and parameters.

Simulation consists of integrating the system of SDE (1) which, for interesting nonlinear cases, cannot be done analytically. Therefore it is customary to find either:

• global approximations that allow analytical solutions for theoretical work [Moran et al., 2007] or

• local approximations in order to calculate numerical values of the state vector x ${}_{\mathit{\text{t}}{}_{\mathit{\text{k}}}}$ = x(t _k) for time steps t _k = k Δt by integrating the system from t to t + Δt where Δt is the integration step [Valdes et al., 1999a, b].

The basis of such approximations is the Taylor‐Ito expansion [Kloeden and Platen, 1995; Jimenez et al., 1999] of a SDE around a reference point x ₀. Retaining only first order terms this leads to the following approximation for the SSM (1) around a reference value x ₀

(7)

with b(x ₀, Θ) = {b _i(x ₀, Θ)} = {Tr(ΣΣ^t F _i,xx (x _t, Θ))}, Note that for a deterministic SSM Σ = 0, we are dealing with a ODE instead of a SDE, and Eq. (7) reduces to the usual Taylor expansion. However for a SDE the Ito calculus takes into consideration a further term.

Global approximations have been mostly of the linear deterministic kind that is to say assuming Σ = 0 and taking x ₀ to be fixed for the whole analysis. Unfortunately, as mentioned before, this ignores the stochastic component of the SSM. Examples with expansion around x ₀ = 0 are some formulations of DCM [Friston et al., 2003; Friston et al., 2007]. This is a rather arbitrary choice and for that reason an alternative is x ₀ = x _s, the solution to the steady state equation f(x _s,Θ) = 0. Such is the choice taken for example by [Robinson et al, 2004; Moran et al., 2007, 2008; Zetterberg et al., 1978]. As long as the system is operating around this steady state values the approximation is valid and allows transformation of the equations to the frequency domain and the analytical determination of notable properties of the system. However the linearized system does not preserve some important nonlinear properties. This is evident, for example, when several equilibrium points are present or there is a stable attractor such as a limit cycle which will not appear in the globally linearized discretization which only has a single point attractor.

For this reason many articles examine simulated trajectories of the state space variables. For the EEG most [Lopes da Silva et al., 1974; Jansen and Rit, 1995; Babajani et al., 2006] have used off the shelf methods developed for Ordinary Differential Equations (ODE). This can be problematic for several reasons. In the first place, even for deterministic SSM (where the noise component is set to zero), these methods are not guaranteed to reproduce the properties of the original continuous dynamical system, in other words they may alter the dynamical invariants of the original continuous system [Biscay et al., 1996; Carbonell et al., 2007]. For SDEs the situation is even worse. Tong has shown [Tong, 1990] that simulations of stochastic systems with polynomial based methods (such as Runge‐Kutta) are almost sure to explode. Additionally, many of these methods are computationally very expensive and are not well suited to migrate up to large scale simulations. Avoiding polynomial expansions and based on the Taylor‐Ito expansion in Eq. (7) one can obtain integration methods by retaining only the constant term which lead to the Euler (ODE) and Euler‐Maruyama (SDE) methods. These however have poor orders of convergence.

These are pitfalls avoided by the fully stochastic Local Linearization technique of Ozaki (henceforth designated the LL Method). The LL method integrates the linear approximation of the SSM (7) around the current point x ₀ = x _t in the orbit of the system in state space:

(8)

This integral has an exact solution which can be stated as

(9)

with _t a stochastic process with mean 0 and covariance matrix

Expression (9) is quite imposing but several different fast and accurate methods for calculating it are available. These numerical variants are known as “LL‐schemes” [Jimenez and Carbonell, 2005; De la Cruz et al., 2007] and care should be taken because some have proven to be more accurate than others. The major computational burden in these schemes is the evaluation of matrix exponentials such as e ${}^{\mathbf{\text{F}}{}_{\mathbf{\text{x}}}\text{(}\mathbf{\text{x}}{}_{\mathit{\text{t}}}\text{,Θ)Δ}\mathit{\text{t}}}$.

The LL method is accurate, stable and conserves the dynamical properties of the original continuous time system. It was first introduced by [Ozaki, 1989, 1992a], later elaborated by [Biscay et al., 1996] and extensively developed in recent years [Carbonell et al., 2002, 2005, 2006, 2007, 2008; Carbonell and Jimenez, 2008; De la Cruz et al., 2007; De la Cruz et al., 2006; Jimenez et al., 1998, 1999, 2002, 2005, 2006; Jimenez, 2002; Jimenez and Biscay, 2002; Jimenez and Carbonell, 2006]. Though originally restricted in its applications to neuroimaging [Valdes et al., 1999a, b; Riera et al., 2004, 2006, 2007], the LL method has gained popularity in recent years. A recent article incorporates it for the first time into the DCM of fMRI signals [Stephan et al., 2008] though using a less accurate LL‐scheme.

ESTIMATION OF STATE SPACE MODELS

Given data [y ₁,…,y ${}_{\mathit{\text{N}}{}_{\mathit{\text{t}}}}$] gathered at N _t time points, in our case the EEG/fMRI measurements, it is often of interest to estimate information about the SSM under examination. The two main estimation problems are

• Assuming Θ known, to estimate the x _t. Predicted estimates are obtained sequentially from data gathered previously to time t, [y ₁,…,y _t−1]. Filtered estimates are obtained sequentially from data gathered up to time t, [y ₁,…,y _t]. Finally, smoothed estimates are obtained with the whole sample of observations, [y ₁,…,y ${}_{\mathit{\text{N}}{}_{\mathit{\text{t}}}}$].

• Assuming either filtered or smoothed estimates of the states, to estimate the parameters of the system, Θ.

The methods for estimation depend critically on the simulation (approximation) method selected. As discussed in the previous section, the type of approximation determines the properties that the estimated system will be able to exhibit. For example, with a global linear deterministic approximation of the SSM [Robinson et al., 2004; Moran et al., 2008] around a stable operating point x _s, and a linear observation equation the SSM is essentially treated as a linear device. The SSM equations (including a stochastic input) how read

(10)

with f = f(x _s,Θ), F = f _x(x _s,Θ), and G = g(x _t,Θ,v(t)).

Transforming these equations to the frequency domain and rearranging terms yields

(11)

where the subscript ω indicates discrete frequencies and noise variables now have a complex multivariate Gaussian distribution. This expression can then be used to estimate the parameters Θ. In practice expression (11) is not used for parameter estimation [Robinson et al., 2004; Moran et al., 2008]. Rather the spectra (variance of complex values) of y _ω are used as the data. This discards all the phase information available in the time series. It would be interesting to see if improvements in estimation could be obtained by using (11) or directly the linear SSM (10) with the methods to be described next.

Instead of global linearization, an alternative is to respect the full nonlinearities of the system by repeatedly generating EEG/fMRI simulations in the time domain (or equivalently calculating their mean and covariance at each point) and maximize their likelihood given the data by adjusting the parameters Θ. In short, we advocate the use of the the LL‐innovation method, designed to preserve the dynamical properties of the continuous time models. This method, outlined in Figure 4, was originated by T. Ozaki [1992b] and further developed in [Biscay et al., 1996; Valdes et al., 1999a, b]. It consists of the following steps:

• 1The same LL discretization scheme described in the section on SSM simulation is used to generate a linear SDE for a given position at a trajectory in state space.
• 2Once a locally linearized discrete state evolution is available, it can be fed together with the linearized observation equation and the data to any of several variants of the Kalman Filter in order to get predicted and filtered estimates of the states.
• 3Of extreme importance is that these prediction estimates may be used to estimate the innovations ϵ_t = y _t − E[g(x ${}_{\mathit{\text{t}}}^{\mathit{\text{t}}\text{−1}}$, e _t)] or prediction error, the extra information about a state space orbit, given the past data and after accounting for observation noise. As shown early on by Ozaki [1992b], with a properly chosen model for the dynamics, the innovations will be distributed as Gaussian white noise even if the system is highly nonlinear and the noise processes are not Gaussian.
• 4
  Finally in view of the Gaussian nature of the innovations, the log likelihood of the SSM given the data may be calculated as:

  (12)

  with Ψ(Θ)=Σ(Θ)Σ(Θ)^T.

The importance of obtaining the innovations is that it removes temporal dependencies and whitens the observed time series x _t into an independent series. To perform this whitening step we need to find a suitable dynamic SSM providing the prediction of the time series by using past observations. Whitening, i.e. removing the temporal dependencies on the past, and finding a suitable causal dynamic model is in fact the same thing. By whitening the causal relations are extracted from the observed time series, and a relevant dynamic model is obtained.

The concepts described so far are supported by a very strong mathematical theorem given by [Levy, 1956]; see also theorem 41 in [Protter, 1990] which states that “for any continuous‐time Markov process x(t) the corresponding innovations can be represented, under mild conditions, as the sum of two white noise processes, namely a Gaussian noise process and a Poisson noise process”. This theorem is a stronger version of the well‐known theorem for Markov diffusion processes [Ito, 1951, 1985; Doob, 1953] according to which “any dynamic process can be represented by a differential equation driven by Gaussian white noise, if the process is Markov and continuous (i.e. without any discontinuous jump)”. The case of additional observation noise has been treated by [Frost and Kailath, 1971]. Consequently we expect that, under the assumption of continuous dynamics, the time series of resulting innovations, for an optimal predictor, will be uncorrelated (in fact, independent) and Gaussian, even if, due to possible nonlinearities in dynamics, the process is non‐Gaussian distributed. This theorem implies that, if we employ a properly chosen model for the dynamics, the prediction errors will be distributed as Gaussian white noise. Then the log‐likelihood function for the time series may be calculated using the standard Gaussian likelihood, even though the original observed time series may have displayed nonlinearities and non‐Gaussian distribution.

The methodology based on Levy's theorem has been employed in time series analysis since the early 1990s [Ozaki, 1992b]. In the neurosciences it has been applied to the identification of dynamic causal models, such as the Zetterberg model for EEG time series [Valdes et al., 1999a, b] and Balloon model for fMRI time series [Riera et al., 2004]. Finally, the log‐likelihood function for the time series may be calculated using the standard Gaussian likelihood, even though the original observed time series may have displayed nonlinearities and non‐Gaussian distribution. Thus the parameter set Θ may be adjusted (using an optimization technique) to yield the maximum likelihood estimators. An evident extension is to augment the likelihood with a penalty function (negative log prior) to carry out Bayesian estimation.

The LL‐innovation approach to SSM estimation is not the only one currently available [Singer, 2008; Jimënez et al., 2006]. Popular alternatives are:

• 1The Expectation Maximization (EM) algorithm developed by [Shumway and Stoffer, 1982]. which alternates between estimating the smoothed estimates and the parameters Θ.
• 2Monte Carlo based methods that select a set of sample points or “particles” and follow their trajectories sequentially [Chen, 2003]. Particle methods can cope with quite general SSM but are currently limited to very small scale models.
• 3A series of methods derived from Variational Bayes techniques which reformulates the SSM by means of a path integral [Kappen, 2008; Wiegerinck and Kappen, 2006; Archambeau et al., 2007a, b]. Use of mean field approximations promise computationally very efficient estimation procedures.
• 4A variant of these Variational techniques is the quite recent Dynamic Expectation Maximization (DEM) [Friston et al., 2007, 2008; Friston, 2008] which reformulates the SSM in terms of generalized coordinates and therefore eschews use of sequential methods such as the Kalman filter.
• 5The Ensemble Kalman Filter (EnKF) [Evensen and van Leeuwen, 2000; Evensen, 2003] a variant of particle methods developed by geophysicists that is suited for very large scale SSM. It is to be noted that the term “ensemble” used here refers to the set of particles to be integrated using nonlinear SDE and not to “ensemble learning” which is a synonym for Variational Bayes methods.

It is to be noted that any combination of these methods may be combined with the LL integration technique to develop new estimation methods. These approaches are currently being explored. Nevertheless the LL‐innovation approach outlined here is a quite feasible alternative. Just as an example, when compared to the EM method, the latter has a higher computational cost (with a backward pass of the Kalman smoother) and yields state estimates that are not always reliable. It is also recognized that the EM algorithm may be quite slow in convergence.

SMALL SCALE EEG/fMRI MODELS

This type of modeling will illustrated with a EEG/fMRI model consisting of a single voxel housing the classical Jansen and Rit neural mass for a single cortical column [Jansen and Rit, 1995; Zetterberg et al., 1978] which is essentially the same as the Zetterberg model. This cortical module will be a component in subsequent more complicated models and is shown diagrammatically in Figure 3B. It only contains 3 interconnected populations of excitatory pyramidal (Pyr) and stellate (St) cells as well as inhibitory interneurons (Inh). Using the notation introduced for EEG/fMRI SSM, x ^ePSP(t) = {x ${}_{\text{1}}^{\text{ePSP}}$(t),x ${}_{\text{3}}^{\text{ePSP}}$(t),x ${}_{\text{5}}^{\text{ePSP}}$(t)} for Pyr, Inh, St cells respectively. The system matrices are:

(13)

Note that the only population with a stochastic input is St, assumed to be the input from the thalamus.

In this model the nPCD is generated by the transmembrane potential of Pyr. This is just the difference between EPSP and IPSP generated by St and Inh, nPCD = z ${}_{\text{1}}^{\text{ePSP}}$(t) − z ${}_{\text{3}}^{\text{ePSP}}$(t). For the sake of simplicity, the lead field is assumed to be K = 1, thus the EEG is equal to the nPCD. Loosely speaking this is as though we were actually measuring the Local Field Potential (LFP). The VFFS has two components u _e = z ₁ + z ₂, the sum of EPSP and u _i = z ₃ the IPSP. As mentioned before, the VFSS feeds into the Metabolic Hemodynamic Model (MHM) [Sotero and Trujillo‐Barreto, 2007] (see Appendix C), a system of ordinary differential equations (ODE) depending on additional state variables x ₇(t),…,x ₁₄(t). The observed BOLD is generated by the Balloon model then is transformed to BOLD by the equation y ₂(t) = g ^BOLD (x ₁₃(t),x ₁₄(t)) in Appendix C. Summarizing, for this cortical module the EEG/fMRI model the complete state vector is x(t) = {x ₁(t),…,x ₁₄(t)}. Usual values for constants in this article are listed in Appendix B. A typical simulation obtained using the LL method for integrating the SDE is shown in Figure 2C. Matlab code is provided in the supplementary material to generate this figure.

Using this type of approach, Neural Mass Modeling, has been producing for decades seemingly correct simulations of EEG rhythms (David and Friston, 2003; Lopes da Silva et al., 1974; Zetterberg et al., 1978) that not only produce EEG like signals but also predict nonlinear bifurcation behavior that can be interpreted, for example, as an explanation of epileptic seizures [Breakspear et al., 2006; Lopes da Silva et al., 2003]. In addition, as already mentioned, [Moran et al., 2007; Rowe et al., 2004] have carried out valuable theoretical analysis of EEG models [Moran et al., 2008; Robinson et al., 2004] by means of global linearization around system steady states and passing to the frequency domain. They also have fitted small scale neural mass models to EEG data by comparison of data versus model spectra.

However, to our knowledge, the first time full nonlinear neural mass models were fit to actual EEG recordings by means of the LL‐innovation method [Valdes et al., 1999a, b] used to estimate the parameters of Zetterberg's model [Zetterberg et al., 1978] for the alpha rhythm.

Fitting the parameters of this neural mass model corresponds to solving the inverse problem from the EEG to the dynamics of the ePSPs in Figure 1A. In our example, the parameters to be estimated are Θ = {c _1,2,c _2,1,c _2,3,c _3,2,μ,σ}. Unsurprisingly, the LL‐innovation approach found parameter values that, when used to produce simulated produce recordings, provided traces very similar to the original data. A new aspect was the calculation of the likelihood of the fit useful for model comparison. Even more, examination of the Gaussianity and independence of the innovations allowed the identification of several alpha rhythm recordings that could not be explained by this model‐thus providing a criterion for falsification of the theory.

What did come as a surprise was a predictive aspect of this model. The Zetterberg model exhibits a stochastic Hopf bifurcation, in which changes in parameter values transform the topology of the stable manifold in state space from a point attractor to a limit cycle. The parameters estimated for each EEG recording corresponded either to a point attractor or to a limit cycle. This model driven fingerprint for dynamic behavior was then independently confirmed with a data driven procedure. This involved fitting non parametric nonlinear time series models and ascertaining if it's “skeleton” (dynamics when the dynamic noise is turned off) was of the appropriate type. The identification of a bifurcation in a data set through model fitting is a topic of major interest. Of course, there are many types of bifurcations, each of which bear their dynamical signature in data and not all of which are displayed by a given model [Izhikevich, 2006]. Hence a complementary approach to model fitting is the visual analysis of the bifurcation structure of a given model and comparison to appropriate data as for example in [Breakspear et al., 2006]. A relevent review of combining model and data driven exploration of nonlinear dynamics can be found in [Valdés et al., 1999].

A number of biophysical models have been proposed for BOLD signals [Stephan et al., 2004]. The global approximation around a steady state and analysis in the frequency domain has also been applied to this type of model [Robinson et al., 2006].

Regarding fully nonlinear models, [Sotero and Trujillo‐Barreto, 2007] have developed a biophysical model of the coupling between neuronal activity and the BOLD signal (metabolic/hemodynamic model, MHM) that allowed explicit evaluation of the role of both excitatory and inhibitory activity. In addition to glycolysis, the “glycogen shunt” is assumed in the astrocytes. They also assume that cerebral blood flow is not directly controlled by energy usage, but it is only related to excitatory activity. Appendix C summarizes this model. By means of simulations with the LL approach, these authors successfully predicted the appearance of negative BOLD phenomena. In a subsequent article [Sotero et al., 2008], they were able to use the LL‐innovation method to fit a stochastic version of this model to BOLD signals obtained in a motor task. Using the Bayesian Information Criterion (BIC) allowed selection among several competing models. In particular it was found that observations seemed to be generated by only an excitatory population rather.

With regard to EEG/fMRI fusion, [Riera et al., 2006, 2007] were the first to use the full LL‐innovation approach for estimating a local electro vascular coupling model. LL simulations explained the continuous dynamics of electrical and vascular states within a cortical unit. They explicitly dealt with the mismatch in temporal resolution of EEG and fMRI by incorporating this information into the model. The innovation approach was then used to estimate state variables and system parameters from the EEG and BOLD signals in selected regions of interest. They applied this algorithm to recordings obtained from two subjects while passively observing a radial checkerboard with a white/black pattern reversal. The EEG and fMRI data from the first subject was used to estimate the electrical/vascular states and parameters of the model in V1.

These examples of small scale models illustrate the feasibility of creating neural mass based methods for EEG/fMRI fusion. Unfortunately it is only at this small scale that the LL‐innovation estimation has been applied. At larger scales only simulations have been possible as yet.

MEDIUM SCALE EEG/fMRI SIMULATIONS

Small scale simulations must be considered only as a proof of concept for EEG/fMRI fusion. More realistic situations must involve at least thousands of neural masses. Several issues become critical then. One problem is that of specifying the connectivities between distant neural populations since in the cortical module model the matrix C only contains local intra module connections (which we will denote as contained in the matrix C ₀). [Babajani and Soltanian‐Zadeh, 2006] postulated exponential decay with distance of short range (SR) connectivity strength of both excitatory and inhibitory populations to neighboring voxels which can be included in the connectivity matrix C _SR. Thus in their case they use the connectivity matrix C = C ₀ + C _SR. With these specifications, these authors successfully simulated EEG and the associated BOLD signals using decay parameters described in the literature. A distinctive feature of their model is the inclusion of a large number of cortical modules per voxel which addresses the issue of generator synchronization discussed on the EEG and BOLD. However, the integration methods used were standard creating the doubt if the dynamical behavior observed corresponded to the original continuous time system.

Another example of a medium sized simulation can be found in [Sotero et al., 2007]. Here the generation of EEG rhythms was studied using a model with several dozen cortical regions comprising thousands of neural masses. These regions were coupled with connectivity coefficients obtained from a neuroimaging database. This simulation included a number of new features:

• Modification of the cortical module model to include Pyr to Pyr connections and 16 SDE3.

• Inclusion of the thalamus as a single module as a set of 12 SDE with local connectivities which was also coupled with the cortical modules. The thalamic module consisted of an excitatory thalamic relay population receiving visual dynamic noise input. The relay population projects to cortex as well as to the inhibitory reticular thalamic population which feed back to the relay population.

• Use of the connectivity matrix C = C ₀ + C _SR + C _LR which includes in C _LR the coupling strength of inter‐area or Long Range (LR) connections between large cortical areas.

• Additionally, the model was modified to take into consideration conduction delays collected in the matrix T. These were used to modify the PSP functions have variable longer time courses [Jansen and Rit, 1995; Sotero et al., 2007].

• A realistic lead field K was used to generate the EEG

The matrices C _LR and T were defined according to their anatomical connections probability (ACP) matrix [Iturria‐Medina et al., 2007]. The ACP gives the probability that any two areas are connected at least by a single nervous fiber according to diffusion weighted Magnetic Resonance Imaging (DWMRI) techniques and Graph Theory. This matrix is estimated as follows:

• 1The cerebral volume was represented as a non‐directed weighted graph G _brain,0 = [N ₀,A ₀,W ₀], where N ₀ is the set of voxels (nodes) having a non‐zero probability of belonging to some cerebral tissue, A ₀ is the set of white matter links (arcs) between contiguous voxels in N ₀, and W ₀ is a set of real numbers representing arcs weights. The weight of an arc is chosen so that it represents the probability that contiguous linked nodes are really connected by nervous fibers.
• 2An iterative algorithm was employed for finding the most probable trajectory between any two nodes, which is assumed to be the hypothetical nervous fiber pathway running between these points. A node‐node anatomical connectivity measure (ranging between 0, not connected, and 1, perfectly connected) is defined as the lowest weight of the arcs set belonging to the most probable path.
• 3After computing the node‐node connectivity measure between any two nodes of the brain surface, a threshold value of 0.6 was applied to retain values higher than 0.6, which were assigned to the matrix C _LR (element c_ij represents the connectivity between nodes i and j).
• 4The delay matrix T = {τ_i,j} i,j = 1,…,N _m was estimated as the average path length between voxels (nodes), multiplied by a conduction velocity of 10 m/s. Both matrices were then averaged over neighboring voxels as to obtain the corresponding matrices for the reduced set of vertices actually used in the simulation.

The previously described model results in quite a large system of SDE. In LL schemes, the limiting step for their solution is calculating matrix exponentials. Fortunately this can now be done efficiently for sparse systems using Krylov subspace methods [Jimenez, 2002; Jimenez and Carbonell, 2005], a feature taken advantage of with the empirical sparseness of the estimated anatomical connectivity matrix as described by [Iturria‐Medina et al., 2007].

Simulations of different types of human brain rhythms were carried out in order to test the model. Changing parameter values allowed the simulation of different EEG rhythms in a similar fashion as described in [David et al., 2006a, b; David and Friston, 2003]. The simulated alpha rhythm was reactive to increase of thalamic input (visual stimulation). However alpha activity was only obtained by using in simulations the connectivity patterns estimated from DWMRI. Alpha disappeared when the elements of C _LR were randomly reshuffled. This shows that the connectivity pattern assumed for EEG/fMRI simulations can be quite critical. [Sotero and Trujillo‐Barreto, 2008] then went on to augment this EEG model with their MHM model (Appendix C) for BOLD generation. The VFFS (the input to the MHM) was the number of active synapses within the voxel. The simulated EEG and BOLD signals where then subjected to the data driven EEG/BOLD correlation method described in Figure 1B. Strikingly, the observed pattern of positive BOLD‐EEG correlations in thalamus and negative BOLD‐EEG correlations in the occipital cortex present in real data (Fig. 2A) was also obtained for the simulated data (Fig. 2C).

LARGE SCALE EEG/fMRI SIMULATIONS

The previously described medium scale simulations have at most modeled a total of 1032 neural masses– the basic limitation for larger simulations is just computational. We now report a new type of integration scheme, the LL‐algebraic method, which is particularly adapted to the analysis of very large networks of neural masses. The motivation for this new type of LL scheme comes from techniques used for the analysis of electrical circuits [Schuster and Unbehauen, 2006] and other modular systems, in which the nonlinear components are described by ODE or SDE and their interconnections through algebraic equations. These types of systems which combine differential equations and algebraic equations are known as Differential Algebraic equations (DAE) and Stochastic Differential Algebraic equations (SDAE) [Denk and Winkler, 2007] and can be solved very efficiently. For example, the technique we shall use involves solving iteratively and separately the algebraic equations and the SDE [Vijalapura et al., 2005]. In fact it can easily be seen from that our neural mass models are already in a differential algebraic formulation in which Equation (3), the formula that specifies neural mass dynamics, is the differential part, and (4), the part that specifies connectivities, is the algebraic part.

The LL‐algebraic integration method alternates between two steps:

• 1Assuming that the inputs z _i(t) to each neural mass have been calculated, Eq. (3) is now a second order random differential equation (RDE). RDEs are functions of stochastic process that do not enter the equation linearly and are not necessarily Gaussian white noise (as is the case for SDE). RDEs do not pose any special problem since methods for their LL integration have been recently developed [Carbonell et al., 2005].
• 2Given the outputs from all neural masses, the inputs z(t) may be calculated using x(t) using expression (4) z(t) = Cx(t).

Two factors may be used to speed up the simulations by orders of magnitude. One is that the numerical integration in step 1 may be carried out analytically (IV) for example using the program Mathematica 6.0 (Wolfram Research Inc.). A program for this derivation is provided in the supplementary material. Denoting the states of the i‐th neural mass states by the integration step found analytically is:

(14)

The second speed up factor is that, as discussed in the previous section, brain connectivity matrices are very sparse– speeding up step 2. An additional bonus of this method for integration is that the second step can now be modified to include explicitly conduction delays

(15)

This contrasts with the suboptimal approach of slowing down the PSP functions as has been done in [Jansen and Rit, 1995; Sotero et al., 2007].

A similar approach can be used to optimize computations for the MHM model. Careful examination of Appendix C reveals that it can be decoupled into pairs of RDE that are linked by algebraic identities. Thus these may be solved using the approach outlined above of interleaving differential and algebraic steps. The differential integration steps have also been solved analytically. Finally we mention that these integration steps for both neural masses and the MHM are all in a format that allows the use of vectorized MATLAB operations making simulation of hundreds of thousands of neural masses possible in approximately real time on a state‐of‐the‐art desktop PC. A demonstration program in MATLAB is available in the supplementary material.

In order to show the feasibility of this approach for large scale simulations, the T2 image of a normal subject was transformed to MNI space and segmented into three different brain tissues (cerebral spinal fluid, gray matter and white matter). Each individual gray matter voxel was labeled based on an anatomical atlas (constructed by manual segmentation) using the transformation matrix obtained in the previous step. All segmentation procedures were implemented by using SPM5 (Statistical Parametric Mapping, FIL, UCL) and the IBASPM (Individual Brain Atlases using SPM) toolbox (Cuban Neuroscience Center, http://www.fil.ion.ucl.ac.uk/spm/ext/#IBASPM). Then, the surfaces for the thalamus, gray and white matter (for both hemispheres) were extracted using the marching cubes algorithm as well as, the previously computed individual atlas. This yielded a set of 65,728 vertices for further analysis. This was down‐sampled to 16138 vertices for use in the simulation.

A Jansen‐Rit cortical module was placed at each of the cortical vertices (8203 for the left hemisphere, 8362 for the right hemisphere). A thalamic module was placed at each of the vertices on the thalamic surfaces (438 for the left and 465 for the right). This yielded a total of 106,614 RDEs for the neural masses to which 99,390 additional ones are added to include the MHM for each voxel. The connectivity and delay matrices were calculated as described in the section on medium scale simulations. However instead of calculating these parameters only for large cortical areas, the long range connection matrices contain all 16138 by 16138 connections and delays. Figure 5A illustrates the connections of the visual cortex. These matrices were also found empirically to be quite sparse (Fig. 5B).

Figure 5.

Connectivity matrix used for large scale simulations. Sample estimated nerve fiber pathways between voxels of the left and right occipital poles (left panel) of cortical tessellation shown in Figure. From these pathways anatomical connectivity values between these voxels were estimated. For use in the large scale neural mass simulations shown in Figure 2a, in reality all the voxels on both cortical and thalamic surfaces were used to calculate the overall connectivity matrix. This connectivity matrix (dimensions 16138 × 16138) is summarized for purposes of illustration here to a region to regional representation corresponding to 90 well‐known anatomical areas (right panel). The same technique also estimates the length of the fiber connections that is used to infer conduction delays between voxels. Note the extreme sparseness of connections.

Simulations were carried out based on the parameter set described in Appendix B. Resting state EEG epochs of 15.3 s duration were simulated with a Gaussian white input to the relay cells of the posterior right thalamus with mean 20 and variance 2 pulses/second. Changing levels of thalamic stimulation were achieved by increasing the input to the thalamus to a mean of 100 pulses/second during 2 s. A continuous Morlet wavelet transform was used to estimate the power in the alpha band of the Local Field Potential at each of the cortical and thalamic modules. A reference signal was calculated for each module by convolving the power at 9 Hz with a standard hemodynamic response function. The correlations between the reference signal and the simulated BOLD signal are shown in Figure 2D (left side view). It is to be noted that this simulation produces the same empirical pattern shown in Figure 2A–C.

DISCUSSION AND CONCLUSIONS

This article presents a general framework for EEG/fMRI fusion on which both data driven and model driven methods are based. For data driven methods it provides a conceptual framework for improvement. For example the framework suggests that instead of the usual approach of directly correlating hrf convolved EEG power changes with BOLD [Goldman et al., 2002; Martinez‐Montes et al., 2004], an alternative approach might be to solve the spatial inverse problem for the EEG and the temporal inverse problem for BOLD and directly compare estimates of nPCD and VFSS (Appendix A and Fig. 1C). However independently of the level of comparison, a consistent finding for all data driven analyses of EEG power in the alpha band has been that of positive correlations between EEG alpha power and BOLD in both frontal cortices and thalamus; and negative ones for the occipital region. These findings provide empirical constraints that should be a test of the predictive powers of more theoretical approaches to EEG/fMRI fusion.

We argue that the appropriate level of description for model driven fusion is the mesoscopic level that can be implemented using neural mass modeling. In this article a general model Neural Mass‐Metabolic Hemodynamic Model is set out as a State Space Model (aka DCM) that couples a neural mass EEG/fMRI model with a metabolic hemodynamic model. This model generalizes many others that have been presented before and is based upon a particularly simple form. This form consists of stating each neural mass as a distinct module and separating out the synaptic connections and conduction delays between populations as algebraic constraints, thus allowing the use of efficient numerical techniques developed for differential‐algebraic equations.

Of primary importance is to implement computer simulations that preserve the dynamical properties of the original continuous time system. We show that the Local Linearization (LL) method for integrating ordinary, stochastic, and random differential equations is appropriate for use in neural mass modeling. LL simulations of small and medium sized network were able to produce many types of EEG rhythms and were consistent with the appearance of negative BOLD signals. Additionally they reproduced the EEG/BOLD correlations found by data driven fusion methods. These simulations clearly show that the emergence of EEG rhythms depends critically on the use of realistic anatomical connectivity information estimated from DWMRI. A new LL‐algebraic method allows simulations with hundreds of thousands of neural populations and full voxel to voxel connectivities and conduction delays.

The neural mass models used in this article are derivations of the Jansen model [Jansen and Rit, 1995] and correspond to mean field approximations of integrate and fire neurons. The present framework can easily accommodate more realistic neural mass model that go beyond such simplifications by considering the second moments of neural masses [Marreiros et al., 2008] or mean field approximations of neurons with intrinsic properties [Robinson et al., 2008].

The LL integration method can be combined with many other procedures as an alternative to the Extended Kalman Filtering, the usual staple for engineering applications. In particular the LL‐innovation method for estimation consists of the discretization of the continuous time system in order to permit Kalman filtering estimate of the states as well as the data innovations or prediction errors [Galka et al., 2004; Kalman, 1960]. Under very general conditions these innovation errors will be Gaussian so the likelihood or Bayesian criteria may be easily calculated.

The LL‐innovation method has already been applied to the estimation of small scale EEG network which revealed hidden dynamical characteristics of alpha rhythm recordings that distinguish point attractor versus limit cycle behavior. This classification was then independently verified with non parametric data driven modeling. Estimation for BOLD signals allowed model selection and distinguished the contribution of inhibitory and excitatory PSPs to fMRI. Finally, combined EEG/fMRI models were able to carry out joint estimation of physiological parameters. However to avoid trivial model fitting it is important to clearly identify predictions of the EEG/fMRI model that would allow its falsification. These might be susceptible to verification via behavioral, TMS, or pharmacological modification of brain states.

EEG/fMRI should soon become a more effective tool for the explanation of inter individual differences in resting state or event related EEG. For example the increasing availability of extensive EEG/fMRI/DWMRI data sets may decide between contrasting views of the origin of brain oscillations. Such a comparison begs to be carried out between local [Lopes da Silva et al., 1974] and global [Nunez et al., 2001] models of the EEG. A perhaps more complex issue is that of EEG/fMRI fusion becoming a bridge to understand cognitive functions. For this, a junction must be made between our type of modeling and that of neural information processing systems.

Until now the LL‐innovation for estimation approach has been limited to small scale models. Fortunately there are promising developments in large scale state and parameter estimation problems, the Ensemble Kalman Filter [Evensen and van Leeuwen, 2000] and Dynamic Expectation Maximization [Friston et al., 2008] being two recent examples. Additionally, special methods are being developed for the estimation of differential algebraic systems [Becerra et al., 2001; Gerdin et al., 2007; Jorgensen et al., 2007]. It is to be expected that the combination of these techniques with the new large scale integration techniques discussed in this article will allow the estimation of realistic models with appropriate resolution in the near future. One of the most complicating factors is that even with prior DWMRI constraints the number of connectivities to be estimated may become very large. In this case the use of Bayesian estimation methods geared to finding sparse models might be necessary [Valdes‐Sosa et al., 2006].

Finally it should be emphasized that this article has limited itself to the analysis of EEG and fMRI data recorded concurrently. Many useful experiments are carried out with non concurrent data so that modifications of our model driven framework for this situation are worthwhile. A promising approach is to apply Bayesian data augmentation methods for datasets with at least partial overlap (for example to combine MEG/EEG and EEG/fMRI).

Supporting information

Additional Supporting Information may be found in the online version of this article.

Mathematica functions: There are 4 Mathematica notebooks that find the anlaytic expression for a Local Linearization (LL) step of integration of the corresponding deterministic or stochastic differntial equations. The notebooks are: cbf.nb cbvdoxy.nb glucose.nb neuralmass.nb For those that don?t have Mathematica we also append the corresponding Mathematica session in pdf form. The analytical equations are automatically translated to matlab and are the basis of the matlab demos. For more information view readme.txt included in .zip file.

APPENDIX A. TEMPORAL DECONVOLUTION OF THE HEMODYNAMIC RESPONSE FUNCTION

For a given voxel the following estimator for the VFFS was used:

(A1)

where v denotes the vector of VFFSs for time instants t = 1, …, N _T, b the BOLD waveform, H the matrix form of the hemodynamic response, L the one dimensional Laplacian operator toeplitz ([−1 2–1,] T), and λ a regularizing parameter. Note that this is a temporal spline inverse solution that generalizes with a smoothness constraint the deconvolution method of Glover et al. [1999].

APPENDIX B. EXTENDED NEURAL MASS MODEL. VALUES OF PARAMETERS AND THEIR PHYSIOLOGICAL INTERPRETATION

 

Table  .

Parameters with the same value in all simulations
Parameter	Physiological interpretation	Value
A	Average synaptic gain for Excitatory, Inhibitory neurons	3.25 mV, 22 mV
e 0, v 0, r	Parameters of the nonlinear sigmoid function	5 s−1, 6 mV, 0.56 mV−1
a,b,a t, b t	Average synaptic time constants for Cortex and thalamic Excitatory and Inhibitory populations.	100 (s−1), 50, 100, 40
c 1, c 3	Synaptic contacts made by pyramidal cells on excitatory and inhibitory interneurons within a cortical module	150, 40
c 2, c 4	Synaptic contacts made by excitatory and inhibitory interneurons on pyramidal cells within a cortical module	120, 40
c 5	Synaptic contacts between pyramidal cells within a cortical module	150
c 6, c 7	Synaptic contacts made by pyramidal cells on pyramidal cells of different cortical module corresponding to short and long range connections.	50,10
c 7t	Synaptic contacts made by pyramidal cells on thalamic TC cell	100
c 1t, c 3t	Synaptic contacts made by thalamic TC cells on RE and on stellate cells.	50, 80
c 2t	Synaptic contacts made by RE neurons on TC cells	50

APPENDIX C. METABOLIC/HEMODYNAMIC MODEL

In Sotero and Trujillo‐Barreto (2007) the role that inhibitory and excitatory activities play in the generation of the BOLD signal was studied with the help of a biophysical model. All the variables are normalized to baseline values. According to the model, changes in excitatory (u _e(t)) and inhibitory (u _i(t)) neuronal activities are linked to the corresponding changes in glucose consumption, x ₇(t) and x ₉(t) respectively, by means of linear differential equations:

Then, the total glucose consumption is calculated as a weighted average of the excitatory and inhibitory contributions. The glucose variables were then directly related to the metabolic rates of oxygen for excitatory (m _e(t)) and inhibitory (m _i(t)) activities, as well as to the total oxygen consumption (m(t)):

For describing CBF (x ₁₁(t)) dynamics, the MHM uses the model introduced in Friston et al. (2000) but with the additional assumption that CBF is not coupled to inhibitory activity:

As in earlier approaches, in the MHM (Sotero and Trujillo‐Barreto, 2007) the Balloon model (Buxton et al, 2004) is employed for linking the output of the metabolic and vascular models to normalized cerebral blood volume (x ₁₃(t)) and deoxyhemoglobin content (x ₁₄(t)).

Knowing x ₁₃(t) and x ₁₄(t), the BOLD signal is calculated as in Buxton et al. (2004):

The interpretation and values of MHM parameters are displayed in the following table: 

Table  .

Parameter	Interpretation	Value
a e	Efficacy of glucose consumption response to excitation	1
a i	Efficacy of glucose consumption response to inhibition	1
τe	Time‐constant of the excitatory glucose consumption impulse response.	1
τi	Time‐constant of the inhibitory glucose consumption impulse response	0.8
c	Steepness of the sigmoid function x	2.5
d	Position of the threshold of the sigmoid function x	1.6
δe	Delay between excitatory neuronal activity and corresponding glucose consumption.	0.1 s
δi	Delay between inhibitory neuronal activity and corresponding glucose consumption.	0.1 s
δf	Delay between excitatory neuronal activity and CBF response	0.2 s
γ	Baseline ratio of excitatory to inhibitory synaptic activity in the voxel	5
ε	Efficacy of blood flow response to excitation	0.6
τs	Constant for CBF signal decay	1.5 (Friston et al., 2000)
τf	Constant for CBF autoregulatory feedback	2.4 (Friston et al., 2000)
τ0	Transit time through the balloon	1 s (Buxton et al., 2004)
α	Coefficient of the steady state flow‐volume relationship	0.4 (Buxton et al., 2004)
a1	Weight for deoxyhemoglobin change	3.4 (Buxton et al., 2004)
a2	Weight for blood volume change	1 (Buxton et al., 2004)
${}_{\mathit{\text{V}}{}_{\text{0}}}$	Baseline blood volume	0.02 (Friston et al., 2000)

APPENDIX D. EXPLICIT EQUATIONS FOR LL INTEGRATION OF A NEURAL MASS

As described in (Carbonell et al., 2005) the local linearization of a Random Differential (RDE) is as follows. Let's assume that a k‐dimensional random process q(t), t∈ [t ₀,T] and a nonlinear function f are given and define the d‐dimensional RDE

Then, given a step size Δt, the local linearization scheme that solves numerically the equation above at the time instants t _n = t ₀ + nΔt, n = 0,1,…. is given by:

where L = [I _d, 0_dx2], r = [0_1x(d+1),1]^T and the (d + 2) × (d + 2) matrix C _n is defined by blocks as

(A2)

Now consider the neural mass:

(A3)

The required Jacobian matrices are

(A4)

Substituting expressions (A3) and (A4) into the formula (A2) produces an expression which can be solved symbolically making it possible to obtain expression (14) in the main text. A Mathematca 6.0 (Wolfram Research Inc.) program to do this is included in the supplementary material. A similar analysis is valid for the MHM equations (Appendix C). Note that these equations for the MHM are organized in pairs with algebraic constraints linking them. Thus it is possible to use the same strategy as just outlined for the neural mass.