Multisensory integration processing during olfactory‐visual stimulation—An fMRI graph theoretical network analysis

Abstract

In this study, we aimed to understand how whole‐brain neural networks compute sensory information integration based on the olfactory and visual system. Task‐related functional magnetic resonance imaging (fMRI) data was obtained during unimodal and bimodal sensory stimulation. Based on the identification of multisensory integration processing (MIP) specific hub‐like network nodes analyzed with network‐based statistics using region‐of‐interest based connectivity matrices, we conclude the following brain areas to be important for processing the presented bimodal sensory information: right precuneus connected contralaterally to the supramarginal gyrus for memory‐related imagery and phonology retrieval, and the left middle occipital gyrus connected ipsilaterally to the inferior frontal gyrus via the inferior fronto‐occipital fasciculus including functional aspects of working memory. Applied graph theory for quantification of the resulting complex network topologies indicates a significantly increased global efficiency and clustering coefficient in networks including aspects of MIP reflecting a simultaneous better integration and segregation. Graph theoretical analysis of positive and negative network correlations allowing for inferences about excitatory and inhibitory network architectures revealed—not significant, but very consistent—that MIP‐specific neural networks are dominated by inhibitory relationships between brain regions involved in stimulus processing.

1. INTRODUCTION

Human evaluation and perception of the environment is based on integrated information from different sensory modalities. Rather than evaluating different sensory information individually, the human brain integrates and fuses simultaneously occurring inputs to deliver a holistic perception based on all our senses. This computational approach to process a multitude of incoming sensory information is described as multisensory integration processing (MIP) (Stein & Stanford, 2008). While the cellular mechanisms of sensory integration have been partly understood to follow the so called principle of inverse effectiveness, firstly described by Stein and Meredith, (1993), the macroscale (whole‐brain level) computational mechanisms are not understood yet. So far, several individual brain areas for MIP have been identified including the superior temporal sulcus (STS), the intraparietal sulcus (IPS), and the prefrontal cortex (PC) (Beauchamp, Argall, Bodurka, Duyn, & Martin, 2004; Ghazanfar & Schroeder, 2006; Senkowski, Schneider, Foxe, & Engel, 2008). Nevertheless, the identification of brain regions, which seem to be involved in MIP, does not necessarily contribute to the question how the brain as one complex organ computes simultaneously occurring sensory information. Studying whole‐brain neural networks processing a specific stimulus aims to unravel the time‐locked underlying architecture of the brain`s computational processes. Graph theoretical measurements deliver hereby the necessary tool to study how the topology of networks shape and modulate brain function (Sporns, 2003; van den Heuvel & Spornss, 2013). Graph theoretical studies of functional network organization have only recently been extended to task‐associated changes in network architectures (Bassett et al., 2011; Bolt, Laurienti, Lyday, Morgan, & Dagenbach, 2016; Cole et al., 2013; Stanley, Dagenbach, Lyday, Burdette, & Laurienti, 2014; Stevens, Tappon, Garg, & Fair, 2012). Previous research on neural networks analyzed with graph theoretical parameters revealed that most structures participating in MIP exhibit so‐called hub‐like characteristics, defined by a central network position receiving and transmitting information (Honey, Kotter, Breakspear, & Sporns, 2007; Zamora‐Lopez, Zhou, & Kurths, 2010). Also, it has been suggested that multisensory integration is flexible and context‐dependent, which is in concordance with the above described involvement of cortical regions within heteromodal association cortices (Stevenson & Wallacee, 2013; van Atteveldt, Murray, Thut, & Schroeder, 2014). Therefore, the variability of multisensory integration through context‐specific neuronal modulation suggests the need to first understand the basic context‐independent MIP and then pursue from there. So far, only few studies aimed to explicitly investigate the integrative relationship of structures processing bimodal information of the olfactory and visual system with regards to MIP, but lack the investigation on the whole‐brain macroscale using network scientific tools (de Araujo, Rolls, Velazco, Margot, & Cayeux, 2005; Gottfried & Dolann, 2003). Moussa et al (2011) investigated connectivity changes on a whole‐brain level during resting state, visual and/or auditory stimulation (MIP underlying visual and auditory stimulus combinations). This study differed compared to our attempt in that sense, that auditory as well as visual masking procedures were conducted to explicitly study connectivity changes elucidated by these early sensory processing areas and therefore do not include areas associated with higher cognitive functions.

The methodological procedure analyzing these integrational processes during olfactory and visual stimulation on a whole‐brain level allows discussing and evaluating the results on the structural as well as functional macroscale, which is of outermost importance for such highly complex mechanisms as MIP. In detail, studying MIP‐specific network nodes does require the considerations that hubs are multi‐connected brain regions and that they are highly important for integrational processes and therefore show strong involvement in various higher cognitive functions (van den Heuvel & Spornss, 2013). Making use of this rather novel network science development, we applied a graph‐theoretical analysis on stimulus‐induced changes in network organization to examine MIP‐specific network characteristics during bimodal stimulation of the olfactory and visual system.

To identify MIP‐specific network hubs, we hypothesized a significantly increased correlation between the nodes in networks including aspects of sensory integrative mechanisms in contrast to those lacking MIP. To describe the topology of the resulting networks, we assessed clustering coefficient as a measurement of segregation and global efficiency as a measurement of integration. Modularity (Newmans’ Q; Newman, 2006) was computed to assess the networks tendency to subdivide into so called modules, which are confined groups of highly interconnected nodes. Degree, describing the “connectedness” of a given network, density and strength were also computed to statistically compare and describe networks including aspects of MIP and those without integrative mechanisms. The underlying hypothesis was that we would find a significantly higher degree, as well as density within networks integrating sensory information. This reasoning was justified by the requirement of increasing the “interconnectedness” of the network due to MIP within integrational brain regions. Last, rather based on a suggestion than on a hypothesis; to approach the underlying microscale chemoarchitecture via the whole‐brain macroscale architecture, we studied networks with and without aspects of MIP with graph metrics for only positive network correlations and only negative network correlations.

2. MATERIALS AND METHODS

2.1. Subjects

Thirty‐four healthy subjects aged between 18 and 45 years were screened for exclusion and inclusion criteria and 18 participants (7 men) with an average age of 27.4 years (SD: 4.0 years; age range 21–36 years) were included in the study. Exclusion criteria were smoking, drug abuse, having breathing problems such as asthma or problems caused by allergies, regular medication intake, using nose drops at the time of testing, exhibiting thyroid dysfunctions, or showing any neurological or psychiatric disorders. Participants were also excluded if their evaluation regarding pleasant and unpleasant odors or congruent and incongruent odor–picture combinations differed from the precategorized pleasant and unpleasant odors or congruent and incongruent odor‐picture combinations. Additionally, the olfactory screening using the olfactory identification test MONEX‐40 (Freiherr et al., 2012) had to reveal normal olfactory functioning (scoring values above 26). Prior to the first visit, participants were instructed not to drink alcohol the day before testing, not to use extensively smelling shampoo or other hygiene and beauty products, only to drink water one hour before testing, not to chew gum during testing and to wear contact lenses in case of strongly impaired vision (vision correction was applied where needed for the subsequently used video goggles during the experiment). The study was approved by the ethics committee of the hospital of RWTH Aachen University (EK 017/11) and all participants provided written informed consent prior to participation.

2.2. Experimental paradigm

While lying in the MRI scanner, participants were exposed to three different odors and the three corresponding pictures of the categories unpleasant, pleasant, and neutral. The unpleasant setup included a picture and the smell of dirty socks, the pleasant setup consisted of a picture and the smell of roses and the neutral setup showing a white picture and exposing the participant to an odorless airflow served as a baseline (BL). For an illustration of the visual stimuli, see Figure 1.

Figure 1.

Visual Stimuli. (a) A rose representing the pleasant visual stimulus. (b) Dirty socks representing the unpleasant visual stimulus. (c) A white picture representing a neutral visual stimulus [Color figure can be viewed at http://wileyonlinelibrary.com]

In addition to the BL, the paradigm included eight conditions consisting of different congruent and incongruent odor–picture combinations of pleasant and unpleasant stimuli for bimodal stimulation and pleasant or unpleasant stimulus content for unimodal stimulation. This resulted in four bimodal and four unimodal stimuli and the BL as illustrated in Table 1: (a) olfactory pleasant and visual pleasant (OPVP); (b) olfactory pleasant and visual unpleasant (OPVU); (c) olfactory unpleasant and visual unpleasant (OUVU); (d) olfactory unpleasant and visual pleasant (OUVP); (e) olfactory pleasant (OP); (f) olfactory unpleasant (OU); (g) visual pleasant (VP); (h) visual unpleasant (VU); and (i) white picture and odorless air—baseline (BL).

Table 1.

Experimental conditions

		Olfactory stimulus
		Rose	Socks	Neutral
Visual stimulus	Rose	OPVP	OUVP	VP
	Socks	OPVU	OUVU	VU
	Neutral	OP	OU	BL

The stimulation paradigm is depicted in Figure 2: A black fixation cross was presented for 3 s, after which it turned green for the duration of 1 s indicating a soon to be expected stimulus. Olfactory and visual stimuli were presented for 1.5 s, followed by a black fixation cross in the center of the screen, indicating the end of the trial. To ensure a synchronous perceptual onset with the visual stimulus, the odors were elicited 100 ms before visual stimulus; the estimated odor onset stimulus delay. Timely precision of stimulus presentation was achieved using software package Eprime. An evaluation slide in which the participant was asked to evaluate the pleasantness of the odor was presented last, but this evaluation itself is not relevant to this study, but the additional time interval is important to avoid adaptation of the nose. Different colors of the fixation cross indicated the following information to prepare the subject for the upcoming event: (a) black fixation cross: fixate eyes on the fixation cross; (b) green fixation cross: attention, stimulus to be expected; (c) blue fixation cross: breathe in through the nose (an odor is coming); (d) yellow fixation cross: breathe in through the nose, (no odor is presented). Using both the yellow and the blue fixation cross as an indicator to breathe in prevents stimulus‐unrelated activity within the olfactory cortex, potentially caused by attentional modulation (Sobel et al., 1998; Zelano et al., 2005). Each condition was presented eight times per scanning session, resulting in 16 trials per condition, due to two consecutive scanning sessions.

Figure 2.

Experimental paradigm. A black fixation cross turned green after 3 s indicating that a stimulus can be expected. Visual and/or odor stimuli were presented for 1.5 s, followed by a black crosshair and an evaluation slide [Color figure can be viewed at http://wileyonlinelibrary.com]

Visual stimuli were presented using video goggles (Video‐Audio‐ System, Resonance Technology Company, Inc, Los Angeles, CA). Odor application inside the scanner was enabled using a constant‐flow olfactometer (OM6b, Burghart Medizintechnik, Wedel, Deutschland, Kobal, 1985; Kobal & Hummel, 1988). Two odors, a rose odor for pleasant and a dirty socks odor for unpleasant, were used consisting of the following chemical components: rose: 40% solution of phenylethyl alcohol (Th. Geyer GmbH & Co.KG, CAS‐No.: 60‐12‐8) diluted in propylene glycol (Fluka, CAS‐No.: 57‐55‐6) and dirty‐socks: 0.1 M solution of isovaleric acid (99%) (Sigma‐Aldrich GmbH; CAS‐No.: 503‐74‐2) in propylene glycol (Fluka, CAS‐No.: 57‐55‐6). Both odors are expected to provoke a purely olfactory stimulation and to not exhibit any trigeminal effects.

2.3. fMRI data acquisition

Brain imaging data were obtained using a 3‐T MRI scanner (Philips; Achieva, Philips Medical Systems) in the Hospital of RWTH Aachen University with an eight‐channel head coil. BOLD images were acquired using T2*‐weighted echo‐planar imaging (Lepisto et al., 1996) covering the whole brain in 46 slices (slice thickness = 2.5 mm, no gaps, echo time (TE) = 22 ms, repetition time (TR) = 2.5 s, flip angle = 70°, field of view 240 × 240 mm, matrix size = 96 × 96, pixel size = 2.5 mm × 2.5 mm, interleaved slice acquisition). The slice package was positioned parallel to the AC/PC line and then adjusted in a 20° angle (half‐coronal positioning) on the sagittal localizer image (Deichmann, Gottfried, Hutton, & Turner, 2003). The anatomical scans were acquired using a high‐resolution three‐dimensional T1‐weighted gradient sequence (inversion time (TI) = 1.048 ms, TE = 4.6 ms, TR = 9.9 ms, field of view 256 × 256 mm, pixel size = 1 mm × 1 mm, flip angle 8°, 145 slices, slice thickness = 1.0 mm).

2.4. fMRI data preprocessing

Functional image preprocessing was conducted using the Statistical Parametric Mapping software (SPM12, Welcome Trust Centre for Neuroimaging, London, UK, http://www.fil.ion.ucl.ac.uk/spm) (Penny, Friston, Ashburner, Kiebel, & Nichols, 2011), implemented in Matlab R2014a (The MathWorks, Inc., Natick, Massachusetts, USA). All images were converted from PAR/REC‐format to NIfTI‐format with r2agui toolbox (http://r2agui.sourceforge.net/). Each subjects' anatomical images were oriented along the anterior commissure (AC)–posterior commissure (PC) line before the functional images were corrected for slice timing. After realignment of the EPI volumes to correct for head movements (Andersson, Hutton, Ashburner, Turner, & Friston, 2001), the structural image of each participant was co‐registered to the mean realigned EPI volume. Also, each individual dataset was manually screened for data quality via inspection for image artifacts and excessive head motion (>2–3 mm). The co‐registered T1 images were then segmented using all six tissue probability maps delivered by SPM12 and the resulting deformation fields were utilized to normalize functional and anatomical images to the Montreal Neurological Institute (MNI) reference space. All individual EPI volumes were smoothed using an 8 mm full‐width half‐maximum Gaussian kernel to account for anatomical variability after spatial normalization across participants. All the analysis following preprocessing was conducted using SPM8. Functional images were then analyzed using a general linear model (GLM). On the single subject level, a design matrix was defined including one regressor for each of the 9 conditions. Brain activation during each trial was modelled using the canonical hemodynamic response function (HRF). The design matrix also included only five of the six motion parameters, which are usually measured (x‐, y‐, z‐translation and ‐rotation) and were estimated during the motion correction step as multiple regressors to minimize signal‐correlated motion effects. We excluded motion regressor y‐translation, as we experienced measurement errors along this regressor caused by the scanner. A high‐pass filter of 128 s was applied to the data and common parameter estimation was performed with a one‐lag autoregressive model AR(1) to account for serial correlations in fMRI time series due to aliased biorhythms and unmodeled neuronal activity.

2.5. Functional connectivity estimation

Functional connectivity between brain regions time‐locked to a given stimulus was studied by conducting a ß‐series correlation analysis, which was introduced by Rissman et al. (2004). Göttlich, Beyer, and Krämer (2015) created BASCO, a Matlab‐based toolbox, with which the ß‐series correlations can be extracted. The method is implemented based on a GLM, where the evoked brain activity in each voxel is modeled by a separate covariate time‐locked to the stimulus onset after the BOLD signal has been convolved with the HRF. The methodological pipeline for computing condition‐specific functional connectivity based on fMRI measurements is illustrated in Supporting Information, Figure 1. To ensure a stable model fit for the ß‐series correlation method, for which a rather slow event‐related data with stimulus durations not smaller than the TR is suggested, the duration of stimulus presentation was set to 2.6 s. Temporal derivatives of the HRF were not included following the same procedure as Rissman et al. (2004) and suggested by Göttlich et al., (2015). In contrast to a standard univariate analysis of brain activity, this approach computes a regressor for each individual trial, rather than using a single regressor for all trials of a single experimental condition. This way, a single ß‐value for each voxel and each trial is calculated. To conduct a region of interest (Pettorossi et al., 2005) based functional connectivity analysis time locked to a given stimulus, these ß‐values were averaged across a priori chosen spherical (5mm) ROIs. Two‐hundred sixty‐four putative functional ROIs covering cerebellum and cortex by Power et al. (2011) plus 17 olfactory ROIs adapted from Seubert, Freiherr, Djordjevic, and Lundstrom (2013) were used (Supporting Information, Table 1). The resulting 281 ROIs were carefully screened for possible overlaps to prevent artificially induced connectivity. The correlations between the ß‐series were calculated using Fisher z‐transformed Pearson‐Moment correlation. The resulting 281 × 281 ROI‐based connectivity matrices (with autocorrelations set to zero) were used for further network‐statistics and graph theoretical measurements.

2.6. Extracting condition‐specific ß‐series correlations

To explicitly study MIP, the combined unimodal stimulus network (CUSN) and the bimodal stimulus network (BSN) were compared. Therefore, fMRI data individually time‐locked to bimodal conditions were merged (OPVP + OPVU + OUVU + OUVP) resulting in an averaged ß‐series correlation representing the BSN. The same was done with all unimodal stimulus conditions (OP + OU + VP + VU) resulting in the CUSN. To study MIP‐specific neural network properties resulting from the subconditions (OPVP, OPVU, OUVU, OUVP), the same principle was applied to combine the fMRI data of the two equivalent unimodal stimuli. For an explanatory summary of conditions and comparisons, see Table 2. Using this approach, the bimodal neural networks are assumed to process sensory information based on two sensory input channels and the combined unimodal networks are assumed to process the same information, but lacking aspects of MIP (neuronal processing was conducted unimodally).

Table 2.

Bimodal and combined unimodal stimulus conditions based on the extraction or combination of the condition specific ß‐series correlations

Bimodal condition		Combined unimodal condition
Main condition
BSN: OPVP + OPVU + OUVU + OUVP	vs	CUSN: OP + OU + VP + VU
Subconditions
OPVP	vs	OP + VP (combined‐OPVP)
OPVU	vs	OP + VU (combined‐OPVU)
OUVU	vs	OU + VU (combined‐OUVU)
OUVP	vs	OU + VP (combined OUVP)

2.7. Network‐statistics

Network‐statistics were conducted using MATLAB R2016a and Statistics and Machine Learning Toolboxes Version 10.2 (The MathWorks, Inc., Natick, MA). The ß‐series ROI‐based correlation matrices for BSN and CUSN from each subject were used to test for network differences resulting in bimodal network matrices (BNM) and combined unimodal network matrices (CUNM). A two‐sided paired‐samples t‐test was used to test for differences in any network element i,j of the correlation matrix over all 18 subjects between BNM and CUNM. This results in revealing significantly stronger correlated network nodes within the BNM with the aim to filter out MIP specific structures. To account for multiple comparisons a Bonferroni correction with an α‐value of 1.27097e‐6 was applied (for a review, see Armstrong, 2014). As this is a very strict threshold, we additionally used a slightly more liberal α‐value of 1.27097e‐5 to study differences between the BNM and the CUNM. Network‐statistics for bimodal stimulus networks compared to combined unimodal stimulus networks for subconditions regarding pleasant, unpleasant, congruent, and incongruent stimulus content were also conducted. This was accomplished using the same principle as comparing BNM to CUNM. This procedure was thought to underline the findings in the main analysis and to study information processing specifically for olfactory and visual stimuli containing pleasant or unpleasant contents.

Network differences were not computed using the network based statistic toolbox (NBS) by Zalesky, Fornito, and Bullmore (2010) due to several reasons, which we will briefly outline: NBS identifies connected graph components as a set of suprathreshold links for which a path can be found between any two nodes using a breadth first search (Ahuja, Magnanti, & Orlin, 1993). Permutation testing is used to ascribe a p value controlled for the family‐wise error (FWE) to each connected component based on its size. This means, that NBS computes its statistic on a component, rather than on a link basis. “Hence, with the NBS, it is never possible to declare individual links as being significant, only the component to which they belong can be declared significant” (Zalesky et al., 2010). Therefore, using the NBS toolbox offers a great possibility to identify significant differences of network subcomponents. With regards to our experimental design, as well as the aim of identifying individual MIP‐specific network hubs in our study, this approach seems to be not ideal. The justification lays mainly in the fact that little is known about functional connectivity underlying sensory integration. MIP seems to be very context and sensory dependent, which conceptually hinders the aim for an MIP‐network identification. Furthermore, identifying significantly stronger correlated network nodes using a paired‐samples t‐test computed on each matrix element i,j, as it was conducted in this study, does not require to threshold the connectivity matrices. No need of a threshold offers an immense advantage as no general solution in this matter could be found (Langer, Pedroni, & Jancke, 2013; van den Heuvel et al., 2017), which renders this methodological step a matter of experimentation and therefore limits comparability of gained results.

2.8. Graph theoretical parameters

Roughly speaking, a network is a large connection dataset. The brain is a complex network on multiple spatial and time scales. Complex systems can be quantitatively described when we draw them as mathematical graphs. Graph theoretical parameters assessing topological characteristics were calculated using algorithms of the Brain Connectivity Toolbox (BCT) by Rubinov and Sporns (2010). Nodal Graph metrics were averaged across the entire network to allow a whole‐brain network analysis. The following graph metrics were computed to describe condition‐specific network differences: degree, density, strength, clustering coefficient and global efficiency. Degree of a node reflects the number of connections of a given node. Averaged over the whole network this parameter represents the degree of “interconnectedness.” Density is the fraction of present connections (degree) to all possible connections (281 × 281 in our study), which makes density and degree proportional to one another. Because degree and density are computed using binary matrix information and not weighted information, they fully correlate to one another. Node strength represents the sum of weights of edges, which connect two nodes (Rubinov & Spornss, 2010). This parameter includes information about how strong a node is connected to other nodes in the network by using weighted links information. Degree, density, and strength have been computed to descriptively analyze networks including aspects of MIP and networks lacking those to subsequently statistically compare the resulting graph theoretical parameter. The underlying hypothesis was, that we would find a significantly higher degree, and density within networks including MIP, due to the required “interconnectedness” for integrating sensory information within multimodal brain regions. Two major network description regarding segregation and integration of different brain areas within a network are assessable with graph theory. Clustering coefficient describes the network's tendency to build “separated communities” consisting of several interconnected nodes (clusters) within the network and is therefore one parameter to describe the degree of segregation. Also, a high clustering coefficient has been suggested to reflect robustness of a network (Ash & Newthh, 2007). Global efficiency is the average inverse shortest path length in the network (Latora & Marchiorii, 2001). The average path length, also called the characteristic path length, is a measurement of integration by representing the total sum of individual edge lengths in a weighted graph. Edge lengths are inversely related to edge weights, meaning, large weights are considered to reflect strong correlations and a close proximity and small weights represent the opposite (Rubinov & Spornss, 2010). Global efficiency is due to its inverse relationship to the characteristic path length a measure of integration by representing how efficient (spatially) information can travel from any node to its target node (Hart, Ypma, Romero‐Garcia, Price, & Suckling, 2016). Networks display a so called “small‐world” characteristic, first described by Watts and Strogatz (1998), if they are highly clustered and resemble a small characteristic path length. This means, they have to be simultaneously highly segregated and integrated (for a review, see Bassett & Bullmorere 2016). Watts and Strogatz described “small‐world” features as “…enhanced signal propagation speed, computational power, and synchronizability.” Last, modularity after Newman (2006) was assessed to identify the degree to which networks tend to form modules which are nonoverlapping groups of nodes. The computation is achieved by maximizing the number of within‐group edges, and by minimizing the number of between‐group edges. In regards to bimodal sensory processing, assessing modularity seems to be of importance as it may reflect whether modules occur within sensory specific processing areas by “building” clearly delineated groups of nodes, or whether processing is distributed more equally over the whole network.

To analyze network specific graph theoretical parameters of bimodal sensory processing and combined unimodal sensory processing, only the subcondition‐specific matrices were used for analysis. BSN, and CUSN were not considered for graph theoretical analysis, due to their composition of four individually combined task‐specific fMRI datasets (see above). Combining all bimodal as well as all unimodal fMRI data to obtain the overall “conditions” BSN and CUSN could falsify the resulting network specific graph metrics. The main reason was to avoid a cluster shift of the individual clusters within the individual bimodal, and individual unimodal networks of the four subconditions by combining the respective fMRI data (Table 2). Graph theoretical measurements for OPVP, OPVU, OUVU, and OUVP were based on uncombined fMRI data, which was acquired by condition‐specific bimodal stimulus presentation within the scanner. The approach combining unimodal stimulus networks (e.g., OP + VP) to compare these to the equivalent bimodal network (e.g., OPVP) is valid. The crucial point is that the difference between these networks (e.g., OPVP vs. OP + VP) is object of the study to explicitly study MIP‐specific network characteristics.

All computed graph metrics represent whole‐brain parameters per network and as input matrices weighted, unsigned and undirected connectivity matrices were used. To test for differences between the calculated graph metrics (negative and positive), a paired‐samples t‐test with a significance threshold of α = 0.05 was conducted. Graphs were thresholded to zero meaning we acquired only positive and only negative (absolute values) networks following a similar procedure as van den Heuvel et al. (2016, see Table 3 for a summary on the conducted graph theoretical comparisons). The networks underwent no further thresholding procedures as it has been shown that especially weak, long‐distance connections contribute significantly to cognitive functioning and therefore seem to have a further unidentified impact on a networks’ architecture and information transfer (Santarnecchi, Galli, Polizzotto, Rossi, & Rossi, 2014).

Table 3.

Summary on graph theoretical subcondition comparisons

Bimodal subconditions		Combined unimodal subconditions
Positive OPVP	vs	Positive combined‐OPVP
Negative OPVP	vs	Negative combined‐OPVP
Positive OPVU	vs	Positive combined‐OPVU
Negative OPVU	vs	Negative combined‐OPVU
Positive OUVU	vs	Positive combined‐OUVU
Negative OUVU	vs	Negative combined‐OUVU
Positive OUVP	vs	Positive combined‐OUVP
Negative OUVP	vs	Negative combined‐OUVP

2.9. Visualization of condition‐specific networks

Visualization of the condition‐specific networks was achieved using BrainNet Viewer, a Matlab‐based toolbox (Xia, Wang, & He, 2013). The matrices resulting from network‐statistics were binarized for visualization of the remaining MIP‐specific network nodes The brain model we used as a surface template was ICBM152, which is congruent to MNI space and therefore to our used coordinates (Collins, Zijdenbos, Baaré, & Evans, 1999). Anatomical labeling of the MNI coordinates was conducted using xjView toolbox (http://www.alivelearn.net/xjview).

3. RESULTS

3.1. Condition‐specific network‐statistics

No statistical differences were observed using the Bonferroni corrected α‐value of 1.27097e‐6. Following network differences have been established using an α‐value of 1.27097e‐5: The comparison between BNM > CUNM revealed six nodes, which are significantly stronger correlated in BNM (Figure 3a). The comparison OPVU > combined‐OPVU revealed one significant network difference between the left precentral gyrus and the left superior temporal gyrus (Figure 3b). The comparison between OUVP > combined‐OUVP revealed four nodes, which are significantly stronger correlated in OUVP (Figure 3c). Comparing OUVU> combined‐OUVU revealed two nodes (Figure 3d). The statistical comparisons between OPVP > combined‐OPVP revealed no significant differences using an α‐value of 1.27097e‐5.

Figure 3.

Condition‐specific network‐statistics. (a) Significant network nodes in BNM > CUNM. (b) Significant network nodes in OPVU > combined‐OPVU. (c) Significant network nodes in OUVP > combined‐OUVP. (d) Significant network nodes in OUVU > combined‐OUVU. Upper part displays network nodes with corresponding MNI coordinates below [Color figure can be viewed at http://wileyonlinelibrary.com]

3.2. Graph theoretical parameters

The results for each calculated graph metric were statistically compared (N = 18) and box plotted for differences between bimodal and combined unimodal networks characteristics using a significance threshold of α = 0.05. All boxplots as well as detailed statistical test result tables can be found in the Supporting Information, Figures 2–9 and Tables 2–9. In the following, only significant test results will be reported.

The statistical comparisons between positive OPVP and positive combined‐OPVP revealed significant differences in degree and density (p = .0006), clustering coefficient (p = .0191), global efficiency (p = .0001), and modularity (p = 3.96E‐04). The bimodal network differed hereby with decreased means in degree and density, but increased values in clustering coefficient, global efficiency, and modularity (Supporting Information, Figure 2 and Table 2).

The statistical comparisons between negative OPVP and negative combined‐OPVP showed significant differences in all five measured graph parameters. With degree and density (p = .000589), clustering coefficient (p = 7.39E‐04), strength (p = 6.75E‐06), and global efficiency (p = 2.54E‐10) differing form the equivalent combined unimodal network with increased whole‐brain parameter (Supporting Information, Figure 3 and Table 3).

Positive OUVU compared to positive combined‐OUVU revealed significant differences in clustering coefficient (p = .0167), strength (p = .0411), and global efficiency (p = .0002). The bimodal network showed hereby also increased whole‐brain parameter (Supporting Information, Figure 4 and Table 4).

The comparison between negative OUVU and negative combined‐OUVU exhibited significant differences in clustering coefficient (p = .0002), strength (p = .0109), and global efficiency (p = 4.57E‐05). Again, the bimodal network displayed increased whole‐brain parameter (Supporting Information, Figure 5 and Table 5).

Positive OPVU and positive combined‐OPVU differed in clustering coefficient (p = .0262), strength (p = .0459), and global efficiency (p = .0012) with the bimodal network showing higher values for these parameters than the equivalent combined unimodal network (Supporting Information, Figure 6 and Table 6).

The statistical comparisons between negative OPVU and negative combined‐OPVU revealed significant differences in clustering coefficient (p = .0011), strength (p = .0388), and global efficiency (p = .0001). The bimodal network differed hereby with increased whole‐brain parameter (Supporting Information, Figure 7 and Table 7).

Positive OUVP compared to positive combined‐OUVP differed significantly in clustering coefficient (p = .0002), strength (p = .0009), and global efficiency (p = 4.23E‐06) with positive OUVP exhibiting greater values (Supporting Information, Figure 8 and Table 8).

Differences between negative OUVP and negative combined‐OUVP could be found in clustering coefficient (p = .0010) and global efficiency (p = .0038) with negative OUVP displaying increased graph theoretical parameters (Supporting Information, Figure 9 and Table 9).

A summary of significant differences between bimodal stimulus networks and combined unimodal stimulus networks for positive and for negative correlations is displayed in Table 4.

Table 4.

Summary of significant differences in graph metrics between bimodal and combined unimodal networks

	OPVP vs c‐OPVP	OUVU vs c‐OUVU	OPVU vs c‐OPVU	OUVP vs c‐OUVP
Positive network
Degree	Yes↓	‐	‐	‐
Density	Yes↓	‐	‐	‐
Clustering coefficient	Yes↑	Yes↑	Yes↑	Yes↑
Strength	‐	Yes↑	Yes↑	Yes↑
Global efficiency	Yes↑	Yes↑	Yes↑	Yes↑
Modularity	Yes↑	‐	‐	‐
Negative network
Degree	Yes↑	‐	‐	‐
Density	Yes↑	‐	‐	‐
Clustering coefficient	Yes↑	Yes↑	Yes↑	Yes↑
Strength	Yes↑	Yes↑	Yes↑	‐
Global efficiency	Yes↑	Yes↑	Yes↑	Yes↑
Modularity	‐	‐	‐	‐

Note. c = combined; ↑ indicating bimodal condition differs by significantly increased values; ↓ indicating bimodal condition differs by significantly decreased values.

4. DISCUSSION

This study aimed to identify context‐independent MIP‐specific neural network characteristics based on the stimulation of the olfactory and visual system. To describe the topology of the networks including aspects of MIP, we assessed graph metrics clustering coefficient as a measurement of segregation and global efficiency as a measurement of integration. Furthermore, we hypothesized that the neural networks underlying bimodal stimulus presentation would exhibit (a) significantly stronger correlation between MIP‐specific network nodes and (b) significantly increased graph theoretical parameter degree and density due to the required increased “interconnectedness” for integrative mechanisms. Modularity was computed to specifically aim for the networks topology regarding the distribution of sensory processing over a given network. Furthermore, we approached the microscale inhibitory and excitatory network computations for MIP via the whole‐brain graph metric degree for negative‐only and positive‐only matrices.

4.1. Condition‐specific network‐statistics reveal MIP‐specific network nodes

Comparing the BNM to the CUNM, three connections with significantly stronger correlated behavior emerged. One of those was the connection between the right insula and the right putamen. As in our study, various studies implementing multisensory stimulation have shown the involvement of the right insula. It has been observed that its functional role is important for integrating multimodal information including sounds and vibrotactile stimuli (Renier et al., 2009), audio and visual stimuli (Naghavi, Eriksson, Larsson, & Nyberg, 2007), audio‐visual integration of speech (Olson, Gatenby, & Gore, 2002), tactile‐visual matching (Banati, Goerres, Tjoa, Aggleton, & Grasby, 2000) and auditory‐visual stimulus‐onset‐(a)synchrony (Bushara, Grafman, & Hallett, 2001). The right putamen, a hypothesized attention‐related brain region has been linked to modulation during sensory integrational processing (Hopfinger, Buonocore, & Mangun, 2000; Li et al., 2015). Based on our findings, we can conclude that the insula connected to the right putamen is part of a functional MIP‐specific network. Further, a connection between the right precuneus and the left supramarginal gyrus was evident. The central precuneus shares connections to the dorsolateral prefrontal, dorsomedial prefrontal and to the multimodal lateral inferior parietal cortex, which is hypothesized to be a cognitive/associative region (Margulies et al., 2009). Interestingly, the precuneus has been suggested to be involved in memory‐related imagery (Fletcher et al., 1995), which contributes to object identification (for a review, see Stevenson & Casese 2005). An odor‐based fMRI study by Royet, Delon‐Martin, and Plailly (2013) revealed strong coactivation between the precuneus and other regions involved in mental smell imagery in students, but not in professional perfumers. The authors concluded that the difference between both groups was due to the connections of the precuneus to the superior parietal lobe which is active when the need for top–down assistance to memory retrieval is maximal (Ciaramelli, Grady, & Moscovitch, 2008), as it was for students but not for the professional perfumers. The left supramarginal gyrus, which connects to the right precuneus, is known to be involved in integrational processes of multimodal sensory information as space perception (Cabeza & Nybergg, 2000), bimodal audio‐visual cueing stimuli (Yu, Li, & Sun, 2016), and the proprioception and resolution of conflicting sensory information (Tsakiris, Longo, & Haggard, 2010). Due to the functional aspect of the precuneus regarding memory‐related visual imagery and the multimodal character of the supramarginal gyrus, this functional connection seems to be an important contributor for an MIP‐specific network processing olfactory and visual information. Finally, we established a connection between the left middle occipital gyrus and the left inferior frontal gyrus (orbitofrontal cortex, OFC). Anderson et al. (2003) showed that neurons within the OFC encode the state‐dependent reward value of smells; however, neutral somatosensory, auditory, and visual stimuli are also processed in this area (McCrea, 2008). Interestingly, previous research revealed the involvement of the inferior fronto‐occipital fasciculus (IFOF) connecting the two identified brain regions, the middle occipital gyrus and the inferior frontal gyrus. The IFOF is a large white matter tract connecting the frontal, temporal, and occipital lobes (Kvickstrom et al., 2011) consisting of major efferent and afferent neural projections (Kier, Staib, Davis, & Bronen, 2004). The function of this fiber tract is not fully understood. Based on its structural connections, it seems to be important for the connection of the prefrontal cortex with the auditory and the visual association cortices in the temporal lobe (Kier et al., 2004; Petrides & Pandyaa, 1988). Thomas et al. (2008) have linked the IFOF to face recognition, thus indicating a functional role of identifying visual information with regards to a general approach of object identification as further investigated by Tavor et al. (2014). Further, Martino, Brogna, Robles, Vergani, and Duffau (2010) suggested: “that the IFOF may have an important role in the subcortical network underling the semantic system.” The semantic system (underlying object recognition) is of interest when studying MIP‐specific network components based on olfactory and visual stimuli. Olofsson, Rogalski, Harrison, Mesulam, and Gottfried, (2013) studied patients with primary progressive aphasia to investigate the interactions between olfactory input and lexical access. The magnitude of temporal cortical thinning within the patients correlated with impairments in odor familiarity and odor matching to visual cues, whereas they observed the inferior frontal gyrus to be correlated with both odor naming and matching. These results show, that the inferior frontal gyrus seems to play an important part in matching visual with olfactory cues and in retrieving odor‐matching semantic information, and that the temporal association area is linking odor object representations to transmodal networks. Furthermore, Zelano, Montag, Khan, and Sobel (2009) made a very interesting observation when studying odor memory: “when an odorant was easily named, its maintenance in working memory was reflected in sustained activity preferentially in the area of the inferior frontal gyrus. This is consistent with maintenance of this information within a phonological loop.” Baddeley's theory on working memory states that its function is based on three systems, the phonological loop, the visuospatial sketchpad, and the episodic buffer that combines information from multiple sources (Baddeley, 1992). While the visuospatial sketchpad is thought to act as a maintainer and manipulator for visuospatial imagery, the phonological loop is hypothesized to maintain speech‐based information. This enables the brain to link visual semantics and language to process “on‐line” information within the episodic buffer (Buchsbaum & D'Espositoo, 2008). The effects of task and stimulus complexity on working memory activation have been introduced before and are of high importance when discussing MIP on a functional whole brain macroscale (for a review, see Quak, London, & Talsma, 2015). Paulesu, Frith, and Frackowiak (1993) originally suggested the left supramarginal gyrus (BA 40) as the primary neural correlate for storing phonological information. This PET study has been critically discussed, but results have been replicated in various studies using different methodologies (Celsis et al., 1999; Hartwigsen et al., 2010; Stoeckel, Gough, Watkins, & Devlin, 2009). The two odors used in this study could be easily named by the participants, especially because visual input—when congruent—was provided. In this context, the factor congruent or incongruent stimulus presentation does not play an important role, because if the two completely different (unpleasant and pleasant) odors have been identified for the first time with the matching picture, they could be kept separated and named (in mind) and therefore produce results comparable with the results obtained by Zelano and colleagues. At this point, we would like to draw the attention to the previously discussed MIP‐specific functional connectivity between the right precuneus, which is hypothesized to be important for memory‐related imagery, and the left supramarginal gyrus. Based on the involvement of four out of the six so far identified MIP‐specific network nodes we hypothesize the following model for integrating information based on bimodal stimulus presentation of the olfactory and visual system (Figure 4).

Figure 4.

Hypothesized MIP‐specific information flow for network nodes which are significantly stronger correlated in BNM compared to CUNM. The precuneus being important for memory‐related imagery is connected to the supramarginal gyrus, which functions as a phonological store. Both regions implement their integrated memory retrieved visual and language information into the temporal association cortex. Here, another information stream via the IFOF connecting the middle occipital gyrus processing visual information with the inferior frontal gyrus activated by namable odor working memory fuses in, allowing a complex computation of integrating incoming multisensory information [Color figure can be viewed at http://wileyonlinelibrary.com]

The network comparison between OPVU and combined‐OPVU revealed one significant connection which lies between the left superior temporal gyrus and left precentral gyrus. During the search for neural correlates of crossmodal integration of intranasal stimuli of pleasant and relatively unpleasant mixtures, Bensafi et al. (2012) observed the superior temporal gyrus to be significantly activated. Furthermore, the superior temporal gyrus region has been shown before to be of crossmodal nature when processing auditory and visual information (Calvert, 2001). Another prominent functional role supporting phonological processing in object naming and other verbal tasks has been assigned to the superior temporal gyrus by Hickok and Poeppel (2004). Benoit, Raij, Lin, Jaaskelainen, and Stufflebeam (2010) observed increased activity for incongruent (compared to congruent) audio‐visual stimuli (McGurk effect) in the precentral gyrus. Here, our data confirm the previous finding that the superior temporal gyrus is a central structure in multisensory integration processing, with the connection to the precentral gyrus being of upmost importance during incongruent stimulation.

OUVP compared to combined‐OUVP revealed two functional connections. The emerged connection between the right fusiform gyrus and the left postcentral gyrus will be discussed first. The function of the fusiform gyrus adjusting the lingual gyrus, the parahippocampal gyrus and the inferior temporal gyrus is highly discussed as a face perceptional area (Kanwisher, McDermott, & Chun, 1997; McCarthy, Puce, Gore, & Allison, 1997), but has also been linked to within‐category non‐faces identification (Gauthier, Skudlarski, Gore, & Anderson, 2000; Tarr & Gauthierr, 2000). The postcentral gyrus has been found to respond to and interpret information for the sense of touch (Nii, Uematsu, Lesser, & Gordon, 1996; Puce et al., 1995; Uematsu, Lesser, & Gordon, 1992). It is unclear why the two brain regions show involvement in seemingly “stimulus‐foreign” processing. One explanation could be that so‐called multisensory neurons, which are located within so far classified unisensory areas, serve as connectors between these unisensory cortices (Foxe & Schroederr, 2005; Macaluso & Driverr, 2005). Other studies observed similar “stimulus‐foreign” co‐/activations within regions, which were not hypothesized to be involved in certain stimulus processing beforehand (for a review, see Stein & Stanfordrd, 2008). The second connection we established comparing OUVP to combined‐OUVP involved the right superior temporal gyrus and the left supramarginal gyrus. Both structures have been discussed already, but here the right (not left) superior temporal gyrus is concerned. How much the function changes recruiting the right instead of the left superior temporal gyrus as an MIP‐specific network node is not assessable. The incongruency in our stimulus design for condition OUVP (unpleasant olfactory and pleasant visual stimulus) causing an irritation between what you see and smell and the corresponding memorized object character might have required a stronger semantic association reflected in an increased connectivity between the supramarginal gyrus and the superior temporal gyrus.

OUVU compared to combined‐OUVU revealed one functional connection between the right fusiform gyrus and the left insula to be stronger correlated in OUVU. Both brain areas have been discussed above, but this contralateral network connection seems to play an important role for integrating congruent stimuli.

Interestingly, the network comparisons within the subconditions revealed pronounced increased connectivity within the conditions for incongruent stimulus presentation. This could be due to elicited low level arousal (Anders, Lotze, Erb, Grodd, & Birbaumer, 2004) caused by conflicting olfactory‐visual information. The overall approach to identify MIP‐specific network nodes was thought to filter out significantly stronger correlated edges. The result is a fragmented MIP‐network. The identified nodes can be interpreted to resemble hub‐like character due to their robust and strongly correlated behavior. To study the explicit hub‐character of the identified nodes further network‐statistics are necessary by explicitly comparing the bimodal networks to a random, but degree‐matched network, as it has been proposed for studying small worldness features (Wang et al., 2009).

4.2. Graph theoretical statistics

The original hypothesis that networks including aspects of MIP would exhibit a significantly increased degree and density was not verified. A significantly increased degree and density has only been observed in one network comparison between negative OPVP and negative combined‐OPVP. The assessment of modularity revealed, except in one network comparison between positive OPVP and positive c‐OPVP, that bimodal sensory networks and combined unimodal sensory networks do not differ in their topology in regards of forming modules. Furthermore, we observed that networks underlying bimodal stimulus presentation, respectively networks including MIP‐specific aspects, are characterized by a “small‐world‐like” topology (Watts & Strogatzz, 1998) due to a higher global efficiency and a higher clustering coefficient. Considering these results, we set up a new assumption: Networks including MIP‐specific aspects are not characterized by an increased “interconnectedness” due to MIP within additional brain regions (degree, density) responsible for integrative mechanisms, but by an increasingly efficient information transfer (global efficiency) and establishment of stronger connected clusters of nodes (cluster coefficient) between brain regions responsible for integrative mechanisms. Thus, networks including aspects of MIP seem to require the same brain regions as networks processing sensory information of a single sensory modality and that processing is equally distributed over the whole‐brain network (modularity). This leads to the conclusion, that networks processing multiple sensory information use the same network of brain areas, which are already involved in processing single sensory information. The bimodal sensory network is shaped by a rearrangement of this network architecture by weighting up or down node‐to‐node connection (strength) to change the efficiency as well as a pronounced incorporation of specific brain regions (cluster coefficient), but not by building distinct modules—for example, in sensory‐specific brain areas or MIP‐specific brain areas. This assumption would strengthen the idea, that the brain always includes aspects of MIP in that sense that brain areas responsible for higher cognitive processes, like identifying a presented odor by retrieving memory elicited semantic information, serve as integrational network components. Their involvement in processing information from more than one sensory channel is especially important for including multiple information in this cognitive process and therefore seem to be stronger involved in networks including MIP. This line of reasoning is justified by the brain regions, which we have obtained via network statistics: the insula, putamen, precuneus, supramarginal gyrus, middle occipital gyrus, inferior frontal gyrus, superior temporal gyrus, pre‐ and postcentral gyrus, and fusiform gyrus have been shown before to be of importance regarding higher cognitive functioning and MIP. This assumed network topology is supported by the idea that the brain is a small‐world network, which changes its topology depending on its wiring‐cost and functional dependencies (for a review, see Bullmore & Sporns, 2012). Nevertheless, the comparison between MIP‐specific networks and degree‐matched random networks are needed to further investigate this network architecture, especially to explore the role of small‐worldness characteristics in networks including aspects of MIP.

4.3. Microscale neuronal computation effects specific macroscale network architecture

It is known that responses to multisensory stimuli on the neuronal level can be measured as the integrated activity of both individual sensory modalities showing multisensory enhancement or superadditivity (Stein & Stanfordd, 2008). Therefore, a clearly distinguishable network processing bimodal sensory information and processing two (time‐wise) separated sensory information was expected. In detail, the bimodal network includes MIP‐specific cellular computations, which affect the information transfer on the macroscale (e.g., white fiber tracts) between ROIs. This characteristic information transfer for multisensory integration on a neuronal level has been observed in many studies using various electrophysiological methods eliciting so called excitatory–excitatory and excitatory–inhibitory connections (Meredith, Nemitz, & Stein, 1987; Meredith & Stein, 1986, 1996; for a review, see Stein & Stanford 2008 or Meredith 2002). The analysis conducted in this study does not offer insights on cellular behavior underlying MIP, but rather offers insights on its macroscale functional architecture hypothesized to be elicited by its microscale neuronal computations. Van den Heuvel et al. (2015) suggested that aspects of microscale cytoarchitectonics and macroscale connectomics are related. Later, Van den Heuvel et al. (2016) linked functional connectivity and intrinsic excitatory and inhibitory chemoarchitecture of cortical areas. Results of this study indicate—even if not significant, but very consistent—that networks including aspects of MIP show less positive and more negative correlations in terms of the degree of the “interconnectedness” (Supporting Information, Tables 2–9). Preliminary and careful inferences could be drawn that MIP‐specific computations are dominantly driven by inhibitory functional connectivity. Interestingly, Moussa et al. (2011) suggested “that when a cortical region is actively processing external information it becomes more inwardly focused and connections to other brain regions are shut down. Such changes in connectivity may provide mechanisms for task‐tailored information processing.” Even though these conclusions by Moussa et al. do not base on a separate analysis of negative‐only and positive‐only graphs, nor on MIP‐specific processing, their suggestion on these explicit network dynamics may point in the same direction as the suggestions we made and that the results and the analysis procedure we obtained may provide further and detailed insights in this matter. To which extend the conclusion made by Moussa et al. regarding a shift toward inhibition in connectivity during sensory processing and our preliminary inference that MIP processing is dominantly driven by inhibitory connectivity needs further clarification. How the identified MIP‐specific network nodes influence this macroscale architecture needs to be further studied, preferably by conducting a seed‐based weighted and directed network analysis. Nevertheless, if used and investigated, this graph theoretical approach to access microscale neuronal computations could lead to a noninvasive tool to reach and study cellular network activity underlying cognitive tasks or sensory processing of the human brain.

5. CONCLUSION

The current experiment is characterized by an olfactory and visual stimulus presentation to study context‐independent MIP using event‐related fMRI. The suggested model describing the hypothesized information flow between identified MIP‐specific network nodes includes brain area‐specific aspects of working memory, memory‐related imagery, object recognition, stimulus‐provoked phonology retrieval by an overarching semantic association, and a crucial pathway connecting these locally different functional brain areas, the IFOF. These findings are in conclusion with previous observations concluding that the associations of the semantic system affect and modulate olfactory perception, evaluation, and identification. To understand whether the suggested model is sensory‐specific, it would be most interesting to examine MIP‐specific network characteristics underlying other sensory stimulation than the ones used in this study. Using different sensory stimuli would also contribute to the question, whether equivalent semantic information retrieval accompanied by the activation of the corresponding brain areas is exclusive to olfactory and/or visual object identification.

Graph theoretical analysis revealed that networks including MIP‐specific aspects, exhibit a more efficient information transfer characterized by a higher segregated and integrated network topology. Furthermore, based on our data, all four bimodal networks reveal (not significant, but very consistent) a stronger connected negative network suggesting that MIP‐specific neural architecture is dominantly driven by inhibitory interactions between brain regions involved in stimulus‐processing. To conclude, the study at hand offers novel combined methodological approaches to study the whole‐brain macroscale neural network processing bimodal sensory information of the olfactory and visual system. Novel knowledge was gained regarding MIP‐specific network nodes and network topology leading to further research questions and ideas as proposed.

DISCLOSURE STATEMENT

The authors have nothing to disclose.

Supporting information

Additional Supporting Information may be found online in the supporting information tab for this article.

Supporting Table I

Supporting Figures and Tables

Ripp I, zur Nieden A‐N, Blankenagel S, Franzmeier N, Lundström JN, Freiherr J. Multisensory integration processing during olfactory‐visual stimulation—An fMRI graph theoretical network analysis. Hum Brain Mapp. 2018;39:3713–3727. 10.1002/hbm.24206