Neuronal Modeling of Cross-Sensory Visual Evoked Magnetoencephalography Responses in the Auditory Cortex

Abstract

Previous studies have demonstrated that auditory cortex activity can be influenced by cross-sensory visual inputs. Intracortical laminar recordings in nonhuman primates have suggested a feedforward (FF) type profile for auditory evoked but feedback (FB) type for visual evoked activity in the auditory cortex. To test whether cross-sensory visual evoked activity in the auditory cortex is associated with FB inputs also in humans, we analyzed magnetoencephalography (MEG) responses from eight human subjects (six females) evoked by simple auditory or visual stimuli. In the estimated MEG source waveforms for auditory cortex regions of interest, auditory evoked response showed peaks at 37 and 90 ms and visual evoked response at 125 ms. The inputs to the auditory cortex were modeled through FF- and FB-type connections targeting different cortical layers using the Human Neocortical Neurosolver (HNN), which links cellular- and circuit-level mechanisms to MEG signals. HNN modeling suggested that the experimentally observed auditory response could be explained by an FF input followed by an FB input, whereas the cross-sensory visual response could be adequately explained by just an FB input. Thus, the combined MEG and HNN results support the hypothesis that cross-sensory visual input in the auditory cortex is of FB type. The results also illustrate how the dynamic patterns of the estimated MEG source activity can provide information about the characteristics of the input into a cortical area in terms of the hierarchical organization among areas.

Significance Statement

Laminar intracortical profiles characterize feedforward- and feedback-type inputs to a cortical area. By combining magnetoencephalography (MEG) and biophysical computational neural modeling, we obtained evidence of cross-sensory visual evoked activity in the human auditory cortex driven by feedback-type input. The finding is consistent with previous intracortical recordings in nonhuman primates. The results illustrate how patterns of MEG source activity can be interpreted in the context of the hierarchical organization among cortical areas.

Introduction

Activity in sensory cortices is influenced by inputs from other brain regions via layer-dependent feedforward (FF) and feedback (FB) connections, which define a hierarchical organization among the regions (Rockland and Pandya, 1979; Felleman and Van Essen, 1991; Zeki, 2018). In nonhuman primates, the laminar profiles of early auditory cortex responses have FF-type characteristics, whereas those of cross-sensory visual evoked activations are of FB type (for reviews see Schroeder and Foxe, 2005; Ghazanfar and Schroeder, 2006; Kayser and Logothetis, 2007). Human electro- and magnetoencephalography (EEG, MEG) studies have revealed that cross-sensory activations and multisensory interactions can occur in low-order sensory areas very early, within a few tens of milliseconds from the stimulus onset (Giard and Peronnet, 1999; Foxe et al., 2000; Molholm et al., 2002, 2004; Teder-Sälejärvi et al., 2002; Lakatos et al., 2007; Talsma et al., 2007; Raij et al., 2010). In line with evidence from studies in other cognitive domains (Polimeni et al., 2010; Muckli et al., 2015; Kok et al., 2016; Fracasso et al., 2018; Klein et al., 2018; Finn et al., 2019; Norris and Polimeni, 2019; Lawrence et al., 2019b), recent high-field functional MRI (fMRI) studies have shown evidence of FF- and FB-like intracortical depth profiles of auditory cortex hemodynamic signals (De Martino et al., 2015; Ahveninen et al., 2016; Moerel et al., 2018, 2019; Wu et al., 2018; Gau et al., 2020; Chai et al., 2021; Lankinen et al., 2022). However, detailed neurophysiological analysis and computational modeling of cross-sensory effects are still lacking in humans.

Biophysically realistic computational models have been used to investigate laminar connections and cellular and circuit level processes of the neurons in detail, and they can also be used to simulate MEG and EEG signals (Jones et al., 2007; Kiebel et al., 2009; Sanz Leon et al., 2013; Hagen et al., 2018; Neymotin et al., 2020). The Human Neocortical Neurosolver (HNN; Neymotin et al., 2020) provides a cortical column model with FF- and FB-type inputs targeting different layers. With HNN, the cellular and network contributions to MEG and EEG signals from a source-localized region of interest (ROI) can be modeled. HNN has been used to interpret mechanisms of sensory evoked responses and oscillations in healthy and clinical populations (Jones et al., 2007, 2009; Ziegler et al., 2010; Lee and Jones, 2013; Khan et al., 2015; Sherman et al., 2016; Pinotsis et al., 2017; Sliva et al., 2018; Bonaiuto et al., 2021; Kohl et al., 2022; Law et al., 2022). Recently, Kohl et al. (2022) showed that sensory-specific auditory responses in the auditory cortex could be modeled by activating the neocortical circuit through a sequence of layer-specific FF–FB–FF inputs, similar to a prior simulation of somatosensory evoked responses in the somatosensory cortex (Jones et al., 2007). These results are consistent with studies suggesting that, in general, early components of sensory-specific evoked responses are related to FF processes, whereas later components reflect FB influences in activity evoked by auditory (Inui et al., 2006; Kohl et al., 2022), visual (Aine et al., 2003; Inui and Kakigi, 2006), and somatosensory (Cauller and Kulics, 1991; Inui et al., 2004; Jones et al., 2007) stimuli. In the present study, we applied HNN to examine FF and FB influences in cross-sensory evoked activity.

We investigated visual evoked responses in the auditory cortex by comparing the experimentally observed MEG responses with simulated source waveforms from a computational model (HNN). We hypothesized that the cross-sensory visual evoked response observed with MEG can be explained by FB-type input to the auditory cortex.

Material and Methods

Subjects

Eight healthy right-handed subjects participated (six females, age 22–30 years). All subjects gave written informed consent, and the study protocol was approved by the Massachusetts General Hospital institutional review board and followed the guidelines of the Declaration of Helsinki.

Stimuli and experiments

The subjects were presented with noise/checkerboard and letter stimuli in separate runs while MEG was recorded. Here we reanalyzed data from the noise/checkerboard experiment used in our earlier publication (Raij et al., 2010), together with the previously unpublished data from the letter experiment. Equiprobable 300 ms auditory, visual, and audiovisual (simultaneous auditory and visual) stimuli were delivered in an event-related design in pseudorandom order. The auditory noise stimuli were white noise bursts (15 ms rise and decay) and the visual checkerboard stimuli static checkerboard patterns (visual angle 3.5° × 3.5° and contrast 100%, with a peripheral fixation crosshair). The letter stimuli were spoken and written letters of the Roman alphabet (“A,” “B,” “C,” etc.). The subjects’ task was to respond to rare (10%) auditory, visual, or audiovisual target stimuli with the right index finger movement as quickly as possible. In the noise/checkerboard experiment, the target stimulus was a tone pip, a checkerboard with a gray diamond pattern in the middle, or a combination of the two. In the letter experiment, the target stimulus was the letter “K,” spoken and/or written. Data were recorded in three runs with different stimulus-onset asynchrony (SOA; mean 1.5, 3.1, or 6.1 s, all jittered at 1.15 s). There were 375 stimuli per category (auditory, visual, and audiovisual): 150 in the short, 125 in the intermediate, and 100 in the long SOA runs. All subjects were presented with the same order of experiments and stimuli. The auditory stimuli were presented with MEG-compatible headphones, with the intensity adjusted to be as high as the subject could comfortably listen to. The visual stimuli were projected onto a translucent screen. The stimuli were controlled using Presentation 9.20 (Neurobehavioral Systems).

MEG and MRI acquisition and coregistration

MEG was recorded with a 306-channel instrument with 204 planar gradiometer and 102 magnetometer sensors (VectorView; MEGIN) inside a magnetically shielded room (Cohen et al., 2002). Simultaneous horizontal and vertical electro-oculograms (EOG) were also recorded. All signals were bandpass-filtered to 0.03–200 Hz and sampled at 600 Hz.

Structural T1-weighted MRIs of the subjects were acquired with a 1.5 T Siemens Avanto scanner (Siemens Medical Solutions) and a head coil using a standard MPRAGE sequence. Cortical surfaces were reconstructed using the FreeSurfer software (http://www.surfer.nmr.mgh.harvard.edu; Fischl, 2012).

Prior to the MEG recording, the locations of four small head position indicator coils attached to the scalp and several additional scalp surface points were determined with respect to three fiducial landmarks (nasion and two preauricular points) using a 3-D digitizer (Fastrak Polhemus). For the MRI–MEG coordinate system alignment, the fiducial points were first identified from the structural MRIs, and then this initial coregistration was refined using an iterative closest-point search algorithm for the scalp surface locations using the MNE Suite software (Gramfort et al., 2014; http://www.martinos.org/mne/).

Neural currents underlying MEG signals

Macroscopically, the sources of MEG and EEG signals can be described in terms of primary currents, which represent the net effect of active neural currents within a small volume of the brain (for reviews, see Hämäläinen et al., 1993; Lopes da Silva, 2010; Ilmoniemi and Sarvas, 2019; Supek and Aine, 2019). The same primary currents contribute to both MEG and EEG, but with different location- and orientation-dependent weighting. The goal of MEG and EEG source estimation is to identify the spatiotemporal distribution of the primary currents. The primary currents can be assumed to be perpendicular to the cortex, and thus, their physical orientation depends on the sulcal and gyral folding patterns. Notably, MEG is insensitive to the radial component of the primary currents, whereas EEG is sensitive to sources of all orientations. The primary currents in the auditory cortex in the superior temporal lobe are mostly tangentially oriented and thus well suited to be studied by MEG (Gutschalk, 2019). The selective sensitivity of MEG to tangential source components can help to separate source waveforms from multiple simultaneously active regions, such as the cross-sensory visual evoked activity in the auditory cortex and the coinciding widespread occipital activity in the present study.

At the mesoscopic level, the primary currents can be modeled in terms of postsynaptic intracellular currents within a local neural circuitry, with the main contribution coming from the vertical (i.e., perpendicular to the cortex) component of intracellular currents within cortical pyramidal neurons. For symmetry reasons, the net effect of transmembrane currents, horizontally oriented intracellular currents, and any currents within interneurons are expected to largely cancel out (Thio and Grill, 2023). The direction of the vertical currents in a pyramidal cell depends on the locations of the synaptic input, whether the input is excitatory or inhibitory, the dendritic structure, and the dynamics of the transmembrane currents and potentials. For example, excitatory synaptic input to the upper part (tuft) of the apical dendrite of a pyramidal cell results in an initial downward current, whereas excitatory input to the basal dendrites results in an upward current (Linden et al., 2010; Lopes da Silva, 2010; Ahlfors and Wreh, 2015).

Because FF and FB-type connections into a cortical area have characteristic laminar distributions, FF and FB inputs are expected to result in MEG/EEG source waveforms reflecting different patterns of upward and downward intracellular current flow, thereby linking effects at the mesoscopic and macroscopic spatial scales (Ahlfors et al., 2015). Considering typical laminar locations of the soma and the dendrites of pyramidal cells, the FF and FB laminar input patterns can be transformed to characteristic spatial distributions of synaptic inputs into the cells (Jones et al., 2007). In the present study, we estimated MEG source waveforms (primary currents) in the auditory cortex and used HNN to model them as potentially resulting from the sum of vertical intracellular currents in response to FF- and FB-type inputs to the local cortical circuitry.

MEG preprocessing and source estimation

The MEG data were analyzed using MNE-Python (Gramfort et al., 2013). After excluding channels and time segments with excessive noise, independent component analysis was used to identify and remove artifacts related to eyeblinks, eye movements, and cardiac activity. The signals were then low-pass filtered at 40 Hz, and event-related responses were averaged separately for the auditory and visual trials, combining the long, intermediate, and short SOA runs. After excluding epochs contaminated by artifacts, an average of 368.4 (SD 7.6; noise/checkerboard experiment) and 371.5 (SD 4.7; letter experiment) epochs per subject remained in response to auditory and 369.0 (SD 5.7; noise/checkerboard experiment) and 371.4 (SD 4.1; letter experiment) to visual stimulation. In the present study we did not analyze the audiovisual or target trials. The zero level in each channel was defined as the mean signal over the 200 ms prestimulus baseline period.

Source activity was estimated at 4,098 discrete locations per hemisphere on the cortical surface, with an average separation of the source elements being ∼4.9 mm. Each source element was a current dipole oriented normal to the cortical surface, with the positive direction defined as pointing outward, toward the pial surface (called “upward” in HNN modeling). For the forward solution, a single-compartment boundary element model was used. Forward solutions were first computed separately for the three runs with different SOAs and then averaged (Uutela et al., 2001). Minimum norm estimates (MNEs; Hämäläinen and Ilmoniemi, 1994) for the cortical source currents were calculated (Fig. 1A). Both the gradiometer and the magnetometer channels were included in the source estimation. We used depth weighting 0.8 to reduce bias toward superficial currents. For ROI selection, the MNE values were noise-normalized to obtain dynamic statistical parametric maps (dSPMs; Dale et al., 2000).

Figure 1.

Source estimation and computational modeling of MEG data. A, An example of MEG data. Auditory evoked MEG sensor data (left) in one subject at 50 ms after auditory stimulus onset (isocontour line step 5.0 fT) are shown with the corresponding noise-normalized MNE (dSPM) source estimate over an inflated cortical surface (middle) and the estimated time course for the left hemisphere auditory cortex (right, vertical line at 50 ms). B, The HNN model. The MEG response is modeled using a network of neurons in a local cortical area (left). Local network structure (right) consists of pyramidal cells (blue) and interneurons (orange). Excitatory and inhibitory coupling is indicated by a circle and a bar, respectively. The network is activated by proximal (red) and distal (green) drives by input spike trains. Modified from Neymotin et al. (2020).

ROIs and MEG source time courses

Auditory evoked potentials and magnetic fields typically have three main deflections: P50-N100-P200 (or P50m-N100m-P200m for MEG), peaking approximately at 50, 100, and 180 ms, respectively, after the auditory stimulus onset (Picton et al., 1974; Hari et al., 1980; Hämäläinen et al., 1993). The ROIs were determined based on the auditory evoked N100m response, because the signal-to-noise ratio (SNR) of the visual evoked response over the auditory cortex was too low to reliably determine auditory cortex ROIs from the visual evoked data. We identified functional ROIs for the auditory cortex in each hemisphere, separately for each subject, based on the N100m peak of the auditory evoked response. First, anatomically defined regions were selected using the Destrieux atlas parcellation from FreeSurfer (Fischl et al., 2004; Destrieux et al., 2010): Heschl's gyrus, Heschl's sulcus, and the lower part of the planum temporale (masked with the supramarginal gyrus) were combined to cover the primary auditory areas. Then, from these regions the source element with the largest negative deflection between 60 and 110 ms (except for manually set 105 ms in one subject) in the dSPM source time course was identified. Using that source element as a seed point, all source elements that had a magnitude of 30% or more of the peak dSPM value and formed a continuous area around the seed point were selected. The average number of selected elements across subjects, hemispheres, and experiments for the auditory cortex ROIs was 19 (standard deviation, 8.7; range, 3–38). The same procedure was used to determine also additional control ROIs in the occipital cortex (V1, V2, and MT based on the FreeSurfer atlas; Fischl et al., 2008).

The source waveform for an ROI was defined as the sum of the MNE time courses over the selected source elements. Note that the magnitude of the response depends on the number of the source elements that were included in the ROI. Because MNE solution gives a distributed source estimate across the entire cerebral cortex, only a subset of the source elements showing estimated activity is within the ROI. Therefore, in general, the magnitude is expected to be lower than what would be found by the use of a single equivalent current dipole to represent the auditory cortex activity (as used, e.g., by Kohl et al., 2022). Although equivalent current dipoles are usually well suited to describe auditory evoked responses, here the use of a distributed source model (MNE) facilitated the extraction of the cross-sensory visual evoked responses in the auditory cortex occurring simultaneously with strong widespread activity in the occipital visual areas.

Neural modeling with HNN

Activity in the auditory cortex evoked by the auditory and visual stimuli was modeled using HNN (https://jonescompneurolab.github.io/hnn-core/; Neymotin et al., 2020). HNN is a software for simulating neocortical circuits and linking cellular- and circuit-level physiology to the electrical source currents measured by MEG and EEG. Thus, HNN allows developing and testing hypotheses on the neural origins of MEG/EEG signals. The neural currents contributing to the MEG/EEG signals from a source region are modeled in terms of the local network dynamics driven by layer-specific inputs (Fig. 1B). Simulated MEG/EEG source currents are represented as current dipole waveforms calculated from the distribution of intracellular currents in the dendrites of the pyramidal cells. MEG/EEG signals originate mostly from postsynaptic currents in cortical pyramidal neurons (Hämäläinen et al., 1993; Okada et al., 1997), and the magnitude and direction of the source current depends on the type of the synaptic input and its dendritic location (Allison et al., 2002; Jones et al., 2007; Linden et al., 2010; Lopes da Silva, 2010; Ahlfors et al., 2015; Ahlfors and Wreh, 2015), providing a link between the laminar distribution of synaptic inputs and the MEG/EEG source waveforms.

In HNN, a local cortical circuit model has a layered structure with pyramidal neurons whose somata are in the supragranular (layer 2/3) or infragranular (layer 5) layers and whose dendrites span across the layers. The model also includes inhibitory interneurons. External input to the circuit arrives through characteristic layer-specific FF- and FB-type connections. FF-type inputs consist of proximal drives to the basal dendrites of the pyramidal cells (assumed to arrive via the middle cortical layer), whereas FB inputs are represented by distal drive to the apical dendrites of the pyramidal cells. The model has 100 pyramidal neurons in each of layers 2/3 and 5; a scaling factor is used to match the simulated dipole to the magnitude of the recorded evoked response. The parameters of the HNN model originate from known anatomical and physiological cell properties, and the local connectivity within and between cortical layers is based on a large body of literature from animal studies (Jones et al., 2007; Neymotin et al., 2020).

We used HNN to explain the MEG responses to auditory and visual stimuli by different sequences of FF and FB inputs to the auditory cortex. Previously, Kohl et al. (2022) used HNN to model auditory evoked responses in humans, explaining the initial part of the response with a combination of FF and FB inputs. In the present study, we sought to replicate the sensory-specific findings of Kohl et al. and additionally hypothesized that the cross-sensory visual response can be explained by FB-type input alone.

We created two main HNN models for event-related activity in the auditory cortex: one for the response to auditory stimuli and one for the response to visual stimuli. For the HNN modeling, we averaged the MEG source waveforms across subjects, hemispheres, and experiments to improve the SNR of the experimental data. The averaging was justified by the lack of significant effects of hemisphere or experiment on the peak magnitudes in linear mixed-effects (LME) model analysis (see below, Statistical analyses). As a starting point, we used the auditory cortex model by Kohl et al. (2022) for activity in the right hemisphere evoked by auditory stimuli presented to the left ear. Because HNN has a large number of user-defined parameters, we made the following assumptions to limit the parameter space: (1) only the timing parameters of the FF/FB spike-train inputs (mean and standard deviation of a Gaussian distribution) were adjusted, in addition to the overall scaling factor for the simulated source waveforms; all the other parameters were kept unchanged. (2) For those other, internal, model parameters, we used the values optimized by Kohl et al. (2022) for sensory-specific auditory evoked response; here we assumed these values to be the same for the responses to both visual and auditory stimuli. (3) The simulations were limited to the time window of 0–130 ms for the auditory and 0–150 ms for the visual response, to focus on the early part of the responses. All simulated HNN waveforms were smoothed in the default 30 ms window (Hamming window convolution).

We first manually adjusted the timing of the FF/FB inputs to achieve a close initial fit to the MEG responses. Thereafter, we further tuned the model parameters using Bayesian optimization implemented in scikit-optimize (gp_minimize function, “expected improvement” as the acquisition function; Head et al., 2020) for estimating $\mu$ (mean input spike timing) and $\sigma$ (temporal distribution of input spikes) for each model by minimizing the root-mean-square-error (RMSE) between the simulated signals $s_{\mathrm{model}}(t)$ and the measured signal $s_{\mathrm{meas}}(t)$:

$$\sqrt{{\displaystyle \frac{1}{T}\sum _t{(s_{\mathrm{meas}}(t)\text{–}\mathrm{c}{s}_{\mathrm{model}}(t))}^2}},$$

where the optimal non-negative linear scaling factor is

$$c=\max \left(0,\sum _t{s}_{\mathrm{meas}}(t)\cdot s_{\mathrm{model}}(t)/\sum _t{s}_{\mathrm{model}}{(t)}^2\right),$$ (1)

and $T$ is the number of time points in the simulated time window. The bounds for the search space were: for auditory model ${\mu }_{\mathrm{FF}}$, 20…50; ${\mu }_{\mathrm{FB}}$, 55…95; ${\mu }_{\mathrm{FF}2}$, 90…130; for visual model ${\mu }_{\mathrm{FF}}$, 25…55; ${\mu }_{\mathrm{FB}}$, 80…120; for both models ${\sigma }_{\mathrm{FF}}$, 1…5; ${\sigma }_{\mathrm{FB}}$, 5…20; ${\sigma }_{\mathrm{FF}2}$, 5…20.

As HNN has a large number of parameters, it is possible that even after optimizing our main models, some other combination of parameter values could explain the waveforms equally well or better. Therefore, we formed alternative models by varying the number and timing of the FF and FB inputs. We focused on the comparison of FF + FB versus FB models for explaining the early part of the MEG activity evoked by auditory and visual stimuli.

Statistical analyses

To determine whether the magnitudes of the estimated MEG source waveforms (averaged across experiments and hemispheres for each subject) were significantly different from zero, we used t-tests with a threshold p < 0.05 at each of the 150 time points in the 0–250 ms window. The p-values were Bonferroni adjusted by a factor of 300 (for the two stimulus types and 150 time points). To evaluate between-subject consistency of the magnitudes of the largest defections in the evoked responses in each hemisphere and experiment, the average value over time points within ±10 ms windows around the peak latencies were calculated for each subject.

To evaluate whether there were significant effects between hemispheres and experiments in the measured MEG data, we calculated an LME model for each peak of interest. We applied an LME model with fixed effects for the hemisphere (left, right) and experiment (noise/checkerboard, letter) and a random effect for subjects by using the fitlme function in Matlab with the following model (in Wilkinson notation):

$$\mathrm{magnitude}\sim 1+\mathrm{hemisphere}+\mathrm{experiment}+(1|\mathrm{subject}).$$ (2)

To evaluate the HNN models, we compared the goodness of fit, as quantified by the RMSE between the measured and modeled MEG source waveforms, with the variability in the estimated MEG source waveforms. To obtain null distributions, the MEG source waveforms for auditory and visual evoked responses were resampled by drawing from 32 signals (8 subjects × 2 hemispheres × 2 experiments) 10,000 times with replacement, and for each resample the RMSE deviation from the average source waveform was calculated. The same was done for waveforms from 32 simulation runs for each HNN model (FF + FB and FB), and the RMSE between each of the 10,000 resampled simulations and the average of the MEG source waveforms was calculated. The distributions of the RMSE values for each model were compared with the corresponding experimental MEG null distributions. The model was considered overfit (RMSE smaller than experimental variability) if the median of its distribution was within the 5th percentile of the MEG null distribution.

Results

MEG source waveforms in the auditory cortex in response to auditory and visual stimuli

Estimated MEG source waveforms for auditory and visual evoked activity in the auditory cortex ROIs, averaged over subjects, experiments, and hemispheres, are shown in Figure 2. The auditory evoked response showed a characteristic biphasic P50m-N100m waveform, with a positive (upward current, directed toward the pial surface) peak at 37 ms and a negative (downward, toward the white matter) peak at 90 ms after the onset of the auditory stimuli. These peak latencies are similar to those reported previously for auditory noise burst stimuli (Hari et al., 1987). The cross-sensory visual evoked response in the auditory cortex had a monophasic peak at 125 ms after the appearance of the visual stimuli. The source magnitudes at the peak latencies were significantly different from zero (t-test; p < 0.05; Bonferroni adjusted; Table 1). The magnitude of the visual evoked response was ∼13% of the magnitude of the auditory N100m. The downward direction of the source current for the visual response was opposite to that of the early auditory P50m response, but the same as the direction of the prominent auditory N100m.

Figure 2.

MEG source activity in the auditory cortex. The estimated source waveforms in response to the auditory (blue) and visual (orange) stimuli: mean and standard deviation (colored shading) across subjects, hemispheres, and experiments. Positive and negative values correspond to upward and downward cortical currents, flowing in the direction toward the pial matter and the white matter, respectively. The gray shading indicates time points that differed significantly from zero (t-test, p < 0.05, Bonferroni adjusted).

Table 1.

Peak magnitudes of the MEG source estimates for the auditory and visual evoked responses in the auditory cortex

	Auditory		Visual
	P50m	N100m	N125m
Magnitude ± SD (nAm)	0.24 ± 0.29	−1.8 ± 1.32	−0.24 ± 0.26
t (31)	4.63	−7.67	−5.18
p value	6.3 × 10−5	1.2 × 10−8	1.3 × 10−5

Mean, standard deviation (SD), and statistical significance (t-test; p-values Bonferroni adjusted by the factor of 300) are shown. Negative values indicate downward-directed source current.

The estimated MEG source waveforms are illustrated separately for the noise/checkerboard and letter experiments in the left and the right hemispheres in Figure 3. The magnitude of the auditory N100m was larger for the letter than that for the noise stimuli in the left hemisphere, but similar in the right hemisphere; this lateralization is expected for responses to phonetic versus nonverbal stimuli (Gootjes et al., 1999; Parviainen et al., 2005). The anatomical overlap of ROIs across subjects (Fig. 3, middle panel) suggested that the prominent auditory evoked responses originated mostly in the Heschl's sulcus and the anterior part of the planum temporale. There were no clear differences in the location of the ROIs between the noise/checkerboard and letter experiments; however, for the letter stimuli, the location extended to the Heschl's gyrus in half of the subjects. The peak latencies of the auditory evoked responses were similar within a few milliseconds in both experiments. For the visual evoked response, there was a negative deflection with the peak latency within the range from 110 to 135 ms in both experiments in both hemispheres.

Figure 3.

MEG source waveforms in the left and right hemisphere auditory cortex in response to auditory and visual stimulation, shown separately for the noise/checkerboard and letter experiments. The source waveforms were averaged over subjects. The locations of the functional ROIs morphed to common anatomical space (“fsaverage” from FreeSurfer) are shown in the middle; the color bar indicates how many subjects’ individual ROIs overlapped at each cortical location. The black lines illustrate the Heschl's gyrus (anterior), Heschl's sulcus (middle), and part of planum temporale (posterior).

Variability of the estimated source waveforms among individual subjects is illustrated in Figure 4. The magnitudes at the peak latencies for each subject (Fig. 4, black dots) were submitted to LME models, separately for each of the three peaks (Table 2). The LME analysis confirmed the statistical significance of each peak (intercept p < 0.05) but indicated no significant differences between the hemispheres or experiments. Since no hemisphere- or experiment-specific differences were found in the magnitudes of the MEG source waveforms, we used the grand average MEG source waveforms (averaged over hemispheres, experiments, and subjects) in the subsequent HNN modeling.

Figure 4.

Individual variability of the estimated MEG source waveforms. A, Noise/checkerboard experiment. B, Letter experiment. Continuous lines and shading, mean ± standard deviation across subjects; black dots, response magnitudes for individual subjects, calculated as the average over ±10 ms time windows around the peak latencies in the grand average data. LH, left hemisphere; RH, right hemisphere.

Table 2.

LME model p-values for the magnitude of the peaks in the MEG source waveforms

	Auditory		Visual
	P50m	N100m	N125m
Intercept	0.042	4.7 × 10−8	3.6 × 10−5
Hemisphere	0.34	0.41	0.22
Experiment	0.14	0.26	0.72

No significant effects of hemisphere (left vs right) or experiment (noise/checkerboard vs letter) were found, but the intercept was significant for each peak.

The observed weak visual evoked activity in the auditory cortex partially coincided with strong activity in occipital visual cortical regions (Fig. 5). The estimated auditory cortex source waveforms could potentially reflect artifactual spread in the MEG source estimates due to activity in other cortical regions responding to the visual stimuli. We examined this possibility into two ways. First, we observed that the time course of the estimated sources for visual cortex ROIs had prominent deflections for both the onset (with peak latencies at ∼100 ms) and the offset (∼400 ms) of the visual stimuli, whereas in the auditory cortex the response was mainly seen for the onset only (Fig. 5A). If the onset and offset responses share a common spatial distribution in the occipital cortex, then also the potential artifactual spreading to the auditory cortex is expected to be the similar after the onset and the offset of the visual stimuli. However, this was not found in the data. Second, the spatial maps of the source estimates for the visual evoked responses had a gap between the weak auditory cortex activity and the strong occipital cortex activity (Fig. 5B). Artificial spread would be expected to be spatially uniform rather than forming separate foci in the auditory cortex. These observations argue against the possibility of the cross-sensory visual evoked response in the auditory cortex to artifactually result from spread from visual cortex.

Figure 5.

Evaluation of potential artifactual spatial spread in the estimated MEG source activity from visual cortex to the auditory ROIs. A, Source time courses (MNE, averaged across subjects and experiments) in response to visual stimuli for occipital areas V1, V2, MT (green), and the auditory cortices (V AC, orange). B, Spatial maps of the MNE source estimate for the visual evoked activity at the time of the largest peak in the response to visual stimuli in the auditory cortex.

Neural modeling with HNN

HNN was used to construct computational models for the observed grand average MEG source waveforms in the auditory cortex. In accordance with our a priori hypothesis, the main HNN model for the response to auditory stimuli had a combination of FF and FB inputs, whereas the main model for the cross-sensory response to visual stimuli had only FB input (Fig. 6).

Figure 6.

HNN simulations of the auditory cortex activity in response to auditory (left) and visual (right) stimuli. A, Simulated source waveforms (gray) overlaid with the experimentally observed grand average MEG source waveforms (blue, auditory; orange, visual). Simulated waveforms with the optimized (thick gray, average; thin gray, 10 individual simulation runs) model parameters are shown. B, Histograms of the timing of the inputs sampled from a Gaussian distribution with a model-specific mean and standard deviation (red, FF; green, FB). C, Layer-specific contributions to the simulated source waveforms after optimization (green, layer 2/3; purple, layer 5; gray, 10 respective individual simulation runs). Positive values correspond to upward (toward pial surface) and negative values to downward (toward white matter) flowing intracellular currents within the model pyramidal cells. D, Spiking activity of the pyramidal and basket cells in layers 2/3 and layer 5 (10 simulation runs).

The initial manual tuning values for the Gaussian time distribution of the inputs were µ_FF = 35 (σ_FF = 3.0) ms for the FF and µ_FB = 75 (σ_FB = 13.3) ms for the FB input in the auditory model, and µ_FB = 105 (σ_FB = 13.3) ms for the FB input in the visual model. The initial scaling factor was found to be 53 for the auditory and 5 for the visual simulation, reflecting the large overall difference in the response magnitudes. These scaling factors are considerably smaller than those in the work by Kohl et al. (2022). This was expected because a different method for estimating the sources was used, as noted in the Materials and Methods. It is noteworthy that the scaling factor does not affect the HNN simulation itself but is used to bring the simulation result to the same scale as the measurement.

Fine-tuning with Bayesian hyperparameter optimization resulted in only minor adjustments to the timing parameters. The optimized values were µ_FF= 35 (σ_FF = 1.0), µ_FB= 74 (σ_FB = 12.6), and scaling factor, 50, in the auditory model, and µ_FB= 107 (σ_FB = 18.9), scaling factor, 5, in the visual model (Table 3). The temporal distributions of the inputs are illustrated in Figure 6B. For both the auditory responses (P50m-N100m) and the visual responses (negative deflection peaking at 125 ms), the simulated source waveforms captured the main features of the experimentally observed MEG results (Fig. 6A).

Table 3.

Optimized HNN parameters for auditory and visual models

Model	${\mu }_{\mathrm{FF}}({\sigma }_{\mathrm{FF}})$	${\mu }_{\mathrm{FB}}({\sigma }_{\mathrm{FB}})$	${\mu }_{\mathrm{FF}2}({\sigma }_{\mathrm{FF}2})$	Scaling	RMSE
A: FF + FB + FF	35 (1.0)	76 (14.7)	90 (15.2)	60	0.15
A: FF + FB	35 (1.0)	74 (12.6)	-	50	0.23
A: FB	-	81 (18.0)	-	42	0.22
A: FF	35 (2.9)	-	-	0	0.73
V: FF + FB	44 (1.0)	101 (15.1)	-	6	0.041
V: FB	-	107 (18.9)	-	5	0.042
V: FF	40 (2.9)	-	-	0	0.087

The mean $\mu$ and standard deviation $\sigma$ (milliseconds) describe the temporal distribution of the inputs. The scaling factor was used to match the simulated waveform to the experimentally observed evoked response waveform. RMSE, root-mean-square error calculated between simulated and measured waveform. The main models are highlighted in bold.

Further insights to the generation of the source currents can be obtained by plotting separately the contributions from layer 2/3 and in layer 5 pyramidal cells (Fig. 6C) and the sequences of the spiking activity of the four cell types included in the HNN model (Fig. 6D). The simulated source waveforms in Figure 6A are the sums of the layer 2/3 and layer 5 currents. In the model for the auditory evoked response, the FF input was assumed to arrive through the middle cortical layer and then excite the basal dendrites of the pyramidal cells in both layers 2/3 and 5 (Fig. 6C, left). The net result of the FF input was an initial upward (positive) peak. The subsequent arrival of the FB input to the distal parts of the apical dendrites of the pyramidal cells resulted in the reversal of the net current to be downward (negative). These results for sensory-specific auditory responses are consistent with Kohl et al. (2022), who reported µ_FF (σ_FF) = 47 (3.0) ms and µ_FB (σ_FB) = 81 (13.3) ms for the first two inputs for the auditory evoked response. In the model for the cross-sensory visual evoked response in the auditory cortex, the FB input arriving distally drove the net source current downward within the apical dendrites of both layer 2/3 and layer 5 pyramidal cells (Fig. 6C, right). In both the auditory and the visual evoked response, the net current in layer 5 was negative. However, the current in layer 2/3 stayed initially positive for the auditory but not for the visual evoked response.

As HNN has a large number of parameters, it is possible that our chosen models are not the only ones that can reproduce the experimentally observed MEG source waveforms. However, HNN can serve us as a valuable hypothesis testing tool for testing different models. Alternative models with different combinations of FF and FB inputs are shown in Figure 7, and the corresponding optimized HNN parameters are listed in Table 3.

Figure 7.

Alternative models for auditory (A) and visual (B) responses. The main models (auditory, FF + FB, and visual, FB) are framed. The experimentally observed MEG source waveforms (blue, auditory stimulus; orange, visual stimulus) are overlayed with the simulated waveforms (thin gray, 10 individual simulation runs; thick gray, average of the individual runs). Histograms below the waveforms show the temporal distribution of FF (red) and FB (green) inputs to the HNN model of the auditory cortex neural circuit. In rightmost column, FF only simulations are additionally scaled (auditory ×100, visual ×15) to illustrate their waveforms compared with the MEG signal: However, the optimal non-negative scaling factor in these cases was actually 0 (dashed line).

For the auditory evoked responses, inclusion of a later second FF input to the model had only little effect on the simulated source waveforms within the 0–130 ms time window (A: FF + FB + FF2 vs A: FF + FB; Fig. 7A). Removing the initial FF input, however, resulted in a notable difference in the early part of the response (A: FB, 30–80 ms), during which the first upward deflection (P50m) was seen in the MEG data. Models with FF alone (A: FF, V: FF) could not produce the shape of the prominent negative deflections in the measured MEG response waveforms. For the FF only models, the optimization procedure resulted in “0” for the optimal non-negative scaling factor, implying that a flat line would give an equally small RMSE for the responses as any source waveform generated by an FF input alone. The rightmost column in Figure 7 shows the FF models scaled up to illustrate the mostly positive magnitude shape of the waveform.

For the visual evoked response, the difference between models with and without an FF input (V: FF + FB vs V: FB) was most pronounced in the early part (30–80 ms) of the simulated source waveforms (Fig. 7B). As expected, the V: FF + FB, which has more free parameters, gave a slightly better fit to the measured MEG signal than V: FB. However, considering the magnitude of the response with the baseline noise level (Fig. 2) suggests that the additional FF input in the model for the response to the visual stimuli may be explaining merely noise in the data (overfit).

We evaluated the FF + FB and FB models statistically by comparing their RMSE values with the MEG null distributions (Fig. 8). Variability in the resampled MEG responses is illustrated in Figure 8A. The resampled simulated source waveforms for the FF + FB and FB models, together with the MEG average, are shown in Figure 8B. The histograms of RMSE between the resampled simulations and the MEG average are shown in Figure 8C, overlaid with the MEG null distribution (i.e., the RMSE between the resampled MEG signals and their average). The median RMSE of the auditory FB model was just within the 95th percentile of the MEG null distribution (median RMSE for FB = 0.11; 95th percentile of MEG null distribution = 0.26). Adding the FF drive improved the model fit, but it did not overfit, staying above the 5th percentile (median RMSE for FF + FB = 0.078; 5th percentile of MEG null distribution = 0.052). The visual FB model provided a good fit, with the median RMSE being above the 5th but below the 95th percentile of the MEG null distributions (median RMSE for FB, 0.023; 5th and 95th percentiles of MEG null distribution = 0.018 and 0.060, respectively). However, the visual FF + FB model was within the 5th percentile (median RMSE for FF + FB = 0.017), suggesting an overfit to the data. To summarize, including the early FF in the model improved the fit in the auditory case but resulted in model overfit in the visual case. Thus, these results support our main hypothesis that the cross-sensory visual response can be explained with just FB input to the auditory cortex.

Figure 8.

Evaluation of the model fit for auditory (left panels, blue) and visual (right panels, orange) models. A, Resampled MEG source waveforms; the black line shows the average of the resamples. B, Resampled simulations for the FF + FB and FB models (in color), together with the MEG average (black). C, Histograms of the MEG null distributions, calculated as the RMSE between the resampled simulations and the MEG average source waveforms, superimposed with histograms of the RMSE for the HNN models. The histograms are visualized as kernel density plots (continuous probability density curve). The vertical dashed lines indicate the 5th and 95th percentiles of the MEG null distributions.

Discussion

The MEG data indicated a cross-sensory event-related response in the auditory cortex, peaking at ∼125 ms after the appearance of the visual stimuli. The direction of the estimated source current for this response was the same as that for the auditory N100m response, pointing downward, toward the white matter. The main shape of the visual evoked response waveform could be reproduced by an HNN model with FB-type input. In contrast, for the biphasic P50m-N100m auditory evoked response, a model with both FF and FB inputs were needed, as reported previously (Kohl et al., 2022). The experimental and modeling results are consistent with the hypothesis that cross-sensory visual input to the auditory cortex is of FB type (Schroeder and Foxe, 2002).

Characterization of cross-sensory visual evoked activation in auditory cortex

Recently, Kohl et al. (2022) presented an HNN model with a sequence of FF and FB inputs explaining several properties of auditory evoked responses in the auditory cortex. With only minor adjustments to the input timings and the overall scaling, the model could be adapted to explain the MEG source waveforms for the auditory evoked responses observed in the present study. A sequence of FF–FB (and –FF) inputs has been shown to model well also somatosensory responses in the somatosensory cortex (Jones et al., 2007). In contrast, to explain the early part of the cross-sensory visual response in the auditory cortex, we found that a model with FB input only, without a preceding FF input, was adequate. Models with only FF input failed to reproduce the prominent negative deflections in the MEG source waveforms. Thus, FB inputs appear to have an essential role in the generation of the evoked responses, as has also been concluded previously from studies using dynamic causal modeling (see Garrido et al., 2007).

The FB-type characteristics in response to visual stimuli are consistent with previous primate electrophysiological studies (Schroeder and Foxe, 2002). Multicontact electrode recordings in the macaque have shown early activity in the granular (middle) layer of auditory cortex in response to auditory stimuli, suggesting FF-type input, whereas cross-sensory visual evoked activity appeared first in supra- and infragranular layers, consistent with FB-type characteristics (Schroeder and Foxe, 2002). Similar laminar properties in the auditory cortex have also been seen in human fMRI studies (Gau et al., 2020; Chai et al., 2021; Lankinen et al., 2022). In the high-field laminar fMRI study of Lankinen et al. (2022), in which we used the same stimuli as in the noise/checkerboard experiment of the present MEG study, blood oxygenation level dependent signal depth profiles in the auditory cortex showed different curvature for auditory versus visual stimuli, consistent with the hypothesized FF- versus FB-type input patterns.

There are several possible neural pathways for the visual information to reach the auditory cortex. Anatomical tract-tracing studies in nonhuman primates have demonstrated anatomical pathways from V2 and prostriata to nonprimary auditory areas (Cappe and Barone, 2005; Smiley and Falchier, 2009; Falchier et al., 2010), but not from low-level visual areas to primary auditory cortex. Another possible pathway is through higher-level auditory association areas or polysensory regions, such as superior temporal gyrus or sulcus, which receive input from visual areas and then project that to the auditory cortex as a FB-type laminar pattern (supragranular layers) (Cappe and Barone, 2005; Smiley and Falchier, 2009). In addition, several thalamic, limbic, and cortical association areas have multisensory responses and could mediate cross-sensory inputs to auditory cortex (Smiley and Falchier, 2009). The relatively long latency of the visual response observed here is consistent with what would be expected from input from polysensory areas such as the superior temporal sulcus (Foxe and Schroeder, 2005). However, the present analyses focused on activity within auditory cortex only and thus did not attempt to identify the origin of the inputs. That type of information could be deduced, for example, from Granger causality measures between estimated source waveforms in multiple cortical areas (Milde et al., 2011; Michalareas et al., 2016; Gow et al., 2021). Another way to trace the possible pathways of the visual information to auditory cortex would be to use repetitive transcranial magnetic stimulation to temporarily suppress local brain activity, thus enabling causal inferences.

Electrophysiological studies in primates have shown different characteristics for visual and somatosensory cross-sensory inputs to the auditory cortex: FB type for visual input to auditory association and primary cortices (Lakatos et al., 2009) and somatosensory input to primary auditory cortex (Lakatos et al., 2007), but FF type for somatosensory input to auditory association cortex (Schroeder and Foxe, 2002). The role of different types of cross-sensory inputs to the auditory cortex may have important implications to theories of multisensory processing. FB-type inputs are commonly associated with modulatory influences, whereas FF-type inputs are more directly related to active sensory processing (Giard and Peronnet, 1999; Molholm et al., 2002; Schroeder and Foxe, 2005; Smiley and Falchier, 2009; Ahveninen et al., 2024). In the present study, however, we only examined responses to stimuli presented in a single sensory modality at a time, and the cortical circuit was assumed to be in a quiet state when the input arrived; thus, our model did not address modulation of ongoing or multisensory activity.

Complementary approaches to noninvasive detection of FF and FB processes

The present approach of combining MEG and cellular-level computational modeling complements other noninvasive methods for studying the organization of cortical processes in the human brain. The millisecond-scale time resolution of MEG and EEG allows the investigation of fast dynamics of brain activity, which is not attainable with hemodynamic fMRI. High-field fMRI, however, can provide laminar-level spatial resolution for making inferences about FF and FB activity (De Martino et al., 2018; Norris and Polimeni, 2019; Lawrence et al., 2019a). With certain strong assumptions about the location and extent of the spatial distribution, layer-specific source localization in MEG has also been demonstrated (Bonaiuto et al., 2018a,b). FF/FB influences can also be inferred from directed connectivity measures for MEG source estimates at specific frequency bands (Michalareas et al., 2016). Although we here used MEG to measure brain activity, the present analysis and modeling approaches would be applicable also to the more widely available EEG.

The present results also support the view that the direction of MEG source currents can be useful for inferring information about the hierarchical organization of cortical processing (Ahlfors et al., 2015). In particular, FB-type input to the supragranular layer, with excitatory synaptic connections to the distal part of the apical dendrites of pyramidal cells, is likely to be a major contributor to the downward-directed MEG source currents (Lopes da Silva, 2010; Ahlfors and Wreh, 2015). There was a general correspondence between the source direction and the type of input in the HNN model: the upward-directed source current during the auditory P50m response was associated with FF input in HNN, whereas FB inputs were needed to model the downward source currents during the auditory N100m and the visual response peaking at 125 ms. A close relationship between the direction of MEG source currents and FF- versus FB-type inputs has also been found in HNN modeling of somatosensory response in the primary somatosensory cortex (Jones et al., 2007). Furthermore, the direction of the MEG source currents in inferior occipitotemporal cortex has been found to reverse between two experimental conditions for which a cognitive neuroscience theory for visual object recognition predicted FF versus FB inputs to the area (Ahlfors et al., 2015).

Limitations of the current study

Localizing weak cross-sensory visual evoked activity in the auditory cortex is challenging because of potential long-range interference in the MEG source estimates from coinciding occipital cortex activity. In the present study, however, both the shape of the time courses and the patterns in the spatial distributions of the source estimates (Figs 5, 6) suggested that it was unlikely that the visual evoked activity in the auditory cortex was due to artifactual long-range cross talk caused by spatial spread in the source estimates. Short-range spread in the source estimates can also confound the interpretation of the source waveforms. If the true location of the visual responses were not within the auditory cortex ROI but, for example, in the opposite side of the superior temporal gyrus, the source direction could become incorrectly identified. Combining MEG with high-resolution fMRI could help to confirm the location of the activity. As the MEG signals are mainly due to source currents oriented normal to the cortex (Hämäläinen et al., 1993), the individual anatomy of a subject can affect the resulting source estimates through cortical folding and signal cancellation/superposition depending on the local geometry of the cortex (Mosher et al., 1993; Ahlfors et al., 2010). For example, a greater degree of cortical folding, and thus possible signal cancellation, in the left auditory cortex can result in bias toward stronger responses in the right hemisphere (Shaw et al., 2013), and the orientations and strengths of equivalent current dipoles can vary substantially between subjects in the auditory cortex (Edgar et al., 2003). Thus, the average source waveforms give information only about the most prominent features in a typical response. It is also possible that there was simultaneous activity in multiple auditory areas in the supratemporal plane. Most of the individual subjects’ ROIs were located directly at the primary auditory regions, at or near the at Heschl's sulcus, whereas Schroeder and Foxe (2002) recorded from an auditory association area just posterior to primary auditory region in the macaque. In nonhuman primates, FF-type patterns are typical throughout the core and belt regions of auditory cortex (Schroeder et al., 2001). Without further data, for example, intracranial recordings, it is difficult to conclusively resolve the exact locations of the sources of the observed cross-sensory MEG response.

In biophysical computational neural modeling, one has to find a balance between the complexity associated with a large number of adjustable parameters and the simplified representations of the cortical circuitry. We used neural circuit parameters of the pre-tuned HNN model for auditory evoked responses in the auditory cortex by Kohl et al. (2022) and only adjusted a small number of selected parameters, focusing on the timing of the FF and FB inputs. Given the limited SNR of the experimental source waveforms, we did not attempt to vary the neural connectivity parameters. We cannot exclude the possibility that there could be some combinations within the high-dimensional parameter space that could explain the responses with a very different circuit model than the one reported here. Useful for future studies, it has been recently demonstrated that combining simulation-based inference to HNN modeling can help in parameter estimation (Tolley et al., 2023).

We modeled only one local region (auditory cortex) receiving one-directional external inputs. Avenues for future research could include connecting other areas of interest to the network, calculating directed (effective) connectivity measures between cortical areas, and combining complementary information from MEG/EEG data with layer-specific fMRI to build a more detailed picture of the FF and FB influences.

Conclusions

The combined MEG and HNN modeling results support the hypothesis that cross-sensory visual input to the auditory cortex is of FB type. The results also illustrate how the dynamic patterns of the estimated MEG source activity can provide information about the characteristics of the input into cortical areas in terms of hierarchical organization.