Audio-Tactile and Peripersonal Space Processing Around the Trunk in Human Parietal and Temporal Cortex: An Intracranial EEG Study

Abstract

Interactions with the environment happen within one’s peripersonal space (PPS)—the space surrounding the body. Studies in monkeys and humans have highlighted a multisensory distributed cortical network representing the PPS. However, knowledge about the temporal dynamics of PPS processing around the trunk is lacking. Here, we recorded intracranial electroencephalography (iEEG) in humans while administering tactile stimulation (T), approaching auditory stimuli (A), and the 2 combined (AT). To map PPS, tactile stimulation was delivered when the sound was far, intermediate, or close to the body. The 19% of the electrodes showed AT multisensory integration. Among those, 30% showed a PPS effect, a modulation of the response as a function of the distance between the sound and body. AT multisensory integration and PPS effects had similar spatiotemporal characteristics, with an early response (~50 ms) in the insular cortex, and later responses (~200 ms) in precentral and postcentral gyri. Superior temporal cortex showed a different response pattern with AT multisensory integration at ~100 ms without a PPS effect. These results, represent the first iEEG delineation of PPS processing in humans and show that PPS and multisensory integration happen at similar neural sites and time periods, suggesting that PPS representation is based on a spatial modulation of multisensory integration.

Introduction

The space immediately adjacent to and surrounding the body—defined as peripersonal space (PPS; di Pellegrino et al., 1997; Rizzolatti et al. 1981, 1997)—is particularly relevant for behavior, as it is where physical interactions with the environment occur (Graziano and Cooke 2006; Làdavas and Serino 2008). The ecological significance of the PPS is evidenced in that the primate brain has developed a frontoparietal network encoding preferentially multisensory stimuli occurring near to (as opposed to far from) the body. That is, neurons located in monkey posterior parietal cortex (i.e., intraparietal sulcus [IPS]) (Duhamel et al., 1997, 1998), parietal area 7b (Leinonen and Nyman 1979a, 1979b; Fogassi et al. 1996; Graziano et al. 1997), and ventral premotor cortex (vPM) (Fogassi et al. 1996; Graziano et al. 1997) have been reported to respond to tactile stimuli applied to different body parts. These regions also respond to visual (Schlack et al. 2005) or auditory cues (Graziano et al. 1999; Schlack et al. 2005) if they occur in a similar spatial position as the tactile stimuli.

A homologous PPS neural network is postulated to exist in humans, supported by numerous psychophysical (Spence et al. 2004; Salomon et al. 2017) and neuropsychological (Farnè and Làdavas 2000; Maravita and Iriki 2004) studies demonstrating enhanced processing of tactile stimulation when a task-irrelevant visual or auditory object is present near versus far from the body. These studies rely on the congruent presentation of multisensory stimuli in the environment (Canzoneri et al. 2012; Serino et al. 2015) are body part centered (hand: (Canzoneri et al. 2012); face: (Teneggi et al. 2013); trunk: (Galli et al. 2015; Noel, Grivaz et al. 2015; Noel, Pfeiffer et al. 2015; Noel et al. 2018)). The existence of a homologous PPS neural network in humans is further supported by fMRI studies, which have demonstrated a close association between the areas encoding for PPS in nonhuman primates and humans (Bremmer et al. 2001; Makin et al. 2009; Gentile et al. 2011; Brozzoli et al. 2011, 2012; Ferri et al. 2016; Grivaz et al. 2017). In addition to the above mentioned PPS areas described in monkeys, human fMRI has equally revealed primary somatosensory cortex (S1), parietal operculum (Tyll et al. 2013), insula (Schaefer et al. 2012), cingulate cortex (Holt et al. 2014), and the lateral occipital cortex (Gentile et al. 2013) as brain regions encoding PPS (for a review, see Grivaz et al. 2017).

The characterization of the areas encoding PPS in humans, however, has quasiexclusively mapped the perihand representation (Makin et al. 2007; Gentile et al. 2011; Brozzoli et al. 2011, 2012), with only a few studies investigating the periface space (see Bremmer et al. 2001; Sereno and Huang 2006; Holt et al. 2014 for exceptions) and even fewer on the peritrunk space (see Huang et al. 2012 for an exception). Moreover, while the encoding of PPS is primarily taken to be subsumed by multisensory networks, most of the evidence on PPS-related neural response is based on the finding that PPS neurons or regions respond both to tactile and visual (or auditory) stimulation. Yet, only one single electrophysiological study (Avillac et al. 2007) has demonstrated clear multisensory integration, that is, a nonlinear integration of stimuli from different modalities, leading to a multisensory supra-additivity or subadditivity (see Gentile et al. 2011 for fMRI evidence). Surprisingly, this nonlinear multisensory integration for the PPS effect (around the trunk) has not been shown in electrophysiological studies in humans. Finally, evidence on the PPS system in humans mainly comes from fMRI studies. Thus, the existing literature has left several open questions such as the characterization of the spatiotemporal brain dynamics of the PPS processing (on the trunk), to what extent regions showing a PPS effect are also multisensory, and whether multisensory integration and PPS processing occur at similar time periods. Answering these questions would provide insight on whether the spatial modulation of multisensory processing characterizing PPS representation occurs in parallel with multisensory integration or follows it hierarchically.

Here, we address the issues raised above, by recording intracranial electrical brain activity in humans, via surgically implanted electrodes in 6 patients with pharmacoresistant epilepsy. By combining high temporal and spatial resolution, intracranial recordings overcome some of the limitations of the techniques used in previous PPS experiments. Patients received tactile stimuli on the trunk while a task-irrelevant auditory stimulus approached the body. Because of the novelty of the study (and therefore limited evidence to generate hypothesis-driven analysis), and to avoid biases induced by prior assumptions, we used a data-driven methodology. To test multisensory PPS processing, we adopted a 2-step analysis approach, in which we first identified electrodes demonstrating multisensory integration—defined as showing nonlinear sensory summation of response to multisensory stimuli (i.e., A + T vs. AT; Giard and Peronnet 1999, for a review, see Besle et al. 2008)—and then, within the resulting set of multisensory electrodes, we search for those showing a neural response that is modulated by the distance between the location of tactile and auditory stimulation (see Quinn et al. 2014 for a similar analytic approach in the visuotactile domain). By comparing the sites and the timing of multisensory integration and PPS processing, we investigated whether multisensory brain areas also encode for PPS. As additional analysis, we investigated whether other brain regions (not showing multisensory integration) also encode for PPS.

Methods

Participants

Intracranial EEG data (i.e., local field potentials; LFPs) were recorded from 6 epileptic patients (3 females, 2 left-handed, mean age: 33 ± 4.8 [mean ± standard error of the mean (s.e.m.)], see Supplementary Table 1 for age, gender, handedness, and epilepsy focus of each patient) who were either implanted stereotactically with depth electrodes and/or grid electrodes were placed on the cortical surface (P-1, P-2, and P-5) for clinical purposes (i.e., presurgical evaluation in pharmacoresistant epilepsy, see Table 1 for details). Written informed consent was obtained from all patients to take part in the procedures, which were approved by the local ethics committee.

Table 1.

Age, gender, handedness, epilepsy focus, comorbidities, number of implanted electrodes, and medications (taken at the time of the recording) of the patients.

Patient	Age	Gender	Handedness	Epilepsy focus	Comorbidity	Medication	Number of implanted electrodes	Number of electrodes with multisensory integration	Number of PPS electrodes
P-1	33	Male	Left	Right postcentral epilepsy	Migraine, anxiety	No antiepileptic drugs	88 (Grid 64e + 3×8e depth)	6	2 (Right hemisphere)
P-2	29	Female	Left	Left parietal opercular epilepsy	None	Trileptal; Vimpat Urbanyl	108 (Grid 64e + 5×8e depth + 1×4 strip)	2	0
P-3	55	Female	Right	Right temporal epilepsy	None	Trileptal	48 (6×8e depth)	3	2 (Right hemisphere)
P-4	33	Male	Right	Left temporal epilepsy	Migraine	Carbamazepine	76 (8×8e depth + 2×6 strip)	2	0
P-5	33	Male	Right	Right postcentral epilepsy	None	Topiramate; Benzodiazepines	60 (Grid 32e + 3×8e depth + 1×4 strip)	6	1 (Right hemisphere)
P-6	19	Female	Right	Right temporal epilepsy	Mild anorexia (not confirmed by DSM-IV), depression	Valproate acid Lamotrigine	120 (15×8e depth)	1	1 (Right hemisphere)

Material and Apparatus

Tactile and auditory stimuli were administered during the task (see Procedures). Tactile stimulations were applied to the patient’s chest, on the upper part of the sternum, by activation of a vibrotactile motor (Precision MicroDrives, shaftless vibration motors, model 312-101, 3 V, 60 mA, 9000 rpm, 150 Hz, 5 g, 113 mm² surface area, maximal rotation speed reached in 50 ms). Tactile stimulation lasted 100 ms and was controlled via a purpose-made microcontroller (Arduino^TM, http://arduino.cc, refresh rate 10 kHz) and driven by in-house experimental software (ExpyVR, http://lnco.epfl.ch/expyvr, direct serial port communication with microcontroller). The auditory stimulus consisted of a white noise sound, which was approaching from the front, and centered on the patient’s body, presented via insert earphones (model ER-4P; Etymotic Research). To give the impression that the sound was approaching from the front, sounds were prerecorded from 2 arrays of 8 speakers (2 m length in total) and head model binaural microphones (Omni Binaural Microphone, http://3diosound.com, see Serino et al. 2015, 2017 for detail regarding the external auditory setup).

Procedures

During the experiment, the patient was comfortably lying in bed, with the upper part of the body reclined forming approximately a 135° angle with the rest of their body. The patient was asked to keep their eyes closed for the duration of the experiment, and they were equally instructed to be attentive to the approaching sound and tactile vibrations. No overt task was requested from the patients.

The experiment consisted in 3 different types of trials: 1) Auditory trials (unisensory audio; A), which consisted of an approaching sound, with a maximal simulated distance from the body of 2 m (and lasted a total of 3 s; speed: 0.66 m/s); 2) vibrotactile trials (unisensory tactile; T), which consisted of 3 successive stimulations, administrated 500, 1500, and 2500 m after the onset of the trial; and 3) audio-tactile trials (multisensory; AT), in which the tactile stimulation were administrated 500 ms (far distance, equivalent to 1.7 m from the body), 1500 ms (middle distance, equivalent to 1 m), and 2500 m (close distance, equivalent to 0.3 m) after the initiation of the trial and auditory stimulus onset. To prevent anticipation effects on the vibrotactile stimulation, a jitter of 0–200 ms (steps of 50 ms) was used for each delay of stimulation. This small temporal jitter allowed us to induce some variability in the timing of tactile stimulation, while not altering the spatial position of the sound when tactile stimulation was administered. A total of 85 trials for each condition were presented, in a randomized manner. The intertrial interval was shuffled randomly between 1.4, 1.7, and 1.9 s. In total, the experiment lasted approximately 20 min.

Electrodes Implantation, Intracranial EEG recordings, and Preprocessing

In total, 500 electrodes (depth and grid) were implanted in 6 patients, covering diverse cortical and subcortical areas including the postcentral and precentral gyrus, insula, temporal and parietal operculum, amygdala, hippocampus, frontal, and temporal cortex (see Fig. 1 for the location of all recording sites). All implantation sites were determined purely based on clinical requirements. Three different types of electrodes were used for the recording: standard electrodes (contact size: 2.4 mm, interelectrode spacing: 10 mm) “short spacing” electrodes (contact size: 1.32 mm, interelectrode spacing: 2.2 mm), and “micro” electrodes (contact size: 1.6 mm, interelectrode spacing: 5.0 mm).

Figure 1.

Locations of all recording sites in 3D MNI space. MNI coordinates of electrodes from all 6 patients (500 electrodes in total) plotted on the Colin27 MRI template (on selected sagittal and axial planes). Note that locations are in 3D MNI space, and not located on the surface of MRI slice shown (thus, recording sites behind the depicted MRI slice are marked with faded color). In black, the implanted electrodes not showing a response (vs. baseline, cluster-corrected) to stimuli, in orange, the electrodes showing a response to audio stimuli only, in yellow, the electrodes showing a response to tactile stimuli only, in red, the electrodes showing a response to audio-tactile stimuli, and in white the electrodes showing a response to at least 2 conditions.

For each patient, intracranial EEG signals were simultaneously recorded across all sites (Micromed System PLUS, Micromed, Mogliano Veneto, Italy) with a sampling rate of 2048 Hz, and an online high-pass filtered at 0.02 Hz. The external reference electrode was located at position Cz (i.e., vertex). Continuous intracranial EEG data were down-sampled to 512 Hz for analyses. Signals were filtered with a band-pass filter between 1 and 40 Hz. Initial peristimulus EEG epochs were generated (800 ms pretrial onset—auditory stimulus in the case of A and AT trials—to 3000 ms post-trial onset), and each epoch was centered to zero. Data were further re-epoched from 100 ms prestimulus onset to 300 ms poststimulus onset. Baseline correction on the 100 ms prestimulus onset was applied, only on the electrodes that were already identified as responsive versus baseline (see Statistical Analysis for details).

In each patient, electrodes and trials showing excessive noise (i.e., >6 interquartile range) were excluded, and thus 480 clean electrodes out of 500 implanted electrodes were used for further analysis. On stripes and depth electrodes, bipolar signals were computed by subtracting intracranial EEG signals from 2 adjacent electrodes (e.g., A1–A2, A2–A3) from within each electrode shaft, to eliminate the influence of the common external reference and remote sources (Lachaux et al. 2003). In the case of grid electrodes, as bipolar referencing is not suitable (Lachaux et al. 2012, but see Mercier et al. 2017 for other procedures), we computed the average of the grid as a reference (i.e., local reference). After preprocessing the number of trials for the tactile conditions was 80.3 ± 1.2 (mean ± sem), 79.7 ± 1.7 (mean ± sem) for the auditory conditions, and 77.5 ± 1.5 (mean ± sem) for the audio-tactile condition. The number of trials retained per condition was not significantly different (F[2,15] = 0.97; P = 0.39).

To compute the Montreal Neurological Institute (MNI) coordinates for each electrode, a postimplant computed tomography (CT) image was coregistered to the normalized preoperative magnetic resonance imaging (MRI) using Cartool Software (Brunet et al. 2011). The midpoint between 2 depth electrodes was considered as the location of the corresponding bipolar derivation, and for the grid electrodes, the exact position was used. Then, locations of the electrodes were visualized on the Colin27 MRI brain template using the BrainStorm toolbox (Tadel et al. 2011). The anatomical description was assessed using Talairach coordinates (http://talairach.org/; Lancaster et al. 1997, 2000), with a 1 mm cube around the coordinates defined above.

Statistical Analysis

With the current experiment, we wanted to assess whether (nonlinear) multisensory integration is mandatory for a PPS effect. Therefore, we conducted 2 distinct analyses. The first analysis was conducted based on the classic approach to study PPS in nonhuman primates, in which PPS is defined as a multisensory modulation of tactile processing due to an external sensory stimulation, as a function of the distance of these stimuli from the body in space (see Introduction). Therefore, to identify PPS electrodes, we used a 2-step statistical approach. First, we first identified electrodes responding to multisensory AT stimuli (vs. baseline), and among those electrodes, we investigated which responded in a manner suggesting multisensory integration (AT vs. A + T; see below). Second, among the electrodes showing a multisensory integration effect, we characterized those that had a PPS effect—a multisensory response that is dependent on the distance of exteroceptive signals (e.g., auditory information) to the body (see below for more details; a similar approach has been previously used in iEEG studies, e.g., Quinn et al. 2014). A second analysis aimed at identifying whether electrodes not showing multisensory integration did show a PPS effect. First, we first identified electrodes responding to the AT stimuli (vs. baseline). Second, among the electrodes showing a response, we identified those that had a PPS effect. For each of the above mentioned steps, statistical significance within each electrode was assessed through (temporal) cluster-based permutation statistics (Maris and Oostenveld 2007) as implemented in the Fieldtrip toolbox (Oostenveld et al. 2011). The advantage of this test is that differences between conditions can be identified without prior assumptions about the temporal distribution of effects. Therefore, it is a data-driven approach. The cluster-level statistic was calculated as the maximum sum (maxsum) of the t-values within the cluster. Statistical significance at the cluster level was determined by computing a Monte Carlo estimate of the permutation distribution of cluster statistics, using 5000 resampling of the original data, yielding a distribution of cluster-level statistics under the null hypothesis that any differences between conditions are due to chance. Within a single electrode, a cluster was taken to be significant if it fell outside the 95% confidence interval of the permutation distribution for that electrode. The determination of significant temporal clusters was performed independently for each electrode. This method controlled for false alarms within an electrode across time points.

Active (Unisensory and Multisensory) Electrodes

To evaluate and select active electrodes for latter between-conditions testing, we applied the cluster-based, nonparametric statistical procedure (see above for details). Electrodes demonstrating a significant response (poststimulus period 0–300 ms) relative to a baseline (−100 to 0 ms) to A, T, and/or AT trials were considered as active electrodes (no baseline correction was applied for this analysis).

Audio-Tactile Multisensory Integration

Among the active AT electrodes, we first selected those showing a response revealing significant multisensory integration (i.e., demonstrating either a supra-additivity or subadditivity effect, A + T vs. AT), and then among the electrodes evidencing multisensory integration we investigate which had a “PPS effect”, that is, a nonlinear modulation of tactile response depending on the distance of the sounds from the body. To identify both the “multisensory” and “PPS” electrodes a modified version of the cluster-based, nonparametric statistical procedure outlined by Maris and Oostenveld (2007) was applied. To assess statistically a multisensory integration effect, we applied a cluster-based permutation statistic individually to each electrode (which showed to be active in comparison to baseline), with the contrast AT versus A + T.

PPS Effect

To investigate the PPS effect, we conducted 2 distinct analysis (see statistical analysis above). One the one hand, we first selected the electrodes showing an AT multisensory integration (among the responding electrodes). Then among those electrodes, we selected those showing a PPS effect. To identify a PPS effect (i.e., a modulation according to the distance of the sound from the body) the following procedure was applied: 1) we first computed the difference AT-A (for the electrodes showing multisensory integration), providing us with the LFPs for the PPS in response to the tactile stimulus and 2) to assess a statistically PPS effect we applied a cluster-based permutation statistic individually on the PPS–LFPs, and we computed a one-way ANOVA (with the contrast Far vs. Middle vs. Close). These analysis steps were applied to each electrode independently. This 2-step approach was chosen for several reasons. First, it provides the possibility to compare the PPS effect with the condition in which only the T stimulus was administered (our control condition), and therefore assess that the PPS processing effect is different from the T stimulation effect (i.e., which may indicate habituation/expectation effect). Second, this approach allows us to control for an eventual effect due to a change in sound intensity as a function of the distance from the body. That is, with this approach we can certify that putative PPS effects are not purely due to a change in sound intensity (at the stimulus is closer to the body), and that it is different from simple tactile habituation.

On the other hand, to provide a characterization of the PPS effect as detailed as possible, we conducted one additional analysis in which we assessed whether any PPS effect might be present also in electrodes that do not show multisensory integration. For this complementary analysis, we applied the cluster-based, nonparametric statistical procedure (P < 0.01, see above for details). Electrodes demonstrating a significant response (poststimulus period 0–300 ms) relative to a baseline (−100 to 0 ms) during the poststimuli onset to AT trials were considered as active electrodes. Among those electrodes, we assessed which electrodes also showed a PPS effect. That is, to assess a statistically PPS effect we applied a cluster-based permutation statistic (P < 0.01) individually on the PPS-LFPs, and we computed a one-way ANOVA (with the contrast Far vs. Middle vs. Close) for each electrode/condition.

Control Analysis

To investigate if any anticipation/habituation effect had occurred and could account for the PPS effect, we computed a similar analysis as for the “PPS effect” on the condition in which only the tactile stimulus was presented. Electrodes demonstrating both multisensory integration and a PPS effect but no tactile habituation effect (or at least with different response pattern) can safely be considered to be electrodes evidencing a multisensory effect that is space-dependent, that is, putatively recording activity from “PPS brain areas.”

Results

Active Unisensory and Multisensory Electrodes

We first investigated which electrodes showed a significant response versus baseline period, across the 480 implanted electrodes (Fig. 1, 480 out of 500 electrodes were included after preprocessing), from all 6 patients, in any of the tested conditions. Our data show that 75 electrodes (~16% out of 480) were responsive to T stimulation, 61 electrodes (13% out of 480) were responsive to A stimulation, and 99 (~21% out of 480) were responsive to AT stimulation (for a summary of activation see Fig. 1 and Fig. S2, that show which electrode is responding and to which condition). It should be noted that some of the electrodes responding to one condition can also be responsive to one (or more) of the other conditions.

Location of Unisensory and Multisensory Electrodes

Among the responsive electrodes, we assessed the distribution of unisensory (T and A) and multisensory (AT) responses. The electrodes responding to the T stimulus were predominantly located in the postcentral gyrus (PCG), but also in the insula and inferior frontal gyrus (IFG). The electrodes responding to the A stimulus were predominantly located in the superior and inferior temporal gyrus (STG and ITG). The electrodes responding to the AT stimuli were in the PCG, the precentral gyrus (pre-CG), the midtemporal gyrus (MTG) and STG, IFG, insula, and parahippocampal gyrus (PHG) (Fig. 1 and Table 2 for a summary). Next, to give an overview of the distribution of the electrodes location (and quantify the proportion of electrodes), we grouped them into larger brain regions. 42% (213 electrodes) of the electrodes were implanted in frontal areas, among those electrodes ~11% (23 electrodes) showed a response to A, ~11% (24 electrodes) to T and ~12% (26 electrodes) to AT, the rest of the electrodes were not responsive electrodes. 18% (86 electrodes) of the electrodes were implanted in the temporal areas, among those electrodes ~17% (15 electrodes) showed a response to A, ~8% (7 electrodes) to T and ~23% (20 electrodes) to AT. The 12% (56 electrodes) of the total number of electrodes in the current study were implanted in the parietal areas, among those electrodes ~9% (5 electrodes) showed a response to A, ~43% (24 electrodes) to T and ~46% (26 electrodes) to AT. Finally, 2% (13 electrodes) of all electrodes implanted in the current study were implanted in the insula, among those electrodes ~23% (3 electrodes) showed a response to A, ~31% (4 electrodes) to T and ~46% (6 electrodes) to AT (for additional details see Fig. 1, and Fig. S1).

Table 2.

Number of significant electrodes showing AT multisensory integration and PPS effect, sorted by brain regions (P < 0.05, cluster-corrected).

Region	Number of AT multisensory electrodes	Timing of AT multisensory integration (mean ± sem)
Postcentral gyrus	7	228 ± 18 ms to 279 ± 6 ms
Superior temporal gyrus	3	100 ± 40 ms to 210 ± 57 ms
Parahippocampal gyrus	3	112 ± 39 ms to 239 ± 25 ms
Inferior frontal gyrus	1	245–291 ms
Precentral gyrus	1	222–299 ms
Insula	1	63–296 ms
Total	16

Location of the Electrodes Showing Audio-Tactile Multisensory Integration

Among the responsive AT multisensory electrodes (99), 20 electrodes (~19% of active electrodes) showed significant multisensory integration (i.e., AT vs. A + T). This multisensory integration occurred principally within the PCG (7 electrodes, 35% of the AT multisensory electrodes), but also within the STG (3 electrodes, 15% of the AT multisensory electrodes), within the PHG (3 electrodes, 15% of the AT multisensory electrodes), the pre-CG (1 electrode, 5% of the AT electrodes), IFG (1 electrode, 5% of the AT electrodes), and the insula (1 electrode, 5% of the AT electrodes). Four electrodes were situated in the white matter (Fig. 2 for more details). The closest cortical brain regions which could have generated the responses at these 4 electrodes are: insula (1 electrode), STG (1 electrode), and IFG (2 electrodes).

Figure 2.

Locations of electrodes showing an AT multisensory integration and peripersonal space (PPS) effect, in 3D MNI space. MNI coordinates of electrodes from all 6 patients, electrodes showing specifically a significant multisensory integration profile are highlighted in black (20 electrodes, see Table 1 for the position), electrodes showing both an AT multisensory integration and PPS effect are highlighted in white (6 electrodes, see Table 2 for electrodes positions). Note that locations are in 3D MNI space, and not located on the surface of selected MRI slice (thus, recording sites behind the depicted MRI slice are marked with faded color).

Timing of Audio-Tactile Multisensory Integration

On average, across the 20 electrodes showing AT multisensory integration, the effect occurred from 151 ± 18 ms (mean ± sem) to 244 ± 15 ms (mean ± sem) poststimulus onset (i.e., tactile). Within the insula the effect occurred from 63 to 296 ms poststimulus onset, and a supra-additive nonlinear (AT > A + T) neural response was observed. Within the STG the response occurred from 100 ± 40 ms (mean ± sem) to 210 ± 57 ms (mean ± sem), and supra-additive nonlinear (AT > A + T; neural response interactions between multisensory and the sum of the constituent unisensory stimuli) was observed. Within the PHG the effect occurred from 112 ± 39 ms (mean ± sem) to 239 ± 25 ms (mean ± sem) and a supra-additive nonlinear (AT > A + T) neural response was observed between the multisensory response and the sum of the constituent unisensory stimuli. Within the precentral and postcentral gyri and IFG the effect occurred from 181 ± 25 ms (mean ± sem) to 254 ± 22 ms (mean ± sem) (see Table 2 for a summary of the timing). In the precentral and postcentral gyri multisensory integration occurred as a supra-additive neural response (AT > A + T) (Fig. 3, for an exemplary LFP for AT multisensory integration). In the IFG the multisensory integration occurred as a sub-additive nonlinear neural response (AT < A + T).

Figure 3.

Exemplar LFP for AT multisensory integration. The left panel shows the position of the electrode (in white), on a selected plane. The electrode was located in the postcentral gyrus (PCG). The middle panel shows the LFPs responses for 3 conditions: A (orange), T (yellow) and AT (red). The right panel shows the LFPs for the AT (red) and the SUM of A + T (green), with a multisensory integration at 148–253 ms after the stimulus onset. The lines indicate the average over trials; the shaded areas indicate the 95% CI, and the black lines indicate the time period with a significant AT multisensory integration (P-value <0.05, cluster-corrected).

Location of the Electrodes Showing a PPS Effect

Among the 20 electrodes showing AT multisensory integration, 6 electrodes (30% of AT multisensory electrodes, and 6% of active AT electrodes) showed a PPS effect. From the electrodes characterized as coding for PPS, 3 electrodes were located in the PHG (50% of the PPS electrodes), 2 electrodes were found in the PCG (34% of the PPS electrodes), and 1 electrode was located in the insula (17% of the PPS electrodes; see Fig 2, Table 3 for details). Importantly, for all the locations where a PPS effect was observed, the response profile differed as a function of the distance from the trunk in such a way that PPS-dependent multisensory integration did not linearly decrease with distance. In fact, this profile was more similar to a step-function (see Fig. 4, right panel).

Table 3.

Timing of the significant effect of AT multisensory integration and PPS effect, separated for brain regions.

Region	Number of PPS/Multisensory electrodes	Timing of PPS effect (mean ± sem)
Postcentral gyrus	2/7	44 ± 3 ms to 92 ± 12 ms and 202 ± 18 ms to 265 ± 7 ms
Superior temporal gyrus	0/3	n.a.
Parahippocampal gyrus	3/3	193 ± 23 ms to 299 ± 1 ms
Inferior frontal gyrus	0/1	n.a.
Precentral gyrus	0/1	n.a.
Insula	1/1	39–129 ms and 141–296 ms
	6/16

Figure 4.

Exemplar LFP for AT multisensory integration and PPS effect. The top left panel shows the position of the electrode, on a selected plane. The electrode was located in the parahippocampal gyrus (PHG). The top right panel shows the LFPs responses for 3 conditions: A (orange), T (yellow) and AT (red). The bottom left panel shows the LFPs for the AT (red) and the SUM of A + T (green), with a multisensory integration at 52–170 ms and 193–270 ms after stimulus onset. The bottom right panel shows the LFPs for the PPS effect at 151–298 ms after stimulus onset. The bottom right. The lines indicate the average over trials; the shaded areas indicate the 95% CI, and the black lines indicate the time period with a significant PPS effect (P-value <0.05, cluster-corrected).

In addition to the electrodes showing AT multisensory integration and a PPS effect (6 PPS electrodes among 20 AT multisensory electrodes), we also observed 4 electrodes showing a PPS effect, without showing a nonlinear AT multisensory integration effect (4 electrodes among 48). Among those electrodes, 3 were located in the IFG and 1 was located in the STG.

Timing of the PPS Effect

On average, across the 6 electrodes, the PPS effect occurred from 139 ± 26 ms (mean ± sem) to 226 ± 31 ms (mean ± sem) poststimulus onset. Within the insula the effect occurred from 39 to 129 ms and from 141 to 296 ms. Within the PCG the PPS effect occurred from 44 ± 3 ms (mean ± sem) to 92 ± 12 ms (mean ± sem) and from 202 ± 18 ms (mean ± sem) to 265 ± 7 ms (mean ± sem). Within the PHG the PPS effect occurred from 193 ± 23 ms (mean ± sem) to 299 ± 1 ms (mean ± sem) (see Table 3 for a summary of the timing).

Tactile Habituation (Control Analysis)

To ascertain that the above-described PPS effect was not merely due to tactile habituation, we investigated if a “time-dependent” effect (i.e., a significant difference on unisensory tactile responses as a function of the delay of tactile stimulation) was observed with the T stimulus alone. Among the 20 electrodes showing AT multisensory integration, 2 electrodes showed a possible anticipation/habituation effect in the tactile condition. These effects occurred within the PHG. Here, 2 distinct time periods showed a tactile habituation effect, on average the first time period occurred from 143 ± 16 ms (mean ± sem) to 213 ± 35 ms (mean ± sem) poststimuli onset, and the second time period occurred from 188 to 281 ms (only on 1 electrode—Fig. S2). No other electrode showed a modulation in response to the T stimulus as a function of time.

These tactile habituation effects occurred during (at least partially) different time-points than the PPS effects, and the modulation was “linearly” dependent on the temporal order of the stimuli presentation. That is, the effect on the T condition showed a modulation of the LFPs for the first versus second administrated tactile stimulus (over both time significant time periods), and for the second versus third (over the second significant time period) administrated tactile stimulus. This modulation pattern was different from what was observed for the PPS effect, where we see a difference, for instance, between the far and the middle/close distance (conceptually similar to a step-function, see Fig. S2).

Discussion

Intracranial EEG human recordings were performed in 6 patients suffering from pharmacoresistant epilepsy, who were presented with vibrotactile stimulation and concurrent sounds approaching their trunk, in an effort to unravel the neurophysiological substrates of audio-tactile PPS surrounding the trunk. Crucially, and overlooked in most previous studies, PPS is defined as a multisensory spatial extent (e.g., Graziano et al. 1997; Serino et al. 2015; Ferri et al., 2016). Therefore, here we first identified brain responses that exhibited AT multisensory integration, indexed by nonlinearity compared with the sum of the unisensory constituents of the multisensory stimuli (i.e., A + T vs. AT). Subsequently, within this subset of multisensory integration responses, we identified those electrodes that showed a modulation of the response as a function of distance from the body—that is, a PPS response. Broadly, results demonstrated that 99 (21%) of the 500 electrodes, implanted for clinical purposes, were responsive to the multisensory AT stimulation, with 75 electrodes (21%) and 65 (16%) being only responsive to T or the A unisensory stimulation, respectively. In addition, 19% (20 electrodes from the 99 electrodes showing an AT multisensory response) of these active electrodes specifically exhibited multisensory integration (defined as a nonlinear summation of the response to AT stimuli, differently from the sum of A + T stimuli). These were located predominantly in the PCG, STG, insula and PHG. The AT multisensory response occurred, respectively, on average from 181, 100, 63, and 112 ms poststimuli onset. Among these 20 AT multisensory integration electrodes, 30% (6 electrodes) also showed a PPS effect, that is, a nonlinear modulation of the response to tactile stimuli as a function of the distance of the sounds from the body. Crucially, the spatial modulation of the responses did not linearly decrease with distance from the body (as it may be the case for responses related to tactile anticipation), but differentiated between the far versus middle and/or near positions, suggestive of the presence of an electrophysiologically defined boundary between PPS and the far space (between 30 and 100 cm from the body), in agreement with behavioral data in humans (Noel, Grivaz et al. 2015; Noel, Pfeiffer et al. 2015; Serino et al. 2015; Pfeiffer et al. 2018). The brain regions demonstrating a multisensory PPS effect were located most prominently in the PCG, but also within other cortical structures, namely, the insula and the PHG (see Grivaz et al. 2017 for independent corroborative evidence). The PPS effect in those brain regions occurred, respectively, on average at 44, 39, and 193 ms poststimuli onset. Hence, we present neurophysiological evidence, for the first time in humans, for the encoding of audio-tactile multisensory PPS in an extended cortical network. These human findings corroborate and extend those described in nonhuman primate studies, which demonstrated PPS processing mostly tested around the face and the hand in the parietal lobe (Duhamel et al. 1997, 1998; Graziano et al. 1997, 1999; Schlack et al. 2005) and fMRI studies in humans showing specific processing for stimuli presented close to the hand and face (parietal lobe, primary somatosensory cortex, and insula; Grivaz et al. 2017). The present results describe neural mechanisms of the human trunk PPS that has been suggested to also be of particular importance for bodily self-consciousness (Blanke et al. 2015; Noel, Grivaz et al. 2015; Noel, Pfeiffer et al. 2015; Serino et al. 2015) and also reveal evidence for trunk PPS coding within the limbic system (i.e., PHG). In the following, we discuss our results with respect to multisensory integration, PPS, and the conjunction between the two processes, in terms of brain location and timing of the effects.

Location of Audio-Tactile Multisensory Integration

Our results corroborate and extend previous literature, by showing multisensory neural processing in response to audio-tactile multisensory pairings (vs. visuotactile in Avillac et al. 2007), in humans (vs. monkeys) and in LFPs recordings (vs. single units). Research regarding multisensory integration—in particular, electrophysiological studies in nonprimary sensory areas—have focused mostly on audio–visual and visuotactile integration. Much less is known about AT multisensory integration. Classically, multisensory integration has been considered to occur in higher-order temporal, parietal and occipital regions (Jones and Powell 1970, more recently Quinn et al. 2014). However, this view has been challenged by studies, in both monkeys and humans alike, providing evidence for early multisensory neural modulations (Schroeder and Foxe 2002; Lakatos et al. 2007) occurring in regions traditionally considered purely unisensory. Many of these modulatory effects in primary sensory areas have been demonstrated via somatosensory or visual effects in the primary auditory areas (Schroeder and Foxe 2002; Lakatos et al. 2007; Besle et al. 2008). However, both fMRI and EEG experiments have highlighted the posterior superior temporal plane (Foxe et al. 2002; Murray et al. 2005), and not primary sensory areas, as brain regions implicated in veritable, overt or suprathreshold, audio-tactile multisensory integration. Other studies localized AT multisensory integration in the posterior parietal cortex, the somatosensory area SII and insula, rather than auditory association cortices (Lütkenhöner et al. 2002; Gobbelé et al. 2003; Renier et al. 2009). Thus, even though clinical purposes limited brain coverage in our study, the location of the electrodes showing stronger AT multisensory integration responses in our data corroborate and extend previous literature. That is, most of the electrodes demonstrating multisensory integration in our study were located in PCG, but also in the STG and insula. Despite, most of the multisensory electrodes being located in the PCG and thus anterior to the VIP region studied in monkeys (Avillac et al. 2007), this difference might be partially due to a different location of VIP in humans (more anterior and ventral) than in monkeys (Sereno and Huang 2014).

The Timing of Audio-Tactile Multisensory Integration

A key advantage of recording intracranial LFPs as opposed to the BOLD response is that the former allows for indexing and characterizing time-resolved computations, in combination with high spatial resolution. In monkeys, AT multisensory integration has been reported to occur at early latencies (<100 ms) (Schroeder and Foxe 2002; Schroeder et al. 2001, 2003). In humans, AT multisensory integration has been reported to occur both at early latencies (Murray et al., 2005) and at later latencies >100 ms (Lütkenhöner et al. 2002; Gobbelé et al. 2003). Late multisensory integration is supported by Quinn et al. (2014), who reported multisensory responses to visuotactile stimulation from iEEG recordings at latencies ranging from 145 to 313 ms poststimulus onset. Multisensory integration at later latencies is also consistent with Valdés-Conroy et al. (2014), who performed an ERP study as a function of visual depth and reported a significant amplitude modulation in evoked responses within 150–200 ms from stimulus onset. At first, our results show that the latency of the AT multisensory integration responses occurs on average between 151 and 244 ms, which is in agreement with later processes of multisensory integration. However, it is important to note that our results show 3 distinct temporal response patterns. That is, we also found an early AT multisensory integration occurring at ~60 ms after the stimulus onset, within the insula and PHG, and a later effect occurring at ~100 ms within the STG, followed by the even later PCG and IFG responses, suggesting that AT multisensory integration occurs over 2 (at least partially) distinct time periods, over more ventral (earlier effects) and more dorsal (later effect) regions (Renier et al. 2009).

Location of PPS Effect

The present results extend the findings of a recent fMRI meta-analysis (Grivaz et al. 2017) study in humans, that aimed at identifying areas that consistently coded for PPS. This later study identified a portion of the PCG, including regions of area 1, 2, and 3b, as well as areas 5 and 40, as crucial PPS areas. Similarly, the present (and other) studies found PPS-like responses in the insula (Cappe et al. 2012; Schaefer et al. 2012) in humans. Thus, locating the bulk of multisensory PPS neural responses to the PCG (present study) is corroborated by recent functional neuroimaging literature on PPS processing.

We also found multisensory PPS responses in the insula and PHG. Regarding the insula, although it is a known area of multisensory convergence (Bushara et al. 2001; Calvert et al. 2001; Rodgers et al. 2008; Renier et al. 2009), its direct electrophysiological implication in the multisensory mapping of PPS has not been previously established. However, the insula has been linked to changes in body ownership, self-identification, and self-location after multisensory illusions, such as the rubber hand illusion (Tsakiris et al. 2007; Brozzoli et al. 2012; Blefari et al. 2017; Grivaz et al. 2017), the enfacement illusion (Apps et al. 2015), and the full body illusion. It is therefore considered a key area for the processing of multisensory cues underlying bodily self-consciousness (Blanke 2012; Seth 2013; Salomon et al., 2016). Notably, the illusions often used to study bodily self-consciousness rely on the manipulation of the spatiotemporal congruency of tactile cues on the body and visual cues from the external space, and have been shown to induce remapping of the PPS around the hand (Brozzoli et al. 2012), face (Maister et al. 2015), and trunk (Noel, Grivaz et al. 2015; Noel, Pfeiffer et al. 2015; Park et al. 2017). The present finding of PPS-related activity in the insula, hence, reinforce theoretical postulations (Blanke 2012; Serino et al. 2013; Blanke et al. 2015) and psychophysical results (Noel, Grivaz et al. 2015; Noel, Pfeiffer et al. 2015; Salomon et al. 2017) highlighting the association between bodily self-consciousness and PPS representation. Lastly, the PHG has previously been categorized as a multisensory region (Tanabe et al. 2005), and a number of studies have suggested that the PHG is part of a network involved in processes relating to bodily self-consciousness (e.g., Tsakiris et al. 2007) as well as spatial navigation and self-location (Guterstam, Björnsdotter, Bergouignan et al. 2015; Guterstam, Björnsdotter, Gentile et al. 2015). The present data underline the PHGs involvement in multisensory PPS processing. This finding deserves further research concerning the potential role of this region as a hub between multisensory PPS processing, self-related processing, and its well-described role in memory and spatial navigation (Guterstam, Björnsdotter, Bergouignan et al. 2015; Guterstam, Björnsdotter, Gentile et al. 2015). Interestingly, experimental alterations of bodily self-consciousness have been suggested to alter memory formation through activation of the hippocampal formation (Bergouignan et al. 2014). Thus, it may be proposed that the PHG might serve as a gateway between the lower-level (multisensory) aspects of PPS and the implication of trunk-centered PPS in higher-order level of cognition such as egocentric processing (Canzoneri et al. 2016). A speculation that remains to be further tested (see Berthoz 2000, for a similar speculation).

The Timing of PPS Effect

Evidence on the timing of AT PPS effect is currently lacking based on both animal and humans studies. The only evidence concerning the timing of the PPS effect is provided by studies investigating visuotactile PPS. At first, our PPS results may appear somewhat late compared with previous electrophysiological findings of visuotactile PPS, as the present PPS responses occurred on average between 129 and 226 ms. For instance, evidence from single cell recording in monkeys (Avillac et al. 2007) show visuotactile, PPS-related responses occurring already at 68 ms. In humans, Sambo and Forster (2009) performed a visuotactile ERP study as a function of the spatial disparity of visuotactile stimuli in depth and observed a modulation in ERP amplitude at electrodes over the superior temporal lobe already at 100 ms poststimulus onset. Similarly, Cappe et al. (2012) showed an effect of distance for audio–visual stimuli starting at ~75 ms poststimuli onset. Although on average our PPS effect occurred somewhat later compared with previous evidence, it is worth noting that our results show distinct response patterns. That is, the insula and PCG show the first response at ~40 ms after stimulus onset, which is compatible with an early visuotactile PPS response. In addition, we also found a later response (~150 ms after stimulus onset) in the insula, PCG, and PHG, which is in line with later responses observed in previous studies. These results suggest that the PPS responses (on the trunk) occur during at least 2 distinct time periods, and can occur simultaneously over different brain areas, mainly overlapping with AT responses (see next section).

Audio-Tactile Multisensory Integration and PPS Effect

Another finding worth highlighting about AT multisensory integration and the PPS effect is that both these processes appear to co-exist spatially and temporally (i.e., in the same electrodes and during similar time periods). AT multisensory integration occurred on average from 151 to 244 ms poststimuli onset, while PPS effect is discernable on average from 139 to 226 ms poststimuli onset. If we look in greater detail into the different regions where both multisensory integration and PPS effect occurred, the timing of the two processes also overlapped. This observation hence provides—for the first time—evidence speaking in favor of a PPS representation, which is not yielded after a series of processes whereby multisensory integration occurs first, and in different brain regions, and then forwarded to regions forming a PPS representation of the space surrounding a body. On the contrary, our results show that AT multisensory integration and PPS effect are concurrent during two time periods and across several brain regions. These electrophysiological findings suggest that PPS processing is based on a form of multisensory integration which, in addition, shows a clear spatial modulation of the response, in agreement with previous suggestions from neuropsychology (Farnè and Làdavas 2000) and psychophysical (Noel, Grivaz et al. 2015; Noel, Pfeiffer et al. 2015; Serino et al. 2015) studies. Our data add critical insight by demonstrating that the representation of the space near (vs. far) from one’s body results from the processing of events/objects in the world involving the response of multisensory brain regions located in the PCG, but also in deeper and more medial areas (such as the insula and the PHG), likely harboring multisensory neurons with bodily anchored and depth-restricted RFs (Fogassi et al. 1996; Graziano et al. 1997, Avillac et al. 2007; see Magosso 2010 for a computational model of multisensory PPS representation). Our results also indicate that few electrodes, located in the IFG, STG and CG, show a PPS effect without showing multisensory integration. However, it should be noted that those electrodes were spatially adjacent (or at least in the same brain regions) to electrodes showing multisensory integration and PPS effect.

The present findings further show that the PPS effect is not due to mere habituation and/or anticipation effect. First, the experimental design used included a randomized order of conditions (Audio [A], Tactile [T], and Audio-tactile [AT]) across trials. Therefore, despite the onset of a sound for the A and AT, the participant could not predict if a tactile stimulation would occur or not. Second, if indeed our results had been biased by some form of warning cue, we would have expected that the response to the first tactile stimulation (i.e., far condition for the AT) to be stronger compared with the 2 other tactile stimulation conditions (i.e., middle, close for the AT). This, however, was not the case. Our results instead show a stronger effect for tactile stimuli in the middle and close (vs. far) position (Fig. 4). Third, there was a 500 msecs between the onset of the auditory stimulus and the occurrence of the first tactile stimulus in the AT conditions. Thus, more than a warning stimulus, the auditory stimulation might act as a cue to expect the occurrence of the tactile stimulation at the far location. However, given the design of our task, this was also the case for the AT middle and near locations. The expectation is therefore equivalent among the 3 time points/distances, or eventually should increase linearly. This was the reason why we added the control analysis, which showed a very different pattern of modulation of the response in the T condition as compared with the AT condition.

As a last note, the experimental design used in the current study aimed at studying multisensory processing involving a tactile response, and more precisely a modulation of tactile processing due to the presence of an auditory cue within the PPS, in line with the neurophysiological mechanisms of PPS as suggested by previous animal research. This may have biased our results toward finding PPS effects more likely within tactile rather than auditory areas. Despite this, we observed results in both tactile and auditory areas. Nevertheless, because the electrodes were exclusively implanted for clinical purpose, the current dataset does not allow us to clearly state for or against such bias, nor to make any specific claim about the prevalence of our effect in any specific region of the brain.

Conclusion

We described electrophysiological responses linked to the multisensory integration of AT events and distinguished them from unimodal A or unimodal T responses as well as from simple AT summation responses. In addition, we showed that among these AT multisensory responses, LFPs from specific sites were modulated by the distance between the A and T component in a way that distinguishes near, peripersonal or bodily space (the trunk PPS) from spatial compartments far from the body. AT multisensory integration was first observed in the insula and during later phases in the STG, PCG, PHG, and IFG. A similar spatiotemporal response pattern was observed for the PPS effect but limited to insula, PHG, and PCG. Taken together, the present findings show—for the first time—that AT multisensory integration and PPS effect share common spatial and temporal processes, which go beyond previous single unit reports of multisensory integration in putative PPS neurons in area VIP (Avillac et al. 2007) to a number of other cortical areas, while also indexing multisensory PPS in humans, via audio-tactile, as opposed to visuotactile integration, and around the trunk as opposed to the hand or face.

Funding

Bertarelli Foundation and the Swiss National Science Foundation [Grant number 320030_166643] to Olaf Blanke. Andrea Serino is supported by the Swiss National Foundation [Grant PP00P3_163951/1].