An analytical method to separate modality‐specific and nonspecific sensory components of event‐related potentials

Abstract

Several models have been developed to analyse cortical activity in response to salient events constituted by multiple sensory modalities. In particular, additive models compare event‐related potentials (ERPs) in response to stimuli from two or more concomitant sensory modalities with the ERPs evoked by unimodal stimuli, in order to study sensory interactions. In this approach, components that are not specific to a sensory modality are commonly disregarded, although they likely carry information about stimulus expectation and evaluation, attentional orientation and other cognitive processes. In this study, we present an analytical method to assess the contribution of modality‐specific and nonspecific components to the ERP. We developed an experimental setup that recorded ERPs in response to four stimulus types (visual, auditory, and two somatosensory modalities to test for stimulus specificity) in three different conditions (unimodal, bimodal and trimodal stimulation) and recorded the saliency of these stimuli relative to the sensory background. Stimuli were delivered in pairs, in order to study the effects of habituation. To this end, spatiotemporal features (peak amplitudes and latencies at different scalp locations) were analysed using linear mixed models. Results showed that saliency relative to the sensory background increased with the number of concomitant stimuli. We also observed that the spatiotemporal features of modality‐specific components derived from this method likely reflect the amount and type of sensory input. Furthermore, the nonspecific component reflected habituation occurring for the second stimulus in the pair. In conclusion, this method provides an alternative to study neural mechanisms of responses to multisensory stimulation.

Four stimulation types (auditory, visual and electrical delivered through surface or pin electrodes) were used in the study (A). Twelve stimulation blocks were carried out, each of them containing 48 trials either of unimodal (A, V and E_X) or multimodal stimuli (i.e., bimodal AV, E_XA, E_XV and trimodal E_XAV). Each trial consisted of two stimuli separated by 1.5 s. Intertrial intervals varied randomly between 8 and 12 s (B). A 62‐channel EEG was recorded in response to the different stimulus combinations (C), and an average ERP waveform was computed. Using an analytical method based on the additive model, we extracted modality‐specific and nonspecific components (D).

Abbreviations

A

auditory stimulus

AV

auditory–visual stimulus

C

nonspecific component

EA

electrical–auditory stimulus

EAV

electrical–auditory–visual stimulus

EEG

electroencephalogram

E_P

pin electrical stimulus

ERP

event‐related potentials

E_S

surface electrical stimulus

EV

electrical–visual stimulus

GFP

global field power

ICA

independent component analysis

V

visual

VAS

visual analogue scale

1. INTRODUCTION

The brain has diverse processing strategies to integrate and evaluate information from sensory inputs to navigate through daily life. Among them, top‐down modulatory mechanisms can enhance certain events in detriment of other sensory inputs, making them more salient (Misselhorn et al., 2016). In this context, saliency is defined as the ability of a stimulus to stand out relative to the sensory background or relative to the preceding stimuli (Ronga et al., 2013). Saliency has been researched extensively, particularly in relation to the nociceptive system (Iannetti et al., 2008; Liang et al., 2019; Senkowski et al., 2014). In general, the perceived intensity and the novelty of the stimuli are the determining features for saliency, and most experimental paradigms manipulate these features to investigate spatiotemporal changes in cortical activity, commonly through the assessment of event‐related potentials (ERPs) in response to differences in sensory inputs (Luck & Kappenman, 2011; Ronga et al., 2013; Valentini et al., 2011). Closely related to saliency is the concept of habituation, which was defined by Picton et al. (1976) as a decline in the magnitude or probability of a response upon repetition of the elicited stimulus. In other words, as saliency declines with the repetitions of a particular stimulus, habituation arises as a reduction of the amplitude of subsequent cortical responses.

Several models have been developed to analyse changes in cortical activity in response to salient events constituted by multiple sensory modalities (Colonius & Diederich, 2017; Gondan & Minakata, 2016; Stevenson et al., 2014). Particularly, the additive model is by far the most popular approach; it compares the cortical responses to events presented in two concomitant sensory modalities (i.e., bimodal stimulation) with the responses to events of each modality in isolation (i.e., unimodal stimulation). The additive model assumes that the neural activity in response to bimodal stimulation should be equal to the algebraic sum of the unimodal responses if both dimensions of a bimodal stimulus are independently processed. Any deviation from this sum should be attributed to the bimodal nature of the stimulation, that is, cortical activity that is associated exclusively with the interaction between modalities (Berman, 1961; Foxe et al., 2000; Giard & Peronnet, 1999).

In 2006, Gondan and Röder proposed an improvement to this model by introducing new terms to properly quantify the interactions between modalities, taking into consideration common (i.e., nonspecific) components. To that end, they compared the response elicited by a single tactile and a trimodal auditory–visual–tactile stimulus with the response elicited by auditory–visual and visual–tactile stimuli. The aim was to assess the components stemming from sensory interactions, discarding the nonspecific components (Gondan & Röder, 2006; Mouraux & Iannetti, 2009). However, nonspecific components are relevant by themselves, since they likely carry information about stimulus expectation and evaluation, motor preparation and attentional orientation.

In this work, we shift the focus from sensory interactions arising from multisensory stimulation to the analysis of modality‐specific and nonspecific components that constitute the ERP. We propose to analyse the contribution of each component to responses from multiple sensory modalities, since we hypothesize that they might be particularly related to underlying neural mechanisms linked to specificity, saliency and habituation. To this end, we developed an experimental setup that delivered four stimulus types (visual, auditory and two somatosensory modalities) in three different conditions (unimodal, bimodal and trimodal stimulation) and adapted the additive model to assess the contribution of modality‐specific and nonspecific components to the ERPs.

2. MATERIALS AND METHODS

2.1. Participants and ethical approval

Twenty‐four healthy volunteers participated in the experiment. Written informed consent was obtained from all volunteers prior to participation. All experimental procedures were approved by the local ethics committee of Region Nordjylland, Denmark (approval number VN‐20110027), and conducted in accordance with the Declaration of Helsinki.

2.2. Experimental design

Volunteers were asked to lay in a comfortable chair in a dim, silent room while receiving different stimuli corresponding to one, two or three of the sensory modalities addressed in this study: visual ( $V$), auditory ( $A$) or somatosensory (surface ( $E_S$) and pin ( $E_P$) electrical stimulation). All sensory stimuli were delivered to or near the dorsum of the foot so they were perceived as coming from the same spatial location, since this could influence the responses (Gondan et al., 2005). The schematic representation of the experimental setup is shown in Figure 1 (top).

FIGURE 1.

Top: Experimental setup. All stimuli were delivered to or near the dorsum of the right foot. Visual (V) stimuli consisted on a reverting checkerboard pattern in a computer screen; auditory (A) stimuli consisted on 800 Hz tones delivered by loudspeakers; electrical (E_S and E_P) stimuli consisted on a train of pulses delivered through electrodes placed on the right foot. EEG data were recorded using a 62‐channel EEG cap and a g.tec amplifier. Bottom: Experimental design for session 2. This session was divided into 12 stimulation blocks, with 48 trials each. A trial consisted of two stimuli separated by 1.5 s interstimulus interval. Intertrial interval varied randomly between 8 and 12 s. Block condition (unimodal or multimodal) and electrode type (E_S and E_P) were randomized, as well as the type of stimuli within the block modality (A, V or E_X for unimodal condition and AV, E_XA, E_XV or E_XAV for the multimodal condition).

Two experimental sessions separated by 24 h were conducted. In the first session, volunteers were familiarized with the experimental protocol, and the somatosensory stimulation intensities were determined. The second session was divided into 12 stimulation blocks. Each block contained 48 trials, and each trial consisted of two stimuli separated by 1.5 s, so that the response of the first stimulus is finished before the second one arrives. Intertrial intervals varied randomly between 8 and 12 s (Mouraux & Iannetti, 2009; Valentini et al., 2011). An interstimulus interval greater than 1 s was chosen to assess the habituation phenomenon (Ronga et al., 2013), which we hypothesized that could be related to changes in the nonspecific component.

Each block contained unimodal (i.e., individual $A$, $V$ and $E_X$ stimuli, where X stands for surface or pin) or multimodal stimuli (i.e., bimodal $\mathit{AV}$, $E_{X}A$, $E_{X}V$ and trimodal $E_X\mathit{AV}$ stimuli). Conditions (unimodal or multimodal) and electrode types (surface or pin) were randomized between blocks, and stimulus modality ( $A$, $V$ or $E_X$ for unimodal condition and $\mathit{AV}$, $E_{X}A$, $E_{X}V$ or $E_X\mathit{AV}$ for the multimodal condition) were randomized within the block. After the first stimulation block for each condition and each electrode, volunteers rated the saliency relative to the sensory background of the different stimuli in a visual analogue scale (VAS) ranging from 0 to 10, where 0 represented a stimulus that failed to capture their attention, and 10 represented a stimulus that completely captured it. A schematic representation of the stimulation paradigm is shown in Figure 1 (bottom).

2.3. Multimodal stimulation

A custom‐made software was programmed in LabView (National Instruments Corp., Austin, Texas, USA) to control all stimulus modalities simultaneously. Auditory stimulation was performed through a loudspeaker. Each auditory stimulus consisted of a sinusoidal wave with a frequency of 800 Hz and a duration of 25 ms, presented at a loud but comfortable listening level (approx. 85 dB SPL). Visual stimulation was performed through a computer screen with a refresh rate of 100 Hz. The stimuli consisted of a black‐and‐white checkerboard that displayed its negative image for 25 ms, reverting afterwards to the original colours (Luck & Kappenman, 2011; Mouraux & Iannetti, 2009).

Electrical stimulation was performed through non‐invasive electrodes placed on the foot dorsum, using the intermediate cuneiform bone as reference. Two different types of electrodes were used as cathode. The pin electrode consisted of 16 blunt stainless‐steel pins fixed in a circle with a diameter of 10 mm, in which each pin had a diameter of 0.2 mm and protruded 1 mm from the base. This design achieves high current density and facilitates spatial summation, thus favouring the activation of nociceptive afferents (Biurrun Manresa et al., 2010; Klein et al., 2004). The surface electrode was a standard electrostimulation electrode (15 × 15 mm, type Neuroline 700, Ambu A/S, Ballerup, Denmark) used for non‐nociceptive somatosensory stimulation (Lelic et al., 2012). In both cases, a large surface electrode (50 × 90 mm, type Synapse, Ambu A/S, Ballerup, Denmark) was placed on the foot arch as anode. These two types of electrodes were chosen because they predominantly activate two different peripheral afferents (Lelic et al., 2012): tactile (mediated by Aβ fibres) and nociceptive (mediated by Aδ fibres). These afferents then relay on the spinal cord and ascend to the brain via two separated pathways (the dorsal column, Baldry & Thompson, 2005, and the spinothalamic tracts, Felten et al., 2016, respectively). Thus, we hypothesized that ERPs in response to these two types of stimuli might result in different modality‐specific components.

Each electrical stimulus consisted of a train of five constant‐current pulses of 1 ms pulse width, delivered at 200 Hz by a computer‐controlled electrical stimulator (BIOPAC STMISOLA, BIOPAC Systems Inc., Goleta, California, USA), that were perceived by the volunteers as a single stimulus coming from the dorsum of the foot. Stimulation intensity was set in both cases as a multiplier factor of the detection threshold for each subject, the latter being determined using a staircase method. This multiplier factor (either 1.5, 2, 3 or 5) was chosen considering the lowest stimulation intensity that elicited a visible evoked response in the EEG during the first experimental session, after averaging 30 trials (Hu et al., 2014).

2.4. Electrophysiological recordings

Continuous EEG data were recorded using a 62‐channel EEG cap (g.tec medical engineering GmbH, Schiedlberg, Austria), based on the extended international 10–20 system. A common ground electrode was located along the sagittal midline, between electrodes FPz and Fz, and a clip electrode on the left earlobe was used as reference. Data was recorded with a band pass filter (0.1–500 Hz) and sampled at 2400 Hz per channel.

2.5. Data pre‐processing

EEG data were analysed offline in MATLAB (Mathworks, Inc., Natick, Massachusetts, USA), using the EEGLAB (Delorme & Makeig, 2004) and Letswave (NOCIONS‐Institute of Neuroscience Université Catholique de Louvain) toolboxes. For each volunteer and each condition, continuous EEG data were resampled at 256 Hz, band‐pass filtered between 1 and 30 Hz using a zero‐phase, second‐order Butterworth filter, notch‐filtered at 50‐Hz and re‐referenced to the average of all channels. The time window of interest was defined by segmenting the data into epochs. Each epoch had a duration of 4 s, including a pre‐stimulus interval of 1 s. The epochs were visually inspected to discard noisy channels and large artefacts (i.e., movements and muscle activity). In order to remove intrinsic artefacts related to the electrical stimulation, eye movements and blinks, the epochs were evaluated using Infomax Independent Component Analysis (ICA) (Lee et al., 1999). The ICA algorithm separated the EEG signals into statistically independent components of different brain and artefact sources. The clean EEG signals were obtained by eliminating the contributions of the artefactual components. These components were identified by inspecting their time course, spectra and scalp topography (Jung et al., 2000). The rejected channels were spatially interpolated with a spherical spline. Finally, the epochs were averaged across trials and baseline‐corrected using the mean amplitude of the pre‐stimulus period to obtain the ERPs. As a result of the pre‐processing stage, we obtained one average waveform for each subject, channel, and condition. ERP features (peak amplitudes and latencies) were identified automatically using the peak_finder() function in the Python package MNE (Gramfort et al., 2013) and then manually reviewed by a human observer (ELY).

2.6. Data analysis

2.6.1. Nonspecific component

To investigate the features of nonspecific components (also referred to as common component, denoted $C$), we used an analytical method based on the model developed by Gondan and Röder (2006) to isolate multisensory interactions and the original additive model (Giard & Besle, 2010). The original additive model poses that the response, for example, to a unimodal auditory stimulus can be expressed as $A+C$, in which $A$ represents an auditory‐specific component and $C$ is the nonspecific component (analogous expressions can be derived for visual and electrical stimuli). In time, the response to a bimodal audiovisual stimulus can be expressed as $\mathit{AV}+I_{\mathit{AV}}+C$, in which $\mathit{AV}$ is the algebraic sum of the unimodal specific components (i.e., $A+V$) and $I_{\mathit{AV}}$ stands for the non‐linear component that arises from the interaction between modalities (similar expressions can be derived for the remaining bimodal combinations: $E_{X}A$ and $E_{X}V$). The non‐linear interactions allow modelling for non‐linear sub‐additive or super‐additive effects in multimodal conditions (Senkowski et al., 2007).

Subtracting the responses between bimodal and unimodal stimuli, we obtain:

$$\left[\left(\mathit{AV}+I_{\mathit{AV}}+\mathrm{C}\right)\right]-\left[\left(A+C\right)+\left(V+C\right)\right]=I_{\mathit{AV}}-C$$ (1)

Gondan and Röder (2006) proposed to add a third stimulus and to quantify responses to trimodal stimulation. For example, if the added modality is electrical, the response to a trimodal stimulus can be expressed as $\mathit{EAV}+I_{\mathit{AV}}+I_{\mathit{EA}}+I_{\mathit{EV}}+C$, in which $\mathit{EAV}$ is the algebraic sum of the three modality specific components and $I_{\mathit{XY}}$ represents the non‐linear interactions between any two modalities. Working algebraically with responses from unimodal, bimodal and trimodal stimuli, it is possible to obtain an expression for a bimodal interaction, as in the following example:

$$\left[\left(E,+,C\right),+,\left(\mathit{EAV},+,I_{\mathit{AV}},+,I_{\mathit{EA}},+,I_{\mathit{EV}},+,C\right)\right]-\left[\left(\mathit{EA},+,I_{\mathit{EA}},+,C\right),+,\left(\mathit{EV},+,I_{\mathit{EV}},+,C\right)\right]=I_{\mathit{AV}}$$ (2)

Finally, by subtracting Equation (2) from Equation (1), it is possible to isolate the nonspecific component:

$$\left[I_{\mathit{AV}}\right]-\left[I_{\mathit{AV}}-C\right]=C$$ (3)

We carried out this analysis for all stimulus combinations listed in Table 1. It is relevant to note that the resulting expression for $C$ is always the same, regardless of the choice of combinations for unimodal, bimodal and trimodal responses. We hypothesized that changes in the nonspecific component reflect differences in brain activity related to saliency and habituation.

TABLE 1.

Possible combinations of the original additive model and Gondan and Roder's additive model from which to compute the nonspecific component (C)

Original additive model	Result	Gondan and Röder's additive model	Result
$\left[\mathit{EA}+I_{\mathit{EA}}+C\right]-\left[\left(E+C\right)+\left(A+C\right)\right]$	$I_{\mathit{EA}}-C$	$$\begin{gathered} \left[\left(V,+,C\right),+,\left(\mathit{EAV},+,I_{\mathit{EA}},+,I_{\mathit{EV}},+,I_{\mathit{AV}},+,C\right)\right] \\ -\left[\left(\mathit{AV},+,I_{\mathit{AV}},+,C\right),+,\left(\mathit{EV},+,I_{\mathit{EV}},+,C\right)\right] \end{gathered}$$	$I_{\mathit{EA}}$
$\left[\mathit{EV}+I_{\mathit{EV}}+C\right]-\left[\left(E+C\right)+\left(V+C\right)\right]$	$I_{\mathit{EV}}-C$	$$\begin{gathered} \left[\left(A,+,C\right),+,\left(\mathit{EAV},+,I_{\mathit{EA}},+,I_{\mathit{EV}},+,I_{\mathit{AV}},+,C\right)\right] \\ -\left[\left(\mathit{AV},+,I_{\mathit{AV}},+,C\right),+,\left(\mathit{EA},+,I_{\mathit{EA}},+,C\right)\right] \end{gathered}$$	$I_{\mathit{EV}}$
$\left[\mathit{AV}+I_{\mathit{AV}}+C\right]-\left[\left(A+C\right)+\left(V+C\right)\right]$	$I_{\mathit{AV}}-C$	$$\begin{gathered} \left[\left(E,+,C\right),+,\left(\mathit{EAV},+,I_{\mathit{EA}},+,I_{\mathit{EV}},+,I_{\mathit{AV}},+,C\right)\right] \\ -\left[\left(\mathit{EA},+,I_{\mathit{EA}},+,C\right),+,\left(\mathit{EV},+,I_{\mathit{EV}},+,C\right)\right] \end{gathered}$$	$I_{\mathit{AV}}$

2.6.2. Modality specific components

Following our assumptions, the subtraction of the nonspecific component $C$ from the recorded unimodal response should results in modality‐specific components, that is, components that corresponds uniquely to each stimulus modality (either $A$, $V$ or $E$).

2.7. Statistics

Unless stated otherwise, values are expressed as mean ± standard deviation (for normally distributed variables), and as median [25th–75th percentile] for non‐normally distributed variables. Differences in saliency, peak latencies and peak amplitudes of the ERP components were assessed using a linear mixed model with random intercept, with condition (unimodal, bimodal and trimodal), stimulus number (first or second) and/or electrode type (surface or pin) as fixed factors, depending on the hypotheses. The restricted maximum likelihood estimation was used for model fitting, the Satterthwaite approach was used to estimate denominator degrees of freedom for the F statistic, and the Holm procedure was used to correct p values for multiple post hoc comparisons. This analysis was performed in Jamovi V.1.6 (The jamovi project, Sydney, Australia). Spatiotemporal differences in amplitudes of the nonspecific component were analysed using a non‐parametric cluster‐level permutation test, using the Python package MNE (Gramfort et al., 2013). This methodology defines clusters of significant differences in time and space, by grouping the time points for which the p value in individual paired t tests is smaller than 0.05, while controlling the false alarm rate. The size of each cluster was defined as the sum of the t values within the cluster. The permutations are performed (1000 in total), by shuffling the data between conditions. Each permutation will result in a new set of clusters that are used to build the permutation distribution. Finally, clusters, which size value is over a 95th percentile of the z distribution from the largest cluster obtained during the permutation testing, are considered significant.

3. RESULTS

3.1. General considerations

Twenty‐four volunteers participated in the first session of the experiment. Three volunteers did not show reliable ERPs at any of the somatosensory stimulation intensities and were discharged from the experiment. The remaining 21 volunteers completed both experimental sessions. Due to technical issues with the EEG equipment (i.e., substantially noisy recordings in multiple channels and blocks), data from two volunteers were discarded.

In the first session, detection thresholds were 100 [65–235] μA and 900 [745–1250] μA for the pin and surface electrode, respectively, whereas in the second session the detection thresholds were 100 [55–120] μA and 870 [400–1175] μA for the pin and surface electrode, respectively. For the stimulation intensity, we used a multiplier factor of 1.5 for 6 volunteers, of 2 for 4 volunteers, of 3 for 7 volunteers and a multiplier factor of 5 for 2 volunteers.

3.2. Psychophysical results

Figure 2 shows the saliency relative to the sensory background reported by the volunteers, in which it can be observed that the scores are monotonically increasing with the addition of simultaneous stimuli ( $F_{\mathrm{2,184}}=31.0,p<0.001$). Post hoc tests revealed that the saliency for the unimodal stimuli was lower compared with the bimodal (mean difference ± standard error: $1.38\pm 0.26$, $t_{184}=5.36$, $p<0.001$) and to the trimodal stimuli (mean difference ± standard error: $2.53\pm 0.34$, $t_{184}=7.48$, $p<0.001$). Furthermore, the saliency for bimodal stimuli was lower compared with the trimodal stimuli (mean difference ± standard error: $1.15\pm 0.33$, $t_{184}=3.50$, $p<0.001$).

FIGURE 2.

Saliency scores relative to the sensory background for each type of stimulation, ranging from 0 (failed to capture subject's attention) to 10 (completely capture subject's attention). Grey dots represent each individual score. Boxes extends from the first quartile to the third quartile (25th–75th percentile).

3.3. Spatiotemporal features of ERPs

After pre‐processing, an average of 34.3 ± 4.9 trials per stimulation type were used to obtain the corresponding ERPs.

3.3.1. Unimodal responses

ERP responses to unimodal stimulus are shown in Figure 3. For auditory stimulation (Figure 3, top), responses to the first stimulus showed an N1 peak located at approximately 110 ms over the vertex and a P2 peak located at around 190 ms with a positive polarity over the vertex and inverting its polarity for electrodes placed near the Sylvian fissure (Winkler et al., 2013). Responses to the second stimulus present a reduction in amplitude as well as a slight reduction in latency (N1: 100 ms and P2: 150 ms after stimulus onset).

FIGURE 3.

Unimodal auditory, visual and electrical (pin and surface) event‐related potentials. Average responses to unimodal stimulus are shown in solid colours with the shading corresponding to its standard deviation. Each plot is accompanied by the topographic representation of the peak marked by an arrow. Responses are shown at Cz for auditory and electrical stimuli and at Oz for visual stimulus. Top: Average response to unimodal auditory stimulus. Arrows indicate peaks at 110, 190 ms and 1.6 and 1.65 s. Middle: Average response to unimodal visual stimulus. Peaks are marked at 120, 230 ms and 1.6, 1.74 ms. Bottom: Average responses to surface (solid line) and pin (dotted line) electrical stimulation. For stimulation with surface electrode, peaks are marked by light blue arrows at 100, 210 ms and 1.59, 1.7 s. For stimulation using the pin electrode, peaks are marked by dark blue arrows at 140, 320 ms and 1.62 and 1.77 s.

Responses to visual stimulation (Figure 3, middle) presented a clear P1 peak at approximately 120 ms, more prominent over occipital electrodes and a N1 peak at approximately 230 ms after stimulus onset (Hillyard & Anllo‐Vento, 1998). A reduction in latencies was observed for the second stimulus, but we did not observe changes in amplitude.

Figure 3 (bottom) shows the average waveforms observed in Cz in response to electrical stimuli, since the somatotopic representation of the feet is located centrally, over the medial wall of the anterior parietal lobe (Sanchez Panchuelo et al., 2018). A N1–P1 complex (latencies approximately 30 and 50 ms, respectively) was observed in response to electrical stimulation with the surface electrode. The complex was followed by a larger N2 deflection at 100 ms and by a positive P2 peak at 210 ms. For the second stimulus, responses presented a N1–P1 complex (latencies approximately 20 and 50 ms, respectively), followed by a N2 deflexion (latency 90 ms) and by a P2 peak (latency 200 ms).

The responses to electrical stimulation with the pin electrode did not display a clear N1–P1 complex. For the first stimulus, the latencies for N2–P2 complex were approximately 140 ms and 320 ms, respectively, whereas for the second stimulus, the N2–P2 latencies were 120 ms and 270 ms, respectively. From these results, it could be observed that responses to electrical stimulation with the pin electrodes presented a delay in comparison to the responses to stimulation with the surface electrodes, in line with previous results (Lelic et al., 2012).

3.3.2. Bimodal responses

The averaged responses to simultaneous stimulation of two sensory modalities are depicted in Figure 4. For simultaneous auditory and visual stimulation (Figure 4, top), the responses to the first stimulation presented a negative deflection around 110 ms over the vertex, followed by a positive peak at approximately 190 ms. The responses to the second stimulation showed these deflections at 100 ms and 190 ms. There was a reduction in amplitude of the response to the second stimulus in comparison to the preceding stimulus.

FIGURE 4.

Bimodal event‐related potentials. Average responses to bimodal stimulus are shown in solid colours with the shading corresponding to its standard deviation. Each plot is accompanied by the topographic representation of the peak marked by an arrow. All responses are shown at Cz. Top: Averaged response to auditory–visual stimulation. Arrows indicate peaks at 110, 190 ms and 1.6, 1.69 s. Middle: Averaged response to electrical‐auditory stimulation with surface (solid line) and pin (dotted line) electrodes. Arrows in light purple indicate peaks at 100, 190 ms and 1.59, 1.67 s obtained from stimulation with surface electrode. Arrow in deep purple indicate peaks at 120, 190 ms and 1.6, 1.67 s obtained from stimulation with pin electrode. Bottom: Averaged response to electrical‐visual stimulation with surface (solid line) and pin (dotted line) electrodes. Arrows in light lilac indicate peaks at 100, 200 ms and 1.59, 1.7 s obtained from stimulation with surface electrode. Arrow in dark lilac indicate peaks at 120, 210 ms and 1.61, 1.71 s obtained from stimulation with pin electrode.

The responses to simultaneous auditory and electrical stimulation with the surface and pin electrodes (Figure 4, middle) presented the same morphology for both types of electrodes, but differing in latency and amplitude. For the first stimulus, electrical stimulation with the surface electrode resulted in a negative peak over the vertex with a latency approximately of 100 ms and a positive peak at about 190 ms. Electrical stimulation with the pin electrode resulted in a negative peak delayed 20 ms in average in comparison with the negative peak obtained from surface stimulation. However, the positive peak did not show such delay, appearing with a latency of 190 ms regardless of the electrode type. For the second stimulus, the differences in latency were less pronounced.

For electrical and visual stimuli (Figure 4, bottom), a negative vertex potential was visible at approximately 100 ms and at 120 ms for electrical stimulation with the surface and the pin electrode, respectively. This difference in latency was less evident for the positive peak, which occurred at about 200 and 210 ms for stimulation with surface and pin electrodes, respectively.

3.3.3. Trimodal responses

The responses to simultaneous trimodal stimulation for both types of electrical electrodes are shown in Figure 5. The trimodal stimulation using the surface electrode resulted in a positive peak approximately at 49 ms, followed by a negative deflection at approximately 100 ms, ending with second positive peak at the vertex at 190 ms.

FIGURE 5.

Trimodal event‐related potentials. Average waveform at Cz, with the solid line showing the response to stimulation using surface electrode and the dotted line showing the response to stimulation using pin electrode. The arrows indicate the latencies of the most prominent peaks where the topographical distributions are shown. Light magenta arrows show peaks at 100, 190 ms and 1.6, 1.69 s for stimulation using surface electrodes. Dark magenta arrows show peaks at 110, 190 ms and 1.6, 1.69 s for stimulation using pin electrodes.

The trimodal stimulation using the pin electrode did not show a clearly visible first positive peak; the negative deflection appeared with a 10‐ms delay compared with the stimulation using the surface electrode, with no observable latency differences of the following positive peak. However, the response elicited using the pin electrode were smaller in amplitude compared to the responses using the surface electrode, more prominent around the 190‐ms peak. As expected, topographic representations at the time of occurrence of the first negative peak showed a central distribution and positive activity at occipital regions for both types of electrodes. Although there was a slight latency difference between the responses to both types of electrical stimuli, it was less prominent than for bimodal and unimodal stimulation.

3.4. Nonspecific component (C)

The resulting waveform for the nonspecific component ( $C$) at the vertex is shown in Figure 6. Statistical analysis did not reveal differences between the waveforms derived from either type of electrode. Topographic representations showed a central negativity around 100 ms after stimulus onset, surrounded by a positivity in occipital and frontal regions. This phenomenon reversed its polarity at around 310 ms, being positive in the vertex region and negative in the occipital and frontal areas (Figure 6, bottom).

FIGURE 6.

Nonspecific component (C). Averaged waveform at Cz derived from stimulation with surface and pin electrodes. Arrows show peaks at 130 and 310 ms, together with the corresponding topographical distribution at said latencies.

We fitted a linear mixed model for the peak amplitudes recorded at Cz, to assess differences in C relative to the first versus the second stimulus. For the negative peak, results showed significant main effects of stimulus number ( ${\mathit{\text{F}}}_{\text{1,38.2}}\text{=23.5257,} \mathit{\text{p}}\text{<0.001}$), but no significant main effect of electrode type ( ${\mathit{\text{F}}}_{\text{1,44.3}}\text{=0.0239,} \mathit{\text{p}}\text{=0.878}$) and no significant interaction ( ${\mathit{\text{F}}}_{\text{1,38.5}}\text{=0.4892,} \mathit{\text{p=}}\text{0.489}$). For the positive peak, results showed significant main effect of stimulus number ( ${\mathit{\text{F}}}_{\text{1,37.1}}\text{=11.2524,} \mathit{\text{p}}\text{=0.002}$), but not electrode type ( ${\mathit{\text{F}}}_{\text{1,43.9}}\text{=3.9786,} \mathit{\text{p}}\text{=0.052}$) or interaction ( ${\mathit{\text{F}}}_{\text{1,37.3}}\text{=0.0361,} \mathit{\text{p}}\text{=0.85}$). Furthermore, we extended the statistical analysis spatially by performing cluster‐based analysis comparing the responses of the first and second stimuli. Results are shown in Figure 7. For the nonspecific component, derived from responses to electrical stimulation using the surface electrodes (Figure 7, left) most of the clusters of significant differences occurred between 200 and 400 ms, even though early small‐sized clusters were also observed. For the responses elicited by the pin electrode (Figure 7, right), most clusters were found between 300 and 400 ms, but also were found later, at approximately 600 ms.

FIGURE 7.

Statistical analysis comparing first versus second stimulus of the nonspecific component (C). Left: Analysis performed on C derived from the responses to electrical stimulation with surface electrode. Right: Analysis performed on C derived from the responses to electrical stimulation with pin electrode.

3.5. Modality‐specific responses

Figure 8 shows the recorded unimodal responses to auditory, visual, and electrical stimulation (for both types of electrodes), together with each modality‐specific component that resulted from subtracting nonspecific component (C) from the unimodal response. In general, the amplitudes in response to the first stimulus compared to the responses to the second stimulus were maintained across the specific responses. However, this observation was not noticed in the original recordings, since the amplitude of the responses diminished from the first to the second stimulus. Global field power (GFP) analysis for specific and nonspecific component is provided in the supporting information.

FIGURE 8.

Modality‐specific components and recorded unimodal responses. In each plot, averaged responses to recorded unimodal (solid line) and calculated specific components (dotted line) are shown. From top to bottom: auditory, visual, electrical with surface electrodes stimulation and electrical with pin electrode stimulation responses are depicted.

To quantitatively assess modality‐specific components in response to the first versus the second stimulus we analyse the peak amplitude data by fitting a linear mixed model. For both the negative and positive peaks we found no significant main effect for stimulus number (Negative: ${\mathit{\text{F}}}_{\text{1,77.5}}\text{=2.89,} \mathit{\text{p=}}\text{0.093}$. Positive: ${\mathit{\text{F}}}_{\text{1,71.5}}\text{=1.226,} \mathit{\text{p=}}\text{0.264}$) nor interactions (Negative: ${\mathit{\text{F}}}_{\text{3,77.7}}\text{=1.74,} \mathit{\text{p=}}\text{0.229}$. Positive: ${\mathit{\text{F}}}_{\text{3,71.6}}\text{=0.776,} \mathit{\text{p=}}\text{0.511}$). We were only able to find significant main effect of stimulus type (Negative: ${\mathit{\text{F}}}_{\text{3,82.3}}\text{=18.74,} \mathit{\text{p<}}\text{0.001}$. Positive: ${\mathit{\text{F}}}_{\text{3,76}}\text{=6.543,} \mathit{\text{p}}\text{<0.001}$).

We further analysed latencies for modality‐specific components of electrical stimulus to test for differences in conduction velocities related to our specificity hypothesis. To that end, we fitted a linear mixed model for the latencies of the negative and positive peaks. Results showed significant main effects for electrode type (Negative: ${\mathit{\text{F}}}_{\text{1,41.2}}\text{=26.345,} \mathit{\text{p}}\text{<0.001}$. Positive: ${\mathit{\text{F}}}_{\text{1,36.5}}\text{=24.59,} \mathit{\text{p}}\text{<0.001}$), but no significance for main effect of stimulus number (Negative: ${\mathit{\text{F}}}_{\text{1,38.4}}\text{=0.570,}\mathit{\text{p=}}\text{0.455}$. Positive: ${\mathit{\text{F}}}_{\text{1,32.8}}\text{=2.16,} \mathit{\text{p=}}\text{0.152}$) or interactions (Negative: ${\mathit{\text{F}}}_{\text{1,38.4}}\text{=0.108,} \mathit{\text{p=}}\text{0.745}$. Positive: ${\mathit{\text{F}}}_{\text{1,32.8}}\text{=1.8,}\mathit{\text{p=}}\text{0.189}$).

4. DISCUSSION

In this study, we implemented an experimental setup to deliver sensory stimulation through simultaneous combinations of up to three different stimulus modalities. These included visual, auditory, and electrical stimulation, the latter delivered using two types of electrodes (surface and pin). EEG signals were recorded in response to unimodal, bimodal and trimodal stimulation for two consecutive stimuli. For the analysis, we proposed an analytical method based on the original additive model and a modified model by Gondan and Röder (2006). The nonspecific neural activity from the ERPs was identified and subsequently subtracted from the unimodal responses to obtain modality‐specific components.

4.1. Neural processes underlying multisensory ERPs

ERPs have been used in numerous studies to describe different aspects of cognitive, sensory and motor processing of the human brain (Williams et al., 2021; Woodman, 2010). ERPs present a locked response to the event that generates them, and thus, they can be associated to the neural processing of a given stimulus along with background information (Luck & Kappenman, 2011). Several studies have focused on auditory and visual stimulus, as they are easier to evoke, and their elicited cortical responses are well known (Giard & Peronnet, 1999; Hillyard & Anllo‐Vento, 1998; Teder‐Sälejärvi et al., 2002; Winkler et al., 2013). Other studies have focused on tactile and electrical stimulation to study the somatosensory system (Eimer & Forster, 2003), whereas laser and thermal stimulation have been employed for the activation of the nociceptive system (Hu et al., 2014; Manfron et al., 2020). Regarding ERP analysis, examples of the use of a multimodal approach introducing up to three simultaneous stimuli have been found along the literature (Misselhorn et al., 2016), although it is more frequent to find only two, being auditory–visual the most popular ones (Stevenson et al., 2014; Teder‐Sälejärvi et al., 2002).

In the present study, we implemented an experimental setup that not only uses the most common stimulation types (auditory and visual) but that it also includes two types of electrical stimulation by using different electrodes, that is, surface and pin, each of them followed by a second stimulus of the same sensory modality. For unimodal stimulation, we obtained similar morphology and peak latency as previously reported in the literature for auditory and visual responses (Hillyard & Anllo‐Vento, 1998; Winkler et al., 2013). For electrical stimulus, we observed differences in latency between the ERP peaks generated from stimulation using the pin electrode and the ERP peaks generated using the surface electrode. This observation could be attributed to differences in conduction velocity of the nerve fibres preferentially activated by each type of electrode (Aβ and Aδ for the surface and pin electrodes, respectively (Hugosdottir et al., 2019; Lelic et al., 2012), as this could also be observed when bimodal and trimodal stimulation were applied, although less pronounced in the latter.

The approach employed in this study is akin to the experimental paradigm presented and used by Mouraux and Iannetti (2009), in which they did not observe differences in saliency due to stimulus modality. Although we did not specifically test for differences in saliency between sensory modalities (we focused instead of differences due to the number of concomitant modalities), we observed an overall large dispersion in saliency values, which could account for the lack of difference in previous studies, and likely reflects that the subjective assessment of saliency is variable for different participants. Interestingly, this dispersion was not reflected in the amplitudes of the ERPs, which were comparable across modalities.

4.2. Contributions of modality‐specific and nonspecific components to multisensory ERPs

Previous studies have addressed the problem of separating modality‐specific from nonspecific neural activity in different ways. For instance, Mouraux and Iannetti (2009) tackled this issue by performing a variant of the ICA algorithm to concatenated ERPs in response to isolated visual, auditory, nociceptive, and non‐nociceptive stimuli. Components with contribution to all sensory modalities were categorized as multimodal, and those which contribute only to one sensory modality only were categorized as modality specific. However, by assuming that each neural generator is independent and the resulting signals are linearly combined, ICA does not take into consideration the possibility of signals arising from the interaction between modalities (which by definition cannot be independent from the modality specific components). Additionally, ICA assumes that net responses are instantaneous linear combinations of the independent sources (i.e., they should be active at the same time), which was not the case in that experimental design, since the signals were concatenated from unimodal responses to each isolated stimulus modality.

Alternatively, the additive model has been applied to establish the presence of interaction between different sensory modalities by adding or subtracting the ERPs of unimodal and bimodal stimulus (Naumer & Kaiser, 2010). However, it has been criticized because of the lack of consideration about the nonspecific activity and its influence in the interactions (Teder‐Sälejärvi et al., 2002). To overcome this limitation, Gondan and Röder (2006) proposed a modification to the traditional additive model, in which a tactile stimulus was introduced in order to balance the nonspecific component in both parts of the addends of the model's equation, in order to obtain components that represent sensory interactions.

We developed a method to further analyse ERP data based on the traditional additive model and on the modified version developed by Gondan and Röder (2006). With the present approach, we were able to isolate a nonspecific component ( $C$) and examine its spatiotemporal features. We found that deriving the nonspecific component using recorded responses to electrical stimulation using pin or surface electrodes did not yield significant differences among them. This represents two possible scenarios: either there was an actual difference between the nonspecific components derived from the two electrodes, but we did not observe it due to lack of power, or such difference did not exist or was negligible, suggesting that the nonspecific component likely reflects common activity due to the processing of sensory stimulation and it is not linked to a specific modality.

Our experimental procedure also involved the presentation of a second stimulus identical to the first one after a 1.5 s interval, to evaluate potential changes due to repetitive stimulation. Studies have been carried out to explore the adaptation to repeated stimuli (i.e., habituation), which involves a decrease in responses amplitudes after presenting the same stimulus more than once. It has been reported for visual (Bruin et al., 2000), auditory (Woods & Elmasian, 1986) and somatosensory systems (Mancini et al., 2018). One of our main hypothesis was that the nonspecific component was related to the attentional processing of the incoming stimulus, thus taking part in the habituation phenomenon. In this regard, there is a clear reduction in amplitude in the nonspecific component between the first and the second stimuli (Figures 6 and 7). It is straightforward to suggest, based on these results, that habituation is a phenomenon that primarily affects common or nonspecific components of the ERP responses, since the stimulation intensity (i.e., the afferent input) does not change between successive stimuli. Indeed, under this hypothesis, little or no changes should be expected in the modality‐specific components between the first and second stimuli in the train. In this way, the amplitude of the nonspecific component could be a proxy to assess the saliency of a stimulus relative to the preceding stimuli (Ronga et al., 2013). It must be noted that the saliency relative to the sensory background (that we measured by the ability of a stimulus to capture one's attention) might be further affected by other factors (e.g., the cognitive load during the rating task) and thus might not be necessarily correlated to changes in amplitude of the nonspecific component.

The modality‐specific components were computed by subtracting the nonspecific component from the unimodal responses. The statistical analysis did not reveal significant differences between peak amplitudes of the first versus the second stimulus, which could indicate that the neurophysiological mechanisms generating the modality‐specific component are not involved in the habituation process. Alternatively, we might not have had enough statistical power to detect such differences, if any. In line with our observations of the ERPs, the latencies of the modality‐specific components for electrical stimulus resulted in differences between electrode types. This supports our hypothesis in relation with specificity. Differences in latencies due to conduction velocities of the fibres should be reflected on the specific components, given that this is a distinctive feature of each stimulus, and not common to all modalities.

In this work we were able to separate nonspecific from modality‐specific components of the ERP responses. By doing so, we were able to analyse known phenomena of the ERP such as habituation and specificity and test hypotheses related to these components. Keeping in mind that the components might represent different neural mechanisms, that is, nonspecific components likely reflect preparation, attentional and arousal mechanisms and modality‐specific components may reveal specific processing of a particular stimulus, new research may benefit from our model in order to individually assess each of these mechanisms.

4.3. Limitations and future perspectives

A few limitations associated to the experimental design should be considered. First, we used synchronized bimodal and trimodal stimuli to elicit concomitant responses, but we cannot account for transduction delays that might arise between the different sensory modalities. This might be of particular importance if one wants to extend the first come, first served considerations of somatosensory volleys and cortical responses to account for audiovisual responses as well (Garcia‐Larrea, 2004). Additionally, there were practical limitations with the stimulation software and the setup that prevented a fully randomized design, the most salient one being the need to switch the electrical stimulator between electrode types. Furthermore, the signal‐to‐noise ratio of the ERPs could be improved by presenting more salient events. However, this might be a challenging aspect to address, as the stimulation intensities could not be much higher, since that would compress the saliency range between unimodal, bimodal and trimodal stimuli. Another option would be to increase the number of trials. However, while the number of trials is within the current recommendations for this type of experiment (Thigpen et al., 2017), it could not be substantially increased either, as the current number of channels, conditions and trials already made for a long experiment (around 3 h in total, including the setup). In future experiments, an increment on the number of trials and the SNR could be achieved by using a single non‐audiovisual modality, for example, nociceptive laser stimulation or non‐nociceptive strong vibrotactile stimuli.

Finally, we proposed an additive model to separate ERP components, which reflect changes in the ERP waves due to external factor (e.g., stimulus saliency or modality, habituation). However, it must be kept in mind that these components might or might not reflect neural activity from individual, well‐localized cortical sources. Future work should be directed towards source localization of these components, to establish whether the modality‐specific and interaction components correspond to established unisensory and multisensory cortical areas, respectively.

5. CONCLUSION

We developed an analytical model to isolate modality‐specific and nonspecific components from ERPs elicited by multimodal sensory stimulation. We observed that the spatiotemporal features of the modality‐specific components likely reflect the amount of sensory input, whereas the nonspecific component is linked with common activity due to the cognitive processing of this input. In conclusion, this method provides an alternative to study neural mechanisms of responses to multisensory stimulation.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

AUTHOR CONTRIBUTIONS

Experimental design: F.A.J., J.B.M., O.K.A.; data analysis: C.A.M., E.L.Y.; drafted paper: E.L.Y.; manuscript revision: C.A.M., F.A.J., O.K.A., J.B.M.

PEER REVIEW

The peer review history for this article is available at https://publons.com/publon/10.1111/ejn.15798.

Supporting information

Data S1. Supporting Information

Young, E. L. , Mista, C. A. , Jure, F. A. , Andersen, O. K. , & Biurrun Manresa, J. A. (2022). An analytical method to separate modality‐specific and nonspecific sensory components of event‐related potentials. European Journal of Neuroscience, 56(7), 5090–5105. 10.1111/ejn.15798

DATA AVAILABILITY STATEMENT

The data and codes supporting the findings of this study are openly available at Open Science Framework at https://osf.io/jm62z/.