The Sound and the Fury: Late Positive Potential is Sensitive to Sound Affect

Abstract

Emotion is an emergent construct of multiple distinct neural processes. EEG is uniquely sensitive to real-time neural computations, and thus is a promising tool to study the construction of emotion. This series of studies aimed to probe the mechanistic contribution of the Late Positive Potential (LPP) to multi-modal emotion perception. Experiment 1 revealed LPP amplitudes for visual images, sounds, and visual images paired with sounds were larger for negatively rated stimuli than for neutrally rated stimuli. Experiment 2 manipulated this audio-visual enhancement by altering the valence pairings with congruent (e.g. positive audio + positive visual) or conflicting emotional pairs (e.g. positive audio + negative visual). Negative visual stimuli evoked larger early LPP amplitudes than positive visual stimuli, regardless of sound pairing. However, time frequency analyses revealed significant midfrontal theta-band power differences for conflicting over congruent stimuli pairs suggesting very early (~500ms) realization of thematic fidelity violations. Interestingly, late LPP modulations were reflective of the opposite pattern of congruency, whereby congruent over conflicting pairs had larger LPP amplitudes. Together, these findings suggest that enhanced parietal activity for affective valence is modality-independent and sensitive to complex affective processes. Furthermore, these findings suggest that altered neural activities for affective visual stimuli are enhanced by concurrent affective sounds, paving the way towards an understanding of the construction of multi-modal affective experience.

Emotion is an emergent construct formed from the interaction of multiple primitive sub-processes (Barrett, 2006; Russell, 2009). The study of emotion faces dual challenges in understanding the role of each sub-process during the context of ecologically valid interactions. While emotional images are commonly used to investigate affective processing, complex multisensory emotional signals, like video events, should be expected to reflect more ecologically valid emotional experiences. Research using functional magnetic resonance imaging (fMRI) has successfully elucidated neural dynamics relevant to salient emotional signals elicited by emotion-rich film clips (Beauregard et al., 1998; Kilpatrick & Cahill, 2003). In contrast, research using electroencephalography (EEG) has been strongly reliant on static emotional images. It is thus an important long-term goal to move from emotional snapshots towards more dynamic, ecologically valid emotional stimuli (i.e. emotional video presentations) in the field of EEG. However, prior to the use of highly complex stimuli, it is imperative to gain a mechanistic understanding of how the emotional brain integrates signals from varied sensory modalities.

In line with a growing understanding of the emergent nature of cognitive experience (Aisa et al., 2008, McClelland et al., 2010), recent work has aimed to define emotion as an outcome of the integration of primitive neural processes (Barrett, 2006; Barrett et al., 2007; Coan, 2012; Lindquist and Barrett, 2012; Lindquist et al., 2012). In other words, differential integration of distinct neural computations underlies the myriad of emotional phenomenon that we experience. In the light of this perspective, research in the field of emotion may aim to address the generic neural mechanisms involved in distinct facets of emotion processing, for example investigating emotional signals as privileged pieces of sensory information. In short, this mechanistic view addresses perceived emotional outputs as a product of early stimulus processing and the communication of these specialized signals between neural areas in regards to the emotionality of the signal being processed.

EEG is particularly sensitive to real-time canonical neural computations (Cavanagh and Castellanos, 2016; Fries, 2009; Turkheimer et al., 2015; Womelsdorf et al., 2014), and thus is an excellent tool for the study of the construction of emotion (Dennis and Hajcak, 2009; Hajcak et al., 2010; Hajcak et al., 2011, Lamm et al., 2013, Liu et al., 2012, Moser et al., 2006; Olofsson, 2008). Indeed, EEG components reveal prioritized emotional processing for salient emotional content (see Hajcak et al., 2011, for review). Of these components, emotion processes appear to be closely associated with the P3, a positive-going deflation peaking approximately 300ms post stimulus onset, and the late positive potential (LPP), a centro-partietal long positive elevation that follows the P3 (Dennis and Hajcak, 2009). The LPP appears to be specifically modulated by the emotional intensity of a stimulus (Brown et al., 2012; da Silva, 2015; Cuthbert et al., 2000; Dennis and Hajcak, 2009; Dillon et al., 2006; Franken et al., 2009; Hajcak and Olvet, 2008; Horan et al., 2013; Liu et al., 2015; Wang et al., 2011; Wesley et al., 2015; Zhang et al., 2013), task relevant stimuli (Gable and Adams, 2013), self-relevant stimuli (Gray et al., 2004), biological imperative stimuli (Hajcak, 2011) and idiosyncratically salient stimuli such as cocaine cues presented to cocaine abusers (Franken et al., 2008), or videogame cues presented to videogame addicts (Thalemann, 2007). LPP amplitude is modulated by emotional reappraisal (Dennis and Hajcak, 2009; Foti and Hajcak, 2008), and attentional control (Hajcak et al., 2009). Taken together, the LPP seems to directly rely on the induction of affective meaning and the modulation of cognitive interpretation, making it a promising tool for understanding the ultimate construction of emotion. However, a mechanistic interpretation of the LPP has eluded investigators. It is clear that LPP is modulated by visual salient information; however, it is not clear whether this signal represents an increase in visual gain for example, or if it is a marker of more general parietal integrative processes (or both, or neither). In this report, we investigate these questions using affective sounds.

Similar to emotional picture processing, literature pertaining to emotional sound processing has also supported the notion that emotional sounds may be processed as specialized pieces of sensory information. These studies have demonstrated that emotional sound processing is related to enhanced attentional network activities (Brosch et al, 2009) and can lead to a more enhanced emotional perceptual experience (Baumgartner et al, 2006). Neuroimaging studies investigating emotion-induced music processing have shown robust effects for valence type music processing (Baumgartner et al, 2006; Blood & Zatorre, 2001; Koelsch, 2010; Sammler et al., 2007; Steinbeis & Koelsch, 2011), with amygdala and orbitofrontal cortex activation during the processing of emotional compared to neutral music pieces. Furthermore, other physiological measures, such as pupillary response (Partala, & Surakka, 2003) and heart rate (Nakahara et al., 2009; Sammler et al., 2007), are modulated by valence specific sounds.

Human EEG studies have also demonstrated early scalp potential differences between valence-specific emotional sounds, yet none (to our knowledge) have examined detailed neural responses to complex affective soundscapes. Similar to emotion picture processing (Schwartz et al., 1975), early brain potential lateralization has also been demonstrated between valence specific sounds (Altenmüller et al., 2002). These early sound induced voltage potentials are modulated for emotional auditory signals but are absent during the encoding of neutral or non emotional signals (Sauter & Eimer, M, 2010). Furthermore, these signals have been shown to modulate visual target-evoked P1, exhibiting an effect whereby affective sound processing may lead to a priming of attention centers of the visual cortex, thus suggesting increased cross modal perceptual processes for emotional signals as well as an early sensory processing boost during multimodal processing (Brosch et al, 2009). One study in particular showed that frontal midline theta power (Sammler et al., 2007) differed between valence specific sounds, whereby pleasant musical pieces were associated higher theta power than unpleasant musical pieces. Indeed, evoked potentials from auditory stimuli exhibit similar properties to that of visual evoked signals. While these studies demonstrate support that affective sounds are specialized pieces of sensory information, very few studies have investigated the extended processing of emotion-rich auditory signals (i.e. the LPP). One work in particular (Erhan et al., 1998) investigated late potentials during the auditory processing of nonsense syllables (e.g. ba, pa) voiced with different emotional intonations. This study revealed significant late (>1000ms) negative over positive amplitude differences for prosodic sounds. If the LPP is sensitive to emotional information, then it stands to reason that complex emotionally evocative sounds (e.g. a car crash or children playing) should produce similar late potentials.

In the current study, we conducted two independent experiments whereby participants were presented with emotional stimuli to a unimodal sensory input (vision or auditory) or with combined sensory modalities. While previous studies have leveraged arousal (Schupp et al., 2000), valence (Yuan et al., 2007) and motivational intensity (Weinberg & Hajcak, 2010) as defining features that enhance the LPP, we used valence as a fixed condition to facilitate the simplest discriminating feature that could be commonly interpreted across modalities. These experiments revealed that emotional sounds both elicit and modulate neural computations associated with image viewing, demonstrating that the LPP represents a multi-modal response to affectively relevant stimuli.

Experiment 1

In Experiment 1, we predicted that the LPP for positive and negative valenced stimuli would differ from each other, as well as from less valenced stimuli. Moreover, we also expect temporal differences between sensory modalities: with earlier differentiation for images, later differentiation for sounds, yet unknown differentiation for mixed modalities.

Method

Participants

Participants were recruited from the University of New Mexico subject pool. Students received class credit for participation. The group consisted of 33 participants (19 females) with a mean age of 19.86 (SD = 2.98). Participants were excluded from participation if they met any of the following criteria: history of head injury that resulted in loss of consciousness for more than five minutes, history of epilepsy, history of any psychiatric or neurological disorder, or currently on any psychiatric or neurological drugs. The Institutional Review Board of the University of New Mexico approved the study protocol.

Materials

The stimulus material consisted of 80 positive and 80 negative images selected from the International Affective Picture System (IAPS) database (Lang et al., 2008) as well as 40 positive and 40 negative sounds selected from the International Affective Digitalized Sounds (IADS) database (Bradley and Lang, 2007). In order to identify experimental stimuli sets that were matched for ratings of arousal and valence between the two valence types, we used an algorithm to equate normative valence and arousal ratings for the selection of our stimuli. First, positive and negative image sets were created using images defined by valence ratings +/− 1 from the mean. Valence was then defined as the absolute distance from the mean, and the matching algorithm contrasted these two sets in two dimensional space by taking the Euclidean Distance of each rating pair based on the Pythagorean theorem (sqrt(Valence²+Arousal²)). The best 160 matches between positive and negative sets were then kept (Figure S1 & Table S1). IADS stimuli were selected via the same manner as the IAPS images. Finally, a computer algorithm randomly selected visual and auditory stimuli into one of the three presentation conditions (image only condition, sound only condition, valence matched audio-visual condition) for each participant.

Procedure

Six practice trials of each experimental condition (image-only, audio-only, and audio-visual) were presented before the main experiment was started. Visual, audio, and audio-visual trials were presented in randomized order that differed between all participants. After each presentation of the emotional stimuli, participants were asked to rate the stimuli on separate Self Assessment Manikin (SAM; Bradley and Lang, 1994) rating scales to indicated how pleasant (1 – 9) and how arousing (1 – 9) they found each stimuli using the corresponding keyboard button. For audio-visual conditions, participants were asked to rate the image only.

During experimental trials, all emotional stimuli were presented for 5000ms (the duration of IADS stimuli). For sound only conditions, participants were presented with a fixation cross for the duration of the sound presentation. For audio-visual trials, image and auditory presentation co-terminated at 5000ms. Overall, the experimental portion consisted of 240 trials (40 positive images, 40 negative images, 40 positive sounds, 40 negative sounds, 40 positive audio-visual pairs, and 40 negative audio-visual pairs) and took an average of 55 minutes.

Data acquisition and preprocessing

Electrophysiological data were collected with a 64Ag–AgCl electrodes embedded in a stretch-lycra cap with a sampling rate of 500Hz with low and high cutoffs at .01–100 Hz. CPz served as the reference electrode and FPz as the ground electrode. Data was recorded with a Brain Vision system (Brain Products GmbH, Munich, Germany). Vertical electrooculogram (VEOG) activity generated by blinks was recorded by two auxiliary electrodes placed superior and inferior to the left pupil.

EEG data were offline re-referenced to an average reference. Very ventral electrodes (FT9, FT10, TP9, TP10) were then removed, as they tended to be unreliable. Data were then epoched around stimulus onset (−2000 to 9,000ms). Data were then visually inspected to identify bad channels to be interpolated, and bad epochs to be rejected. Trials contaminated with ocular activity or muscular artifact greater than 100 μV was identified and rejected via an Independent Component Analysis (ICA)-based correction process from EEGLab, FASTER (Nolan et al., 2010) before averaging. Following ICA, experimental epochs were cut down to only include the stimuli presentation duration (−500 to 5000ms).

Afterwards, all remaining EEG data segments for each experimental condition were baseline corrected (−200 to 0 ms before stimulus onset) and averaged to calculate event related potentials (ERP). ERPs were averaged over posterior electrode sites (Pz, P1, P2, P3, P4, CPz, CP1, CP2, CP3, CP4, POz, PO3, PO4) and then quantified into three temporal windows based on waveform morphology common to all conditions following the image onset; a temporal window around the early LPP (400 – 1100ms), middle LPP window (1100 – 3000ms) and a late LPP window (3000–5000ms).

Statistical analyses

Greenhouse-Geisser adjusted ANOVAs and planned comparison decompositions were used for data analyses. Valence was the only main or interaction effect of theoretical interest. ANOVA effect sizes were reported as partial-η², while planned comparison effect sizes were reported as d.

Due to the dichotomous nature of the data selection process (matched positive and negative stimuli), this condition can be understood as a fixed effect (Maxwell & Delaney, 2003), whereby we assume experimental stimuli to relate to one of two dimensions of valence (i.e. positive and negative). Therefore, we initially tested these experimenter-defined conditions of positive and negative stimuli. Note that neutral stimuli were not included in these conditions, as it is difficult to find affectively neutral sounds since the tonal characteristics of realistic sounds can be strongly valenced. For this analysis a 2(valence) × 3 (modality) × 3 (time) repeated measures ANOVA was conducted. Significant interactions were then followed up within each modality. We report these fixed condition analyses in the Supplement (Figure S2), and describe the idiosyncratically informed condition analyses (detailed below) in the main text.

In order to provide a more sensitive test of affective experience, we leveraged individual ratings to derive idiosyncratically meaningful discrimination between conditions. This method of stimuli selection was motivated in part by a lack of standardized affectively neutral sounds (imagine how off putting 5 seconds of a dripping faucet or restaurant scene would be). Sounds characterized as neutral from the IADS were used during piloting and during the practice section of the experiment and were found to elicit varying ratings of valence (however, ratings of arousal were consistently low). These varied ratings of valence for seemingly neutral sound stimuli motivated our decision to use the idiosyncratic ratings from each participant for condition creation. This individual differences approach facilitated the creation of high-intensity negative and positive stimulus sets as well as low-intensity valence-neutral stimulus set. We discriminated positive and negative trials by applying a median split for ratings of valence for each sensory modality (positive trial = top half of the median split while negative trials = bottom half of median split). We then computed the Euclidean Distance in order to find the 20 most valence-intense stimuli in each valence set. Finally, we identified the 20 least valenced and arousing stimuli (the 10 least positively valence-intense stimuli and the 10 least negatively valence-intense stimuli). The procedure correctly identified these least valence-intense stimuli and was then employed for each participant, individually, in order to create an idiosyncratic neutral stimuli set for each modality. Figure 1 displays mean ratings of arousal and valence across participants for individual stimuli used for the idiosyncratically informed analysis. We analyzed this model with a 3(valence) × 3(modality) × 3(time) repeated measures ANOVA with all follow up comparisons split by modality.

Figure 1.

Experiment 1: grand means for ratings of valence, absolute valence (difference from ‘5’), and arousal. Participant’s subjective ratings along these two dimensions of emotional experience were used to create three de novo experimental conditions (see legend).

Results

ERP results

Figure 2 depicts the grand average waveforms for stimulus-locked ERPs based on idiosyncratic ratings. Time window-specific LPP amplitude means and standard deviations are presented in Table S3. This analysis revealed a significant 3(valence) × 3(modality) × 3(time window) interaction (F_{3.54, 113.24} = 3.48, p = .013, η² = 0.1). To follow up this significant interaction a series of analyses were conducted comparing mean ERP amplitudes for valence conditions across time window for each sensory modality separately.

Figure 2.

Experiment 1: EEG activities by modality condition. LPP amplitudes were enhanced for negative > neutral images and sounds. For paired audio-visual conditions, negative valence was larger than positive and neutral conditions in an early time window. Partial η² effect sizes for the main effect within each modality are presented for each temporal window. Topographic plots display scalp potentials for emotion stimuli (positive + negative divided by 2) minus neutral stimuli (scaled: +/− 2 μV). † = p < .10; * = p < .05; ** = p < .01; *** = p < .001

Image

A 3(valence) × 3(time) repeated measures ANOVA was conducted on LPP modulations for emotional picture viewing. This analysis revealed a significant main effect for valence (F_{1.84, 58.78} = 3.83, p = .031, η² = .11), whereby negative images elicited larger LPPs across all time windows than neutral images (t₃₂ = 2.42, p = .021, d = .42), and marginally larger than LPPs for positive images (t₃₂ = 1.83, p = .077, d = .32). There was no significant valence by time interaction. These results support earlier work (Cuthbert et al., 2000; Dennis and Hajcak, 2009; Dillon et al., 2006; Franken et al., 2009; Hajcak and Olvet, 2008; Horan et al., 2013; Liu et al., 2015; Wang et al., 2011) suggesting that more intense emotional images (especially negative stimuli) evoked larger LPP amplitudes than stimuli that are rated as more neutral.

Sound

The 3 (valence) × 3 (time) ANOVA revealed a marginally significant main effect for valence (F_{1.88, 60.09} = 2.69, p = .08, η² = .08) as well as a significant valence by time interaction (F_{2.17, 68.58} = 3.46, p = .033, η² = .10). Planned comparison ANOVAs tested for valence effects within each of the three LPP time windows. ANOVAs over the early and middle LPP failed to reveal a significant difference between LPP amplitudes for the three valence types (F_{1.93, 61.84} = .11, p = .893, η² = .003; F_{1.87, 59.76} = 2.08, p = .137, η² = .06, respectively), but in the late time window there was a significant main effect of valence (F_{1.82, 58.08} = 4.02, p = .027, η² = .11). Planned comparison t-tests revealed that sound-evoked LPP amplitudes were significantly larger for negative sounds than neutral sounds (t₃₂ = 2.55, p = .016, d = .44), marginally larger for negative sounds than positive sounds (t₃₂ = 1.79, p = .083, d = .32), and there was no significant difference between positive and neutral sounds (t₃₂ = 1.05, p = .30, d = .19).

Audio-Visual

For valenced matched audio-visual pairs, there was no main effect for valence (F_{1.76, 56.36} = 1.18, p = .313, η² = .04). There was, however, a significant valence by time interaction (F_{2.18, 69.74} = 3.43, p = .034, η² = .10). Planned comparisons did not reveal a main effect of valence during the middle or late LPP time windows (F_{1.72, 55.05} = 1.11, p = .33, η² = .03; F_{1.95, 62.38} = 1.69, p = .193, η² = .05, respectively), yet there was a main effect in the early LPP time window (F_{1.45, 46.34} = 5.58, p = .013, η² = .148). Multiple comparison t-tests revealed that early LPP amplitudes evoked during the presentation of negative audio-visual pairs was significantly greater than positive audio-visual pairs (t₃₂ = 4.4, p < .001, d = .77) and neutral audio-visual pairs (t₃₂ = 2.44, p = .02, d = .43). Finally, there was no significant difference between positive and neutral audio-visual evoked LPP’s (t₃₂ = .12, p = .91, d = .02).

Discussion

In Experiment 1 we found that sound valence caused a similar increase in LPP amplitude as pictures (albeit later in time), and combined modalities had larger and earlier differentiation between the two valence conditions. Although the difference in the valence contrast effect sizes between modalities was minimal, this was still a surprising finding. The results of Experiment 1 support our first hypothesis stating that we would observe similar valence trends for LPP modulations (valenced > neutral) across modalities; however, these modulations would be temporally different. We suspect that these later ERP components reflect visual vs. auditory temporal processing differences, whereby sound affect unfolds slower than visual affect. Our second hypothesis that the LPP would be modulated in line with a negativity bias was also supported. Very few studies have tested for a negativity bias in LPP, yet those that have tested for it tend to find significant effects (See Table S2). The results from the fixed condition analysis demonstrate a negativity bias (negative over positive) for LPP amplitudes elicited by emotional picture processing while, surprisingly, ratings of arousal (a proxy of motivational intensity; Harmon-Jones, Gable & Price, 2013) were higher for positive pictures than negative pictures.

These outcomes reveal two novel findings. First, the LPP disassociations for sound-only stimuli suggest that complex emotional sounds produce and modulate the LPP. This suggests that LPP, long believed to relate to emotional images, may be a domain-general computation that is generically related to affective phenomena (i.e. pictures and sounds). The second novel finding in Experiment 1 is the larger differentiation between negative and positive evoked potentials elicited in the paired audio-visual stimuli condition. This effect is also in line with the suggestion that the LPP reflects a domain-general computation that is generically related to affective phenomena (Ito & Cacioppo, 2005).

Recently, Brosch and colleagues (2009) provided evidence that very early neural modulations (P1 amplitudes) recorded from visual areas were enhanced during emotional sound presentation but not for non emotional sounds. They posit that affect-modulated sounds prime the attention center of the visual cortex faster than non emotional sounds and consequently, lead to better processing of the target. This line of research may help to support the enhanced neural activities shown from the audio-visual results in Experiment 1. This finding suggests that multiple modalities boost affective processing above and beyond evocative visual stimuli. The findings from studies investigating multimodal processes have suggested that a kind of sensory modality “cross-talk” between multiple modalities not only influences one another (Beauchamp et al., 2004; de Gelder and Vroomen, 2000; Lerner et al., 2003; McGurk & MacDonald, 1976; Keil et al., 2012; Setti et al., 2013; Spence & Squire, 2003; Stein & Stanford, 2008; van Wassenhove et al., 2007), but also expedite processing speeds (Dalton et al., 2000; Rowland et al., 2007). Taken together, these findings suggest that emotion processes share similar integrative features as reflected by the LPP, and that this reflection of emotional signals may be strengthened when affective information is presented to multiple sensory modalities.

Experiment 2

Experiment 1 revealed that paired pictures and sounds elicited larger effect sizes between positive and negative stimuli pairs earlier in time than each modality alone. While this effect may be due to enhancement of the affective experience, it may also be an artifact of increased processing demand. In order to investigate this question, we tested whether these modality (audiovisual boost) and valence (negativity bias) effects interacted with each other by manipulating the pairing of audio-visual valences. Based on findings suggesting the modulatory role that emotion intensity has shown to influence LPP amplitude (Hajcak et al., 2010), we predicted that both arousal and LPP amplitude would be greater for valence conflict (positive image + negative sound, negative sound + positive sound). Second, we predicted that self-reported ratings of stimuli pairs will be modulated by conflict and non conflict audio-visual pairs, whereby conflict pairs will lead to higher levels of arousal ratings but lower levels of absolute valence ratings (c.f. Brown, 2014). Furthermore, we hypothesized that perceived conflict for mismatched emotional audiovisual pairs may elicit enhanced frontal midline theta activity. Cavanagh and Frank (2014) have proposed that frontal midline theta acts as a marker for cognitive control. This signal of control is ultimately modulated while an agent attempts to evaluate conflicting states in an uncertain environment. When presented emotionally conflicting audio-visual pairs, we expected these systems of control would be engaged in order to better aid the processing of these conflicting stimuli.

Method

Participants were 43 students (32 female) recruited from the University of New Mexico subject pool, with a mean age of M = 21.02 (SD = 3.52). Inclusion and exclusion criteria were the same as Experiment 1. IAPS images (160 positive, 160 negative) were chosen using the same algorithm as Experiment 1. These images were then randomly placed into two algorithmically matching experimental sets and were paired with a similarly chosen 80 (40 positive, 40 negative) emotional sounds. Following four practice trials where participants were presented neutral images (shapes) paired with example IADS sounds (a man yawning and the strumming of a harp), participants were presented 80 valence-congruent and 80 valence-incongruent stimuli pairs in random order. The experiment thus had four experimental conditions: positive-image+positive-sound, positive-image+negative-sound, negative-image+positive-sound, negative-image+negative-sound. In this 2 × 2 design, a cross-over interaction would support the hypothesis that an organism’s affective experience would be modulated by the congruency of valence specific audio-visual pairs. Statistical interactions were followed up with planned comparison t-tests. Since audio-visual pairs effectively modulated the LPP in <3000ms in Experiment 1, we omitted the third temporal window (3000ms – 5000ms), and thus stimuli were presented for 3000ms. As with Experiment 1, limitations in the size of the sound library required that each sound stimulus needed to be presented twice, once with an incongruent valenced image (e.g. a positive image paired with a negative sound) and once with a congruent valenced image (e.g. a negative image paired with the same sound). Note that specific picture and sound pairings were always randomized between participants. EEG data preprocessing, temporal windows (400 – 1100ms and 1100 – 3500ms), and acquisition methods were the same as Experiment 1. Time-Frequency measures were computed using custom-written Matlab functions (Cavanagh et al., 2009) by multiplying the fast Fourier transformed (FFT) power spectrum of single trial EEG data with the FFT power spectrum of a set of complex Morlet wavelets (defined as a Gaussian-windowed complex sine wave: e^i2πtfe^{-t^2/(2xσ^2)}, where t is time, f is frequency (which increase from 1-50Hz in 50 logarithmically spaced steps), and defines the width (or ‘cycles’) of each frequency band, set according to 4/(2πf)), and taking the inverse FFT. The end result of this process is identical to time-domain signal convolution, and it resulted in estimates of instantaneous power (the magnitude of the analytic signal), defined as Z[t] (power time series: p(t) = real[z(t)]² + imag[z(t)]²). Each epoch was then cut in length (−250 to 1,750ms). Power was normalized by conversion to a decibel scale (10 × log10[power(t)/power(baseline)]), allowing a direct comparison of effects across frequency bands. Based on previous literature, the region of interests for time frequency analysis for detecting conflict and control signals was set a priori over frontal midline site (FCz) and the analytic time window150 - 450ms after audio-visual stimulus onset. The experiment took an average of 32 minutes.

Results

Self-reported ratings

Similar to Experiment 1, ratings of absolute valence and arousal were compared across experimental conditions (again, participants were asked to rate the image only; see Figure 3). Table S4 displays means and standard deviations for self-report ratings. Ratings of absolute valence were analyzed by a 2 (image) × 2 (sound) repeated measure ANOVA, where an interaction would indicate an effect of the modality-valence congruency manipulation. This analysis revealed a significant main effect for image type (negative > positive; F_{1, 42} = 85.36, p < .001, η² = .67), yet no main effect for affective sounds (F_{1, 42} = .11, p = .739, η² = .003). There was a cross-over interaction, (F_{1, 42} = 14.34, p < .001, η² = .255), whereby congruent audio-visual pairs were rated as more valenced than conflicting pairs, in contrast to the hypothesis that these would be less valenced.

Figure 3.

Experiment 2: grand means for ratings of valence, absolute valence (difference from ‘5’), and arousal. Findings include main effects for absolute valence ratings (negative > positive) and arousal ratings (positive > negative).

A second 2(image) × 2(sound) repeated measures ANOVA was conducted on ratings of arousal. There was a main effect of image (positive > negative; F_{1, 42} = 30.57, p < .001, η² = .421), sound (positive > negative; F_{1, 42} 11.71, p = .001, η² = .218), and a significant interaction (F_{1, 42} = 17.09, p < .001, η² = .289). Planned comparison t-tests revealed that ratings of arousal were lower for positive images when they were paired with negative sounds compared to positive sounds (t₄₂ = 4.37, p < .001, d = .67). Together, these findings replicate findings from Experiment 1 such that negative (vs. positive) stimuli were perceived as more polarized on a good-vs. bad dimension, whereas positive (vs. negative) stimuli were perceived as more arousing.

Time frequency results

Figure 4 depicts the grand averages for stimulus-locked time frequency decomposition, revealing a common, early burst of theta band power over mid-frontal electrodes. A 2(image) × 2(sound) repeated measures ANOVA was conducted on frontal midline theta power (4-8Hz). There was no main effect of image (F_{1, 42} = .28, p = .60, η² = .007), but there was a main effect for sound (F_{1, 42} = 4.11, p = .049, η² = .09), where theta power for positive sounds was larger than negative sounds. There was also a significant cross-over interaction (F_{1, 42} = 4.58, p = .038, η² = .10), whereby conflicting pairs were greater than congruent pairs. This significant interaction suggests very early emotion signal detection for congruent and incongruently valenced audio-visual stimuli when that stimulus is presented to multiple sensory modalities. Furthermore, it provides evidence that the mixed valence manipulation was experienced as incongruent.

Figure 4.

Experiment 2: Time-frequency power plots from the FCz electrode. (A) Time-frequency power collapsed across all experimental conditions. Early (< 500ms) midline frontal theta band power (4-8 Hz) is evident across all experimental conditions. (B) Theta power envelope. frontal midline theta power peaked about 300ms after audio-visual stimuli onset. (C) Line graphs for region-of-interest over the theta burst (150 – 450ms), split by condition. There was a significant interaction between experimental conditions where stimulus pairs with conflicting picture and sound valence had significantly higher theta power than affectively congruent stimuli pairs. * p < .05

ERP results

Figure 5 depicts the grand average waveforms for stimulus-locked LPP modulations between sensory modalities. Time window-specific LPP amplitude means and standard deviations are presented in Table S5. A series of 2(image) × 2(sound) repeated measures ANOVAs were conducted on centro-pariatal ERP amplitudes for audio-visual stimuli pairs for each of the two specified temporal windows (early and late LPP). As predicted, there was a main effect of image (negative > positive) in the early LPP time window (400 – 1100ms; F_{1, 42} = 7.35, p = .01, η² = .149), replicating the findings from Experiment 1. However, there was no main effect for sound (F_{1, 42} = .11, p = .748, η² = .002) and no interaction (F_{1, 42} = 2.65, p = .111, η² = .059). There were no main effects of image or sounds in the late LPP (1100-3500ms; F_{1, 42} = .25, p = .671, η² = .006; F_{1, 42} = 1.08, p = .304, η² = .025, respectively), but there was a significant interaction (F_{1, 42} = 4.29, p = .044, η² = .093), whereby congruent pairs were larger than conflict pairs, in contrast to our original hypothesis.

Figure 5.

Experiment 2: ERPs time-locked to the presentation of emotional audio-visual pairs, collapsing across auditory conditions. Negative images, regardless of sound condition, led to a main effect of enhanced amplitudes at the early LPP temporal window (400 – 1100ms). In the late LPP window, there was an interaction between affective modalities, but no main effects. Topographical plots display scalp potentials for negative visual stimuli minus positive visual stimuli (scaled +/− 2 μV). ** p < .01 * p < .05

Discussion

Experiment 2 revealed an expected main effect of valence for the visual modality in early LPP activities, replicating findings of a negativity bias from Experiment 1. Our initial hypothesis that valence-conflicting pairs would be rated as more arousing than congruent pairs was not supported. In fact, ratings of arousal for positive images were significantly attenuated when they were paired with valence-conflicting negative sounds. This significant interaction was also observed in the EEG, albeit with differing effects by neural indicator. Frontal midline theta was enhanced for valence conflict, yet late LPP amplitudes were smaller for valence conflict. While this latter finding was unexpected, together these findings demonstrate that complex multimodal affective meaning is widely represented across a variety of neural measures as time unfolds. These findings bolster the conclusions from Experiment 1 that neural processes sensitive to visual stimuli may be predictably modulated by the inclusion of sound.

General Discussion

In the present study, we provide evidence showing that (1) LPP dynamics are influenced by emotional sound processing and (2) image processing is influenced by the accompaniment of valence specific sounds. This study provides evidence that LPP modulation is sensitive to valence-specific stimulus integration within image and sound conditions. While previous reports have clearly noted an LPP modulation by visually salient information, it was not clear what process this signal represented. The findings reported here suggest that LPP amplitude reflects a meaning-related modulation that is independent of the visual modality.

In Experiment 1, there was an unexpected finding of valence-specific differentiation for audio-visual stimuli pairs in the early LPP. This finding did not indicate if this differentiation was due to either an enhancement of the affective experience or the doubled processing demand. In Experiment 2, we manipulated the valence of audio-visual pairings to dissociate affective experiences and discovered that neural processing for visual stimuli is sensitive to valence specific auditory stimuli, suggesting a multimodal induced affective experience enhancement. These auditory and multi-modal effects are novel and suggest that the LPP is not only a signal of visual attention, but that it reflects higher-order affective integrative processes regardless of the sensory modality being exposed.

This set of experiments is unique in that we matched conditions by normative ratings of valence and arousal for stimuli selection and individually tailored conditions sets per participant. A limitation of the current study was the absence of any “neutral” stimuli. However, we believe we addressed this issue by using idiosyncratic ratings provided by each participant (positive conditions = high arousal + high valence; negative conditions = high arousal + low valence, neutral conditions = low arousal + low absolute valence). These ratings allowed us to create valence conditions tailored to each individual participant, thus providing us with more generalizable effects. We analyzed the conditions with both fixed and idiosyncratic condition groupings, with highly similar outcomes between the two. This matching procedure may explain the unexpected finding that positive stimuli had larger ratings of arousal whereas negative stimuli had larger ratings of absolute valence. This finding was replicated between the two studies, yet LPP amplitudes were uniformly larger for negative valence between modalities and studies.

While this finding may seem to challenge the widespread notion that LPP is primarily sensitive to arousal (Schupp et al., 2000), a negativity bias has been observed as well (Table S2). These constructs are not orthogonal in any case (Lang et al., 2008), as demonstrated in both these experiments by larger valence ratings for negative stimuli yet larger arousal ratings for positive stimuli. These co-linear facets more likely reflect a common feature of motivational intensity (Weinberg & Hajcak, 2010). In this report, we leveraged the simplest and thus most parsimonious affective dimension (valence) to contrast affective modulation of the LPP across modalities. While we (and others) have provided evidence that the LPP is sensitive to a negativity bias in valenced stimuli, we do not think that this is the specifically defining characteristic of LPP modulation, and we expect future studies to demonstrate a sensitivity of the LPP to higher-order affective dimensions (Lindquist et al., 2012).

In summary, we found that complex affective sounds meaningfully modulate the LPP, either alone or in combination with affective images. These findings suggest that the LPP is sensitive to general parietal integrative processes and is influenced by multi-modal emotional stimuli. Audio-visual stimuli are likely to act as a closer analog of ecologically valid affective sources and are an important methodological stepping-stone. A closer understanding of the primitive processes that contribute to LPP generation will help achieve the dual aims of understanding affect in terms of mechanistic sub-processes during highly controlled, yet realistic, affective events. In conclusion, the LPP reflects a domain-general computation that is generically related to affective phenomena.