Representing the Good and Bad: fMRI signatures during the encoding of multisensory positive, negative, and neutral events

Abstract

Few studies have examined how multisensory emotional experiences are processed and encoded into memory. Here, we aimed to determine whether, at encoding, activity within functionally-defined visual- and auditory-processing brain regions discriminated the emotional category (i.e., positive, negative, or neutral) of the multisensory (audio-visual) events. Participants incidentally encoded positive, negative, and neutral multisensory stimuli during event-related functional magnetic resonance imaging (fMRI). Following a 3-hour post-encoding delay, their memory for studied stimuli was tested, allowing us to identify emotion-category-specific subsequent-memory effects focusing on medial temporal lobe regions (i.e., amygdala, hippocampus) and visual- and auditory-processing regions. We used a combination of univariate and multivoxel pattern fMRI analyses (MVPA) to examine emotion-category-specificity in mean activity levels and neural patterning, respectively. Univariate analyses revealed many more visual regions that showed negative-category-specificity relative to positive-category-specificity, and auditory regions only showed negative-category-specificity. These results suggest that negative emotion is more closely tied to information contained within sensory regions, a conclusion that was supported by the MVPA analyses. Functional connectivity analyses further revealed that the visual amplification of category-selective processing is driven, in part, by mean signal from the amygdala. Interestingly, while stronger representations in visuo-auditory regions were related to subsequent-memory for neutral multisensory stimuli, they were related to subsequent-forgetting of positive and negative stimuli. Neural patterning in the hippocampus and amygdala were related to memory for negative multisensory stimuli. These results provide new evidence that negative emotional stimuli are processed with increased engagement of visuosensory regions, but that this sensory engagement—that generalizes across the entire emotion category—is not the type of sensory encoding that is most beneficial for later retrieval.

1. Introduction

Emotional experiences are inherently multisensory (e.g., Robins, Hunyadi, & Schultz, 2009; Klasen, Chen, & Mathiak, 2012; Klasen, Kreifelts, Chen, Seubert, & Matrhiak, 2014). In everyday life, emotional reactions are often elicited by events that include both visual and auditory information: We hear a crash as two cars collide, or we hear laughter as children play together. Memories of those moments can similarly evoke multisensory re-experiencing: we may hear the echo of the laughter as well as the visions of the smiling faces. Yet the vast majority of research examining the neural processes underlying human’s reactions to—and episodic memories for—emotional stimuli has used visual stimuli. Indeed, even the study of memory for neutral materials has generally focused on the visual domain. For example, two meta-analyses of functional magnetic resonance imaging (fMRI) studies employing univariate analyses examined the neural correlates associated with the successful encoding of emotional episodic memories (Murty, Ritchey, Adcock, & LaBar, 2010; Dahlgren, Ferris, & Hamann, 2020). Consistent neural activity was identified in the medial temporal lobe, namely the amygdala and hippocampus, among other regions. These findings are consistent with the well-known memory enhancing effects of emotional arousal and the role of amygdala in enhancing neural activity related to memory encoding in the medial temporal lobe (e.g., Hamann, 2001; Kensinger, 2004; Phelps, 2004; Kensinger & Schacter, 2008). However, the large majority of the stimuli included in the meta-analyses were visual stimuli (e.g., printed words, objects, faces, scenes, scrambled scenes, or line drawings of scenes). This focus has left open many questions about how multimodal stimuli are initially processed and later remembered, and about how the emotion category (i.e., whether the event is positive, negative or neutral) may affect certain aspects of processing.

The majority of research examining how emotion category is represented in the brain has focused on the amygdala (e.g., Jin, Zelano, Gottfired & Mohanty, 2015; Namburi, Al-Hasani, Calhoon, Bruchas, & Tye, 2016), or other known emotion-processing regions, such as the medial prefrontal cortex (Kim, Wan, Wedell, & Shinkareva, 2016). However, several recent studies suggest that emotional content is encoded (Mickley & Kensinger, 2008) and reinstated (Bowen & Kensinger, 2017; Kark & Kensinger, 2015; Kark & Kensinger, 2019) in sensory processing regions, such as the ventral visual stream (for reviews see: Bowen, Kark, & Kensinger, 2018; Kensinger & Ford, 2020). Interestingly, emotional auditory stimuli enhance neural activity in auditory cortex compared to neutral auditory stimuli (Viinikainen, Katsyri, & Sams, 2012), early cortical processing of visual stimuli (Gerdes et al., 2013) and subjective emotional experience of emotional stimuli (Gao, Wedell, Kim, Weber, Shinkareva, 2018). Yet, it remains unknown how multisensory stimuli impact reinstatement of neural traces during memory retrieval. Here, we focus on how emotion category is represented within regions that process visual or auditory information.

There is intriguing evidence that negative memories may be particularly likely to be re-experienced with visual detail and that “flashbacks” of negative events are more likely to include visual detail rather than details from other modalities (Ehlers & Steil, 1995; van der Kolk & Fisler, 1995). The sparse literature available to elucidate how emotion-category-specificity affects multimodal processing suggests the intriguing possibility that, rather than supporting cross-modal connections, negative emotionality may bias processing toward the modality that confers the emotional content (e.g., Scherer & Larsen, 2011) or, in the case of multiple modalities conveying affect, toward the visual modality (e.g., Spreckelmeyer, Kutas, Urbach, Altenmuller, & Munte, 2006; Klasen, et al., 2014; Gao et al., 2018). These findings raise the interesting possibility that negative emotionality may not enhance all sensory processing – or at least may not enhance the likelihood that all sensory content is encoded into a memory – a possibility that can be directly assessed by examining not only how well sensory regions represent emotion category information during event experience but also how well that the strength of that representation varies as a function of memory.

Based on this literature, we hypothesized that visuo-sensory regions would be particularly sensitive at coding for emotion category, and more so than auditory regions. We also hypothesized that the ability to represent emotion-category-related information would be stronger when information was subsequently remembered. However, an interesting possibility—consistent with evidence that negative memories tend to be associated with more visual re-experience than re-experience in other modalities—is that visual sensory processing may be particularly related to the processing and encoding of negative events (e.g., Mickley & Kensinger, 2008; Bowen et al., 2017; Kark & Kensinger, 2015, 2019), while auditory sensory processing may be less related.

To address these hypotheses, participants performed a sensory functional localizer scan that allowed us to identify visual regions and auditory regions in a way that was unbiased with regard to emotionality or to the memory task. During the encoding phase of the memory task, positive, negative, and neutral pictures were presented and paired with related sound-clips and neutral verbal labels. Univariate analyses examined how emotional category (i.e., positive, negative, or neutral) related to the mean signal in the visual or auditory processing regions revealed in the localizer scan. In a complementary fashion to the mean signal analysis, in the MVPA, we computed a metric of emotional-category-selectivity based on the neural patterning associated with each emotion category. We then tested whether emotional-category-selectivity in neural patterning differed in visual or auditory processing regions. We assumed that the emotion-category-specific neural patterning would be an indicator of the representational strength of this information at encoding (cf., Kuhl, Rissman, & Wagner, 2012; Koen, Hauck, Rugg, 2019).

During the retrieval phase of the memory task, studied and novel neutral verbal labels were presented alone (i.e., without the picture or sound). We then examined how encoding-related activity in the a priori sensory regions identified by the localizer related to subsequent memory for each emotional category, using both univariate and multivoxel pattern analysis (MVPA) approaches. In addition to examining visual and auditory processing regions, we also examined effects within the hippocampus and amygdala, two canonical regions associated with the successful encoding of emotional episodic memories (Murty et al., 2010; Dahlgren et al., 2020).

2. Materials and Methods

2.1. Participants

Forty-five participants (aged 19–38 years) consented to participate in the study. All participants had normal or corrected-to-normal vision, were fluent in English, had no history of major medical, neurological, psychiatric or sleep disorders, were not taking any psychoactive medications at the time of study participation, and did not have any contraindications for MRI. Data from seven participants was excluded for the following reasons: 6 participants for an insufficient number of trials (i.e., fewer than 10 trials of hits or misses of a given valence; for other studies using a similar cut-off for MVPA and univariate analyses, see Ritchey, Montchal, Yonelinas, & Ranganath, 2015; Thakral, Wang, & Rugg, 2017; Srokova, Hill, Koen, King, & Rugg, 2020; Carpenter, Thakral, Preston, & Schacter, 2021) and 1 participant who did not complete the retrieval portion of the behavioral task. The remaining sample, consisting of 38 participants, had a mean (± 1 standard error of the mean) age of 24.6 ± 0.75 (23 females). All experimental procedures were approved by the Boston College institutional review board and informed consent was obtained prior to participation. All participants were compensated for their time.

2.2. Experimental procedure

The pictures and verbal labels employed in the current study were taken from the prior study of Ford, Morris and Kensinger (2014). Study stimuli were selected from a set of three-hundred positive, negative and neutral pictures (100 from each emotion category). As detailed in Ford et al. (2014), pictures were selected such that arousal ratings were equated for positive and negative pictures, and positive and negative pictures were higher in arousal than neutral pictures. Each picture was paired with a neutral label that later served as a retrieval cue (e.g., ‘Desert Reptile’; see, Figure 1, left). New to this study design, each picture and neutral label was additionally paired with a semantically-related, emotion-category-matched sound clip, such as a rattle snake rattling its tail (Figure 1, left), a child laughing, or ambient nature sounds. To confirm neutrality of the labels, a pilot study was conducted where five participants viewed all titles and determined whether they were neutral, positive, or negative, and titles were replaced if more than two participants rated them as either positive or negative (for full details, see Ford et al., 2014). Sounds were selected from the International Affective Digital Sounds database (IADS; Bradley & Lang, 2007) and online databases of Creative Commons Licensed sounds (Freesound.org and YouTube.com) and were selected to correspond to the emotion of the scene (i.e., positive, negative, or neutral) and to be thematically appropriate for the scene (e.g., a negative image with a rattlesnake was paired with the sound of a rattlesnake’s tail (Figure 1, left); a positive image of athletes was paired with the sound of a crowd cheering).

Figure 1.

Left. During the encoding/study phase, positive, negative, and neutral pictures were presented and paired with related sound-clips and neutral verbal labels. Right. During retrieval/test phase, studied and novel neutral verbal labels were presented alone (i.e., without the picture or sound) and participants were asked to indicate the vividness of their memory, ranging from 0 (no memory) to 3 (extremely vivid). Participants also completed a separate functional localizer task to identify visual and auditory processing regions. Due to the IAPS usage agreement, a representative negative picture is shown.

We ran a number of analyses to consider the potential effect of stimulus features on subsequent memory and to address whether there is a potential confound between emotionality and other perceptual stimulus features. First, we analyzed the memorability of the stimuli employed at encoding (i.e., the % participants for whom the encoded item was a subsequent hit (vs. miss)). A one-way ANOVA with factor Emotion Category on the memorability data just approached significance (F(2, 297) = 3.14, p = 0.045), as expected given the reported tendency for emotional items to be remembered more than neutral items. We then examined whether the memorability of the emotional items would be tied to individual stimulus features, including visual salience, auditory frequency range, auditory peak frequency, and auditory loudness as measured in root mean square (RMS)). Across all stimuli, although memorability was weakly correlated with frequency range, it did not survive Bonferroni correction for multiple corrections (Spearman’s rho = 0.13, p = 0.03). No other correlations were significant (rhos < 0.05). Correlations were also conducted on the negative, positive, and neutral stimuli separately. For negative stimuli, memorability did not significantly correlate with any of the individual stimulus features (rhos < 0.07). For positive stimuli, memorability was weakly correlated with frequency range but it did not survive Bonferroni correction for multiple corrections (rhos = 0.20, p = 0.04), all other correlations were null (rhos < 0.15). For neutral stimuli, memorability was only correlated with frequency range (rho = 0.27, p < 0.01), all other correlations were null (rhos < 0.08). We also examined whether there were differences in these stimulus characteristics as a function of emotional category, using a one-way ANOVA with a factor of Emotion Category. The ANOVAs on visual salience, frequency range, and peak frequency were all null (Fs(2, 293) < 1.24, ps > 0.20). The ANOVA on the RMS values was significant (F(2, 297) = 5.20, p < 0.01, partial η² = 0.03). Follow-up comparisons revealed that this main effect was driven by greater RMS values for neutral relative to negative stimuli (t(297) = 3.22, p < 0.01, d = 0.46), with all other comparisons not significant, critically between the two emotional categories, positive and negative (ts(297) < 1.63, ps > 0.20). Taken together, we found no clear association between memorability and the stimulus features of the visual pictures and auditory sounds, and critically, we found no differences in stimulus features between the positive and negative stimuli. These findings indicate that stimulus features cannot account for the differences in subsequent memory nor in differences revealed between emotion categories.

One-hundred and fifty of the 300 picture-sound-label triplets were randomly divided into 5 sets of 30 triplets each (10 positive, 10 negative, and 10 neutral). Triplets were ordered pseudo-randomly with triplets of a particular emotion category appearing no more than twice in a row. These five sets of picture-sound-label triplets were employed in each of the 5 encoding fMRI runs described below. To avoid order effects, four versions of the encoding task and two versions of the retrieval task were created, counterbalanced across participants.

Participants completed the study and test phase in the fMRI scanner, with the test phase separated from the study phase by a 3-hour delay that included a 2-hour nap opportunity (nap data not reported here). The study and test phase each were broken into 5 fMRI scan runs, for a total of 10 fMRI runs (5 study fMRI runs followed by 5 test fMRI runs). Prior to each scan, participants were instructed on, and practiced, the task they would complete in the scanner. During the encoding/study phase (Figure 1 left), each study trial comprised the presentation of a single picture, corresponding label, and associated auditory sound for 3 s, following which a 3 s response screen was presented where participants judged the emotionality of the multi-featural stimulus/event on a 5-point scale, ranging from 0 (not emotional) to 4 (very emotional). We informed participants that “emotional intensity” refers to the “instant feeling” the stimulus had on them personally, irrespective of whether the feeling was positive, neutral or negative. Further, we told participants we were interested in their personal reaction, not how they thought other people in general should feel. Thus, emotional intensity ratings were qualitatively similar to the construct of arousal. The trial ended with a jittered fixation screen ranging from 6–12 s. Participants were not instructed that their memory for the stimuli would later be tested, and thus encoding was incidental.

Immediately following the encoding phase, participants exited the scanner and were outfitted with polysomnography and provided a 120-min nap opportunity (these data were collected for an analysis outside the scope of the current study). Following the nap, participants were given a surprise cued-recall test during a second fMRI scan. During the surprise test, 150 neutral labels presented during the study phase were presented as memory cues, randomly interspersed with 150 unstudied/novel neutral labels (300 labels presented in total). Each trial of the test phase (Figure 1, right) comprised the presentation of the neutral cue label for 3 s, followed by a 3 s response screen asking participants to rate the vividness of their memory of the associated picture and sound on a 4-point scale, ranging from 0 (no memory) to 3 (extremely vivid). We informed participants that memory vividness ratings could be based on how vividly they remembered the details of the picture-sound pair and/or how vividly they remembered their reaction or thoughts about the picture-sound pair. Participants selected the 1 of 4 choices, ranging from 0 (i.e., no memory of the cue label) or 1–3 if they had some memory of the content associated with the cue label (i.e., either somewhat vivid, vivid, or extremely vivid memory). Instructions were adapted from vividness ratings used by Kark & Kensinger (2019). The trial ended with a jittered fixation screen ranging from 0–6 s.

Prior to the encoding phase, participants completed a visual-auditory functional localizer task during fMRI. During the localizer, participants cycled through pictures and sounds presented in a blocked design. Each block comprised the presentation of 7 stimuli, presented for 3 s (either picture or sounds presented in isolation) separated by a 21 s fixation period, totaling 3 visual and 3 auditory blocks. Stimuli employed for the localizer task did not overlap those employed in the primary memory task described above.

2.3. Image acquisition and analysis

Functional and anatomic images were acquired on a 3 Tesla Siemens Prisma scanner equipped with a 32-channel head coil. Functional images were acquired with an interleaved multi-slice echoplanar imaging sequence (TR = 1.5 s, TE = 28 ms, matrix size of 104 × 104, field-of-view = 208 mm, 69 slices, 2mm³ resolution). For each encoding run, 315 images were acquired (for one participant, 301 images were acquired for the final run due the participant exiting the scanner early), and for the localizer run, 130 images were acquired. The first 4 functional images of each run were discarded to allow for equilibrium effects and were not included in analyses described below. Anatomic images were acquired with a magnetization-prepared rapid gradient echo sequence (1 mm³ resolution).

fMRI data were analyzed using both a univariate general linear model (GLM) and MVPA. Univariate analyses were conducted using Statistical Parametric Mapping (SPM12, Wellcome Department of Cognitive Neurology, London, UK). MVPA was conducted using the Princeton MVPA Toolbox (https://code.google.com/p/princeton-mvpa-toolbox/) and custom MATLAB scripts.

2.3.1. Univariate analyses

Functional image preprocessing included slice-time correction, two-pass spatial realignment, and normalization into Montreal Neurological Institute (MNI) space (resampled at 2mm). For univariate analyses, functional images were smoothed with a 4 mm full-width-half-maximum Gaussian kernel. Anatomic images were normalized into MNI space using an analogous procedure to that employed for the functional images.

Univariate analyses for the encoding data (concatenated across 5 runs) and localizer data were conducted in a two-step mixed-effects GLM. In the first step, neural activity associated with each event was modeled with a boxcar function from event onset to the reaction time for the emotionality judgment during encoding. For the localizer, each visual and auditory block was modeled with a 21 second boxcar function. The associated blood-oxygen-level-dependent (BOLD) response was modeled via convolution with a canonical hemodynamic response function yielding regressors in a GLM that modeled the BOLD response for each event. For the analyses of the encoding data, there were six events of interest (emotion category [3] by subsequence memory [2]): neutral hits, neutral misses, negative hits, negative misses, positive hits, and positive misses. Subsequent misses were those studied/old trials that went on to be given a 0 or ‘no memory’ response. Hits corresponded to studied/old trials that were given a response other than 0. For the localizer analysis, there were two events of interest: visual and auditory trials. The design matrix for the encoding analysis also included 5 columns to regress out linear drift for the 5 concatenated runs. An AR(1) model was used to estimate and correct for non-sphericity of the error covariance (Friston et al. 2002). Temporal smoothing was conducted before estimation of the parameter estimates using the default high-pass filter of 128 s in SPM12.

In the second step, parameter estimates for the events of interest and for each participant were entered into a repeated measures ANOVA with participants modeled as a random effect. Unless otherwise noted, an individual threshold of p < 0.005 was combined with a cluster extent threshold of 28 voxels to yield a threshold corrected for multiple comparisons of p < 0.05 (Slotnick et al. 2003; Slotnick 2017; for other studies employing the identical individual voxel threshold, see Thakral, Madore, Kalinowski, & Schacter, 2020). This cluster extent was computed using a Monte Carlo simulation with 10,000 iterations. The Monte Carlo simulation modeled activity in each voxel using a normally distributed random number (mean of zero and unit variance) and Type-I error was assumed to be equal to the individual voxel threshold p-value (p < 0.005) in a volume defined by the functional acquisition dimensions. Spatial correlation was simulated by smoothing with a 5.25 mm FWHM Gaussian, which was estimated using the residual mean-square image of the first level model of the encoding analysis. To evaluate effects within the hippocampus, a mask was created by manually tracing the hippocampus using the across-participant mean normalized anatomical image based on standard anatomical landmarks (Frisoni et al., 2015; for similar approaches, see Thakral et al., 2015, 2020). Correction for multiple comparisons (to p < 0.05) was affected by imposition of a cluster extent threshold of 8 voxels within the hippocampal mask computed using the same Monte Carlo simulation described above.

2.3.2. Multi-voxel analyses

2.3.2.1. Feature selection

MVPA analysis was conducted to complement the univariate analysis described above. MVPA was used to assess whether neural patterning in regions sensitive to visual and auditory processing code for the subsequent memory of emotional events, and if this differs as a function of emotion category. Thus, MVPA was conducted within regions associated with the multi-featural events participants encoded: 1) two sensory ROIs (i.e., visual and auditory ROIs) identified by the localizer data and 2) two a priori chosen anatomical ROIs (i.e., the hippocampus and amygdala). The sensory features used for the MVPA were the voxels that showed the largest mean signal differences across the two modalities (visual and auditory) and emotion categories (positive, negative, neutral) as estimated from the univariate GLM analysis described above. To identify visual and auditory-selective voxels, the top 500 voxels showing the largest t values for the visual > auditory and auditory > visual contrasts were identified (Figure 2).

Figure 2.

The visual/auditory features used for the MVPA. The top 500 voxels showing the largest t values for the visual > auditory (green) and auditory > visual (red) contrasts as identified from the localizer data.

Given the known link between the hippocampus and amygdala and the successful encoding of emotional information (e.g., Murty et al., 2010; Dahlgren et al., 2020), MVPA was also conducted within these two areas. The hippocampus and amygdala were defined as the left and right hippocampus and left and right amygdala labels of the Automated Anatomical Labeling (AAL) atlas (Tzourio-Mazoyer et al., 2002) as implemented in the WFU PickAtlas Tool (Maldjian, Laurienti, Kraft, & Burdette, 2003).

2.3.2.2. Multi-voxel pattern similarity analyses

For the purposes of the MVPA, for each individual participant, the unsmoothed data from the five study scans were concatenated and subjected to a least-squares-all GLM to estimate the BOLD response to individual trials (Rissman, Gazzaley, D’Esposito, 2004; Ritchey, Wing, LeBar, & Cabeza, 2013; Mumford, Davis, Poldrack, 2014). Each event was modeled as a boxcar function as described above and convolved with a canonical HRF. The design matrix also included 5 columns to regress out linear drift for the 5 concatenated runs.

The MVPA procedure, which is illustrated in Figure 3, was conducted in a similar fashion to prior subsequent memory studies (e.g., Kuhl et al., 2012; LaRocque et al., 2013; Koen et al., 2019; Srokova, et al., 2020). The similarity measures were derived from the single-trial beta estimates from the least-squares-all GLM, and based on Fisher z- transformed Pearson’s correlations. On an individual participant basis and for each ROI, a within-category similarity metric was computed for each emotion category. This was computed as the correlation across voxels between each encoding trial and all other encoding trials belonging to the same emotion category (i.e., r_within), with the resulting correlation averaged across all trials. A between-category similarity metric was computed as the correlation between encoding trials belonging to different emotion categories (i.e., r_between), with the resulting correlation averaged across all trials. Both r_within and r_between correlations were computed across trials from different encoding scans to avoid potential bias due to temporal autocorrelation (Mumford et al., 2014). A similarity index was computed as the difference between the r_within and r_between correlations. Following the logic of prior studies (e.g., Liang, Wagner, & Preston, 2013; Koen et al., 2019; Srokova et al., 2020), the similarity index is a metric of neural selectivity as it reflects the extent to which different emotion categories evoke consistent patterns of neural activity. To the extent that a given ROI is sensitive to a given emotion category, within category correlations should be greater than the corresponding between category correlation (i.e., a positive similarity index), as the correlation is greater across events sharing the same emotionality, relative to those events differing as a function of emotion category. Critically, the similarity index was computed separately for subsequent hits and misses and for each emotion category allowing us to assess whether the consistency of neural patterning during encoding is differentially predictive of memory success or failure, and whether this differed as a function of emotion category. For example, to compute within-negative-hit similarity, we compared a given negative trial that was subsequently remembered to all other negative trials that were also subsequently remembered. For between-negative-hit similarity, we compared a given negative trial that was subsequently remembered to all other positive and neutral trials that were also subsequently remembered. The analogous procedure was taken for subsequent miss pattern similarity (see also, Kuhl et al., 2012).

Figure 3.

Overview of the MVPA. A within-category similarity metric was computed for each emotion category (i.e., r_within). This was computed as a correlation across voxels between each encoding trial and all other encoding trials of the same emotion category (e.g., correlating a given negative trial [e.g., ‘Desert Reptile’] to all other negative trials [e.g., ‘Baton’]). A between-category similarity metric (i.e, r_between) was computed as the correlation between rather than within a given emotion category (e.g., correlating a given negative trial [e.g., ‘Desert Reptile’] with a neutral trial [e.g., ‘Man with Parted Hair’] and separately to a positive trial [e.g., ‘Soccer Player’]). The similarity index was computed as the difference between the average r_within and r_between correlations, and reflects the extent to which a particular emotion category elicits consistent neural patterning. Due to the IAPS usage agreement, representative negative, positive and neutral pictures taken from other sources are shown.

3. Results

3.1. Behavioral Results

Table 2 lists the mean (± 1 standard error) emotional intensity ratings and associated reaction times during encoding as a function of subsequent memory and emotion category. A two-way repeated measures ANOVA on the emotional intensity ratings with factors Memory (hits and misses) and Emotion Category (positive, negative and neutral) revealed main effects of Subsequent Memory (i.e., greater emotional intensity for subsequent hits > misses; F(1, 37) = 18.83, p < 0.001, partial η² = 0.34) and Emotion Category (F(1, 37) = 123.81, p < 0.001, partial η² = 0.77). Follow-up paired t-tests (collapsed across Memory) revealed that negative events were rated as higher in emotional intensity than both positive and neutral events (ts(37) > 3.69, p < 0.001, d > 0.60), and positive events were rated as higher in emotional intensity than neutral events (t(37) = 13.77, p < 0.001, d = 2.23). The interaction of Memory by Emotion Category was not significant (F < 1). The analogous ANOVA conducted on the RTs associated with the emotional intensity rating failed to reveal any significant findings (Fs < 1.18, ps > 0.29).

Table 2.

Mean (± 1 standard error) emotionality rating and reaction time (ms) during encoding as a function of subsequent memory and emotion category.

Emotional Intensity	Hit	Miss
Positive	2.22 (0.08)	1.96 (0.10)
Negative	2.52 (0.09)	2.34 (0.10)
Neutral	1.24 (0.06)	1.10 (0.07)
Reaction Time	Hit	Miss
Positive	942.80 (0.04)	938.06 (0.04)
Negative	964.10 (0.04)	929.46 (0.04)
Neutral	952.72 (0.04)	942.58 (0.04)

3.2. Univariate Results

3.2.1. Generic emotion-selective effects

The univariate analysis was first used to identify the regions commonly demonstrating generic effects of emotional category (positive, negative, neutral) and those associated with the processing of each sensory (i.e., visual/auditory) feature of the event encoded. This was achieved via a three-step procedure. First, directional contrasts between the two classes of content (visual > auditory and vice versa, each at p < 0.005) were conducted on the data from the localizer to identify the neural regions differentially active for each of the individual sensory features of the multi-featural event (see Figure 4, green and red, respectively).

Figure 4.

Visual and auditory-selective effects. Neural regions identified with the visual > auditory contrast are depicted in green and neural regions identified with the auditory > visual contrast are depicted in red.

Second, the encoding data were employed to identify the neural regions differentially active for each category of emotion associated with the encoded event. To achieve this aim, the same procedure described above for the MVPA feature selection was employed where emotion-selective regions were identified through exclusive masking. For example, the contrast of negative > neutral + positive (at p < 0.005), was exclusively masked with the analogous contrasts for the remaining two emotion categories (e.g., the contrast [negative > neutral + positive] was exclusively masked with the contrasts of [positive > negative + neutral] and [neutral > positive + negative], exclusive mask threshold of p < 0.05). Emotion category effects are illustrated in Figure 5, with negative-emotion-selective effects shown in red, positive-emotion-selective effects shown in green, and neutral-emotion-selective effects shown in blue. Note that neutral-emotion-selective effects failed to be identified at the corrected threshold of p < 0.005, but when the cluster extent threshold was relaxed to 20 voxels, neutral-emotion-selective effects were identified in the dorsolateral prefrontal cortex (see Figure 5, bottom center), among other regions. As is apparent from Figure 5, negative-emotion-selective effects were identified in regions previously associated with the processing of negative emotional content, such as the amygdala (see Figure 5, bottom right), but also in auditory- (e.g., lateral superior temporal cortex) and visual-processing regions in occipital cortex. Positive-emotion-selective effects were observed in regions previously associated with the processing of positive emotional content, such as the ventromedial prefrontal cortex (see Figure 5, bottom left), as well as within visual-processing regions in occipital cortex. In contrast to the negative-emotion-selective effects, positive-emotion-selective effects were not identified in auditory-processing regions, such as the lateral temporal cortex.

Figure 5.

Emotion category effects. Negative-emotion-selective effects are displayed in red, positive-emotion-selective effects are shown in green, and neutral-emotion-selective effects are shown in blue. Mean parameter estimates are shown for each emotion category and respective brain region (left, ventromedial prefrontal cortex, middle, dorsolateral prefrontal cortex, and right, left amygdala). Note that error bars are not plotted as a result of potential noise, and significance tests were not run on these data. Neutral-emotion-selective effects were identified at the relaxed cluster extent threshold of k = 20, relative to the corrected cluster extent threshold of k = 28 (see text for full details).

To statistically test for overlap between the previously identified emotion-category-specific and visual/auditory-selective processing, we performed a conjunction analysis by inclusively masking each of these effects (i.e. Figure 4 with Figure 5; each contrast was set at the original threshold of p < 0.005 with a cluster extent of 28 voxels; joint probability of 2.90 × 10⁻⁴; Fisher, 1950; Slotnick & Schacter, 2004). As illustrated in Figure 6 and detailed in Supplementary Table 1, during the encoding of negative events, both visual- (occipital) and auditory-processing (superior temporal) regions were recruited as indicated by significant overlap between the negative-emotion-selective effects and each of the directional contrasts associated with the localizer (Figure 6, top). A different pattern emerged when examining overlap between the encoding of positive events and visual/auditory activity (Figure 6, bottom): although there was overlap between positive-emotion-selective and visual-processing-selective activity, the amount of overlap was reduced relative to negative-emotion-selective activity. In addition, common positive-emotion-selective and auditory-processing-selective activity was only observed in a single region, the ventromedial prefrontal cortex.

Figure 6.

Emotion-category-selective effects that overlap those with the localizer identified via a conjunction analysis. Top. Negative-emotion-selective effects that overlap the visual > auditory contrast (cyan) and the auditory > visual contrast (magenta). Bottom. Positive-emotion-selective effects that overlap the visual > auditory contrast (cyan) and the auditory > visual contrast (magenta)

3.2.2. Emotion-category-selective subsequent memory effects

The above analyses identified overlapping neural correlates associated with the generic processing of emotional category and visual/auditory processing. We then performed hypothesis-driven analyses to assess whether, as predicted, emotion-category-selective subsequent memory effects (that is, subsequent memory effects that were reliable for events associated with one of the three emotion categories, but not the other) overlapped regions not only demonstrating generic emotion-category effects (see Figure 5), but also those sensitive to the processing of the sensory features comprising the events encoded (i.e., the visual and auditory content; see Figure 4). The generic emotion-category-selective effects described above were inclusively masked with the corresponding subsequent memory contrast for the same emotion category at p < 0.005 (e.g., the negative-emotion-selective effects (Figure 5, red) were inclusively masked with negative-hit > negative-miss at p < 0.005). The outcome of this procedure was then exclusively masked with the analogous contrasts for the remaining two emotion categories (e.g., positive-hit > positive-miss and neutral-hit > neutral-miss; each at p < 0.05; for similar procedures, see Gottlieb, Uncapher, & Rugg, 2010). This analysis failed to identify any significant voxels.

We then went on to identify subsequent memory effects unconstrained by generic effects of emotion category. Neutral-emotion-selective subsequent memory effects were identified by exclusively masking the neutral-hit > neutral-miss contrast with both the positive-hit > positive-miss and negative-hit > negative miss contrasts (each at p < 0.05). As shown in Figure 7A and listed in Supplementary Table 2, this procedure identified clusters in the lateral parietal cortex (left angular gyrus), frontal cortex (middle frontal gyrus and anterior prefrontal cortex, and the left hippocampus, among other regions. An analogous procedure was employed to identify negative- and positive-emotion-selective subsequent memory effects. While no significant positive-emotion-selective subsequent memory effects were observed, a single cluster was identified where negative-emotion-selective subsequent memory effects were significant; the right hippocampus (see Figure 7B, Supplementary Table 2).

Figure 7.

Emotion-category-selective subsequent memory effects. A. Neutral-emotion-selective subsequent memory effects in the hippocampus (left) and at the whole-brain level (right). B. Negative-emotion-selective subsequent memory effects in the hippocampus (no significant effects were observed at the whole-brain level). Note that error bars are not plotted as a result of potential noise, and significance tests were not run on these data.

To determine if the emotion-category-selective subsequent memory effects identified overlapped those engaged by the processing of the visual and auditory information associated with the encoded events, we conducted a final analysis where the visual > auditory contrast and auditory > visual contrast from the localizer were inclusively masked with the significant negative-emotion-selective and neutral-emotion-selective subsequent memory effects (i.e., a statistical test of overlap between Figure 4 and Figure 7). This analysis failed to identify any significant voxels.

3.3. MVPA Results

3.3.1. Visual/Auditory ROIs

A three-way repeated measures ANOVA on the similarity indices (Figure 8) with factors ROI (visual and auditory), Emotion Category (positive, negative and neutral), and Memory (hit and miss), revealed a main effect of ROI (F(1, 37) = 119.89, p < 0.001, partial η² = 0.76), main effect of Emotion Category (F(2, 74) = 108.44, p < 0.001, partial η² = 0.75), and with no significant main effect of Memory (F < 1). The main effect of Emotion Category was driven by greater similarity for negative relative to positive and neutral events. While the ROI by Memory by Emotion Category or ROI by Memory interactions were not significant (Fs < 4.03, ps > 0.05), the ANOVA did reveal a significant ROI by Emotion Category interaction (F(2, 74) = 17.67, p < 0.001, partial η² = 0.32). Follow-up paired t-tests revealed that similarity was greater for negative events in the auditory ROI relative to the visual ROI (t(37) = 3.16, p = 0.003, d = 0.51), with similarity greater for positive and neutral events in the visual relative to the auditory ROI (ts(37) > 5.30, ps > 0.001, ds > 0.86). As is apparent in Figure 8, the latter ‘greater’ similarity in the visual relative to the auditory ROI for positive and neutral events, was not numerically greater than 0. This suggests that the difference across ROIs reflects less dissimilarity among positive and neutral events in the visual than auditory ROI. The Memory by Emotion Category interaction was also significant, F(2, 74) = 25.16, p < 0.001, partial η² = 0.41. Across both the visual and auditory ROIs, a subsequent memory effect was identified for neutral events (i.e., greater similarity for hits > misses; t(37) = 11.00, p < 0.001, d = 1.20), with the reverse pattern observed for both positive and negative events, where similarity was significantly less for subsequent hits than misses (ts > 3.25, ps < 0.002, ds > 0.52)

Figure 8.

Similarity index (within-between similarity) for each emotion category and subsequent memory computed from the visual- and auditory-selective ROIs. Error bars denote mean [± 1 standard error] similarity index.

3.3.2. Anatomical ROIs

3.3.2.1. Amygdala

Figure 9 illustrates the pattern similarity metrics for the amygdala. A three-way repeated measures ANOVA on the similarity indices with factors Hemisphere (left and right), Emotion Category (positive, negative and neutral), and Subsequent Memory (hit and miss), revealed a Hemisphere by Emotion Category by Memory interaction (F(2, 74) = 73.96, p < 0.001, partial η² = 0.67). To breakdown the three-way interaction, three additional ANOVA’s were conducted for each category of emotion with factors Hemisphere and Memory. Each ANOVA revealed significant Hemisphere by Memory interactions (Fs(1, 37) > 11.94, ps < 0.002, partial η² s > 0.24). For each hemisphere and emotion category, subsequent memory effects were then assessed with paired t-tests. The direction of each effect (i.e., hit > miss for negative events and miss > hit for positive events) was equivalent across hemispheres (ts(37) > 4.35, ps > 0.001, ds > 0.71 and ts(37) > 5.05, ps > 0.001, ds > 0.82, respectively), only the magnitude differed (i.e., larger subsequent-memory effect in the right hemisphere for negative events and larger subsequent-forget effect in the left hemisphere for positive events). In contrast to negative and positive events, the subsequent-memory effect for neutral items was specific to the left hemisphere (i.e., hit > miss, t(37) = 9.58, p < 0.001, d = 1.55) with no effect observed in the right hemisphere (t(37) < 1). Taken together, pattern similarity in the amygdala was greater for subsequent hits than misses for both negative and neutral events, but the latter effect was specific to the left hemisphere. In contrast, pattern similarity in the amygdala was greater for subsequent misses than hits for positive events.

Figure 9.

Similarity index (within-between similarity) for each emotion category and subsequent memory computed from the left and right amygdala. Error bars denote mean [± 1 standard error] similarity index.

3.3.2.2. Hippocampus

Figure 10 illustrates the pattern similarity metrics for the hippocampus. Akin to the amygdala, a three-way repeated measures ANOVA on the similarity indices with factors Hemisphere (left and right), Emotion Category (positive, negative and neutral), and Memory (hit and miss) revealed a Hemisphere by Emotion Category by Memory interaction (F(2, 74) = 73.46, p < 0.001, partial η² = 0.67). Separate 2-way ANOVA’s were conducted for each emotion category. For negative events, there was no Hemisphere by Memory interaction (F < 1). Both the main effect of Hemisphere (i.e., greater similarity in the left relative to the right hemisphere; F(1, 37) = 20.82, p < 0.001, partial η² = 0.36), and replicating the amygdala, the main effect of Memory was significant (i.e., hit > miss; F(1, 37) = 3.38, p < 0.02, partial η² = 0.15). For positive events, there was a main effect of Hemisphere (i.e., right > left; F(1, 37) = 123.84, p < 0.001, partial η² = 0.77) and a main effect of Memory (i.e., miss > hit; F(1, 37) = 479.90, p < 0.001, partial η² = 0.93). These main effects were qualified by a significant Hemisphere by Memory interaction (F(1, 37) = 162.88, p < 0.001, partial η² = 0.82). Despite the interaction, the direction of the subsequent-forgetting effect (i.e., miss > hit) in each hemisphere was equivalent (ts(37) = 19.73, ps < 0.001, ds > 3.20). For neutral events, akin to negative events, there was a main effect of Hemisphere (i.e., right > left; F(1, 37) = 109.24, p < 0.001, partial η² = 0.75) and a main effect of Memory, with greater similarity for hits relative to misses (F(1, 37) = 26.49, p < 0.001, partial η² = 0.72). Although these main effects were qualified by a significant Hemisphere by Memory interaction (F(1, 37) = 48.61, p < 0.001, partial η² = 0.57), the direction of the subsequent-memory effect (i.e., hit > miss) in each hemisphere was equivalent and significant (ts(37) = 2.17, ps < 0.04, ds > 0.35). Taken together, the pattern similarity results from the hippocampus parallel those of the amygdala: a subsequent-memory effect was observed for negative and neutral events (i.e., hit > miss), but a subsequent-forgetting effect (i.e., miss > hit) was observed for positive events.

Figure 10.

Similarity index (within-between similarity) for each emotion category and subsequent memory computed from the left and right hippocampus. Error bars denote mean [± 1 standard error] similarity index.

3.3.2.3. Amygdala to Sensory ROI Functional Connectivity

We performed a final analysis to directly compare how univariate mean signal within the amygdala was related to the similarity within the two sensory ROIs and tested whether this relationship changes as a function of both emotional category and subsequent memory. Given the effects of emotional memory enhancement observed in the amygdala, as evidenced in its enhanced mean signal for positive and negative events that are subsequently remembered (e.g., Hamann, Ely, Grafton, & Kilts, 1999; Kensinger & Schacter, 2008), we tested whether amygdala activity would differentially correlate with similarity in the two sensory ROIs, providing evidence that the amygdala contributes to the category-selectivity of emotional information in sensory regions, and more importantly whether this relationship would differ as a function of subsequent memory and emotional category. Following the work of prior studies also conducting a ‘functional connectivity’ analysis between mean signal within medial temporal lobe regions and pattern similarity metrics within content-selective regions (e.g., Tompary, Duncan, & Davachi, 2016), in our analysis, on an individual participant-level basis, we extracted the encoding-related mean signal from our amygdala ROI. The amygdala functional ROI included voxels within the AAL anatomical ROI employed for the MVPA (left and right amygdala). The functional connectivity analysis collapsed across hemispheres as mean signal analyses identified emotion-sensitive effects in each hemisphere (see Figure 5). We correlated the trial-by-trial univariate mean signal from the amygdala with the similarity index computed in each sensory ROI for each trial type of interest (i.e., subsequent-hit-negative, subsequent-miss-negative, subsequent-hit-positive, subsequent-miss-positive, subsequent-hit-neutral, and subsequent-miss-neutral). Correlations were Fisher z- transformed before statistical tests were performed.

Figure 11 illustrates averaged across-participant correlations between similarity indices and univariate amygdala activation as a function of ROI (visual, auditory), subsequent memory (hit, miss) and emotional category (negative, positive, neutral) for each ROI. A three-way repeated measures ANOVA on the functional connectivity metrics with factors ROI, Emotion Category, and Subsequent Memory, revealed a significant three-way interaction of ROI by Emotion Category by Memory interaction (F(2, 74) = 12.43, p < 0.001, partial η² = 0.25). The ANOVA also revealed significant Emotion Category by Memory and ROI by Emotion Category interactions (Fs(2, 74) > 3.96, ps < 0.001, partial η²s > 0.21). Other than a main effect of Emotion Category (F(2, 74) = 57.05, p < 0.001, partial η² = 0.61), all other ANOVA effects were not significant (Fs < 2.44, ps > 0.10).

Figure 11.

Amygdala to the visual- and auditory-selective ROI functional connectivity for each emotion category and subsequent memory. Error bars denote mean [± 1 standard error] connectivity.

To decompose the significant three-way interaction, we conducted separate ANOVAs on data from each ROI. In the Visual ROI (left), the main effect of Emotion Category was significant (F(2, 74) = 21.20, p < 0.001, partial η² = 0.36) but the main effect of Memory was not (F(2, 74) = 3.29, p > 0.05). The Memory by Emotion Category interaction was significant (F(2, 74) = 3.17, p < 0.05, partial η² = 0.08). Follow-up paired t-tests revealed that functional connectivity between the amygdala and the visual ROI was greater for subsequent hits than misses for only negative events (t(37) = 2.51, p < 0.05, d = 0.23), with no difference among the two positive or neutral events (ts(37) < 1.61, ps > 0.10). In the Auditory ROI (right), the ANOVA revealed a significant main effect Emotion Category (F(2, 74) = 73.30, p < 0.001, partial η² = 0.67), with no main effect of Memory (F < 1). The main effect of Emotion Category was qualified by a significant Memory by Emotion Category interaction (F(2, 27) = 17.55, p < 0.001, partial η² = 0.32). In direct contrast to the Visual ROI, follow-up paired t-tests revealed that functional connectivity between the amygdala and the auditory ROI was greater for subsequent misses than hits for positive events (t(37) = 5.62, p < 0.001, d = 0.25), with no difference among the two negative events (t < 1). For neutral events, functional connectivity was greater for hits than misses (t(37) = 3.91, p < 0.001, d = 0.39). The above analyses identified differential connectivity between the amygdala and the two sensory ROIs as function of emotional category: higher mean signal in the amygdala was related to higher similarity for hits than misses for only negative events, with this effect specific to the visual ROI, as no such effect was present in the auditory ROI. Functional connectivity between the amygdala and auditory ROI was greater for hits than misses for neutral events, with the opposite pattern for positive events.

4. Discussion

Employing a univariate analysis and MVPA of fMRI data, the present study provides novel insights into how multimodal emotional stimuli are initially processed and later remembered. We contrasted the neural correlates of the encoding of the associations between neutral verbal labels and the emotion category information carried by the co-presented multisensory (i.e., visual and auditory) information, and further compared encoded events as a function of subsequent memory (i.e., those events remembered with some degree of vividness relative to events that were not remembered). We asked whether the selectivity of neural patterning in visual and auditory regions differed by emotion category and whether emotion-category-selectivity was predictive of subsequent memory in visual and auditory processing regions as well as two regions known to be associated with the encoding of emotional episodic memory (i.e., the hippocampus and amygdala). Below, we first describe the generic emotion-selective effects identified by the univariate and MVPA results. Second, we go on to describe the subsequent memory effects identified by each analytical approach.

4.1. The effects of emotion on sensory processing of multisensory stimuli

The univariate analysis revealed that negative-category-selective regions overlapped extensively with those regions functionally defined as visual-selective or auditory-selective regions. Although positive-category-selective regions also showed some overlap, it was less extensive, and there was generally greater recruitment of visual and auditory regions under negative relative to positive event encoding. The latter findings are consistent with our hypotheses that negative emotionality of the presented stimuli enhances sensory processing during perception.

The results do not strongly support one possibility we outlined, which was that the information conveyed by negative multisensory events may be more strongly represented within a single modality, the visual domain. The MVPA did reveal differential sensitivity to emotion category information within visual and auditory processing regions, but the results were such that neural patterning in auditory vs. visual regions was more diagnostic for negative information while neural patterning within visual vs. auditory regions was more diagnostic of positive and neutral information. Moreover, there was greater similarity within both visual and auditory ROIs for negative relative to positive or neutral stimuli, suggesting that both modalities were strongly represented for negative stimuli. And, in the univariate analyses, it was the positive-category-selective regions that showed overlap only with visual, and not with auditory, regions. Thus, the results overall suggest that negative emotion results in a stronger link to sensory representation in auditory and visual regions.

Why might the information conveyed by negative multisensory events be strongly represented within large portions of regions that process each individual modality of the event? One possibility is that participants grant greater attention to the sensory details of negative events. Attention is known to increase category-selectivity in sensory regions as measured in neural patterning (e.g., Reddy & Kanwisher, 2007). Therefore, increased attention to the sensory features of negative events could increase the valence-selectivity of visual and auditory processing regions, as measured with MVPA. Additional and convergent evidence comes from event-related potentials (ERPs). For instance, ERP components associated with the orienting of attention (i.e., the P3a and N1, respectively) are enhanced when a visual (Tartar et a., 2012) or auditory (Thierry & Roberts, 2006) stimulus are negative relative to neutral. These ERP findings were interpreted as reflecting increased attentional resources allocated to stimuli with negative emotionality (for related evidence from fMRI; see Talmi, Anderson, Riggs, Caplan, & Moscovitch, 2008; Pessoa & Ungerledier, 2004). More generally, there are attention-related increases in activation in sensory regions evidenced with fMRI (for a review, see Pessoa, Kastner, & Ungerleider, 2003). Although much of this research did not compare negative to positive stimuli, the results could suggest that attentional-related enhancement of sensory processing is especially strong when events are associated with negative emotionality. This difference could be attributable to the negativity of the stimuli, or to the different goal states evoked by such stimuli (see, Clewett & Murty, 2019). Lastly, the univariate findings are also consistent with prior behavioral studies examining the effects of valence on the capture of attention. For example, in the meta-analysis of Yiend (2010), it was shown that negative information elicits a selective attentional priority and attentional resources relative to both non-emotional and positive information. Whether this bias occurs automatically or in a top-down fashion is a topic for future research.

Another possibility, that need not be mutually exclusive, is that the neurochemistry associated with negative emotion drives modulation of sensory regions by the amygdala (e.g., Vuilleumier & Driver, 2007) or by other regions activated by negative emotion (e.g., Edmiston et al., 2013). Indeed, earlier work has shown that different visual cortical regions are modulated by different categories of emotionality (Mourao-Miranda et al., 2003), with unpleasant relative to pleasant pictures producing significantly greater activity in early visual cortex (i.e., V1/V2). Similarly, we found that negative-emotion-selective effects overlapped visual processing in the lingual gyrus encompassing V1, with no parallel ‘early’ visual effect observed when encoding positive events. Debates continue about whether the impact of emotion on sensory processes occurs in a ‘bottom-up’ or ‘top-down’ manner (for a review, see Mohanty & Sussman, 2013). The current data cannot speak as to which mechanism underlies the present findings, but what the data do indicate is the need for this research to more seriously consider that “emotion” is not a monolithic construct and that the nature of sensory modulations may vary depending on whether the emotion elicited is positive or negative.

The present findings extend other studies also employing MVPA to decode valence of multisensory stimuli. In Kim et al., (2016), participants passively viewed audiovisual clips that varied in both valence (positive/negative) and arousal (high/low). A multivoxel searchlight analysis revealed that valence could be decoded at above chance levels in the medial prefrontal cortex, posterior cingulate cortex, lateral temporal cortex (superior and middle temporal gyri), and middle frontal gyrus. As our analysis focused on sensory regions, the current findings demonstrate that the representation of emotion category information as previously observed in emotion processing regions such as the medial prefrontal cortex (e.g., Peelen, Atkinson, & Vuilleumier, 2010) and amygdala (e.g., Jin, 2015) is preserved at other levels of processing, namely at the level of sensory processing (see also, Chikazoe, Lee, Kriegeskorte, & Anderson, 2014). The current findings also provide further support for theories positing that sensory systems have a direct role in representing emotion category information (e.g., Satpute et al., 2015; Miskovic & Anderson, 2019). Our findings suggest that sensory representations associated with multisensory stimuli differ by emotion category, spanning the spectrum from positive to negative the former emotion category of which has been largely excluded from multisensory classification studies (e.g., Peelen et al., 2010; Kim et al., 2016). Future work aiming to understand the patterning of emotion-category-specific information in sensory regions may require considering the type of emotion perception involved (e.g., comparison of viewing emotional scenes, viewing emotional states of others, or experiencing discrete emotional states). Such work would elucidate the degree of specificity at the neuronal level. One possibility is that the emotion-category-selective patterning observed in sensory regions codes for basic emotion categories employed here or extends to other emotions.

The MVPA findings stand in contrast to the univariate analysis which did not identify a dissociation between visual and auditory regions: while the univariate analysis would suggest that negative valence enhances both visual and auditory processing, the MVPA findings indicate that negative valence biases processing towards the auditory modality, to suggest that the auditory modality may confer negative emotional content. These divergent findings add to the existing evidence that univariate and MVPA analyses can give rise to dissociable results, where reliable emotion-specific information is carried by neural patterning and not the mean signal (e.g., Peelen et al., 2010). An outstanding question is whether it might be possible to further localize emotion-category-specific information across sensory cortices when participants encounter multisensory stimuli. For example, using visual emotional stimuli, Kragel, Reddan, LaBar, & Wager, (2019) were able to decode 11 distinct emotion categories in the human visual cortex. On a broader level, the differential MVPA and univariate results highlight the importance of employing both pattern-based and mean-signal approaches to fMRI data analysis as they can offer unique insights into neural processing. As discussed in prior work (e.g., Davis et al., 2014), each analysis is sensitive to unique sources of variability: voxel-level variability with MVPA and participant-level variability with a univariate analysis. Null results may be a consequence of insufficient sensitivity with the specific type of analysis employed.

4.2. The effects of emotion on memory for multisensory stimuli

Although visual and auditory regions overlapped those associated with the generic processing of negative and, to a lesser extent, positive categories, they did not predict subsequent memory. In fact, the MVPA results revealed that, for positive and negative events, subsequent hits had reduced similarity compared to subsequent misses. This set of findings differs from previous work from our lab suggesting that the engagement of visual processing predicts subsequent remembering of negative pictures (Mickley & Kensinger, 2008; Mickley-Steinmetz & Kensinger, 2009; Kark & Kensinger, 2019; Bowen & Kensinger, 2017), a difference that may be explained by the current use of multisensory emotional events.

In contrast to the results for emotional stimuli, the MVPA results revealed that similarity was higher for subsequent hits than misses for neutral events in both visual and auditory ROIs. These findings are consistent with prior MVPA studies demonstrating that category-selective neural patterning in sensory processing regions is predictive of subsequent memory for neutral information (e.g., face or scene information in ventral temporal cortex; Kuhl et al., 2012). Here, similarity was computed as the difference between within-category (i.e., among neutral events) and between-category (i.e., between neutral and emotional events) correlations, thus the significant subsequent memory effect likely reflects the engagement of distinct processes engaged during successful neutral relative to emotional encoding. One such process may be unique sensory processing (both visual and auditory) which aided the formation of memory for neutral relative to emotional events. The divergence in results for neutral stimuli and emotional stimuli are broadly consistent with the proposal that memory for neutral stimuli is more tied to retrieval of sensory content while memory for emotional stimuli may be more strongly tied to retrieval of other types of content, including affective context (e.g., Yonelinas & Ritchey, 2015; Sharot, Delgado, & Phelps, 2004).

The finding that greater similarity in sensory regions was predictive of subsequent misses for positive and negative events runs counter to our prediction that the greater the emotion-category-specific neural patterning, the higher performance on subsequent memory. One possibility is that the memory for the emotional stimuli relied not on encoding the generic/gist information shared across events of a given emotion category, but on encoding the event/item-specific information that would later allow an individual to link the multisensory event details to the verbal retrieval cue. According to this interpretation, the increased similarity observed for subsequent misses in sensory regions may reflect the emotional information that, although shared across events of a particular category of emotion (either positive or negative), was not diagnostic of the specific event/item information used to guide the subsequent retrieval response due to the shift in item-specific encoding.

4.3. Hippocampus/Amygdala

Replicating prior findings, the univariate analysis did reveal valence-selective subsequent memory effects in the hippocampus for negative events, (for reviews, see Murty et al., 2010; Dahlgren et al., 2020), with additional subsequent memory effects for neutral events observed in the lateral parietal cortex and hippocampus among other regions commonly associated with the encoding of neutral information (for a review, see Kim, 2011). Consistent with the two sensory ROIs, pattern similarity in both the hippocampus and amygdala predicted subsequent memory for neutral events, with greater similarity for subsequent hits relative to misses. Pattern similarity also predicted subsequent memory for negative events. Critically, pattern similarity for negative events was greater than neutral events in both the amygdala and hippocampus. These findings are consistent with the well documented emotional enhancement of memory, where the amygdala enhances episodic memory through modulation of memory-encoding related activity in the hippocampus (for a review see, Phelps, 2004; Murty, et al., 2010; Dahlgren et al., 2020). Therefore, the greater similarity may reflect the operation of emotional memory enhancing processes, engaged to a greater extent during negative relative to neutral event encoding. It is unclear as to why this effect was not also present for positive events. In fact, the reverse was observed with greater pattern similarity among positive events predicting subsequent forgetting. This is particularly surprising because the effects of emotional memory enhancement are known to occur for both positive and negative events in the amygdala (e.g., Hamann, Ely, Grafton, & Kilts, 1999; Kensinger & Schacter, 2008). Moreover, positive emotion has been shown to promote more gist-based processing (e.g., Fredrickson, 2001; Gasper & Clore, 2001; Rowe, Hirsh, Anderson, 2007) and memory (e.g., Denburg, Buchanan, Tranel, & Adolphs, 2003), which should have subsequently led to greater category-level pattern similarity (and subsequent memory). While this raises many questions for future research, we note that prior emotion-enhancing memory effects have been largely, if not exclusively, documented with univariate analyses (for a review see, Dahlgren et al., 2020). In contrast, when using MVPA, differential amygdala sensitivity to positive and negative emotionality has been observed (Jin et al., 2015). Additionally, there is some evidence that lower pattern similarity predicts subsequent memory. When examining only within-category correlations during encoding (e.g., faces to faces or scenes to scenes), LaRocque et al., (2013) found that lower pattern similarity was predictive of subsequent memory in the hippocampus. They interpreted their findings as reflecting the role of the hippocampus in ‘pattern separation’ (Yassa & Stark, 2011), where the hippocampus encodes differentiated representations of a stimulus. Although speculative, one possible explanation for the disparate findings across negative and positive events is that the lower category-selectivity for positive events may reflect the role of the amygdala/hippocampus in the creation of differentiated positive event representations. In contrast, during negative event encoding, the hippocampus and amygdala may prioritize the encoding of the commonalities across negative events which then supports later memory. This interpretation is in line with evidence to indicate that positive and negative events engage different encoding-related processes (e.g., Mickley & Kensinger, 2008). In addition, pattern separation and completion have been linked to unique subfields of the hippocampus (i.e., CA3/dentate gyrus and CA1, respectively; for a review, see Yassa & Stark, 2011). Future studies could acquire fMRI data at a higher resolution than the current study to assess whether the present pattern of findings extends to distinct hippocampal subfields (e.g., if the positive subsequent memory effects reflect ‘pattern separation’ they should be identified in CA3/dentate gyrus).

In a final analysis, we aimed to directly link the amygdala activity and the category-selectivity observed in the sensory regions through a functional connectivity analysis. In this analysis, the trial-by-trial encoding-related mean signal from the amygdala was correlated with the similarity indices on an individual participant basis for each of the six response categories (i.e., subsequent memory by emotional category). This analysis revealed that higher mean signal in the amygdala was related to higher similarity for hits than misses for negative events, with this enhancement present in only the visual but not auditory ROI. Across both ROIs, amygdala to sensory ROI connectivity was also higher for hits than misses for neutral events, but in only the auditory ROI was connectivity significantly greater for misses than hits for positive events. These findings suggest that the known memory enhancing effect of emotion as measured by amygdala univariate signal (e.g., Hamann, Ely, Grafton, & Kilts, 1999; Kensinger & Schacter, 2008), amplified the representational strength of negative event neural patterning in visual but not auditory regions. Critically, this amplification of category-selectivity was predictive of later memory success, as functional connectivity between the amygdala and visual ROI was greater for subsequent hits than misses. In contrast to negative events, functional connectivity was greater for misses than hits for positive events in the auditory ROI. This latter finding is consistent with the MVPA analysis of the amygdala ROI (Figure 9) and provides convergent evidence to suggest that in addition to neural patterning in the amygdala itself, connectivity to auditory regions may support the formation of differentiated positive event representations (see above).

Although we cannot make claims about the directionality of these correlations, the findings pertaining to the negative events lend support to the hypotheses raised in our Introduction that visual sensory processing may be particularly related to the processing and encoding of negative events (e.g., Mickley & Kensinger, 2008; Bowen et al., 2017; Kark & Kensinger, 2015, 2019), while auditory sensory processing may be less related. More importantly, our analysis suggests that the amplification of category-selective processing negative event information in visual regions is driven in part, by mean signal in the amygdala.

4.4. Limitations

There a few limitations of the current study that deserve mention. First, all events in the current study comprised multisensory (visual and auditory) information. The current study is thus limited in that we could not identify neural regions associated with multisensory ‘integration’, which would necessitate a comparison to events consisting only of visual or auditory information (for a review, see Klasen et al., 2012). It may be the case that the effect of emotion on sensory processing and later memory may be present in regions coding for the integrated event information. In the current study, similarity was computed at the category-level of valence where similarity reflects the information shared across events belonging to a given emotion category. Therefore, it is unknown whether the effects of emotion on neural patterning are also present at the level of individual events (cf., Xue et al., 2010). This is an important avenue for future work. A final limitation of the current study is that memory success was operationalized with a single vividness rating, reflecting the total vividness of the retrieved picture and associated sound. It will be important for future studies to assess whether the current findings extend to objective indices of episodic memory (e.g., source memory) and also whether the effects of emotion differ as a function of the individual features comprising a multisensory emotional event, for example by collecting a vividness rating for each feature comprising the multisensory event.

4.5. Conclusions

Using both mean signal and pattern-based analyses of fMRI data, we demonstrate that levels and patterns of neural activity in visual and auditory regions are more closely tied to negative-category-specific effects than to positive- or neutral-category-specific effects. We further reveal that, in contrast to prior results using unimodal visual stimuli, this sensory engagement for negative multisensory stimuli is not beneficial for emotional memory: while neural patterning in sensory regions is linked to subsequent memory for neutral stimuli, it is linked to subsequent forgetting for emotional stimuli. These results emphasize that whether stimuli elicit positive or negative emotions can affect the way that visual and auditory regions represent emotional information. These results underscore that the emotionality of multisensory events has a differential impact on encoding processes.

Table 1.

Mean [± 1 standard error] number of encoding trials for each level of vividness rating during retrieval as a function of the emotion category

	0 ‘No Memory’	1 ‘Somewhat Vivid’	2 ‘Vivid’	3 ‘Extremely Vivid’
Positive	15.74 (0.70)	7.84 (0.64)	5.13 (0.48)	6.66 (0.65)
Negative	14.47 (0.66)	7.45 (0.71)	4.97 (0.59)	5.87 (0.71)
Neutral	14.18 (0.75)	8.37 (0.57)	5.61 (0.57)	6.92 (0.80)