Hearing something emotional influences memory for what was just seen: How arousal amplifies effects of competition in memory consolidation

Abstract

Enhanced memory for emotional items often comes at the cost of memory for the background scenes. Because emotional foreground items both induce arousal and attract attention, it is not clear whether the emotion effects are simply the result of shifts in visual attention during encoding or whether arousal has effects beyond simple attention capture. In the current study, participants viewed a series of scenes that each either had a foreground object or did not have one, and then, after each image, heard either an emotionally arousing negative sound or a neutral sound. After a 24-hour delay, they returned for a memory test for the objects and scenes. Post-encoding arousal decreased recognition memory of scenes shown behind superimposed objects but not memory of scenes shown alone. These findings support the hypothesis that arousal amplifies the effects of competition between mental representations, influencing memory consolidation of currently active representations.

Arousing central details are especially likely to be remembered, but at the cost of peripheral details, a “memory narrowing” (Burke, Heuer, & Reisberg, 1992) or "memory trade-off” (Kensinger, Garoff-Eaton, & Schacter, 2007) effect. For instance, in the weapon-focus effect, people remember the weapon in a real or simulated crime vividly, but at the cost of memory for scene details (Loftus, Loftus, & Messo, 1987; Steblay, 1992). Likewise, memory tends to be better for negative than for neutral foreground objects (Kensinger, Garoff-Eaton, et al., 2007; Kensinger, Gutchess, & Schacter, 2007). This negative foreground-object enhancement comes at the cost of memory for the background scenes, which are remembered less well than when shown behind neutral foreground objects.

What are the mechanisms of these emotionally induced trade-offs? It could be that the physiological experience of arousal changes how people process information. Arousal-biased competition (ABC) theory (Mather & Sutherland, 2011) proposes that emotional arousal amplifies biased competition processes, increasing the strength of high priority representations while suppressing other competing representations. Both top-down goal relevance and bottom-up perceptual salience determine the priority and initial activation levels of the representations. When a central foreground item is in front of a scene, and perceptual resources are deployed to the foreground item, it tends to dominate attention and suppress processing of the scene (Yi, Woodman, Widders, Marois, & Chun, 2004). In the ABC framework, emotional arousal should amplify the effects of competition between representations of the more salient foreground object and the suppressed background scene. However, it is also possible that the trade-offs seen in previous research are not due to emotion, per se, but instead to emotional stimuli that command more attention because they are more interesting, surprising, goal relevant or perceptually salient, leading to even fewer resources spent attending to whatever is in the background. In the current study, we tested the ABC claim that emotional arousal can modulate competition between stimuli, even if neither of the competing stimuli is inherently emotional.

In addition, we wanted to go a step further than just separating the source of arousal from the memoranda being tested. Our lab’s recent work indicates that when arousal is induced before stimuli are shown, encoding is enhanced for perceptually salient and impaired for less salient stimuli (Lee, Itti, & Mather, 2012; Sutherland & Mather, 2012). But these effects could be due entirely to arousal amplifying the effects of stimuli competition during initial perception. ABC theory proposes that arousal amplifies the effects of competition during memory consolidation, as well as during initial perception. To be able to remove potential influences of perceptual biases under arousal, we wanted initial perception of the stimuli to be identical in the arousing and neutral conditions.

To do this, we manipulated emotion immediately after presentation of each visual stimulus. Many previous studies have shown that people tend to have poorer memory for neutral pictures or words that were shown just before an emotional stimulus than before something neutral (cite oddball studies here). However, emotional stimuli also sometimes enhance memory for preceding stimuli (Anderson et al., 2006; Knight & Mather, 2009; Sakaki, Fryer, & Mather, 2014). Arousal-biased competition theory predicts that the effects of emotional arousal on preceding stimuli will depend on the relative priority of those stimuli. Thus, scenes shown behind a foreground object should be more likely to be forgotten when something emotional happens next than scenes shown alone.

In our study, we manipulated emotional arousal by presenting negative or neutral sounds after showing visual images of scenes and objects. Some scenes and objects were shown individually and some were paired, with the object superimposed on the scene (see Figure 1A). Recognition memory for the scenes and objects was tested after a 24-hour delay. Our prediction was that the scenes would be remembered better if they were shown alone than if shown with a competing foreground object, and that this difference between solo and background scenes would be larger when the scenes were followed by emotional sounds than by neutral sounds. In contrast, since priority would be high both for solo objects and for objects shown in front of scenes, we did not anticipate that emotion would influence these two conditions differently. We also asked participants to make “remember” vs. “know” judgments about the items they remembered (with “remember” corresponding to memories that included contextual details) to examine if the predicted emotional memory effects were due primarily to changes in memory binding efficacy under arousal (Mather, 2007).

Figure 1.

Figure 1A shows the trial structure. On every trial, participants were given a cue, asking them to pay attention to either a background or an object. They were then presented with either an object alone, a scene alone, or a combination (which is an object superimposed on a scene). A sound was played, and the sound was either negative or neutral. They were then given a judgment of learning question, which asked them to indicate how likely they were to remember the cued image. A jittered ITI followed, of either four seconds, six seconds, or eight seconds. Figure 1B shows that scenes presented as part of a combination had significantly lower overall (remember + know) recognition rates when followed by negative sounds than by neutral sounds (* p < .05). Emotion did not significantly influence memory for preceding solo scenes.

Methods

Participants

Fifty-three adults (18–35 years of age, M = 20.33, SD = 3.07, 14 males, 68% Caucasian) participated and were compensated with course credit or $15 per hour for their participation. Four additional participants completed only the first day of the study and so were not included in analyses.

Equipment and Stimuli

The experiment was presented using MatLab experimental software, including Psychtoolbox (Brainard, 1997). Composite image stimuli were assembled in-house using Adobe Photoshop and images acquired through various sources, including the Internet and an object and scene database (Goh, et al. 2007). We used three different types of image stimuli, an object presented alone (object-only trials), a scene presented alone (scene-only trials), or a combination image with an object superimposed centrally on a scene (combination trials). Some sound stimuli were created in house using Audacity; most sounds were a part of the International Affective Digital Sounds (IADS; Bradley & Lang, 2007). Negative sounds included people crying and animals growling. Neutral sounds included water running and people yawning. The sounds were pre-rated on valence (M_negative = 2.63, SD_negative = .76, M_neutral = 4.99, SD_neutral = .56) and arousal (M_negative = 6.46, SD_negative = 1.39, M_neutral = 2.20, SD_neutral = 1.05) by a different group of 40 participants. We did not include positive sound stimuli because they tend to be less arousing than negative ones and we also wanted to minimize the number of trials to avoid participant fatigue.

Procedure

Participants came for two sessions. On the first day, participants completed informed consent procedures, the Center for Epidemiological Studies-Depression questionnaire (Radloff, 1977; M_CESD =12.94, SD_CESD = 8.94), Positive and Negative Affect Scale (Watson, Clark, & Tellegen, 1988; M_{Positive PANAS} = 29.11, SD_{Positive PANAS} = 7.07, M_{Negative PANAS} =13.14, SD_{Negative PANAS} =4.62), and the encoding task. To avoid ceiling effects, and to ensure the consolidation period included a night of sleep, which preferentially enhances memory for emotional events (e.g., Wagner, Gais, & Born, 2001), participants completed a recognition task twenty-four hours later.

Encoding

We manipulated scene priority during encoding using both bottom-up salience and top-down goals. Bottom-up perceptual salience of scenes was manipulated by having the scenes either shown alone or in competition with scene-incongruent objects spatially centered on top of the scenes (“combination” trials). Both centrality (Burke et al., 1992), and incongruency (Desimone & Duncan, 1995) increase salience of foreground objects, and we used saliency map software (Itti & Koch, 2000) to check that the object was the most salient part of each scene-object combination image. As a control to make sure that appearing with a scene did not impair memory of the foreground objects, we also included a condition with objects shown alone. Thus, during encoding participants saw three types of trials: scene-only trials, object-only trials, and combination trials, across 180 trials split into six blocks.

We manipulated top-down goal relevance by having different top-down goal cues in scene-only and combination trials. At the beginning of each scene-only trial, participants were instructed to pay attention to the scene. In contrast, at the beginning of each scene-object combination trial, participants were instructed to pay attention to the object, to further lower the relative priority of the scene. In addition, the judgment of learning after the sound ended re-emphasized the prioritization of either the scene or the object.

Each trial consisted of a top-down goal cue (2 seconds; “BACKGROUND” for scene-only trials, or “OBJECT” for object-only and combination trials), an image (2 seconds; scene-only, object-only or combination images), a sound (4 seconds; negative or neutral), and a judgment of learning (2 seconds + a jittered inter-trial interval of 4, 6, or 8 seconds; see Figure 1 for complete trial structure). During the judgment of learning, participants were asked to rate on a scale from one (not at all likely) to four (very likely) how likely they were to remember the previously shown image.

Memory Test

After a 24-hour delay, participants completed a recognition task. The participants were told that they would be viewing images from the day before, plus more images that were new. Scenes and objects that were part of a combination were viewed separately, for a total of 240 old items. Sixty additional images (30 scene and 30 object images) served as lures and were counterbalanced with old items across participants. Participants were asked to respond either “new,” for new items, or “remember” or “know,” for old items (Rajaram, 1993). If they remembered the image in addition to contextual details as to how the image was presented, they were instructed to select the “remember” key. Participants were told that examples of contextual details included location information, another image that was paired with it (in the case of a combination), or a particular sound that followed. If they remembered the image but not the contextual details as to how the image was presented, they were asked to select the “know” key.

Results

ABC theory predicts that arousal should impair memory consolidation of low priority background scenes but not of foreground objects. Old items could be differentiated by whether they were seen in a combination image or alone and by associated sound type, but new items were either just objects or scenes. Thus, we focused on recognition of old items in our analyses. For each type of item, we computed the proportion of all old items judged “remember,” “know” and “new” (Table 1). A repeated-measures ANOVA compared proportions of old items correctly judged as old (via either a “remember” or “know” judgment) across sound type (negative and neutral), scene condition (scenes presented alone and scenes presented as part of a combination), and memory type (whether the item was judged as "remember" or "know"). A main effect of scene condition, F(1,52) = 329.81, p < .001, partial η² = .86, indicated successful manipulation of scene priority (scenes presented alone were better remembered than scenes presented as part of a combination). There was no significant main effect of sound type (p = .37), but there was a significant scene condition by sound type interaction, F(1,52) = 5.86, p = .02, partial η² = .10. Consistent with the ABC prediction, scenes presented as part of a combination had lower overall (remember + know) recognition rates when followed by negative sounds (M = .29, SE =.02) than by neutral sounds (M = .32, SE =.02), t(52) = −2.276, p = .027. This suggests that arousal suppressed low priority information. In contrast, the emotionality of the sounds did not significantly influence memory for preceding solo scenes (M_arousing=.68, SE=.02; M_neutral = .67 SE=.02), t(52) = 1.02, p = .31 (see Figure 1B). There was no three-way interaction between sound type, scene condition and memory type, F(1,52) = .29, p = .59, partial η² = .01, indicating that emotion influenced recollection and familiarity in similar ways.

Table 1.

Mean proportions of items identified as “Remember,” “Know,” or “New” at test (standard errors are in parentheses).

				Remember		Know		New
Type	Sound Type	Condition		Mean	SE	Mean	SE	Mean	SE
Encoded Items	Negative	Alone	Scenes	.41	(.03)	.27	(.02)	.32	(.02)
			Objects	.49	(.03)	.29	(.03)	.22	(.02)
		Combination	Scenes	.10	(.01)	.19	(.02)	.72	(.02)
			Objects	.44	(.03)	.28	(.03)	.28	(.02)
	Neutral	Alone	Scenes	.40	(.03)	.27	(.02)	.33	(.02)
			Objects	.50	(.04)	.27	(.03)	.23	(.02)
		Combination	Scenes	.12	(.01)	.20	(.02)	.68	(.02)
			Objects	.42	(.03)	.30	(.03)	.27	(.02)
New Items	N/A	N/A	Scenes	.05	(.01)	.15	(.02)	.80	(.02)
			Objects	.08	(.01)	.16	(.02)	.76	(.02)

We predicted that since priority would be high both for solo objects and for objects shown in front of scenes, emotion would not influence these two conditions differently. In another repeated-measures ANOVA, we examined the proportion of old objects recognized across sound type (negative and neutral), object condition (objects presented alone and objects presented as part of a combination), and memory type (whether the item was judged as "remember" or "know"). There were no significant effects of emotion for the objects.

Discussion

Most previous studies examining how emotion influences the selectivity of memory have induced emotion via the same stimuli used to examine memory (e.g., Kensinger, Garoff-Eaton et al., 2007; Loftus, Loftus, & Messo, 1987; Steblay, 1992; Mather, Gorlick, & Nesmith, 2009). But this confounding of the source of arousal and the memoranda means that any resulting effect of emotion could be due to the perceptual or semantic qualities of the emotional stimuli rather than to arousal, per se. In addition, using the memoranda to induce emotion also makes it impossible to isolate the effects of post-encoding memory consolidation from effects of initial attention during stimuli presentation.

In the current study, we tested the hypothesis that negative emotional arousal increases competition between currently active mental representations, such that dominant representations suppress competing representations even more than they would otherwise. We induced emotion immediately after participants viewed some solo or background scenes and tested memory for the scenes a day later. As task-irrelevant scenes shown behind an attended object have low priority, we predicted they would be impaired by subsequent emotion, whereas this impairment would not occur for task-relevant scenes. We further heightened the discrepancy in priority by cueing participants to focus on foreground objects rather than background scenes, when they were shown together. These manipulations of the priority of scenes were successful, as people were much less likely to remember scenes shown behind objects than scenes shown alone. But most important for our hypotheses, we found these effects of differential prioritization were even stronger when the images were followed by emotional sounds. That is, emotion impaired consolidation of low priority scenes but did not impair (instead slightly, albeit not significantly, enhanced) consolidation of high priority scenes. These differential effects of emotion depending on priority are in line with the hypotheses of arousal-biased competition theory. By separating the induction of emotion from the memoranda tested, we could counterbalance which scenes were low and high priority across participants independently of our manipulation of emotion. In addition, we kept the initial perceptual experience identical across conditions by not having the auditory emotion manipulation occur until after the visual stimuli were shown. Thus, our results reflect the influence that emotion has on competing mental representations, rather than on visual attention processes.

Previous research indicates that emotionally arousing central objects are remembered better than neutral central objects, but at the cost of memory for the background scenes (Kensinger, 2009). However, memory for background scenes can instead be enhanced by emotional foreground objects if the encoding task focuses attention on the background scenes (Kensinger, Garoff-Eaton, et al., 2007). The current study provides evidence that these discrepancies about whether there is enhanced or impaired memory for background scenes shown behind emotional objects occur because the impact of emotional arousal on memory for visual information depends on the relative priority of that information.

More generally, the current study helps address why arousal often causes retrograde amnesia for previously presented neutral information, but sometimes causes retrograde enhancement (for a review see Mather & Sutherland, 2011). For instance, in one study, retrograde enhancement was found for neutral information that participants were trying to remember but retrograde impairment for neutral items that were simply rated (Knight & Mather, 2009). This pattern is consistent with the importance of stimuli priority in determining effects of emotion. Indeed, a recent study (Sakaki, et al. 2014) demonstrated that the typical retrograde amnesia for neutral items shown in a list just before an emotional oddball item (the “oddball - 1 item”) is a result of the oddball - 1 item having low priority as it is just one of a larger set of items that has no particular salience or priority. If the oddball - 1 is prioritized, then emotional oddballs actually enhance later memory for that preceding item. Our study extends these results to show that emotion also modulates competition between different components of an image.

ABC theory argues that it is the arousing nature of stimuli that is critical for amplifying biased competition processes (Mather & Sutherland, 2011), and indeed, in the oddball experiment described above, both positive and negative emotionally arousing oddballs yielded retrograde enhancement and impairment depending on the oddball - 1 item priority (Sakaki, et al. 2014). However, in the current study, one limitation is that we cannot be certain that our manipulation is due to arousal rather than negative valence. One more limitation is that our study only revealed suppression effects of emotion on low priority items and not enhancement effects on high priority items. For instance, previous research suggests that solo target stimuli that are the focus of attention can be enhanced by subsequent emotional stimuli (Anderson, Wais, & Gabrieli, 2006; Knight & Mather, 2009; Sakaki, et al. 2014), yet we did not find a main effect of emotion on memory for preceding objects (we did not predict an interaction between emotion conditions and solo versus combination presentation of objects because objects were high priority in both cases). Future research is needed to see if the retrograde effects of emotion are indeed stronger in terms of suppressing weaker representations than in enhancing dominant representations or if we failed in the current study to endow the “high priority” information with enough priority. For instance, our cue label “background” for the scenes shown alone may have led participants to prioritize those scenes less than they would have otherwise.

Another question for future research is whether our effects depended on the 24-hour delay between encoding and test. We designed the study with this delay because prior research suggests that the effects of emotional arousal on memory consolidation are enhanced by sleep (Wagner et al., 2001). For instance, in one study, emotion-induced retrograde enhancement of items was seen a week after encoding, but not within the same session (Knight & Mather, 2009). Thus, our results may depend on the night of sleep between encoding and retrieval.

Previous research indicates that memory for contextual details is sometimes better for emotional items than for neutral items (Mather, 2007; Schmidt, Patnaik & Kensinger, 2011). Thus, we were interested in whether emotional effects in our paradigm would be specific for recollective memory that includes contextual details, rather than for recognition memory in general. To examine this, we included a “remember” vs. “know” rating on the memory test. Emotion effects did not vary significantly by rating type, suggesting that arousal broadly influenced recognition for preceding information and that effects were not specific to associative information (see also Knight & Mather, 2009).

In sum, emotional arousal impaired memory for low priority information but did not impair memory for high priority information. The fact that emotion influenced memory consolidation differently for low and high priority information supports the ABC framework. More generally, the findings indicate that emotion makes us even more likely to forget information that doesn’t seem important at the time.