Effect of emotional valence on retrieval-related recapitulation of encoding activity in the ventral visual stream

Abstract

While prior work has shown greater retrieval-related reactivation in the ventral visual stream for emotional stimuli compared to neutral stimuli, the effects of valence on retrieval-related recapitulation of successful encoding processes (Dm effects) have yet to be investigated. Here, seventeen participants (aged 19–35) studied line drawings of negative, positive, or neutral images followed immediately by the complete photo. After a 20-minute delay, participants performed a challenging recognition memory test, distinguishing the studied line drawing outlines from novel ones. First, results replicated earlier work by demonstrating that negative and positive hits elicited greater ventral occipito-temporal cortex (VOTC) activity than neutral hits during both encoding and retrieval. Moreover, the amount of activation in portions of the VOTC correlated with the magnitude of participants’ emotional memory enhancement. Second, results revealed significant retrieval-related recapitulation of Dm effects (Hits > Misses) in VOTC (anterior inferior temporal gyri) only for negative stimuli. Third, connectivity between the amygdala and fusiform gyrus during the encoding of negative stimuli increased the likelihood of fusiform activation during successful retrieval. Together, these results suggest that recapitulation in posterior VOTC reflects memory for the affective dimension of the stimuli (Emotional Hits > Neutral Hits) and the magnitude of activation in some of these regions is related to superior emotional memory. Moreover, for negative stimuli, recapitulation in more anterior portions of the VOTC is greater for remembered than forgotten items. The current study offers new evidence for effects of emotion on recapitulation of activity and functional connectivity in support of memory.

Introduction

There is accumulating interest in the effect of emotion on the encoding-to-retrieval overlap (Dew, Ritchey, LaBar, & Cabeza, 2014; Hofstetter, Achaibou, & Vuilleumier, 2012; Ritchey, Wing, LaBar, & Cabeza, 2013; Smith, Henson, Dolan, & Rugg, 2004; Taylor, et al., 1998) because the overlap of processes engaged during encoding and retrieval influences the probability of a memory coming to mind (Morris, Bransford, & Franks, 1977; Tulving & Thomson, 1973). Episodic memory retrieval is thought to represent a state of ‘mental time travel’ back to when the event was initially experienced (Tulving & Thomson, 1973). These theories suggest that, the more the brain begins to resemble its original encoding state during search, the greater the chance of successful memory retrieval. Consistent with this suggestion, extensive research has shown retrieval-related reactivation in visual processing regions that were engaged during encoding (Bosch, Jehee, Fernandez, & Doeller, 2014; Danker & Anderson, 2010; Gordon, Rissman, Kiani, & Wagner, 2013; Ritchey, et al., 2013; Sakai & Miyashita, 1991; Slotnick, 2009; Staresina, Cooper, & Henson, 2013; Staresina, Henson, Kriegeskorte, & Alink, 2012; Tanaka, et al., 2014). Indeed, previous studies have consistently demonstrated modality-, content-, and context-specific recapitulation processes for neutral remembered items (Danker & Anderson, 2010; Johnson & Rugg, 2007; Kuhl, Rissman, Chun, & Wagner, 2011; Maratos, Dolan, Morris, Henson, & Rugg, 2001; Otten & Rugg, 2001; Skinner, Grady, & Fernandes, 2010; Skinner, Manios, Fugelsang, & Fernandes, 2014; Smith, et al., 2004; Thakral, Wang, & Rugg, 2015; Wheeler, Petersen, & Buckner, 2000; Woodruff, Johnson, Uncapher, & Rugg, 2005).

Negative events tend to be remembered with more visual detail than positive or neutral events and can elicit greater, and more expansive, activity throughout the ventral visual pathway (Murty, Ritchey, Adcock, & LaBar, 2011). Previous studies have separately reported greater recruitment of visual processing regions during successful encoding (Kensinger, Garoff-Eaton, & Schacter, 2007; Mickley & Kensinger, 2008; Mickley Steinmetz & Kensinger, 2009; Todd, Schmitz, Susskind, & Anderson, 2013) and successful retrieval (Keightley, Chiew, Anderson, & Grady, 2011; Kensinger & Schacter, 2007; Mitchell, Mather, Johnson, Raye, & Greene, 2006; Taylor, et al., 1998) of negative visual stimuli, compared to positive and neutral visual stimuli. For instance, more expansive reactivation in the fusiform gyrus has been reported during retrieval of negative scenes, compared to neutral scenes (Hofstetter, et al., 2012). Similarly, Smith et al. (2004) reported overlapping activation in the left fusiform gyrus during encoding and recognition of objects previously presented on negative backgrounds, relative to positive and neutral backgrounds. These results suggest that enhanced recollection for negative information may be associated with enhanced overlap between visual processes engaged at encoding and retrieval. However, by holding memory constant and assessing memory only for remembered information (i.e., hits), the foregoing findings have not been able to assess the relation between the encoding-to-retrieval overlap and memory accuracy (i.e., activity that distinguishes remembered from forgotten information [Hits > Misses]). This distinction is important because when activity is compared for remembered items (hits) with different emotional valences, the differences that are revealed could be related to explicit memory processes (i.e., specific to hits) but they also could be related to emotion processes that are not specific to explicit memory (i.e., also present for misses).

To circumvent this problem, several studies have taken the approach of comparing activity to hits and to misses. These studies have revealed retrieval-related reactivation of the same visual processing regions that distinguish successful (subsequent hits) from unsuccessful encoding (subsequent misses). In other words, the same neural regions that are associated with successful encoding of neutral stimuli (i.e., encoding differences due to memory [Dm effects] (Paller & Wagner, 2002; Wagner, Koutstaal, & Schacter, 1999)) can be reactivated at the time of their retrieval (Gordon, et al., 2013; Hayes, Nadel, & Ryan, 2007; Karanian & Slotnick, 2015; Morcom, 2014; Prince, Daselaar, & Cabeza, 2005; Prince, Dennis, & Cabeza, 2009). For instance, Morcom (2014) reported retrieval-related reactivation of Dm effects in the left middle occipital/inferior temporal gyrus for neutral faces and scenes. Prince, Dennis, and Cabeza (2009) found right fusiform gyrus engagement during successful encoding and retrieval of faces.

The effect of emotion on retrieval-related recapitulation of Dm effects has yet to be investigated. Thus, it remains an open question whether the retrieval-related recapitulation of VOTC activity is greatest for emotional, or specifically for negative, items. That is, does recapitulation of VOTC activity occur more often during retrieval of negative items compared to neutral and positive ones? And, is that reactivation tied to explicit retrieval (hits) as compared to unsuccessful retrieval (misses)? Here, we also investigate whether greater activation in VOTC regions during retrieval of emotional trials is associated with superior emotional memory enhancement across participants, as prior studies have shown greater memory-related activation of OTC is associated with superior memory performance (Gron, et al., 2003; McIntosh, et al., 1999).

In the current study, we take three approaches to answer the preceding questions regarding the effect of emotion on retrieval-related recapitulation activation. First, we take the approach used in previous research, restricting analyses to Hits, and examining the role of VOTC reactivation in memory for the affective dimension of a stimulus (i.e., Negative Hits > Neutral Hits and Positive Hits > Neutral Hits overlap). Second, we investigate VOTC regions that correlate with the magnitude of emotional memory enhancement demonstrated across participants. Third, we test how emotional valence affects recapitulation processes that differentiate memory success from memory failure by comparing overlap in the encoding and retrieval processes for Hits > Misses for each valence.

In addition to examining encoding-to-retrieval overlap in activity, the present study also sought to examine whether VOTC connectivity with the amygdala overlapped at encoding and retrieval. Extensive evidence has confirmed a role for the amygdala during the successful encoding and consolidation of emotional memories (see Hamann, 2001; LaBar & Cabeza, 2006 for reviews). Furthermore, prior animal work has demonstrated anatomical projections from the amygdala to visual cortex (Amaral, Behniea, & Kelly, 2003; Amaral & Price, 1984; Freese & Amaral, 2005) and human work has shown functional connectivity between the amygdala and VOTC regions during emotion-related tasks (Herrington, Taylor, Grupe, Curby, & Schultz, 2011; Mickley Steinmetz, Addis, & Kensinger, 2010; Ritchey, et al., 2013; Robinson, Laird, Glahn, Lovallo, & Fox, 2010; Smith, Stephan, Rugg, & Dolan, 2006; Vuilleumier, Richardson, Armony, Driver, & Dolan, 2004) including emotional memory encoding (e.g., Kensinger & Schacter, 2007) and retrieval (e.g., Ritchey, et al., 2013).

Despite the evidence for functional connectivity between the amygdala and VOTC, the effects of negative and positive valence on the encoding-to-retrieval overlap of this network have yet to be investigated. Based on previous evidence for stronger connectivity between the amygdala and VOTC with increased arousal of negative stimuli (Mickley Steinmetz, et al., 2010), we anticipated that retrieval-related recapitulation of Dm effects associated with amygdala and VOTC connectivity during encoding would be strongest for negative items. Conversely, given prior work suggesting decreased connectivity between amygdala and VOTC with increased arousal of positively valenced stimuli (Mickley Steinmetz, et al., 2010), and evidence for reduced sensory processing of positive compared to negative stimuli (Mickley Steinmetz, et al., 2010; Nielen et al., 2009; Proverbio, Adorni, Zani, & Trestianu, 2009) we predicted less expansive or absent amygdala and VOTC connectivity during encoding and retrieval of positive stimuli.

Here, participants studied low-resolution line drawings of negative, positive, and neutral International Affective Picture System (IAPS) (Lang, Bradley, & Cuthbert, 2008) images followed by the full color photo. The novel use of line drawings as retrieval cues created a challenging visual recognition test without the confounding factor of re-presenting the original full color photo during test. The challenging recognition test also allowed for sufficient numbers of missed trials to compare hits to misses for each valence as well as sufficient variation in memory performance for us to examine the relationship between the magnitude of VOTC activation during retrieval and emotional memory enhancement across participants. This paradigm was used to examine the effect of emotion on retrieval-related recapitulation of VOTC regions engaged during successful encoding, using three approaches. Encoding-to-retrieval overlap was examined first when memory was held constant (i.e., Emotional Hits > Neutral Hits) and second when activity distinguished memory success from memory failure (i.e., hits vs. misses). Third, amygdala connectivity was assessed to examine whether encoding-to-retrieval overlap may be guided by connectivity between the amygdala and VOTC.

Materials and Methods

Participants

Twenty-four participants were recruited from Boston College and the greater Boston area. Seven participants were excluded from the analysis: six participants (aged 19–25, 3 females) due to insufficient number of trials per condition (i.e., fewer than 8) and one due to a technical failure resulting in loss of behavioral data (age 25, female). This resulted in 17 participants (7 females) aged 19 to 35 (M = 23.9; standard deviation [SD] = 4.0).

Participants were healthy, right-handed native speakers of English, with normal or corrected-to-normal vision. Participants reported no history of neurologic or psychiatric problems, learning disorders, head injury or medications affecting the central nervous system. Before entering the scanner, participants were screened for MRI environment contradictions. The Boston College Institutional Review Board approved this study and written informed consent of study procedures was obtained from all participants. Participants were compensated $25/hour for their participation.

Stimuli

Stimuli were 300 photos (100 negative, 100 positive, and 100 neutral) selected from the International Affective Picture System (IAPS) database (Lang, Bradley, & Cuthbert, 2008). Using IAPS normative data, negative and positive photos were pre-matched on arousal (Negative items M = 5.54, SD = 0.62; Positive items M = 5.43, SD = 0.61) and absolute valence (Negative items M = 2.04, SD = 0.79; Positive items M = 2.07, SD = 0.58). Two-tailed independent sample t-tests confirmed negative items were more arousing (t(198) = 23.14, p < 0.001) and valenced (t(198) = 19.28, p < 0.001) than neutral photos (Arousal M = 3.28, SD = 0.76; Absolute valence M = 0.42, SD = 0.31). Positive items were also more arousing (t(198) = 22.17, p < 0.001) and valenced (t(198) = 25.01, p < 0.001) than neutral photos. Line drawings of these IAPS photos where created using Adobe Photoshop and an in-house MATLAB script (see Figure 1c for depictions of stimuli).

Figure 1.

Stimuli and task structures. a) During encoding, participants were shown line drawing outlines of IAPS images (1.5 seconds), followed by the complete color photo (2 seconds). Participants made an approach/back away judgment during the presentation of each photo. b) During the recognition task, participants were shown studied line drawings intermixed with novel line drawings. At both encoding and recognition, fixation periods between trials were jittered (0.5–9.5s). c) Sample of line drawings of IAPS images.

Procedures

Following instruction and a brief practice, participants were scanned while they studied line drawings of IAPS photos that had negative, positive, or neutral valence followed by the complete color photo (i.e., they viewed each line drawing outline for 1.5 seconds immediately followed by the complete photo for 2 seconds) (Figure 1a). Participants were presented with 150 image sets (50 of each valence). In order to ensure participants were actively encoding the images, participants were asked to indicate, via an MRI-safe button box, whether they would ‘Approach the scene’ or ‘Back away from the scene’ for each image pair presentation.

Following a 20-minute delay, while still in the scanner, participants were given a surprise recognition memory test. After instruction and brief practice, participants were shown the line drawing outlines of the 150 previously studied photos intermixed with 150 line drawing outlines of photos they never studied (Figure 1b). All participants saw the same items at recognition; two study lists were used to vary which half of the items each participant had studied. Due to a programming error, one neutral photo was repeated and therefore removed from all fMRI analyses, leaving a total of 299 stimuli for analysis. Stimuli in the scanner were presented using MacStim (White Ant Publishing, Melbourne, Australia) on a MacBook and viewed using a rear projection system viewed through a mirror mounted to the head coil. For each item, participants were asked to make a recognition decision that combined an old-new recognition judgment and a confidence rating (1 = sure the corresponding photo was not studied, 2 = unsure, but think the corresponding photo was not studied, 3 = unsure, but think the corresponding photo was studied, 4 = sure the corresponding photo was studied) (Figure 1). All responses were made using an MRI-safe button box. The novel use of line drawings allowed for three advantages: 1) The ability to cue specific memories without re-presenting the full color images seen at encoding, 2) The use of less emotional retrieval cues than the original images, and 3) The creation of a challenging recognition task that yielded a sufficient number of remembered and forgotten items for analysis.

After completion of the retrieval task, participants were removed from the scanner and were shown the IAPS images presented earlier during the encoding phase and were asked to rate each photo’s valence and arousal on a 1–7 scale. These post-scan ratings confirmed that negative images were rated as more negative than neutral images (t(16) = 22.20, p < 0.001) and positive images were more positive than neutral images (t(16) = 15.99, p < 0.001). Post-scan arousal ratings confirmed that negative and positive images were both significantly more arousing than the neutral images (p < 0.001 for both comparisons). Although the stimuli were selected so that the negative and positive stimuli would be of equal arousal and absolute valence (i.e., distance from neutral valence), the post-scan arousal and valence ratings given by the participants showed that the negative images were rated as more arousing (t(16) = 8.11, p < 0.001) and valenced (t(16) = 6.49, p < 0.001) than the positive and neutral images.

After the experiment, participants were debriefed, compensated for their time, and dismissed.

FMRI Data Acquisition and Pre-Processing

Structural and functional images were acquired using a Siemens Tim Trio 3 Tesla scanner equipped with a 32-channel head coil. A functional localizer and auto-align scout were followed by collection of whole-brain T₁-weighted anatomical images (MPRAGE, 176 sagittal slices, 1.0 mm³ voxels, TR = 2530 ms, TE1 = 1.64 ms, TE2 = 3.5 ms, TE3 = 5.36 ms, TE4 = 7.22 ms, Flip angle = 7 degrees, 256 field of view, base resolution = 256) and T₂-weighted echo-planar images were acquired in an interleaved fashion, with the slices oriented perpendicular to the long-axis of the hippocampus (47 slices, 3.0 mm³ voxels, TR = 3000 ms, TE = 30 ms, Flip angle = 85 degrees, 216 field of view, base resolution = 72). A total of six functional runs were collected (2 encoding, 4 retrieval). A distortion-unwarping protocol was employed to correct for asymmetric distortions inherent in the coronal slice acquisition. This method implemented a pixel-by-pixel displacement map, derived from a pair of alternately-phase encoded spin-echo EPI images acquired before the first blood-oxygen-level dependent (BOLD) scan, to correct each time-point of the subsequent BOLD series as they were reconstructed on the scanner (Benner, van der Kouwe, Mainero, Holland, & Dale, 2011; Holland, Kuperman, & Dale, 2010). The first 4 scans of each run were discarded to allow for equilibrium effects; due to a programming error, stimuli were presented during these discarded scans for three participants. A diffusion weighted scan and a resting-state scan were included to create a delay between encoding and recognition scans; these data will not be discussed here.

Images were pre-processed and analyzed using SPM8 (Wellcome Department of Cognitive Neurology, London, United Kingdom) implemented in MATLAB 2009b. All functional images were reoriented, slice time corrected, realigned, co-registered, spatially normalized to the Montreal Neurological Institute template (resampled at 3mm during segmentation and written at 3 mm during normalization), and smoothed using a 6 mm isotropic Gaussian kernel. Global mean intensity and motion outliers were identified using Artifact Detection Tools (ART) (available at www.nitrc.org/projects/artifact_detect). Individual scan runs with more than 10 total outliers were excluded from analysis. Global mean intensity outliers were defined as scans with a global mean intensity that differed by more than 3 standard deviations from the mean. Acceptable motion parameters were set to ± 5 mm for translation and ± 5 degrees for rotation. In total, three scan runs were excluded across the sample from analysis (one encoding run for two different participants and one retrieval run for another participant).

General Linear Model

Here, we used fMRI analyses to 1) replicate prior findings that report greater reactivation in VOTC for emotional hits to neutral hits, 2) examine the correlation between retrieval activity and emotional memory enhancement, and 3) examine the effect of emotion on activity predictive of successful retrieval during each phase by comparing activity related to correctly remembered items (‘hits’, defined as all correct ‘old’ responses, collapsing across confidence) to forgotten items (‘misses’, defined as all incorrect ‘new’ responses, collapsing across confidence) by valence. Collapsing across confidence ensured enough trials per condition. There were an average of 30.1 analyzed hits/condition (range: 10–42) and 15.2 analyzed misses/condition (range: 8–33) for the encoding fMRI analyses and an average of 31.5 analyzed hits/condition (range 13–42) and 16.1 analyzed misses/condition (range: 8–27) for the retrieval fMRI analyses¹.

At the first-level of analysis and for each participant, two separate, rapid event-related design matrices were created: (1) an 8-column encoding regression matrix including 6 conditions of interest (subsequent hits and misses by valence) and 2 columns to regress out linear drift across the two concatenated encoding runs; and (2) an 11-column retrieval model including 6 conditions of interest (hits and misses by valence), 1 nuisance regressor that combined correction rejections (CRs) and false alarms (FAs), and 4 columns to regress out linear drift across the four concatenated retrieval runs.

Encoding-to-Retrieval Overlap

Ten whole-brain fixed-effects contrast analyses were conducted: comparing subsequent hits to subsequent misses for each valence during encoding (3 contrasts total), comparing hits to misses for each valence during retrieval (3 contrasts total), comparing one of the emotional (negative or positive) subsequent hits to neutral subsequent hits during encoding (2 contrasts total), and comparing one of the emotional (negative or positive) hits to neutral hits during retrieval (2 contrasts total).

At the random-effect level, the results of the fixed-effects contrasts were entered into 10 separate one-sample t-tests. Conjunction analyses were then used to examine the spatial overlap between encoding and retrieval (i.e., Subsequent Hits > Subsequent Misses ∩ Retrieval Hits > Misses). Five separate conjunction analyses were conducted. These conjunction analyses were achieved by inclusively masking retrieval activity with corresponding encoding activity. For example, to examine recapitulation of Dm effects for negative items, the comparison of Negative Hits > Misses during retrieval was inclusively masked with Negative Subsequent Hits > Subsequent Misses during encoding. To examine memory for the affective dimension of the stimulus, the comparison of Negative Hits > Neutral Hits during retrieval was inclusively masked with Negative Subsequent Hits > Neutral Subsequent Hits during encoding.

Correlations Between Behavioral Memory Performance and Retrieval Activity

We sought to demarcate regions that were correlated, across participants, with a larger emotional enhancement of memory (Emotional Item d′ > Neutral Item d′). First-level parameter estimates for emotional hits were entered into a one-sample t-test to examine the correlation between retrieval activity and emotional memory enhancement (Emotional Item d′ greater than Neutral Item d′).

Amygdala Functional Connectivity

Functional connectivity analyses were implemented using the generalized psychophysiological interactions (gPPI) toolbox (available at http://brainmap.wisc.edu/PPI; McLaren, Ries, Xu, & Johnson, 2012). To identify the amygdala seed region engaged during encoding and retrieval, we first used the general linear models for encoding and retrieval defined above to run two omnibus contrasts, one at encoding and one at retrieval, comparing all negative and positive trials (combining hits and misses) to baseline. In order to isolate the region of the amygdala engaged during encoding and retrieval of emotional images, the probabilistic bilateral amygdala mask published by Hammers et al. (2003) was applied to a conjunction of the encoding and retrieval omnibus contrasts (joint probability p < 0.005, k=10). This conjunction revealed a spatial overlap in 11 voxels of the left amygdala (MNI coordinates xyz = −30, −4, −17) and right amygdala (MNI coordinates xyz = +30, −4, −17). These clusters were used as seed regions for each participant for all subsequent gPPI analyses (see violet left amygdala cluster shown in Figure 4).

Figure 4.

Recapitulation of amygdala connectivity related to Dm effects for negative information. Amygdala seed region for all gPPI analyses visualized on the average of the anatomical scans for the seventeen participants (shown in violet). Conjunction analysis revealed that regions of the left fusiform gyrus that were engaged with the amygdala during successful encoding of negative items were reactivated during retrieval of negative items (shown in red). See Table 3 for all regions.

At the first-level and for each participant, the gPPI toolbox was used to: (1) create a psychological/task regressor (2) create the physiological variable by estimating the BOLD signal observed in the amygdala seed region; and (3) calculate the psychophysiological interaction term (PPIs). This interaction term identified memory-dependent changes in functional connectivity between the amygdala seed region and the entire brain, revealing the regions that were more strongly correlated with the amygdala during hits compared to misses. We conducted separate PPI analyses for positive and negative images, and separate PPI analyses for encoding and retrieval. This led to 4 single-subject PPI analyses, each analyzed in a one-sample t-test at the random-effects level.

These thresholded PPI contrast maps were used in two separate conjunction analyses to examine common amygdala connectivity for encoding and retrieval that was predictive of memory for negative and positive images. Specifically, two PPI contrast image conjunctions were computed: 1) connectivity during encoding of subsequent Negative Hits > connectivity during encoding of subsequent Negative Misses ∩ connectivity during retrieval of Negative Hits > connectivity during retrieval of Negative Misses and 2) connectivity during encoding of subsequent Positive Hits > Positive Misses ∩ connectivity during retrieval of Positive Hits > Positive Misses. To examine whether amygdala engagement during encoding guided retrieval-related reactivation, the encoding connectivity masks of Negative Hits > Negative Misses and Positive Hits > Positive Misses were conjoined with their corresponding retrieval univariate contrast of Hits > Misses (described above).

Data Reporting and Visualization

Unless otherwise specified, an individual voxel threshold of p < 0.005 (uncorrected) was enforced for all contrasts. In order to correct for multiple comparisons at p < 0.05, a 17 voxel extent (k) was determined using Monte Carlo simulations (https://www2.bc.edu/sd-slotnick/scripts.htm). While we focus our discussion on clusters that reach the corrected threshold, we report all clusters with at least 10 contiguous voxels in the tables to avoid Type II error (see Lieberman & Cunningham, 2009 for discussion). Unless otherwise specified, all visualizations of activation shown in the figures reflect a 17-voxel extent threshold. In all conjunction analyses, the joint probability for each individual voxel was set to p = 0.005 by setting the individual contrast thresholds for each voxel at p = 0.0243 (calculated using the Fisher equation; (Fisher, 1973).

All MNI coordinates from SPM8 were converted to Talaraich Coordinates using the GingerAle tool (http://www.brainmap.org/ale/). Unless otherwise specified, we report the whole-brain results from all analyses in the figures and tables. However, because our hypotheses were specific to the medial temporal lobe (MTL) and OTC, we restrict the in-text reporting to those regions.

Results

Behavioral Results

Memory Performance

Overall, old items were remembered 65.8% (SD = 11.02) of the time and new items were incorrectly endorsed as old 23.3% (SD = 10.4) of the time. Recognition memory performance (as calculated by d′) collapsed across valence was above chance (M_dprime = 1.22, SE = 0.1, range: 0.53–1.89; t(16) = 12.55 p < 0.001) and was equivalent (F(2,32) = 1.59, p = 0.221) for negative (M_dprime = 1.19, SE = 0.11, range: 0.53–2.32), positive (M_dprime = 1.31, SE = 0.13, range: 0.31–2.21), and neutral (M_dprime = 1.17, SE = 0.08, range: 0.73–1.83) stimuli. This equivalence is important, as it reduces concern that the results could be confounded by valence differences in the numbers of accurately remembered items included in the fMRI analyses^SN1.

Reaction Times

Reaction times (RTs) were entered into a 2×3 repeated-measures ANOVA with factors of response type (hit or miss) and valence (negative, positive, neutral). To be expected, there was a significant main effect of response type whereby RTs for hits (M = 1.62, SD = 0.16) were faster than RTs for misses (M = 2.01, SD = 0.27) (F(1, 16) = 49.09, p < 0.001). Importantly, there was no significant main effect of valence (F(2, 32) = 1.12, p = 0.34) nor a significant response type by valence interaction (F(2, 15) = 1.88, p = 0.187). Thus, a time-on-task explanation could not account for any valence differences.

Neuroimaging Results

Greater recapitulation of VOTC activity for emotional compared to neutral stimuli

Consistent with prior research, we first held memory success constant, considering only the trials that were hits, and examined whether the valence of those hits affected the recapitulation of VOTC activity. Conjunction analyses of the contrasts comparing negative hits and neutral hits during encoding and retrieval (i.e., Negative Hits > Neutral Hits at encoding ∩ Negative Hits > Neutral Hits at retrieval) showed significant overlap in the VOTC, including peak activations in the bilateral fusiform gyrus (BA37), left inferior temporal gyrus (BA20), middle occipital gyrus (BA19), and right temporal pole (BA38) (activity shown in red in Figure 2). The conjunction of encoding and retrieval comparisons of positive hits to neutral hits (i.e., Positive Hits > Neutral Hits at encoding ∩ Positive Hits > Neutral Hits at retrieval) showed significant overlap in bilateral middle occipital cortex (BA19) and cuneus (BA19) activity (activity shown in blue in Figure 2, activity shown in cyan reflects voxels that overlap with the Negative Hits > Neutral Hits conjunction). See Table 1 and Supplementary Table 1 for all regions. There was no significant overlap in MTL regions.²

Figure 2.

Regions showing significant spatial overlap for emotional hits compared to neutral hits and correlations with emotional memory enhancement in VOTC. Compared to neutral items, conjunction analysis confirmed greater encoding-to-retrieval overlap in VOTC for negative hits (in red) and, positive hits (in blue), with some overlap between these regions (in cyan). Emotional memory enhancement at retrieval was positively correlated with VOTC activation (activity in green and yellow). See Table 1 and Supplementary Table 1 for list of all regions.

Table 1.

Regions that showed spatial overlap of emotional activity greater than neutral activity during encoding and retrieval.

Lobe	Region	Hem	Approximate Brodmann’s area	MNI Coordinates (x,y,z)	TAL Coordinates (x,y,z)	Cluster remains when individual maps thresholded at p < 0.005?
Encoding ∩ Retrieval (Negative Hits > Neutral Hits)
Occipital	Middle occipital gyrus	R	19	51,−76,4	46,−73,2	Y
Temporal	Fusiform gyrus	L	37	−42,−49,−20	−40,−45,−19	Y
		R	37	36,−46,−23	32,−43,−20	Y
	Inferior temporal gyrus	L	20	−24,2,−41	−23,4,−33	N
	Middle temporal gyrus	L	21	−54,−1,−20	−51,−1,−15	Y
	Parahippocampal cortex	L	30	−18,−34,−5	−18,−33,−4	N
	Temporal pole	R	38	42,20,−35	38,20,−25	N
Frontal	Superior frontal gyrus	R	8	12,50,49	10,40,52	Y
Parietal	Inferior parietal lobule	L	40	−60,−31,31	−57,−33,28	N
	Precuneus	R	7	9,−52,40	7,−54,35	N
Other	Posterior cingulate	L	29	−3,−46,13	−4,−46,12	N
Encoding ∩ Retrieval (Positive Hits > Neutral Hits)
Occipital	Cuneus	R	19	9,−94,22	7,−91,16	N
	Middle occipital gyrus	L	19	−42,−67,13	−40,−65,9	N
		R	19	54,−73,−2	49,−70,−3	N

Note: MNI=Montreal Neurological Institute, TAL=Talaraich.

Correlation between retrieval-related VOTC activation and memory performance across participants

During retrieval of emotional items (Emotional Hits), bilateral activity in the VOTC including occipital and temporal portions of the fusiform gyrus (BA37; MNI coordinates xyz = +42, −46, −26, k=50; BA18; MNI coordinates xyz = −27, −88, −23, k=22; BA18; MNI coordinates xyz = −24, −100, −17, k=15;), left inferior temporal gyrus (BA20; MNI coordinates xyz = −45, −40, −23, k=33), right middle occipital gyrus (BA19; MNI coordinates xyz = +48, −73, +19, k=32; BA18; MNI coordinates xyz = +30, −88, +16, k=15) and a small portion of the right inferior temporal gyrus (BA20; MNI coordinates xyz = +42, −7, −38, k=11) correlated with the emotional enhancement of memory (i.e., Emotional d′ > Neutral d′; activity shown in green in Figure 2). Portions of these regions overlapped with those that showed encoding-to-retrieval overlap for Negative Hits > Neutral Hits (overlap shown in yellow in Figure 2).

Importantly, RTs for emotional hits did not correlate with d’ values (p > 0.45 for both comparisons), and thus the results are not a result of time-on-task effects.

Retrieval-related recapitulation of Dm effects for remembered negative items

We next examined whether valence affected the recapitulation of activity specifically tied to successful (vs. unsuccessful) memory retrieval (i.e., recapitulation of Dm effects). For negative items, conjunction analyses revealed significant spatial overlap in VOTC regions related to successful subsequent memory (i.e., Hits > Misses at encoding) and successful retrieval (i.e., Hits > Misses at retrieval). Of particular relevance, we found significant overlap in the bilateral inferior temporal gyri (ITG; see activity in red in Figure 3 and see Table 2 and Supplementary Table 2 for all regions). There was no evidence of recapitulation effects for positive items in these regions^SN2 and no spatial overlap of activity related to subsequent memory and successful retrieval for positive information or neutral information at our a priori thresholds (see Supplementary Table 2 for regions active during encoding only or during retrieval only and Supplementary Note 3 for thresholds at which spatial overlap was revealed for neutral and positive items).

Figure 3.

Retrieval-related recapitulation of Dm effects for negative hits in ITG. Conjunction analysis revealed extensive overlap in VOTC regions recruited during successful encoding and retrieval of negative stimuli (in red, see Table 2 and Supplementary Table 2 for all regions.). Regions of the ITG that showed a significant valence by response type interaction are shown in green (Negative Hits > Misses, Positive Misses > Hits; joint probability p < 0.005), with the spatial overlap of the joint probability maps shown in yellow.

Table 2.

Regions that showed spatial overlap of successful memory activity during encoding and retrieval of negative photos.

Lobe	Region	Hem	Approximate Brodmann’s area	MNI Coordinates (x,y,z)	TAL Coordinates (x,y,z)	Cluster remains when individual maps thresholded at p < 0.005?
Temporal	Inferior temporal gyrus, Middle temporal gyrus	L	20/21/37	−60,−46,−11	−57,−43,−11	Y
		L	20/21	−48,−7,−26a	−45,−6,−20	Y
	Inferior temporal gyrus	R	20	45,−13,−29a	41,−12,−22	N
	Middle temporal gyrus	R	21	57,−34,−11	52,−33,−8	N
Frontal	Inferior frontal gyrus	L	45	−48,32,4	−45,28,10	Y
	Superior frontal gyrus	L	8	−12,38,55	−13,29,56	Y
Parietal	Inferior parietal lobule	L	40	−48,−55,49	−46,−57,42	Y
		R	39	54,−61,43	48,−63,38	Y
Other	Anterior cingulate	L	32	−21,17,−29	−20,17,−21	N

Note:

^aDenotes a VOTC cluster that also shows a significant interaction with positive items, whereby memory the relationship for Negative Hits vs Negative Misses is stronger than Positive Hits vs. Positive Misses.

To confirm the differential recapitulation for negative stimuli compared to positive stimuli, a follow-up conjunction analysis was used to identify regions that showed a valence by response type interaction across both the encoding and retrieval phases. The analysis, revealing the regions that showed a stronger relation to Negative Hits vs. Negative Misses than to Positive Hits vs. Positive Misses at both encoding and retrieval, confirmed that recapitulation in the bilateral anterior inferior temporal gyrus was stronger for negative stimuli^SN4 (right ITG: MNI coordinates xyz = +48, −4, −35; left ITG: MNI coordinates xyz = −48, −10, −32; see green and yellow regions in Figure 3). No regions showed a valence by response type interaction with the stronger relation to Positive Hits vs. Positive Misses as compared to Negative Hits vs. Negative Misses.

Effect of valence on retrieval-related recapitulation of amygdala connectivity

For negative stimuli, there was evidence that the VOTC regions that covaried with the left amygdala during successful encoding were those that were reactivated during successful retrieval, while there was no significant overlapping amygdala and VOTC connectivity across both phases. Specifically, a conjunction analysis revealed that regions of the bilateral fusiform gyrus (BA37) more strongly connected to the left amygdala during the successful encoding of negative information and more active during the successful retrieval (Hits > Misses) of negative items (see activity in Figure 4 and Table 3 for all regions). These results suggest that regions engaged with the amygdala during successful encoding of negative stimuli are likely to reengage during successful retrieval.

Table 3.

Regions that showed left amygdala connectivity during successful encoding of negative photos that were reactivated during retrieval (Encoding Amygdala Connectivity [Negative Hits > Negative Misses] ∩ Retrieval Activity [Negative Hits > Negative Misses]).

Lobe	Region	Hem	Approximate Brodmann’s area	MNI Coordinates (x,y,z)	TAL coordinates (x,y,z)	Cluster size (no. of voxels)
Temporal	Fusiform gyrus	L	37	−39,−52,−11	−37,−49,−11	37
		R	37	42,−55,−14	38,−52,−13	47
	Superior temporal gyrus	L	22	−54,−46,13	−51,−45,11	27
Frontal	Medial frontal gyrus	L	9	−6,47,10	−7,41,17	92
		L	6	0,−4,55	−2,−10,53	28
		L	10	−3,56,−2	−4,51,7	26
Frontal	Precentral gyrus	L	6	−54,−4,25	−51,−7,25	17
Parietal	Superior parietal lobule	L	7	−30,−43,67	−30,−48,60	23
Other	Cingulate gyrus	L	31	−6,−34,37	−7,−37,34	82
		L	32	0,44,−14	−1,40,−5	25
		R	24	6,2,28	4,−2,29	17
	Claustrum	L	NA	−33,2,−14	−31,1,−9	32
	Insula	L	13	−42,−7,10	−40,−9,12	65
		R	13	45,−4,10	40,−7,13	20

At standard thresholds, there was no significant retrieval-related reactivation of VOTC regions that covaried with the amygdala during encoding of positive items^{SN5 and SF1}. Analysis of the right amygdala seed region revealed no significant connectivity with the VOTC for either valence^SN6.

Discussion

The present findings offer novel evidence for effects of emotion on retrieval-related recapitulation in the ventral visual stream. The results reveal that not only is there more retrieval-related recapitulation for negative items but also this recapitulation is tied to the successful retrieval (Hits vs. Misses) of those items. Further, amygdala connectivity during the initial encounter with an emotionally arousing stimulus influences later retrieval-related recapitulation processes in VOTC for negative stimuli, suggesting that connectivity between the amygdala and VOTC guides successful encoding and influences downstream recapitulation processes.

There are three key results, which both support and extend prior research, confirming the importance of recapitulation for memory accuracy and also suggesting that the neural processes that support ‘mental time travel’ back to an encoding event may differ as a function of emotional valence. First, we found greater activation of occipital cortex and VOTC during encoding and retrieval of emotional stimuli, compared to neutral stimuli, and a subset of activity in the fusiform gyrus, middle occipital gyrus, and inferior temporal gyrus was tied to the magnitude of the behavioral emotional memory enhancement demonstrated across participants. Second, spatial overlap of successful encoding and retrieval processes was found most strongly for negative items. Specifically, we found retrieval-related recapitulation in regions of the bilateral inferior temporal gyri that showed Dm effects during encoding of negative items (Figure 3), which suggests VOTC engagement facilitates retrieval of higher-order visual representations of negative stimuli. Importantly, we found anterior regions of the inferior temporal gyrus that survived a response type by valence interaction analysis across both phases, which suggests a stronger relationship to negative item Dm effects than positive item Dm effects. Third, we found enhanced connectivity between the left amygdala and the fusiform gyrus during successful (vs. unsuccessful) encoding of negative information, and later reactivation of these fusiform regions during successful retrieval (Figure 4). These findings suggest that amygdala connectivity with VOTC regions during the initial experience of a negative event increases the likelihood of their reactivation during retrieval.

According to the cortical reinstatement hypothesis, the more the brain resembles its original state during encoding, the more likely it is to make a correct match during retrieval (Morris, et al., 1977; Tulving & Thomson, 1973). While previous studies have examined the effect of emotion on encoding-to-retrieval similarities for remembered items, to our knowledge, our study is the first to examine the effects of valence on recapitulation of Dm effects and amygdala connectivity predictive of memory success both within and across participants. Our results corroborate prior evidence for recapitulation of the broader affective dimension for both negative and positive stimuli, compared to neutral stimuli. However, recapitulation of the affective dimension appears to be more robust for negative items, as evidenced both by more voxels that survived the conjunction of Negative Hits > Neutral Hits, compared to Positive Hits > Neutral Hits.

Our results also revealed a connection between VOTC recapitulation and behavioral memory performance. Across participants, increased engagement of the fusiform gyrus, middle occipital gyrus, and inferior temporal gyrus during successful retrieval of emotional stimuli was positively correlated with emotional memory enhancement. In other words, those participants who re-engaged VOTC to a greater extent during the retrieval of emotional stimuli also showed an enhanced ability to discriminate those emotional stimuli relative to neutral stimuli. These results suggest a link between VOTC activation and emotional memory success.

Critically, the current study was designed to compare recapitulation effects for successful vs. unsuccessful memory processes within-subject, sorting items based on their memory success (i.e., Hits vs. Misses). By using low-resolution line drawings of IAPS images as retrieval cues, a challenging, visual recognition task was created without the confounding factor of re-presenting the original full color photo during test, a common confound in other recapitulation studies. This approach allowed for cueing of memories for specific items, rather than cueing source or general stimulus class judgments (e.g., cue word previously studied with a face or scene or cue previously studied with emotional or neutral stimuli) as has been done in the bulk of foregoing recapitulation work. Although it is possible that the line drawings carried some level of intrinsic valence that induced an affective reaction during retrieval, the second and third analyses (Negative Hits > Negative Misses) controlled for that potential confound by examining memory success from memory failure within each valence category. It is likely that the line drawings cue memories for the previously studied photos and provide access to the emotional states that accompanied the original perception of those items. It is also possible that the line drawings cue memories of the prior affective state even without the perceptual content of the image returning to mind, although the recapitulation of ventral temporal activation may minimize the plausibility of this alternative.

Although there were weak patterns of recapitulation processes for positive and neutral information, the effects only reached significance for negative stimuli and interaction analyses confirmed that the patterns were strongest for negative stimuli. Taken together, these data provide additional support for recapitulation processes reflective of encoding and retrieval of negative stimuli (i.e., Negative Hits > Neutral Hits) and provide novel evidence for recapitulation processes supporting successful memory for negative items (i.e., Negative Hits > Negative Misses) within the broader negative valence stimulus category.

Retrieval-related recapitulation of Dm effects in higher-order visual processing regions such as regions of the anterior inferior temporal gyrus, as opposed to posterior visual processing regions or primary visual cortex, is in agreement with prior work that suggests fine-grained representations might not be necessary for retrieval of higher-order sensory representations, such as complex scenes (Bosch, et al., 2014). Wheeler and Buckner (2003) found activation of the inferior temporal cortex supported visual object memory, suggesting an efficient retrieval process by way of top-down activation of higher-order visual processing regions that store visual memory representations, without reactivation of low-level visual processes that were engaged during initial perception.

The results further suggest that—while there was no significant overlap in amygdala connectivity Dm effects in VOTC—the VOTC regions engaged with the amygdala during successful encoding of negative stimuli have a higher likelihood of being reactivated during the retrieval of those stimuli. Specifically, we found reactivation in the bilateral fusiform gyrus that covaried with the amygdala during encoding of negative stimuli. This finding aligns with an extensive literature demonstrating anatomical and functional connections between the amygdala and visual processing regions (Amaral, et al., 2003; Amaral & Price, 1984; Freese & Amaral, 2005; Herrington, et al., 2011; Mickley Steinmetz, et al., 2010; Ritchey, et al., 2013; Robinson, et al., 2010; Smith, et al., 2006; Vuilleumier, et al., 2004), but emphasizes that these interactions with VOTC may be more relevant during the successful formation of memories for negative visual stimuli than during their conscious retrieval. These findings suggest that the amygdala may serve an enhancing purpose during encoding to augment VOTC-based memory traces for negative stimuli, enabling these traces to be more readily re-accessed later during retrieval, independent of amygdala engagement. In other words, the role of the amygdala during the encoding of negative visual stimuli^SN7 has downstream effects on patterns of activation during retrieval.

Limitations

We encountered an unanticipated confound of higher post-scan arousal ratings for negative than positive stimuli. The stimuli had been equated on normative data from the IAPS (Lang, Bradley, & Cuthbert, 2008) and it is unclear whether the ratings of the studied images given by our participants were influenced by previous exposure during the task. Further work is needed to clarify if the degree of functional separation for negative and positive memories will hold when subjective arousal is equated at both the time of encoding and of retrieval.

It is also worth noting that the current results might be constrained to relatively short retrieval delays. It has been suggested that recollection memory might rely on perceptual processes over short delays (Uncapher & Rugg, 2005). Thus, recapitulation of VOTC activation may be particularly strong after shorter delays. While our study reveals recapitulation effects after a delay of 20 minutes, future research is needed to investigate how visual memory stores evolve into longer-term memories.

General Conclusions

In conclusion, this study was the first to show retrieval-related recapitulation of VOTC activity engaged during successful encoding of emotional stimuli, and to demonstrate that this reactivation is strongest for negative stimuli. Here, we replicated prior work that shows greater OTC activity for emotional stimuli, compared to neutral, and we reveal that portions of the VOTC are associated with emotionally enhanced memory across participants. While positive and negative stimuli elicited greater VOTC activity during successful encoding and retrieval relative to neutral items, recapitulation in VOTC supported memory most strongly for negative items. Moreover, the results provide evidence that some of the VOTC regions that are reinstated during negative memory retrieval are also those that covary with the amygdala during encoding. Together, these results suggest that the VOTC supports encoding and retrieval processes for emotional—and particularly negative—information, likely enhanced by the amygdala during encoding. Together, the current study offers new evidence demonstrating differential effects of valence on recapitulation and connectivity profiles, which is key to understanding the effect of emotion on cortical reinstatement and phenomenological experience of memory.

Highlights.

• Examines the effect of emotion on the links between recapitulation and memory

• Greater recapitulation in VOTC for emotional stimuli compared to neutral stimuli

• Retrieval activity in VOTC associated with superior emotional memory performance

• Recapitulation of Dm effects in anterior VOTC strongest for negative stimuli

• VOTC regions reactivated for negative stimuli correlate with amygdala at encoding