Skip to main content
PMC home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2007 Apr 18;27(16):4366–4373. doi: 10.1523/JNEUROSCI.3310-06.2007

Independent Effects of Emotion and Working Memory Load on Visual Activation in the Lateral Occipital Complex

Jan Gläscher 1,✉, Michael Rose 1, Christian Büchel 1
PMCID: PMC6672316  PMID: 17442821

Abstract

Emotional salience and working memory (WM) load are known to affect object processing in the ventral stream. However, the combined effect, which could be either synergistic or antagonistic, remains unclear. Using functional magnetic resonance imaging and a three-factorial design, we investigated the effects of WM load and emotional salience on object processing in the ventral visual stream. Twenty-three female subjects were shown blocks of task-irrelevant and more or less degraded visual stimuli that varied in emotional valence (negative or neutral) and phase coherence rendering them more or less noisy. Superimposed on these pictures, subjects saw colored squares on which they had to perform a demanding WM task (one-back, two-back). This WM task absorbs attentional resources normally available for the perceptual analysis of visual objects and can therefore be interpreted as a manipulation of attention. We hypothesized that attenuated processing resources in the lateral occipital complex (LOC) for these task-irrelevant pictures under high WM load (Rose et al., 2005) could be regained when they were of negative emotional valence. Our results indicate that both emotional salience and WM load critically depend on a minimum level of phase coherence of the stimuli to affect LOC activation. Furthermore, the influences of emotional salience and WM load do not interact with each other in LOC. Rather, emotional salience exerts a general multiplicative gain effect while preserving the difference in activation between low and high WM load. A connectivity analysis suggests that the emotional modulation might originate in the amygdalo-hippocampal junction.

Keywords: emotion, working memory load, phase coherence, lateral occipital complex, amygdala, psychophysiological interaction, multiplicative gain, emotion and attention interaction

Introduction

Perceptual processing in the ventral visual stream in humans is not only driven by the quality of the sensory information coming from the retina but is also subject to modulatory influence from different cognitive systems, including attention, working memory (WM), and emotional salience. These modulatory effects have been independently demonstrated (Pessoa et al., 2002a; Vuilleumier, 2005), yet the question of whether these effects interact remains unclear.

A common way to manipulate the quality of sensory input is the degradation of the visual stimulus (Rainer et al., 2001). Attentional manipulations can range from direct to indirect by either explicitly directing attention to a specific spatial location (Posner, 1980) or by absorbing attentional resources in a cognitively demanding task and measuring the quality of task-irrelevant processing. One example of the latter is WM, a closely related phenomenon (Kastner and Ungerleider, 2000), that can impose considerable load on the system, leading to an attenuation of the processing of irrelevant stimuli because resources are exhausted (Lavie, 1995).

Recently, we tested the effects of WM on the processing of background stimuli of different visual integrity (Rose et al., 2005). Subjects performed a WM task while simultaneously viewing task-irrelevant pictures at different levels of phase coherence. High WM load led to a more gradual increase in blood oxygenation level-dependent (BOLD) signal across the different levels of phase coherence in the lateral occipital complex (LOC) known to be involved in object recognition (Grill-Spector and Malach, 2004).

Emotional salience, often manipulated through stimulus valence, also exerts a modulatory influence on ventral stream areas (Pessoa et al., 2002a; Anderson et al., 2003; Vuilleumier, 2005), a phenomenon sometimes called natural selective attention (Keil et al., 2005), because emotionally evocative stimuli are thought to modulate the attentional selection of available stimuli. Visual perception benefits from emotional salient stimuli (Reinders et al., 2005), probably mediated through the elicited arousal (Sabatinelli et al., 2005), and with likely involvement of the amygdala (Vuilleumier et al., 2004), a structure that maintains multiple anatomical connections with ventral visual areas (Amaral et al., 1992).

Here we report findings from a study that allowed us to characterize the combined or independent influences emotional salience and WM load on the processing of task-irrelevant stimuli. In addition to phase coherence and WM load (Rose et al., 2005), we also include an emotional manipulation of the background stimuli in a design involving the factors emotion, load, and phase coherence (see Fig. 1A). [The terms emotion, load, and phase coherence are in quotation marks whenever we refer to the specific realization of our factors in this experiment. We use the terms emotional salience, WM load, and attention without quotation marks when we discuss these concepts at a theoretical level or in reference to other studies.] We hypothesized that the attenuated processing resources in LOC during the demanding WM task could be regained through emotionally salient background pictures. Furthermore, we expected that the emotional influence would originate in the amygdala (Morris et al., 1998; Pessoa et al., 2002b; Das et al., 2005).

Figure 1.

Figure 1.

Open in a new tab

A, Factorial design and experimental task. The three factors WM load (levels of one-back and two-back), emotion (levels of negative and neutral), and phase coherence (levels of 0, 33, 66, and 100%) yielded a total of 16 experimental conditions. B, Example for experimental task. Subjects were confronted with blocks of 10 visual stimuli drawn form the International Affective Picture System. Stimuli were presented for 500 ms (SOA of 1500 ms), and each block contained three n-back targets, the last of which occurred on either position 9 or 10 in the block. The top row represents an example from the condition negative, two-back, 100% phase coherence, whereas the bottom row is an example for the condition neutral, one-back, 33% phase coherence. On completion of six scanning sessions, subjects were confronted with a surprise recognition task asking for OLD/NEW judgments on all targets presented in the scanner and an equal amount of distracter stimuli.

Materials and Methods

Volunteers.

We scanned 23 right-handed female subjects (mean ± SD age, 24.4 ± 2.8 years). The issue of potential differences in brain activation in male and female subjects has not been decisively settled yet (Hamann and Canli, 2004). Whereas one study did not find significant gender differences in the activation of primary and early extrastriate visual areas to emotional stimuli (Sabatinelli et al., 2004), a body of literature implicates gender differences in size, connectivity, and lateralization of activation in the amygdala (for review, see Cahill, 2006). Given that we hypothesized an emotional influence on ventral stream activation originating in the amygdala, we decided to rule out potential gender differences by restricting our sample to only one gender (females). All subjects were free of any neurological or psychiatric disorder and had normal or corrected-to-normal vision. All subjects signed a consent statement approved by the local ethics committee.

Experimental design and task.

Our experimental design was a 2 × 2 × 4 factorial blocked design with the within-subject factors emotion (negative, neutral), load (one-back, two-back), and phase coherence (0, 33, 66, 100%) (Fig. 1A). The entire scanning procedure was completed in six runs of 7.25 min each. Within each run, subjects saw 16 blocks of stimuli corresponding to each of the 16 different experimental conditions. Thus, subjects were exposed to each experimental condition six times (in six sessions). Subjects were instructed that they would see blocks of 10 stimuli at different degrees of phase coherence and emotional valence, with a colored square overlaid on top of each image. They were told that they should perform either a one-back or a two-back task on these colored squares and that they should focus on the task and disregard the pictures in the background (for examples of the task, see Fig. 1B). Subjects also were instructed to press a button whenever an n-back target appeared.

Each block consisted of 10 picture stimuli presented for 500 ms with a stimulus onset asynchrony (SOA) of 1500 ms. A colored square was overlaid on each picture (Fig. 1B). There were six possible colors (red, green, blue, yellow, brown, and magenta) whose brightness adjusted for maximal discriminability in the scanner environment. Background pictures, however, were presented in their original unmodified brightness. Blocks were separated by 10 s of rest in which subjects were instructed by a number (1 or 2) which n-back task they should perform during the next block. The number appeared immediately after the termination of the stimulus block and remained visible until 500 ms before the start of the next block. The stimulus sequence within each block was randomized as well as the block sequence within each run. To further control for potential order effects, we performed this randomization procedure for each subject individually. Each block contained three n-back targets. Target position was also randomized with the restriction that the last target always occurred at position 9 or 10 within the block to enforce the cognitive task set throughout the entire block.

After scanning, subjects were confronted with a surprise recognition task in which all targets and an equal number of novel (i.e., distracter) stimuli were presented at 100% phase coherence in a randomized order specific for each subject. Subjects were instructed to submit OLD/NEW judgments for each item. It was also pointed out to them that they would probably make more NEW than OLD judgments because, as a result of the degradation of the phase coherence, some of the target pictures were only barely recognizable or not at all. Thus, subjects were specifically instructed to decide for each item separately and not to match the total number of OLD and NEW responses.

Stimulus selection.

We selected a total of 160 negative and 160 neutral pictures from the International Affective Picture System (IAPS) (Lang et al., 1999) based on the normative ratings supplied with these stimuli (excluding erotica). Mean ± SD normative valence ratings were 2.30 ± 0.79 and 5.35 ± 0.82 on a nine-point Likert scale for negative and neutral pictures, respectively (1, negative; 5, neutral; 9, positive). Mean ± SD arousal ratings were 5.95 ± 1.03 and 3.51 ± 0.9 on a nine-point Likert scale for negative and neutral pictures, respectively. The pictures in each class were then randomly assigned into the target category presented during scanning and the distracter category presented along with the targets during the recognition task. To ensure that target pictures and distracters did not significantly differ in mean normative valence and arousal ratings, we calculated a one-way ANOVA on both valence and arousal ratings between targets and distracters and rejected all random assignments that did not meet our statistical criterion (F < 0.79; p > 0.5). Negative and neutral target pictures were then randomly assigned into four phase coherence categories nested within the two load conditions. This assignment was similarly controlled for nonsignificant differences between valence and arousal ratings for all n-back and phase coherence levels. To further control for potential effects attributable to stimulus selection, we performed this random assignment for each subject separately. Thus, each subject received not only an individual event train but also an individual selection of stimuli into the different factor levels.

The four levels of phase coherence (0, 33, 66, and 100%) were realized by partial random scrambling of the phase information of each image (Rainer et al., 2001; Reinders et al., 2005). First, we performed a Fourier transform on each image yielding the amplitude and phase components of each image. A new phase component for each image was then rebuilt using the following algorithm: new phase = phase coherence × original phase + (1 − phase coherence) × random phase. Before rebuilding the new scrambled images with an inverse Fourier transform, we averaged the amplitude components of all images, thus ensuring that all presented stimuli had the same amplitude spectrum and differed only in their phase components. Finally, the picture borders of these new stimuli were smoothed with a 51 × 38 pixel full-width at half-maximum (FWHM) Gaussian filter to remove sharp picture edges from the stimuli that are known to elicit activations in primary and, to a lesser extent, in extrastriate visual areas (Kourtzi and Kanwisher, 2000). Similarly, the colored squares on which subjects had to perform the n-back task were also smoothed to ensure a gradual transition to the underlying picture stimulus. Figure 1B visualizes the results of this image scrambling for the negative two-back condition at 66% phase coherence and a neutral one-back condition at 33% phase coherence.

Data acquisition.

Functional imaging was performed on a 3 T Siemens (Erlangen, Germany) Trio scanner. Forty contiguous descending transversal slices of echo-planar T2*-weighted images were acquired in each volume, with a slice thickness of 2 mm and 1 mm gap (repetition time, 2290 ms; echo time, 25 ms; flip angle 70°; field of view, 192 mm2; matrix, 64 × 64). One hundred ninety images were acquired in each session lasting 7.25 min. We discarded the first four images before data processing and statistical analysis to compensate for the T1 saturation effects.

Data processing.

Image processing and statistical analyses were performed using SPM2 (http://www.fil.ion.ucl.ac.uk/spm). All volumes from all sessions were corrected for differences in slice acquisition, realigned to the first volume, spatially normalized to a standard echo planar imaging template included in the SPM software package (Friston et al., 1995) using third-degree B-spline interpolation, and finally smoothed with an isotropic 10 mm FWHM Gaussian filter to account for anatomical differences between subjects and to allow for valid statistical inference at the group level.

Statistical analysis.

For the analysis of the behavioral data during n-back performance in the scanner, we averaged the number of errors and the reaction times (RTs) for each subject in all 16 experimental conditions. These data were then subjected to a repeated-measures ANOVA. For the performance during the recognition task, we calculated the discriminability index d′ in each subject for all experimental conditions. Because some subjects did not score any hits (H) in the low phase coherence conditions nor any false alarms (FA) in the high phase coherence conditions, we applied the correction to hit rate (HR) and false alarm rate (FAR) as recommended by Snodgrass and Corwin (1988): HR = (H + 0.5)/(H + FA + 1) and FAR = (FA + 0.5)/(FA + CR + 1) (CR, correct rejections). In addition, because the distracters were never presented during the functional magnetic resonance imaging (fMRI) experiment, FAR could be only calculated for the two “emotion” conditions (because they were derived from the normative IAPS ratings) but not for the different conditions of load or phase coherence. Thus, for the d′ indices for negative pictures, we subtracted the z-score of FARneg from all z-scores of HRneg for all load and phase coherence conditions. For the neutral pictures, we did the analogous. These d′ indices revealed a highly similar pattern across the different experimental conditions as the percentage correct scores, which we calculated using hits and misses (data not shown). Statistically, we tested for significant effects in these d′ indices in the context of a repeated-measures ANOVA. When applicable, the degrees of freedoms were adjusted according to a Greenhouse–Geisser correction.

The analysis of the imaging data commenced at the single-subject level by specifying a model including all experimental conditions as separate regressors for all six scanning sessions. Because we were concerned about confounding brain activations in which subjects were not performing the n-back task well, we rejected those experimental blocks in which subjects committed more than two errors. This led to a rejection rate of 0.91% across all experimental blocks in all subjects. These experimental blocks were modeled as a separate nuisance regressor. In addition, we included the six rotational and translational parameters from the rigid body transformation obtained during image realignment. The serial autocorrelation of the BOLD time series was modeled using a first-order autoregressive model, and the high-pass filter was set to 1120 Hz. After model estimation, we created linear contrasts that averaged all six instances of each experimental condition. These contrast images were then included in a second-level group analysis in the context of a repeated-measures ANOVA, correcting appropriately for potential nonspherical distribution of the error term. We then created linear contrasts testing for the main effects and the two-way and three-way interaction terms, thus allowing us to address the question of interacting or independent influences of emotion and attention on visual processing in LOC.

To test whether the sources of the emotional modulation that we observed in LOC could originate in the amygdala, we performed a psychophysiological interaction (PPI) analysis (Friston et al., 1997) with the seed voxel located in the amygdalo-hippocampal junction as determined by the main effect of emotion (100% phase coherence only). As the psychological/stimulus dimension, we chose the load × phase coherence interaction known to modulate the activation in LOC (Rose et al., 2005). This analysis then tests for a significant covariation of the interaction of the time series in the seed region and the stimulus variable with any other parts of the brain. Hence, for every subject, we created new design matrices comprising three regressors: (1) the psychophysiological interaction term (time series × stimulus variable), (2) the time series in the seed region, and (3) the psychological/stimulus variable. The seed voxel for each subject was selected as the maximum effect size from the contrast for the main effect of emotion in a 6 mm sphere centered on the peak voxel in the amygdalo-hippocampal junction in the group random effects analysis. After the estimation of these single-subject PPI analyses, we submitted the parameter estimate images for the PPI regressor to a second-level one-sample t test.

Our statistical threshold in both analyses was set at p < 0.05 corrected for multiple comparisons using Gaussian random field (GRF) theory. For our target regions in the LOC and the amygdala, we used reduced search volumes corresponding to the approximate anatomical extension of the areas. Specifically, in LOC, we applied a reduced spherical search volume (10 mm radius) centered on coordinates found in our initial study using the same task (Rose et al., 2005). Similarly, we used an anatomical masking image derived from the automated anatomical labeling template (Tzourio-Mazoyer et al., 2002) as the search volume for the amygdala activation that served as the seed voxel for a PPI. Finally, for statistical inference in our PPI target region in LOC, we placed a 10 mm spherical search volume on our peak activation found in the first analysis. For all other regions, the statistical threshold was set to p < 0.05 corrected for whole-brain volume using GRF theory controlling effectively for an inflation of the type I error rate attributable to multiple comparisons. For display purposes, we show all statistical parametric maps at an uncorrected statistical threshold of p < 0.001 with a cluster extent threshold of at least five contiguous voxels. Plots of mean parameter estimates in Figures 3–6 are based on 33 voxels from a 6 mm sphere centered on the peak voxel in conjunction analysis for LOC and in the main effect of emotion (100% phase coherence only) for the amygdala.

Figure 3.

Figure 3.

Open in a new tab

Two-way interaction terms emotion × phase coherence (red) and load × phase coherence (blue) in LOC. A, B, Lateral and ventral view of the interaction terms rendered on a template brain without a cerebellum. C, Both interaction effects on a transverse slice of the same template brain. Note the extensive overlap of both interaction effects in LOC (purple/magenta), which coincides with the conjunction analysis of both interaction terms. D–G, Mean parameter estimates for 33 voxels from a 6 mm sphere centered on the peak voxel of the conjunction analysis (arrows) in the left and right LOC (error bars indicate SEM). The one-back and two-back conditions exhibit approximately the same difference in phase coherence slopes in both emotion conditions (load × phase coherence interaction). However, the slopes are overall steeper in the negative emotion condition (emotion × phase coherence interaction).

Figure 4.

Figure 4.

Open in a new tab

Psychophysiological interaction analysis investigating the source of emotion modulation in LOC. A–C, The left amygdalo-hippocampal junction (C, parameter estimates of the “load × phase coherence” interaction in A and B) provided the seed region. D, Results of a second-level one-sample t test of the psychophysiological interaction across subjects. For visual reference, the result of the conjunction analysis of “emotion × phase coherence” and “load × phase coherence” (see also Fig. 3) is shown as a magenta outline. Note the close vicinity of the target region of the PPI and the results of the conjunction analysis in LOC.

Results

Behavioral data

We selected the mean number of n-back errors and RTs as a performance measure during the WM task in the scanner. Overall, the mean percentage of errors across all conditions and subjects in the nonrejected blocks was very low (1.5%). However, when we analyzed the data in the context of a repeated-measures ANOVA, we observed a highly significant main effect of load (errors, F(1,22) = 50.10, p < 0.0001; RTs, F(1,22) = 27.93, p < 0.0001): during the two-back condition, subjects committed more errors and were considerably slower than during the one-back condition (Fig. 2A,B). A trend for a main effect of emotion was observed in the RTs (negative RTs < neutral RTs; F(1,22) = 4.01; p < 0.06). No other main effects or interaction terms were significant (all F values <1.52; p > 0.2).

Figure 2.

Figure 2.

Open in a new tab

Behavioral results. A, B, Performance on the WM task in the scanner using mean number of errors (A) and reaction times to n-back targets (B) in all experimental conditions. The closely spaced bars represent the four different phase coherence levels (0, 33, 66, and 100%) from left to right. Note the significant main effect of load (two-back > one-back) in both measures. C, D, Performance (d′) during the recognition task for negative (C) and neutral (D) targets. Note the significant load × phase coherence interaction term in both emotion conditions. Error bars indicate the SEM across subjects.

For the recognition performance, we analyzed the corrected d′ discriminability indices in a repeated-measures ANOVA testing for all experimental main effects and interaction terms. The main effects of load (F(1,22) = 30.33; p < 0.0001) and phase coherence (F(1.56,34.33) = 40.23; p < 0.0001) were further qualified by a significant load × phase coherence interaction term (F(2.24,49.18) = 5.25; p < 0.01) (Fig. 2C,D). This interaction can be best de-scribed by a reduction in phase coherence slope during the two-back condition (linear increase of d′ across the different phase coherence levels). However, no effects of emotion or significant interaction terms involving the factor emotion were observed (all F values <1; p > 0.3).

Imaging data

We began the analysis of the imaging data by identifying the brain areas responding to the main effects of our experimental design common to all subjects. The main effect of load (two-back > one-back) elicited activations in a frontoparietal network comprising the superior and inferior parietal lobule, and superior and middle frontal gyrus often seen in working memory tasks (Owen et al., 2005) (Table 1). Additional activations occurred in the anterior cingulate cortex (ACC) and in the thalamus (Table 1). The main effect of phase coherence (linear increase from 0 to 100% phase coherence) evoked strong activations in the entire ventral visual stream with peaks in LOC and in the fusiform gyrus (Table 1). Because the factor emotion is conveyed through the background pictures, which also were manipulated in their degree of phase coherence, we decided to test for the main effect of emotion (negative > neutral) by only including those pictures of 100% coherence. This test revealed activations in our target areas LOC and amygdala (Table 1). In addition, we found activations in the fusiform gyrus, medial orbitofrontal gyrus, perigenual ACC, and in the colliculi (Table 1).

Table 1.

Peak voxel and corresponding Z values of significantly activated clusters in experimental main and interaction effects and psychophysiological interaction analysis

Experimental effect
 Left hemisphere
 Right hemisphere
    Region x y z Z x y z Z
Main effect load
    IPL −42 −42 42 13.81+ 42 −39 42 11.85+
    SPL −6 −69 51 12.86+ 18 −66 57 12.45+
    MFG −27 3 54 13.81+ 30 3 57 12.43+
    IFG −48 15 27 12.22+ 45 18 39 6.99+
    ACC 6 27 42 9.20+
    Thalamus −12 −3 3 9.77+ 12 −3 3 9.60+
Main effect phase coherence
    LOC −45 81 −3 19.16*** 45 −78 −3 19.66***
−27 −72 −12 16.40*** 42 −72 −15 17.88***
    FG −36 −54 −18 17.08+ 39 −57 −18 17.85+
Main effect emotion (100% phase coherence only)
    LOC −48 −78 −3 5.52*** 51 −72 −3 7.18***
    FG −42 −54 −18 4.07 45 −51 −27 3.37
    Amygdala −27 −9 −12 3.24**
    Med. OFC 3 33 −24 3.18
    Perigenual ACC 15 48 0 3.27
    Colliculi 6 −33 −3 3.29
Interaction emotion × phase coherence
    LOC −42 −78 −3 5.47*** 51 −72 0 6.64***
    TPJ −48 −36 15 3.92 54 −42 12 3.75
    Amygdala −27 −9 −12 3.24**
    Perigenual ACC 12 42 6 4.02
    Colliculi 9 −30 −3 4.00
Interation load × phase coherence
    FG −33 −42 −21 5.02+ 39 −36 −21 4.46
    LOC −45 −66 −9 4.84*** 42 −66 −12 3.84**
−45 −75 −9 4.74*** 48 −75 −6 3.64**
    SOS −24 −75 24 3.96 30 −63 30 3.26
Conjunction of emotion × phase coherence and load × phase coherence
    LOC −45 −75 −9 4.74*** 42 −78 −12 3.61**
−45 −66 −9 4.35*** 42 −66 −12 3.84**
Target voxel of psychophysiological Interaction from left amygdalo-hippocampal junction
    LOC −51 −63 −9 3.37*
Open in a new tab

*p < 0.05 (small-volume corrected)

**p < 0.01 (small-volume corrected)

***p < 0.001 (small-volume corrected)

+p < 0.05 (corrected for whole-brain volume) (see Materials and Methods).

IPL, Inferior parietal lobule; SPL, superior parietal lobule; MFG, middle frontal gyrus; IFG, inferior frontal gyrus; FG, fusiform gyrus; Med. OFC, medial orbitofrontal cortex; TPJ, temporoparietal junction; SOS, superior occipital sulcus.

Closer inspection of the parameter estimates in LOC across all different experimental conditions suggested the presence of an interaction effect between load and phase coherence because the phase coherence slopes differed between the two load conditions (Fig. 3). This was formally tested by the contrast for the load × phase coherence interaction effect, which tests for a steeper phase coherence slope in the one-back compared with the two-back condition independent of the different emotion conditions (parameter plots in Fig. 3) (for a visualization of this interaction as difference scores, see supplemental Fig. 2, available at www.jneurosci.org as supplemental material). We found a strong effect for this interaction in LOC (Fig. 3, Table 1), thus confirming previous results from our laboratory (Rose et al., 2005). The fusiform gyrus and the superior occipital sulcus also exhibited this interaction effect (Table 1).

Furthermore, the parameter estimates in LOC also suggest differences in the phase coherence slopes for the two emotion conditions (Fig. 3). This was confirmed by a formal test for the emotion × phase coherence interaction effect, which tests for steeper phase coherence slopes during the presentation of negative versus neutral background pictures independent of the different load conditions (parameter plots in Fig. 3) (for a visualization of the interaction in difference scores, see supplemental Fig. 2, available at www.jneurosci.org as supplemental material). This contrast yielded significant activations in the same areas in LOC and the amygdala (Table 1). Furthermore, right perigenual ACC, right colliculi, and bilateral temporoparietal junction also showed this effect.

Figure 3A–C focuses on the overlap of the emotion × phase coherence (shown in red) and the load × phase coherence interaction effects (shown in blue) in the LOC. To derive a formal statistic for this overlap, we performed a conjunction analysis (Nichols et al., 2005) of these two interaction effects. The result of this analysis corresponds to the purple/magenta areas in Figure 3A–C (for the peak voxel and its statistical value of this conjunction analysis, see Table 1).

The third two-way interaction of load × emotion did not elicit any significant activations even at an uncorrected threshold of p < 0.001.

Our design allowed us to specifically address the question of whether the modulatory effects of emotion and working memory load interact in their influence on visual processing in LOC. This can be formally tested by the three-way interaction effect of emotion × load × phase coherence. However, at our selected statistical threshold, this contrast did not reveal any significant effects. Only when we relaxed our statistical threshold to p < 0.05 (uncorrected) did a bilateral three-way interaction effect emerge in a region corresponding to visual area V4. Because this effect is far from reaching our selected level of statistical reliability, we refrain from an extensive interpretation of this trend.

In conclusion, object processing in LOC depends on the visual integrity of the stimuli (i.e., the main effect of phase coherence). In addition, the two other modulatory influences we set out to investigate also critically depend on a minimum level of phase coherence, because their influence is only revealed in interaction with the phase coherence factor (i.e., an emotion × phase coherence and a load × phase coherence interaction). Finally, emotion and load do not interact (no direct two-way and no three-way interaction). Rather, the emotional salience of the background stimuli exhibits a multiplicative gain effect on the phase coherence slopes for both load conditions while preserving the slope differences between the two (see plots of parameter estimates in Fig. 3).

What could be the source of this emotional influence on visual processing in LOC? A potential candidate for this modulatory influence is the amygdala, which is known to maintain extensive bidirectional anatomical connections to all visual processing areas in the ventral stream (Amaral et al., 1992). In our paradigm, the left amygdalo-hippocampal junction exhibited a main effect of emotion, which was further qualified by an emotion × phase coherence interaction effect (Table 1, Fig. 4). The right amygdala also exhibited a trend for the same interaction effect but did not survive statistical thresholding with the appropriate search volume. Thus, we restricted this analysis to the left amygdalo-hippocampal junction. To investigate this question, we conducted a PPI analysis (Friston et al., 1997) by multiplying the time series of the left amygdalo-hippocampal junction with the load × phase coherence interaction contrast to construct the PPI term. We then tested which brain areas covaried significantly with this PPI term. Interestingly, we found a target region in the left LOC area immediately adjacent to and partially overlapping with the above mentioned conjunction analysis of emotion × phase coherence and load × phase coherence (Fig. 4, magenta outline). Thus, this PPI analysis lends support to the notion that the amygdalo-hippocampal junction might be the source of an emotional modulation of LOC processing.

Discussion

Using a three-factorial design, we investigated the combined effects of emotional salience and working memory load on visual processing in the lateral occipital complex. Twenty-three female subjects performed a working memory (n-back) task of two levels of difficulty while being simultaneously presented with task-irrelevant background stimuli of different degrees of emotional salience and phase coherence. We observed overlapping but independent interaction effects of emotion × phase coherence and load × phase coherence in ventral visual areas. Furthermore, a psychophysiological interaction analysis revealed that the left amygdalo-hippocampal junction could be the potential source of the emotional modulation of LOC.

The load × phase coherence interaction confirms previous observations (Rose et al., 2005) and is not directly modulated by emotion (no three-way interaction). However, a colocalized emotion × phase coherence interaction suggests that visual processing in LOC is also modulated by emotionally salient stimuli, albeit independent of load. Thus, both load and emotion critically depend on a minimum level of phase coherence. Considering load as a manipulation of attentional resources available for irrelevant stimuli, it is interesting to note that our findings are consistent with a recent EEG study using IAPS stimuli and a spatial attention task that reports enhancement of steady-state visually evoked potentials by attention and emotional salience but no interaction among these factors (Keil et al., 2005). The observed effect of emotion in this study, however, is not a simple additive increase in LOC activation. Rather, the increase in phase coherence slopes suggests that emotion acts as a multiplicative gain factor; whereas the small activation to low coherent pictures remains almost unchanged in both emotion conditions, the large activation to high coherent negative pictures is further exaggerated (Fig. 3). To state it differently, the gain effect of emotion leads to an increase of activation to stimuli that convey a lot of signal and to a decrease of activation when stimuli convey a lot of noise.

In summary, on top of a modulation of visual processing in LOC by the quality of the sensory information (i.e., main effect of phase coherence), there are two additional independent (i.e., non-interacting) influences: (1) WM load decreases processing resources for irrelevant background stimuli, a conjecture consistent with perceptual load theory (Lavie, 1995), and (2) if these background stimuli are coherent and emotionally salient, this leads to an enhanced allocation processing resources, i.e., a multiplicative gain effect of emotion. Thus, emotional salience increases the distinctiveness of the perceptual representation in LOC. This interpretation is consistent with the idea that biologically relevant stimuli are prioritized in the perceptual analysis to facilitate immediate and appropriate behavioral responses (Globisch et al., 1999; LeDoux, 2000).

Our PPI analysis suggests that the source of the emotional modulation might originate in the amygdalo-hippocampal junction. This finding is consistent with a recent fMRI study demonstrating close covariation of amygdala and inferior temporal cortex activation when viewing emotionally evocative stimuli from the IAPS (Sabatinelli et al., 2005). The activation in these areas increased with rated arousal, suggesting it as a mediating factor for the activation in these brain areas. Building on this foundation, the present connectivity analysis demonstrates that the amygdala activation modulates the load × phase coherence effect on the activation in the target area LOC. This observation gains support from previous studies demonstrating effective connectivity between the amygdala and face-sensitive areas in the ventral visual pathway (Morris et al., 1998; Hariri et al., 2000; Pessoa et al., 2002b; Das et al., 2005) and prefrontal regions (Iidaka et al., 2001; Hariri et al., 2003). More direct evidence for a modulatory influence of the amygdala on ventral visual areas is provided by a recent study involving patients with lesions in the amygdala and/or hippocampus and control subjects (Vuilleumier et al., 2004). The authors showed that patients with amygdala damage lack the facilitating influence of this structure on the response of ventral visual areas to emotional facial expressions that they observed in patients with hippocampal lesions and controls.

Unlike the emotional modulation in LOC, we found no effect of emotion in the behavioral recognition data. This is somewhat surprising because highly arousing stimuli (e.g., negative pictures) tend to be better remembered than neutral stimuli (emotional memory) (for review, see LaBar and Cabeza, 2006). Although most studies on emotional memory report an encoding and retrieval advantage for emotional items (for examples of the basic experimental paradigm, see Cahill et al., 1996; Hamann et al., 1999; Canli et al., 2000), these studies always presented their stimuli for several seconds and never used a dual-task paradigm. Although our imaging data showed a clear effect of emotional salience (emotion × phase coherence interaction), we suggest that the attentional demands of the concurrent WM task may have prevented a measurable role of the emotional salience in the recognition behavior.

A possible limitation of the generalizability of these findings lies in the restriction to only female subjects in this study. There is some evidence that males and females do not differ significantly in their ventral stream activation to emotional pictures (Sabatinelli et al., 2004). However, other studies addressing distal influences originating in the amygdala either did not have enough power to make conclusion about gender effects (Morris et al., 1998) or they did not report any (Das et al., 2005). Thus, additional studies focused on gender differences are needed to resolve this issue.

We also explored our data with respect to two controversial papers addressing the relationship between emotion and attention in their influence on activation of ventral visual areas and the amygdala (Vuilleumier et al., 2001; Pessoa et al., 2002b). When we plotted our data in LOC only for the factors emotion and load at 100% phase coherence (thus ignoring the factor phase coherence), we found two non-interacting main effects for emotion and load (supplemental Fig. 1, available at www.jneurosci.org as supplemental material), similar to the results in the fusiform face area as reported by Vuilleumier et al. (2001). Likewise, when we plotted our data for the factors emotion and phase coherence only for the one-back condition (thus ignoring the factor load), we observed an interaction of emotion and phase coherence (supplemental Fig. 1, available at www.jneurosci.org as supplemental material) that is similar but not identical to the interaction of emotion and attention shown by Pessoa et al. (2002b). Although our three-factorial design does not directly resemble the theoretical framework of the two studies, the above mentioned observations raise the possibility that the differences between the studies of Vuilleumier et al. (2001) and Pessoa et al. (2002b) may also involve different processes beyond a mere difference in difficulty of the attentional task (Pessoa et al., 2002b). However, our data can only raise some new ideas on this controversy and leave it to future studies that can explicitly address these issues.

In conclusion, we demonstrated that both emotion and WM load critically depend on the quality of the sensory information to affect visual object processing in LOC. Furthermore, both emotion and WM load do not interact; rather, emotion exerts a general multiplicative gain effect while preserving the effects of WM load (i.e., a difference in phase coherence slopes). This emotional effect potentially originates from the amygdalo-hippocampal junction. Thus, this gain effect suggests an enhanced allocation of processing resources in ventral visual areas for highly visible stimuli that convey biological and behavioral relevance.

Footnotes

This work was supported by the Studienstiftung des Deutschen Volkes and the Deutsche Akademie der Naturforscher Leopoldina (LPD) [Bundesministerium für Bildung und Forschung (BMBF)–LPD Grant 9901/8-140] (J.G.), the Volkswagenstiftung and BMBF (M.R., C.B.), and Deutsche Forschungsgemeinschaft (C.B.). We thank the Physics group and the radiographer team at NeuroImage Nord in Hamburg. In addition, we thank Eszter Schoell for helpful comments on this manuscript.

References

  1. Amaral DG, Price JL, Pitkanen A, Carmichael ST. Anatomical organization of the primate amygdaloid complex. In: Aggleton JP, editor. The amygdala: neurobiological aspects of emotion, memory, and mental dysfunction. New York: Wiley; 1992. pp. 1–66. [Google Scholar]
  2. Anderson AK, Christoff K, Panitz D, De Rosa E, Gabrieli JD. Neural correlates of the automatic processing of threat facial signals. J Neurosci. 2003;23:5627–5633. doi: 10.1523/JNEUROSCI.23-13-05627.2003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Cahill L. Why sex matters for neuroscience. Nat Rev Neurosci. 2006;7:477–484. doi: 10.1038/nrn1909. [DOI] [PubMed] [Google Scholar]
  4. Cahill L, Haier RJ, Fallon J, Alkire MT, Tang C, Keator D, Wu J, McGaugh JL. Amygdala activity at encoding correlated with long-term, free recall of emotional information. Proc Natl Acad Sci USA. 1996;93:8016–8021. doi: 10.1073/pnas.93.15.8016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Canli T, Zhao Z, Brewer J, Gabrieli JD, Cahill L. Event-related activation in the human amygdala associates with later memory for individual emotional experience. J Neurosci. 2000;20:RC99. doi: 10.1523/JNEUROSCI.20-19-j0004.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Das P, Kemp AH, Liddell BJ, Brown KJ, Olivieri G, Peduto A, Gordon E, Williams LM. Pathways for fear perception: modulation of amygdala activity by thalamo-cortical systems. NeuroImage. 2005;26:141–148. doi: 10.1016/j.neuroimage.2005.01.049. [DOI] [PubMed] [Google Scholar]
  7. Friston KJ, Ashburner J, Frith CD, Poline JB, Heather JD, Frackowiak RS. Spatial registration and normalization of images. Hum Brain Mapp. 1995;2:165–189. [Google Scholar]
  8. Friston KJ, Buechel C, Fink GR, Morris J, Rolls E, Dolan RJ. Psychophysiological and modulatory interactions in neuroimaging. NeuroImage. 1997;6:218–229. doi: 10.1006/nimg.1997.0291. [DOI] [PubMed] [Google Scholar]
  9. Globisch J, Hamm AO, Esteves F, Ohman A. Fear appears fast: temporal course of startle reflex potentiation in animal fearful subjects. Psychophysiology. 1999;36:66–75. doi: 10.1017/s0048577299970634. [DOI] [PubMed] [Google Scholar]
  10. Grill-Spector K, Malach R. The human visual cortex. Annu Rev Neurosci. 2004;27:649–677. doi: 10.1146/annurev.neuro.27.070203.144220. [DOI] [PubMed] [Google Scholar]
  11. Hamann S, Canli T. Individual differences in emotion processing. Curr Opin Neurobiol. 2004;14:233–238. doi: 10.1016/j.conb.2004.03.010. [DOI] [PubMed] [Google Scholar]
  12. Hamann SB, Ely TD, Grafton ST, Kilts CD. Amygdala activity related to enhanced memory for pleasant and aversive stimuli. Nat Neurosci. 1999;2:289–293. doi: 10.1038/6404. [DOI] [PubMed] [Google Scholar]
  13. Hariri AR, Bookheimer SY, Mazziotta JC. Modulating emotional responses: effects of a neocortical network on the limbic system. NeuroReport. 2000;11:43–48. doi: 10.1097/00001756-200001170-00009. [DOI] [PubMed] [Google Scholar]
  14. Hariri AR, Mattay VS, Tessitore A, Fera F, Weinberger DR. Neocortical modulation of the amygdala response to fearful stimuli. Biol Psychiatry. 2003;53:494–501. doi: 10.1016/s0006-3223(02)01786-9. [DOI] [PubMed] [Google Scholar]
  15. Iidaka T, Omori M, Murata T, Kosaka H, Yonekura Y, Okada T, Sadato N. Neural interaction of the amygdala with the prefrontal and temporal cortices in the processing of facial expressions as revealed by fMRI. J Cogn Neurosci. 2001;13:1035–1047. doi: 10.1162/089892901753294338. [DOI] [PubMed] [Google Scholar]
  16. Kastner S, Ungerleider LG. Mechanisms of visual attention in the human cortex. Annu Rev Neurosci. 2000;23:315–341. doi: 10.1146/annurev.neuro.23.1.315. [DOI] [PubMed] [Google Scholar]
  17. Keil A, Moratti S, Sabatinelli D, Bradley MM, Lang PJ. Additive effects of emotional content and spatial selective attention on electrocortical facilitation. Cereb Cortex. 2005;15:1187–1197. doi: 10.1093/cercor/bhi001. [DOI] [PubMed] [Google Scholar]
  18. Kourtzi Z, Kanwisher N. Cortical regions involved in perceiving object shape. J Neurosci. 2000;20:3310–3318. doi: 10.1523/JNEUROSCI.20-09-03310.2000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. LaBar KS, Cabeza R. Cognitive neuroscience of emotional memory. Nat Rev Neurosci. 2006;7:54–64. doi: 10.1038/nrn1825. [DOI] [PubMed] [Google Scholar]
  20. Lang PJ, Bradley MM, Cuthbert BN. National Institute of Mental Health Center for the Study of Emotion and Attention. Gainesville, FL: University of Florida; 1999. International affective picture system (IAPS): technical manual and affective ratings. [Google Scholar]
  21. Lavie N. Perceptual load as a necessary condition for selective attention. J Exp Psychol Hum Percept Perform. 1995;21:451–468. doi: 10.1037//0096-1523.21.3.451. [DOI] [PubMed] [Google Scholar]
  22. LeDoux JE. Emotion circuits in the brain. Annu Rev Neurosci. 2000;23:155–184. doi: 10.1146/annurev.neuro.23.1.155. [DOI] [PubMed] [Google Scholar]
  23. Morris JS, Friston KJ, Buchel C, Frith CD, Young AW, Calder AJ, Dolan RJ. A neuromodulatory role for the human amygdala in processing emotional facial expressions. Brain. 1998;121:47–57. doi: 10.1093/brain/121.1.47. [DOI] [PubMed] [Google Scholar]
  24. Nichols T, Brett M, Andersson J, Wager T, Poline JB. Valid conjunction inference with the minimum statistic. NeuroImage. 2005;25:653–660. doi: 10.1016/j.neuroimage.2004.12.005. [DOI] [PubMed] [Google Scholar]
  25. Owen AM, McMillan KM, Laird AR, Bullmore E. N-back working memory paradigm: a meta-analysis of normative functional neuroimaging studies. Hum Brain Mapp. 2005;25:46–59. doi: 10.1002/hbm.20131. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Pessoa L, Kastner S, Ungerleider LG. Attentional control of the processing of neural and emotional stimuli. Brain Res Cogn Brain Res. 2002a;15:31–45. doi: 10.1016/s0926-6410(02)00214-8. [DOI] [PubMed] [Google Scholar]
  27. Pessoa L, McKenna M, Gutierrez E, Ungerleider LG. Neural processing of emotional faces requires attention. Proc Natl Acad Sci USA. 2002b;99:11458–11463. doi: 10.1073/pnas.172403899. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Posner M. Orienting of attention. Q J Exp Psychol. 1980;32:3–25. doi: 10.1080/00335558008248231. [DOI] [PubMed] [Google Scholar]
  29. Rainer G, Augath M, Trinath T, Logothetis NK. Nonmonotonic noise tuning of BOLD fMRI signal to natural images in the visual cortex of the anesthetized monkey. Curr Biol. 2001;11:846–854. doi: 10.1016/s0960-9822(01)00242-1. [DOI] [PubMed] [Google Scholar]
  30. Reinders AA, den Boer JA, Buchel C. The robustness of perception. Eur J Neurosci. 2005;22:524–530. doi: 10.1111/j.1460-9568.2005.04212.x. [DOI] [PubMed] [Google Scholar]
  31. Rose M, Schmid C, Winzen A, Sommer T, Buchel C. The functional and temporal characteristics of top-down modulation in visual selection. Cereb Cortex. 2005;15:1290–1298. doi: 10.1093/cercor/bhi012. [DOI] [PubMed] [Google Scholar]
  32. Sabatinelli D, Flaisch T, Bradley MM, Fitzsimmons JR, Lang PJ. Affective picture perception: gender differences in visual cortex? NeuroReport. 2004;15:1109–1112. doi: 10.1097/00001756-200405190-00005. [DOI] [PubMed] [Google Scholar]
  33. Sabatinelli D, Bradley MM, Fitzsimmons JR, Lang PJ. Parallel amygdala and inferotemporal activation reflect emotional intensity and fear relevance. NeuroImage. 2005;24:1265–1270. doi: 10.1016/j.neuroimage.2004.12.015. [DOI] [PubMed] [Google Scholar]
  34. Snodgrass JG, Corwin J. Pragmatics of measuring recognition memory: applications to dementia and amnesia. J Exp Psychol Gen. 1988;117:34–50. doi: 10.1037//0096-3445.117.1.34. [DOI] [PubMed] [Google Scholar]
  35. Tzourio-Mazoyer N, Landeau B, Papathanassiou D, Crivello F, Etard O, Delcroix N, Mazoyer B, Joliot M. Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain. NeuroImage. 2002;15:273–289. doi: 10.1006/nimg.2001.0978. [DOI] [PubMed] [Google Scholar]
  36. Vuilleumier P. How brains beware: neural mechanisms of emotional attention. Trends Cogn Sci. 2005;9:585–594. doi: 10.1016/j.tics.2005.10.011. [DOI] [PubMed] [Google Scholar]
  37. Vuilleumier P, Armony JL, Driver J, Dolan RJ. Effects of attention and emotion on face processing in the human brain: an event-related fMRI study. Neuron. 2001;30:829–841. doi: 10.1016/s0896-6273(01)00328-2. [DOI] [PubMed] [Google Scholar]
  38. Vuilleumier P, Richardson MP, Armony JL, Driver J, Dolan RJ. Distant influences of amygdala lesion on visual cortical activation during emotional face processing. Nat Neurosci. 2004;7:1271–1278. doi: 10.1038/nn1341. [DOI] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES