Working Memory for Social Cues Recruits Orbitofrontal Cortex and Amygdala: A Functional Magnetic Resonance Imaging Study of Delayed Matching to Sample for Emotional Expressions

Abstract

During everyday interactions, we continuously monitor and maintain information about different individuals and their changing emotions in memory. Yet to date, working memory (WM) studies have primarily focused on mechanisms for maintaining face identity, but not emotional expression, and studies investigating the neural basis of emotion have focused on transient activity, not delay related activity. The goal of this functional magnetic resonance imaging study was to investigate WM for two critical social cues: identity and emotion. Subjects performed a delayed match-to-sample task that required them to match either the emotional expression or the identity of a face after a 10 s delay. Neuroanatomically, our predictions focused on the orbitofrontal cortex (OFC) and the amygdala, as these regions have previously been implicated in emotional processing and long-term memory, and studies have demonstrated sustained OFC and medial temporal lobe activity during visual WM. Consistent with previous studies, transient activity during the sample period representing emotion and identity was found in the superior temporal sulcus and inferior occipital cortex, respectively. Sustained delay-period activity was evident in OFC, amygdala, and hippocampus, for both emotion and identity trials. These results suggest that, although initial processing of emotion and identity is accomplished in anatomically segregated temporal and occipital regions, sustained delay related memory for these two critical features is held by the OFC, amygdala and hippocampus. These regions share rich connections, and have been shown previously to be necessary for binding features together in long-term memory. Our results suggest a role for these regions in active maintenance as well.

Introduction

Everyday, we process and maintain information about the identity and changes in emotional expression of those with whom we interact. This study focuses on how this continuously changing social information is maintained in working memory (WM). Early WM studies explored how objects and locations, initially processed separately by dorsal and ventral visual streams, were maintained in WM. It was initially postulated that object “what” and location “where” data were maintained separately in prefrontal cortex (PFC) (Wilson et al., 1993), but a seminal study challenged this idea of anatomical segregation, suggesting instead that individual PFC neurons can hold both spatial and object information across a delay (Rao et al., 1997). In the present study, instead of examining two nonsocial cues (e.g., what and where), we used a delayed match to sample task to examine two salient social cues: Emotion and Identity. We contrasted two competing predictions. The first, based on evidence that emotion and identity are processed by different anatomical regions (Hasselmo et al., 1989), was that delay related activity would be anatomically segregated for emotion and identity. The second prediction, based on the demonstration that PFC neurons bind together information in WM, was that both emotion and identity are maintained by the same neuroanatomical system. A summary of neuroanatomical regions that we predicted would be critical for social WM follows.

Studies exploring face perception have demonstrated that attending to emotional expression activates superior temporal cortex, whereas attending to face identity activates inferior temporal and fusiform cortex (Hasselmo et al., 1989; Narumoto et al., 2001; Vuilleumier et al., 2001; Andrews and Ewbank, 2004; Winston et al., 2004). Both areas have strong anatomical connections with the orbitofrontal cortex (OFC) (Barbas, 1988; Seltzer and Pandya, 1989; Morecraft et al., 1992). A strong functional connection also exists between the amygdala and fusiform gyrus during face processing (Fairhall and Ishai, 2007).

The OFC is implicated in processing emotional information (Iidaka et al., 2001; Beer et al., 2006; Barbas, 2007; Eippert et al., 2007; Wright et al., 2008), and may maintain visual information in WM when task demands are high (Schon et al., 2008). Consistent with its role in social reinforcement learning (Kringelbach and Rolls, 2003; Rolls, 2004), the OFC contains face-selective neurons (Thorpe et al., 1983; Rolls et al., 2006) and damage to the OFC can impair face identification (Hornak et al., 2003).

Facial expressions consistently activate the amygdala (Schneider et al., 1997; Morris et al., 1998), and the amygdala enhances long-term memory for emotional stimuli (Hamann et al., 1999; Kilpatrick and Cahill, 2003; Dolcos et al., 2004; Kensinger and Schacter, 2006). Face-selective neurons exist in the amygdala (Leonard et al., 1985), and patients with amygdala damage have impaired recognition of expressions (Adolphs et al., 2002; Adolphs and Tranel, 2003).

Strong anatomical connections exist between amygdala, OFC, and additional medial temporal lobe (MTL) structures (Saunders et al., 1988; Barbas, 2000; Ghashghaei and Barbas, 2002; Barbas and Zikopoulos, 2006; Ishikawa and Nakamura, 2006; Furtak et al., 2007), suggesting that these regions play a critical role in WM for social cues.

Materials and Methods

Participants

Nineteen healthy individuals (six male; mean age, 23 ± 3) from the Boston community, with no history of neurological or psychiatric illness, were recruited for this study. Vision was either normal or corrected to normal. All subjects were screened to ensure magnetic resonance imaging (MRI) environment compatibility. Eligible individuals who agreed to participate gave signed informed consent in accordance with the Partners Human Research Committee and the Institutional Review Board of Boston University.

Procedure

Stimuli.

Stimuli for the task were selected from the University of Pennsylvania database of facial expressions (Gur et al., 2002). We selected three moderate positive, negative, and neutral valenced expressions from photographs of 24 individual actors of varying age, sex, and race, for a total set of 72 unique stimuli. All stimuli were cropped to 350 × 467 pixels at 28.35 pixels/cm resolution (12.35 × 16.47 cm), put on a gray background, and converted to gray-scale. An oval mask was used to isolate the central facial features (see Fig. 1). All peripheral features (e.g., hair, clothes, eye color) were removed. Control stimuli were Fourier phase scrambled versions of the original face stimuli.

Figure 1.

DMS and CON task. In all tasks, a trial consisted of three time-locked components: a 2 s face presentation (sample), followed by a 10 s delay period, and by a 2 s face presentation (test). Trials were separated by a variable-length ITI. During EMO_DMS trials, subjects matched the emotional expression of the face, and facial identity was irrelevant. During ID_DMS trails, subjects matched the identity of the face and the emotional expression was irrelevant. In the case of an EMO-match trial (top), the test face showed the same emotional expression as the one shown by the sample face, but was from a different actor. In the case of an ID-match trial (middle), the test face was from the same actor as that seen during the sample phase, but showed a different emotional expression. During CON trials (bottom), Fourier phase scrambled versions of the face stimuli were presented and subjects were instructed to respond to the number presented in the center of the test stimulus.

Delayed match-to-sample task.

During MRI scanning, subjects performed a delayed match-to-sample task (DMS) consisting of eight runs. The DMS task required subjects to make match or nonmatch judgments by comparing a test stimulus to a sample stimulus separated by a short delay period (see Fig. 1). Subjects performed two types of the DMS task that differed only in instruction. The first type required the subject to match the emotional expression (EMO) of the face, and the second type required the subject to match the identity (ID) of the face. For the EMO_DMS task, the test face was always a different identity than the sample face, and for the ID_DMS task, the test face was always a different emotion than the sample face (Fig. 1). This prevented the subject from seeing an identical stimulus for the sample and test phases of a trial. Emotional expressions and identities were counterbalanced throughout the 8 functional MRI (fMRI) runs, and the two tasks were equated in terms of the emotional and identity information that was presented. The emotional expressions were characterized as positive, negative, and neutral for counterbalancing and analysis purposes only, however the subjects were not provided with these verbal labels. Behaviorally, subjects were simply told to match the emotion or the identity, and were not informed of the number of individuals or types of emotions presented. Labeling the emotion was not a factor in selecting a response. Each trial consisted of 2 s presentations of a sample face and a test face with an intervening 10 s delay period. The length of the delay period was kept fixed because previous work has shown that different delay lengths can result in different neural response patterns (Elliott and Dolan, 1999). A variable-length intertrial interval (ITI) (6–26 s, counterbalanced using Optseq2) (Dale, 1999) separated all trials. The variable ITI provided temporal “jitter” for selective averaging across trials (Postle et al., 2000a; Schon et al., 2004).

Visuomotor control task.

One-half of all trials consisted of a visuomotor control task (CON). During these trials, subjects were presented with scrambled versions of the face stimuli and responded to a “1” or “2” presented in the center of the test stimulus. The CON task acted as an effective baseline because it was designed to account for activity that was generalized to low-level visual and motor responses. These trials required no active maintenance, but maintained the subjects' attention to the stimuli by requiring an arbitrary response.

Functional runs either consisted of EMO trials or ID trials to minimize brain activation because of task switching. Before each functional scan, the subjects received verbal instruction that the upcoming run was an EMO or ID task. In addition, the color of the central fixation cross present during delays and ITIs reminded the subject of the task type. Subjects performed 12 trials in each of the eight functional runs (six DMS and six CON trials per run; four EMO and four ID runs, randomly ordered), for a total of 96 trials (24 EMO_DMS, 24 ID_DMS, 48 CON).

Before scanning, subjects viewed task instructions and performed a practice version of the task. Responses and reaction times (RTs) were recorded from an MRI compatible button box. We created four different randomized versions of the task so that no >5 subjects performed identical versions. Tasks were designed and presented and behavioral data were recorded using Psyscope 1.2.5 (Cohen et al., 1993).

fMRI data acquisition

Data were acquired on a 3.0 Tesla Siemens (Erlangen, Germany) Trio scanner at the Athinoula A. Martinos Center for Biomedical Imaging (Massachusetts General Hospital, Charlestown, MA). Two high-resolution T1-weighted anatomical images were acquired for each subject. Cognitive tasks were performed during 8 functional T2*-weighted gradient-echo, echo-planar blood oxygen level dependent (BOLD) scans [matrix, 64 × 64; field of view, 200 mm; repetition time (TR), 2 s; echo time, 30 ms; flip angle, 90°; 26 interleaved axial-oblique slices; voxel size, 3.125 × 3.125 × 5 mm; no interslice gap]. Slices were aligned parallel to the line connecting the anterior and posterior commisures and 144 images per slice were acquired during each scan.

fMRI data analysis

Preprocessing.

fMRI data were preprocessed using the SPM2 software package (Statistical Parametric Mapping, Wellcome Department of Cognitive Neurology, London, UK). BOLD images were corrected for differences in slice timing and realigned to the first image within a series. Realignment and unwarping was performed to correct for image distortions caused by susceptibility-by-movement interactions. The images were then normalized into MNI (Montreal Neurological Institute) stereotactic space, resampled to 2 × 2 × 2 mm isotropic voxels, and spatially smoothed using a 4 mm full-width half-maximum Gaussian filter.

fMRI statistical analysis.

Analysis of BOLD activity during the delayed match-to-sample task was assessed with multiple regression with near orthogonal regressors using the SPM2 software package. This approach has been successfully used to assess delay period activity in fMRI studies with delayed-match-to-sample tasks (Courtney et al., 1997, 1998; Postle and D'Esposito, 1999; Schon et al., 2004; Schon et al., 2005). As originally proposed by Courtney et al. (1997), and similar to Postle et al. (2000) and previous work in our laboratory (Schon et al., 2004; Schon et al., 2005), we created six regressors that reflected the comparisons of interest and convolved them with a Gamma hemodynamic response function (HRF) (Boynton et al., 1996) in Matlab 6.5 (MathWorks, Natick, MA). The regressors consisted of the following comparisons: regressor 1 assessed activation caused by nonspecific visual stimulation, regressor 2 assessed the difference between DMS stimuli and CON stimuli, regressor 3 assessed the difference between the sample stimulus and the test stimulus for CON trials, regressor 4 assessed the difference between the sample stimulus and the test stimulus for DMS trials, regressor 5 assessed differences in activity during task delays (across tasks) versus during the ITI, and regressor 6 assessed the difference between activity during the DMS delay versus activity during the CON delay (see Fig. 2).

Figure 2.

Multiple regression analysis. A, Stick functions were used to create six contrasts for comparison of DMS and CON task components. S, Sample; D, delay; T, test. Regressors 2 and 6 were used to assess activity related to face processing and active maintenance, respectively. Bars indicate the positioning of the HRF. Stick functions for delay periods were divided into five bars spread across the five TRs of the delay to account for the sustained time course of the delay period activity (Schluppeck et al., 2006). B, Example of regressors created by convolving the six stick contrasts with a gamma-variate function (HRF) (Boynton et al., 1996). The bottom graph shows all six regressors together.

This procedure generated a statistical parametric T-map (SPM{T}) for each regressor and for each subject. Group analyses were performed on the regressors of interest (regressors 2 and 6) by entering the SPM{T} maps from each subject into second-level random-effects one-sample t tests, treating subjects as a random factor. For regressor 2, the significance level to identify suprathreshold voxels was p ≤ 0.001 with false discovery rate (FDR) correction (Genovese et al., 2002) across the whole brain. For regressor 6, the significance level was p ≤ 0.01, with FDR correction across the whole brain. The statistical threshold was more stringent for regressor 2 than for regressor 6 because more robust activity was expected during face presentations than during the intervening delay period. In both cases, the threshold extent was five contiguous voxels.

Our regressors were designed to isolate brain regions that contribute specifically to the processing of the sample and test stimuli (see Fig. 2, regressor 2) and the maintenance of the sample stimulus during the delay period (see Fig. 2, regressor 6) by contrasting DMS trials to CON trials. These regressors were constructed to account for the slow onset of the BOLD signal (HRF) relative to stimulus presentation. In addition, regressor 6 was separated into five ⅕ size stick functions spread across the five TRs of the delay period to account for the sustained time course and expected weaker signal during this phase of the task (Schluppeck et al., 2006) (see Fig. 2). We compensated for this expectation by using a more lenient significance threshold (described above) for analysis of delay period activity. This regression model allowed us to differentiate neural activity that was sustained across the delay component of the task from transient activity that occurred only during presentation of the face stimuli. However, because of residual collinearity between regressors 2 and 6 caused by overlap of the HRFs, the distinct contribution of a ROI to the stimulus and/or delay components of the task can be determined most accurately by analyzing the time course of activation within that region. Therefore, regions that were initially activated by the sample face presentation and sustained activity across the delay were revealed by both regressors. Further exploration of the contribution of brain regions to the DMS task was accomplished by analysis of the time course of activation within a given region.

In addition, a subsequent analysis was also performed using a 16-regressor model that allowed for all possible comparisons of task conditions and trial components. This analysis searched for regions showing differential EMO and ID task effects that may not have been captured by the six-regressor model. Data from this analysis are provided as supplemental text (available at www.jneurosci.org as supplemental material).

ROI analysis.

The regions of interest (ROIs) chosen for our time-series analysis were the orbitofrontal cortex, the amygdala, the hippocampus, parahippocampal cortex, inferior occipital/fusiform cortex, and superior temporal cortex. ROI peak activation coordinates were derived from second-level one-sample t tests on regressors 2 and 6. In all cases, time-series analysis was conducted on suprathreshold voxels within a 5 mm sphere surrounding the peak coordinate within each ROI (Table 1).

Table 1.

Significant areas of activation from the active maintenance analysis (regressor 6) and the face processing analysis (regressor 2; threshold at p ≤ 0.001_FDR)

Brain areas	t value	p value (FDR)	MNI peak coordinates (mm)
			x	y	z
Active maintenance analysis (regressor 6, DMS delay vs CON delay)
L Orbitofrontal cortex	7.84	<0.001	−46	26	−6
L Middle frontal gyrus	7.70	<0.001	−40	56	14
L Inferior frontal gyrus (triangularis)	6.42	<0.001	−44	30	8
L Insula	6.90	<0.001	−26	32	8
R Precentral gyrus	5.22	<0.001	54	6	46
L Precentral gyrus	4.89	0.001	−40	2	32
Supplementary motor area	7.72	<0.001	0	20	48
L Parahippocampal gyrus	4.49	0.001	−18	−16	−20
L Amygdala	3.63	0.005	−18	−4	−20
L Hippocampal body	5.23	0.001	−32	−30	−8
R Parahippocampal/hippocampus	6.86	<0.001	34	−46	−4
L Inferior parietal cortex	6.14	<0.001	−50	−50	52
L Cerebellum	6.33	<0.001	−42	−54	−28
Cerebellum	5.96	<0.001	−2	−82	−30
Face processing analysis (regressor 2, DMS stimuli vs CON stimuli)
R Inferior occipital cortex	9.87	<0.0001	50	−74	−14
R Inferior temporal gyrus	8.86	<0.0001	58	−62	−10
R Middle occipital gyrus	7.71	<0.0001	28	−82	18
R Fusiform gyrus	6.76	<0.0001	24	−80	−14
R Superior temporal sulcus	6.50	<0.0001	50	−36	0
L Inferior occipital cortex	11.48	<0.0001	−40	−74	−14
L Fusiform gyrus	6.13	<0.0001	−40	−60	−18
L Superior temporal sulcus	6.67	<0.0001	−52	−46	4
R Orbitofrontal cortex	10.87	<0.0001	40	30	−8
R Inferior frontal gyrus (triangularis)	11.63	<0.0001	50	38	6
R Insula	14.64	<0.0001	34	28	2
L Orbitofrontal cortex	7.72	<0.0001	−50	22	−6
L Inferior frontal gyrus (operculum)	11.72	<0.0001	−46	16	22
L Insula	12.91	<0.0001	−32	26	0
Supplementary motor area	14.23	<0.0001	2	30	44
R Cingulate gyrus	12.32	<0.0001	10	20	36
R Anterior cingulate gyrus	8.75	<0.0001	6	38	18
L Parahippocampal cortex	8.63	<0.0001	−16	−36	−6
R Parahippocampal cortex	8.16	<0.0001	16	−36	−6
L Amygdala	6.54	<0.0001	−18	−4	−16
R Amygdala	7.51	<0.0001	20	2	−20
L Inferior parietal sulcus	7.75	<0.0001	−36	−52	42
R Inferior parietal sulcus	7.78	<0.0001	40	−52	48
Cerebellum	7.22	<0.0001	0	−52	−44
L Cerebellum	7.15	<0.0001	−6	−80	−38

Peak coordinates in MNI space are listed using the Talairach coordinate system. t values are reported for magnitude of activation. Regional labels were derived using the WFU_pickatlas toolbox (Maldjian et al., 2003) in SPM2 and the AAL atlas provided with MRIcro (Rorden and Brett, 2000). L, Left; R, right.

Time-series analysis.

The VOI (voxel-of-interest) tool in SPM2 was used to extract signal intensity time courses for each subject from suprathreshold voxels within each ROI. Signal intensities (Y) were adjusted for effects of interest and were extracted for each time point (144 per BOLD scan) and selectively averaged across subjects. Group level time course line charts were generated by plotting Y values across the 13 TRs (26 s) of each trial for the DMS and CON tasks (see Figs. 3–5). Because the grand mean of the data were scaled to 100, the signal intensity values represent percentage signal change with respect to the global mean intensity of the scaled images. The tasks were designed such that the CON task serves as the baseline. Thus, significant positive activations are evident for events where DMS > CON. To contrast activity from DMS and CON trials, signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference (see Fig. 3–5). Signal difference values were selectively averaged by task type (EMO and ID) and emotion condition (positive, negative, and neutral). To differentiate activity related to the individual components of the task, time points were averaged together to compute separate signal amplitudes that represent the sample (4th and 5th TRs), delay (6th–9th TRs), and test (10th and 11th TRs) phases of the DMS trial (see Figs. 3–5). TR bins were selected to capture the peak of the HRF for the sample and test face responses and to account for an initial temporal lag of 6 s (Buckner, 1998). These values were statistically analyzed with t tests and a three-factor repeated-measures ANOVA with the within-subject factors task type (EMO vs ID), emotion condition (positive vs negative vs neutral), and task component (sample vs delay vs test). All statistics were corrected for unequal variance among factor levels whenever necessary.

Figure 3.

Sustained delay period activity in orbitofrontal cortex. A, fMRI results from the active maintenance analysis (regressor 6) demonstrating sustained activity in left OFC, x = −46, y = 26, z = −6 (arrows). Statistical parametric maps are shown superimposed onto the canonical average T1-weighted MNI brain. B, Sustained activity is displayed in the line graph representing the signal intensity time course extracted from suprathreshold voxels within a 5 mm sphere surrounding the ROIs peak activation (green, DMS trials; gray, CON trials). Blue and red horizontal bars indicate time points when sample and test faces were presented and the intervening delay period, respectively. Shaded areas indicate time points that were averaged to obtain signal amplitudes for sample, delay, and test components of the task. C, Transient dissociation between EMO and ID tasks in OFC. Greater activity was found for EMO than ID during sample face presentations. In contrast, significant delay period activity was similar for EMO and ID tasks. The bar graph represents signal difference values (DMS − CON) for sample, delay, and test. Signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference value for each task component. D, Greater OFC activity at test for EMO_DMS trials with negative valence. During test phase face presentations within EMO trials, negatively valenced faces elicited a significantly greater signal than positively valenced faces. L, Left; R, right. Error bars indicate SEM. Asterisks indicate significant difference.

Figure 4.

Sustained delay period activity in medial temporal lobe structures. A, Left amygdala, x = −18, y = −4, z = −20 (arrows). B, Left mid-hippocampal body, x = −32, y = −30, z = −8 (arrows). Statistical parametric maps are shown superimposed onto the canonical average T1-weighted MNI brain. Sustained activity is displayed in the line graphs representing signal intensity time courses extracted from suprathreshold voxels within a 5 mm sphere surrounding the ROIs peak activation (green, DMS trials; gray, CON trials). Blue and red horizontal bars indicate time points when sample and test faces were presented and the intervening delay period, respectively. Shaded areas indicate time points that were averaged to obtain signal amplitudes for sample, delay, and test components of the task. Bar graphs represent signal difference values (DMS − CON) for sample, delay, and test. Signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference value for each task component. Significant delay period activity was similar for EMO and ID tasks in amygdala and hippocampus ROIs. L, Left; R, right. Error bars indicate SEM.

Figure 5.

fMRI results from face processing analysis (regressor 2) demonstrating transient activity that is modulated by task. A, Right inferior occipital cortex, x = 50, y = −74, z = −14 (arrows). B, Right superior temporal sulcus, x = 50, y = −36, z = 0 (arrows). C, Right posterior parahippocampal cortex, x = 16, y = −36, z = −6 (arrows). Statistical parametric maps are shown superimposed onto the canonical average T1-weighted MNI brain. Line graphs represent signal intensity time courses extracted from suprathreshold voxels within a 5 mm sphere surrounding the ROIs peak activation (green, DMS trials; gray, CON trials). Blue and red horizontal bars indicate time points when sample and test faces were presented and the intervening delay period, respectively. Shaded areas indicate time points that were averaged to obtain signal amplitudes for sample, delay, and test components of the task. Bar graphs represent signal difference values (DMS − CON) for sample, delay, and test. Signal intensity values for CON trials were subtracted from corresponding values for DMS trials to compute a signal difference value for each task component. A, Greater activity for ID than EMO during sample face presentations in the right inferior occipital cortex. B, Greater activity for EMO than ID during sample face presentations in the right STS. C, Greater activity for EMO than ID during test face presentations in right parahippocampal cortex. L, Left; R, right. Error bars indicate SEM. Asterisks indicate significant difference.

Behavioral data analysis

Match/nonmatch responses and reaction times were recorded for each trial to ensure the subjects were performing the task as instructed. Accuracy and average RTs were analyzed with t tests.

Results

Behavioral Results

Subjects performed DMS tasks at close to 90% accuracy and showed no significant difference in accuracy between the EMO task (mean ± SE: 85.9 ± 1.7%) and ID task (89.6 ± 2.8%). Reaction times for the two tasks (EMO: 1210 ± 35 ms; ID: 1189 ± 34 ms) also did not differ significantly. Identity trials were excluded for two subjects because of poor task performance. The CON task was performed at close to 100% accuracy (mean ± SE: 99.4 ± 0.6%) and with significantly faster reaction times (mean ± SE: 822 ± 15 ms) than both DMS tasks (EMO vs CON, t₍₂₆₎ = 10.328, p < 0.001; ID vs CON, t₍₂₁₎ = 10.000, p < 0.001).

fMRI results

Sustained delay related activity in the orbitofrontal cortex, amygdala, and hippocampus

Active maintenance of information in working memory was assessed by contrasting DMS delay period activity with CON delay period activity (Fig. 2, regressor 6). This analysis revealed significant delay period activity in left OFC (peak at x = −46, y = 26, z = −6; T = 7.84, Z = 5.92, p < 0.01_FDR) (Fig. 3). This cluster of activation extended dorsally and rostrally into inferior frontal gyrus (triangularis) and laterally into the insula. A repeated measures ANOVA on the signal difference (DMS − CON) time-series in this ROI (Fig. 3C) revealed a significant main effect of task (EMO vs ID) (F_(1,16) = 9.76, p = 0.007) and a significant main effect of component (sample vs delay vs test) (F_(2,32) = 4.20, p = 0.032). t tests revealed significantly greater activation for EMO trials than ID trials (t₍₉₉₎ = 2.262, p = 0.026) during the sample phase in this ROI. T tests on the individual emotion conditions (positive vs negative vs neutral) in the OFC revealed significantly greater activation for negative faces than positive faces (t₍₃₆₎ = 2.613, p = 0.013) (Fig. 3D) during the test phase for EMO trials.

Significant delay period activity was also found in MTL structures including left amygdala (peak at x = −18, y = −4, z = −20; T = 3.63, Z = 3.32, p < 0.01_FDR) (Fig. 4A) and left mid-hippocampal body (peak at x = −32, y = −30, z = −8; T = 5.23, Z = 4.46, p < 0.01_FDR) (Fig. 4B). A repeated-measures ANOVA on the signal difference (DMS − CON) time-series in the left amygdala ROI revealed a significant main effect of component (sample vs delay vs test) (F_(2,32) = 4.92, p = 0.016). A repeated-measures ANOVA and t tests did not reveal any significant differences between any of the conditions in the left hippocampal ROI. In both ROIs, activation appeared to be sustained early in the delay period (Fig. 4A,B, line graphs). Significant activation was also present in left insula, left inferior frontal gyrus (triangularis), the supplementary motor area (SMA), anterior parahippocampal gyrus, right posterior MTL (including the hippocampus and parahippocampal cortex), and left inferior parietal cortex.

In summary, robust activity in the OFC was sustained across the delay period. In OFC, during the sample phase, responses were larger for EMO trials than ID trials, and during the test phase of EMO trials, Negative faces elicited a greater response than Positive faces. Sustained activity was also found in the hippocampus and amygdala early in the delay period, but activity in these MTL areas was not modulated by task. See Table 1 for a complete list of areas of activation revealed by regressor 6.

Transient activity in temporal and occipital cortices and posterior parahippocampal cortex

Transient responses to the presentation of face stimuli were assessed by contrasting DMS sample and test phase activity with CON sample and test phase activity (Fig. 2, regressor 2). This analysis revealed significant transient responses to the sample and test face presentations in right inferior occipital cortex (peak at x = 50, y = −74, z = −14; T = 9.87, Z = 6.78, p < 0.001_FDR) (Fig. 5A). The activation extended anteriorly into inferior temporal gyrus and medially into the fusiform gyrus. A repeated measures ANOVA on the signal difference (DMS − CON) time-series in this ROI revealed a significant main effect of component (sample vs delay vs test) (F_(2,32) = 47.72, p < 0.001) and a significant task (EMO vs ID) by component interaction (F_(2,32) = 3.82, p = 0.042). t tests revealed significantly greater activation for ID trials than EMO trials (t₍₁₀₅₎ = 2.300, p = 0.023) (Fig. 5A) during the sample phase in this ROI.

This analysis also revealed significant transient responses during sample and test periods in right superior temporal sulcus (STS, peak at x = 50, y = −36, z = 0; T = 6.50, Z = 5.23, p < 0.001_FDR) (Fig. 5B). This activation extended along the rostral/caudal axis of the STS. A repeated measures ANOVA on the signal difference (DMS − CON) time-series in this ROI revealed a significant emotion (positive vs negative vs neutral) by component (sample vs delay vs test) interaction (F_(4,64) = 4.59, p = 0.009). t tests revealed significantly greater activation for EMO trials than ID trials (t₍₉₈₎ = 1.987, p = 0.050) (Fig. 5B) during the sample phase in this ROI.

Transient responses were also found in right posterior parahippocampal cortex (peak at x = 16, y = −36, z = −6; T = 8.16, Z = 6.07, p < 0.001_FDR) (Fig. 5C). A repeated measures ANOVA on the signal difference (DMS − CON) time-series did not reveal any significant effects; however, t tests did show significantly greater activation for EMO trials than ID trials (t₍₁₀₅₎ = 2.177, p = 0.032) (Fig. 5C) during the test phase in this ROI. Significant activation was also present in left inferior occipital cortex, left fusiform gyrus, left STS, bilaterally in the insula, bilaterally in orbitofrontal and inferior frontal cortex, left posterior parahippocampal cortex, bilaterally in the anterior MTL (including the amygdala), and in the SMA spanning ventrally along the medial wall into anterior cingulate cortex.

In summary, transient activity related to the sample and test phases of each trial that returned to baseline during the delay period was found in temporal and occipital cortices, including the right inferior occipital cortex and right STS, and in right posterior parahippocampal cortex. The most robust activity was found in the inferior occipital cortex. During the sample phase, responses in inferior occipital cortex were larger for ID trials than EMO trials, whereas responses in STS were larger for EMO trials than ID trials. During the test phase, responses in posterior parahippocampal cortex were larger for EMO trials than ID trials. See Table 1 for a complete list of areas of activation revealed by regressor 2.

Results from the 16-regressor model comparing EMO and ID tasks can be found in the supplemental text (available at www.jneurosci.org as supplemental material).

Discussion

Our goal was to explore WM for two critical social cues, emotion (EMO) and identity (ID). We examined two competing predictions. The first, based on the fact that emotion and identity are processed perceptually by separate systems, was that sustained delay related activity in EMO and ID trials would be anatomically segregated and actively maintained within ventral visual areas. The second, based on the demonstration that PFC neurons can actively maintain disparate stimulus types, such as what and where, was that emotion and identity would be maintained by a single neuroanatomical system. Our results support this second prediction. Neuroanatomically, our predictions and analysis focused on (1) a perceptual system within temporal and occipital cortices (TOC) that is known to separately process identity and emotion and (2) a neuroanatomically interconnected memory system comprised of the OFC, amygdala, and hippocampus, that has been implicated in long-term emotional memory and more recently in visual WM paradigms.

A discussion of these two neuroanatomical systems, one critical for active maintenance and the other critical for transient perceptual dissociation, follows.

Active maintenance of social cues

Our results demonstrate sustained delay related activity within the OFC, the amygdala, and hippocampus for both emotion and identity (Figs. 3, 4, Table 1). Neuroanatomically, strong connections exist between these three structures (Saunders et al., 1988; Barbas, 2000; Ghashghaei and Barbas, 2002; Barbas and Zikopoulos, 2006; Ishikawa and Nakamura, 2006; Furtak et al., 2007). Additional connections from TOC to OFC provide a critical link between the regions we found that were transiently active, and those that demonstrated sustained activity (Barbas, 2007).

Our results extend previous studies of WM demonstrating contributions of prefrontal structures, including the OFC, to the active maintenance of faces (Courtney et al., 1996; Haxby et al., 2000; Sala et al., 2003), and differ from several studies demonstrating active maintenance in TOC during face working memory tasks (Druzgal and D'Esposito, 2003; Postle et al., 2003; Xu and Chun, 2006; Yoon et al., 2006). However, it should be noted that fMRI studies examining WM maintenance in TOC do not consistently find delay related activity, and our results may help explain these inconsistencies. Critically, the present study differs from these earlier studies in that two features, identity and emotion, were varied across trials, rather than simply varying a single stimulus feature, identity. Therefore, it may be that the TOC is sufficient for the active maintenance of simple stimuli, including objects and faces (Druzgal and D'Esposito, 2003; Postle et al., 2003; Xu and Chun, 2006; Yoon et al., 2006), but when multiple stimulus dimensions such as emotion and identity are held across a delay, these TOC regions may no longer be sufficient. Therefore, although one might have predicted sustained activity in the superior temporal sulcus for maintaining emotion and sustained activity in inferior occipital/fusiform cortex for maintaining identity, our data do not support this. Instead, our data suggest that these separate streams of information are relayed from the TOC to the OFC and to the amygdala (Barbas, 2007). These structures, together with the hippocampus, act to bind the two features together and hold this unified information in WM. A similar system for maintaining conjunctions of object and spatial information in WM has been proposed in the lateral PFC in the monkey (Rao et al., 1997; Rainer et al., 1998) and human (Owen et al., 1998; Postle et al., 2000b).

Our data are consistent with the role of the OFC in social reinforcement learning (Kringelbach and Rolls, 2003; Rolls, 2004, 2007), which, in turn, is consistent with its prominent role in rapid stimulus-reinforcement learning (Rolls, 2000, 2004; O'Doherty, 2004; Rolls et al., 2006; Dolan, 2007). A recent review suggests the OFC may be critical for promoting flexible behaviors and guiding response selection, and in addition, is critical for reflecting changes in reinforcement values, and in modifying behavioral responses when predicted outcomes (such as reinforcement) change (for review, see Murray and Izquierdo, 2007). In our task, the subjects need to flexibly switch between selecting their responses based on emotion and identity. This might be accomplished by inhibiting one stimulus dimension and maintaining the other. Or, as we suggest, it might be accomplished by binding together and maintaining both stimulus dimensions, emotion and identity, across the delay, and then flexibly selecting the correct response.

We predicted that our task would engage the amygdala. Face responsive neurons have been identified in the amygdala (Leonard et al., 1985), and imaging studies have demonstrated amygdala activation in response to expressions (Schneider et al., 1997; Morris et al., 1998; Blair et al., 1999; Iidaka et al., 2001; Killgore and Yurgelun-Todd, 2004). Memory studies have implicated the amygdala in the enhancement of long-term encoding of emotionally charged stimuli (Cahill et al., 1996; Canli et al., 2000; Kensinger and Schacter, 2006), and the amygdala exerts an influence on activity in prefrontal cortex and MTL structures during long-term memory encoding (Hamann et al., 1999; Kilpatrick and Cahill, 2003). In addition, amygdala activity has been demonstrated during implicit processing of emotional stimuli (Critchley et al., 2000; Phelps and LeDoux, 2005) or when these stimuli are task-irrelevant distracters (Siegle et al., 2002; Dolcos and McCarthy, 2006; Siegle et al., 2007), which is consistent with the fact that this limbic region showed sustained activity in our task regardless of trial type. In addition to its known role in long-term emotional memory and consolidation (McGaugh, 2004), the sustained activity noted in the present study suggests an additional role for the amygdala in the active maintenance of emotional and socially relevant material in WM.

In addition to sustained activity in the OFC and amygdala, we also found sustained activity within the hippocampus. Previous fMRI data has demonstrated a role for MTL structures, including the hippocampus and surrounding parahippocampal cortex, in the short term maintenance of novel information (Ranganath and D'Esposito, 2001; Stern et al., 2001; Ranganath et al., 2003, 2005; Schon et al., 2004; Schon et al., 2005). Based on the rich connections between the hippocampus, amygdala, and OFC (Barbas, 2000; Insausti and Munoz, 2001; Munoz and Insausti, 2005), we suggest that together these areas form a circuit that binds together and actively maintains emotion and identity information in WM.

Transient responses to face presentations in temporal and occipital regions

Our results demonstrated transient activity during the sample period in the inferior occipital cortex, extending into the fusiform gyrus, that was significantly greater during ID trials than during EMO trials, and transient activity in the superior temporal sulcus that was significantly greater during EMO trials than ID trials (Fig. 5A,B). This result is consistent with data showing that inferior occipitotemporal regions are involved in perception of facial identity (Kanwisher et al., 1997; Haxby et al., 2001), whereas superior temporal regions are involved in perception of emotional cues and may be important for social perception in general (Hasselmo et al., 1989; Narumoto et al., 2001; Vuilleumier et al., 2001; Andrews and Ewbank, 2004; Winston et al., 2004). Neuroanatomically, the superior temporal sulcus and inferior temporal cortex have strong connections with the OFC (Barbas, 1988; Seltzer and Pandya, 1989; Morecraft et al., 1992), and an effective connectivity study has demonstrated a strong functional connection between amygdala and fusiform gyrus during processing of faces (Fairhall and Ishai, 2007).

In addition to regions within the TOC, posterior parahippocampal cortex only responded transiently to face presentations (Fig. 5C). This result is consistent with a study that found sustained activity in the hippocampus, but transient activity in parahippocampal and anterior fusiform gyri, during delayed matching of faces (Ranganath and D'Esposito, 2001).

Linking the two systems

The active maintenance of social cues requires the cooperation between systems critical for perceptually identifying and processing emotion and identity, and systems critical for actively maintaining this information in memory. In studies where only one dimension, such as identity, is varied, it may be possible for posterior TOC to actively maintain this information in the same system where the information is initially perceptually processed. But in more natural social situations, such as when both identity and emotional expressions can vary, then information from these perceptual TOC regions may be relayed to a memory system that is capable of flexibly binding and maintaining this information together. We suggest this system is comprised of the orbitofrontal cortex, the amygdala, and the hippocampus. We conclude that this system, which is already known to play a critical role in binding together emotional information in long-term memory (McGaugh, 2004), is also critical for actively maintaining, and binding together, salient social cues in working memory.