What have Klingon letters and faces in common? An fMRI study on content‐specific working memory systems

Abstract

Neuroimaging studies show that prefrontal, premotor, and parietal cortical regions are part of a working memory network that supports the active retention of information. In two experiments we used fMRI to examine whether prefrontal and posterior cortical areas are organized in a content‐specific way for object and spatial working memory. Subjects performed a delayed matching‐to‐sample task modified to allow the examination of content‐specific retention processes, independent of perceptual and decision‐related processes. In Experiment 1, either unfamiliar geometrical objects (Klingon letters from an artificial alphabet unknown to the participants) or their spatial locations had to be memorized, whereas in Experiment 2, either unfamiliar faces or biological objects (butterflies) were actively memorized. All tasks activated a similar cortical network including posterior parietal (banks of the intraparietal sulcus), premotor (banks of the inferior precentral sulcus) and prefrontal regions (banks of the inferior frontal sulcus), and the presupplementary motor area (pre‐SMA). For geometrical objects and faces for which strategic semantic processing can be assumed, this activation was larger in the left than in the right hemisphere, whereas a bilateral or right dominant distribution was obtained for butterflies and spatial locations. The present results do not support the process‐specific or content‐specific view of the role of the prefrontal cortex in working memory task. Rather, they suggest that the inferior prefrontal cortex houses nonmemonic strategic processing systems required for response selection and task management that can flexibly be used across a variety of tasks and informational domains. Hum. Brain Mapping 11:146–161, 2000. © 2000 Wiley‐Liss, Inc.

INTRODUCTION

Working memory refers to the capability to consciously hold and manipulate information in mind temporarily, a cognitive function that is central to a large variety of higher cognitive processes. Even simple working memory tasks, those in which subjects retain a memory of a visual stimulus for a brief delay, require an ensemble of cognitive operations. These operations include perceptual encoding of a stimulus, the maintenance of its representation, and, usually, a decision process indicating whether the contents of working memory match with a target item. Delay tasks of this kind have often been employed to study the brain regions mediating visual working memory processes [Haxby et al., 1995; Courtney et al., 1996; Smith and Jonides, 1998] Another form of working memory tasks requires subjects to not only retain a memory of a stimulus but to perform computations with working memory representations [Cohen et al., 1997; Smith and Jonides, 1999]. An example for this kind of working memory task is the n‐back task, in which subjects have to decide whether an actual stimulus matches the one n‐back in the sequence. An ensemble of brain regions were shown to be activated in both versions of working memory tasks, including ventromedial temporal, posterior parietal, premotor, supplementary motor areas, and inferior and superior frontal areas.

Several views about the role of the prefrontal cortex in visual working memory tasks have been proposed: One group of models propose a content‐specific subdivision of the prefrontal cortex, with ventrolateral prefrontal regions in the vicinity of the inferior frontal sulcus (BA45/47) being recruited by object working memory tasks and dorsolateral regions (BA 46/9) being engaged by spatial working memory tasks [Goldman‐Rakic, 1987; Wilson et al., 1993]. Recently, the spatial working memory module has been extended by a region near the superior frontal sulcus, anterior to the frontal eye field. [Courtney et al., 1998; Ungerleider et al., 1998; Haxby et al., 2000]. A content‐specific subdivision of the prefrontal cortex has also been proposed by Smith and Jonides [1999]. According to their view, content specificity is mainly a laterality effect, with the right ventral and dorsal prefrontal cortex being specialized for spatial working memory and the corresponding left hemisphere regions for object working memory.

Another group of models proposes a process‐specific subdivision of prefrontal cortices [Petrides, 1994; Owen et al., 1996]. It is assumed that ventral areas are recruited when a task requires that information is maintained across a nondistracted time interval and is concerned with so‐called “first‐order executive processes” such as the active organization of response sequences, whereas dorsal areas are concerned with “secondary‐order executive processes” such as the monitoring and manipulation of working memory contents.

A third view proposes that the ventral prefrontal cortex is not specialized for working memory but rather is more important for stimulus selection and attentional processes [Rushworth et al., 1997]. This view is based on experimental lesion studies with animals showing that lesions in the inferior convexity region (partly homologous to the ventral prefrontal cortex in humans) do not impair task performance in delay tasks for visual pattern and colors. The view that the lateral prefrontal cortex plays no specific role for working memory is also supported by the observation that lesions of the prefrontal cortex do not lead to deficits in verbal or spatial working memory capacity [D'Esposito and Postle, 1999]. In support of this view, left inferior prefrontal cortex activation has been reported in a variety of tasks such as semantic classification [Thompson‐Schill et al., 1997; Gabrielli et al., 1998], task switching [Dove et al., 1999], or episodic retrieval [Kelley et al., 1998; Opitz et al., 2000], which do not involve the temporary maintenance of information. By these data the prefrontal cortex houses operations, such as stimulus selection or response organization, that are required by a variety of tasks but not specifically tied to working memory tasks.

A similar functional ambiguity exists with respect to the role of posterior cortical regions in working memory tasks. On the one hand it is assumed that the same posterior brain regions that are specialized for the perception of particular stimulus attributes are also engaged when these stimulus attributes have to be maintained in working memory. In support of this view, Courtney et al. [1996] found activation in the ventromedial temporal lobes, i.e. a brain region that is part of the ventral, object perception pathway, when faces had to be retained in working memory, whereas regions in the superior and inferior parietal cortex, which are part of the dorsal, spatial perception pathway, were activated when spatial locations were maintained in working memory. This dorsal‐ventral difference between object and spatial working memory tasks in posterior cortex was confirmed by an integrative analysis of ten PET and fMRI studies on visual working memory [Smith and Jonides, 1999]. Conversely, other studies contrasting spatial and nonspatial working memory found a pronounced decrease of ventromedial temporal lobe activation in a nonspatial working memory task relative to a nonspatial perceptual task [Belger et al., 1998] or a negative correlation between ventromedial temporal lobe activation and retention delay in a face working memory task [Haxby et al., 1995]. This pattern suggests that the ventromedial temporal lobes are more engaged by perceptual processes and are recruited for the maintenance of working memory contents only under particular circumstances.

An alternative view of posterior cortex contribution to working memory holds that amodal representation areas in the inferior parietal cortex (BA40) bind sensory fragments of perceptual inputs and store amodal perceptual and task‐related representations in working memory [Clark et al., 2000]. According to this view, activation in the inferior parietal cortex reflects an amodal storage component engaged by phonological as well as visuo‐spatial contents.

The present experiments were designed to investigate the content‐specificity of frontal and posterior cortical regions in visual working memory tasks. We used a spatial and nonspatial delayed response task and measured brain activation in the delay interval with event‐related fMRI. Several features of our modified delayed response task make it particularly suitable to examine the maintenance of various informational contents in working memory. First, the maintenance of spatial and nonspatial features can be examined using the same stimulus materials and without confounding these processes with perceptual encoding, content manipulations, or decision‐related processes. Second, similar to the “difference wave” technique in event‐related brain potential analysis [Rugg and Coles, 1995], in which a physiological signal in an experimental condition of interest is directly contrasted with a baseline condition, the present paradigm contrasts working memory trials with baseline trials without any maintenance requirements.

EXPERIMENT 1: METHODS

Subjects

A total of 16 subjects (7 male) with a mean age of 25 years (range: 22–29 years) participated in Experiment 1. All were right handed and had normal or corrected‐to‐normal vision. All subjects provided informed consent and were paid 15 DM/hr for participation.

Stimuli and Procedure

The basic task layout is illustrated in Figure 1. The subject's task was to memorize two objects and their spatial locations provided by S1 until a cue indicated whether object forms or spatial locations were relevant for the subsequent comparison with S2 (memory tasks) or that a simple visual classification task will be required at S2 (baseline task). In the memory tasks, subjects had to decide whether both objects (or locations) in S2 were identical to the ones shown in S1. In the baseline task they indicated whether the two digits presented in S2 were identical. They indicated their responses (same/different) by pressing one of two response buttons.

Figure 1.

Schematic illustration of the working memory and baseline trials in Experiment 1.

The S1 and S2 were two distinct green‐colored objects from a set of 32. These objects were generated by transforming the Microsoft Windows font Klinzhai (“Klingon letters”) into a 3D representation using the freeware program POVRAY V2.2. The objects were located on a virtual horizontal gray plane. One object was located in the left half of the screen, the other in the right. There were 32 possible locations possible, 16 on each side. The positions varied in spatial depth and on the left‐right dimension. Objects were 4.5 ± 0.7 cm wide with a height of 4.75 ± 0.25 cm. The horizontal distance between the objects in the display was 8.5 ± 2.5 cm. In addition to the objects, S2 contained two digits presented randomly in two of the corners of the screen. Task cues were presented in between S1 and S2 and consisted of the strings ”Obj,“ ”Pos,“ or ”Zif.“ They were presented at the center of a 17″ VGA monitor extending visual angles of 3° horizontally and 0.4° vertically.

The sequence of events in a single trial was as follows: A button press initiated the trial. Each trial began with the presentation of an empty 3D virtual space. Next, two objects were shown. The task cue was presented 4 sec later and after another 4 sec, S2 was presented. Subjects had to respond within a 2,000‐ms time period after S2 onset. Thereafter, feedback indicating correct, incorrect, or timeout responses was provided. The subsequent trial started 4 sec after onset of the feedback stimulus. Subjects were instructed to respond as quickly and as accurately as possible. In 50% of the trials the response associated with the irrelevant information (objects in spatial trials and spatial locations in object trials) was incongruent with the one dictated by the relevant information. Following a training session of 60 trials, all subjects performed a total of 180 trials (60 trials in each of the three tasks) while fMRI was recorded. There were 12 repetitions of each object and each spatial location across the experimental session. Both response types were equiprobable within each task and task order was pseudorandomized such that each task could follow each other with the same probability.

MR image acquisition and analysis

Imaging was performed at 3T on a Bruker Medspec 30/100 system equipped with a standard ‘bird cage’ head coil. Subjects lay on the scanner bed and cushions were used to reduce head motion. In addition to the functional data sets high resolution whole brain images were recorded to improve the localization of activation foci using a T1 weighted 3D segmented MDEFT sequence (128 slices sagital, 1.5 mm thickness, 256 × 256 pixel matrix). To align the echo planar functional images to the 3D images, conventional anatomical images in plane with the functional images were acquired as an intermediate step using a IR‐RARE sequence (TE = 20 ms; TR = 3,750 ms, matrix 512 × 512). Finally, functional images were acquired using a gradient EPI sequence (TE = 40 ms) sensitive to BOLD contrast. Data were recorded from 14 axial slices parallel to the AC‐PC line at a rate of 2 sec per image. Image acquisition was synchronized with the onset of stimulus presentation. In each trial seven EPIs per slice were acquired, yielding a total of 180 × 7 = 1,260 images per slice. Slice thickness was 5 mm with an interslice distance of 2 mm. The field of view was 19.2 mm with a data matrix of 64 × 64 voxels.

FMRI data were processed using the BRIAN software package [Kruggel and Lohmann, 1996]. Prior to statistical analyses, movement artifacts were detected by thresholding the ratio of foreground (brain) to background intensity for each image. Images at which this ratio was lower than 12.5 were classified as artifacts [Kruggel et al., 1998]. Spatial smoothing with a Gaussian Kernel (FWHM = 2.35 mm) was applied, and low‐frequency signal drifts were controlled for by applying a voxel‐based temporal high‐pass filter (0.03 Hz cutoff frequency).

Activation related to task cue onset was identified by the construction of individual and group z‐maps. For each subject, voxel‐wise t‐tests for unequal sample sizes were performed to contrast the activation evoked by the task cue in each of the memory tasks with the baseline task. Only trials with correct responses were entered in this analysis. Because the BOLD response typically reaches its peak level 4–6 sec after event onset [Buckner et al., 1998], the two images chosen for t‐test analyses were those beginning 4 sec after task cue onset. The t‐test statistics were normalized to z‐scores. For each subject the z‐maps were coregistered with the individual high‐resolution 3D data sets and transformed into the Talairach coordinate space [Talairach and Tournoux, 1988]. Group z‐maps were computed by performing t‐tests for known variance at corresponding voxels across subjects [Bosch, 2000]. These z‐maps were thresholded to eliminate voxels for which the across subject z‐scores were < 2.33 (P < .01; one‐tailed; uncorrected for multiple comparisons).

Regions of interest (ROIs) were defined as spheres with a radius of 5 mm, centered on the local maxima of the summed activity of the two memory tasks. If a region did not reach significance in one task, the respective maximum of the other task was chosen. In a second step, z‐values were averaged within each ROI, separately for each task and subject, thereby allowing repeated‐measure ANOVAs with factors ROI and task to be calculated [Bosch, 2000].

As an additional analysis, we computed the event‐related time course of the BOLD response separately for each of the tasks in each of the spherical ROIs. Average raw activation values were obtained from nine images starting with the image immediately prior to S1. Separately for each of the subjects, these values were z‐standardized taking into account the variance across all three tasks. Finally, the resulting time courses were averaged across subjects.

RESULTS

Performance

The corrected recognition scores Pr (hit rate/false alarm rates) were .62 (SEM .03), .61 (SD .04), and .91 (SD .02) in the object, spatial, and baseline task, respectively. A repeated‐measure ANOVA revealed an effect of Task, F(2,30) = 36.07, P < .001. Post hoc tests revealed that the recognition scores were higher in the baseline task than in the two memory tasks, P's < .001 but did not differ between the two memory tasks, P < .66.

The mean reaction times were 936 ms (SEM 38), 812 ms (SEM 29), and 859 ms (SEM 23) in the object, spatial and baseline task, respectively. Again there was a main effect of Task, F(2,30) = 9.52, P < .0006. As revealed by post hoc tests, the subjects responded slower in the object task than in the other two tasks, P < .02, for which reaction times were not significantly different, P < .16.

Activation results

Figure 2 shows the across‐subject activation in the object task (upper panel) and the spatial task (lower panel). Both memory tasks activated similar regions that included posterior parietal areas in the banks of the intraparietal sulcus (IPS), lateral premotor areas along the banks of the inferior precentral sulcus (IPCS), prefrontal regions in the banks of the inferior frontal sulcus (IFS) and in the pre‐SMA [Picard and Strick, 1996]. In the object task, the activation was more intense in the left hemisphere than in the right. In the spatial task it was bilateral, except for the lateral premotor area where it was larger in the right hemisphere. Additional activation was observed in the left and right insula (INS) for both tasks.

Figure 2.

Across‐subject activation patterns in the object and spatial task of Experiment 1. The upper row shows axial, coronar, and lateral views of the activation pattern in the object tasks, the corresponding views in the lower row show the brain areas activated in the spatial task. In this figure the threshold was set to z = 2.33 and the average z‐scores were superimposed on a randomly selected T1 image of an individual brain in Talairach space (for further details see Methods section.)

To quantify the distribution of activation in both tasks, regions of interest (ROI) were defined for the IPS, IPCS, IFS, pre‐SMA, and INS. The mean z‐scores for each ROI in each task are displayed in Figure 3 and the mean Talairach coordinates for each ROI are shown in Table I. In a second step the mean z‐scores were subjected to repeated‐measure ANOVAs. Two analyses were performed: First the bilateral activations in the IPS, IPCS, IFS, and the insula were entered in a three‐way ANOVA with factors Task (two levels), Region (four levels), and Hemisphere (2 levels). Since the pre‐SMA activation did not reveal clear patterns of lateralization a one‐way ANOVA with factor task was performed for this region.

Figure 3.

Activation pattern (mean z‐scores + 1 SEM) at frontal, temporal, and parietal regions for both tasks as a function of left and right hemisphere. IFS: inferior frontal sulcus; IPCS: inferior precentral sulcus; IPS: intraparietal sulcus; INS: insula.

Table I.

Anatomical location (in Talairach coordinates) of the ROIs in Experiment 1 for the object and spatial memory task*

	Object task			Spatial task
	x	y	Z	x	y	z
pre‐SMA	8	7	50	−6	10	50
IFS (R)	39	29	26	38	29	25
IFS (L)	−43	21	26	−43	21	27
IPS (R)	36	−50	45	31	−53	33
IPS (L)	−37	−58	37	−41	−56	39
IPCS (R)	42	−1	29	43	1	28
IPCS (L)	−47	−0	34	−48	−0	34
INS (R)	26	21	12	26	21	12
INS (L)	−31	23	11	−33	22	12

^*IFS: inferior frontal sulcus; IPCS: inferior precentral sulcus; IPS: intraparietal sulcus; INS (insula).

Table II.

Anatomical location (in Talairach coordinates) of the ROIs in Experiment 2 for the butterfly and face memory tasks*

	Butterfly task			Face task
	x	y	Z	x	Y	Z
pre‐SMA	−7	5	51	−6	5	52
ant‐IFS (L)	−39	28	20	−39	27	19
IFS (R)	36	22	28	36	14	42
IFS (L)	−44	17	36	−43	15	32
IPCS (R)	46	−6	36	—	—	—
IPCS (L)	−45	−3	38	—	—	—
IPS (R)	30	−54	38	—	—	—
IPS (L)	−33	−54	34	−36	−56	32

^*IPCS and IPS activation in the right hemisphere in the face memory task was rather small and no clear maximum was discernible. IPCS activation in the left hemisphere for faces was not discernible from the adjacent IFS activation. For all three regions the corresponding coordinates of the object task were used. IFS: inferior frontal sulcus; ant‐IFS: anterior inferior frontal sulcus; IPCS: inferior precentral sulcus; IPS: intraparietal sulcus.

The three‐way ANOVA revealed a highly significant three‐way interaction, F(3,45) = 19.21, P < .0001). To decompose this interaction, Task × Hemisphere ANOVAs were performed separately for each of the ROIs. For the IFS there was a Task × Hemisphere interaction, F(1,15) = 13.69, P < .002. Post hoc tests revealed that IFS activation was larger in the left than in the right hemisphere in the objects task, P < .02, but not in the spatial tasks. A Task × Hemisphere interaction was also obtained for the IPCS, F(1,15) = 19.42, P < .0005. As revealed by post hoc test, the spatial tasks activated the right IPCS more than the object tasks, P < .008. In addition, there was a trend toward a larger left IPCS activation in the object than the spatial task, p < .08. For the IPS there was a Task × Hemisphere interaction, F(1,15) = 9.26, P < .008, indicating that the object task activated the left IPS to a larger extent than the right IPS, P < .01, whereas there was no IPS lateralization in the spatial task, P < .95. Finally, the insula was stronger activated in the object task in both hemispheres, F(1,15) = 12.64, P < .001 and the pre‐SMA was stronger activated by objects, F(1,15) = 7.20, P < .01.

To examine the temporal characteristics of the BOLD response in each of the ROIs in more detail we computed the time courses of the activation pattern for each trial in each of the three tasks. Figure 4 shows these time courses separately for each of the three tasks. Two peaks of the BOLD response were obtained, one at around 4 sec after S1 and another at around 4 sec after S2. Taking in account a temporal delay of the BOLD response of about 4 sec, this pattern of results indicates that the most pronounced changes in hemodynamic activation were evoked by these two stimuli. Notably, the BOLD response was highly similar for the three tasks up to 4 sec after cue onset. At this time point, the BOLD response remained elevated for the two memory tasks and decreased for the baseline task. Memory task‐specific activation patterns were obtained at this time point that extended for 1 to 2 images (i.e., 2 to 4 sec).

Figure 4.

Time course of the activation patterns (mean z‐scores + 1 SEM) for each of the three tasks and the regions of interest analyzed in Experiment 1.

DISCUSSION

Experiment 1

In Experiment 1 we contrasted the pattern of fMRI activation in a nonspatial and spatial working memory task. To ensure that any content‐specific activation patterns were not confounded with encoding and/or decision‐related processes, event‐related fMRI was recorded in a modified delayed matching paradigm. A task cue was presented in the middle of a delay interval and indicated whether a memory‐based comparison for objects or spatial locations or a simple perceptual discrimination task was required at S2. Task‐specific activation was measured starting about 4 sec after the onset of the task cue until about 6–8 sec thereafter. Consistent with prior studies, we found bilateral activation in the banks of the IFS, IPCS, IPS, and in the insula. The hemispheric weighting of this activation pattern was content specific: The right IPCS region was stronger activated in the spatial working memory task, whereas the reversed pattern (larger object memory activation) was obtained for the left IPCS region. Object working memory activated more the left IFS and the IPS, whereas both regions were equally activated across hemispheres in the spatial working memory task. Moreover, the right IPCS region was stronger activated in the spatial working memory task than in the object working memory task, whereas the reversed pattern (larger object than spatial memory activation) was obtained for the left IPCS region.

The interhemispheric distribution of inferior prefrontal activity is consistent with the content‐specific view that assumes that content specificity is expressed as a hemispheric laterality effect in the prefrontal cortex [Smith and Jonides, 1998]. The prefrontal activation pattern found in the present study resembles that of previous working memory studies, particularly showing a different hemispheric pattern as a function of the informational content maintained in working memory. Left hemisphere activation, was found in the ventrolateral prefrontal cortex (i.e., along the left inferior frontal sulcus) in fMRI studies requiring the maintenance of an abstract object's shape [McCarthy et al., 1996, Belger et al., 1998; Postle and D'Esposito, 1999] or of unfamiliar faces in working memory [Courtney et al., 1998]. Using PET, Smith et al. [1995] also found activation in the left inferior frontal region, including the IFS when namable objects had to be retained in working memory, which was absent when verbal rehearsal strategies were controlled. Bilateral or right lateralized prefrontal activation for spatial working memory tasks has been reported in a number of studies, either in the IFS [Jonides et al., 1993; McCarthy et al., 1996; Baker et al., 1996] or in the vicinity of the so‐called frontal eye field [Courtney et al., 1998].

Activation in the lateral premotor cortex and pre‐SMA is less consistently associated with visual working memory tasks. Baker et al. [1996] found larger lateral premotor cortex activation as a function of motor preparation. Note, however, that the lateral premotor cortex is activated by actual movements that are initiated by visual stimuli and seems to play a major role in sensory guided motor behavior [Roland et al., 1980]. Increased activation along the banks of the left precentral sulcus was found during the observation of graspable objects [Grafton et al., 1997; Martin et al., 1996] and it was proposed that this region houses motor schemata for object use [Martin et al., 1996]. Given that the present study used 3D objects presented in a 3D space, it is conceivable that the left dominant inferior lateral premotor activation in the object task reflects the activation of motor programs for grasping the objects with the dominant hand, even though no such movements had to be performed. Conversely, the right hemisphere activation in the spatial task might reflect the fact that spatial mnemonic contents implicitly activate motor programs for whole body movements, which are required for navigation and movement through space.

The banks of the IPS were activated in both tasks. Object memory activated predominantly the left IPS, whereas spatial memory leads to bilateral IPS activation. Coactivations of inferior parietal regions and premotor regions have been reported in two working memory tasks [Jonides et al., 1993; Baker et al., 1996], and during spatial attention shifting [Corbetta et al., 1993] and self‐determined finger movements [Schubert et al., 1998]. The inferior parietal cortex has also been coactivated with regions around the superior and middle frontal gyrus [Klingberg et al., 1997]. Thus it appears that the IPS is a functionally heterogeneous region that is activated in a variety of cognitive tasks. However, the posterior IPS (BA40/39) has consistently been found to be engaged in encoding spatial locations and the processing of spatial relations [Ungerleider and Haxby, 1994; Colby et al., 1995; Carpenter et al., 1999]. Given this, it is conceivable that the present IPS activation is associated with spatial encoding and storage functions [Smith and Jonides, 1998]. In the object task, the IPS activation could reflect the fact that representations of spatial locations were still active in the cue‐S2 interval even though they were irrelevant for the S2 comparisons. The present results therefore seem to support the view that the dorsal processing stream, though its main function is perceptual, is also required when spatial locations are maintained in working memory.

Using the present methodology, we did not find activation in occipito‐temporal regions of the ventral stream in either task. This result was surprising given that different areas in this region have been found to be activated by different visual stimulus characteristics, not only in perceptual tasks [Colby et al., 1995] but also in a variety of nonspatial working memory tasks [Belger et al., 1998, Baker et al., 1996; Courtney et al., 1996; Postle and D'Esposito, 1999; Jiang et al., 2000]. It is conceivable that the absence of ventral stream activation in the object task together with the strong IPS activation in both tasks is related to the 3D spatial character of the stimuli. This might have imposed a spatial encoding strategy by which the objects were encoded by means of the location and orientation of their main axes in the 3D space. To test this assumption a second experiment with two nonspatial working memory tasks was performed. The stimuli for this experiment were selected with the goal to avoid any spatial encoding strategies.

A second issue addressed in Experiment 2 concerned the lateralization pattern in the ventrolateral prefrontal cortex. Left lateralized prefrontal cortex activation in object working memory tasks similar to the one obtained for geometrical objects has been attributed to conceptual or verbal encoding of object features [Smith and Jonides, 1995]. Moreover, the left prefrontal cortex has been shown to be recruited by strategic processes associated with semantic retrieval in a variety of nonworking memory tasks [Thompson‐Schill et al., 1997; Gabrieli et al., 1998] It is conceivable that the left lateralization of IFS activation reflects processes associated with the retrieval of semantic features, such as the functional and/or visual characteristics of geometrical objects. To test this assumption we used two classes of objects that differ in the availability of semantic features, i.e. pictures of unfamiliar faces and of biological objects (butterflies), in Experiment 2. Both classes of stimuli have been shown to activate the ventral stream in perceptual and working memory tasks [Kanwisher et al., 1997; Bentin et al., 1999; Chao et al., 1999]. We assumed that semantic encoding plays a more important role for faces, which we learn to discriminate on an individual level in early childhood. Seeing an unfamiliar face may be associated with the retrieval of semantic concepts like dark skin or gray hair. This form of encoding may be less likely for biological objects such as butterflies because we are simply not used to encode the textures and gradings of single biological objects to the same extent as for faces [Gauthier 2000]. These latter assumptions were tested in an additional dual task experiment.

EXPERIMENT 2: METHODS

Keeping the basic logic of Experiment 1, cueing the subjects on the task‐relevant dimension within a retention interval, the present experiment used two nonspatial memory tasks with different stimuli, each one combined with a baseline task matched for all stimulus characteristics. Subjects saw either pictures of unfamiliar faces or butterflies. Thereafter the task cue indicated whether a memory‐based comparison was required at S2 (memory tasks) or whether a visually based comparison, as in Experiment 1, was to be performed. Recent fMRI studies revealed that perceiving faces activated different parts of the fusiform gyrus than perceiving nonbiological objects [Chao et al, 1999; Ishai et al., 1999; Bentin et al., 1999] or biological objects [Chao et al., 1999]. Given that the ventral temporal cortex is not only involved in object and face perception but that the same regions are also recruited by object and face working memories, we expected differential ventral temporal cortex activation for the butterfly and face working memory task. A second prediction concerned the lateralization of ventrolateral prefrontal cortex activation. Given that left IFS activation reflects processes associated with semantic retrieval we expected left lateralization for unfamiliar faces but not for butterflies.

Subjects

Twelve right‐handed subjects (5 male) with a mean age of 25.5 years (range: 20–33 years) participated in the experiment. All subjects were students from the University of Leipzig, had normal or corrected‐to‐normal vision, and were paid 15 DM/hr for participation. Two subjects also participated in Experiment 1.

Stimuli and procedure

The procedures for stimulus presentation were the same as in Experiment 1. S1 contained either black‐and‐white pictures of three unfamiliar natural faces or two butterflies, whereas at S2 either one face or one butterfly was presented. The total stimulus set included 48 faces (50% female) and 36 butterflies. Each single picture was 4.5 cm wide with a height of 5.5 cm. As in Experiment 1 S2 also contained two digits presented at random in two corners of the screen. The task cues between S1 and S2 consisted of the strings “Mem” or “Zif.”

The subject's task was to memorize the faces or the butterflies until the cue indicated that either a memory‐based comparison (memory tasks) or a visual classification task (baseline task) will be required upon presentation of S2. In the memory task, subjects had to decide whether the face (butterfly) in S2 is one of the faces (butterflies) seen at S1. In the baseline task they indicated whether the two digits were identical. Response requirements were the same as in Experiment 1 and the sequence of events in a single trial was the same as in Experiment 1, except that the cue‐S2 SOA was lengthened to 6 sec.

Following a training session of 48 trials, all subjects performed a total of 192 trials (48 trials in the face memory, 48 in the butterfly memory tasks, and 48 trials in each of the two baseline tasks) while fMRI was recorded. Across the experimental session there were eight repetitions of each butterfly and each face. As in Experiment 1, ‘same’ and ‘different’ responses were equiprobable within each task and task order was pseudorandomized such that each task could follow each other with the same probability.

MR image acquisition and analysis

The procedures for MR image acquisition and analyses were the same as in Experiment 1, except that 8 images per slice were acquired in each trial and 16 rather than 14 axial slices were recorded. Due to technical artifacts, one subject had to be excluded from the analyses. For each subject voxel‐wise t‐tests for unequal sample sizes were performed to compare the activation evoked by the task cue in each of the memory tasks with its corresponding baseline task. To take the longer retention interval into account, three rather than two images per slice beginning 4 sec after cue onset were selected for the t‐test analyses. The ANOVA used the same regions of interest as in Experiment 1. Additional activation patterns were considered significant if they exceeded a threshold of z = 1.65, P < .05; one‐tailed.

Dual task experiment

Fifteen subjects performed the butterfly and face working memory tasks and the baseline tasks either as single tasks or in dual task settings in different sessions. In the dual task session a semantic classification task was inserted in the cue‐S2 interval. This latter task required an animacy judgment for a noun presented in the middle of the cue‐S2 interval. A total of 192 concrete German nouns were used, half of which were animate. The noun was presented 2 sec after cue onset with a duration of 500 ms and the participants had to respond within a 1,300 ms interval. All other aspects of stimulus presentation were identical with Experiment 2 and only the results of the dual task session will be reported here. Reaction times in the (secondary) semantic classification task were analyzed by means of a two‐way repeated measure ANOVA with factors primary task (memory vs. baseline) and object (faces vs. butterflies). Due to the low error rates in the secondary task, data analysis was restricted to reaction times.

RESULTS

Dual task experiment

Mean reaction times in the semantic classification task were 844 ms (SEM 37) when butterflies were maintained in working memory and 871 ms (SEM 43) when faces were currently held in working memory. The mean reaction times while the baseline task was performed amounted to 853 ms (SEM 38) in butterfly trials and 848 ms (SEM 37) in face trials. These observations were confirmed by a significant primary task × object interaction, F(1,14) = 5.47, P < .03. Post hoc testes revealed longer response times in face trials than in butterfly trials in the memory task, P < .02, but not in the baseline task. The result that semantic classifications are prolonged when faces as compared to butterflies are maintained in working memory is consistent with our assumption that semantic encoding is more likely in the face than in the butterfly working memory task. In support of this conclusion, no differential responses in the semantic classification task were obtained in the face and butterfly baseline trials that did not impose working memory demands.

FMRI study

Performance

The corrected recognition scores were .75 (SEM .05) and .74 (SEM .03) in the butterfly and face memory tasks, respectively, and .89 (SEM .02) and .93 (SEM 01) in the corresponding baseline tasks. A two‐way ANOVA with factors Information Type and Task revealed a main effect of task, F(1,10) = 14.49, P < .003, suggesting that task performance was better in the baseline than in the memory tasks, which did not differ one from the other.

The mean reaction times in the butterfly and face memory tasks were 844 ms (SEM 40) and 804 ms (SEM 35), respectively, and 712 ms (SEM 38) and 731 ms (SEM 37) in the corresponding baseline tasks. The two‐way ANOVA revealed a significant Information Type × Task interaction, F(1,10) = 19.85, P < .001. As indicated by post hoc comparisons, the subjects responded slower in the butterfly than in the face task, P < .001, whereas reaction times were not different in the two baseline tasks, P > .23.

Activation results

Figure 5 shows the across‐subject activation in the butterfly (upper panel) and the face memory task (lower panel). As in Experiment 1, an activation network including the pre‐SMA, posterior parietal, lateral premotor, and prefrontal regions was observed in both tasks. There were two major difference to the activation pattern obtained in Experiment 1: first, as evident from Figure 6, the activation for both tasks was left lateralized except for the activation of the IFS by butterflies. Second, both memory tasks activated more anterior portions of the left IFS. The time course of the BOLD response for this latter region, not activated in Experiment 1, is shown in Figure 7. Similar to Experiment 1, the BOLD response was largest at around 4–6 sec after S1 and S2, respectively, and memory specific activation patterns were obtained at around 4 sec after cue onset. In accordance with the longer retention interval this memory specific activation extended for two to three images (i.e., 4 to 6 sec).

Figure 5.

Across‐subject activation patterns in the butterfly and face task of Experiment 2. The upper row shows axial, coronar, and lateral views of the activation pattern in the butterfly tasks, the corresponding views in the lower row show the brain regions activated by faces. In this figure a threshold of z = 1.65 was applied (see Methods section for further details).

Figure 6.

Activation pattern (mean z‐scores + 1 SEM) at frontal, temporal, and parietal regions for both tasks in Experiment 2 as a function of left and right hemisphere. ant‐IFS: anterior inferior frontal sulcus; post‐IFS: posterior inferior frontal sulcus; IPCS: inferior precentral sulcus; IPS: intraparietal sulcus.

Figure 7.

Time course of the activation patterns (mean z‐scores + 1 SEM) for each of the four tasks in the anterior inferior frontal sulcus region in Experiment 2. Solid lines: memory tasks; dashed lines: baseline tasks.

As in Experiment 1, repeated‐measure ANOVAs were performed for the mean z‐scores in each of the ROIs. The three‐way ANOVA (Task × ROI × Hemisphere) performed for the bilateral activations in the IPS, IPCS, and IFS revealed a main effect of Hemisphere, F(1,10) = 6.48, P < .02, and a significant Task × Hemisphere interaction, F(1,10) = 10.44, P < .001. We elaborated that interaction by performing separate two‐way ANOVAs (Task × Region) for each hemisphere. For the right hemisphere there was an effect of Task, F(1,10) = 12.96, P < .004 indicating that the memory task for butterflies elicited more activity in the IFS, IPCS and IPS than memory for faces. The analysis for the left hemisphere ROIs revealed a Task × Region interaction, F(2,20) = 4.10, P < .03. Post hoc tests showed that the face memory task activated the IFS to a larger extent than the object memory task, P < .03, whereas no between‐task differences were obtained in the IPS and the IPCS. Finally, ANOVAs contrasting the pre‐SMA and the left anterior IFS activation in both tasks revealed larger pre‐SMA activation in the face memory task, F(1,10) = 13,47, P < .004, whereas the left anterior IFS activation was not significantly different for the two tasks, F(1,10) = 3.72, P > .08.

DISCUSSION

Experiment 2

The major findings of Experiment 2 were as follows: Both memory tasks activated a similar neuronal circuitry as in Experiment 1, including the supplementary motor areas, regions along the banks of the inferior frontal sulcus, the inferior precentral sulcus, and the intraparietal sulcus. For both tasks additional activation was found in anterior portions of the left IFS. The activation was larger in the left hemisphere with this asymmetric pattern being more pronounced for the face than the butterflies memory task. Notably, while the left anterior IFS showed a similar activation pattern in both task, the posterior left IFS also found to be activated in Experiment 1, showed larger activation in the face as compared to the butterfly task. Moreover, no activation in the ventral temporal lobes was obtained in neither task.

In this experiment, we minimized the spatial processing demands by employing two nonspatial tasks that required the maintenance of intrinsic visual characteristics of faces and butterflies. As in Experiment 1 there was (left lateralized) activation in the intraparietal sulcus and no reliable activation pattern in occipito‐temporal regions of the ventral stream. We used event‐related fMRI in combination with an experimental paradigm that minimized the contribution of perceptual processes to working memory. In addition we employed two classes of stimuli that in perceptual tasks recruit ventral temporal lobe regions. Using this methodology, the study found no evidence to suggest that the ventral temporal lobe, reported to be involved in object and face perception, plays a role when working memories of nonspatial stimuli have to be maintained [Haxby et al., 1995; Belger et al., 1998].

A second issue addressed in Experiment 2 concerned the processes contributing to the left lateralization of ventrolateral prefrontal activation. Based on prior studies that showed that the left ventrolateral prefrontal cortex is recruited by tasks that require meaningful encoding and strategic processes associated with semantic retrieval, we used two classes of stimuli (unfamiliar faces and butterflies) assumed to differ in the availability of semantic features. A dual task experiment revealed larger interference effect for semantic classifications to be performed in face memory trials than in butterfly memory trials, and by this confirmed the view that semantic encoding plays a larger role for faces than for butterflies. While anterior portions of the left IFS showed a similar activation pattern for both tasks, the posterior IFS was selectively activated by the face working memory task. This pattern of results confirms the view that posterior portions of the IFS are selectively involved in strategic processes associated with semantic retrieval as the need to select relevant semantic features (e.g., blond hair, black moustache) from alternatives that are simultaneously activated.

DISCUSSION

In the present study, hemodynamic brain activity was measured during delays while subjects maintained information of different content. The principal finding was that the maintenance of information was associated with activation in a multisite neuronal circuitry including prefrontal, lateral premotor, and posterior parietal regions. This neuronal network showed a hemispheric asymmetry with larger activation of the left hemisphere in nonspatial tasks and bilateral equal or larger activation of the right hemisphere in the spatial working memory task.

The dorsal and ventral visual processing stream

The main issue addressed in the present study was the functional role of prefrontal and posterior cortical areas when information is maintained in working memory. Previous studies suggest a dorsal/ventral dissociation of posterior cortical areas in visual working memory tasks, with maintaining positional characteristics of stimuli activating the dorsal processing stream and maintaining featural details of objects activating the ventral processing stream [Courtney et al., 1996]. The present findings provide some evidence against the view that object and spatial working memory recruit brain areas assumed to be specialized for the perceptual analysis of either information kind. First, all working memory tasks investigated in Experiments 1 and 2 activated posterior parietal regions along the banks of the IPS. While in Experiment 1 this dorsal stream activation pattern could have reflected the presence of spatial stimulus properties in both the spatial and the nonspatial working memory tasks, this explanation appears to be unlikely in light of the results obtained in Experiment 2. Employing stimuli that minimized spatial processing requirements but rather required the encoding of figural details lead to the same posterior parietal activation pattern as the one obtained in Experiment 1. The fact that posterior parietal activation along the banks of the intraparietal sulcus was found in all tasks is inconsistent with the view that this region is specifically involved in spatial working memory. The IPS (BA 40/39) is a large and functionally diverse structure. It is activated by viewing faces, letter strings, or gradings [Puce et al., 1996] or by the control of self‐determined finger movements [Schubert et al., 1998]. Moreover, it is an amodal region that binds the modal, sensory features of perceptual objects [Clark et al., 2000]. It is possible that IPS activation reflects a cognitive function common to all tasks, like the transient storage of perceptual stimulus features, and that the IPS together with prefrontal and premotor regions constitutes a neuronal circuitry representing both, sensory and action oriented aspects of a working memory content. The present data suggest that the left IPS is more involved in the storage of object features, whereas spatial storage recruits this region bilateraly. The finding that left accentuated posterior parietal cortex (BA40) activation was also found in verbal working memory tasks [Paulesu et al., 1993; Jonides et al., 1998] further supports the view that this region houses multiple working memory storage systems including the phonological modality.

A second line of evidence against the dorsal/ventral dissociation of posterior cortical areas in object and spatial working memory tasks can be derived from the absence of any ventral temporal cortex activation in the present experiments. Some previous studies reporting ventral temporal lobe activation in working memory tasks for nonspatial materials used the PET technique and measured hemodynamic activation in task blocks of more than 30 sec that included perceptual processes such as the encoding of visual stimulus attributes [Smith et al., 1995; Courtney et al., 1996; Baker et al., 1996]. In the present study, we minimized the contribution of stimulus‐specific perceptual processing to the hemodynamic activation pattern. The present data suggest that ventral temporal regions, although engaged in the processing of visual intrinsic stimulus characteristics, are not part of the working memory network per se. Consistent with this view, Belger et al. [1998] found fewer occipito‐temporal cortex activation in a shape working memory task relative to a shape perception task when both tasks altered in the same experimental run. The view that occipito‐temporal brain regions are driven by perceptual rather than mnemonic task components is also supported by recent studies of Haxby et al. [1995], who reported a negative correlation between occipito‐temporal activation and the retention delay in a face working memory task, and of Jiang et al. [2000] who showed a reduction of occipito‐temporal activation with stimulus repetition in a face working memory task. An account for the differential involvement of the occipito‐temporal region can be derived from animal studies, recording single cell activity while monkeys performed simple delayed matching tasks. Delay activity in the inferior temporal cortex was found to be stimulus specific, but notably, it was disrupted when new stimuli were presented in the delay interval [Miller et al., 1996]. It is conceivable that the absence of occipito‐temporal activation in the present experiments is due to the fact that beside stimuli requiring object feature analyses, new stimuli (i.e., the task cues) were also presented that may have disrupted the maintenance‐related activation in this brain region.

Pre‐SMA and lateral premotor cortex

Activation in the pre‐SMA was observed in all memory tasks relative to the baseline tasks. It was larger for objects than for spatial locations in Experiment 1 and also larger for faces than butterflies in Experiment 2. However, this pattern of pre‐SMA activation was not in correspondence with the observed response times, which were larger for objects than spatial locations (Exp. 1) and also larger for butterflies than faces (Exp. 2). The pre‐SMA is involved in the preparation of self‐initiated and highly practiced movements. Given this, our findings suggest that memory‐guided decisions were associated with higher motor preparation demands than visually guided decisions in the baseline tasks.

Consistent with previous studies contrasting object and spatial working memory, the left premotor cortex was more activated by nonspatial information, whereas the maintenance of spatial locations in working memory led to right dominant premotor cortex activation. Animal studies have shown that cells in the premotor cortex are visually responsive and can be tuned by action‐relevant stimulus characteristics [Rizzolatti et al., 1988]. Consistent with this view, the lateral premotor cortex is activated in tasks like passive viewing of tools, tool naming and naming tool‐use, or imaging hand movements [Decety et al., 1994; Martin et al., 1996].

Given that the presentation of a graspable object can activate the lateral premotor cortex contralateral to the dominant hand even if the motion itself is not performed [Grafton et al., 1997; Hari et al., 1998], it is conceivable that working memory for objects, be they geometrical objects, faces or butterflies is strongly associated with prerepresented patterns of specific movements by the dominant hand and that the premotor activation reflects the activation of the intrinsic motor acts performable with those objects [Rizzolatti et al., 2000]. Given that navigation in space involves whole body movements rather than movements of the dominant hand, it is tempting to speculate that the right hemisphere premotor cortex activation in the spatial task reflects mnemonic representations of spatially guided whole body movements. Conversely, since similar right premotor activation has been found in a variety of spatial attention and spatial working memory tasks, this area is assumed to play an important role in spatial rehearsal operations, i.e., covertly shifting of attention from one location to another [Awh et al., 1998; Smith and Jonides, 1999]. A strong link between space coding and movement programming has also been proposed by the premotor theory of spatial attention [Rizzolatti and Camarda, 1987]. Thus, the above‐mentioned accounts may not be mutually exclusive, in that the overlapping right premotor activations in spatial attention and spatial working memory tasks could reflect common mnemonic representations of prepared spatially guided movements that are not immediately executed.

Inferior frontal cortex

As outlined in the Introduction, there are conflicting views with respect to the role of prefrontal areas in working memory tasks. The process‐specific view [Owen et al., 1996; Petrides, 1994] assumes that ventolateral areas are concerned with the selection and organization of motor responses when information is maintained across a time interval, whereas dorsolateral regions subserve manipulation and monitoring of information in working memory. The present paradigm stresses the maintenance of information across a time interval (i.e., the cue‐S2 interval) and minimizes manipulation and monitoring demands. Due to the alternating task order and variable stimulus‐response mappings, it moreover requires a between‐trial reorganization of response parameters. The finding of a dominant activation in ventrolateral regions is therefore not inconsistent with the process‐specific view.

In showing mainly inferior prefrontal activation our results argue against the content‐specific view of prefrontal organization holding that dorsolateral regions are mainly concerned with memory for spatial information and ventrolateral regions with nonspatial materials [Ungerleider et al., 1998]. Another form of content‐specific view of prefrontal cortex, as outlined in the Introduction, is hemispheric specific and proposes that object storage activates the left ventrolateral prefrontal cortex and spatial storage activates right ventrolateral prefrontal and right premotor cortices. This view also appears to be partly supported by the present data. According to Smith and Jonides [1998], the left accentuated ventrolateral prefrontal cortex activation in object working memory tasks is part of a verbal rehearsal circuit that also comprises posterior parietal areas, Broca's area, and the lateral premotor cortex. However, left‐accentuated ventrolateral activation was found for geometrical objects and faces but not for butterflies, suggesting that besides verbal rehearsal, additional factors may contribute to these laterality effects.

An indication as to the nature of these factors was provided by Experiment 2: In showing a robust laterality effect in the face working memory that, as revealed by an additional dual task experiment, also interferes with a simultaneous semantic classification task, the present results suggest that the availability of semantic features of objects to be retained in working memory has a strong influence on prefrontal lateralization. If stimuli allow the extraction of meaning or of meaningful units such as functional, perceptual, or categorical properties available for faces and geometrical objects, a variety of codes may be activated, some of which are selected among competing alternatives as building blocks for a rehearsal mechanism. Conversely, these strategic processes associated with semantic retrieval are rather unlikely to occur for butterflies, which were mainly discriminable by complex and noncategorical textures or for metrical spatial locations used in the spatial working memory task of Experiment 1.

CONCLUSIONS

Overall, the results of the present study suggest that the maintenance of working memory contents is realized by a coactivation of perceptual and motor memory networks that are controlled by inferior frontal regions. These inferior frontal regions seem to link together posterior parietal regions housing amodal perceptual representations with premotor regions housing response representations in the service of goal‐oriented behavior. The left inferior frontal cortex seems to be part of partly overlapping but functionally different networks. On the one hand all stimuli in both experiments activated posterior portions and faces, and butterflies in Experiment 2 led to comparable activation in anterior portions of the IFS. This suggests that IFS activation reflects general processing aspects, such as linking of sensory and motor representation and the selection of information, to guide responses. On the other hand, enhanced left‐accentuated activation was obtained in the posterior IFS for stimuli allowing semantic encoding, suggesting that this specific portion of the left IFS is specifically activated by the selection of semantic information. In sum, the present results do not unambiguosly support either the content‐ or the process‐specific view of the role of the prefrontal cortex in working memory task. They suggest, rather, that the inferior prefrontal cortex houses nonmemonic strategic processing systems required for task management can be used flexibly by a variety of tasks and informational domains.