The neural correlates of visual working memory encoding: A time-resolved fMRI study

Abstract

The encoding of information into visual working memory (VWM) is not only a prerequisite step for efficient working memory, it is also considered to limit our ability to attend to, and be consciously aware of, task-relevant events. Despite its important role in visual cognition, the neural mechanisms underlying visual working memory encoding have not yet been specifically dissociated from those involved in perception and/or VWM maintenance. To isolate the brain substrates supporting VWM encoding, here we sought to identify, with time-resolved fMRI, brain regions whose temporal profile of activation tracked the time course of VWM encoding. We applied this approach to two different stimulus categories– colors and faces – that dramatically differ in their encoding time. While several cortical and subcortical regions were activated during the VWM encoding period, one of these regions in the lateral prefrontal cortex – the inferior frontal junction – showed a temporal activation profile associated with the duration of encoding and that could not be accounted for by either perceptual or general attentional effects. Moreover, this region corresponds to the prefrontal area previously implicated in ‘attentional blink’ paradigms demonstrating attentional limits to conscious perception. These results not only suggest that the inferior frontal junction is involved in VWM encoding, they also provide neural support for theories positing that VWM encoding is a rate-limiting process underlying our attentional limits to visual awareness.

1. Introduction

Visual working memory (VWM), the temporary storage and manipulation of visual information, has been the subject of numerous behavioral and neurobiological studies (e.g., Baddeley & Logie, 1999; Funahashi et al., 1989; Haxby et al., 2000; Jolicœur & Dell’Acqua, 1998; Luck & Vogel, 1997; Miller et al., 1996; Munk et al., 2002; Pessoa et al., 2002; Postle et al., 2000; Todd & Marois, 2004; Wheeler & Treisman, 2002; Vogel et al., 2001). These studies, along with many others, have led to a rich understanding of VWM (Luck & Hollingworth, 2008), including its relationship with attention (e.g. Awh, Jonides, & Reuter-Lorenz, 1998; Cowan, 2001; Corbetta, Kincade, & Shulman, 2002; de Fockert, Rees, Frith, & Lavie, 2001; Downing, 2000; Fougnie & Marois, 2006; LaBar, Giteman, Parrish, & Mesulam, 1999; Mayer et al., 2007; Oh & Kim, 2004; Rensink, 2000; Woodman & Luck, 2004; see this special issue). By comparison, the initial process by which information is encoded into working memory is much less understood. Yet, there is evidence that VWM encoding is functionally dissociable from the storage of information in working memory (Woodman & Vogel, 2005), and may therefore rely on at least partly distinct neural processes. Moreover, VWM encoding is capacity-limited (Jolicœur & Dell’Acqua, 1998; Vogel, Woodman, & Luck, 2006), and it has been suggested that that this capacity limit impairs our ability to consciously perceive multiple, temporally proximate events (Akyürek & Hommel, 2005; Akyürek, Hommel, & Jolicœur, 2007; Chun & Potter, 1995; Jolicœur, 1998). The latter suggestion has been drawn from studies of the attentional blink (AB), which reveals a deficit in the conscious registration of the second of two targets presented among distractor items when the second target (T2) is presented close in time to the first target (T1) (Raymond, Shapiro, & Arnell, 1992). According to the VWM encoding account of the AB, T2 may not be encoded — and consciously perceived — until T1 encoding is complete. In support of this model, the AB is contingent on the rate of encoding of the T1 display (Ouimet & Jolicœur, 2007). However, other studies have proposed that it is not WM encoding, but the control of selective attention, that is ultimately responsible for the AB (Di Lollo, Kawahara, Shahab Ghorashi, & Enns, 2005; Olivers van der Stigchel, & Hulleman, 2007, see also Nieuwenstein & Potter, 2006). Dux and Marois (2009) have proposed a reconciliation of these different viewpoints by arguing that it is selective attention to the encoding process that maybe the capacity-limited process underlying the AB.

If encoding of information into working memory is the rate-limiting step underlying the AB, then one might expect the neural substrates of VWM encoding to overlap with those underlying the AB. Neuroimaging, electrophysiological, and lesion studies of the AB have consistently implicated a network of lateral prefrontal and parietal cortical areas as the neural underpinnings of our limited capacity to consciously perceive multiple targets in RSVP streams (Feinstein, Stein, Castillo, & Paulus, 2004; Kranczioch, Debener, Schwarzbach, Goebel, & Engel, 2005; Marcantoni, Lepage, Beaudoin, Bourgouin, & Richer, 2003; Marois, Chun, & Gore, 2000; Marois, Yi, & Chun, 2004; see also Cooper, Humphreys, Hulleman, Praamstra, & Georgeson, 2004; Gross et al., 2004; Hein, Alink, Kleinschmidt, & Müller, 2009; Kessler et al., 2005; Kihara et al., 2007; Kranczioch, Debener, Maye, & Engel, 2007; Martens, Munneke, Smid, & Johnson, 2006; Sergent, Baillet, & Dehaene, 2005; Williams, Visser, Cunnington, & Mattingley, 2008). According to the encoding bottleneck account of the AB, these regions should therefore be involved in the encoding of information into VWM. Despite the contribution of the rate-limited process of VWM encoding to attentional limitationsin conscious perception, and to working memory in general, the neural mechanisms underlying this process are not well established. Several neuroimaging studies have ascribed the encoding of information into VWM to specific brain regions, most notably the parietal and frontal/prefrontal cortices (Andersen, Essick, & Siegel, 1985; Courtney, Ungerleider, Keil, & Haxby, 1997; Linden et al., 2003; Majerus et al., 2007; Munk et al., 2002; Pessoa, Gutierrez, Bandettini, & Ungerleider, 2002; Petrides, 1994, 1996; Postle, Zarahn, & D’Esposito, 2000; Roth, Serences, & Courtney, 2006; Rypma & D’Esposito, 1999; Todd & Marois, 2004, 2005; Zarahn, Aguirre, & D’Esposito, 1997). However, these studies did not provide a pure measure of VWM encoding because the encoding activity could not be dissociated from either perceptual or maintenance-related activity due to the limited temporal resolution of fMRI.

The goal of the present study was to isolate VWM encoding-specific brain activity, and to determine whether this activity is consistent with neuroimaging studies of the AB. Our experimental strategy for isolating the brain substrates of VWM encoding consisted in using fMRI to identify brain regions whose temporal profile of activation tracked the time course of VWM encoding. To achieve this aim, we took advantage of the fact that the duration of WM encoding increases proportional to increasing object complexity (Alvarez & Cavanagh, 2004; Awh, Barton, & Vogel, 2007; Curby & Gauthier, 2007; Eng, Chen, & Jiang, 2005; Ouimet & Jolicœur, 2007). While fMRI cannot reveal the absolute duration of a neurophysiological process, it can be informative about the relative duration of such process (Dux, Ivanoff, Asplund, & Marois, 2006; Formisano & Goebel, 2003; Henson, Price, Rugg, Turner, & Friston, 2002; Liao et al., 2002; Miezin, Maccotta, Ollinger, Petersen, & Buckner, 2000). Hence, b rain regions involved in WM encoding should show differential durations of activity depending on the time it takes to encode objects of different complexity. These latency effects can be estimated by assessing the differences in the time it takes for the hemodynamic response to reach its peak, as time-to-peak is a reliable measure of duration of brain activity (Dux et al., 2006; Henson et al., 2002).

We applied this time-resolved fMRI approach to the encoding of two different stimulus categories that differ extensively in the duration of encoding: colors, which can be encoded in about 50 ms (Vogel et al., 2006), and faces, which may take about ten times longer to be fully encoded (Curby & Gauthier, 2007). Subjects were instructed to encode, in separate trials, either the color or the face identity of a pair of stimuli (Figure 1). Because encoding a pair of colors should take approximately 100 ms while encoding a pair of faces should take about 1,000 ms, the 900-ms difference in peak latency between these two conditions should be readily resolved with fMRI. In a pilot behavioral experiment, we demonstrated that two colors were fully encoded in about 300 ms, whereas approximately 1,200 ms was needed to encode two faces into VWM (see Supplemental Information). In the fMRI scanner, a stimulus presentation of 1,500 ms should therefore yield different times-of-peak activation depending on whether subjects are encoding colors or faces. By contrast, if the stimuli are backward masked after only 500 ms, thereby halting any further encoding (Vogel et al., 2006) of the faces, there should be relatively little peak time differences elicited by the encoding of the colors and faces.

Figure 1.

Trial design. At trial onset, a cue reminded the subject of the task-relevant feature (C, color; F, face). While performing an articulatory suppression task, the subject encoded the task-relevant feature from the memory array, which was presented for 500 or 1,500 ms. Concurrent with encoding the memory array, the subject monitored for a brief change in the luminance in either memory array stimulus. Following a mask and then a 9,000 ms retention interval, the memory probe was presented. Finally, a question mark appeared in the center of the display and the subject indicated if there was (not) a luminance change in the memory array.

2. Methods

2.1. Participants

Twenty-four individuals (11 males; 23 right-handed; age range, 18–31 years) from the Vanderbilt community participated in this experiment for paid compensation. All subjects reported normal color vision and had normal or corrected-to-normal visual acuity.

Six subjects were excluded from analysis: Four subjects were removed because their performance in the face feature condition was not significantly different from chance, even at the long stimulus duration (50% accuracy), and two other subjects were removed because of improper slice prescription at the fMRI scanner. A total of 18 subjects were retained for final analysis.

2.2. Behavioral methods

2.2.1. Stimuli

In order to ensure that a brain region’s activity differences between color and face conditions are not driven by low-level sensory differences between the two stimulus classes, these feature conditions were combined into a single object by applying a color filter to each grayscale face stimulus (Figure 1). As a result, the same stimuli were presented in the color and face feature conditions, but the task changed by instructing the subjects to encode either the colors or the faces in the sample array(Clark et al., 1997).

In each trial, two grayscale, affect-neutral male faces were selected randomly without replacement from a set of six faces. The constituent faces were selected so their features (eyes, noses, lips, etc.) were similar enough to encourage subjects to encode the entire face, rather than rely upon a single feature to discriminate the faces. Each face’s ears, hair, and neck were masked, and the average contrast of each face was adjusted to match the group mean contrast. Similar to the face stimuli, two colors were selected randomly without replacement from a set of six colors (green, magenta, violet, yellow, cyan, and red) and assigned to each face on each trial. A colored face stimulus (0.84° × 0.76° visual angle) was positioned on each side of a black fixation point presented at the center of a gray screen (Figure 1). The total visual angle of the memory array was 0.84° × 2.0° (height × width), which places all stimuli within the fovea, thereby minimizing the need for subjects to make eye movements. Each stimulus in the sample array was followed by a pattern mask made from a face selected randomly from the remaining five faces and divided into a matrix of 54 squares that were vertically flipped, randomly shuffled, with each square assigned one of the seven colors (Figure 1).

2.2.2. WM encoding task trial design

Trial onset was cued with a letter presented for 500 ms at the center of the monitor to remind the subject of the target stimulus feature to encode, “F” for face and “C” for color. Following a 500-ms fixation period, a sample array of two colored faces was then presented for 500 ms or 1,500 ms, followed by two masks for 500 ms. A 9,000-ms retention interval followed the mask in order to dissociate response probe-related activity from encoding-related activity. The response probe consisted of a single colored face presented at fixation for 3 s, during which subjects made a present/absent judgment regarding whether the probe’s target (specific color or face) was present in the sample array. Subjects used their right index and middle fingers to respond to the presence or absence of the target, respectively. The target was present in the probe on 50% of the trials, and was randomly selected from either the right of left sample array stimuli. In target-absent trials, the probe feature corresponding to the target feature originated from one of the four other exemplars of the 6-exemplar set. The probe’s non-target feature (e.g. color in a face trial) independently matched one of the two sample stimuli on 50% of trials: The probability that a colored face probe was the conjunction of the same face and color features used for one of the sample array stimuli was 25%. Subjects were instructed to maintain fixation throughout the trial and to emphasize accuracy in their response. In order to minimize the contribution of verbal working memory, they also performed an articulatory suppression task throughout each trial (subvocally rehearse “the” at a fast but comfortable rate of about 2–3 times/second) (Baddeley, 1992; Todd & Marois, 2004).

2.2.3. Luminance detection task

Because encoding is completed earlier for colors than faces, subjects might stop attending to the sample array sooner in the color than in the face feature condition in the long (1,500 ms) stimulus presentation. This difference in sustained focal attention duration could potentially drive any differences in the duration of brain activity, thereby confounding the signal associated with WM encoding from that of attention. To control for differences in attention duration, subjects were also required to concurrently monitor the sample array for a 500-ms change in luminance that occurred randomly in one of the stimuli in half of the trials. The changing stimulus progressively dimmed (in 16 ms steps) for 250 ms, then brightened back for 250 ms. In the 1,500-ms sample array, the luminance change occurred in one of three temporal bins (0–500, 500–1,000, or 1,000–1,500 ms from the onset of the sample array). At the offset of the VWM probe, a question mark appeared at fixation for 1,500 ms, and subjects indicated if a luminance change occurred in the sample array (Figure 1). Subjects used the left hand to respond to this task, with the index finger and middle fingers corresponding respectively to the presence or absence of a luminance change, and to emphasize accuracy in their response. The magnitude of the luminance change was manipulated after each fMRI run in order to keep the mean detection accuracy between 70% and 80%.

To motivate subjects to perform as well as possible in both the VWM encoding and luminance detection tasks, they were given the opportunity to earn up to $10 over their base compensation for good performance. For each trial that they performed accurately in both the VWM and luminance detection tasks, they were awarded about 6¢ ($10 ÷ 162, the total number of trials in the experiment). Subjects were informed of their cumulative earned reward after each run.

2.3. Training session

Subjects practiced the two tasks before participating in the fMRI session. They performed six runs (16 trials/run) of each feature encoding condition in each 1-hour training session. These training sessions took place in a mock scanner to acclimate subjects to the fMRI scanner environment (e.g., fMRI scanner sequence sounds, visual stimulus presentation, subject positioning) and to help ensure that behavioral performance in the training session would be comparable to that obtained in the fMRI session (Hannula, Simons, & Cohen, 2005). Subjects were trained equally with both VWM feature conditions until their performance in the color task reached asymptote, which was defined as color task performance being equal for both stimulus presentation durations (500 and 1,500 ms presentations). This was achieved after 2–3 training sessions. To maximize the number of number of trials presented in the training session, the WM maintenance period was reduced from 9 s in the fMRI experimental session to 4 s, and the no-event trials (see FMRI methods, below) were excluded.

2.4. FMRI methods

The 1.5 hr fMRI session was subdivided into six runs of 25 trials, with the target feature alternating between runs (so, three runs per target feature). Prior to each run’s onset, subjects were instructed of the target feature to remember. An equal number of 500-ms and 1,500-ms encoding durations, and of target-probe matches and non-matches, were presented in each run. The presentation order of these four trial types and of a no-event fixation trial type were counterbalanced (5 trials per trial type in each fMRI run; see Todd & Marois, 2004). This no-event condition was included to facilitate extraction of time courses to the feature condition trials (Kourtzi & Kanwisher, 2001; Todd & Marois, 2004; Xu & Chun, 2006). In these no-event trials, a cue stimulus (“C” or “F”) was presented at trial onset and followed by 17.5 s of fixation.

A 3-T Philips Intera Achieva scanner was used to acquire T2*-weighted echoplanar images (TR, 1,000 ms; TE, 35 ms; flip angle, 70°; FOV, 24 cm; matrix, 64 × 64). Each scan consisted of 18 contiguous 5-mm axial slices running parallel to the AC–PC line (in-plane resolution, 5 × 5 mm, 1-mm skip). A total of 506 images were collected in each functional run. Low- and 3-D high-resolution T1-weighted anatomical images were acquired using conventional scan sequences.

Stimuli were presented to the subject, lying supine in the MR scanner, using an LCD back-projection video system. Stimuli were presented to subjects using PsychToolBox for MATLAB. Manual responses were collected from button boxes, using the right hand button box for the WM task and the left hand button box for the luminance detection task.

3. Data analysis

3.1. Behavioral analysis

WM capacity estimates were calculated using Cowan’s (2001) k formula, defined as k = (hit rate − correct rejection rate) × set size, where set size = 2. Although response time (RT) was not emphasized, RT was recorded in order to verify that any differences in k were not due to speed-accuracy tradeoffs. Performance accuracy and RT in the luminance detection were also recorded.

3.2. FMRI analysis

All pre-processing and imaging data analysis was performed using BrainVoyager QX v.1.09 (Brain Innovation). Preprocessing of functional data included 3-D motion correction, slice scan-time correction, intra-session image alignment, linear trend removal, and spatial smoothing using a 6-mm FWHM Gaussian kernel. Data were transformed into standardized Talairach coordinate space for voxel-wise and regions-of-interest (ROIs) analyses.

The analytical strategy consisted in first defining ROIs activated during the VWM encoding phase of the task in group-averaged, random-effect SPMs (collapsing across stimulus features and durations). The resulting foci of activation were then used as regional landmarks for isolating ROIs in individual subjects using a fixed-effects analysis of the same encoding phase (The individual definition of ROIs maximized the sensitivity of the subsequent time course analyses). Only ROIs that could be defined in at least half of the 18 subjects were submitted for time course analysis. The time courses of the individually defined ROIs were then subjected to peak activation latency analyses to determine whether they exhibited temporal profiles of activation that were consistent with either simple effects of stimulus duration or with encoding-specific responses. Importantly, the means to isolate WM encoding-related ROIs do not statistically bias the main peak-latency analysis (Kriegeskorte, Simmons, Bellgowan, & Baker, 2009; Vul, Harris, Winkielman, & Pashler, 2009), as this analysis involves finding an interaction on the signal peak latency between conditions (delayed peak latency for face relative to color at the 1500-ms condition but not at the 500-ms condition; described in detail below) whereas the ROIs were isolated based on a main effect of encoding (collapsed across all encoding conditions) on signal amplitude.

3.2.1. Voxel-wise analysis at the group and individual levels

A voxel-wise multiple regression analysis was first performed to identify foci of activation associated with VWM encoding at the group level by defining an open contrast for the encoding phase (collapsing across features and durations). Regressors for the WM encoding phase were defined as those corresponding to the presentation of the sample array (one volume for the 500-ms duration, two volumes for the 1,500-ms duration). These regressors were then convolved with a canonical, two-gamma hemodynamic response function (Boynton et al., 1996). The resulting random-effects statistical parametric map (SPM) was thresholded using q(FDR) < 0.05 in order to control for the false discovery rate. Above-threshold activated areas in the group SPMs were then used as landmark regions in a subsequent fixed-effects, voxel-wise analysis using the same contrast and statistical threshold in order to isolate regions-of-interest at the individual subject level for the subsequent latency time course analysis.

3.2.2. Region-of-interest analysis

Regions-of-interest (ROIs) were defined at the individual subject level using a cluster threshold of 10 contiguous above-threshold voxels around the peak-activated voxel. Group-level ROIs were created by sorting the individual subjects’ ROIs into common anatomical regions. Only regions localized in at least half of the 18 subjects were submitted for time course analysis (the following regions were not further analyzed because they were not observed in at least half of the subjects: basal ganglia (globus pallidus, caudate, putamen, etc.), thalamus, cuneus, precuneus, superior temporal sulcus).

For a given ROI, time courses were extracted for each combination of features (color/face) and durations (500/1,500 ms). An ROI’s mean percent signal activation change was calculated using the no-event condition as a baseline, on a per-run basis for each subject, and averaged across all runs. The 12-s period following sample array presentation was defined as the encoding response for each time course. Time-of-peak measures of each condition were taken at the highest response amplitude within that 12-s period. The peak latency difference between color and face feature conditions for each sample array duration was then computed in each subject and averaged across subjects. Finally, these latencies were subjected to a repeated measures ANOVA with feature condition and stimulus duration as factors (Both Dim and No-Dim trials were included in the analysis, but the same overall pattern of results was obtained when we restricted the analysis to the No-Dim trials). It is important to note that while increased duration of neural activity will lead to a change in both BOLD response amplitude and peak response latency (because this hemodynamic response is an integral of brain activity over time), peak latency is unaffected by simply changing the intensity of brain activity (Henson et al., 2002, Miezin et al., 2000). Thus, a change in peak latency can be unambiguously ascribed to a change in duration of neural activity rather than a change in amplitude of neural activity. This principle holds regardless of whether the relationship between neural activity and the hemodynamic response is linear or not (Henson et al., 2002; see also supplementary information in Dux et al., 2006).

No corrections for multiple statistical comparisons were applied to the ROI analysis. Our a priori hypothesis that the lateral frontal and intra-parietal cortex should exhibit encoding-related activity (see Introduction) obviates the need for correcting for multiple comparisons for these regions. Applying such corrections for the remaining ROIs do not change the results’ outcome.

4. Results

4.1. Behavioral results

4.1.1. VWM encoding task

An ANOVA with feature (color, face) and stimulus duration (500, 1,500 ms) as factors revealed main effects for both factors (feature, F(1,17) = 161.09, p < 0.001; duration, F(1,17) = 21.14, p < 0.001) and a significant interaction (F(1,17) = 11.14, p = 0.004; Figure 2A). Planned comparisons revealed no difference in color WM capacity estimates between the two durations (500 vs. 1,500 ms, t(17) = 1.30, p = 0.21, two-tailed), but more information was encoded into VWM in the face condition at the longer duration (500 vs. 1,500 ms, t(17) = 4.16, p < 0.001).

Figure 2.

A–B: Visual working memory task performance. (A) Group mean k values for the 500-ms (dark bars) and 1,500-ms (light bars) durations for color (blue/red) and face (azure/pink) conditions. (B) Reaction times for the 500-ms and 1,500-ms durations for color and face conditions. C–D: Luminance detection task performance. (C) Group mean accuracy for the 500-ms (dark bars) and 1,500-ms (light bars) durations for color and face conditions. (D) Reaction times for the 500-ms (blue/red bars) and 1,500-ms (azure/pink bars) durations for color and face conditions. Error bars represent standard error of the mean.

An ANOVA of RT showed an effect of feature condition (F(1,17) = 63.74, p < 0.001): Subjects responded faster in the color than the face condition. There was no effect of sample array duration on reaction time (F < 1; Figure 2B).

4.1.2. Luminance detection task

An ANOVA of luminance detection task performance revealed an effect of duration (F(1,17) = 9.18, p < 0.008; Figure 2C), indicating that luminance detection accuracy was greater for the longer than shorter duration. There was a main effect of feature condition as well (F(1,17) = 7.61, p = 0.01), with accuracy higher in the color than the face condition. This effect of feature did not reflect a speed-accuracy tradeoff, because there was no difference between features conditions on RT (t(17) = 1.01, p = 0.28; Figure 2D). Importantly, and unlike for the WM task, there was no interaction between stimulus duration and feature condition on accuracy (F < 1). Thus, the differential VWM performance of the color and face tasks at the different stimulus durations did not result from a trade-off between the VWM and luminance tasks. Moreover, the relatively high performance in the luminance detection task in all conditions suggests that this task was successful in maintaining subjects’ attention to the display throughout the experimental trials.

4.2. FMRI results

Brain regions involved in VWM encoding were isolated using the following approach. We first identified, on voxel-wise, group -level SPMs, regions that were activated during the encoding phase of the task, irrespective of the feature and stimulus presentation duration, because these regions will include those that are involved in WM encoding. Using this activation map as a mask, we then localized the corresponding regions in individual subjects using a fixed-effects analysis. Finally, the time course of the ROIs that could be identified in at least half of the participants were then examined for a temporal pattern of activation that was sensitive to the duration of stimulus presentation or of VWM encoding.

The group-level statistical parametric map (SPM) revealed several cortical and subcortical regions recruited during VWM encoding (Table 1). Seven of these ROIs showed a main effect of stimulus duration as assessed by peak latency, including the intra-parietal sulcus (IPS), fusiform gyrus (FFG), middle occipital gyrus (MOG), anterior cingulate cortex (ACC), anterior insula, the inferior frontal junction (IFJ), and a middle frontal gyral (MFG) region (Table 2). An example of a duration sensitive region is the right FFG shown in Figure 3A. This FFG ROI was sensitive to the presentation duration of the stimuli (F(1,17) = 14.69, p = 0.001), but it showed neither an effect of feature (F(1,17) = 2.52, p = 0.13) nor an interaction of duration and feature (F < 1) (Figure 3B,C).

Table 1.

Regions localized in the group level SPM during the encoding phase.

Talairach coordinates of activation peaks for brain regions showing increased activity during the VWM encoding phase in the group-level random-effects SPM analysis (q(FDR) < 0.05). ACC, anterior cingulate cortex; FFG, fusiform gyrus; aIFS, anterior inferior frontal sulcus; IPS, intraparietal sulcus; MFG, middle frontal gyrus; MOG, middle occipital gyrus; VO, ventral occipital; IFJ, inferior frontal junction; PCG, precentral gyrus; STG, superior temporal gyrus; mPFC, medial prefrontal cortex; TPJ, temporoparietal junction

Regions	Peak Tal Coordinates
	x	y	z
Cingulate sulcus	−12	20	31
	12	20	28
ACC	0	14	43
FFG	−30	−49	−8
	36	−49	−5
aIFS	−49	35	10
	36	35	19
Insula	−27	17	10
	27	23	1
Internal capsule	−12	11	1
	12	11	1
IPS	−21	−58	46
	21	−58	46
Occipital	−33	−70	7
	27	−86	9
MOG	−27	−82	16
VO	−30	−79	−5
IFJ	−36	8	22
	39	2	34
PCG	−42	−1	43
	36	−1	43
Putamen	24	5	4
STG	50	−40	19
mPFC	0	8	55
Thalamus	3	−13	8
TPJ	−53	−38	24
	49	−36	28

Table 2.

Regions localized as candidate ROIs sensitive to encoding duration.

Statistical results for individually-defined ROIs showing an effect of stimulus presentation duration at encoding. The table denotes, for each ROI, the number of subjects for which an ROI could be individually defined (subject count), the Talairach coordinates for the mean peak-activated voxel for these subjects, the mean time-of-peak of activation in the encoding phase, and ANOVA F values for the Stimulus Feature and Stimulus Duration factors, and their interactions. Of all the candidate regions, only the left LPFC responded in a manner consistent with a hypothesized region sensitive to encoding duration, i.e., showing significant effects of duration, feature, and the resulting interaction. ACC, anterior cingulate cortex; FFG, fusiform gyrus; aInsula, anterior insula; IPS, intraparietal sulcus; MFG, middle frontal gyrus; MOG, middle occipital gyrus; IFJ, inferior frontal junction

Regions	Subject Count	Tal Coordinates			Mean Time-of-Peak (ms)				Peak Latency Effects (F scores)
					500		1,500		Duration	Feature	Interaction
		x	y	z	Color	Face	Color	Face
ACC	9	−9	21	30	6,556	7,556	7,889	8,222	6.86*	2.07	1.07
	9	6	18	32	6,600	6,800	8,300	7,700	27.16**	< 1	1.21
FFG	18	−34	−52	−9	6,889	7,167	7,833	8,167	31.32**	3.77	< 1
	18	34	−53	−7	7,056	7,389	7,667	8,000	14.69**	2.52	< 1
aInsula	14	31	21	5	6,308	6,846	6,923	7,538	4.70*	3.87	< 1
IPS	18	−25	−53	46	7,056	7,167	7,722	7,833	10.88**	< 1	< 1
	18	26	−53	48	6,778	7,222	7,611	8,222	18.20**	5.99*	< 1
MOG	16	−30	−82	10	6,625	6,500	7,563	7,313	31.96**	< 1	< 1
	14	30	−80	10	7,214	7,000	7,786	7,643	14.62**	< 1	< 1
IFJ	18	−38	8	26	7,000	7,500	7,222	8,278	4.94*	8.77*	5.12*
	18	38	9	25	6,500	7,167	7,111	7,944	13.96**	9.00*	< 1
MFG	14	−44	−1	39	7,143	6,929	7,857	8,000	20.47**	< 1	< 1
	12	40	0	40	6,667	7,417	7,333	8,250	7.62*	5.67*	< 1

^*p ≤ 0.05;

^**p ≤ 0.005.

Figure 3.

Group-level SPMs and time courses for the right FFG (top panels) and left IFJ (bottom panels) in the VWM encoding task. (A) SPM of right FFG (long blue arrow). (B) and (C), FFG ROI time courses. The first activation peak corresponds to the encoding (and perceptual) phase of the task, whereas the second peak corresponds to the probe/response phase. Note the similar peak activation latencies for the color and face conditions (denoted by small blue and red arrows, respectively) at both the 500-ms (B) and 1,500-ms (C) durations. (D) SPM of the left IFJ (long blue arrow). (E) and (F), IFJ ROI time courses. Note that the peak latency was delayed in the face condition relative to the color condition at the 1,500-ms stimulus duration (F), but not at the 500-ms (E) duration. See Table 2 and text for statistics. Green arrow, time of encoding array onset (time = 0 s); orange arrow, time of probe onset. Short, dark blue arrow, group mean time-of-peak for color feature; short red arrow, group mean time-of-peak for face feature condition.

Only one region showed a time course of activation that corresponded to the pattern expected of a brain region involved in VWM encoding. The left inferior frontal junction (IFJ, Figure 3D), in the posterior aspect of the lateral prefrontal cortex, showed not only peak latency effects for feature condition (F(1,17) = 8.77, p = 0.009), and stimulus duration (F(1,17) = 4.94, p = 0.040), but also a significant interaction (F(1,17) = 5.12, p = 0.037). Specifically, planned comparisons showed no effect of feature at the 500-ms duration (t(17) = 1.64, p = 0.12; Figure 3E), but there was an effect at the 1,500-ms duration (t(17) = 3.86, p = 0.001; Figure 3F). In contrast, the right IFJ and the left and right IPS showed duration and feature effects, but no interactions (Table 2).

The current experiment was designed to investigate latency effects during VWM encoding. To be sure, areas that showed the peak latency effects described above can be unambiguously associated with encoding, but this analysis does not distinguish between areas involved in only encoding from those involved in both WM encoding and maintenance, as both areas would show a peak latency effect (however, the analysis does rule out areas solely involved in maintenance, as those areas would have shown delayed onsets of activity with encoding load instead of peak latency differences). Because the slow-event design also permits the analysis of activity associated with the maintenance of information in VWM, we can ask whether the IFJ is activated only during VWM encoding, or whether it also contributes to the maintenance of information in VWM. To answer this question, maintenance load effects were tested within the color and face feature conditions using the BOLD response at 11-s post-stimulus array onset for the 500-ms duration and as 12-s post-stimulus array onset for the 15000ms duration as estimates of maintenance-related activity (the 1-s spread between the two duration conditions reflects the 1-s difference in the sample array duration.). Based on previous studies (Pessoa et al., 2002; Todd & Marois, 2004; Zarahn et al., 1997), the BOLD signal should be dominated by maintenance-related activity after 10 s of retention, though one cannot completely rule out minor contributions of encoding-related activity. This maintenance-related activity was compared to baseline activity, defined as the mean level of activity for the two volumes directly preceding the onset of the sample array (qualitatively similar results were found using the signal at the onset of the sample array, time = 0 s, as the baseline). For the IFJ, all feature-duration condition combinations were modulated, either significantly or marginally, above baseline (Face-500, t(17) = 2.05, p = 0.056; Face-1,500, t(17) = 4.21, p < 0.001; Color-500, t(17) = 2.75, p = 0.014; Color-1,500, t(17) = 2.01, p = 0.061; two-tailed). These data provide evidence for a role of the IFJ in VWM maintenance. Furthermore, the behavioral finding of a greater memory load in the 1,500-ms than 500-ms duration in the face condition was also replicated. Controlling for differences in baseline between the two conditions, activation in the 1,500-ms condition was greater than in the 500-ms condition (t(17) = 3.92, p = 0.001). The behavioral finding showing a null effect of duration on the memory capacity estimates in the color condition was also replicated neurally (t(17) = 0.93, p = 0.363). Thus, the same left IFJ region that is sensitive to encoding duration also appears to be involved in the storage of that information in VWM.

In contrast to the IFJ, the right FFG was not significantly modulated above baseline during the maintenance phase (Face-500, t(17) = 0.87, p = 0.396; Face-1,500, t(17) = 0.82, p = 0.422; Color-500, t(17) = 1.17, p = 0.276; Color-1,500, t(17) = 0.72, p = 0.484). The same pattern was observed for the left FFG (Face-500, t(17) = 0.003, p = 0.998; Face-1,500, t(17) = 1.68, p = 0.111; Color-500, t(17) = 0.65, p = 0.523; Color-1,500, t(17) = 0.07 p = 0.942). Also diverging from the IFJ’s behavior during maintenance, controlling for baseline differences between the two durations revealed no duration-sensitive difference in activity in either the color (right FFG: t(17) = 0.94, p = 0.363; left FFG: t(17) = 0.93, p = 0.366) or face (right FFG: t(17) = 1.50, p = 0.153; left FFG: t(17) = 1.64, p = 0.119) conditions. Thus, the FFG does not play a reliable role in helping to maintain encoded representations in WM.

5. Discussion

The goal of the present study was two-fold. First, it aimed at isolating the neural substrates of VWM encoding, un-confounded by activity associated with perceptual processing and VWM maintenance. At the same time, the study also aimed at determining whether the brain regions associated with VWM encoding correspond to those previously implicated in the attentional blink, as predicted by VWM encoding limitation accounts of the AB (Chun & Potter, 1995; Ouimet & Jolicœur, 2007). The results of our study suggest that the IFJ is preferentially involved in VWM encoding. Furthermore, because this is the same area that has been previously implicated in several AB studies (see below), our findings also support the WM encoding limitation account of the AB.

The experimental design of our study allows us to rule out several alternative accounts of the IFJ activation. By using time-resolved fMRI to identify brain regions that index the VWM encoding duration, we were able to dissociate brain regions involved in VWM encoding from perceptual brain regions, whose responses covary with the time of stimulus presentation, and from regions uniquely involved in VWM maintenance, which would not exhibit a time-to-peak sensitivity to the encoding duration of WM. Moreover, the luminance detection task required the subjects to continuously attend to the display during its presentation, thereby allowing us to rule out perceptual attention, or general attention or effort, as the process responsible for the IFJ activation. Finally, while the IFJ’s location is near Broca’s area, there are several reasons why the recruitment of this brain region in the present VWM encoding task is unlikely to reflect verbal WM contributions to encoding. First, subjects were instructed to perform an articulatory suppression task concurrent with the visual WM task, and their rehearsal was monitored during the training phase and feedback was provided if they failed to properly rehearse. Second, the anatomical focus of the IFJ in our study (intersection of Brodmann’s areas 9 & 44) is inconsistent with the location of Broca’s area in verbal WM studies (Brodmann’s areas 44 & 45) (Grodzinsky & Santi, 2008; Brass, Derrfuss, Forstmann, & von Cramon, 2005). We therefore conclude that the present IFJ activation reflects a neural signature of the encoding of information in visual WM, a signature that is not feature-specific as it was robustly activated by the encoding of both faces and color stimuli.

The conclusion that the IFJ is involved in VWM encoding is consistent with WM studies showing that this region’s BOLD response amplitude reflects the selection and transfer of visual information into WM (Courtney et al., 1997; Roth et al., 2006). This conclusion is also consistent with evidence that the IFJ is involved in selecting targets to encode into WM (Derrfuss, Brass, & von Cramon, 2004; Petrides, 1994, 1996; Roth et al., 2006). Moreover, the IFJ is engaged during the selection of task -appropriate responses to sensory stimulations (e.g. Marois et al., 2006; Dux et al., 2006, 2009), and in rule retrieval (Brass & von Cramon, 2004; Bunge, Kahn, Wallis, Miller, & Wagner, 2003; Bunge, 2004), suggesting that this brain region may have a role in both encoding of information into, and retrieval of information from, WM.

Analysis of the activation time course at the end of the delay period suggests that the IFJ is not only involved in encoding information into WM, but also in maintaining it. Like other prefrontal areas, the IFJ most likely supports WM maintenance by participating in manipulation and organization processes necessary to maintain the fidelity and accessibility of the stored representations in WM (Curtis & D’Esposito, 2003). That the same region indexing WM encoding duration is also involved in maintaining this information suggests that encoding and maintenance are not completely dissociable processes. At face value, this conclusion does not seem to support behavioral studies positing that VWM encoding and maintenance are dissociable processes (e.g., Woodman & Vogel, 2005). A more cautious interpretation of the current findings is that, although the same region is recruited in encoding and maintenance, different neural subpopulations may contribute uniquely, or at least differently, to each WM phase. If there is some partitioning of neural resources between these phases, this would provide neural support for behavioral evidence of a dissociation between these two WM phases (e.g., Woodman & Vogel, 2005). It is also of course possible, if not likely, that there are other brain regions, which went undetected in the present study, that may specifically contribute to VWM encoding or maintenance (see also below). Evidently, more work is needed to definitively address the issue of the neural association/dissociation between WM encoding and maintenance.

It should not be concluded from the present study that only the IFJ contributes to VWM encoding. The experimental approach adopted here was far more conservative than previous VWM encoding studies (Courtney et al., 1997; D’Esposito, Postle, Jonides, Smith, 1999; Druzgal & D’Esposito, 2003; Haxby, Petit, Ungerleider, & Courtney, 2000; Motes & Rypma, 2009; Munk et al., 2002; Pessoa et al., 2002), and therefore was not the most sensitive method to detect all encoding-related activity. While previous studies used activation amplitude as a measure of encoding activity, we used activation latency. Moreover, the latency measurements had to be distinct from those associated with mere duration of stimulus presentation (and perceptual attention) in order to be considered VWM-specific. Thus, while our present results strongly implicate the IFJ in VWM encoding, they cannot rule out the possibility that other brain regions also participate in VWM encoding.

In addition to the IFJ, candidate brain regions for VWM encoding are likely to be found among the many that showed sensitivity to the presentation duration of the sample array. Indeed, several of these areas have been observed in amplitude-based studies of VWM encoding (e.g. Courtney et al., 1997; Munk et al., 2002; Pessoa et al., 2002). Among these regions, it is worth mentioning the visual cortex areas, specifically the MOG and FFG ROIs, as these regions may not be specific to the presentation duration of the stimulus array, as there is some evidence that they may also be sensitive to the encoding and accumulation of task-relevant information into WM (Ploran et al., 2007; see also Miller, Deouell, Dam, Knight, & D’Esposito, 2008). The location of the FFG ROI is similar to that of the fusiform face area (FFA), which is sensitive to the processing of faces (Kanwisher, McDermott, & Chun, 1997; McCarthy, Puce, Gore, & Allison, 1997) and shows increased activation during the encoding of faces into memory (Haxby et al, 1996; Ranganath, DeGutis, & D’Esposito, 2004). The MOG ROI is close to another region frequently associated with face processing, the occipital face area (OFA) (Gauthier, Skudlarski, Gore, & Anderson, 2000). In addition, the FFG and MOG are close to a color-selective occipital region, V4/V8, lying adjacent to the FFA (Corbetta, Miezin, Dobmeyer, Shulman, & Petersen, 1991; Zeki et al., 1991), which may account for the color-related activation observed in the present study. However, because we did not perform functional localizer tasks to individually define the FFA/OFA and V4/V8 regions, it remains unclear whether the visual cortex activation here is indiscriminate to the particular stimulus feature.

The IPS is another area that showed sensitivity to the duration of stimulus presentation, and that may ultimately be shown to play a role in VWM encoding as well. Previous research indicates that response amplitude changes during encoding in this brain region are correlated with behavioral performance (Pessoa et al., 2002; Todd & Marois, 2005). The sensitivity of the IPS to the duration of the sample array presentation observed in the present experiment might reflect the sustained deployment of attention to the stimuli during array presentation (Colby, Duhamel, & Goldberg, 1996; Kastner & Ungerleider, 2000), and this attentional modulation could facilitate encoding by increasing the salience of targets in the visual scene (Gottlieb, Kusunoki, & Goldberg 1998; Bisley & Goldberg, 2003).

5.1. VWM encoding limitations and the Attentional Blink

In our VWM task, the IFJ’s response profile likely reflects the time-consuming encoding of the task-relevant features (Braver et al., 2001). The sensitivity of this brain region to the duration of WM encoding, which cannot be accounted for by perceptual processing or WM maintenance, and which generalizes across different featural information, is consistent with a capacity-limited model of WM encoding operating as a central bottleneck of information processing (Jolicœur & Dell’Acqua, 1998). Such a bottleneck has been postulated to account for the deficit in conscious perception of multiple targets in the AB paradigm (Chun & Potter, 1995; Jolicœur, 1998). The present study provides neural support for the VWM encoding limitation of the AB because the IFJ area isolated in the present study (range of peak-activated voxel coordinates, x: −48 to −30, y: −4 to +32, z: +16 to +37) corresponds very well to the prefrontal region previously observed in neuroimaging studies of the AB (lateral prefrontal cortex peak coordinates (x, y, z) in Kranczioch et al., 2005: −43, −1, +39; in Marois et al. 2004: −48, +8, +35). Taken together, these studies suggest that brain regions typically associated with the AB may also be the regions placing capacity limitations on the rate of encoding information into WM. Of course, we cannot rule out the possibility that WM encoding and the AB activate different neural populations of these brain regions, and a more definitive answer to this issue will await the use of multi-voxel pattern analysis (Haynes & Rees, 2005; Kamitani & Tong, 2005).

It is worth emphasizing that a neural overlap between brain regions involved in WM encoding and in the AB is consistent with behavioral findings showing that it is the deployment of selective attention, and not VWM encoding, that is responsible for the AB (DiLollo et al., 2005; Olivers et al., 2007, see also Nieuwenstein & Potter, 2006). Dux & Marois (2009) proposed a reconciliation of these accounts by positing that it is selective attention to WM encoding that gives rise to this deficit. Consistent with this unifying account of the AB, the IFJ area is not only engaged by WM encoding and by the AB paradigm, but it has also been shown to exert a central role in the control of selective attention (Asplund, Todd, Snyder, & Marois, 2010). Evidently, the tight relationship between attention and WM is not solely restricted to the maintenance of information in working memory, but may extend to WM encoding as well.