Common and distinct neural regions for the guidance of selection by visuoverbal information held in memory: Converging evidence from fMRI and rTMS

Abstract

Recent research indicates that working memory (WM) and attention interact, with attention automatically biased to stimuli that match the contents of WM. Though there is behavioral evidence for verbal guidance (written words) as well as guidance by more visual cues in WM, we have limited understanding of how these two representational formats influence the guidance of visual selection at a neural level. Here, we present converging evidence from functional MRI and transcranial magnetic stimulation (TMS), which indicates that both common and distinct neural regions mediate the influence of visuoverbal representations on WM. Colored shapes, but not words, in WM activated the superior frontal gyrus (SFG) and recognition memory areas in the temporal lobe when the contents of WM matched a stimulus in a subsequent search display. rTMS to the SFG disrupted WM effects from colored shapes. The lateral occipital cortex, however, tended to be more activated with written word cues, and rTMS to the lateral occipital complex tended to disrupt effects from written words more than from colored shapes in WM. There was also evidence for cue validity effects from colored shapes and written stimuli operating through different subthalamic nuclei. We discuss the evidence for understanding the neural systems mediating attention effects from WM. Hum Brain Mapp, 2012. © 2011 Wiley Periodicals, Inc.

INTRODUCTION

The mechanisms of attention and working memory (WM) work in a closely orchestrated way [Soto et al., 2008, for a review]. Behaviorally, the focus of attention can be automatically attracted to the space occupied by stimuli matching the current contents of WM [Downing, 2000; Olivers et al., 2006; Soto and Humphreys, 2006; Soto et al., 2005, 2006, 2007; though see Downing and Dodds, 2004; Woodman and Luck, 2007, for an alternative view]. Neural accounts of attention, including the influential biased competition model [Desimone and Duncan, 1995], suggest that WM influences attention through re‐entrant activation of visual pathways [Chelazzi et al., 1993]. To date the neurophysiological and neuroimaging studies on WM‐based guidance of attention have only examined effects from visual WM, when WM items were visually identical to the items presented in the search array. Nevertheless, recent behavioral work demonstrates that visual attention can be driven through more verbal relationships between WM and the visual array. For example, semantic associations between the representation of a target in WM (e.g., a motorbike) and a distractor in the search display (“helmet”) can attract attention to the distractor [Belke et al., 2008; Huang and Pashler, 2007; Moores et al., 2003]. Similarly, holding a word in WM can automatically draw attention to a matching item in a subsequent search task [Soto and Humphreys, 2007]. There is limited work dealing with how, at a neural level, verbal knowledge in WM can bias visual selection [though see Grecucci et al., 2010, for data on emotional stimuli], and current models [such as biased competition; Desimone and Duncan, 1995] are silent on this topic.

This study aims to assess the relations between the neural structures supporting attentional guidance from representations in WM, which can be of a more visual or verbal nature. The work is built around a procedure used by Soto et al. [2005] to assess automatic interactions between WM and attention. Soto and Humphreys [2007] presented observers with a feature cue to be held in WM throughout a trial, either in the form of a visual image or in the form of a verbal description presented visually (i.e., “Red”). Subsequently, a search array appeared. This was composed of several lines, each appearing within a different colored shape, with the task being to identify the orientation of a tilted line target that was presented among vertical distracters. On invalid trials, the colored shape held in WM reappeared in the search display surrounding a vertical distracter. On valid trials, the WM stimulus reappeared surrounding the search target (i.e., the tilted line). On neutral trials, the WM stimulus did not appear in the search displays. A memory test followed each search trial to ensure that participants were holding the cues in memory. Relative to the neutral baseline, search performance was slower in the invalid condition and faster in the valid condition (i.e., there was a WM‐cue validity effect), indicating that attention was biased to a stimulus matching the content of WM. This attentional bias is also evident on oculomotor measures [e.g., the probability of directing a first saccade to the search target; Soto et al., 2005], and it happens even under experimental conditions that used just invalid and neutral cues [so the WM item never validly cues the search target; Soto et al., 2005]. One interpretation for the similar behavioral effects on selection that emerge from words and colored shape cues in WM is that participants spontaneously transform one stimulus to the other—for example, generating a visual image from the verbal description [though see Soto and Humphreys, 2007, 2008, for arguments against this]. According to this translation account, the guidance of selection from verbal information in WM should rely on similar neural processes to those that guide attention from visual WM.

The Neural Basis of Cueing From Irrelevant Items in WM

The neural substrates of visual guidance from WM were examined in a functional MRI (fMRI) study by Soto et al. [2007]. Two patterns of neural response were evident. First, there was an effect of re‐presenting the cue in the search display, which occurred for valid and invalid trials alike [the cue reappearance effect; see also Grecucci et al., 2010]. The cue reappearance effect was seen in regions of the frontal [including the superior frontal gyrus (SFG)] and middle temporal cortices. In contrast, dorsolateral, orbitofrontal cortex and the thalamus were sensitive to the validity of the WM cue. These last regions were more activated when the search target appeared within the WM cue (on valid relative to neutral trials), and their response decreased when a distracter fell within the WM cue (on invalid trials compared with neutral).

This prior work on attentional guidance from WM used fMRI to delineate the neural correlates of performance [Grecucci et al., 2010; Soto et al., 2007]. One limitation of this is that the causal roles of any activated structures in attentional guidance cannot be inferred. Arguments for a causal role of neural activation require interventions that change neural states. Here, we present evidence from an fMRI‐guided repetitive transcranial magnetic stimulation (rTMS) study to demonstrate the common and distinctive neural processes that support attentional guidance from colored shapes and words initially presented for WM. One possibility is that attentional guidance from verbal WM is supported by a neural system independent of the circuits that implement visual guidance. This would fit models advocating the compartmentalization of WM into distinct subsystems including the verbal phonological loop, the visuospatial sketchpad, and more abstract “episodic” representations that integrate information from the different modality‐specific systems [see Baddeley, 1986, 1992, 2003]. Effects from visual WM may be based on feedback from dorsolateral and SFG (including the frontal eye fields), which biases activation in striate and extrastriate cortex [Chelazzi et al., 1993; Motter, 1993; Ruff et al., 2006]. In contrast, attentional effects from verbal WM may operate in networks involved in semantic and categorical processing of perceptual input, including the left inferior frontal gyrus [Gabrieli et al., 1998; Gold and Buckner, 2002] and the left occipital–temporal cortex, BA37 [Büchel et al., 1998; Goldberg et al., 2006; Grecucci et al., 2010; Martin et al., 1995]. These different possibilities were examined here by assessing the effects of rTMS on WM‐based guidance of attention, focusing on the SFG and the lateral occipital complex in the left hemisphere—both of which are activated in a first study examining WM and attention using fMRI. By applying TMS to these areas, we show for the first time that they play a causal role in WM‐based guidance of attention, with differential effects emerging, respectively, for more‐visual and more‐verbal cues (colored shapes and written words).

STUDY 1: NEURAL CORRELATES OF ATTENTIONAL GUIDANCE BY MORE VISUAL AND MORE VERBAL REPRESENTATIONS IN WM: AN fMRI STUDY

Methods

Participants

Ten right‐handed healthy volunteers (five females), who were unaware of the purpose of the experiment, participated. They were aged between 19 and 26 years, and their vision was normal or corrected to normal. No participant had a prior history of neurological or neuropsychiatric disorders, and all had normal or corrected‐to‐normal vision. The study conformed to the Declaration of Helsinki and approved by the local ethics committee. All participants provided written informed consent and received a payment of £50 for their participation in the fMRI and the rTMS experiments.

Design and procedure

A within‐subjects full‐factorial design was used with the following factors: WM‐cue type (written word and visual form) and WM‐cue validity (valid, neutral, and invalid). Cue validity defined the relation between the WM item and the target item in the search task, while WM‐cue type defines the form of the WM stimulus, presented at the study and memory‐test phases. Note that the cue type only manipulated the form in which the information was presented and not the actual semantic content of the WM stimulus (see next paragraph for details). The task was programmed and run using E‐Prime [Psychology Software Tools, 2002].

Figure 1 illustrates the sequence of events in different conditions. Each trial began with the presentation of a fixation cross for 500 ms to alert the participants of the beginning of a trial. This was followed by a blank screen for another 500 ms, and then the memory cue appeared at fixation for 928 ms (comprising the study phase). The cue was either a colored shape (visual cue) or a written verbal description of a colored shape (i.e., “Red Square”—the verbal cue). The cue was followed by a blank screen for 188 ms and then the target display for 100–183 ms (see below) and a random dot mask for 1,500 ms. The target display was composed of four differently colored shapes displayed at the corners of an imaginary square centered at fixation, with an eccentricity of 5.73° of visual angle. Visual objects were simple geometrical shapes, a 1.43° × 1.43° square, a 2° × 1.91° diamond, a 1.62° × 1.81° triangle, a 1.81° × 1.81° circle, and a 2.38° × 1.3° hexagon, which were drawn in five possible colors (i.e., red, green, blue, yellow, and pink). Each object contained a line of 0.573° length. Three of the lines were vertical, and one, the target, was tilted 16° to the left or to the right. The task was to identify the orientation of the target line by pressing one of two buttons (one for left and the other for right responses). Observers were required to be as fast and as accurate as possible within the limited time window of 1,500 s (when the mask was present). A memory test followed 500 ms after completion of the line orientation task. On visual WM trials, the memory test comprised a colored object, and the observers had to indicate whether it was identical or not to the memory item (based on the conjunction of shape and color). In the verbal condition, the memory test consisted of a written verbal description of the colored shapes, and observers had to indicate whether this was identical to the initial display (for both the named color and shape). Participants responded “same” in the memory test if dimensions of both the cue and probe object matched; otherwise, they responded “different.” On “different” trials, the cue and test items could have just their color in common, just their shape in common, or neither attribute in common. Responses in the memory task were made by pressing one key for “same” and one for “different” stimuli. The total length of a trial was ∼4.75 s from the study phase with the WM cue to the memory test phase. The trial onset was jittered by varying the intertrial interval (range 2–6 s). This enabled the estimation of the hemodynamic responses for the individual events.

Figure 1.

Illustration of the experimental display sequences in the different conditions. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

The observer was first familiarized with the line discrimination task and carried out training trials with different time exposures. The training took place in the Birminghan University Imaging Centre (BUIC) mock scanner using projection devices and setup similar to the MRI scanner. During the training blocks, the target duration was initially set to 1,000 ms; it was subsequently reduced progressively across blocks of trials, i.e., 500, 250, and 183 ms, and then further diminished in 17‐ms steps until performance was about 75% correct during the initial 10 trials of a block (accuracy feedback was provided on each trial). The target display time ranged from 100 to 183 ms, depending on the participant. This presentation time was then used in all the experimental sessions for a given participant (for the fMRI and the three rTMS sessions). We used this performance criterion to avoid ceiling effects. Note that presentation times were too short to make an eye movement, and observers were explicitly trained to keep central fixation through a trial. Observers received several training trials in the combined WM and attention tasks, prior to the experimental session. Feedback was not provided during the experimental blocks. On a third of the trials, the cue was valid and contained the target; on another third, the cue was neutral; and on the remaining third, it was invalid, containing a distracter. Observers were instructed about these contingencies. There were 240 trials (40 per condition) in the fMRI study, which was run in five separate sessions with eight trials per condition in each session.

Data acquisition

We used a Phillips 3T Achieva system to acquire blood oxygenated level–dependent (BOLD) contrast‐weighted echoplanar images (EPI) during the functional scans. Thirty seven oblique slices, with 2‐mm thickness and 1 mm gap, were acquired, resulting in an in‐plane resolution of 3 × 3 mm², with 80° flip angle, 35‐ms echo time, and 2,020‐ms slice repetition time. Images were acquired using an eight‐channel phase array coil with a sense factor of 2. To minimize susceptibility artifacts, slices were tilted 30° along the frontal–temporal cortex [Deichmann et al., 2003]. The fMRI experiment was run in five sessions that were concatenated for the analysis. In addition, high‐resolution (1 × 1 × 1 mm³), T1‐weighted images were acquired (with 8° flip angle, 3.8‐ms echo time, and 8.4‐ms slice repetition time) using an eight‐channel phase array coil with a sense factor of 2. The T1 images were used to guide the rTMS stimulation (Methods Study 2).

Data analysis

The data were analyzed using SPM5 (Wellcome Department of Imaging Neuroscience, London; http://www.fil.ion.ucl.ac.uk/spm). EPI volumes were spatially realigned and unwarped to correct for movement artifacts [Ashburner and Friston, 2003] and movement by distortions interactions [Andersson et al., 2001], transformed to the Montreal Neurological Institute (MNI) standard space, and smoothed using a 9‐mm Gaussian kernel to account for residual intersubject differences.

Statistical voxel‐based analysis was performed in two steps. First, for each subject, we estimated the effect size on each condition averaged across the five sessions. To do this, we used a model (first‐level model) that included a regressor of the onsets of the WM cue on each correct response trial across the six different experimental conditions, following our 2 (cue type: verbal and visual) × 3 (validity: valid, neutral, and invalid) design. Incorrect trials were modeled separately. Note that the main aim of the study was to test for an interaction between WM and visual selection. Hence, the critical period of the trial was in its first 1.5 s corresponding to the joint presentation of the WM cue and the search array. Further, for each of the above regressors, we included the reaction time (RT) of each event as a covariate to control for RT differences. All the regressors were convolved with the canonical hemodynamic response function. To correct for residual signal changes due to head movement after the unwrap procedure, the six realignment parameters were included in the design matrix. An additional set of harmonic regressors was used to account for any low‐pass frequency variance within the data along time with a cutoff of 1/128 Hz.

Consistent effects across subjects (random‐effects and second‐level analysis) were then tested using ANOVA with a 2 (cue type) × 3 (validity) within‐subject factor. In the model, we assumed neither independency nor equal variance between the conditions. We used conjunction analyses [Friston et al., 1999] to identify common regions involved in more verbal (written word) and more visual (colored shape) WM guidance of attention. Dissociated neural structures were identified based on the pattern of the simple effects using exclusive masking; for example, we identified regions that showed a validity effect for visual WM but not (P > 0.05, uncorrected) for the more verbal condition; these regions were also tested for significant cue × validity interaction, which is reported in the tables. For cortical structures, we report clusters at P < 0.001 (uncorrected unless specified otherwise), which were larger than 45 mm³.

fMRI: Results and Discussion

Behavioral data

The proportions of correct responses in the WM and search tasks were 0.95 and 0.76, respectively. Note that search display durations were selected to comply with a level of search accuracy of about 75% (see Methods Section of Study 1). In the subsequent analyses, performance in the search task was assessed only on those trials with correct WM responses. A 3 (validity) × 2 (cue type) repeated‐measures ANOVA on search RTs showed a significant validity effect [F(2,18) = 5.77, P < 0.018], with faster responses on valid compared to invalid trials. There was no interaction between cue validity and cue type [F(2,18) < 1, P > 0.1]. Figure 2 illustrates the pattern of results. We note that although performance benefits (RT valid − RT neutral) were numerically higher for colored shape stimuli in WM than for written words in WM, a paired t test showed that the size of the benefits did not differ between conditions [t(9) = 1.6, P = 0.32]. Likewise, the size of the performance costs (RT invalid − RT neutral) was similar across the different cueing conditions [t(9) = 1.15, P = 28].

Figure 2.

Behavioral data: Search RTs.

The accuracy data matched the RTs. There was a reliable validity effect [F(2,18) = 6.62, P < 0.009], with more accurate target identification on valid compared to invalid trials. Cue validity did not interact with cue type [F(2,18) < 1, P > 0.1], and no differences between the size of the performance benefits across cueing conditions were apparent (see Supporting Information Fig. 1). The data suggest that colored shape and written cues were equally effective in modulating attention in search. These behavioral results replicate previous findings [Soto and Humphreys, 2007].

fMRI data

WM‐cue type effects

We first delineated brain regions that showed a differential response to the two types of WM cue, irrespective of cue reappearance or validity (Table I, Fig. 3). Early visual regions within the posterior and middle occipital cortex responded more on trials with written compared with colored shape cues in WM. Similarly, the left fusiform gyrus (FFG), potentially corresponding to the visual word form region [Fiez and Petersen, 1998], showed a larger response under conditions of word rather than visual cues. In contrast, colored shape cues activated the more lateral dorsal banks of the occipital cortex and the right posterior FFG, when compared with word cues. These results suggest that the written cues were more demanding of early visual processes compared with the colored shape cues. However, this had no effect on the extent of the selection bias from WM, as the behavioral data showed no differential validity effects for the two cue types.

Table I.

Cue‐type effects

Anatomy	BA	Z	MNI (x, y, z)
A. Verbal cue > visual cue
IOG
R	17, 18	6.29*	24, −96, 2
L	17, 18	5.73*	−16, −98, 4
FFG (L)	37	3.38	−38, −44, −24
B. Visual cue > verbal cue
MOG
R	19	3.24	38, −78, 6
L	19	3.41a	−42, −72, 6
pFFG (R)	19	3.14a	30, −66, −14
Cerebellum		3.37	−12, −40, −16

Anatomical labels are based on the Duvernoy Human Brain Atlas.

All clusters of activation reported were threshold at P < 0.001, uncorrected. Cluster sizes are larger than 45 mm³.

BA, Broadman areas, based on the MRICROn; IOG, inferior occipital gyrus; pFFG, posterior fusiform gyrus; MOG, middle occipital gyrus.

^* P < 0.05, FEW‐corrected.

^aCluster size larger than 10 mm³.

Figure 3.

The effect of cue type (verbal vs. visual) on the fMRI data. SPMs overlaid on a canonical T1 image. In red are regions showing an increased response in the visual cue condition compared with the verbal cue condition. In green are regions showing an increased response in the verbal cue condition compared with the visual cue condition. The graphs depict the responses (estimated effect size) of three regions across the different conditions. The coordinates in MNI space and the anatomical labels for the regions are written above the graphs. L, left; R, right; MOG, media occipital gyrus; FFG, fusiform gyrus; IOG, inferior occipital gyrus. Error bars are standard error of the means. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

WM‐cue reappearance effects

Next, we asked which neural systems mediate the attentional effects from WM. We first tested for brain regions that responded to the reappearance of the cue in the search display [i.e., (valid + invalid)/2 versus neutral]. This comparison was computed separately for each cue type and conjointly, to assess dissociable and common components (Fig. 4 and Table II). There were no areas that reliably increased activity for both types of cue but some regions did decrease their activity for re‐presentation of both types of cue (notably the right SFG). Cue‐specific decreases were observed with word cues in the left insula and with colored shape cues in the left temporal gyrus. The pattern of decreased activation is consistent with repetition suppression effects for regions associated with recognition memory. In line with the interpretation proposed by Soto et al. [2007], the current results suggest that the right SFG encodes passive memory traces from different types of visual events and is sensitive to repetition suppression [see also Kristjánsson et al., 2007].

Figure 4.

Cue repetition effects in the fMRI data. SPMs overlaid on a canonical T1 image. In red are regions showing increased responses to the repetition of the cue in the visual condition (but not verbal). In green are regions showing increased responses to the reappearance of the cue in the verbal condition (but not visual). Differential effects across cue types were tested using exclusive masking (see Methods section of Study 1). The graphs depict the responses (estimated effect size) of the regions across the different conditions. The coordinates in MNI space and the anatomic labels for the regions are written above the graphs. L, left; R, right; SFG, superior frontal gyrus; IFG, inferior frontal gyrus; pCG, posterior cingulate gyrus. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Table II.

Cue repetition effects: (valid + invalid)/2 versus baseline

Anatomy		BA	Z (min‐t 9)	MNI (x, y, z)
A. Conjunction visual and verbal
Reappearance increases
No above threshold voxels
Reappearance decreases
mSFG (R)		32	3.92 (2.90)	10, 46, 32
SFG (R)		6	3.84 (2.49)	22, 6, 36
Anatomy	BA	Z	F(2,18)	MNI (x, y, z)
B. Verbal (but not visual P > 0.05)
Reappearance increases
Post‐CG	26	3.08	8.25	−8, −44, 22
Reappearance decreases
mSFG (R)	9	3.53	6.58	6, 56, 34
Insula (L)	48	3.32	11.5	−40, −16, 18
C. Visual (but not verbal P > 0.05)
Reappearance increases
Frontal
SFG
L	8	3.44	6.65	−14, 32, 56
L	9	3.07	7.72	−22, 54, 32
R	8	3.30	–	14, 34, 56
IFG (L)	45	2.85*	6.54	−50, 22, 2
Temporal
ITG
L	20	3.74	14.1	−54, −8, −34
R	38	2.99*	4.28	50, 10, −20
MTG
L	20	3.22	–	−58, −30, −10
R	21	3.05	6.73	66, −32, −2
PHG
L*	20	2.45*	6.06	−28, −20, −24
R**	20	2.25*	7.64	30, −26, −24
Parietal (parieto‐occi)	19	3.06	–	−14, −56, 32
Subcortical
SN		3.61	5.59	−12, −20, −18
Cerebellum		3.03	8.94	−12, −62, −32
Reappearance decreases
tTempG	41	3.50	5.85	−34, −40, 18

Index: Anatomical labels are based on the Duvernoy Human Brain Atlas.

Z statistic of simple effect in one modality, threshold at P < 0.001, uncorrected.

F(1,54) statistic of the interaction between modality and cue reappearance.

BA, Broadman areas, based on the MRICROn; min‐t ₉, the t statistic of the smallest effect of the two comparisons; post‐CG, posterior cingulate gyrus; IOG, inferior occipital gyrus; mSFG, middle superior frontal gyrus; IFG, inferior frontal gyrus; ITG, inferior temporal gyrus; MTL, middle temporal gyrus; PHG, parahipocampal gyrus; parieto‐occi, parieto‐occipital sulcus; SN, substantia nigra; tTempG, transverse temporal gyrus.

Cortical clusters sizes larger than 45 mm³ are reported, except at *P < 0.005, uncorrected, and **P < 0.01, uncorrected.

Cue‐specific increases in activation were also observed. Reappearance following a colored shape (but not a written) cue elicited larger responses compared to neutral trials in the left superior frontal cortices (SFCs; BA8) and inferior and middle temporal gyri; at a lower threshold, there were also increased responses in the parahippocampal gyrus (PHG). Increased responses to reappearance of a search item corresponding to a word cue were present in the left posterior cingulate gyrus. We suggest that these increases in activation are sensitive to the match between an active representation in WM and the search display.

WM‐cue validity effects

Subsequently, effects of cue validity were examined, looking for contrasting activation patterns across valid trials (when the WM cue was colocated with the search target) and invalid trials (when the WM cue appeared at a location different to the search target). Table III and Figure 5 presented the results of tests for common and dissociated effects from written and colored shape WM cues.

Table III.

Cue‐validity effects: valid > baseline > invalid

Anatomy	BA	Z (min‐t 9)	MNI
A. Conjunction visual and verbal
CG (R)	6, 23, 48	4.01 (2.63)	22, 22, 24
IPS (R)	7	3.22 (2.00)	28, −54, 62
Insula
L	48	3.65 (2.34)	−36, −22, 32
R	48	3.32 (2.08)	38, −22, 14
LOS
L	19	3.75 (2.42)	−32, −80, −6
L	37	3.49 (2.22)	−50, −56, −8
PHG (R)	20	3.45 (2.18)	34, −20, −12
Anatomy	BA	Z	F(2,18)	MNI (x, y, z)
B. Verbal (not visual P > 0.05)
LOS (L)	19	3.49	—	−40, −74, 2
MOG (L)	39	3.28	4.3	−48, −76, 18
pCS (R)	3	3.96	—	48, −26, 40
CG	24	4.02	8.48	8, 16, 28
FP
L*	10	2.28*	8.48	−14, 50, −2
R*	10	2.34*	6.73	32, 58, 6
Insula
R	48	3.26	10.86	46, −4, −4
L	48	3.21	4.68	−32, 8, 14
Pulvinar
L		3.93	7.62	−26, −34, −2
R		3.65	6.67	28, −24, 4
Hipp (L)		4.06	9.51	−26, −16, −14
Putamen (L)		3.60	5.73	−28, −8, 8
Caudate
L		3.93	7.11	−10, 20, −6
R		3.23	8.04	22, 22, −2
SN (R)		3.39	–	16, −18, −22
Cerebellum		3.69	6.63	−16, −26, −24
C. Visual (not verbal P > 0.05)
SFS (L)	6	3.14	5.91	−16, −8, 54
SFG (R)	8	3.05	5.73	12, 28, 56
STG (R)	41	2.94	–	50, −34, 14
Thalamus (R)		3.24	4.25	34, −16, −12

All clusters of activation reported were threshold at P < 0.001, uncorrected.

Index: Anatomical labels are based on the Duvernoy Human Brain Atlas.

Z statistic of simple effect in one modality, threshold at P < 0.001, uncorrected.

F(1,54) statistic of the interaction between modality and cue reappearance.

BA, Broadman areas, based on the MRICROn; min‐t ₉, the t statistic of the smallest effect of the two comparisons; CG, cingulate gyrus; IPS, intraparietal sulcus; LOS, lateral occipital sulcus; MOS, middle occipital sulcus; pCS, posterior cingulate sulcus; FP, frontopolar; Hipp, hippocampus; SN, substantial nigra; SFS, superior frontal sulcus; SFG, middle superior frontal gyrus; STG, superior temporal gyrus.

Cortical cluster sizes larger than 45 mm³ except the one marked with *P < 0.005, uncorrected.

Figure 5.

Cue validity effects in the fMRI data. In yellow are regions showing increased responses on valid compared to invalid trials in both visual and verbal conditions. In red are regions showing increased responses to valid trials relative to invalid trials in the visual condition (but not verbal). In green are regions showing cue validity effects in the verbal condition (but not visual). Differential effects across cue types were tested using exclusive masking (see Methods). The graphs depict the responses (estimated effect size) of four regions across the different conditions. The coordinates in MNI space and the anatomical labels for the regions are written above the graphs. L, left; R, right; mCG ‐ middle cingulate gyrus; LOC ‐ lateral occipital sulcus. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

A pattern of increased responses on valid relative to invalid trials, common to colored shape and written cues, was observed in the middle cingulate, the intraparietal sulcus, the insula, and the posterior occipital cortices. Dissociable mechanisms were also observed. The validity of colored shape cues modulated responses of the left SFCs (but inferior to the region showing a cue‐repetition effect) and the thalamus while the validity of written cues affected responses in the bilateral pulvinar, the lateral occipital cortex (LOC; including the lateral occipital sulcus and middle occipital gyrus), and, at a lower threshold, bilateral frontal polar cortex (BA10), among other regions. We note that in the left lateral occipital sulcus, there was no reliable interaction of modality and cue, though the validity effect tended to be more substantial for written than for colored shape cues. However, a reliable interaction was observed in the neighboring middle occipital gyrus (Table III). Finally, similar to previous results [Grecucci et al., 2010; Soto et al., 2007], we did not observe any above‐threshold increase in responses for invalid relative to valid trials.

Finally, we assessed the time course of the BOLD response in the SFG and LOC ROIs across the different conditions. These results can be found in the Supporting Information Analyses and Supporting Information Figure 2.

Overall, the data demonstrate both behavioral and fMRI effects from the cues in WM. Behaviorally, the validity effects did not differ for colored shape and written word cues. The fMRI data suggested that cue repetition (regardless of the validity of the cue for search) led to both decreased and increased neural responding in different areas. Most notably, decreased responses were found across both cue types in the right SFG, which we link to passive activation by the search display of a memory trace from the cue, which facilitates processing. Regions showing enhancement to the repetition of the cue in the display showed a more differentiated response according to the modality of the cue. For example, there was increased activity in the left SFG and IFG for colored shape cues, along with increased activation bilaterally in regions of temporal and parahippocampal cortex. With words, there was a similar increase in the left postcentral gyrus.

Effects of cue validity on neural responses (i.e., increased activation on valid relative to invalid trials) were present in a number of regions, including the LOC as well as the cingulate gyrus, the insula, and the IPS. Of particular relevance to Study 2 here, effects associated with the LOC tended to be stronger for written word cues (though there were also some effects from colored shape cues). There were also validity effects specific to colored shape cues, most clearly in the left SFG (but anterior to the effects of cue reappearance). There was also some evidence for modality‐based validity effects in different thalamic nuclei.

The general pattern of these results follows that reported by Soto et al. [2007] and Grecucci et al. [2010]. We note two changes. One is that validity effects in the IPS have not been previously reported; the second is that there was previously stronger evidence for a validity effect in the anterior prefrontal region BA10. Our visual stimuli were similar to those used by Soto et al. [2007], and any differences with these items likely reflect a change in the presentation conditions. We used brief exposures of the search display here while Soto et al. [2007] had long exposures. With long but not short exposures, participants may check that they are indeed responding to the search target and not the cue, a process we previously linked to area BA10 [Soto et al., 2007]. With short exposures, this effect may be weakened and, along with it, the effects of validity in this brain area. In addition, effects reflecting the initial engagement of attention may be more apparent with the short exposures used here, which may lead to validity affecting activation of the IPS if there is an initial effect of cueing spatial attention. However, given that cueing from WM can occur in patients with damage to posterior parietal cortex and impaired spatial orienting [Soto and Humphreys, 2006], this activity does not appear to be necessary for generating attentional cueing from WM.

STUDY 2: MODULATION OF WM GUIDANCE OF ATTENTION FOLLOWING rTMS

fMRI data only provide correlational evidence for distinct effects of cue reappearance and cue validity, along with evidence for modality‐specific components. To address whether activation of the observed neural regions were necessary to generate the behavioral interaction between WM and attention, we conducted a subsequent rTMS experiment where we sought to disrupt activation at critical sites. There were two main questions. First, would disruption of activity in areas sensitive to cue reappearance (but not cue validity) modulate the cue validity effect on behavior? This would provide the first evidence for brain regions responding to cue reappearance also playing a key role in the validity effects. To test this, rTMS was applied to the left SFG where effects of cue reappearance were observed. A similar rationale was used to test the behavioral consequences of interfering with regions differentially activated under the different cue validity conditions—here, we applied rTMS to the left LOC. Second, would rTMS to areas showing modality‐specific activation profiles modulate the validity effects in just one or both modalities? A modality‐specific effect would suggest that activation is not simply stronger in one modality, but rather that there are different neural regions supporting effects for each cue type. This was evaluated by applying rTMS to left SFG and the left LOC, where differential BOLD activity was found, respectively, for colored shape and written word cues.

Methods

Participants

Eight (four females) of the participants in the fMRI study also took part in the rTMS experiment. None of the participants had a history of neurologic or neuropsychiatric disorder, and none had a relative who suffers from epilepsy. After the rTMS protocol was explained, all the participants signed an informed consent form. The study was approved by the local ethic committee.

Design and procedure

The experimental design was identical to the one used in the fMRI study. Importantly, the presentation duration of the search array was kept constant for each participant depending on his/her initial threshold (75% accuracy, see above). Small changes were also introduced to optimize the procedure for the rTMS protocol. The main constraint was that effects of rTMS were expected to wash out after about 7.5min (see below); consequently, memory for the WM cue was tested on only 10% of the trials in order to maximize the number of search trials post‐TMS. The intertrial interval jitter was removed making the average length of a trial ∼3 s. There were 144 trials (24 per condition) in each rTMS session (see below).

TMS protocol

Two target regions in the left hemisphere were chosen for the rTMS stimulation sites: the SFC and the LOC. SFC and LOC depicted the two patterns of neural response observed during the guidance of attention from WM, the WM‐cue reappearance effect and the WM‐cue validity effect, respectively. The responses in the SFG and the LOC also revealed selective responses to one type of WM cue (respectively to visual and verbal cues). The target regions were chosen based on the results of our previous studies [Grecucci et al., 2010; Soto et al., 2007] replicated in the current study (Study 1). Furthermore, given the rTMS protocol that was used (see below), in this section these regions were anatomically localized sufficiently far apart to reduce any effects of carryover stimulation from one region to the other.

The sites targeted by rTMS were localized for each subject individually based on their brain responses as measured by fMRI. Clusters of activity in the two target regions (left SFC and the LOC), reflecting cue appearance and then cue validity effects, were identified on a single subject basis guided by the group results and our previous observations [Grecucci et al., 2010; Soto et al., 2007]. For this purpose, we coregistered the fMRI data to the T1 anatomical images, keeping both in the subject's native space (non‐normalized images). We used an identical statistical model as described earlier to identify within each participant the foci of maximal response based on the areas previously identified at the group level. The functional foci in the left SFC were derived by examining the cue reappearance effect [i.e., ([1]Valid + [1]Invalid)/2 − [−2]Neutral] collapsed across the cue type, and the left LOC was identified using the validity effect (i.e., [1]Valid − [0]Neutral − [−1]Invalid). Note that these contrasts were not based on conjunction statistics and hence tested main effects independent of cue type. We used main effects of cue reappearance and validity to localize the rTMS sites because our main interest was to test the behavioral consequences of rTMS on WM guidance of (i) registering the reappearance of the cue and (ii) modulating the processing of the search target due to its match to the cue (the validity effect). However, having identified the regions to stimulate, we then examined whether there were differential effects of cue type (colored shape vs. written word) on the rTMS effects.

The clusters identified on individual MRIs were then used to guide the positioning of the TMS coil. Performance in the combined WM and search task was assessed after a 15‐min train of 1 Hz TMS applied at a fixed intensity (55% of the maximum stimulator output). Prior work has shown that this TMS protocol leads to inhibition of cortical excitability in the targeted area that outlasts the TMS train itself, at least for half of the train duration [Robertson et al., 2003], i.e., 7.5 min here. Performance was also tested after a sham TMS condition with the coil tilted 45° away from the scalp, with the lower end of one of the wings touching just below the inion. This sham protocol has been used before [Kosslyn et al., 1999] to control for nonspecific effects of the TMS train itself (i.e., tactile sensations on the scalp, arousal from the stimulation context), while ensuring that the brain is not stimulated.

We used a Polaris infrared tracking device to measure the position of the subject's head, and Brainsight software was used to coregister the subject's head with a T1 high‐resolution MRI scan. This procedure was followed at the beginning of each session. Each rTMS session (Sham, left LOC, and left SFC) was carried out in separate days, with at least 1 day in‐between to allow sufficient time for the rTMS effect to reverse. The order of the rTMS sessions was counterbalanced across participants. The behavioral data were analyzed using SPSS 15.00.

rTMS: Results and Discussion

The stimulated rTMS sites: for descriptive purposes, we transformed the results (and TMS sites) in native space for individual subjects into the MNI normalized space (Fig. 6A). The average fMRI Z‐statistic at the individual subject level for the effect of WM‐cue reappearance in the left SFC was 2.43 (±0.55 std), and the averaged location in MNI space (x, y, z) of the peak was −25 (±6 std), 20 (±13 std), 52 (±8 std), respectively. The average Z‐statistic for the effect of WM‐cue validity in the left LOC was 3.11 (±0.85 std), with an average peak in MNI space (x, y, z) at −49 (±15 std), −63 (±17 std), 1 (±10 std), respectively. These functionally defined clusters, corresponding to left SFC and LOC, were used to guide the rTMS protocol by means of neuronavigation (Brainsight). We used a paradigm similar to the one used in the fMRI study though memory was tested only on 10% of the trials (see Methods Section in Study 1). Accuracy on memory catch trials was good, averaging 92.3% across the different TMS conditions (LOC: 92.4%; SFG: 93.1%; Sham: 91.4%), which did not differ (F < 1, P = 0.761). The mean proportion of correct line orientation responses was 75.6% ± 1.9 standard error of mean (SEM) in the sham condition, 77.4% ± 2.2 SEM in the left SFC condition, and 76.6% ± 1.2 SEM in the left LOC condition. These differences were not reliable (F < 1, P = 0.673). This demonstrates that the rTMS did not affect overall accuracy in the memory and search tasks. There was also no effect of rTMS on the effects of cue validity on accuracy (F < 1, P = 0.814). Note that rTMS effects on accuracy measures of performance are scarce in the literature; rTMS effects are most evident in RTs [Walsh and Pascual‐Leone, 2003]. Accordingly, clear effects of rTMS were observed in the RT data (Fig. 6B).

Figure 6.

(A) Illustration of the different TMS sites across participants. (B) Search RTs as a function of cue validity and cue type, across the different TMS sites: (a) sham; (b) SFG; (c) LOC.

A 3 (cue validity) × 2 (cue type) × 3 (rTMS condition) repeated‐measures ANOVA was conducted on RTs for correct search trials. There was no main effect of the site of the rTMS [F(1,7) = 0.084, P = 0.78], indicating that any effects of TMS were specific to the experimental manipulation and could not be accounted by overall changes on visual or motor processing. There was also no main effect of cue type (P > 0.05). However, there was a main effect of validity [F(2,14) = 11.93, P < 0.003, and partial eta square _pη² = 0.63], with faster responses on Valid than Neutral trials and faster RTs on Neutral than Invalid trials (P < 0.05 for all comparisons). More importantly, however, we observed a significant three‐way interaction [F(4,28) = 7.126, P = 0.004, and _pη² = 0.5), indicating that the size of the validity effect varied across different rTMS and cue‐type conditions (Fig. 6B). In order to unravel the source of this interaction, we performed several additional analyses.

We computed a separate 2 (cue type) × 3 (cue validity) repeated‐measures ANOVA for each rTMS site. There was a reliable validity effect following sham rTMS [F(2,14) = 10.26, P = 0.002, and _pη² = 0.6) and following rTMS to the left SFC [F(2,14) = 4.35, P = 0.034, η² = 0.38]. This overall effect was eliminated following rTMS to the LOC [F(2,14) = 1.56, P = 0.24, and _pη² = 0.18]. For both the SFC and LOC stimulation, cue validity interacted with cue type [left SFC, F(2,14) = 4.21, P < 0.037, and _pη² = 0.37; left LOC, F(2,14) = 3.97, P < 0.043, and _pη² = 0.36]. This was not the case for sham TMS (P > 0.2). rTMS to left SFC eliminated the validity effect with visual cues [F(2,14) = 0.57, P = 0.57, and _pη² = 0.07] while verbal cueing effects remained [F(2,14) = 6.5, P = 0.01, and _pη² = 0.48]. In contrast, rTMS to the left LOC eliminated validity effects from written words in WM [F(2,14) = 0.72, P = 0.5, and _pη² = 0.09]; there was also a trend for an attenuated effect from colored shapes [F(2,14) = 3.02, P = 0.08, and _pη² = 0.3].

These rTMS findings show that rTMS to the left SFC was primarily effective in reducing cueing from a colored shape in WM. This is consistent with the left SFC playing a particular role in directing attention from visual WM. The results after left LOC were also differentially affected by the modality of the cue, with the effects being greater after written cues than colored shape cues, though the data suggested validity effects in both cases.

In summary, the rTMS results indicated that the left SFC and the left LOC play a causal role in guiding attention from the contents of WM, with validity effects reducing after stimulation to both sites relative to sham TMS. As for the fMRI data, there was also some evidence for modality‐specific effects. rTMS to the left SFC modulated validity effects following colored shape cues. rTMS to the left LOC reduced the validity effects after both types of cue, with the effects tending to be strongest following written relative to colored shape cues. There were both reduced benefits and costs from cues. It is noteworthy that the effects of rTMS here were tightly related to the experimental manipulations, while overall RTs and accuracy were not affected. This rules out effects from disruption solely to search or to memory.

GENERAL DISCUSSION

fMRI Results

The fMRI results confirmed prior reports showing that WM guidance of visual selection can be associated with the WM cue reappearance effect and the WM cue validity effect [Grecucci et al., 2010; Soto et al., 2007]. In different brain regions, the cue reappearance effect led to decreases (repetition suppression) and increases in activation, which, we propose, reflect, respectively, passive reactivation of memory traces (repetition suppression) and the engagement of active WM processes (repetition enhancement). In connection to this, our TMS data (Study 2) indicate that the latter, enhancement of activity, is linked to the cue validity effects on behavior.

Soto et al. [2007] demonstrated that “passive” priming due to the repetition of the memory cue in the search display was associated with decreased neuronal responses [see also Kristjánsson et al., 2007] while the presentation of a stimulus matching the properties of a cue in WM was associated with enhanced neuronal responding in recognition memory areas (e.g., the PHG). Given the opposite direction of these effects, it is difficult to argue that our findings are based on simple amplification of a common underlying neural response found under priming and WM conditions alike; rather at least some mechanisms of bottom–up priming and top–down guidance from WM seem qualitatively different. This is supported by prior behavioral work, which has repeatedly shown little effects of bottom–up cueing on visual search when observers are merely exposed to cues that are not committed to memory [Downing, 2000; Olivers et al., 2006; Soto and Humphreys, 2007; Soto et al., 2005, 2006, 2007]. Also, the presence of a cue validity effect with verbal WM cues further indicates that cueing effects cannot be merely attributed to bottom–up priming from the visual properties of the initial stimulus.

The current data go beyond previous findings by showing common and dissociated effects of cue type format on WM‐based guidance of attention. A robust repetition enhancement effect for visual cues was seen in superior frontal and temporal cortices [Table II, see also Soto et al., 2007] while repetition enhancement effects from verbal cues were observed only in posterior cingulate gyrus. The lack of common regions showing increased responses to the reappearance of both types of WM cue suggests that the neural mechanisms of WM processing are critically dependent on the modality of the cue—more verbal vs. more visual [see Baddeley, 1986, 1992, 2003]. There was some evidence for a supramodal reappearance effect (with both visual and verbal cues) in the right SFG, though here reappearance of a WM item in the search display led to a decrease in response. We suggest that the repetition suppression effect in both cueing conditions here may reflect a common bottom–up component contributing to the deployment of attention by the contents of WM.

BOLD responses to the WM‐cue validity effect were observed for both cue types in the lateral occipital sulcus, cingulate gyrus, insula and right parietal cortex. In addition the validity of both cue types elicited a thalamic response, with more ventrolateral loci responding to cue validity in the more‐visual cueing condition (colored shapes) and with the pulvinar responding to cue validity in the more‐verbal (written words) condition. Further, the responses of the LOC and insula showed a larger validity effect for verbal cues compared with visual cues, while the left superior frontal sulcus was sensitive to the validity of a match to visual WM. This last effect was inferior to the region of the left SFG linked to cue reappearance. The modality specificity of these effects is consistent with a contrast between verbal and visual components of WM, which may modulate different regions—or common regions to different degrees—according to the degree of match between WM and search items. We consider these possibilities further after reviewing the TMS data.

TMS Results

The rTMS study concentrated on the causal role of the left SFG and LOC in the cue validity effects. The region of the left SFG that we stimulated was linked to effects of cue reappearance in the fMRI data, not the cue validity effect. Despite this, TMS to this region modulated the behavioral effects of cue validity. This demonstrates that regions sensitive to cue reappearance are necessary for cue validity effects to emerge. We interpret this as indicating that regions responding to cue reappearance provide a source of activation to areas that pool activation from the cue with activation from the search target, to generate effects of cue validity (e.g., in LOC and thalamus, among other regions). The enhanced neural response on valid compared to invalid trials, in the latter areas, may reflect this pooling of activity when the WM and search target coincide at the same location. Reduced activation, on invalid relative to neutral trials, might reflect competition when the WM and search‐based activation are directed to different locations. It is also possible, however, that there was a spread of TMS‐induced neural suppression beyond the stimulated region of the left SFG to affect the more inferior region that showed an effect of validity in the fMRI activation profile. The effects of TMS on LOC follow the fMRI data more directly, given that activity in the LOC reflected the cue validity effect. Here, we suggest that TMS reduced the sensitivity of the LOC to the joint top–down effects from WM and the bottom–up effects from the search target. The possibility of a spread of suppression to neighboring regions, in this case concerned with semantic representations should also be noted. For both stimulation sites, further work (e.g., combining TMS and fMRI) is required to test for any spread of suppression with the current TMS protocols.

Effects of Cue Type

Interestingly, we found some distinction between the neural areas supporting the effects of colored shape and written cues on attention. The left SFG showed stronger effects of cue reappearance with colored shape than with written cues, and effects of TMS to this region only disrupted the cue validity effect from colored shape cues. These data suggest that the left SFG is particularly involved in maintaining information in visual WM and then in the application of that information to modulate attention. Activation of the left LOC tended to be stronger with verbal than with visual cues, and the effects of rTMS to this region were more pronounced following verbal than visual cues.

Superior frontal gyrus

The finding that the left SFG was only involved in the cue reappearance effect for colored shape cues fits with this region, supporting visual memory representations. Prior work has demonstrated that SFG [around the frontal eye fields, see Paus, 1996] is involved in the encoding of the prior history of items [Bichot and Schall, 1999], and direct connections between the SFG and visual cortex may support strong visually based activation of this region [Schall et al., 1995]. Our finding that activation in the SFG modulates attentional guidance from visual cues extends prior work that links the SFG to covert and overt spatial selection [Corbetta et al., 1998; Henik et al., 1984; Hopfinger et al., 2000; Makino et al., 2004; Rosen et al., 1999]. Suppression of this activity, through rTMS, may reduce any feedback from this region to processing regions involved in visual selection. We also found evidence for a separate effect of cue reappearance in the left posterior cingulate gyrus for written cues. Though not a region usually associated with verbal WM, the left posterior cingulate gyrus has been implicated in studies assessing the comprehension of spoken language [Awad et al., 2007], and it may play a role in translating and maintaining a verbal representation of a written cue. We suggest, then, that there were separate sources of WM for the colored shape and written cues. In addition, we found that distinct regions of the thalamus were differentially sensitive to the validity of the colored shape (ventrolateral) and the written name (pulvinar), consistent with there being independent feedback routes to early processing, from the visual and verbal representations in WM through the thalamus. These representations may feedback to visual regions such as the LOC.

Lateral occipital cortex

LOC activation has been reported in crossmodal sensory processing between touch, audition, and vision regardless of the sensory input modality [Amedi et al., 2007; Pascual‐Leone and Hamilton, 2001]. Greater activation of the LOC when the cue and target information converge (on valid trials) would fit with this region, providing a driving signal that biases visual processing (and attention) to those stimuli. The stronger effects from written than visual cues in LOC could come about because the change to ongoing (visual) processing may be greater following feedback from verbal relative to visual WM. For example, bottom–up priming may affect both valid and invalid trials with a visual memory cue, while this will be absent with written cues. As a consequence, any changes in activation from the presence of the cue may be greater from written words than from colored shape cues. Alternatively, there may be stronger feedback links to this area from anterior language‐related regions than from visual WM. The LOC is usually linked to visual object recognition [Grill‐Spector et al., 1998, 2000; Kourtzi and Kanwisher, 2000] and to visual WM for complex objects [Xu and Chun, 2006]. However, regions neighboring the LOC are known to be involved in the semantic processing of written words and in processing semantic associations between words and pictures [Mechelli et al., 2007; Price and Mechelli, 2005]. These regions may support relatively strong feedback from verbal WM (written cues). A third possibility is that the reading and maintenance of a word in WM were more effortful than maintaining a single colored shape. Because of this increased effort, there was greater feedback from verbal WM in this case. In this last case, we cannot assume that the “driver” of the LOC was modality‐specific.

It might also be argued that the effects from verbal cues here reflect conscious visual imagery. Against this, however, we note that (i) memory cues and memory probes were always presented in the same format, either as colored shapes or as verbal descriptions, and so any visual recoding of the verbal materials would need to be retransformed again to carry out the memory test; (ii) verbal cues, like the visual cues, did not reliably predict where the search target was located or in which object it fell; hence, there was no incentive to attend strategically to visual stimuli matching the verbal description; (iii) the results are consonant with prior studies showing semantic influences in visual search [Belke et al., 2008; Huang and Pashler, 2007; Huettig and Altmann, 2005; Moores et al., 2003; Parasuraman and Martin, 2001]; finally, (iv) visual object processing (including imagery) ought to be stronger in the visual condition relative to the verbal case, and so, on an imagery account, LOC activation should be stronger following colored shape relative to written cues. This was not the case. Therefore, we conclude that visual imagery alone cannot account for the large effect of rTMS to the left LOC in the written cue condition.

CONCLUSIONS

Taken together, these findings demonstrate that there can be guidance of visual attention from representations in WM that are either of a more visual or of a more verbal nature (with colored shape or written word cues, respectively). This guidance is supported by interacting and dissociable neural systems. These systems include areas that respond to the reappearance of a WM cue in a subsequent display, along with areas that reflect the validity of the cue (in relation to a search target), which are responsive to specific types of cue.

A final important aspect of this work may be the potential implication for the assessment of cognitive function and cognitive recovery in neurological patients after brain insult. Current views contend that patients with posterior occipitotemporal brain lesions ought to show deficits mainly in visual object perception and recognition. In light of the current findings, however, we speculate that posterior brain lesions may also lead to attentional deficits that may not be of a strict visual nature but might also reflect impaired top–down guidance of vision from more verbal, semantic aspects of the visual information. Future work ought to throw light on this possibility.

Supporting information

Additional Supporting Information may be found in the online version of this article.

Supporting Information

Supporting Figure 1

Supporting Figure 2