Perception and action selection dissociate human ventral and dorsal cortex

Abstract

We test theories about the functional organization of the human cortex by correlating brain activity with demands on perception versus action selection. Subjects covertly searched for a target among an array of 4, 8, or 12 items (perceptual manipulation) and then, depending on the color of the array, made a saccade towards, away from, or at a right angle from the target (action manipulation). First, choice response times increased linearly as the demands increased for each factor, and brain activity in several cortical areas increased with increasing choice response times. Second, we found a double dissociation in posterior cortex: activity in ventral regions (occipito-temporal cortex) increased linearly with perceptual, but not action, selection demands; conversely, activity in dorsal regions (parietal cortex) increased linearly with action, but not perceptual, selection demands. This result provides the clearest support of the theory that posterior cortex is segregated into two distinct streams of visual processing for perception and action. Third, despite segregated anatomical projections from posterior ventral and dorsal streams to lateral prefrontal cortex (PFC), we did not find evidence for a functional dissociation between perception and action selection in PFC. Increasing action, but not perceptual, selection demands evoked increased activation along both the dorsal and ventral lateral PFC. Although most previous studies have focused on perceptual variables (e.g., space versus object), these data suggest that understanding the computations underlying action selection will be key to understanding the functional organization of the PFC.

INTRODUCTION

The primate visual system is divided into two distinct anatomical pathways: a ventral pathway that projects from visual cortex to the temporal lobe, and a dorsal pathway that projects from visual cortex to the posterior parietal lobe (Ungerleider & Mishkin, 1982). Visual information is transformed for object perception in the ventral pathway and for action selection in the dorsal pathway (Goodale & Milner, 1992; Ungerleider & Mishkin, 1982; but see also Konen & Kastner, 2008; Cusack, Mitchell, & Duncan, 2009). Lesions of the ventral pathway selectively impair visual perception (e.g. visual-form agnosia) (Farah, 1990), whereas lesions of the dorsal pathway selectively impair spatial attention and the visual control of action (e.g. optic ataxia) (Battaglia-Mayer & Caminiti, 2002). Previous neuroimaging studies on healthy humans have reported that the posterior ventral cortex is activated in perceptual tasks, such as object recognition (Kourtzi & Kanwisher, 2000), whereas posterior parietal cortex (PPC) is activated in spatial attention and movement tasks (Culham et al., 2003; Ikkai & Curtis, 2008). These studies indicate that the posterior ventral and dorsal pathways have distinct roles in visual information processing.

The two posterior pathways continue as largely segregated afferents to the frontal lobes, forming a ventral occipito-temporo-prefrontal pathway and a dorsal occipito-parieto-prefrontal pathway (Averbeck & Seo, 2008; Macko et al., 1982; Romanski, 2004). What is the fate of these segregated streams of processing in the frontal lobes? Theories are mostly agnostic and relevant data is scant on this issue. Goldman-Rakic (Goldman-Rakic, 1996) extrapolated from the specialized object processing in the ventral “what” pathway and spatial processing in the dorsal “where” pathway of posterior cortex (Ungerleider & Mishkin, 1982) to the ventral and dorsal lateral PFC, respectively. Her influential “domain specificity” hypothesis proposed that the lateral PFC is specialized for working memory, with the ventrolateral (VLPFC) and dorsolateral (DLPFC) prefrontal cortex maintaining the type of information that they receive from posterior pathways; i.e., VLPFC maintains non-spatial information from the “what” pathway and DLPFC maintains spatial information from the “where” pathway. Consistent with this hypothesis, studies have reported object or cue-related selectivity in VLPFC (Courtney, Petit, Maisog, Ungerleider, & Haxby, 1998; Scalaidhe, Wilson, & Goldman-Rakic, 1999; Wilson, Scalaidhe, & Goldman-Rakic, 1993) and spatial selectivity in DLPFC (Funahashi, Bruce, & Goldman-Rakic, 1989, 1993). However, other studies either did not find a spatial-nonspatial dissociation (Owen, Evans, & Petrides, 1996; Postle, Berger, Taich, & D’Esposito, 2000; Postle & D’Esposito, 1999) or proposed an altogether different functional architecture of the PFC (Badre & D’Esposito, 2007; Desposito, Postle, Ballard, & Lease, 1999; Desposito, Postle, & Rypma, 2000; Fuster, 2001; Koechlin, Ody, & Kouneiher, 2003). Moreover, lesions of the ventral and dorsal PFC do not provide conclusive support for the domain specificity model (Curtis & D’Esposito, 2004). Indeed, the presence, if any (Duncan & Owen, 2000), of a functional parcellation of the PFC is a current topic of intensive investigation in cognitive neuroscience (Badre, 2008; Badre & D’Esposito, 2009; Botvinick, 2008). Surprisingly, previous studies and theory have largely ignored motor factors, despite the fact that the PFC is thought to be at the apex of the motor hierarchy (Fuster, 2001).

Here, we test for a double dissociation between perception and action selection in the ventral and dorsal pathways of the human cortex. We predict that if the posterior visual streams can be clearly segregated according to perceptual and motor functions, then this functional organization may propagate to the lateral PFC. We test this hypothesis with a unique visuomotor search task that parametrically manipulates the demands on perception and action selection.

METHODS

Subjects

Nineteen neurologically healthy subjects (5 males, 18 right handed, age between 18 and 39) were recruited for participation and were paid for their time. All subjects had normal or corrected-to-normal vision. Subjects gave written informed consent and all procedures were in compliance with the safety guidelines for fMRI research and approved by the human subjects Institutional Review Board at New York University. One subject’s data were discarded from the analyses due to excessive movement (> 10 mm) during fMRI data acquisition.

Behavioral Procedures and Factorial Design

The experimental stimuli were controlled by E-Prime (Psychology Software Tools, Inc., Pittsburgh, PA) and projected (Eiki LC-XG100) into the bore of the scanner on a screen that was viewed by the subjects through an angled mirror. Subjects fixated a central white cross against a black background until a search array was presented. Examples of three of the nine possible search array displays are shown in Fig. 1a. Search arrays consisted of one target (letter “T”) and 3, 7, or 11 distractors (letter “L”), both of which were in Arial Black font to increase the target-distractor similarity, and thus to induce inefficient search slope. Both were 1.5 degrees of visual angle high and wide, and presented within an invisible annulus with an outer radius of 5.75 degrees of visual angle. Search arrays were visible for 3 sec, while subjects covertly searched for the target (i.e., gaze remained at fixation). Target and distractors could be presented in any of 3 colors (yellow, magenta, and cyan), and in any of 4 orientations (0, 90, 180, and 270 degrees). A variable inter-trial interval (ITI: 3, 5, 7, 9, or 11 sec) was used between search trials.

Fig. 1.

(a) Visuomotor search task, example trials. Subjects fixated a white cross during a variable intertrial interval (3~11 sec) that served as a baseline. Upon presentation of the search array, subjects covertly searched for the letter “T” among letter “L” distractors. By increasing the number of items in the array, we increased the demands on perceptual selection. The left, middle, and right panels show examples of trials in which there were 4, 8, and 12 item search arrays. By increasing the spatial compatibility between the location of the target and the required saccade, we increased the demands on action selection. The left, middle, and right panels show examples of trials in which subjects looked towards the target (prosaccade), away from the target (antisaccade), or rotated 90° clockwise from the target. The color of the search array indicated which saccade transformation to apply. Although the color-transformation assignment was counterbalanced across subjects, in this example, cyan, yellow, and magenta instructed a prosaccade, antisaccade, or rotation saccade, respectively (indicated by the white dotted arrow, invisible to subjects). (b) The average saccade response time (RT) data from one representative subject and (c) averaged across all subjects (N = 18). Each subject’s RT was converted into z-scores before averaging. Error bars are SEM.

We used a fully crossed factorial design with two factors, perceptual selection and action selection, each with 3 levels yielding a total of nine trial types. In order to increase the demands on perceptual selection processes, we manipulated the set size of the array. The arrays consisted of 4, 8, or 12 letter items. Placeholders (dots, 0.6 degrees of visual angle) were used to populate the entire array to approximate an equal visual display (e.g. visual luminance, spatial coverage). Letters and placeholders were arranged so that each quadrant contained one letter and two placeholders in set size 4 trials, and two letters and one placeholder in set size 8 trials. Only letters were used in set size 12 trials. In order to increase the demands on action selection, we manipulated the spatial compatibility between the location of the target and the required saccade (i.e., saccade transformation). The color of the search array specified the saccade transformation (Fig. 1a). The color instructed subjects to look to the target (prosaccade), 180 degrees opposite from the target (antisaccade), or rotated clockwise or counterclockwise 90 degrees from the target (rotation saccade). The color-saccade transformation assignment was counterbalanced across subjects and the order of trial types was pseudo-randomized. The location of the target and the color of the search array were pseudo-randomized so that neither the search nor saccade target appeared in the same place on more than two trials in a row, and the same color did not repeat on more than two trials. Each scanning session consisted of 8 blocks of four trials per condition, yielding a total of 36 trials per block and a total of 32 trials per condition in a scanning session. One subject terminated the experiment after 5 blocks (20 trials per condition). Subjects were trained on the task outside of the scanner on the day before the scanning session (~1 hour).

Oculomotor Procedures

Eye position was monitored in the scanner at 60 Hz with an infrared videographic camera equipped with a telephoto lens (ASL 504LRO; Applied Sciences Laboratories, Bedford, MA; modified with a Sony HAD CCD) that focused on the right eye viewed from the flat surface mirror mounted inside the RF coil. Nine-point calibrations were performed at the beginning of the session and between blocks when necessary. Eye-movement data were transformed to degrees of visual angle, calibrated using a third-order polynomial algorithm that fit eye positions to known spatialpositions, and scored offline with in-house software (GRAPES). Any trials with unwanted/incorrect saccades were discarded (e.g. overt search, corrective saccade, saccade to wrong item). Only trials in which the first saccade landed on the correct target and remained there until the search array offset were further analyzed. Saccadic reaction times were estimated with semiautomatic routines that relied on the velocity of the eye reaching about 30°/s to determine the onset of saccades. The data were also inspected visually, trial by trial, and corrections were made if necessary.

fMRI Procedures

fMRI data were collected using a 3T head-only scanner (Allegra, Siemens, Germany) at the Center for Brain Imaging at New York University. Images were acquired using custom radio frequency coils (NM-011 transmit head-coil and NMSC-021 four-channel phased array receive coil; NOVA Medical, Wakefield, MA) placed over lateral frontal and parietal cortices. During each fMRI scan, a series of volumes was acquired using a T2*-sensitive echo planar imaging pulse sequence (repetition time, 2000 ms; echo time, 30 ms; flip angle, 80°; 36 slices; 3 × 3 × 3 mm voxels; 192 × 192 mm FOV). High-resolution (1 mm isotropic voxels) MP-RAGE three-dimensional T1-weighted scans were acquired for anatomical registration, segmentation, and display. To minimize head motion, subjects were stabilized with foam padding around the head.

BOLD Analytic Procedures

Post hoc image registration was used to correct for residual head motion [MCFLIRT (motion correction using the Linear Image Registration Tool from Oxford University’s Center for Functional MRI of the Brain)] (Jenkinson, Bannister, Brady, & Smith, 2002). Additional preprocessing of the fMRI data was as follows. First, we bandpass-filtered the time series of each voxel (0.05 to 0.25 Hz) to compensate for the slow drift typical in fMRI measurements (Zarahn, Aguirre, & D’Esposito, 1997), and divided the time series of each voxel by its mean intensity to convert to percent signal modulation and compensate for the decrease in mean image intensity with distance from the receive coil.

The fMRI response was modeled with an impulse time-locked to the onset of the search array convolved with a canonical hemodynamic response function (HRF) (Polonsky, Blake, Braun, & Heeger, 2000). Each level of both factors (e.g. set size 4-prosaccade, set size 4-antisaccade, …, set size 12-rotation saccade, etc.) were modeled separately in the design matrix and entered into a modified general linear model (GLM (Worsley & Friston, 1995)) for statistical analysis using VoxBo (http://www.voxbo.org). For each subject, we used the software Caret (http://brainmap.wustl.edu/caret) for anatomical segmentation, gray-white matter surface generation, flattening, and multi-fiducial deformation mapping to the PALS atlas (Van Essen, 2005). Registering subjects in a surface space using precise anatomical landmark constraints (e.g., central sulcus, sylvian and calcarine fissures, etc.) results in greater spatial precision of the alignment compared to standard volumetric normalization methods. Statistical maps for contrasts of interest were created using the beta-weights estimated from each subject’s GLM. For overall task-related activity, we contrasted all trial types with the ITI baseline. To estimate linear increases in activation, we used weights of −1, 0, +1 to model increasing task demands associated with the appropriate levels of set size and saccade transformation. These parameter maps were then deformed into the same atlas space, and t-statistics were computed for each contrast across subjects in spherical atlas space. We used a nonparametric statistical approach based on permutation tests to help address the problem of multiple statistical comparisons, which are even more problematic when one performs statistical analyses on surfaces. First, we constructed a permuted distribution of clusters of neighboring surface nodes with t-values > 2.5. We chose a primary t-statistic cutoff of 2.5 because it is strict enough that intense focal clusters of activity would pass but not so strict that diffuse large clusters of activity are lost. In the case of a one-sample comparison, where measured values are compared to the test value of 0, the signs of the beta values for each node were randomly permuted for each subject’s surface, prior to computing the statistic. One thousand iterations, N, of this procedure were performed to compute a permutation distribution for each statistical test performed. Then, we ranked the resulting suprathreshold clusters by their area. Finally, corrected P-values at α = 0.05 for each suprathreshold cluster were obtained by comparing their area to the area of the top 5% of the clusters in the permuted distribution. Where C = Nα+1, we considered clusters ranked C^th or smaller significant at t = 2.5. The permutation tests controlled for Type I error by allowing us to formally compute the probability that an activation of a given magnitude could cluster together by chance (Holmes, Blair, Watson, & Ford, 1996; Nichols & Holmes, 2001).

In order to examine the relationship between measured behavior (RT) and neural activity (BOLD signal), we computed statistical maps that reflected correlations between evoked BOLD activity and saccade response times (RT) on a trial-by-trial basis. To do so, excluding incorrect trials, we regressed RT (convolved with a HRF) against BOLD time-courses. The resulting beta maps from each subject were entered into a second-level analysis in which we estimated the reliability of the effects across subjects (N=18) with a t-statistic (corrected for multiple comparisons as described above).

Time-series Analytic Procedures

We used ROI-based analyses of the time courses of BOLD signal change. First, on each subject’s high-resolution anatomical scans, we traced around grey matter of several ROIs motivated by past studies of visual search and oculomotor processing (Curtis and Connolly, 2008; Ikkai & Curtis, 2008; Jerde et al., 2008; Srimal & Curtis, 2008) and preliminary inspection of task-related deviations from baseline for all trial types in single subjects. In posterior cortex, these areas included the collateral sulcus (CoS, along ventro-medial surface of occipital to temporal lobes), the horizontal segment of the intraparietal sulcus (hIPS, extended from the junction with postcentral sulcus to the junction with parieto-occipital sulcus), the superior parietal lobule (SPL, the gyrus posterior to the postcentral sulcus, anterior to the parietal-occipital sulcus, and lateral to the interhemispheric sulcus, medial to hIPS), and the inferior parietal lobule (IPL, the gyrus posterior to the postcentral sulcus, anterior to the parietal-occipital sulcus, and lateral to the hIPS). The IPL was chosen because of its unique negative activation during all trial types (Fig. 2a).

Fig. 2.

Surface-based statistics. (a) Overall task-related activations independent of trial type. Lateral and medial surfaces of the right (RH) and left (LH) hemispheres are shown. (b) posterior view and (c) lateral view of regions showing significant linear effects of set size (red), saccade transformation (green), or both (yellow).

In the lateral PFC, ROIs included the superior precentral sulcus (sPCS, along the precentral sulcus and lateral to the junction with superior frontal sulcus), the superior frontal sulcus (SFS), the inferior frontal sulcus (IFS), and the dorsal and ventral middle frontal gyrus (d/vMFG). SFS and IFS were defined as the grey matter within these sulci but not extending past the gyral convexity. Dorsal and ventral MFG were separated by the fundus of the intermediate frontal sulcus. As can be seen in Fig. 4a, SFS, dMFG, vMFG, and IFS divide the PFC into 4 ROIs from dorsal to ventral along the lateral PFC. We were not able to identify the intermediate frontal sulcus in one of the subjects. For this subject, all ROIs but d/vMFG were included in the analysis, which led to N = 17 for d/vMFG and N = 18 for all other ROIs. Within each ROI, we used an F-test to select 20 voxels (540 mm³) with the strongest overall task effect. These voxels showed a consistent deviation from baseline during the task. The selection is unbiased by activation (could be negative or positive relative to baseline) or trial type (none of the 2 factors x 3 levels are given unique weight). BOLD data were converted into percent signal change, and time-courses time-locked to the onset of the search array were deconvolved using AFNI (http://afni.nimh.nih.gov/afni), with no hemodynamic response assumed. The estimated impulse response functions were averaged across voxels within an ROI and averaged across subjects from analogous ROIs to visualize time-series. Error bars are standard deviations between subjects at each time point. For an individual subject, the average of 3 TRs around the peak of the impulse response function (time points 4, 6, and 8 sec) from each condition was extracted from each ROI and used as a dependent variable in statistical analyses of the time-courses.

Fig. 4.

Time-series data. (a) ROIs were drawn for each subject based on anatomical landmarks including the collateral sulcus (CoS), superior parietal lobule (SPL), horizontal segment of the intraparietal sulcus (hIPS), and inferior parietal lobule (IPL) in posterior cortex. ROIs in prefrontal cortex (from dorsal to ventral) included the superior precentral sulcus (sPCS), superior frontal sulcus (SFS), dorsal and ventral middle frontal gyrus (dMFG, vMFG) and inferior frontal sulcus (IFS). BOLD time-series from posterior (b) and frontal (c) ROIs. Red lines depict responses evoked by each array set size regardless of saccade transformation. Green lines depict responses evoked by each saccade transformation regardless of set size. Asterisks indicate a significant difference within the gray epochs of the average signal in given ROIs. Error bars represent between-subject standard error. For CoS, hIPS, and sPCS, time-series from the right hemisphere are shown (“r-“). Left hemisphere ROIs showed the same activation patterns (see Results).

Retinotopic Mapping Methods and Analyses

In 4 subjects, we identified subregions within the visual cortex with standard retinotopic mapping procedures (Engel et al., 1994; Larsson & Heeger, 2006; Sereno et al., 1995). Stimuli were controlled by MGL (http://justingardner.net/mgl), and stimulus presentation and analysis were performed following procedures in Heeger and colleagues (Gardner, Merriam, Movshon, & Heeger, 2008; Larsson & Heeger, 2006). During the scanning session, observers viewed high-contrast checkerboard stimuli presented within a rotating wedge aperture while fixating the center of the display. Wedge apertures subtended 90° of polar angle and extended from 0.6 to 12.6° eccentricity, moving in the stepwise increment of 10° visual angle per TR. Regions outside the aperture were gray. Each scan lasted 252 sec, yielding 10.5 cycles, but we discarded the first half cycle of each scan before analysis. All subjects completed at least 6 scans of wedges (3 clockwise and 3 counterclockwise). One subject had 5 and 4 clockwise and counterclockwise scans, respectively, and another subject had 4 clockwise and counterclockwise scans. The time-series for each scan were coarsely corrected for hemodynamic delay by shifting the time-series of each voxel by 4 secs. The time-series for the counterclockwise wedges were then time-reversed and averaged with the time-series for clockwise wedges. Phase information was then projected onto the flattened surface generated in Caret (Van Essen, 2005). Here, we identified V1, dorsal and ventral V2 and V3 boundaries using phase reversals in the angular component of the visual field representation. These ROIs were then transformed back to each subject’s functional space, and time-series were extracted.

RESULTS

Behavioral Results

As predicted, subject performance was better when the set size was smaller, and when the saccade transformation was simpler. Across trial types, subjects were on average 86% accurate. A repeated measures ANOVA of accuracy revealed significant linear effects of set size, F (1,17) = 22.10, P < 0.001, effect size = 0.57, and saccade transformation (prosaccade < antisaccade < rotation saccade), F (1,17) = 22.65, P < 0.001, effect size = 0.59, but critically the two factors did not interact, F (4,68) =0.69, P > 0.05. Although a different number of trials contributed to each condition, our post-hoc power analysis revealed large power for the linear analysis of set size (power = .991) and saccade transformation (power = .993). Similarly, response times (RTs) increased linearly as the set size increased (F (1,17) = 179.49, P < 0.001, effect size = 0.91) and as the saccade transformation became more difficult, F (1,17) = 87.28, effect size = 0.84. Neither factor showed a quadratic effect (P values > 0.05). Fig. 1b shows the RT data from a representative subject and Fig. 1c shows the average normalized RT of all subjects. The slope for the set size effect collapsed across saccade transformation was 40 msec/item, and the slope for the saccade transformation effect collapsed across set size was 211 msec/transformation (prosaccade, antisaccade, rotation saccade). Univariate ANOVAs within each subject revealed that the main effect of set size was statistically significant in all subjects (all P values < 0.003); RT was fastest in set size 4 trials, followed by set size 8, and slowest in set size 12 (linear contrast, all P values < 0.003). All but two subjects (16/18) showed a significant main effect of saccade transformation; saccade initiation was fastest for prosaccades, followed by antisaccades, and slowest for rotation saccades. One of the two subjects who did not show a main effect of saccade transformation approached significance, P = 0.08. The interaction of set size and saccade transformation on RT was not significant in all but 2 subjects. Overall, our manipulations of perceptual selection (set size) and action selection (saccade transformation) had robust, reliable, and predictable effects on behavioral performance that make interpreting the neuroimaging data straightforward.

Imaging Results: Cortical Surface Statistical Analyses

Performance of the visuomotor search task evoked BOLD activity in occipital, posterior temporal, parietal and a large extent of frontal cortices bilaterally (Fig. 2a). Moreover, We found robust positive correlations between behavioral RTs and bilateral activation in these same areas (occipital and parietal cortices, superior and inferior PCS (s/iPCS), insula, and preSMA (Fig. 3)), indicating a strong coupling of neural activation with task performance. Additionally, right central sulcus, right inferior frontal gyrus, left intermediate sulcus, and left IFS showed positive correlations with the RT. We found a significant negative correlation in the right IPL. Positive correlations may reflect the greater neural activity (duration or magnitude) associated with the more demanding level of the factor (e.g., saccade transformation: rotation saccade > antisaccade > prosaccade) that resulted in longer RTs. These data nicely link theory (experimental conditions), measured behavior (RT), and neural activity (BOLD signal).

Fig. 3.

Surface statistics showing cortical regions that significantly correlated with saccadic RT. The medial surface of the left hemisphere and lateral surface of the right hemisphere are shown. Positive (orange/yellow) and negative (blue) correlations with RT are depicted (P < 0.05, corrected).

In posterior cortex, we observed a clear distinction between areas showing a linear effect of set size and of saccade transformation (Fig. 2b). The linear effect of set size (set size 12 > set size 4, Fig. 2b, red) was observed bilaterally in the area extending from the occipital lobe to the medial surface of the temporal lobe (P < 0.001, corrected), whereas the linear effect of saccade transformation (rotation saccade > prosaccade, Fig. 2b, green) was observed in bilateral posterior-to-anterior parietal lobe (P < 0.001, corrected). A negative linear effect of saccade transformation was observed in the right IPL (P < 0.03, corrected) and left posterior cingulate (P < 0.03, corrected). We also observed an overlap between these linear effects at the lateral side of the occipital cortex (Fig. 2b, yellow). This region, located just posterior to the parieto-occipital sulcus in the right hemisphere, could be sensitive to both set size and saccade transformation, thus making it a bridge between visual processing for perception and action. Overall, these data show a clear functional double dissociation between perceptual selection and action selection in the ventral and dorsal regions of the posterior human brain, respectively.

We did not find a double dissociation in the frontal cortex, where no clusters showed a significant linear effect of array set size (Fig. 2c). However, the left precentral sulcus (P = 0.003, corrected) showed a linear effect of saccade transformation. In the right hemisphere, the effect was marginal along the precentral sulcus (P = 0.103 and 0.068 for the superior and inferior precentral sulcus, corrected), but the pattern was similar to that of the left hemisphere.

To summarize, the cortical surface statistics revealed a clear distinction between posterior ventral and dorsal areas as the demands on perceptual selection and action selection increased, respectively. Conversely, such effects were not observed in frontal areas. We considered the possibility that this result may be partly due to poor inter-subject anatomical co-registration. Although we used inflated surface-based registration to ensure higher inter-subject alignment consistency (Van Essen, 2005), registration in frontal cortex is generally poor compared to posterior cortex due to large individual differences in sulcal folding patterns. Evidence for individual variability is exemplified by the fact that one subject did not have an intermediate frontal sulcus. To minimize these effects, we performed time-series analyses on regions-of-interest (ROIs) based on anatomical criteria applied on an individual subject basis. With this approach, we parcellated the lateral frontal cortex into four anatomical ROIs spanning the dorsal to ventral convexity. Namely, we identified the superior frontal sulcus (SFS), dorsal middle frontal gyrus (dMFG), ventral middle frontal gyrus (vMFG), and inferior frontal sulcus (IFS). For comparison purposes, we included additional ROIs that included the collateral sulcus (CoS), horizontal segment of the intraparietal sulcus (hIPS), superior (SPL) and inferior (IPL) parietal lobule, and superior precentral sulcus (sPCS). Only the right hemisphere ROI was analyzed for the IPL because of its unique negative activation during the task vs. baseline contrast (Fig. 2a).

Imaging Results: ROI Time-series Analyses

In three of our ROIs (CoS, hIPS and sPCS), right hemisphere activation was larger than left hemisphere activation (CoS: F (1,17) = 6.586, P < 0.03, hIPS: F (1,17) = 8.818, P < 0.01, sPCS: F (1,17) = 7.790, P < 0.03). Therefore, the right and left hemispheres were analyzed separately for these ROIs. It is important to note that although the right hemisphere activation was overall larger in these ROIs, the same patterns of time series (Fig. 4b,c) and linear effects (see below) were observed in the right and left hemispheres. For the other ROIs that did not show an effect of hemisphere, we collapsed across hemispheres to increase statistical power. Fig. 4 plots the subject-averaged time series from each of the ROIs, time-locked to the onset of the search array. Data from the right hemisphere (“r-”) are shown for CoS, hIPS and sPCS. As a dependent variable, we averaged the signal from the three time points around the peak of the hemodynamic response for each subject (see Methods; gray epochs in Fig. 4b,c). To be completely consistent in our ROI criteria, we omitted the one subject who did not have an intermediate frontal sulcus from the analyses involving the dMFG and vMFG.

In ventral posterior cortex, CoS (Fig. 4b) showed a significant linear increase in the BOLD signal as the set size increased (left: F (1,17) = 4.80, P < 0.05; right: F (1,17) = 9.77, P < 0.01), but no effect of saccade transformation (both F values < 1, P > 0.05). In 4 subjects, we collected additional BOLD data to create retinotopic maps of visual cortex in the occipital lobe (Fig. 5a). Similar to anatomically defined CoS, we found a significant linear effect of set size in V2d (F(1,3) = 12.94, P < 0.05), V2v (F(1,3) = 12.10, P < 0.05), and V3d (F(1,3) = 14.97, P < 0.05) (Fig. 5b). This effect was not reliable, but approached significance in V1 (F(1,3) = 7.63, P = 0.070) and V3v (F(1,3) = 7.86, P = 0.068). None of the retinotopically-defined ROIs showed a significant linear effect of saccade transformation (all F values < 1, all P values > 0.05).

Fig. 5.

(a) Retinotopic visual areas in left hemisphere of an example subject. Boundaries were first identified on flattened surfaces. Here, an inflated medial surface is shown for presentation. Colors plotted represent the phase angle giving maximum correlation at the task frequency. The key on the left can be used to match the color to the visual field, where UVM = upper vertical meridian, LVM = lower vertical meridian, RHM = right horizontal meridian, and LHM = left horizontal meridian. (b) Time-series plots from retinotopically-defined ROIs. Color schematics are as in Fig. 4.

In dorsal posterior cortex, hIPS and SPL showed a significant linear increase as the saccade transformation demands increased (right hIPS: F (1,17) = 75.77, P < 0.001, left hIPS: F (1,17) = 77.47, P < 0.001; SPL: F (1,17) = 64.88, P < 0.001), but no effect as set size increased (all F values < 1, all P values > 0.05) (Fig. 4c). Thus, consistent with the surface statistics, the time-series extracted from posterior ventral and dorsal ROIs showed a double dissociation: BOLD responses in occipital and temporal cortex increased significantly with greater set size, whereas activation in PPC increased significantly with a greater demand on saccade transformation. A region-by-condition interaction between hIPS and CoS was significant for a linear effect of set size, F(1, 17) = 11.19, P < .005, and saccade transformation, F(1,17) = 43.89, P < .001.

The right IPL was the only cortical region across both hemispheres that was negatively activated during the task, but neither linear effect of set size or saccade transformation was significant, F(1,17) < 1.12, P > 0.05. There was a significant quadratic effect of set size, F(1,17) = 5.37, P < 0.05. Post hoc paired t-test revealed that the quadratic effect was driven by the significant difference between set size 4 and set size 8, t(17) = 2.35, P < 0.05.

Similar to the time-courses from PPC, the PFC ROIs (left and right sPCS, SFS, vMFG, and IFS) showed a significant linear increase in activation for saccade transformation (all F values > 8.0, all P values < 0.03); except dMFG, F (1,16) = 3.38, P = 0.08 (Fig. 4c). None of the PFC ROIs showed an effect of set size (all F values < 0.33). Consistent with the behavioral data, no ROI showed an interaction between set size and saccade transformation, or a quadratic effect of either factor.

DISCUSSION

We used rapid event-related fMRI to measure cortical activity during a visual search task that parametrically manipulated the demands on perceptual and action selection. Previous fMRI studies have reported a dissociation between posterior ventral and dorsal regions for perception and action, but the tasks involved action recognition (Shmuelof & Zohary, 2005), not action itself, or examined perception and action in separate experimental conditions and subsequently compared the resulting activation across regions (Cavina-Pratesi, Goodale, & Culham, 2007). The present study used a single search task that manipulated the demands on perception and action selection in a factorial design (Fig. 1). Consistent with past behavioral studies, response times increased as the number of potential targets increased (Duncan & Humphreys, 1989; Treisman, 1991; Treisman & Gormican, 1988) and the difficulty of the response transformation increased (Everling, Dorris, Klein, & Munoz, 1999; Fischer & Weber, 1992; Fischer, Deubel, Wohlschlager, & Schneider, 1999). Neurally, we found a double dissociation in posterior cortex: brain activity in ventral regions (occipito-temporal cortex) increased linearly with perceptual, but not action, selection demands, whereas activity in dorsal regions (PPC) increased linearly with action, but not perceptual, selection demands (Fig. 2b). In the PFC, no double dissociation was observed; rather, activity increased only as action selection demands increased (Fig. 2c). Below, we relate these findings to previous studies and discuss their implications.

Perceptual Selection

Activation in occipito-temporal regions increased linearly as the set size increased during visual search. The difficulty of saccade transformation had no effect. The present task required the deployment of attention to extract the target (letter T) from distractors (letters L). We conclude that the activation in posterior ventral areas with increasing set size reflects the individuation of items, which is a critical component of the visual selection process. In this regard, increased activity reflects either the number of selection operations and/or the time required for selection to take place. Occipito-temporal regions are active during the processing of object categories or features, such as faces (Downing, Liu, & Kanwisher, 2001; O’Craven, Downing, & Kanwisher, 1999) color, and texture (Cant & Goodale, 2007; Cant, Large, McCall, & Goodale, 2008), and activation in these areas increases when attention is directed to that feature (Downing et al., 2001; O’Craven et al., 1999; Wojciulik, Kanwisher, & Driver, 1998). Furthermore, lesions of human posterior ventral areas disrupt search tasks (Humphreys, Riddoch, & Quinlan, 1985). Imaging studies have shown that word and letter recognition activate the more medial portions of the fusiform gyrus (Cohen et al., 2002; Dehaene et al., 2002; James et al., 2005; Joseph et al., 2006; Polk et al., 2002), near where we found activation along the collateral sulcus. Although we did not run a functional localizer for the lateral occipital area (LO), we note that linear effects were not observed in this region for the set size or saccade transformation contrasts (Fig. 2c). The lack of LO activation in these contrasts was expected, since the manipulations of set size and saccade transformation were embedded within the same visual display, which contained essentially the same number, type, and arrangement of objects.

Traditionally, the set size effect in visual search refers to the linear increase in RT as the number of distractors increases. Neuroimaging studies have not agreed on which brain areas (e.g., prefrontal, parietal, or occipital) show an effect of set size (Anderson et al., 2007; Leonards, Sunaert, Van Hecke, & Orban, 2000; Muller et al., 2003). In monkey electrophysiological studies, firing rates in both frontal eye field (FEF) (Cohen, Heitz, Woodman, & Schall, 2009) and lateral intraparietal area (LIP) (Balan, Oristaglio, Schneider, & Gottlieb, 2008) neurons correlate with set size, although there is debate about the interpretation of these results (Balan & Gottlieb, 2009; Cohen et al., 2009). A consistent finding in these studies is that neuron firing rates decline proportionately as the set size increases, presumably due to competitive interactions produced by more stimuli in the stimulus array.

When interpreting past studies, one must consider two potential confounds in manipulating set size by increasing the number of distractors. First, greater set sizes in traditional visual search tasks contain more visual stimuli in the display, which would likely drive neural activity in brains areas involved in low-level visual processing. And second, increasing the number of items in the display might result in increased competitive interactions among neurons for which the potential targets lie within their receptive fields (Schall, Hanes, Thompson, & King, 1995). Therefore, it remains unclear whether processes related to attentional selection or low-level perception cause the neural changes that have been associated with increasing set size. We avoided these concerns by using placeholders in the smaller set sizes, which equated the amount of retinal stimulation across set sizes. This design allowed us to manipulate the amount of relevant visual information in the task, and therefore the perceptual selection demands, without co-varying the overall amount of sensory information that fell within the receptive fields of individual neurons. Importantly, our saccadic RT data show a classical set size effect even though 12 items were always present, namely a linear increase in RT as the target-to-distractor ratio increased. Furthermore, the fact that we observed a robust set size effect in occipital and temporal regions indicates that our experiment taxed perceptual processing, and the significant correlations between saccadic RT and BOLD activation in occipital cortex, PPC, and PFC indicate that neural activation reflected the specific demands of the task (Fig. 3).

Given the involvement of the PPC in the control of attention, one might expect to observe a set size effect in this region since the demand on visual attention presumably increases with set size. For example, it is reasonable to think that the number and direction (Gourtzelidis et al., 2005; Jerde et al., 2008) of covert shifts of attention would increase with the number of potential targets, and load-dependent increases have been observed in PPC for attention and working memory (Mayer et al., 2007). However, we did not find a set size effect in PPC. One possible explanation is that parietal activation might have saturated at set size 4 (Mitchell & Cusack, 2008). Another possibility is that visual search tasks have limited utility in testing claims about the neural effects of covert shifts of attention (Wardak, Ibos, Duhamel, & Olivier, 2006; Wardak, Olivier, & Duhamel, 2004) or claims about serial versus parallel processing of spatial attention (Leonards et al., 2000), since it is impossible to reliably measure the occurrence, timing, or direction of attention shifts during search. Indeed, in the present experiment, it is plausible that the number of covert attention shifts may not have differed appreciably across set sizes of 4, 8, and 12 potential targets. For each set size, target search may only have required attention shifts to quadrants of the stimulus array, whereupon the target is identified, rather than to each item until the target is found. Since we have no evidence that increasing set sizes requires more attention shifts, the lack of a set size effect in PPC may not be surprising.

The right IPL was uniquely deactivated during the task (Fig. 2a). The right temporo-parietal junction, including the right IPL, is involved in filtering distractors during visual search (Shulman, Astafiev, McAvoy, d’Avossa, & Corbetta, 2007). Shulman et al. (2007) reported that the degree of deactivation was correlated with performance during visual search, such that deactivation was greater when the target was detected. In the current task, deactivation was larger in set size 8 compared with set size 4 conditions. This result suggests that the right IPL plays a role in enhancing the search process by filtering sensory information from distractors.

Action Selection

As the demands on action selection, but not perceptual selection, increased, neural activity increased in PPC and PFC (Fig. 2b,c). The greater BOLD response in the PPC and PFC for antisaccades compared to prosaccades is consistent with past human imaging studies (Brown, Vilis, & Everling, 2007; Curtis & Connolly, 2008; Curtis & D’Esposito, 2003a, 2003b; DeSouza, Menon, & Everling, 2003). During antisaccade trials, subjects invert the saccade vector by 180 degrees and execute a volitional saccade to the mirrored location. Rotation saccades are more demanding than antisaccades because subjects must calculate the spatial coordinates of the 90-degree rotation, as well as the direction of the rotation, and then execute a saccade to the internalized location (Fischer et al., 1999). The activation in PPC and PFC as a function of action selection indicates that these regions are involved in the spatial-motor transformation, a conclusion that is consistent with previous studies of sensorimotor integration in parietal cortex (Andersen & Buneo, 2002; Buneo & Andersen, 2006) and with the proposed role of the lateral PFC in action selection (Miller & Cohen, 2001; Passingham, 1993; Rowe, Toni, Josephs, Frackowiak, & Passingham, 2000; Wise, Boussaoud, Johnson, & Caminiti, 1997).

Previous studies have tested theories about the functional differences of the ventral and dorsal PFC, mainly using working memory tasks (Curtis & D’Esposito, 2004; Desposito et al., 2000; Goldman-Rakic, 1996). Invariably, these studies probed spatial working memory with memory-guided delayed response tasks (DR), and object working memory with delayed-matching or non-matching-to-sample tasks (DMTS, DNMTS). Motor factors confound the spatial-object distinction because the subject knows the forthcoming motor response throughout the delay of “spatial” DR tasks, but not during “object” DMTS and DNMTS tasks. In fact, most theories of PFC consider only stimulus features and various forms of cognitive control, but not motor factors. Given the largely segregated inputs to VLPFC and DLPFC from temporal and parietal cortex (Averbeck & Seo, 2008; Macko et al., 1982), it is rather surprising that no one, to our knowledge, has systematically examined the hypothesis that the human VLPFC and DLPFC are selectively engaged in the selection of perception and action, respectively. Rather, past studies of the distinction between ventral and dorsal pathways have focused almost exclusively on the posterior cortex (Cavina-Pratesi et al., 2007; Culham et al., 2003). In the lateral PFC, recent data and models have put forward a hierarchical view of its functional organization (Badre, 2008; Botvinick, 2008; Koechlin et al., 2003; Koechlin & Summerfield, 2007), with anterior regions controlling more abstract or temporally distant characteristics of cognition and behavior, whereas posterior regions control more immediate and sensorimotor characteristics. However, these models make no claims concerning the functional specialization of the ventral and dorsal regions along the anterior-posterior axis of the PFC (Koechlin & Summerfield, 2007), highlighting the importance of the findings of the current study.

Conclusions

By imaging the human brain during a demanding visual search task, we provide the first clear neuroimaging evidence of a functional double dissociation between the posterior ventral and dorsal pathways for perceptual selection and action selection, respectively (Goodale & Milner, 1992; Milner & Goodale, 1995, 2008). Despite largely segregated anatomical projections from posterior ventral and dorsal streams to lateral PFC, we did not find evidence for a functional dissociation between perception and action selection in PFC. Increasing action, but not perceptual, selection demands evoked increased activation along both the dorsal and ventral lateral PFC. These data suggest that understanding the computations underlying action selection will be key to understanding the functional organization of the PFC.