EEG-Informed fMRI Reveals Spatiotemporal Characteristics of Perceptual Decision Making

Abstract

Single-unit and multiunit recordings in primates have already established that decision making involves at least two general stages of neural processing: representation of evidence from early sensory areas and accumulation of evidence to a decision threshold from decision-related regions. However, the relay of information from early sensory to decision areas, such that the accumulation process is instigated, is not well understood. Using a cued paradigm and single-trial analysis of electroencephalography (EEG), we previously reported on temporally specific components related to perceptual decision making. Here, we use information derived from our previous EEG recordings to inform the analysis of fMRI data collected for the same behavioral task to ascertain the cortical origins of each of these EEG components. We demonstrate that a cascade of events associated with perceptual decision making takes place in a highly distributed neural network. Of particular importance is an activation in the lateral occipital complex implicating perceptual persistence as a mechanism by which object decision making in the human brain is instigated.

Introduction

Even the simplest decisions we make on a day-to-day basis are thought to involve a complex cascade of spatially and temporally dependent cortical activations. What cortical areas are activated, their relative timing and causality remains of intense interest in cognitive neuroscience as ultimately these are the questions that must be answered to truly infer the underlying neural networks. The localization restrictions of EEG and the temporal precision constraints of fMRI suggests that only a combination of these approaches can enable the recovery of the network processes underlying decision making in the human brain.

A number of single unit and functional magnetic resonance imaging (fMRI) studies have attempted to localize neural activity related to perceptual decision making (Kim and Shadlen, 1999; Shadlen and Newsome, 2001; Paulus et al., 2002; Heekeren et al., 2004; Grinband et al., 2006). These studies have found a variety of areas modulated by decision accuracy and decision uncertainty, including areas in the frontal and parietal cortices, the thalamus, as well as the striatum. Using a cued paradigm and single-trial analysis of electroencephalography (EEG) we reported previously temporally specific components related to decision accuracy as well as decision difficulty (Philiastides and Sajda, 2006; Philiastides et al., 2006). Specifically, we collected EEG data while subjects were asked to either perform a face-categorization task (e.g., face vs car) or discriminate the color of an image (e.g., red vs green) (Philiastides and Sajda, 2006; Philiastides et al., 2006). Discriminating between face and car trials, using a machine learning approach, we identified two face-selective components reflective of decision accuracy: an “early” (170 ms after stimulus presentation) and a “late” one (≥300 ms), the strength of which was proportional to the strength (i.e., percent coherence) of the stimulus.

The strength of the early component was comparable for both the face versus car and color-discrimination tasks. Its presence in the color discrimination, although our subjects were not explicitly categorizing the type of image, indicated that this component is associated with an early perceptual event. The late component, however, was shown to be more closely linked to a postsensory/decision event, as it was a better match to the psychophysical data (compared with the early one), systematically shifted later in time and became more persistent as a function of decision difficulty, disappeared when a relevant decision was no longer needed (i.e., during the color discrimination), and correlated significantly with the mean drift rate in a diffusion model simulation.

In addition, we found a “difficulty” component situated between the early and late components (≈220 ms after visual stimulation), the strength of which was highly correlated to the difficulty of the task and predicted the onset time of the late component. We also showed that the difficulty component was likely to represent a top-down influence of attention on decision making, rather than a mere bottom-up processing of the stimulus, by virtue of the fact that, for the same stimulus evidence, it was reduced during a trivial color discrimination.

In this work, we use a technique where we drive the analysis of fMRI data collected for the same task based on previous information realized through single-trial EEG analysis. Using this EEG-informed fMRI design, we are able to identify the cortical regions that are correlating with each of our EEG temporally specific components. We summarize our results in a form of a spatiotemporal diagram and show that a cascade of events associated with perceptual decision making take place in a highly distributed neural network.

Materials and Methods

Participants.

Twelve subjects (seven males, five females; mean age, 30.2 years) participated in the study. Four of these subjects also participated in the original EEG experiments. All had normal or corrected to normal vision and reported no history of neurological problems. Informed consent was obtained from all participants in accordance with the Columbia University Institutional Review Board.

Stimuli.

We used a set of 20 face (Max Planck Institute face database) and 20 car grayscale images (image size 512 × 512 pixels, 8 bits/pixel). All images were equated for spatial frequency, luminance, and contrast. They all had identical magnitude spectra and their corresponding phase spectra were manipulated using the weighted mean phase (Dakin, 2002) technique to generate a set of images characterized by their percentage phase coherence. We then colorized the images with subtle red and green tones. We performed this adjustment by manipulating the hue (H), saturation (S), and value (V) color space of the different percentage coherence images (red, H, 0.04; S, 0.17; V, unchanged; green, H, 0.34; S, 0.23; V, unchanged). A Dell (Round Rock, TX) Precision 530Workstation with nVidia (Santa Clara, CA) Quadro4 900XGL graphics card and E-Prime software (Psychological Software Tools, Pittsburgh, PA) controlled the stimulus display. A liquid crystal display projector was used to project the images through a radio frequency shielded window onto a front projection screen. Each image was subtended 11° × 8° of visual angle.

Behavioral paradigm.

Similarly to the EEG experiments we designed previously (Philiastides and Sajda, 2006; Philiastides et al., 2006), subjects were originally presented with one of two possible visual cues for 250 ms, which was followed by a variable delay in the range 1–4 s before the next image presentation. The cue indicated whether subjects would perform a face-versus-car (FC) categorization task or simply discriminate the color of the image [i.e., red vs green (RG)] (see Fig. 1a). Images were presented in interleaved manner. Each image was presented for 50 ms and was followed by another delay period that was randomized in the range 1–4 s (mean delay, 2.5 s) (Wager and Nichols, 2003) during which subjects had to make a choice by pressing one of two buttons. Participants were instructed to press the left button of an MR compatible Lumitouch button box (Photon Control, Burnaby, British Columbia, Canada) with their index finger for a face or a red choice and the right button with their middle finger for a car or a green choice. Image presentation events and button presses were recorded on separate channels at a sampling rate of 1000 Hz using a National Instruments (Austin, TX) analog/digital card on a personal computer running LabVIEW software.

Figure 1.

Experimental paradigm and behavioral performance. a, Colorized images at two phase coherence levels were presented in random order. At each trial subjects were initially shown one of two possible cues before the presentation of an image. The cue indicated whether subjects would perform the face-versus-car categorization task or simply discriminate the color of the image (i.e., red vs green). Subjects were required to make a decision by pressing a button. b, The proportions correct for the three suprathreshold conditions were statistically indistinguishable. The performance for the perithreshold condition however was significantly lower. c, Correspondingly, all subjects had comparable mean RTs for the three suprathreshold conditions whereas the mean RT for the perithreshold condition was significantly higher. Error bars represent ±1 SEM.

We used only two phase coherence levels for this study. The high coherence level (45%) was chosen so that subjects performed nearly perfectly at the face-versus-car discrimination. The second coherence level (in the range 32.5 to 37.5% depending on the subject) was chosen so that each subject was at perithreshold performance (i.e., 80% correct). A block of trials consisted of a total of 80 trials across all conditions. Each subject performed a total of four blocks (i.e., 40 trials per condition) while we simultaneously recorded functional MRI data. All subjects were trained before the main experiment (practice sessions took place on different days but always within 2–3 d of the main experiment). Subjects qualified for the main experiment when their performance became consistent across blocks.

EEG-informed fMRI analysis.

Figure 2 illustrates how we used the experimental observations from our earlier EEG recordings (Philiastides and Sajda, 2006; Philiastides et al., 2006) to inform the fMRI analysis. Our goal was to incorporate the strength of each component (i.e., early, difficulty, and late) with respect to the difficulty (i.e., percent coherence) and the type of task (i.e., face vs car or red vs green) into the fMRI regressor design. Specifically, we modulated the fMRI regressor heights based on whether the subject was discriminating FC or RG and whether the presented image was at the low (LC) or high coherence (HC) level.

Figure 2.

Schematic representation of our EEG-informed fMRI analysis design. For simplicity, we modeled all of the high strength regressor amplitudes with 1s and the low strength amplitudes with 0.5s. The significant reduction in strength for the late component during the color discrimination we modeled with 0.2s. Each regressor event was centered on the peak time of each component as identified in the EEG (i.e., at 170, 220, and 300 ms for each component respectively) and had a duration of 50 ms to match the presentation time of the stimulus. Regressor amplitudes outside these 50 ms event windows were set to zero. To absorb the variance of uncorrelated processes we included two unmodulated regressors, one centered at the onset time of the stimulus and one at the response time. Note that only the four representative trial types (i.e., FC LC, FC HC, RG LC, RG HC) are shown here. All like regressor events were modeled in the same way.

The strength of both the early and late components for the FC discrimination is highest at the HC level, whereas the strength of the difficulty component at the same coherence is low. For the LC level however, the strength of the early and late components is reduced while that of the difficulty component is increased. For the RG discrimination the strength of the early component is comparable with that of the FC discrimination for both the LC and HC level trials. The late component, however, is significantly reduced during the color discrimination regardless of coherence level. Finally, the strength of the difficulty component during the RG condition is low in all cases because the color discrimination is always trivial.

For simplicity, and considering the results of our single-trial EEG analysis (which is sensitive to signals up to 300 Hz), we modeled all of the high strength regressor amplitudes with a 1 and the low strength amplitudes with a 0.5. The significant reduction in strength for the late component during the color discrimination we modeled with a 0.2. Each regressor event was centered on the peak time of each component as identified during the EEG analysis (i.e., 170, 220, and 300 ms for the early, difficulty, and late components, respectively), and had a duration of 50 ms to match the presentation time of the stimulus. We also included two unmodulated regressors (all amplitudes set to 1), one centered at the onset time of the stimulus and one at the response time to absorb the variance of uncorrelated processes such as early visual processing and button pushes. Also note that both the early and late components were face-selective and therefore we applied this procedure on face trials only. Additionally, we added regressors to absorb the variance of all car trials as well as the variance of all of the cues.

In a separate analysis, we more precisely modeled the relative associations between our EEG components as well (e.g., late component better match to the psychophysical data than the early one). In this case, we used the actual output of the linear discriminator from the single-trial analysis of the EEG data. Specifically, we averaged the discriminator output [denoted “y” by Philiastides and Sajda (2006) and Philiastides et al. (2006)] associated with each component and each experimental condition across both trials and subjects. This ensured that we had a single value describing each component at each experimental condition. We also normalized these values to be in the range 0–1. We then used these values to modulate the amplitude of the different regressor events. Table 1 summarizes these values and compares them to the amplitudes used in the simplified design. Note that although the relative associations between the different trial types within each component regressor remained similar to the simplified analysis, the relative difference between the early and late components (e.g., for the same coherence level the strength of the late component is higher than that of the early) is also captured. The fMRI activation results using this design remained essentially unchanged when compared with those of the simplified version. This finding was not particularly surprising given the noisy nature of the fMRI and the fact that each regressor is ultimately processed to have zero mean in the general linear model (GLM) analysis. In light of these results, we used the simpler model for all subsequent analysis.

Table 1.

Normalized EEG discriminator amplitudes used to modulate the fMRI regressor heights for each component at each one of the four trial types.

	FC_LC	FC_HC	RG_LC	RG_HC
Early	0.24 (0.50)	0.53 (1.00)	0.24 (0.50)	0.53 (1.00)
Difficulty	0.36 (1.00)	0.15 (0.50)	0.15 (0.50)	0.15 (0.50)
Late	0.39 (0.50)	0.78 (1.00)	0.12 (0.20)	0.12 (0.20)

Note that, unlike the simpler model (values shown in parenthesis), this formulation also captures the relative association between the early and late components at each trial type. The fMRI results using this design remained essentially identical to those of the simplified model.

Data acquisition/analysis.

Whole brain fMRI data were collected from each subject on a 1.5T scanner (Philips Medical System, Aachen, Germany) measuring 24 slices of 64 × 64 voxels with an in-plane resolution of 3.125 mm, a slice thickness of 5.5 mm, a field of view (FOV) of 200 mm, echo time (TE) of 40 ms, and repetition time (TR) of 2.0 s. Initial scout scans were performed to localize slice planes parallel to the anterior commissure–posterior commissure line. Structural scans were performed using an SPGR sequence (80 slices; 256 × 256; FOV, 200 mm). fMRI data preprocessing included motion correction, slice-timing correction, high-pass filtering (≥50 s), and spatial smoothing (with a Gaussian kernel of 6 mm full-width at half maximum).

The main analysis was performed on whole-brain data within the framework of a GLM as implemented in FMRIB Software Library (Smith et al., 2004). All regressors were convolved with a hemodynamic response function (double-gamma function) before the GLM analysis. For each subject, we performed a first-level analysis to analyze each of the four trial blocks individually. Subsequently, we performed a level-two analysis to compute within subject group results (fixed effects). Finally, to compute across subject activation maps, we ran a higher-level analysis (mixed effects) to combine all level-two results. Statistically significant activations were determined using a cluster detection statistic, with a height threshold of z > 1.8 and a cluster probability of p < 0.05 corrected for whole-brain multiple comparisons using Gaussian random field theory. To demonstrate the consistency of the significant activations across subjects and to establish their uniqueness to each of the components of interest, we performed two additional analyses (see Results).

Localizer scans.

Separate localizer scans were performed to identify the fusiform face area (FFA) of all subjects, for subsequent region-of-interest analysis. Subjects passively viewed 12 alternating blocks of unmodulated, unmasked face and nonface images. Stimuli were presented for 750 ms with 250 ms interstimulus intervals, in blocks of 16 consecutive stimuli. A 12 s rest period was interleaved between blocks.

Results

Behavioral performance

In this study, we used the same paradigm as with our previous EEG recordings (Fig. 1a), using only two phase coherence levels, while simultaneously recording fMRI data. The HC level was chosen so that subjects performed nearly perfectly at the face-versus-car discrimination (suprathreshold performance) while the LC level was chosen so that each subject was at perithreshold performance (i.e., ≈80% correct) (Fig. 1b). All subjects were at suprathreshold performance during the color discrimination task regardless of coherence. The proportions correct for the three suprathreshold conditions were statistically indistinguishable (two-tailed t tests, all p > 0.2). The performance for the perithreshold condition, however, was significantly lower (left-tailed t tests, all p < 1 × 10⁻¹⁰).

Reaction times (RTs) were treated as a measure of the degree of difficulty between the different experimental conditions. The difficulty between the three suprathreshold conditions was equivalent as evident by their corresponding mean RTs (two-tailed t tests, all p > 0.75) (Fig. 1c). The mean RT for the low-coherence face-versus-car discrimination, however, was significantly higher than the other three (right-tail t tests, all p < 1 × 10⁻⁶).

We trained our subjects to ensure that this pattern of behavioral performance (both in terms of percent correct and RT) was also observed on an individual subject basis and was not merely an artifact of averaging across several subjects and experimental sessions. Training our subjects to perform consistently across sessions and across each other was critical for designing the fMRI analysis as outlined in Figure 2.

Early and difficulty component localization

For the early component, we observed significant activations in areas implicated in early visual processing. In particular, we found activations in the right and left superior temporal gyrus (STG) and superior temporal sulcus [STS; Brodmann area (BA) 39,41] as well as significant activations in the cuneus (CU; BA 17,18) (Fig. 3). In addition, we identified significant voxels in an area around the fusiform gyrus (FG; BA 37) bilaterally (with a right bias). We used a separate localizer scan to identify the FFA in each subject. We found that the early component explains a significantly higher percentage of the signal in the FFA than the difficulty and late components (right-tail, paired t tests, p < 6 × 10⁻⁶ and p < 2 × 10⁻⁶, respectively) (Fig. 4a).

Figure 3.

Areas correlating with our early, difficulty, and late components. Statistically significant activations are corrected for whole-brain multiple comparisons and are rendered on the mean brain across all subjects. Images are radiological convention.

Figure 4.

Fusiform gyrus activations. a, Mean percentage signal change in the FFA, as identified by separate localizer scans, across subjects and for each of the three components. The highest percent signal change in the FFA can be explained primarily by the early component. Error bars represent ±1 SEM. b, Fusiform gyrus activations correlating with the early (green) and late (yellow) components respectively as revealed by the group analysis. Significant activations are corrected for whole-brain multiple comparisons. Together these results suggest that the early component activations might have arisen primarily by the source seen for the localizer scans. The late component activations however are likely independent of those of the localizer scans.

For the difficulty component, a number of brain regions that are typically associated with the human attentional network were activated (Fig. 3). Specifically, we observed activations in the supplementary eye fields (SEF) and anterior cingulate cortex (ACC; BA 32), and activations in the frontal eye fields (FEF; BA 6) that extended ventrally into the dorsolateral prefrontal cortex (DLPFC; BA 9,46). We also observed significant anterior insular activations (AI; BA 13).

It was previously suggested that the ACC and DLPFC are functionally connected as activity in the ACC can predict activity in the DLPFC (Kerns et al., 2004). To test this hypothesis, we computed the mean percent signal changes in these regions (as explained by the difficulty component) for each subject separately and performed a correlation analysis. We indeed found a significant correlation between the signal changes in these regions (r = 0.8344; p < 8 × 10⁻⁴). We also found significant correlations between the ACC and AI (r = 0.7823; p < 3 × 10⁻³). The correlation results are depicted in Figure 5.

Figure 5.

Correlation results between the percent signal change (as explained by the difficulty component) in ACC and DLPFC (a) and ACC and AI (b). In both cases we found significant correlations suggesting that these areas are functionally connected.

Late component localization

More importantly, significant activations for the late component were identified in an area covering the inferior and middle occipital gyri bilaterally (BA 18,19) in a region commonly referred to as lateral occipital complex (LOC). We also found significant voxels in the right ventrolateral prefrontal cortex (rVLPFC), which extended into the orbitofrontal cortex (BA 11,47). Figure 3 illustrates these activations for the late component.

Because of the intersubject variability of face-selective areas in the fusiform gyrus, it is not typically considered common practice to display activations maps of these regions across subjects. Even with this variability however, it could still be quite informative to visualize these activations to get an estimate of their magnitude and their spatial extent. Interestingly, when we did this we found that a region in the anterior portion of the fusiform gyrus, which was clearly nonoverlapping with the activations seen for the early component, correlated significantly with the late component (Fig. 4b).

Given the confound of the intersubject variability, this finding cannot unequivocally be attributed to the existence of different subregions in the fusiform gyrus that are selective to each of the early and late components independently (i.e., selective to different cognitive stages). The fact however that the FFA activations, as identified by the separate localizer scans, do not reflect a correlation with the late component (Fig. 4a) suggest that this might be an interesting possibility. Although lack of better evidence prevents us from making strong claims regarding this observation, it is worth re-emphasizing the potentially restrictive nature of functional localizers (Friston et al., 2006), which are typically performed independent from and under dissimilar behavioral conditions than the main experiment.

Establishing activation consistency across subjects

To demonstrate the consistency of our results across subjects we performed an additional test. For each subject we applied a more lenient threshold (z = 1.8, uncorrected) to the statistical parametric maps generated using the fixed-effects group analysis and binarized them (i.e., set all voxels that passed threshold to 1 and 0 otherwise). We then summed the binary maps across all subjects. The range of values for each voxel was 0–12 (i.e., the total number of subjects). We repeated this process for each of the three components (early, difficulty, and late) and took a closer look at the voxels that had passed significance across subjects (Fig. 3). The maps in Figure 6a illustrate the results of this test and demonstrate that the activations were consistently present across subjects.

Figure 6.

Establishing subject consistency and activation significance. a, Consistency of activations for each of the three components across subjects. Colored voxel represent the number of subjects that showed significant activations. A voxel value of 12 indicates that voxel was activated in all subjects in our dataset. b, Unthresholded z-map for a permutation test where we randomly relabeled all trials, remodeled the fMRI data based on the principles of Figure 2 and repeated the group analysis. The original activations for the early, difficulty and late components, the boundaries of which are depicted by the green outlines, are no longer present (no other activations survive after correction for multiple comparisons either). This negative result reinforces the significance of the original effects which appear to only correlate with the “true” trial labels.

Establishing component separability

To substantiate the claim that our design/model is suitable to separate fMRI signal changes related to each of our main components of interest (i.e., early and late), we ran two supplementary analyses. Specifically, we separately orthogonalized each component regressor with respect to the other and re-run the analysis (i.e., early with respect to late, and vice versa). This manipulation ensured that the common variance was absorbed by the regressor we orthogonalized to each time. We found that in both cases each of the significant activations remained attached to the EEG component that was originally associated with. We view this finding along with the consistency in which these activations appeared in each subject as strong evidence of the separability of our components.

Another important point to consider is whether the time differences between the EEG components (i.e., 170, 220, and 300 ms) contribute to the separability of their corresponding regressors. Therefore, we also analyzed the data such that the onset times of all three EEG components was identical (set at stimulus onset) and we found that the results were quite similar to those of the original analysis. It appears that the different components are primarily separated by the amplitude manipulation of the regressor events rather than time. The main reason we centered our regressor events around the actual component time was to be precisely consistent with our EEG findings.

Establishing activation significance/uniqueness

To further establish the significance of these activations we performed an exercise motivated by a traditional machine learning approach. We performed this exercise much in the same way one would perform a bootstrapping test, where the trial labels are randomly permuted and the main effect is reanalyzed a number of times. If the original activations simply arose by chance one might expect to see the same activations appear again. However, if the activations are significant then they would only correlate with the “true” trial labeling and they would not appear in these permutation tests.

For each subject, we randomly permuted all trial labels and re-designed the analysis using the EEG-informed fMRI design depicted in Figure 2. We then performed a mixed-effects group analysis as before to acquire group results. We repeated this process a number of times and in all cases the activations seen earlier for the early, difficulty, and late components were no longer present. Figure 6b shows group results from an unthresholded z-map for one of these permutation tests. The boundaries of the original activations are overlaid on the z-map, which clearly shows that these regions no longer correlate significantly with our three components of interest. We view this finding as additional evidence that the main activations are indeed significant.

Unmodulated regressor activations

The two unmodulated regressors captured activations from a number of additional regions. Statistical significance was established in the same manner as with the rest of the activations (i.e., using a cluster statistic and corrected for multiple comparisons). Cortical activations in the precentral gyrus (primary motor cortex; BA 4), regions of the somatosensory cortex in the postcentral gyrus (BA 1,2,3) and activations in the putamen and the anterior portion of the DLPFC (corresponding to the superior frontal gyrus, BA 10) correlated primarily with the regressor placed at the reaction time. For the regressor centered on the onset of visual stimulation we observed earlier visual activations around the middle temporal gyrus (MTG) and middle occipital gyrus. Specifically, we found bilateral activations in the temporo-occipital junction (BA 37) along with activations in the dorsal portion of the MTG (BA 39). In addition we found a right-hemisphere activation in the ventral portion of the MTG (BA 21). Early visual cortex activations (BA 17/18) also correlated with this regressor, although ultimately they did not pass the cluster statistic.

Discussion

In this study we used information derived from our previous EEG recordings to drive the analysis of fMRI data collected during a perceptual decision making task. Specifically, this EEG-informed fMRI analysis was intended to ascertain the cortical origins of each of our temporally specific EEG components (i.e., early, difficulty, and late) that they would otherwise be unobtainable given the low spatial resolution of the EEG and the ill-posed nature of the inverse problem. Identifying the regions associated with each component will allow one to start constructing a spatiotemporal diagram of the events associated with perceptual decision making in the human brain.

Early visual perception

Based on our previous EEG findings, we have always speculated that our early component (≈170 ms) is related to an early visual perception event. Our fMRI results substantiate that claim. We found bilateral activations (with a right bias) in the FFA that covaried with our early component (Fig. 4). In addition, we found significant activations in the STG/STS bilaterally as well as activations in the CU (Fig. 3).

Both the FFA and STG/STS have been implicated previously in early visual processing of faces using neuroimaging (Puce et al., 1995, 1998; Kanwisher et al., 1997; Haxby et al., 2000; Hoffman and Haxby, 2000) and field potential recordings on the cortical surface (Allison et al., 1999; Puce et al., 1999b,a). Moreover, it was proposed that the sources of the face-selective N170 component seen in electrophysiological recordings are generated in these regions (Horovitz et al., 2004). Our data provide additional support for this claim. Finally, the CU is also known to be involved in early visual processing, and it has been reported extensively as such in the neuroimaging literature (Cohen et al., 1997; Ganis et al., 2004; Heekeren et al., 2004), including studies on visual extinction and spatial neglect of faces (Vuilleumier et al., 2001).

Task difficulty and the influence of attention

The difficulty component in our EEG is found ≈220 ms after the onset of visual stimulation. The strength of this component is highly correlated to the difficulty of the task and it predicts the onset of the late component. We also showed that this component is likely to represent a true top-down influence of attention on decision making, rather than a mere bottom-up processing of the stimulus, by virtue of the fact that, for the same stimulus evidence, it is reduced during an easy color (red vs green) discrimination (Philiastides et al., 2006). In the fMRI, we found activations that correlated with this component at ACC, SEF, FEF, DLPFC, and the AI. All of these regions have previously been associated with the attentional network in the human brain and it is not surprising that they covaried with the difficulty component.

The ACC has been shown previously to participate in conflict monitoring and error detection and to signal the need for greater cognitive adjustments (Carter et al., 1998; MacDonald et al., 2000; Kerns et al., 2004; Brown and Braver, 2005). In addition, experiments in primates and humans found attention-related fluctuations in traditionally occulomotor structures such as the SEF and FEF (Corbetta et al., 1998; Schall et al., 2002; Kelley et al., 2007). Microstimulation experiments in primates have also revealed causal links between occulomotor control areas in the frontal cortex and regions primarily involved in specifying the visual properties of the stimulus (Moore and Armstrong, 2003; Moore et al., 2003). Therefore, it is not unlikely that there are reciprocal interactions between the attention-related activations we see here and the areas involved in early visual perception that correlated with our early component.

The DLPFC was also shown to exert attentional control by representing and maintaining the attentional demands of a task (MacDonald et al., 2000; Milham et al., 2001). DLPFC was also implicated in directing efficient (Bor et al., 2003; Olesen et al., 2004) and successful (Pessoa et al., 2002; Sakai et al., 2002) working memory encoding and maintenance. Furthermore, the DLPFC appears to be functionally connected to ACC as activity in ACC was shown to predict signal changes in DLPFC (Kerns et al., 2004). Indeed, we too found such correlations between activity in the ACC and both DLPFC and AI (Fig. 5). Although these results suggest that these regions covary together, it still remains unclear whether the correlations between them reflect a direct link or the common influence of some other region(s).

Finally, the AI appears to be involved in the integration of multimodal information as evident by its widespread efferent and afferent projections to and from both the frontal and parietal cortices (Mesulan and Mufson, 1982a,b; Selemon and Goldman-Rakic, 1988). Furthermore, anterior insular activations were recently shown to covary with categorical uncertainty (Grinband et al., 2006) during decision making.

We also found, in the EEG, an association between the difficulty and late components, namely that the strength of the difficulty component predicts the onset time of the late component (which shifts later in time as a function of the task difficulty). We also showed that the difficulty component represents a top-down influence of attention on decision making (Philiastides et al., 2006). Consistent with this finding is evidence suggesting that there are task-related top-down influences of attention and task demands (Hopf et al., 2006; Large et al., 2007), as well as working memory load (Rose et al., 2005) on the activation of neural populations in the LOC as a whole, an activation that was shown to covary with our late component. These results not only reinforce the finding that LOC is associated with the late component but they also speak to the interconnectivity (direct or otherwise) between the LOC and the regions that correlated with the difficulty component (e.g., ACC, DLPFC, AI, etc.)

Perceptual persistence in decision making

There is converging evidence that the ventral stream of visual processing plays a critical role in the long-term representation and categorization of objects. A number of neuroimaging studies have consistently shown that the LOC, part of the ventral visual stream, is highly active during object perception and recognition (Malach et al., 1995; Halgren et al., 1999; Grill-Spector et al., 2000, 2001; James et al., 2000, 2002; Grill-Spector et al., 2004). Additional evidence implicated the LOC in “perceptual persistence,” where a percept assembled by lower-visual areas can be stored at least briefly in the LOC (Ferber et al., 2003; Large et al., 2005) even after the stimulus has been removed. It has also been proposed that a possible mechanism of such “persistence” involves feedback pathways in the ventral stream allowing the representation of a stimulus to remain in the visual system via “local” loop reverberations (Lollo, 1977; Coltheart, 1980; Van-Rullen and Koch, 2003), even after the corresponding objects have disappeared from the retina.

Evidence from our original EEG experiments suggests that the late component is more closely associated with a postsensory/decision-related event. Unlike the early component, whose onset time remains unchanged regardless of the task difficulty, the late component systematically shifts later in time as a function of difficulty. At the same time EEG activity becomes more persistent as evident by the increased temporal spread/duration of this component [see Philiastides et al. (2006), their Fig. 9]. We originally proposed a decision making strategy that allocates a fixed amount of time for the bottom-up processing of the stimulus (early component) while allowing a variable amount of time for the top-down evaluation of the evidence (late component). In addition we interpreted the delayed onset and temporal persistence of the late component across difficulty conditions as additional time in a “loop,” consistent with the hypothesis of perceptual persistence by means of local reverberations. The fact that LOC, which was shown to be critically involved in perceptual persistence during object categorization, correlated significantly with the late component provides strong support to this interpretation.

The concept of perceptual persistence is especially convincing considering the brief stimulus durations used here (50 ms), which essentially necessitate the presence of a process that will allow the stimulus to reverberate. Although it remains to be seen whether such process applies to longer presented stimuli, it currently explains why the late component is a better match to the behavioral data and why it disappears when a trivial nonobject categorization is required (i.e., color categorization). It also reconciles the fact that, in the EEG data, the late component appears more correlated with the stimulus than the response and as such is unlikely to reflect the final output of the decision stage. Instead the late component can be thought of as part of a postsensory process that ultimately drives the final stage of decision making (i.e., accumulation to threshold), which is why we found a strong correlation between the strength of this component and the mean drift rate in a diffusion model (Philiastides et al., 2006). In fact, diffusion model simulations have recently suggested that for briefly presented visual stimuli, the mean drift rate would lie somewhere in visual cortex able to maintain a short-term memory/representation for integrating evidence about the decision (Smith et al., 2004). Our findings are consistent with this interpretation and they suggest that LOC is a likely candidate for this type of processing.

In addition, the order in which the late component appears in the EEG is consistent with the order in which LOC appears in the information processing stream in the ventral visual pathway. Specifically, our diffusion model simulations placed this component at the end of early visual perception and at the onset time of the diffusion process (Philiastides et al., 2006). In line with our findings, novel evidence from experimental findings suggest that occipitotemporal areas in the ventral visual pathway form a cascade of stages with increasing degree of signal uncertainty and place the LOC at the end of this cascade (VanEssen et al., 1992; Lennie, 1998; Schmolesky et al., 1998; Grill-Spector et al., 1999).

Recall that our late component was face selective, which is why we included only face trials in our original EEG-informed fMRI design. This finding however could be merely attributable to the strong neural generators for faces (compared with other objects), which in turn yield strong dipolar fields easily recorded with scalp electrodes. It therefore remains possible that the neural correlates of this component could generalize well across other object classes that have not been shown to generate as strong EEG signatures. For this reason we rerun the analysis (with the same design used for face trials) (Fig. 2) using only car trials. We found a similar activation in the LOC (Fig. 7) in a region dorsally to the one activated for faces. This finding suggests that, during perceptual decision making, LOC might represent the perceptual persistence of objects in general with a subregion [LO-faces (Grill-Spector et al., 2004)] dedicated to faces.

Figure 7.

LOC activations seen for the late component during the analysis of face (yellow) and car (blue) trials. Activations are corrected for whole-brain multiple comparisons and are rendered on the mean brain across all subjects.

The other activation that correlated with our late component was in the rVLPFC. Current evidence implicates VLPFC in decision making tasks that involve uncertainty or risk (Goel and Dolan, 2000; McClure et al., 2004). Moreover, activity in VLPFC appears to subserve the active maintenance and retrieval of information held in short-term memory to facilitate various executive processes such as active selection, comparison and/or judgment of stimuli (Petrides, 2002; Prince et al., 2005; Hanakawa et al., 2006). Hence, the VLPFC appears to be in line with the perceptual persistence interpretation (also seen in LOC) as a means of driving the final decision making stages.

In terms of effective and functional connectivity, there are a number of studies, ranging from anatomical investigation of cortical connectivity in primates to transcranial magnetic stimulation in humans, which demonstrate that the VLPFC has extensive cortical connections to ACC, DLPFC, and the insula (Cavada et al., 2000; Cohen et al., 2005; Rounis et al., 2006). These findings provide further support of the influence of the difficulty component (i.e., ACC, DLPFC, AI) on the late component (VLPFC). Moreover, the VLPFC was also shown to receive (primarily nonspatial) visual information from the inferior temporal cortex (“what” pathways) (Ungerleider et al., 1998).

Beginnings of a spatiotemporal diagram

Figure 8 summarizes our findings in a form of a spatiotemporal diagram. Although our EEG-informed fMRI approach may not produce a unique set of neuronal sources (as is often the case with traditional fMRI experiments as well), enables the fusion of these modalities in a manner that circumvents the disparate time scales on which the two datasets are acquired. Using this technique we were able to identify the cortical regions that correlated with each of our EEG temporally specific components and in doing so we demonstrated that a cascade of events associated with perceptual decision making takes place in a highly distributed neural network. These include early visual perception (early component), task/decision difficulty (difficulty component) and postsensory/decision-related events (late component). Furthermore, taking into consideration anatomical and functional connectivity evidence from the literature we were able to start making inferences regarding the interconnectivity between the participating regions.

Figure 8.

Beginnings of a spatiotemporal diagram of the processes involved in perceptual decision making. Using an EEG-informed fMRI analysis we were able to identify the cortical regions correlating with our early, difficulty and late EEG components and begin making inferences about their interconnectivity. For details, see Discussion.