The evolution of a disparity decision in human visual cortex

Abstract

We used fMRI-informed EEG source-imaging in humans to characterize the dynamics of cortical responses during a disparity-discrimination task. After the onset of a disparity-defined target, decision-related activity was found within an extended cortical network that included several occipital regions of interest (ROIs): V4, V3A, hMT+ and the Lateral Occipital Complex (LOC). By using a response-locked analysis, we were able to determine the timing relationships in this network of ROIs relative to the subject's behavioral response. Choice-related activity appeared first in the V4 ROI almost 200 ms before the button press and then subsequently in the V3A ROI. Modeling of the responses in the V4 ROI suggests that this area provides an early contribution to disparity discrimination. Choice-related responses were also found after the button-press in ROIs V4, V3A, LOC and hMT+. Outside the visual cortex, choice-related activity was found in the frontal and temporal pole before the button-press. By combining the spatial resolution of fMRI-informed EEG source imaging with the ability to sort out neural activity occurring before, during and after the behavioral manifestation of the decision, our study is the first to assign distinct functional roles to the extra-striate ROIs involved in perceptual decisions based on disparity, the primary cue for depth.

1. Introduction

For primates, one of the main cues to depth perception is horizontal disparity, the difference between the retinal coordinates of a given feature. Over the last decades, the cortical mechanisms for disparity processing have been well investigated in macaque using single-cell recording (Hubel and Wiesel, 1970; Poggio and Poggio, 1984; Cumming and DeAngelis, 2001) and in human using fMRI (Backus et al., 2003; Neri et al., 2004; Durand et al., 2009). Disparity processing is important for perception and action (Melmoth et al., 2009), but the neural basis of its contribution to behavior is poorly understood. Electrophysiological studies, supported by microstimulation in the recorded areas, have demonstrated causal effects in both ventral and dorsal visual pathways in the decision process for various disparity tasks (DeAngelis et al., 1998; Uka and DeAngelis, 2006; Shiozaki et al., 2012). However, only a few areas have been explored, and none of the single-cell studies have recorded neural responses from multiple areas at the same time, making it difficult to characterize the entire cortical network involved in disparity-based decisions. With its high spatial resolution and large field of view, fMRI permits precise localization of the areas whose responses are related to disparity judgments (Chandrasekaran et al., 2007). However, the slow dynamics of the BOLD response does not allow precise characterization of the sequence of activity leading to the subject's response. Given its temporal resolution on the order of milliseconds, a technique like EEG may provide a better tool for deciphering the temporal characteristics of decision-making. EEG has been used to investigate perceptual judgments in various tasks (VanRullen and Thorpe, 2001; Philiastides and Sajda, 2006), including disparity tasks (Kasai and Morotomi, 2001). These studies have analyzed evoked responses at the scalp, which makes it challenging to determine the cortical areas involved in the decision.

In the current study, we used a high-density EEG imaging technique, which when coupled to fMRI-defined regions of interest (ROIs), allowed us to examine the dynamics of the responses directly at the cortical level (Cottereau et al., 2012a). The subjects performed a reaction-time disparity discrimination task. We were particularly interested in the decision-related activity within those ROIs whose disparity tuning properties we had previously characterized: V1, V4, V3A, Lateral Occipital Complex (LOC) and hMT+ (Cottereau et al., 2011; Cottereau et al., 2012bc). Using a response-locked analysis (see also Ales et al., 2013), we were able to establish that among our five visual ROIs, the V4 ROI is the first to exhibit decision-related activity. We also found that all the extra-striate visual areas exhibit significant post-decision activity.

2. Materials and Methods

2.1. Subjects

The 11 participants (6 males, 5 females, age range, 24-69 years) were volunteers, with normal stereopsis and normal or corrected-to-normal visual acuity. They were given instructions and detailed information about the experiments and provided written informed consent before participating in the study in accordance with Helsinki Declaration; the human subjects review committee of the Smith-Kettlewell Eye Research Institute approved the study.

2.2. Stimulus display

Stereoscopic stimuli were displayed using a system in which orthogonally polarized images from two matched Sony Trinitron monitors (Model 110GS), were combined via a beamsplitter and viewed through appropriately oriented polarized filters placed immediately in front of the eyes. Each eye could see the image from only one screen; the viewing distance was 80 cm. Each screen had a resolution of 1024 by 768 pixels and was refreshed at 85Hz. The luminance of the background was 4.52 cd/m². The luminance of the dots was 85.88 cd/m².

2.3. Experimental protocol

The task consisted of a disparity discrimination judgment. The base stimulus (Fig. 1) was a 7.5-degree diameter central disk, surrounded by a large annulus (15 degree diameter); both were composed of dynamic random dots (90% contrast) that were refreshed every 47 ms (21.25 Hz). Each dot consisted of a square of 6.5 arcmin on a side. The dot density was 30 dots per square degree of visual angle. The central disk alternated at 1 Hz (square wave) between a fixed crossed disparity of 5 arcmin and the fixation plane (0 arcmin). Intermittently (30% of time), the disparity value was increased to (5+δd) arcmin (“Odd step”). The subjects were asked to detect these changes by pressing a button with their right index finger. The disparity value of non-target stimuli was the same for each subject and was set to 5 arcmin. The size of the odd step δd was determined individually prior to the EEG session in order to obtain 80% correct discrimination of the incremental change in disparity (‘Hits’). Subjects did not receive feedback regarding the correctness of their responses.

Figure 1.

Experimental protocol. a) Left and right monocular images used to define the center-surround display. Nonius lines are provided above (left eye) and below (right eye) the fixation point to facilitate fusion. The dots are refreshed at 21.25Hz. b) 3D view of the stimulus. c) Temporal properties. The disk moves between 0 and 5 arcmin (crossed domain) at 1Hz. 30% of the time, the disparity increment equals (5 + δd) arcmin and the subject has to detect the event and press the button.

To facilitate fusion of the two monocular images, the stimuli also contained a pair of nonius lines (one in each eye) and a binocularly visible fixation point superimposed on the center of the disk-annulus (see Fig. 1). These nonius lines, combined with the fixation point and the large static annulus constituted a stable zero-disparity reference that permitted the subjects to maintain their fixation at the horopter during the disparity step of the disk. To assess the stability of fixation, we asked our subjects if they experienced misalignment of the nonius lines during the recordings. All of them reported that the lines remained aligned. Given that the sensitivity for nonius misalignment is typically below 2 arcmin (McKee and Levi, 1987), we conclude that eye position was not driven by the stimulus. Previous psychophysical measurements of fixation stability (Cottereau et al., 2011) confirmed that subjects can hold their fixation during the type of disparity modulations presented here. The recordings were performed in blocks of continuous trials that lasted 11 seconds (i.e. 11 trials per block). No odd steps were displayed during the first second. There was at least one non-target trial between two odd step trials. The inter-block interval was 1 second. For each subject, data collection was continued until the subject reached at least 300 Hits, which typically led to recording sessions of roughly 45 minutes.

2.4. EEG signal pre-processing

The electroencephalogram (EEG) data were collected with 128-sensor HydroCell Sensor Nets (Electrical Geodesics, Eugene OR) and were band-pass filtered from 0.3 to 50 Hz. EEG artifacts were eliminated off-line using standard procedure whose description can be found in (Cottereau et al. 2012c). Before the source imaging procedure (see section 2.8), data were segmented into one-second trials corresponding to one full cycle of the 1Hz disparity display. At t=0, the centre disk is presented with either a crossed disparity of 5 or (5+δd) arcmin for 500 ms and then is returned to the fixation plane for another 500 ms. These one-second trials were baseline corrected by subtracting the average activity over the 100 ms directly preceding their beginning. Trials corresponding to the first second of the stimulus were discarded from the analysis, as they never contain the odd step. The remaining trials were then sorted into four distinct categories: 1) Hits (correct detection of a target), 2) Misses (missed target), 3) Correct Reject (correct detection of a non-target disparity) and 4) False alarms (detection of a non-target disparity). As we will see in the results section, the number of false alarms was too small to permit a proper analysis. This category is therefore not discussed in this study. The Hit responses were also analyzed after temporally alignment to the button press. In this case, trials consisted in one second centered on the subject's response. Once again, these trials were baseline corrected by subtracting the average activity over the 100 ms directly preceding the stimulus onset (i.e. before alignment to reaction time). To eliminate the contribution due to the random dot refresh-rate (f = 21.25 Hz), all data were low-pass filtered at 20 Hz using a zero-phase filter. Because filtering can affect the estimation of onset latencies (VanRullen, 2011) the timing described in this study were double-checked against the unfiltered data. The onset latencies were identical between filtered and unfiltered data.

2.5. Structural and functional Magnetic Resonance Imaging (MRI)

Structural and functional MRI scanning was conducted at 3 T (Siemens Tim Trio, Erlangen, Germany) with a 12-channel head coil. We acquired a T1-weighted MRI data set (3D MP-RAGE sequence, 0.8 × 0.8 × 0.8 mm³ resolution) and a 3D T2-weighted data set (SE sequence at 1 × 1 × 1 mm³ resolution) for tissue segmentation and registration with the functional scans. For functional MRI (fMRI), we employed a single-shot, gradient-echo planar imaging (EPI) sequence (TR/TE = 2,000/28 ms, flip angle 80, 126 volumes per run) with a voxel size of 1.7 × 1.7 × 2 mm³ (128 × 128 acquisition matrix, 220-mm field of view, bandwidth 1,860 Hz/pixel, echo spacing 0.71 ms). We acquired 30 slices without gaps, positioned in the transverse-to-coronal plane approximately parallel to the corpus callosum and covering the whole cerebrum. Once per session, a 2D SE T1-weighted volume was acquired with the same slice specifications as the functional series in order to facilitate registration of the fMRI data to the anatomical scan. The general procedures for these scans (head stabilization, visual display system, etc.) are standard and have been described in detail elsewhere (Brewer et al., 2005). The FreeSurfer software package (http://surfer.nmr.mgh.harvard.edu) was used to extract both gray/white and gray/cerebrospinal fluid (CSF) boundaries. These surfaces can have different curvatures. In particular, the gray/white boundary has sharp gyri (the curvature changes rapidly) and smooth sulci (slowly changing surface curvature), while the gray/CSF boundary is the inverse, with smooth gyri and sharp sulci. To avoid these discontinuities, we generated a surface partway between these two boundaries that has gyri and sulci with approximately equal curvature. This “midgray” cortical surface consisted in a triangular tessellation of 20,484 regularly spaced vertices and was used to define the visual areas and the source space for the EEG current modeling (see section 2.7).

2.6. Functional area definition

Retinotopic field mapping using rotating wedges and expanding/contracting rings produced ROIs corresponding to visual cortical areas V1, V2v, V2d, V3v, V3d, V3A, and V4 in each hemisphere (Tootell and Hadjikhani, 2004). ROIs corresponding to hMT+ were identified with low-contrast motion stimuli similar to those described in (Huk and Heeger 2002). The LOC was defined with a block-design fMRI localizer scan. During this scan, the observers viewed blocks of images depicting common objects (12 s/block) alternating with blocks containing scrambled versions of the same objects. The stimuli were those used in a previous study (Kourtzi and Kanwisher, 2002). The regions activated by these scans included an area lying between the V1/V2/V3 foveal confluence and hMT+ that we identified as LOC. This definition covers almost all regions (e.g., V4d, LOC, LOp) that have previously been identified as lying within object-responsive lateral occipital cortex (Kourtzi and Kanwisher, 2002). The location of the functional visual areas used in this study is provided in supplementary figure 1.

2.7. Forward modeling of the cortical currents

For each subject, the EEG source space was given by his/her “midgray” cortical surface tessellation (see above) and consisted in 20,484 regularly spaced vertices. The distance between connected vertices was on average 3.7mm, with standard deviation of 1.5 mm and range 0.1–11 mm. Current dipoles were placed at each of these vertices. Their orientations were constrained to be orthogonal to the cortical surface to diminish the number of parameters to be estimated in the inverse procedure (Hämäläinen et al., 1993). The FSL toolbox (http://www.fmrib.ox.ac.uk/fsl/) was used to segment from the individual T1- and T2-weighted MRI scans contiguous volume regions for the inner skull, outer skull, and scalp. These MRI volumes were then converted into inner skull, outer skull, and scalp surfaces (Smith 2002; Smith et al., 2004) that defined the boundaries between the brain/CSF and the skull, the skull and the scalp, and the scalp and the air. Following each EEG recording session, the 3D locations of all electrodes and three major fiducials (nasion, left and right peri-auricular points) were digitized using a 3Space Fastrack 3-D digitizer (Polhemus, Colchester, VT). For all observers, the 3D digitized locations were used to co-register the electrodes to their T1-weighted anatomical MRI scans. The source space, the 3D electrode locations, and the individually defined boundaries were then combined with the MNE software package (http://www.nmr.mgh.harvard.edu/martinos/userInfo/data/sofMNE.php) to characterize the electric field propagation with a three-compartment boundary element method (BEM) (Hämäläinen and Sarvas, 1989). The resulting forward model is linear and links the activity of the 20,484 cortical sources to the voltages recorded by our EEG electrodes. Mathematically, this forward model can be written as:

$$\mathbf{M}(\mathrm{t})=\mathbf{G}\mathbf{J}(\mathrm{t})+\varepsilon (\mathrm{t}),$$

where M is a column vector containing the m measurements on the EEG or MEG sensor array at instant t; J is a column vector of the n unknown source amplitudes of all elementary sources in the model with zero mean and a co- variance matrix R (size n * n); G (size m * n) is the forward gain matrix sampled at the sensor array and ε (size m * 1) is an additive nuisance term with zero mean and a covariance matrix C (size m * m).

2.8. fMRI–informed inverse modeling of the cortical currents

A classical solution to the forward model described above is to use an L2 minimum-norm inverse (Hämäläinen et al., 1993). In this case, the solution has a closed form and can be written:

$$\widehat{J}=R{G}^{\prime }{\left(GR{G}^{\prime }+{\lambda }^{2}C\right)}^{-1}M$$

where λ is a regularization parameter. In absence of any prior on the source distribution, the source covariance matrix R is often equal to the identity matrix. In our case, we introduce our knowledge of the functionally defined visual ROIs into this matrix. Our aim was to decrease the tendency of the minimum-norm procedure to smooth activity over very large surfaces and across different functional areas. Two modifications were applied: 1) We increased the variance allowed within the visual areas by a factor of 2 relative to other vertices (i.e. we multiplied the diagonal elements of R corresponding to visual sources by a factor of 2), and 2) we enforced a local correlation constraint within each area using the first- and second- order neighbors on the cortical tessellation with a weighting function equal to 0.5 for the first order and 0.25 for the second (i.e. the off-diagonal elements of R corresponding to neighbor sources belonging to the same functional ROI were increased). This modification of the correlation matrix R therefore respects both retinotopy and areal boundaries and permitted us to dissociate the signals from different areas, unlike other smoothing methods such as LORETA that apply the same smoothing rule throughout cortex (Pascual-Marqui et al., 1994). More details of this approach can be found in Cottereau et al. (2012c). The value of the regularization parameter was estimated using a generalized cross-validation approach (Babiloni et al., 2004). Once the current density was obtained for each cortical source, ROI-level responses were computed by averaging the time courses from all the sources within each ROI.

2.9. Cross-talk

In previous studies (Cottereau et al., 2011; Cottereau et al., 2012bc), we described how we estimated the theoretical cross talk among ROIs for a specific EEG study. Cross talk refers to the neural activity generated in other ROIs that is attributed to a particular ROI, due to the smoothing of the electric field by the head volume. In brief, for each subject, we simulated the cross talk by placing sources in one ROI and estimating their contribution to other ROIs, using the same forward and inverse methods described above. The global cross talk matrix (i.e., averaged across all the subjects who participated in our EEG experiment) is shown in Fig. 2 for seven ROIs (V1, V2, V3, V4, LOC, V3A, and hMT+); the cross talk magnitude shown in the matrix is proportional to activity originating in the ROI where the cross talk is being estimated.

Figure 2.

Simulation estimates of cross talk between source-imaged EEG signals in retinotopically defined ROIs. Grayscale values at row i and column j represent the relative contribution of ROI j to the cortical current density estimate in ROI i. The 5 visual regions of interest (ROIs) discussed in the present study are emphasized by the white dot square. LOC, lateral occipital complex.

From our simulations, it was apparent that there was significant cross talk in ROIs V2 and V3 (the last 2 rows of the matrix). For this reason, we excluded these two ROIs from our analysis and focused on the five other ROIs: V1, V3A, V4, hMT+, and the LOC. These ROIs are more widely separated, and their estimated activities are therefore more reliable. We discuss the influence of cross-talk on our study in section 3.5.

2.10. Cross-subject surface averaging and characterization of the activity in the hand area of the motor cortex

To define an ROI in which to quantify responses to a button-press of the right index finger, we localized the hand area of the motor cortex using cross-subject, surface-based averaging. Averaging was performed with the FreeSurfer image analysis suite (http://surfer.nmr.mgh.harvard.edu/) as documented in (Fischl et al., 1999; Yeo et al., 2010a; Yeo et al., 2010b). First, each subject's cortical surface was inflated to a sphere. Using the pattern of gyral and sulcal curvature, each subject's cortical surface was then aligned to a reference subject. This alignment defines a unique unidirectional transform from each subject to the target subject. The alignment was then used to average the source data across subjects. For cross-subject, surface-based averaging only, the source reconstructions were smoothed over 1^st and 2^nd order neighboring vertices, resulting in a boxcar smoothing 4mm in width. The motor cortex ROI was defined from this average activity as the source with the most significant activity within the hand area in M1 of the contralateral hemisphere (i.e. the left hemisphere) at the instant of the button-press. The hand area in M1 lies at the intersection between the precentral and superior frontal sulcus within an ‘inverted-omega-shaped’ folding of the precentral sulcus (Yousry et al., 1997; Meier et al., 2008).

2.11. Statistical analysis

Differences between the experimental conditions were identified by a permutation test based on methods devised by Blair and Karniski (1993) and described in detail in Appelbaum et al. (2006). Briefly, the null hypothesis that no differences were present between experimental conditions was tested by making synthetic datasets in which the condition labels for an individual subject's data were randomly permuted. For each permutation we calculated t-scores of the difference and found the longest run of consecutive time points with p-values less than 0.05. This procedure provides a nonparametric reference distribution of consecutive significant p-values. We then rejected the null hypothesis if the length of any consecutive sequence of significant t-scores in the original, nonrandomized data exceeded 95% of the values in the null distribution. Because each permutation sample contributes only its longest significant sequence to the reference distribution this procedure implicitly compensates for the problem of multiple comparisons, and is a valid test for the omnibus hypothesis of no difference between the waveforms at any time point. Furthermore, this test not only detects significant departures from the null hypothesis, it also localizes the time periods when such departures occur. However, since the correction procedure is tied to the length of the data and the somewhat arbitrary choice of keeping family-wise error at 5%, we therefore also present the uncorrected significance values (see red/yellow color maps in Fig. 4, 5, 7, and 9 and in the supplementary figures 6 and 7). By evaluating the data using both statistical approaches, we are better able to identify time periods when the responses depart from the null hypothesis. We tested whether reaction time had an effect on the slopes of the responses in the V4 ROI (see section 3.4 and figure 11) using ANOVAs. The independence-of-variance assumption underlying the validity of the ANOVAs was evaluated with Mauchly's sphericity test. The sphericity tests were not significant, so we did not correct the p-values for the ANOVAs.

Figure 4.

Stimulus-locked responses. Comparison between Hits (green) and Correct reject (black) in five visual ROIs (n = 11 subjects). The shaded areas indicate SEM. The colored bars provide the p-values (uncorrected) for the Hits/CR difference. Stars indicate periods that pass run-length correction for multiple comparisons.

Figure 5.

Stimulus-locked responses. Comparison between Hits (green) and Misses (red) in five visual ROIs. See Fig. 4 for the details of the legend.

Figure 7.

Response-locked responses. Comparison between the true response-locked data (blue) and a surrogate dataset computed from the ‘Misses’ (red) in five visual ROIs (n = 11 subjects). The shaded areas indicate SEM. The colored bars provide the p-values (uncorrected) for the Hits/Correct rejects difference. Stars indicate periods that pass run-length correction for multiple comparisons.

Figure 9.

Response-locked responses. Comparison between the true response-locked data (blue) and a surrogate dataset computed from the ‘Hits’ (green) in five visual ROIs (n = 11 subjects). See Fig. 7 for the details of the legend.

Figure 11.

Slopes of the ramping down activity for the slow (white), medium (light gray) and fast (dark gray) trials in the V4 ROIs (n = 11 subjects). The bars give the standard errors.

3. Results

We recorded high-density EEG while subjects performed a disparity discrimination reaction-time task. During a typical trial (see Fig. 1 and section 2.4), a disparity-defined disk alternated its disparity at 1Hz from the fixation plane to a crossed value of 5 arcmin (500 ms in each state). 30% of the time, the disparity onset was increased to 5+δd arcmin. Subjects (n =11) were instructed to detect this disparity increment and press a button with their right index finger as quickly as possible to indicate the detection. Across subjects, the disparity increment δd was equal to 6.7 arcmin on average (2.7 arcmin of standard deviation, range, 2-12 arcmin). On average, subjects detected 83% of the targets (6.8% of standard deviation). The average number of total trials, ‘Hits’, ‘Misses’, ‘Correct Rejects’ and ‘False alarms’ were equal to 1315, 411, 85, 800 and 19 trials per subject, respectively. The corresponding standard deviations were equal to 57, 48, 30, 42 and 13. The average number of false alarms was too small to permit a proper analysis of this category of trials and we therefore do not discuss it in this study.

In the following, we separately analyze target-specific activity whose timing is tied to the onset of the stimulus and target-specific activity that is associated with the time of the behavioral response. In the former case, the analysis is time-locked to the stimulus onset time (stimulus-locked). For the latter, the analysis is time-locked to the behavioral response time (response-locked). We will refer to activity specifically associated with a correct detection of the target (i.e. present in the ‘Hits’ but not in the ‘Misses) as ‘choice-related’ activity.

3.1. Evoked activity locked to the stimulus onset

Sensor level signals

We first describe the EEG signals recorded on the scalp before moving on to a discussion of the responses recovered from functionally defined visual ROIs. Fig. 3a illustrates the three different trial categories. The first two correspond to trials where disparity increments were displayed and were either correctly identified (‘Hits’ in green) or undetected (‘Misses’ in red). The last category comprised trials where no changes occurred and the subjects did not make a button press (‘Correct Rejects’ in black). Fig. 3b displays the global-field power (GFP), which is the spatial standard deviation across electrodes of the evoked potential at a given time instant. This widely used metric is a reference-independent measure of activity that is useful for describing the onset and overall time-course of the evoked potential (Skrandies, 1990).

Figure 3.

Stimulus timeline and stimulus-locked responses (sensor space and reaction time)(n = 11 subjects). a) Stimulus timeline. The ‘Hits’, ‘Misses’ and ‘Correct Rejects’ are illustrated in green, red and black respectively (see the text for their definition). b) Global field power (GFP) corresponding to the different categories. The shaded areas indicate SEM. c) Response time histogram. d) Topographic maps corresponding to the three time instants outlined in b). Responses in b) and d) have been baseline corrected from the 100 ms preceding the stimulus onset.

The Hit response begins around 120 ms after the stimulus onset and peaks at ∼ 250 ms (t1). Sustained activity then continues for another 300 ms and finally the response goes back to baseline. The Correct Reject response has a similar onset time and peak latency, but, by contrast, Correct Reject trials do not show sustained activity and the t1 peak is weaker. Responses on Hit vs Correct Reject trials could differ because the stimulus differs or because Hit trials also produce choice-related activity leading to a motor response. Effects due to stimulus differences are eliminated in the comparison between Hits and Misses because the stimulus properties are the same on these two trial types (see section 4.3 for discussion of the possible impact of stimulus variability on this ‘choice-related activity’ and also for the possible involvement of attention mechanisms). Responses on Miss trials have lower initial peak amplitude and less sustained activity. Thus by time t1 and for several hundred ms after, there is activity in the Hits that is due either to choice-related or response preparation activity.

The behavioral response-time distributions are useful for interpreting the stimulus-locked activity. They have an average median value across subjects of 480 ms (with a corresponding standard deviation of 190 ms; see Fig. 3c). The first differences in the GFP for Hits vs Misses start before 250 ms and are thus at least 230 ms before the median value for the button press and thus could logically contribute to the generation of the motor response. From the distribution of scalp potentials at t1 (250 ms) shown in Fig. 3d, it can be seen that these first differences in GFP reflect amplitude changes within and around occipito-parietal and fronto-temporal cortex for each trial category. The amplitudes for the Hits are larger than for the Misses. Amplitudes for the Misses are also larger than those for the Correct rejects. At t2 (365 ms), the Hit responses have extended over the ventral and frontal cortex. The topography associated with the Misses is more occipital and weaker, while the Correct Reject responses are not significant. Finally at t3 (480 ms), only the Hits show significant responses and these are localized over parietal and motor areas. This latency corresponds to the median value of the reaction-time and therefore partially reflects post-decision activity.

Because our experimental protocol consisted of a continuous presentation of trials within the stimulation blocks, there is an ambiguity on the trial labels when the subjects responded after the stimulus onset of a trial directly following a target trial. In this case, we labeled the trials with the target as ‘misses’ and the following trials as ‘false alarms’. As suggested by our reaction-time distribution (see Fig. 3c), those trials were very rare. Among our small number of false alarms, 7 of them on average across subjects (standard deviation of 5) were ambiguous trials. This number is at least 10 times smaller than our number of ‘Hits’, ‘Misses’ and ‘Correct rejects’. This ambiguity on the trial labels is therefore very unlikely to impact our results. We controlled for this by considering as a ‘Hit’ every trial for which the subjects responded to a target in less than two seconds, relabeling the trials according to this scheme. This operation did not change any of the results described in this study.

Responses in functionally defined ROIs

To localize the underlying cortical sources of the activity described above more precisely, we used EEG source-imaging and examined the responses within functionally defined ROIs. We estimated time-courses in 5 well-separated ROIs (V1, V4, LOC, hMT+ and V3A) using a distributed linear inverse approach (see section 2.8). Group averaged time-courses for the ‘Hits’ and ‘Correct Rejects’ trials were constructed by averaging activity in the individually defined visual ROIs and are shown in Fig. 4.

Except for the V1 ROI, all other ROIs showed significant differences between Hits and Correct rejects. These differences first appear in the V3A ROI between 200 ms and 250 ms after stimulus onset. This differential response is followed by more negative responses for the Hits in the V4 ROI that are sustained up to 900 ms. This is consistent with the activity observed in the global field power (see Fig. 3b). This activity may underlie the “Selection Negativity” recorded at the scalp by Kanai and Morotomi (2001) in their disparity discrimination task. Around 400 ms after stimulus onset, other significant differences appear in the V3A ROI. They are followed at t = 500 ms by differences in ROIs hMT+ and LOC.

To determine if these differences are due to the variation in the disparity input (i.e. 5 vs 5+δd arcmin), we compared ‘Hits’ and ‘Misses’ because the visual input on these trials had the same properties (Fig. 5). The pattern of differences between Hits and Misses is the same as that obtained between Hits and Correct Rejects, with the exception of the first peak in the V3A ROI (Compare Fig. 5 to Fig. 4). Thus, the first differential activity between Hits and Correct Rejects observed in the V3A ROI at roughly 200 ms reflects differences in the stimulus disparity but not choice-related activity. This early difference between Hits and Correct Rejects is not surprising because previous imaging studies of human disparity processing have found that area V3A is extremely sensitive to disparity (Backus et al., 2003; Tsao et al., 2003; Cottereau et al., 2011). This ROI may encode disparity magnitude (Cottereau et al. 2011; Preston et al., 2008). In our measurements, this difference is not contingent on the subject's response, and probably does not determine the choice. On the other hand, differences between Hits and Misses in the V4 ROI from 250 ms to 900 ms, in the V3A ROI around 400 ms and in the LOC and hMT+ ROIs around 500 ms may well reflect choice-related activity. However, response times in our task varied widely between trials, so it is possible that this choice-related activity only appears after the button-press and therefore does not directly reflect the decision process. In the following, we address these issues by aligning the hit responses to the timing of the button press.

3.2. Evoked activity locked to the subject response

Response-locked activity at the scalp comprises a monotonic increase in GFP that begins almost 400 ms before the motor response and continues to increase after the motor response is executed (Fig. 6a). The scalp topography at 250 ms before the button press is consistent with sources in occipital cortex but also in frontal cortex. By contrast, the scalp distribution at the time of the button press is located over motor areas responsible for a press on the button with the right index finger.

Figure 6.

Responses aligned to the button press (n = 11 subjects). a) Global field power (GFP). The shaded areas indicate SEM. b) Topographic maps associated with two time instants (Amp.: Amplitude).

Fig. 7 shows response-locked activity in our five visual ROIs in blue. A possible confound in our analysis is that response-locked averages may also contain a residual from sustained sensory responses locked to the stimulus onset. If these responses are sufficiently long lasting, they can survive the blur introduced by the temporal jitter of response locking. To determine which portion of our results are not contaminated by sensory-related stimulus-locked activity, we compared our response-locked data to a surrogate dataset that was created by applying the same analysis as used for the true response-locked data but using the miss trials. The miss trials contain sensory-related activity, but do not contain response-related activity. For every subject, each miss trial was assigned to a response time randomly picked among the subject actual response times during the hit trials. Once aligned to their assigned response time, trials were averaged (within subjects first and then across subjects). The time-courses corresponding to this surrogate dataset are shown in red.

Because the visual input is the same for ‘Hits’ and ‘Misses’, any difference between these two time-courses necessarily reflects a choice-related mechanism. The results shown in Fig. 7 therefore allow us to establish when choice-related activity occurs relative to the button-press. Consistent with the stimulus-locked data (see Fig. 5), ROI V4, followed by ROI V3A, are the first ROIs to show choice-related activity. These responses appear around 200 ms and 125 ms before the button-press. Choice-related responses can also be observed after the button-press in ROIs V4, V3A and LOC and to a lesser degree in the hMT+ ROI. Decision-making implies the integration of sensory inputs until the subject commits to a motor response (Shadlen and Newsome, 2001). In our data, it appears that only the V4 ROI and possibly ROI V3A might be involved in the decision process, as they are the only ROIs with choice-related activity before the button-press.

3.3. Parsing the sensory-decision continuum

Understanding what aspects of the activity estimated in our visual ROIs relates to decision is not straightforward. What we have access to are two clearly defined events in a trial: the time of stimulus presentation, and the time of behavioral response. Decision is something that happens along a continuum between these two endpoints (Di Carlo and Maunsell, 2005). In order to understand how to interpret the choice-related responses of Fig. 7, we have simulated choice-related activity that reflects two different mechanisms, one time-locked to the stimulus onset and one time-locked to the button-press (Fig. 8). For the button-press timings, we used the reaction-time distribution of our data.

Figure 8.

Models of the decision process. First model (stimulus-locked choice-related activity). a-top) The sensory response, locked to the stimulus onset is shown in black and the reaction time distribution in orange. Choice-related activity is added to the sensory response, increasing it from the black to the gray curve. a-bottom) The case of no button-locked component. b) Corresponding responses locked to the stimulus onset. Both the ‘Hits’ (green) and ‘Misses’ (red) are displayed. c) Corresponding responses locked to the button-press (in blue). The corresponding surrogate time-courses obtained by shuffling the ‘Hits’ (dashed green) and the ‘Misses’ (dashed red). Second model (response-locked choice-related activity). d-top) The sensory response locked to the stimulus onset is shown in black and the reaction time distribution in orange. d-bottom) Choice-related activity is locked to the button-press. e) and f) see b) and c).

One possibility is that choice-related activity is purely stimulus-locked and increases the sensory responses following a target presentation (Fig. 8a). When looking at the stimulus-locked data (Fig. 8b) there is a difference between Hits and Misses. The response-locked activity is also significantly higher (Fig. 8c). This is because the stimulus-locked response is slow enough in relation to the button-press distribution and therefore the shuffling is insufficient to completely remove this response. Next, we simulated a choice-related activity that is locked to the button-press. It adds to the stimulus-locked sensory response of the ROI (Fig 8d). Interestingly, even though in this second model there is no stimulus-locked increase of the response, the two models lead to very similar stimulus-locked responses for the ‘Hits’ and ‘Misses’ (Fig. 8b & e). Both models also lead to significant differences between the response-locked data and the surrogate created from the ‘Misses’ (see section 3.2 and Fig. 8c & f). To establish where along the sensory-decision continuum our responses lie, we created a second surrogate dataset. This was done by applying the same analysis as used for the true response-locked data, but instead of using the correct response times, we used random response times drawn from the subjects’ actual responses. That is, we permuted the trial labels for each response, destroying the actual response-time correspondence (see Ales et al., 2013). The second surrogate dataset and our response-locked data contain the same amount of temporal blur from the hit response. Any difference between the two reflects decision-related activity that is temporally more aligned to the button-press than to the stimulus onset (i.e. the second model, see Fig. 8c & f). Fig. 9 shows the comparison between our response-locked data and this second surrogate dataset.

Only the V3A ROI shows significant differences between the true responses and the surrogate. However, these differences appear after the button-press. Therefore, none of our ROIs shows choice-related activity strongly locked to the button-press as in our second model. The choice-related activity observed in the V4 ROI before the button-press (Fig. 7) is better characterized by our first model. Our two models reflect the extreme cases where choice-related activity is either fully stimulus-locked (model 1) or fully response-locked (model 2). The choice-related activity we observe in our V4 ROI might of course contain both stimulus-locked and response-locked components. The absence of significant difference in Fig.7 does not imply that there is no response-locked component in our V4 ROI but rather that they are more tightly locked to the target onset than to the button-press. Actually, the fact that the peak of the response in the V4 ROI is not higher when this response is aligned to the target onset (compare Fig. 5 and Fig. 7) suggests that this response is also partially locked to the button-press.

3.4. Correlation with reaction time

Activity in the V4 and V3A ROIs clearly precedes the subject's responses (see Fig. 7). To more directly test if the behavioral responses were related to activity within these ROIs, we split the reaction-time distribution in three. For each subject, we grouped together the trials with fast, medium and slow reaction times (see DiCarlo and Maunsell, 2005). This manipulation led to three distributions containing an identical number of trials with average reaction times of 390, 480 and 626ms respectively (the corresponding standard deviation are 54, 78 and 95ms). An ROI whose responses are temporally correlated with the subject's reaction time should show significant differences between the evoked potentials associated with these three response-time distributions. Stimulus-locked, activity should rise first for fast reaction times, then for medium reaction times and finally for slow reaction times. Response-locked, activity should rise shortly before the button press for fast reaction times, indicating a fast integration of the visual input. At the opposite end, activity should occur long before the button press for slow reaction times, indicating a longer integration of the visual input. Activity for medium reaction times should exhibit an intermediate profile. Fig. 10-a) and 10-b) shows the evoked potentials corresponding to these three distributions in our V4 and V3A ROIs.

Figure 10.

Responses to slow (yellow), medium (orange) and fast (red) trials in the V4, V3A and motor ROIs (n = 11 subjects). a) Stimulus-locked time-courses in the three ROIs. b) Response-locked time-courses in the three ROIs. c) Magnification of the responses in V4 between 0 and 500ms for stimulus-locked data (left panel) and between -500 and 0ms for response-locked data (right panel). The colored arrows show the time instants when the negative-going phase of the response peak (for stimulus-locked data) and begin (for response-locked data) for the three types of trials. For response-locked data, a black arrow also shows the time instant when the curves converge. The areas shaded in blue provide the time windows that were used to compute the slopes of the ramping down activity (see figure 11). d) Same magnification for the responses in the motor cortex ROI.

A magnification of the responses in V4 during the first 500ms (i.e. between 0 and 500ms for stimulus-locked and between -500 and 0ms for response-locked) is shown in panel c. Stimulus-locked, V4 responses for fast, medium and slow trials peak respectively at 295, 350 and 388ms. These values are given by the time- points when the negative-going phase of the responses peaked on the evoked potentials (see the colored arrows in the figure). Response-locked, slow, medium and fast trials arise from the baseline at -450, -375, and -300ms respectively. These values are given by the time-points when the negative-going phase of the responses began on the evoked potentials (see the colored arrows in the figure). Interestingly, the activity associated with the slow, medium and fast trials seem to converge to the same level 100ms before the button-press (see the black arrow in the figure). After this time, the three time-courses are very similar. This pattern of responses suggests that reaction time has a direct influence on the early part of the responses in the V4 ROI. Importantly, these effects for response-locked data are still observable if we subtract stimulus-locked activity as in section 3.2 (see supplementary figure 4). In comparison, these effects are not observable in our V3A ROI. For stimulus-locked activity, differences between fast, medium and slow reaction times only appear after 450ms in this ROI. There is no clear trend for response-locked activity.

To further characterize the effect of reaction time on the activity associated with slow, medium and fast trials in the V4 ROI, we used linear regressions to compute the slopes of the corresponding ramping down activity (see Roitman and Shadlen, 2002). These regressions were computed during the time window when all the three curves are ramping down (i.e. from 200 to 295ms for the stimulus-locked data and from -300ms to -100ms for the response-locked data, see the shaded areas in figure 10). The slopes for stimulus-locked and response-locked data in the V4 ROI are provided in figure 11.

Both the stimulus-locked and response-locked data show that the slopes are steeper for short reaction time and shallower for longer reaction times. The influence of reaction time on these slopes was tested using ANOVAs. For stimulus-locked data, there was a relatively small effect (p = 0.049). For response-locked data, this effect was more reliable (p = 0.0001) and remained significant after the subtraction of the stimulus-locked activity (p = 0.01, see supplementary figure 4). Taken together, these results suggest that reaction time is based on a variable duration of integration in the V4 ROI.

3.5. Responses in the motor cortex

A possible confound in our results is that the responses we estimate in our visual ROIs may reflect motor activity. There are two mechanisms by which such a confound could manifest in our data: 1) Responses in early visual ROIs are contaminated by motor activity through volume conduction. 2) Activity in our early visual ROIs is motor related.

To demonstrate that the V4 and V3A ROI responses are not contaminated by motor activity through volume conduction, we estimated the responses from the hand area in the motor cortex of the left hemisphere (i.e. the contralateral hemisphere as subjects were instructed to press the button with their right index). This area was localized from the average across-subject activity at the instant of the button-press (see Supplementary figure 5) and lies at the intersection between the precentral and superior frontal sulcus (see section 2.10). This area exhibits ramping preparatory activity that appears 250ms before the button press. Just before the button-press, the activity in this ROI inverts its polarity and remains significant up to 500 ms after the button-press. This response profile is very different from those observed in ROIs V4 and V3A. Therefore, our response estimates in the V4 and V3A ROIs do not reflect false activation caused by volume conduction of motor activity.

To demonstrate that the trend we observed in the V4 ROI for fast, medium and slow reaction times is not motor related (see section 3.4), we estimated the responses associated with fast, medium and slow reaction times in the hand motor ROI. The time-courses are presented in Fig. 10 (panels a and b, right column and panel d for a magnification of the responses during the first 500ms). Response-locked, there is no observable difference between fast, medium and slow trials in our motor ROI during the ramping preparatory activity. The earliest differences appear after the button-press. This is consistent with the stimulus-locked activity where the differences only appear 500ms after the target onset. Theses differences are long after the trend we observed in our V4 ROI. This pattern of response suggests that reaction time is based on a variable duration of integration in the V4 ROI that is followed by a fixed and stereotypic response in motor cortex.

4. Discussion

The sensory mechanisms associated with stereoscopic vision have been well studied in the human brain (Backus et al., 2001; Parker 2007; Preston et al., 2008), but much less is known about the cortical processes involved in disparity-based perceptual decisions. Using an EEG source imaging technique that estimates responses within functionally defined visual ROIs, we were able to establish which ROIs have responses that reflect the subjects' behavior and when this choice-related activity appears relative to both the stimulus onset and the button-press.

4.1. Cortical networks involved in disparity-based decision

Among our five visual ROIs, only the V1 ROI did not show any choice-related activity (see Fig. 5). In theory, our task could be controlled by neurons tuned to absolute and/or relative disparity (i.e. the difference between the disk and annulus disparities). Psychophysical studies have demonstrated that disparity discrimination is much more difficult when no disparity reference is provided (Westheimer, 1979; Prince et al., 2000; Read et al., 2010; Cottereau et al., 2012bc). Because V1 is thought to be only involved in absolute disparity processing (Cumming and Parker, 1999; Cottereau et al., 2011), the absence of choice-related activity in this ROI is not surprising.

At the population level in human, disparity-evoked responses in ROIs V4, V3A, LOC and hMT+ are affected by a disparity reference and therefore are sensitive to relative disparity (Cottereau et al., 2012b). Because of this, each of these ROIs could be involved in the decision, although the specifics of the neural computations might differ across areas (Cottereau, 2011). Our analysis of the stimulus-locked responses indeed showed significant differences between Hits and Misses in our 4 extra-striate ROIs (see Fig. 5). This is consistent with an fMRI study where the subjects had to perform a disparity-defined shape discrimination task (Chadrasekaran et al., 2007). In that study, BOLD activations in both dorsal and ventral visual areas correlated with the subject's performance. Beyond the literature related to binocular vision, numerous studies have demonstrated that responses in sensory areas are larger when the target is perceived (rightly or wrongly) during a detection/discrimination task (Ress and Heeger, 2001; Moradi et al., 2007; Donner et al., 2008).

In our data, the first sign of choice-related responses occurred at 250 ms after target onset in the V4 ROI (Fig. 5). Choice-specific activity was then found in ROIs V3A, LOC and hMT+. Behavioral responses to the target occurred at a median latency of 480 ms. If we assume a motor delay of less than 100 ms (Roitman and Shadlen, 2002; Kiani et al., 2008, see also the responses estimated from the motor cortex in figure 10 and in supplementary figure 5) that leaves a transitional time of approximately 130 ms during which the decision to respond is made. By looking backwards in time from the moment of the behavioral response, we found that only ROIs V4 and V3A had choice-related activity before the button-press (Fig. 7). Choice-related activity appeared much earlier in the V4 ROI. By modeling this activity (see Fig. 8), we were able to show that choice-related responses in the V4 ROI are temporally more linked to the stimulus onset (Fig. 9) but also probably contain a response-locked component.

Disparity-selective neurons whose responses only depend on the disparity difference between two objects and are invariant to absolute disparity (and therefore to change in convergence) are found in both human (Neri et al., 2004) and macaque (Umeda et al., 2007) V4 using functional imaging (fMRI) and single-cell recordings. These neurons are probably implicated in the disparity discrimination task described in the single-cell study of Shiozaki et al. (2012). They are also likely to be involved in the decision in our task. A smaller proportion of this type of neuron can also be found as early as area V2 in macaque (Thomas et al., 2002; Umeda et al., 2007). Our source-imaging technique does not permit us to extract reliable time-courses from ROIs V2 and V3 as they receive significant cross-talk from other areas (and in particular from the V4 ROI, see section 2.4). From our study, it is therefore difficult to determine if ROIs V2 and V3 also have early choice-related activity.

V3A is extremely sensitive to absolute disparity in both human (Preston et al., 2008) and macaque (Anzai et al., 2011), but its responses are also strongly modulated by the spatial disparity context, which is another form of relative disparity sensitivity (Cottereau et al., 2011; Cottereau et al., 2012b). Because of these properties, we had expected that the V3A ROI would have an early involvement in our disparity task (Cottereau et al., 2012b). Although the V3A ROI showed the earliest target-related activity (see the differences between Hits and Correct rejects in Fig. 4) these early differences reflect a purely sensory mechanism as they disappear in the comparison between Hits and Misses (Fig. 5). In our analysis, choice-related activity appears in the V3A ROI well after in ROI V4 (Figs. 5 and 7). The onset latency of this choice-related activity (i.e. around 125 ms before the button-press) and also the responses observed for fast, medium and slow reaction-times (Fig. 10) make it hard to argue that V3A ROI responses are involved in the decision-making process. Perhaps the strong dependence of area V3A on absolute disparity and therefore on convergence does not make it as appropriate as area V4 to perform our task.

Even though the exact instant when the decision is made is inaccessible (Nienborg et al., 2012), because we used speeded reaction times in this study we have an additional covariate that can be used to tease apart the driver of the neuronal latencies (e.g. perception of the stimulus or response generation). This type of analysis follows from DiCarlo & Maunsell (2005), who showed how neuronal latencies fall along a continuum from latencies purely tied to perception to latencies purely tied to response generation. By sorting the response-locked averages by reaction time, we showed that activity in the V4 ROI is systematically related to the time of choice, arising earlier before the button press on trials with short RTs. This pattern of results suggests that there is a close link between activity in V4 and behavioral choice, as has been previously reported for color/orientation choices in macaque V4 (Mirabella et al., 2007) and disparity (Shiozaki et al., 2012). Shiozaki et al. were also able to influence disparity choices through micro-stimulation, further reinforcing a causal role for V4 in disparity decisions.

A leading model of perceptual decision-making is the drift-diffusion model (see e.g. Gold and Shadlen, 2007). In this model, there is an initial stage that represents the momentary evidence for the stimulus parameter. The output of this stage is fed into a stage that accumulates this evidence for the purpose of making a decision. In many studies, V4 responds in a way that is similar to a stage that represents momentary evidence (Roe et al., 2012). However, one study that examined decisions about color and orientation in V4 neurons (Mirabella et al., 2007) found that while firing rates of neurons in V4 contain a strong response to the stimulus, the response also contains differential activity that encodes choice, and this activity diverges at different rates for fast and slow trials. The present data also contains a pattern of activity that diverges for fast, medium and slow trials (see Fig. 10 and Fig. 11). Both stimulus-locked and response-locked, V4 ROI responses for these three categories are consistent with an accumulation process in this area (Roitman and Shadlen, 2002).

Before the button-press, significant choice-related activity was also found in the frontal and temporal poles (see the supplementary results on the ‘Evoked activity in outside of visual cortex’). Our modeling of the decision process (Fig. 8) suggests that these regions could be more strongly related to the behavioral decision than is the V4 ROI as their responses are time-locked to the button-press. Our source estimates for the frontal and temporal poles are based on anatomically defined areas, and thus caution is in order regarding the exact cortical origin of these responses (see section 4.4).

As noted by Mirabella et al. (2007), cortical area V4 could be directly involved in the process of converting sensory evidence into a response-related format, or, alternatively, choice-related activity measured in the V4 ROI could be the result of feedback from higher-level areas involved in the task. Frontal cortex in both human and macaque are involved in perceptual decision-making (see e.g. Hernandez et al., 2010 or Schall, 2011). These regions could integrate the sensory responses from earlier areas, such as V4 to form the decision. This information could then be fed-back to V4. Because choice related activity is both wide-spread and temporally overlapping, the notion of a sequential segregation between sensory coding and decision making, as suggested by the drift-diffusion model may need to be reconsidered (Mirabella et al., 2007). On this view, V4 may represent a key transitional area along the sensor-motor continuum proposed by DiCarlo and Maunsell (2005), being involved in both extracting an estimate of disparity and in converting this information into a format that is suitable for subsequent generation of behavioral output.

4.2. Post-decision activity

Activity after the button press is by definition after the decision. But the decision must have been complete at some point prior to the button press. Unfortunately, there is no explicit marker that enables us to determine the precise moment of decision. In some studies of decision-related activity, participants are not free to respond at will, but have a fixed response cue (Shadlen and Newsome, 2001; Donner et al., 2009). These studies tend to interpret any choice-related activity as being involved in the decision. A recent study (Nienborg and Cumming, 2009) used a clever stimulus manipulation that enabled a determination of the time at which a decision was complete. Nienborg and Cumming (2009) demonstrated that neural responses associated with the choice survive long after the decision is made. In the current study, significant choice-related activity was found after the button-press (see Fig. 7 and also Fig. 9) in the V4 ROI but also in ROIs V3A, LOC and to a lesser degree in the hMT+ ROI. The time-course of this later activity is very different from the activity estimated in motor cortex and is thus not due to volume conduction. Other studies have described significant post-decision activity during a disparity discrimination task. Using single-cell recordings, Uka and collaborators (Uka et al., 2005) found that the responses in IT, an area analogous to human LOC (Denys et al., 2004), were correlated with the animal's behavior. This correlation appeared only 360 ms after the stimulus onset. Given this long latency, they suggested that the activity in IT appeared after the decision. Nienborg and Cumming (2009) reached the same conclusion for the activity they observed in V2 related to a disparity-task.

4.3. Possible confounds in our ‘choice-related activity’ and limitations of the study

It is clear that attention plays a role in our discrimination task as a subject is less likely to detect a target if his/her attention is not directed to the disc disparity. Is it possible that the differences we observe between the response amplitudes of the Hits and Misses is directly caused by attention mechanisms? In one of our previous studies (Cottereau et al., J. Neuroscience 2012), we tested whether attention had an impact on the EEG responses estimated in our ROIs. The stimuli were identical to those used in the present study. Subjects were asked to view the displays passively, to attend to changes in the disparity modulation of the central disk (directed attention), or to attend to changes in a set of letters superimposed on the center of the disk (divided attention). These manipulations of attention had no significant effect on the signal-to-noise ratio in any ROI. This conclusion is consistent with an earlier study (Tsao et al., 2003) that also found that attention had no consistent effect on disparity-driven BOLD signals in early visual areas. On the other hand, attention has been shown to be a major factor shaping activity in V4 in numerous studies (single-cell physiology and neuroimaging), and various types of visual tasks (see Roe et al., 2012 for a review). Because of these studies and because our attention control was not performed on the same subjects who participated in this decision experiment, it is not possible to exclude attention as a possible cause of the difference we obtained between the response amplitudes of the Hits and Misses.

Another possible confound in our results is that the Hits versus Misses difference might reflect an accuracy effect (i.e., due to fluctuations in selective attention and/or arousal), rather than a decision-related effect. To address this point, future studies should manipulate the task difficulty and characterize the influence of this manipulation on the responses in the V4 ROI. Because of our response criterion (the disparity increment was chosen so that the subjects correctly detected 80% of the targets), we collected only a small number of false alarms for each subject. This number was too small to permit a proper analysis of this category of trials and we therefore do not discuss it in this study. Future studies should decrease the response criterion in order to obtain more false alarms.

Although our stimuli had the same global properties across trials, we cannot exclude the possibility that the differences we observed between Hits and Misses (i.e. ‘choice-related’ activity) were caused by small variations in the stimulus presented on different trials (e.g. edge visibility or spatial patterns in our random-dot stereograms). Our stimuli we generated online but not recorded. To test this hypothesis, future studies could record the seed information and check if any particular stimulus features are related to the hit versus miss choice, for example by using psychophysical reverse correlation (Neri et al., 1999).

4.4. Spatial resolution of fMRI-informed EEG source imaging

We saw in section 2.9 that the V4 ROI receives some cross-talk from ROIs V1 (27%), LOC (34%), hMT+ (32%) and V3A (43%) (see Fig. 2). However our results in the V4 ROI cannot be explained by cross-talk from these ROIs, as the time-courses in the other ROIs are very different from those observed in the V4 ROI. In particular, the early choice-related activity we observed in the V4 ROI for response-locked data (Fig. 7) did not exist in other ROIs. In our simulations, cross-talk from ROIs V2 and V3 to the V4 ROI is rather small (37% and 31% respectively). Of more importance, The V2 and V3 ROIs had more influence on the V1 ROI (43 and 35%) and on the LOC ROI (39 and 43%). If our results were due to cross-talk from ROIs V2 and V3, they would also show up in ROIs V1 and LOC, which is not the case. One could argue that cross-talk could also come from other cortical areas that were not included in the simulations. Given the nature of EEG activity, cross-talk is likely to come from nearby regions (Im et al., 2007) and thus, V4 ROI responses may be partially due to cross-talk from sources in VO-1 and VO-2 (Brewer et al., 2005). We also found that both the stimulus-locked (Fig. 5) and response-locked (Fig. 7) results in the V4 ROI were present at the individual subject level (see supplementary figures 2 and 3). Our effects were found in 10 of the 11 subjects and are therefore not driven by outlier subjects.

The frontal and temporal poles were defined using an atlas based on anatomical landmarks (Desikan et al., 2006, see the supplementary results on the ‘Evoked activity in outside of visual cortex’). No specific correlations were assumed for the neural responses estimated within these regions. Our source localization technique is probably less accurate in this case (see Cottereau et al., 2012a) suggesting that caution is in order regarding the exact origin of the choice-related activity we observed in these two regions.

5. Conclusion

EEG measurements have been used previously to decode the temporal characteristics of decision-making in various perceptual tasks (VanRullen and Thorpe, 2001; Philiastides and Sajda, 2006; Ratcliff et al., 2009). The analyses in these studies were limited to the time-courses measured on the scalp and time-locked to the stimulus. Here (see also Ales et al., 2013), we show that source imaging, combined with a response-locked analysis can recover the sources of the recorded signals and provide important information about the dynamic continuum that links perception to action during a discrimination task.