Here we study the behavioral and neural dynamics of perceptual categorization when decision information varies in multiple domains at different levels of abstraction. Behavioral and modeling results suggest that categorical (more abstract) information is weighted more strongly than perceptual (more concrete) information but that perceptual and categorical domains interact to influence decisions. Frontoparietal brain activity during categorization flexibly represents decision-relevant features and highlights significant dissociations in frontal and parietal activity during decision making.
Keywords: categorization, perceptual decision making, frontal cortex, parietal cortex, diffusion model
Abstract
To date it has been unclear whether perceptual decision making and rule-based categorization reflect activation of similar cognitive processes and brain regions. On one hand, both map potentially ambiguous stimuli to a smaller set of motor responses. On the other hand, decisions about perceptual salience typically concern concrete sensory representations derived from a noisy stimulus, while categorization is typically conceptualized as an abstract decision about membership in a potentially arbitrary set. Previous work has primarily examined these types of decisions in isolation. Here we independently varied salience in both the perceptual and categorical domains in a random dot-motion framework by manipulating dot-motion coherence and motion direction relative to a category boundary, respectively. Behavioral and modeling results suggest that categorical (more abstract) information, which is more relevant to subjects’ decisions, is weighted more strongly than perceptual (more concrete) information, although they also have significant interactive effects on choice. Within the brain, BOLD activity within frontal regions strongly differentiated categorical salience and weakly differentiated perceptual salience; however, the interaction between these two factors activated similar frontoparietal brain networks. Notably, explicitly evaluating feature interactions revealed a frontal-parietal dissociation: parietal activity varied strongly with both features, but frontal activity varied with the combined strength of the information that defined the motor response. Together, these data demonstrate that frontal regions are driven by decision-relevant features and argue that perceptual decisions and rule-based categorization reflect similar cognitive processes and activate similar brain networks to the extent that they define decision-relevant stimulus-response mappings.
NEW & NOTEWORTHY Here we study the behavioral and neural dynamics of perceptual categorization when decision information varies in multiple domains at different levels of abstraction. Behavioral and modeling results suggest that categorical (more abstract) information is weighted more strongly than perceptual (more concrete) information but that perceptual and categorical domains interact to influence decisions. Frontoparietal brain activity during categorization flexibly represents decision-relevant features and highlights significant dissociations in frontal and parietal activity during decision making.
fundamentally, all organisms must be able to link a potentially large number of stimuli with a smaller number of goal-relevant responses. In studies of such stimulus-response mappings, perceptual decision making (PDM) has typically been conceptualized as the ability to define a course of action based on the evidence accumulated over a noise-degraded stimulus. In contrast, studies of rule-based categorization (RBC) have typically been defined by varying how far a perceptually salient stimulus lies from a category boundary. These differences between the two types of decisions, most overtly in level of abstraction, have been emphasized despite the fact that both sets of studies have used overlapping stimuli, ranging from simple features like color, motion, and size (PDM: Gold and Shadlen 2007; Kayser et al. 2010b; Stanford et al. 2010; RBC: de Gardelle and Summerfield 2011; Freedman and Assad 2006; White et al. 2012) to more complex stimuli such as faces, houses, cars, and animals (PDM: Heekeren et al. 2008; Philiastides and Sajda 2006; Tremel and Wheeler 2015; RBC: Freedman et al. 2001; Scholl et al. 2014). Moreover, brain regions participating in PDM and RBC have both been identified primarily by finding areas that modulate their activity as a function of feature salience (Freedman and Miller 2008; Heekeren et al. 2004; Kayser et al. 2010a; Roitman and Shadlen 2002; Seger et al. 2015; Tremel and Wheeler 2015).
In most such studies, choice-relevant information is available from only a single source. For example, variable perceptual salience, such as motion coherence contaminated by varying levels of perceptual noise, is examined only in the context of highly salient categorical information, such as opposing leftward and rightward motion (Kayser et al. 2010b), or, alternatively, variable categorical salience, as defined by different abstract shapes, is present in the setting of highly salient, noiseless visual stimuli (Gauthier and Tarr 1997). Thus how multiple domains of information at differing levels of abstraction contribute to decision making remains an active area of investigation. Recently, novel categorization studies have begun to address the issue of multiple input sources by manipulating the average color (red/blue) and shape (circle/square) of a multiple-element array that formed the basis for a categorical color-shape decision (de Gardelle and Summerfield 2011; Michael et al. 2015), or they have independently varied distance from a category boundary and distance from a prototype (Seger et al. 2015). However, the important questions addressed in these studies required all individual elements within the array to have high perceptual salience, whereas the variabilities were both in the categorical domain—e.g., in the former study, variability in the mean was defined by distance from the category boundary and variability in the variance of the elements was defined about that mean. Here we sought to uncouple salience in the mean stimulus strength from distance to the category boundary in order to introduce variability into distinct domains that differ in abstraction but both influence choice: categorical/more abstract (i.e., distance from the category boundary) and perceptual/less abstract (i.e., mean stimulus strength).
To this end we adapted the well-established PDM paradigm utilizing a random-dot kinematogram (Gold and Shadlen 2007; Heekeren et al. 2006; Kayser et al. 2010b) to a categorization framework. Critically, we varied both the direction of dot motion relative to the category boundary (variable categorical salience) and the coherence of dot motion (variable perceptual salience). Additionally, we used the diffusion model, a well-known computational model successfully applied to model choice behavior in a variety of similar tasks (Kayser et al. 2010b; Palmer et al. 2005; Ratcliff and McKoon 2008; White et al. 2012), to predict the relative importance of categorical and perceptual information. In such models evidence accumulates over time, at a rate determined by feature salience, until a threshold is reached and a choice is made. We predicted that while both domains would interact to determine choice, categorical information would have greater weight within the accumulator model, as this and similar tasks ultimately rely upon a categorical discrimination (e.g., left or right of a category boundary) to generate a categorical response (e.g., left or right button press).
Similarly, we predicted that categorical and perceptual salience would be represented differentially within the brain, depending on their relative importance in determining the response (Kayser et al. 2010a). In accord with previous results relating feature salience and parametric blood oxygen level-dependent (BOLD) amplitudes in PDM (Hebart et al. 2012; Ho et al. 2009; Kayser et al. 2010b), we hypothesized that frontal and parietal activity would be modulated in negative parametric fashion by salience in the most decision-relevant feature. Specifically, BOLD amplitude would increase with decreasing feature salience, be it decreasing motion coherence or decreasing distance from category boundary, to the extent that the feature in question was relevant to the response. In contrast, when faced with a feature that poorly determined the response and was therefore less relevant, the BOLD response to that feature would demonstrate a positive parametric effect, in keeping with our previous work (Kayser et al. 2010a).
Based on our prediction that categorization would be weighted more heavily, we anticipated that a negative parametric effect would be seen more strongly in the parametric variation of categorical information (Seger et al. 2015; White et al. 2012) than perceptual information. However, we expected that under certain conditions this effect could vary. Under conditions of uniformly high categorical salience, for example, when perceptual salience would drive choices, such negative parametric activity would vary primarily with perceptual salience, while under conditions of absent categorical salience activity within frontal regions would vary little with perceptual salience, if at all (Kayser et al. 2010a). Consistent with other work on multisensory integration (Senkowski et al. 2011; Stein et al. 2009; Stevenson et al. 2012), this strong interaction between categorical and perceptual salience would reflect the important flexibility of frontal networks to selectively represent goal-relevant features at different levels of abstraction, especially those that are most proximal to the decision. More generally, this interaction would support the idea that perception is a sensorimotor process (van Atteveldt et al. 2014).
MATERIALS AND METHODS
Ten subjects (ages 18–42 yr; 4 men, 8 women) participated in the study and gave written informed consent to participate in a protocol approved by the Committee for the Protection of Human Subjects at the University of California, Berkeley. All subjects had normal neural anatomy as assessed by a neurologist (A. S. Kayser), were right-handed, and had normal or corrected-to-normal vision. Before scan sessions, subjects were trained on the task for a minimum of five 1.25-h sessions to reduce learning effects in the scanner. Once trained, all subjects underwent four 1.5-h fMRI sessions, each of which consisted of six runs of 64 trials for a total of 4 × 6 × 64 = 1,536 trials per subject.
Subjects performed a visual dot-motion task in which they viewed a variable proportion of coherently moving dots on a background of randomly moving dots (Fig. 1). On any given trial, the dots could move in 1 of 16 different directions; subjects were required to categorize the direction of motion as either leftward or rightward of a boundary, which was an oblique line inclined at either 45° or 135° from the horizontal. We used oblique axes rather than the cardinal axes because motion direction is highly discriminable near the cardinal axes (oblique effect: Ball and Sekuler 1987; Matthews and Welch 1997) and, therefore, restricts the range of categorical uncertainty that can be created.
Fig. 1.
Behavioral task. A: task schematic. On each trial the subject pressed a button to indicate whether the overall movement of a random dot kinematogram (RDK) was to the left or right of an oblique boundary line. The boundary line was presented for 5 s at the start of each block of trials. Each trial started with a central fixation cross that dimmed after 2–12 s, after which the RDK appeared on the screen for 2.5 s. The subject responded within this duration. B and C: perceptual (B) and categorical (C) uncertainty were introduced to manipulate salience. Perceptual salience was manipulated via the introduction of noise into motion coherence, while categorical salience varied with distance of dot motion from the category boundary.
To study the effects of perceptual and categorical salience on subjects’ choices, independent parametric manipulations of each domain were introduced. Perceptual salience was altered by changing dot-motion coherence: salience decreased as motion coherence decreased. Categorical salience was manipulated by varying the direction of motion of the dots: salience decreased as the coherently moving dots moved closer to the boundary line. Subjects were instructed to report their choice via a button press as quickly and accurately as possible, using the index or middle finger of their right hand. Runs with over six aborted trials (in which subjects did not respond within the allotted duration) were excluded from subsequent analyses. One subject was excluded from the study because of a high percentage of aborted trials in two sessions. Specifically, this subject responded outside the response window in >20% of trials within each run in two scanner sessions, rendering both the behavior and the BOLD data problematic. Consequently, he was excluded from both the behavioral and imaging analyses. Nine runs were excluded in all from the remaining nine subjects.
The category boundary was indicated to subjects at the start of each block for a period of 5 s, at the end of which the boundary was removed and the trials commenced. The boundary remained unchanged over a block of trials but alternated between blocks. Motion coherence as well as direction of motion were consistent over a single trial and were varied independently of each other across trials. At the beginning of each trial, a bright central fixation cross was dimmed to indicate onset of the dot-motion stimulus. The stimulus was presented for 2.5 s and subjects were required to respond within that duration, in accord with previous studies in the laboratory (Kayser et al. 2010a, 2010b). At the end of each trial the fixation cross brightened for a jittered intertrial interval of 2–12 s.
In the training sessions conducted outside the scanner, subjects were trained on five different dot-motion coherences (0%, 6%, 12%, 25%, and 100%) and nine different distances from category boundary (0°, ±12°, ±45°, ±78°, ±90°). In the initial two training sessions, subjects were provided with auditory feedback on their performance on individual trials. Thereafter, feedback was discontinued. After the training sessions, psychometric curves were constructed and intermediate values of dot-motion coherence and distance from category boundary were ascertained in order to obtain an intermediate accuracy value of 80%. Subsequently, during the scanner sessions, subjects were presented with dots that moved with four levels of coherence [0%, 25%, 100%, and 1 intermediate coherence (IC) ranging from 12% to 15%] at nine distances from either side of the category boundary {0°, ±45°, ±90°, and 2 intermediate distances (IDs) between 0° and ±45° (i.e., ±ID, ranging from 8° to 20°) and between ±45° and ±90° [i.e., ±(90° − ID)]}. Two subjects were assigned two IC values because they over- and underperformed at 25% coherence. These subjects performed the task at 0%, 4%, 10%, and 100% and 0%, 25%, 35%, and 100% motion coherence, respectively.
For fMRI sessions, the presentation order of dot-motion trials was computed with OptSeq (http://surfer.nmr.mgh.harvard.edu/optseq/) (Dale 1999). Stimuli were programmed in MATLAB in the PsychToolbox environment (Brainard 1997; Pelli 1997), adapted from our previous code. Dot density was fixed at 16.7 dots·°−2·s−1, and dot velocity was fixed at a single value of 5°/s to ensure that motion energy was uniform across levels of motion coherence. Blurring effects (in which consecutive placements of a single dot were seen as forming a line) were avoided by the serial presentation of three interleaved subsets, with each frame containing only one of the subsets. To ensure that dots were initially placed evenly across the viewing aperture, we rejected initial dot placements that showed evidence for an unusually skewed starting configuration. Specifically, we rejected initial random dot configurations that showed a ≥95% chance of deviating from the expected χ2 distribution for the frequency of dots over a 4 × 4 grid covering the viewing aperture (note that the grid was not displayed on the screen). Once set in motion, dots that moved outside the aperture were repositioned on the opposite side of the window to prevent them from collecting in any particular region of the aperture over time. To identify overall motion direction, subjects were instructed to pay attention to the entire field of dots, as the interleaved presentation and variable assignment of motion vectors rendered strategies focusing on any single dot ineffective.
Behavioral Modeling
We used a proportional rate diffusion model as defined by Palmer and colleagues (Palmer et al. 2005) to model behavior in the categorization task. The impetus was to identify how information from two domains, one more concrete (perceptual) and the other more abstract (categorical), combined to form a decision. The diffusion model hypothesizes that decision making consists of a process of evidence accumulation for each of the alternative decisions available to a subject. When a threshold level of evidence is reached for one of the decisions, the subject generates a corresponding response. Importantly, the model permits one to fit both reaction time (RT) and accuracy data with a single set of parameters, thereby simultaneously constraining both RT and accuracy variables and providing a parsimonious and theoretically meaningful explanation for the data.
The Palmer model, derived from the diffusion model of Ratcliff (Ratcliff and McKoon 2008), consists of four variables: 1) A′, bearing on the decision threshold; 2) x, representing stimulus salience (see below); 3) k, a proportionality constant (the “sensitivity”) linking stimulus salience to the drift rate μ (drift rate μ = kx); and 4) TR, the mean residual time in seconds, representing a fixed processing duration independent of evidence accumulation (e.g., for low-level sensory processing or implementation of motor commands). Parameters were derived for each subject with an iterative procedure designed to optimize the log-likelihood of the diffusion model fit (Palmer et al. 2005). Notably, this model produces parameter fits that incorporate all trials, and thus single-trial estimates are not generated.
Three versions of the model were implemented. In the low-level interaction model (Interaction model), A′, k, and TR were assumed constant across all stimulus conditions, while x assumed a value that represented the interaction of perceptual and categorical information. This model embodies the assumption that perceptual and categorical information interact to provide the input to a consistent accumulator process. Thus the behavior of each subject was fit by finding 20 different values of x—reflecting the 20 different stimulus combinations derived from 4 coherence values and 5 distance values (collapsed across positive and negative angles)—and one each of A′, k, and TR. In the two high-level interaction models (Coherence and Distance models), values for x representing motion coherence (or distance from category boundary) were obtained by fitting behavior at the highest distance from category boundary (or motion coherence). These values of x were then used at all other distances from category boundary (or motion coherence) as well, while A′, k, and TR were fit anew for each distance from category boundary (or motion coherence) in a two-step process. First, for each combination of A′ and TR, the best k was found; then, the best fitting combination of A′ and TR was calculated to arrive at the best fit values of A′, k, and TR. This model embodies the assumption that the accumulator process itself may vary quantitatively based on the feature being integrated; we use it to determine which domain informs stimulus strength (x) and which exerts an additional modulating influence (k) on the decision. This multiplication, taken directly from the Palmer model itself (Palmer et al. 2005), represents the simplest type of interaction between two parameters that does not incorporate other constraints on its precise form (van Eeuwijk 1995). Thus each subject’s behavior was fit by using 4 (or 5) values of x to represent coherence (or distance) along with 5 (or 4) values each of A′, k, and TR. Table 1 provides ranges of parameters obtained for all model versions, across subjects. We used the corrected Akaike information criterion (AICc; Anderson et al. 1994; Hurvich and Tsai 1989) to compare the three versions of the model implemented. The AICc is defined as AICc = AIC + 2(k + 1)(k + 2)/(n − k − 2), where AIC = 2k − 2lnL, k is the number of estimated parameters in the model, n is the sample size, and L is the maximum likelihood for the model. The AICc is recommended over the AIC when n/k < 40 for the model with the largest k value (Burnham and Anderson 2004) and permits the comparison of models constructed with different numbers of parameters for a smaller sample size. To evaluate whether AICc values were significantly different between the alternative versions of the model across subjects, we used the Wilcoxon signed-rank test, as the distribution of AICc values does not necessarily permit the application of standard parametric tests.
Table 1.
Parameter ranges across subjects for each model version
| Interaction Model | Distance Model | Coherence Model | |
|---|---|---|---|
| x | |||
| lo | 0.17 (0.11–0.19) | 0.15 (0.07–0.23) | 0.11 (0.05–0.19) |
| hi | 2.51 (2.13–2.89) | 3.03 (2.61–4.83) | 4.83 (3.11–5.53) |
| k | |||
| lo | 0.88 (0.75–0.92) | 0.06 (0.03–0.06) | 0.06 (0.04–0.09) |
| hi | 0.7 (0.43–0.78) | 1 (0.39–1.5) | |
| A′ | |||
| lo | 1.12 (1.12–1.12) | 1.08 (1.08–1.08) | 1.08 (1.08–1.08) |
| hi | 1.2 (1.12–1.2) | 1.13 (1.08–1.31) | |
| TR | |||
| lo | 0.31 (0.22–0.35) | 0.28 (0.26–0.85) | 0.33 (0.2–0.37) |
| hi | 0.34 (0.2–0.39) | 0.46 (0.36–0.67) |
Values are medians (interquartile ranges) for each model parameter (x, stimulus strength; k, sensitivity; A′, decision threshold; TR, nondecision time) for the 3 model versions, where lo and hi refer to lowest and highest stimulus salience levels. For the Interaction model, x (lo) and x (hi) refer to the lowest and highest levels of combined perceptual and categorical salience, respectively. For this model, a single value of k, A′, and TR was subsequently fit across all combinations of perceptual and categorical salience (see materials and methods). For the Distance model, x (lo) and x (hi) refer to the lowest and highest levels of categorical salience, respectively. For the other parameters in this model, lo indicates the lowest and hi indicates the highest level of perceptual salience. Conversely, for the Coherence model x (lo) and x (hi) refer to the lowest and highest levels of perceptual salience, respectively. For the other parameters in this model, lo indicates the lowest and hi indicates the highest level of categorical salience.
MRI Scanning
MRI scanning was conducted on a Siemens MAGNETOM Trio 3-T MR scanner at the Henry H. Wheeler, Jr. Brain Imaging Center at the University of California, Berkeley. Images were collected with a 12-channel phased-array coil. Anatomical images consisted of 160 slices acquired with a T1-weighted MP-RAGE protocol (TR = 2,300 ms, TE = 2.98 ms, FOV = 256 mm, matrix size = 256 × 256, voxel size = 1 × 1 × 1 mm). Functional images consisted of 32 slices acquired with a single-shot gradient echoplanar imaging protocol in a contiguous order (TR = 1,800 ms, TE = 23 ms, FOV = 210 mm, matrix size = 70 × 70, voxel size = 3 × 3 × 3 mm). A projector (Avotec SV-6011, http://www.avotecinc.com/) was used to display the image on a translucent screen placed within the scanner bore behind the head coil. A mirror was used to allow the subject to see the display. The distance from the subject’s eye to the screen was 29 cm, and the presented images subtended a visual angle of 7.5°. Subjects’ responses were recorded with an MRI-safe fiber-optic response pad (Inline model HH-1x4-L, http://www.crsltd.com).
fMRI Preprocessing
fMRI preprocessing was performed with both AFNI (https://afni.nimh.nih.gov/) and FSL (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/). Functional images were converted to four-dimensional NIfTI format and corrected for slice timing offsets. Motion correction was carried out with the AFNI program 3dvolreg, with the reference volume set to the mean image of the first run in the series. Images were then smoothed with a 5-mm FWHM Gaussian kernel. Coregistration was performed with the AFNI program 3dAllineate using the local Pearson correlation cost function optimized for fMRI-to-MRI structural alignment. The subsequent inverse transformation was then used to warp the anatomical image to the functional image space. Anatomical images were normalized with the FSL program fnirt to a standard volume (MNI_N27) available from the Montreal Neurological Institute (MNI; https://www.mcgill.ca/bic/). The same normalization parameters were later applied to native-space statistical maps as necessary for the generation of group statistical maps (see below).
Univariate Analysis
To address a series of hypotheses, we carried out a number of voxelwise fMRI statistical analyses for each subject using the general linear model framework implemented in the AFNI program 3dDeconvolve. The overall effects of motion coherence were assessed by modeling the four coherence values with separate regressors, each of which was derived by convolving a gamma probability density function (peaking at 6 s) with a vector of stimulus onsets for each condition. Effects of distance from category boundary were similarly assessed by modeling the five unique distance values (0°, ID, 45°, 90° − ID, 90°) with separate regressors. Tests of linear trends were carried out for each voxel using the coherence (distance from boundary) vector transformed to zero mean and a sum of squares equal to 1 (Kayser et al. 2010b) and then applied to the estimated β-coefficients computed for each motion coherence (distance from boundary) value. The resulting values were mapped to the MNI template and subjected to group-level analyses. Mapwise significance (P < 0.05, corrected for multiple comparisons) was determined by applying a cluster size correction (20 voxels) derived from the AFNI programs 3dFWHMx and 3dClustSim on data initially thresholded at a value of P < 0.001, uncorrected. Only correct trials were evaluated for all univariate analyses, with the exception of the accuracy map. Both these and other maps were masked by the positive main effect of task (P < 0.05, uncorrected) to remove areas that deactivated during task performance. Importantly, the definition of this mask does not incorporate information about, and is therefore independent of, task demands.
Split parametric map generation.
Split parametric maps were generated in a similar fashion as the procedures noted in the section above. However, tests of linear trends of motion coherence (distance from category boundary) were now assessed at each value of distance from category boundary (motion coherence).
Accuracy map generation.
To identify areas showing parametric representation of subject accuracies, we rank-ordered the mean accuracies in the 20 stimulus combinations into four separate bins. We then averaged the accuracies within each bin, vector-transformed the average accuracies of all bins to zero mean and a sum of squares equal to 1, and applied the weights to the estimated β-coefficients computed for the corresponding stimulus combination.
ANOVA analysis.
An ANOVA was performed with motion coherence and distance from boundary comprising the fixed factors and subjects the random factor.
ROI Selection
To confirm the findings of the whole brain analyses, we also performed region of interest (ROI) analyses to evaluate whether parametric effects could be seen at the level of the BOLD time courses. To avoid an ROI selection bias, 35% of the fMRI data collected (8 individual task runs per subject) was analyzed independently. To generate ROIs, we repeated the univariate main-effects analysis with the reduced data set, masked by the positive main effect of task. Specifically, after single-subject main-effects maps were normalized to MNI space and a group univariate analysis was performed, local maxima were defined on the main-effects group map (thresholded at P < 0.005, uncorrected). Each defined maximum served as the center of a sphere with a diameter of 8 mm. In cases in which neighboring spheres showed any overlap, the sphere with the lesser maximum was excluded. After reverse normalizing the ROIs to each subject’s native space, we selected the top 15 voxels from the training data set within each ROI that demonstrated a positive main effect of task. Each of these sets of voxels was then applied to the primary (and independent) data set. These selection criteria were useful for a number of reasons (Kayser et al. 2010a). First, by constraining ROI selection by the positive main effect of task, we ensured that we were not reporting areas that deactivated during task performance (see also Ho et al. 2009; Tosoni et al. 2008). Second, voxel selection was not based on a significant parametric response to either motion coherence or distance from category boundary. This choice ensured that we were not restricting ourselves to areas that represented stimulus feature alone. Finally, and most importantly, these voxels were selected from an independent data set totaling 72 runs for our subjects. Thus our selection criteria did not influence, and were not influenced by, the data ultimately analyzed.
BOLD Time Course Estimation
Estimates of the hemodynamic responses starting at the onset of the stimulus phase were calculated for each combination of motion coherence and distance from category boundary. To produce an unbiased estimate of the time course, we applied a deconvolution approach to the main data set using piecewise B-spline basis functions (Saad et al. 2006) separated by 100-ms intervals for 20 s after onset using AFNI’s 3dDeconvolve command. Since trial onset times were not synchronous with the transistor-transistor logic (TTL) pulse, across the entire run we were able to sample the time course at different points. The peak amplitude was defined as the first maximum in the average time course after stimulus onset, and time to peak was considered the time from onset to this maximum amplitude.
RESULTS
Behavior on Categorization Task
To study the effects of perceptual and categorical salience on subjects’ choices, we asked nine subjects to categorize the direction of motion of a dot-motion kinematogram as left or right of an oblique category boundary (Fig. 1A). Perceptual salience was altered by changing dot-motion coherence: salience increased as motion coherence increased (Fig. 1B). Categorical salience resulted from the direction of motion of the dots: salience increased as the direction of motion moved further away from the boundary line (Fig. 1C).
From behavioral training sessions performed outside the scanner, we selected one intermediate value for motion coherence (distances from category boundary) such that at the most salient distance from category boundary (coherence) subjects spanned the full range of behavior, from chance (50%) to 100% with an intermediate accuracy of 80% (IC, 4–25%; ID, 8–20°). Before pooling subject data to study group behavior, we compared accuracy across subjects for the intermediate values of motion coherence and distance from category boundary. Subjects did not deviate from the expected accuracy of 80% at the IC/ID from boundary [IC: t(8) = −0.43, P = 0.68; ID: t(8) = −0.08, P = 0.94].
Figure 2 shows behavioral data pooled across nine subjects as a function of distance from category boundary (Fig. 2, A and B) and motion coherence (Fig. 2, C and D). When salience was low in either domain, accuracy was at chance irrespective of stimulus strength in the other domain (Fig. 2, A and C, light gray lines). As salience increased, mean accuracy increased and RT decreased (Fig. 2, darker gray lines). Moreover, for a given level of salience in one domain, accuracy increased and RT decreased as salience increased in the other domain. A three-way mixed-effects ANOVA (coherence × distance from boundary × subject, with subject as a random effect) confirmed the interaction between perceptual and categorical information on both accuracy [F(12,96) = 41.05, P < 10−9] and RT [F(12,96) = 25.52, P < 10−9]. This interaction was not driven solely by the zero-salience conditions. When this three-way mixed-effects ANOVA was repeated while excluding all zero-motion coherence and zero-category distance conditions, the resulting interaction remained strongly significant for both accuracy [F(6,48) = 6.4, P = 0.001] and RT [F(6,48) = 20.62, P < 10−9]. This interaction was also not driven by the highest-salience conditions for both features. When the ANOVA was again repeated while excluding both the zero- and highest-salience conditions for motion coherence and category distance (leaving only 6 of the 20 feature combinations), RT still showed a significant interaction [F(2,16) = 8.12, P = 0.0037], while the interaction effect for accuracy reached trend significance [F(2,16) = 3.25, P = 0.066].
Fig. 2.
Behavior on the categorization task: accuracy (A and C) and reaction times (RTs; B and D) pooled from 9 subjects plotted as a function of distance from category boundary (A and B) and motion coherence (C and D). Progressively darker shades of gray indicate progressively higher coherence (A and B) and category distance (C and D) values. In A and B: ID, intermediate distance from category boundary (8–20°); In C and D: IC, intermediate motion coherence (4–25%, but see materials and methods). Error bars represent SEs.
To determine whether perceptual and categorical information differentially influenced behavior, we looked in greater detail at the very low- and very high-salience conditions. As expected, accuracy and RTs were modulated differentially at low vs. high salience but did not vary by domain of information. As categorical salience increased at high perceptual salience (100% coherence; Fig. 2, A and B, black lines), accuracy ranged from chance (50%) for each subject to 100% while mean RT decreased by an average of 557 ms across all subjects [t(8) = 10.07, P < 10−5]. As perceptual salience increased at high categorical salience (90° from category boundary; Fig. 2, C and D, black lines), accuracy again ranged from 50% for each subject to 100% while mean RT decreased by an average of 794 ms across all subjects [t(8) = 7.62, P < 10−4]. The differential decrease in mean RT between the perceptual and categorical conditions was not significant [t(8) = −1.91, P = 0.09].
In contrast, as perceptual salience increased at low categorical salience (e.g., dots moving along the category boundary), mean RT decreased by 256 ms. Nonetheless, this decrease was not significant [Fig. 2D, light gray line; t(8) = 2.18, P = 0.06]; neither was the differential decrease in RT between the perceptual and categorical conditions [t(8) = −1.91, P = 0.09]. These data demonstrate that both perceptual and categorical information modulate behavior in similar ways at different levels of salience.
Drift Diffusion Model and Domain Interaction
Despite the similarities in their overall effects on accuracy and RT, perceptual and categorical information potentially operate at different levels of abstraction, with motion coherence information representing a more concrete, stimulus-driven percept and categorical distance information being more abstract. Beyond the fact that the two domains interact to produce a decision, we wanted to investigate whether either domain influenced the decision more than the other. To this end we modeled behavior on the task using the proportional rate drift diffusion model defined by Palmer and colleagues (Palmer et al. 2005). We implemented three versions of the model that differed in the way perceptual and categorical information combined to determine the drift rate.
First, to directly evaluate the potential influence of the interaction between motion coherence and category distance, we defined the diffusion model so that drift rate was a function of each unique combination of perceptual and categorical salience (Interaction model). Thus in this version of the model we fit 20 different drift rates, corresponding to 1 drift rate for each combination of 4 motion coherences and 5 (absolute) distances from category boundary for each subject. This version of the model fit behavior quite well (Fig. 3A, solid lines). To confirm that the drift rates represented domain interactions, we fit a regression model separately for each subject with model drift rate as the dependent variable and motion coherence, distance from category boundary, and their interaction as independent variables. The interaction variable was significant (P < 0.05) for all subjects, whereas the separate domain variables were not significant for any subject (Table 2, Interaction model).
Fig. 3.
Proportional rate diffusion model fits. Model fits of low-level Interaction model (A), Distance model (B), and Coherence model (C) (see materials and methods for model details). Behavior on the categorization task, as shown in Fig. 2, A and B, is overlaid on each model plot (dashed lines). Top: accuracy. Bottom: RT. The color scheme is the same as that of Fig. 2. Error bars represent SEs.
Table 2.
Model regression values
| Model | Domain |
||
|---|---|---|---|
| Perceptual Median β | Categorical Median β | Interaction Median β | |
| All perceptual and categorical salience levels included | |||
| Interaction | 0.094 (0) | 0.436 (0) | 2.004 (9) |
| Distance | 0.237 (0) | 0.554 (3) | 1.635 (9) |
| Coherence | 0.355 (0) | 0.236 (1) | 3.069 (9) |
| Lowest perceptual and categorical salience levels excluded | |||
| Interaction | 0.906 (1) | 1.246 (1) | 3.043 (3) |
| Distance | 0.356 (0) | 0.611 (0) | 0.938 (2) |
| Coherence | 0.373 (0) | 0.356 (0) | 2.605 (3) |
Values are median regression coefficients obtained by regressing model drift rates against motion coherence (perceptual), distance from category boundary (categorical), and their interaction with all 20 stimulus combinations included and with lowest-salience conditions in both domains excluded, with numbers of subjects (of 9) exhibiting significant regression coefficients in parentheses.
To identify whether one domain fit choice behavior better than the other—specifically, whether distance from category boundary was more relevant than motion coherence—we implemented two additional versions of the model in which the stimulus strength component (x) of the drift rate was fixed by one domain and the proportionality constant (k) comprising the drift rate was determined by the other domain (see materials and methods). Model fits of the variant in which x was determined by categorical information (Distance model) are shown in Fig. 3B, while fits of the variant in which x was determined by perceptual information (Coherence model) are shown in Fig. 3C. Among all variants, the Distance model fit the data best, as confirmed by the lowest AICc score (Interaction: 1,877.9, Distance: 1,251.7, Coherence: 1,662.8). These differences between the AICc scores were significant across subjects for the Distance model compared with the other two models (Distance < Coherence: P = 0.002; Distance < Interaction: P = 0.002) but not for the Coherence model vs. the Interaction model [Coherence < Interaction: P = 0.1]. As with the Interaction model, we also fit a regression model to the drift rate parameters obtained from the Distance and Coherence models. In both cases, regression analysis revealed an additional significant (P < 0.05) interaction of the two domains in all subjects (Table 2). Significant effects of categorical information were also observed in a small subset of subjects in both model variants (Table 2, Distance, Coherence models). That the Distance model produces the best fit for the data suggests that categorical information is weighted more strongly than perceptual information. However, regression fits indicate, as expected, that both features are informative, in that the two domains interact to produce the final choice.
Presumably, interactions between domains are most important when salience in either domain is low. To address this possibility, we refit the models after excluding the conditions in which perceptual or categorical salience was absent. Here too, the Distance model performed best and the Coherence model the worst, as shown by the AICc scores (Interaction: 1,185.6, Distance: 798.2, Coherence: 1,329.5). These differences between the AICc scores remained significant for the Distance model compared with the other two models (Distance < Coherence: P = 0.002; Distance < Interaction: P = 0.002) but not for the Coherence model vs. the Interaction model (Coherence < Interaction: P = 0.82). Individual-subject domain interactions now retained significance (P < 0.05) in only a subset of subjects (Table 2).
However, it is possible that the differing fits of the Distance and Coherence models might have been biased by the greater number of distance values. To address this possibility, we reran the models while separately excluding the highest two levels (90° − ID and 90°) of distance from the category boundary, thereby matching the number of distance and coherence values. Lower values of the AICc for the Distance model were again seen when we excluded only the 90° − ID condition (Distance: 1,273, Coherence: 1,486.4, Interaction: 1,635.2) or only the 90° condition (Distance: 1,288.9, Coherence: 1,350.5, Interaction: 1,601.8). When the 90° − ID condition was excluded, these differences between the AICc scores were significant for all model comparisons (Distance < Coherence: P = 0.02; Distance < Interaction: P = 0.004; Coherence < Interaction: P = 0.048). When the 90° condition was excluded, these differences between the AICc scores were lower but no longer significant for Distance vs. Coherence (Distance < Coherence: P = 0.29) but remained so for the other two comparisons (Distance < Interaction: P = 0.01; Coherence < Interaction: P = 0.002). Overall, the modeling results suggest that perceptual and categorical information interact to influence the decision but that subjects weight categorical information more strongly because of the nature of the task. The models also suggest that domain interactions are most relevant when salience is low in either domain.
fMRI Analysis
Parametric representation of stimulus features within the brain.
To identify a neural correlate of the above model findings during RBC, we measured BOLD activity in the brain while subjects performed the task. We first identified areas that modulated the amplitude of their response as a function of perceptual and categorical salience. Decision making studies using only a single domain of information have established that nodes in the frontoparietal network (FPN) exhibit a negative parametric effect of stimulus salience (Hebart et al. 2012; Ho et al. 2009; Kayser et al. 2010b; Tosoni et al. 2008). In other words, BOLD amplitude decreases in these areas as stimulus salience increases—e.g., with increasing motion coherence (Kayser et al. 2010b). In contrast, when faced simultaneously with relevant as well as highly salient but irrelevant/unattended stimuli, the FPN response to the irrelevant stimulus feature demonstrates a positive parametric effect, with BOLD amplitude increasing with stimulus salience (Kayser et al. 2010a). Modeling results suggest that categorical information is weighted more strongly than perceptual information, so we hypothesized that the FPN would exhibit a negative parametric effect as a function of categorical salience and a relatively more positive parametric effect as a function of perceptual salience.
Figure 4A shows areas demonstrating a parametric modulation by categorical salience, while Fig. 4B shows the same for modulation by perceptual salience, masked by a positive main effect of task to exclude regions that deactivate during task performance. Here, cold (blue) colors represent a negative parametric effect—i.e., increasing BOLD activity with decreasing categorical or perceptual salience—while hot (red) colors indicate positive parametric effects—i.e., increasing BOLD activity with increasing categorical or perceptual salience. When processing categorical information, FPN areas including dorsolateral prefrontal cortex, anterior cingulate/pre-supplementary motor area, anterior insula, and intraparietal sulcus exhibited a strong bilateral negative parametric effect (Fig. 4A, blue; P < 0.05, corrected; Table 3). Perceptual information, on the other hand, produced a predominantly positive parametric effect (Fig. 4B, red; P < 0.05, corrected; Table 4) within FPN nodes, with the exception of anterior insula, which did not exhibit a significant parametric effect of perceptual salience. To confirm the importance of representations for both features within the FPN irrespective of the direction of the parametric effect, we also performed a whole brain contrast of the absolute value of parametric category distance, collapsed across all values of motion coherence, and the absolute value of parametric motion coherence, collapsed across all values of category distance. Consistent with the above findings, no brain regions survived significance correction (data not shown), indicating that while the sign of the parametric variation may differ in important ways, both features are represented in similar regions of the FPN.
Fig. 4.
Brain regions showing a parametric representation of salience in the categorical and perceptual domains. A: categorical salience (distance from category boundary). B: perceptual salience (motion coherence). Cold (blue) colors indicate negative parametric effects—i.e., increasing BOLD activity with decreasing categorical or perceptual salience—while hot (red) colors indicate positive parametric effects—i.e., increasing BOLD activity with increasing categorical or perceptual salience. All functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task (see materials and methods). Threshold t value = 3.97.
Table 3.
Brain areas showing a parametric effect of distance from category boundary
| Area | Hemi | Coordinates |
t Score | ||
|---|---|---|---|---|---|
| x | y | z | |||
| Negative parametric effect | |||||
| mIPS | L | −18 | −72 | 54 | −7.94 |
| aIPS | R | 39 | −42 | 45 | −7.94 |
| mIPS | R | 24 | −69 | 57 | −7.42 |
| MOG | R | 36 | −75 | 33 | −7.40 |
| aIPS | L | −36 | −45 | 42 | −7.39 |
| IFS | R | 54 | 6 | 33 | −7.33 |
| SMA | L | −3 | 18 | 45 | −6.77 |
| FEF | R | 30 | −6 | 51 | −6.71 |
| mIPS | L | −27 | −81 | 33 | −6.46 |
| aINS | L | −30 | 24 | 3 | −6.45 |
| IFS | L | −48 | 6 | 30 | −6.33 |
| FEF | L | −24 | −9 | 48 | −6.23 |
| Thalamus | L | −9 | −18 | 12 | −5.66 |
| Cerebellum | L | −12 | −54 | −54 | −5.61 |
| SMG | R | 63 | −24 | 39 | −5.53 |
| aINS | R | 36 | 21 | 6 | −5.34 |
| Cerebellum | R | 42 | −42 | −36 | −5.22 |
| MT+ | L | −45 | −69 | −9 | −5.08 |
| Midbrain | L | −3 | −27 | −24 | −5.00 |
| MT+ | R | 51 | −60 | −12 | −4.84 |
| Cerebellum | R | 12 | −78 | −24 | −4.69 |
| OPOLE | R | 15 | −96 | −3 | −4.34 |
| ACC | R | 12 | 24 | 27 | −4.27 |
| Cerebellum | R | 24 | −63 | −57 | −4.21 |
| SMG | L | −51 | −27 | 36 | −4.12 |
List of areas showing significant parametric effects of categorical salience. The functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task. ACC, anterior cingulate cortex; aINS, anterior insula; aIPS, anterior intraparietal sulcus; FEF, frontal eye fields; IFS, inferior frontal sulcus; mIPS, middle intraparietal sulcus; MOG, middle occipital gyrus; MT+, middle temporal area; OPOLE, occipital pole; SMA, supplementary motor area; SMG, supramarginal gyrus.
Table 4.
Brain areas showing a parametric effect of motion coherence
| Area | Hemi | Coordinates |
t Score | ||
|---|---|---|---|---|---|
| x | y | z | |||
| Positive parametric effect | |||||
| mIPS | L | −6 | −72 | 54 | 9.24 |
| aIPS | R | 51 | −42 | 57 | 9.13 |
| PCC | R | 3 | −30 | 27 | 6.77 |
| MOG | R | 45 | −78 | 27 | 6.53 |
| aIPS | L | −45 | −51 | 54 | 6.36 |
| mIPS | R | 15 | −78 | 42 | 6.30 |
| MT+ | R | 54 | −57 | −15 | 5.94 |
| Calcarine gyrus | R | 24 | −60 | 18 | 5.46 |
| SMG | L | −60 | −27 | 39 | 5.29 |
| mIPS | L | −27 | −69 | 42 | 4.96 |
| Fusiform gyrus | R | 27 | −54 | −15 | 4.53 |
| MT+ | L | −48 | −63 | −9 | 4.51 |
| OPOLE | R | 12 | −87 | −21 | 4.40 |
| OPOLE | L | −12 | −81 | −12 | 4.38 |
| Cerebellum | L | −36 | −81 | −24 | 4.22 |
| Negative parametric effect | |||||
| OPOLE | R | 18 | −99 | 3 | −3.80 |
List of areas showing significant parametric effects of perceptual salience. The functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task. aIPS, anterior intraparietal sulcus; mIPS, middle intraparietal sulcus; MOG, middle occipital gyrus; MT+: middle temporal area; OPOLE, occipital pole; PCC, posterior cingulate cortex; SMG, supramarginal gyrus.
Feature integration for categorical choices.
To address our behavioral and modeling results suggesting that perceptual and categorical information interact to influence subjects’ choices, we next evaluated the neural interaction between motion coherence and category distance. To identify FPN nodes wherein motion coherence and distance from category boundary interacted, we carried out a three-factor ANOVA in which subjects were included as a random effect (Fig. 5A; P < 0.05, corrected). A similar FPN demonstrated both the parametric effects of salience and their interaction. Table 5 provides the full list of areas showing significant interactions (P < 0.05, corrected) between perceptual and categorical information.
Fig. 5.
FPN regions showing a domain interaction (A and B) and a parametric representation of accuracy (C). A: to identify areas involved in processing both categorical and perceptual salience, a mixed-effects ANOVA was performed using factors of category distance and coherence. Brain regions that demonstrate a significant interaction between these two task features are shown in hot colors. Threshold F value = 3.09. B: to ascertain that the interaction was not driven solely by a qualitative change in behavior when no motion signal or categorization signal was available, the ANOVA was repeated after exclusion of the low-salience conditions. Brain regions that demonstrate a significant interaction are shown in hot colors. Threshold F value = 4.55. C: the parametric representation of accuracy was assessed by first binning accuracies in the 20 unique stimulus combinations into 4 bins, transforming the weights to a zero-mean vector with sum of squares equal to 1, and applying the resultant parametric weights to the stimulus regressors. The color scheme in C is the same as that of Fig. 4—i.e., negative parametric effects, in which BOLD activity increases with decreasing accuracy, are shown in cold colors, while positive parametric effects are shown in hot colors. All functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task (see materials and methods). Threshold t value = 3.97.
Table 5.
Brain areas showing interaction effects between distance from category boundary and motion coherence
| Area | Hemi | Coordinates |
F Score | ||
|---|---|---|---|---|---|
| x | y | z | |||
| MOG | R | 33 | −75 | 36 | 7.91 |
| FEF | L | −24 | −6 | 51 | 7.36 |
| MT+ | R | 54 | −60 | −9 | 5.80 |
| SMG | R | 63 | −24 | 42 | 5.62 |
| FEF | R | 30 | −9 | 57 | 4.16 |
| IFS | R | 57 | 9 | 36 | 3.80 |
| SMA | R | 3 | 12 | 54 | 3.73 |
| SMG | L | −45 | −30 | 36 | 3.49 |
| aIPS | R | 48 | −42 | 57 | 3.46 |
| mIPS | L | −18 | −72 | 54 | 3.40 |
| aIPS | L | −39 | −48 | 60 | 3.25 |
| mIPS | R | 12 | −72 | 60 | 3.13 |
| IFS | L | −51 | 9 | 33 | 2.72 |
List of areas showing the significant interaction of perceptual and categorical salience. The functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task. aIPS, anterior intraparietal sulcus; FEF, frontal eye fields; IFS, inferior frontal sulcus; mIPS, middle intraparietal sulcus; MOG, middle occipital gyrus; MT+, middle temporal area; SMA, supplementary motor area; SMG:, supramarginal gyrus.
However, one concern about this interaction relates to the inclusion of the 0% motion coherence and 0° category distance conditions, specifically, the concern that the interaction may be driven solely by a qualitative change in behavior when no motion signal or categorization signal is available. To address this possibility, we reran the above ANOVA after excluding all 0% motion coherence and 0° category distance conditions. As shown in Fig. 5B, a significant interaction between the two domains remained evident in many regions within the FPN, even when these zero-salience conditions were not included. Table 6 provides the full list of areas showing significant interactions (P < 0.05, corrected) when low-salience conditions were excluded.
Table 6.
Brain areas showing interaction effects after low-salience conditions were excluded
| Area | Hemi | Coordinates |
F Score | ||
|---|---|---|---|---|---|
| x | y | z | |||
| MOG | R | 42 | −75 | 27 | 5.26 |
| FEF | L | −24 | −6 | 57 | 5.53 |
| FEF | R | 30 | −9 | 54 | 6.26 |
| IFS | R | 54 | 9 | 36 | 5.56 |
| aIPS | R | 45 | −42 | 54 | 5.15 |
| mIPS | R | 27 | −69 | 54 | 5.29 |
List of areas showing the significant interaction of perceptual and categorical when low salience stimuli were eliminated. The functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task. aIPS, anterior intraparietal sulcus; FEF, frontal eye fields; IFS, inferior frontal sulcus; mIPS, middle intraparietal sulcus; MOG, middle occipital gyrus.
To affirm that the FPN nodes reflecting domain interactions were also behaviorally relevant [in that they reflected the more decision-relevant feature(s)] and to ensure that parametric responses to behavioral output as well as sensory input were investigated, we next related FPN activity directly to subject accuracy on the task. To do so, we created parametric weights corresponding to accuracies at different stimulus combinations, rather than to the salience of the stimulus features themselves, and evaluated FPN activity as a function of these weights (see materials and methods). These results are shown in Fig. 5C (P < 0.05, corrected; see also Table 7), with red and blue colors representing positive and negative parametric effects, respectively. The same FPN nodes demonstrating an interaction between stimulus domains also showed a negative parametric effect of accuracy, thus underscoring the relevance of these interactions to behavior. However, the gross parametric effects and ANOVA interaction map shown in Figs. 4 and 5 do not yet account for the parametric BOLD response at individual levels of salience in either domain.
Table 7.
Brain areas showing a parametric effect of accuracy
| Area | Hemi | Coordinates |
t Score | ||
|---|---|---|---|---|---|
| x | y | z | |||
| Negative parametric effect | |||||
| aIPS | R | 27 | −57 | 57 | −5.82 |
| SMA | 0 | 9 | 51 | −5.82 | |
| FEF | L | −27 | −9 | 48 | −5.79 |
| FEF | R | 30 | −9 | 51 | −5.49 |
| ACC | R | 12 | 24 | 30 | −5.38 |
| aINS | L | −30 | 24 | 3 | −5.13 |
| Cerebellum | 0 | −30 | −6 | −4.95 | |
| IFS | R | 54 | 3 | 33 | −4.60 |
| aIPS | R | 39 | −36 | 42 | −4.36 |
| mIPS | L | −21 | −63 | 54 | −4.21 |
| IFS | L | −51 | 0 | 33 | −4.01 |
| Positive parametric effect | |||||
| aIPS | R | 51 | −45 | 57 | 5.87 |
| Ang | R | 51 | −54 | −24 | 5.16 |
List of areas showing the significant parametric effects of accuracy. The functional maps were cluster size corrected to achieve a significance of P < 0.05 and masked by the positive main effect of task. ACC, anterior cingulate cortex; Ang, angular gyrus; aIPS, anterior intraparietal sulcus; FEF, frontal eye fields; IFS, inferior frontal sulcus; aINS, anterior insula; mIPS, middle intraparietal sulcus; SMA, supplementary motor area.
FPN dynamics as a function of feature salience.
To separately study FPN dynamics under low and high feature salience, we did the following: rather than evaluating parametric effects of perceptual salience collapsed across all values of categorical salience, as in Fig. 4, we evaluated the effects of perceptual salience at each level of categorical salience, and vice versa. At high salience in either domain, we expected the FPN to display a predominantly negative parametric effect as a function of the other feature—e.g., at high categorical salience, the choice would be driven primarily by variation in perceptual salience. In contrast, under low-salience conditions, we expected differential activity in the FPN depending on the domain of salience. Specifically, at low categorical salience, for which motion coherence remains decipherable, we expected the FPN to display a positive parametric effect of perceptual salience. On the other hand, at low perceptual salience, decision information is hidden from the subject. Thus we expected to see minimal but potentially differential parametric activity in the FPN under these two conditions.
The split parametric maps within the FPN (P < 0.05, corrected) are displayed in Fig. 6. Figure 6A shows parametric effects of categorical salience at each value of perceptual salience, while Fig. 6B shows the parametric effects of perceptual salience at each value of categorical salience. Consistent with the predictions of the model, at the lowest perceptual salience, when perceptual information was absent, (Fig. 6A, left), there was minimal parametric activity in the FPN. At the lowest categorical salience, on the other hand, parietal—but not frontal—cortex showed a robust positive parametric effect as a function of perceptual salience (Fig. 6B, left). Importantly, as motion coherence increased to perceptible values (Fig. 6A), the parametric effect of categorical salience grew more negative, consistent with its stronger weighting. In contrast, as categorical salience increased to strongly perceptible values (e.g., 45°), a positive parametric effect of perceptual salience could still be demonstrated, in keeping with the reduced weighting of this information. At the other end of the spectrum, when salience was high in either domain, the entire network showed negative parametric modulation as a function of salience in the other stimulus domain (Fig. 6, A and B, right). The corresponding time courses of BOLD activity within three representative nodes of the FPN are shown in Fig. 7. A robust effect of perceptual and categorical salience across feature values is visible in the peak amplitude of the time courses, providing further evidence that the parametric effects were not driven by any single level of salience.
Fig. 6.
Parametric representation of categorical salience at each level of perceptual salience, and vice versa, in frontoparietal areas. A: parametric map of categorical salience at each value of motion coherence. B: parametric map of perceptual salience at each value of distance from category boundary. The color scheme is the same as that of Fig. 4—i.e., negative and positive parametric effects are shown in cold and hot colors, respectively. All functional maps were cluster-size corrected to achieve a significance of P < 0.05 (see materials and methods) and masked by the positive main effect of task. Threshold t value = 3.97.
Fig. 7.
Time courses from 3 representative nodes in the FPN. A: time courses from the supplementary motor area (SMA) (MNI x = 3, y = 12, z = 51). B: time courses from left frontal eye fields (FEF) (MNI x = −24, y = −9, z = 48). C: time courses from left middle intraparietal sulcus (mIPS) (MNI x = −24, y = −63, z = 57). The color scheme is similar to that of Fig. 2. Progressively darker shades of gray indicate progressively higher category distance (top) and higher coherence (bottom). These results confirm the parametric effects noted in Fig. 4. a.u., Arbitrary units.
In addition to the split parametric effects, we evaluated domain-specific contributions and domain interactions by contrasting parametric maps in the high- and low-salience conditions (Fig. 8). This analysis directly compares the parametric changes in brain activity due to categorical and perceptual salience in specific conditions in order to define regions that respond preferentially to one or the other domain. The center panels in Fig. 8 are from Fig. 6: top panels are the extreme panels from Fig. 6A (0%, 100% coherence), and bottom panels are from Fig. 6B (0°, 90° distance from category boundary). Figure 8, A and B, show the contrast of parametric modulation of perceptual and categorical salience at the extreme levels of categorical and perceptual salience, respectively. All nodes of the FPN demonstrated changes in parametric activity along the salience spectrum in both domains, again indicating that both features were used to make categorical choices. Figure 8C shows the contrast of parametric effects at low salience in both domains, while Fig. 8D shows the contrast at high salience in both domains. At low salience, the contrast map is similar to the map showing parametric effects of coherence, highlighting the result that FPN activity at low salience was limited to parietal regions and driven primarily by perceptual information. At high salience, however, all nodes in the FPN exhibited parametric activity during categorization, indicating that both features contributed to the choice.
Fig. 8.
Contrast of parametric activity in one task feature at high and low salience in the other, confirming quantitatively that differences in parametric activation within the FPN distinguish these conditions. Center images are the extreme panels taken from Fig. 6, A and B, demonstrating parametric category activity at 0% and 100% coherence, respectively (top) and parametric motion coherence activity at 0° and 90° from category boundary, respectively (bottom). A: contrast of parametric category activity at 100% vs. 0% coherence. B: contrast of parametric motion coherence activity at 90° vs. 0° distance from category boundary. C: contrast of parametric motion coherence activity at 0° from category boundary with parametric category activity at 0% motion coherence. D: contrast of parametric category activity at 100% coherence with parametric motion coherence activity at 90° from category boundary. The color scheme is the same as that of Fig. 4—i.e., negative and positive parametric effects are shown in cold and hot colors, respectively. All functional maps were cluster size corrected to achieve a significance of P < 0.05 (see materials and methods) and masked by the positive main effect of task. Threshold t value = 3.97.
DISCUSSION
Perceptual decision making (PDM) refers to the process of evaluating incoming sensory information, typically low-level, noise-corrupted stimuli such as motion signals, in order to choose an action from potential alternatives (de Lafuente and Romo 2003; Gold and Shadlen 2007; Heekeren et al. 2008; Ratcliff and McKoon 2008). On the other hand, studies of rule-based categorization (RBC) typically use strong, perceptually salient stimuli that are mapped to different sets, and thereby to different responses, based on generalizable rules defined by category boundaries (Ashby and Maddox 2011). Thus variation in perceptual, as opposed to categorical, decision making may be reflected in more concrete, as opposed to more abstract, representations encoded within different decision-relevant brain regions. Alternatively, because both types of decision involve rule-based mappings from a stimulus to a response, pertinent stimulus features might be represented similarly within decision making circuitry that adapts to encode decision-relevant features. Here we sought to manipulate both sources of stimulus salience independently and simultaneously to characterize how decision making networks incorporate information from these differentially abstract stimulus domains (perceptual/concrete and categorical/abstract). Our results demonstrate not only the activation of similar networks of brain regions in both cases but also the importance of interactions between these two domains, especially when salience is low. However, both modeling and imaging results also suggest that categorical information is weighted more strongly than perceptual information, as driven by task demands.
Modeling Behavior During Categorization Task
Importantly, one feature of both PDM and RBC tasks is the potential to model such behavior with simple integration-to-bound accounts of decision making (Ratcliff and McKoon 2008). Here we demonstrated that a simple proportional rate drift diffusion model was sufficient to account for subject behavior as it varied with both perceptual and categorical salience. Interactions between perceptual and categorical salience were strongly evident in subject behavior, both for accuracy and RT data. Accordingly, the three versions of the diffusion model we tested suggested that interactions between perceptual and categorical information are necessary for decision making, especially when salience in either domain is low. However, as indicated by the quality of model fits, categorical information was weighted more strongly, likely by virtue of the fact that the decision ultimately required a categorical motor decision (i.e., a left or right button press). A similar phenomenon has been observed during the integration of multiple sensory modalities as well: integration is maximal when all modalities being integrated are individually suboptimal, but when any modality is highly salient decisions are weighted heavily on its basis (Senkowski et al. 2011; Stein et al. 2009; Stevenson et al. 2012). Conceptually, this finding argues that the FPN can flexibly incorporate the information that is most relevant to the decision.
While the diffusion model fit these data well, it was less accurate for low categorical salience and increasing perceptual salience—i.e., when most subjects displayed a decrease in RT without a corresponding increase in accuracy. From a behavioral perspective this finding likely results from the fact that, as motion coherence increased, subjects could more readily infer that decision-relevant categorical information was absent, thus prompting them to respond faster. The model’s inability to capture this effect is principally because the equations describing the model (Palmer et al. 2005) fit best when RT and accuracy change in concert with each other. A more complex model, possibly one incorporating collapsing decision thresholds, might be better able to account for systematic variations in RT when categorical evidence is absent. In addition, the model employed here produced summary values for model parameters that were obtained from all the data, rather than single-trial estimates of these values. In the future, models capable of deriving trial-by-trial estimates of categorical and perceptual variables for use in neuroimaging analyses might allow one to demonstrate how brain activity contributes to variation in such decisions across time.
FPN Activity Representing Stimulus Variability
In line with being conceptualized differently, PDM and RBC have also been thought to have distinct neural correlates within frontal and parietal cortices. In work in other animals, decision making paradigms have uncovered the existence of neurons that accumulate evidence over noisy sensory representations in FPN regions including the lateral intraparietal area (Roitman and Shadlen 2002) and the frontal eye fields (Schall 2003) as well as the basal ganglia (Ding and Gold 2010) and other regions (de Lafuente and Romo 2006). In contrast, a review of RBC studies emphasizes the existence of category-selective neurons throughout the primate brain (Seger and Miller 2010), although—consistent with PDM work—such studies have generally focused on the ability of frontal and parietal neurons to flexibly represent variable category salience in the face of consistent sensory stimuli (Freedman and Miller 2008; Swaminathan and Freedman 2012). More recently, reports that emphasize the commonality of perceptual and categorical decision making (Freedman and Assad 2011) have focused on the ability of parietal neurons to represent abstract outcomes that are linked neither to the specific source of salience nor to a specific motor response.
Despite their emphasis on different stimuli (from motion, to objects, to faces and facial expressions), human PDM studies have converged upon a similar FPN that represents these features in a fashion that varies parametrically with salience. Potentially consistent with work in primate LIP, EEG studies have argued for the existence of a centroparietal positivity that appears to represent a general decision making variable strongly correlated with the accumulation of sensory evidence (Kelly and O’Connell 2015). Categorization studies tend to find a similar network (Seger and Miller 2010; White et al. 2012), though with perhaps greater emphasis on frontal regions including the anterior cingulate/supplementary motor area and premotor regions (Grinband et al. 2006), especially during category learning (Ashby and Maddox 2011). Even studies that explicitly address salience in a different manner (e.g., by manipulating probabilities or risk) find its parametric representation within the FPN (Huettel et al. 2005; Lopez-Paniagua and Seger 2013; Wheeler et al. 2015), although the last study found that frontoparietal areas were even more active when the evidence for one response over the other switched during the trial. The authors argue that this sensitivity to switching may implicate a more general function of the FPN than evidence accumulation, such as attention or mental calculation (Wheeler et al. 2015). Alternative possibilities may be that because the BOLD signal is likely to incorporate neurons integrating evidence toward both decision boundaries the signal may be even greater on trials when strong evidence is present for both decisions or that multiple cognitive processes are incorporated within this network. Regardless, here we found that the FPN strongly represented both distance from the category boundary and motion coherence, as well as their interaction, arguing that common brain networks underlie PDM and RBC. However, in keeping with the diffusion model findings that category information might be preferentially weighted, we found a strongly negative parametric effect for category variation but a positive parametric effect for motion coherence, consistent with our previous finding that the BOLD response to more decision-relevant stimulus features varies inversely with stimulus salience—i.e., fewer neural resources are required as stimulus salience increases (Kayser et al. 2010a).
FPN Representation of Multiple Sources of Information
The above findings demonstrate that stimulus domains relevant to the decision will be represented within the FPN; however, how multiple sources of salience are represented remains an active area of investigation. In a recent set of studies manipulating the mean and variance of categorical information (de Gardelle and Summerfield 2011; Michael et al. 2015), FPN activation was found to vary depending on the source of stimulus variability. As salience in the mean decreased (analogous in the present study to dots moving closer to the category boundary), greater BOLD amplitude was seen within the entire network, similar to the negative parametric FPN activity observed here with varying categorical salience. In contrast, variance-related salience produced differential activity within the FPN: parietal areas showed a negative parametric effect, while medial frontal areas responded with a positive parametric variation. Because we did not have a direct correlate of variance-related salience, these results cannot be directly compared with results from our task; rather, we varied motion coherence, which impacts the variance in the category signal, and observed a positive parametric effect within the parietal components of the FPN. Based on our previous work (Kayser et al. 2010a), this positive parametric effect may reflect attentional resources having been allocated away from less relevant features—or, more generally, the relative absence of strong top-down modulation of such feature representations. Nonetheless, these results are in keeping with the fact that multiple stimulus features are represented parametrically within the FPN.
Notably, in a study of an information integration categorization (IIC) task, in which (unlike RBC) the categorization is difficult to verbalize and is typically learned implicitly, results were different (Seger et al. 2015). In this study, both distance from the decision boundary and distance from a prototype stimulus were independently varied. Activation within a cingulo-opercular network was greater for stimuli closer to the decision boundary, potentially reflecting increasing conflict between responses generated by procedural (i.e., striatal) systems, while activity within the superior parietal lobule was sensitive to distance from the prototype (Seger et al. 2015), in keeping with the potential involvement of a perceptual representation memory system (Ashby and Maddox 2011). These findings raise the intriguing possibility—as hypothesized elsewhere (Ashby and Maddox 2011)—that IIC tasks, and/or tasks that depend on a prototype, activate a different set of cognitive processes, and therefore a different network of brain regions, than RBC tasks.
Frontal-Parietal Dissociation and Cognitive Control Within FPN
In addition to distinguishing more relevant from less relevant stimulus features, parametric activity could dissociate frontal from parietal regions. Most notably, when category information was absent (because category distance was at 0°), only variability in motion coherence distinguished the different sensory stimuli, but this feature did not permit a decision to be made in the task. In this circumstance, BOLD activity that was positively parametric with motion coherence was seen within parietal regions, but no differential activity was seen within frontal regions. This finding argues that frontal activity only distinguishes task conditions when decision-relevant information permits selection of an appropriate response. Consistent with this idea, when we parameterized BOLD data by accuracy rather than by stimulus domain in the larger data set, a negative parametric effect was seen in frontal cortex. More broadly, this finding agrees with proposals in which the parietal cortex defines relational metrics that are then used by the frontal cortex to select appropriate actions (Genovesio et al. 2014).
Criticisms, Limitations, and Future Directions
One question that might arise from these results concerns whether choosing different values for categorical distance and motion coherence could shift the weighting of each of these features. Given that the feature values sampled here covered the full range of accuracy values and a range of RTs, we suspect that the weighting, both behaviorally and in the brain, would not differ. More importantly, our data argue that making a categorical decision would require a choice that necessitates evaluating the categorical information available in the task. On the other hand, if subjects were tasked to make judgments about the level of motion coherence rather than the direction of motion, we would hypothesize that the coherence information would be more strongly weighted and the directional information would be less strongly weighted. In fact, our previous work has demonstrated that directing subjects to make decisions based on one feature or another profoundly modulates the way in which the inferior frontal sulcus and intraparietal sulcus process those features in a PDM task (Kayser et al. 2010a). More generally, regions within the FPN should weight the feature(s), or the feature interactions, that are most relevant to the task at hand and may therefore reflect the operation of a flexible, multiple-demand frontal network (Duncan 2013) that is responsible for making first-order stimulus response associations. Thus the important difference in this task is not an unvarying property of the stimulus (e.g., direction of motion vs. motion coherence) or the level of abstraction but a property of the goals that the organism has defined. Asking subjects to judge the level of motion coherence rather than the direction of the stimulus relative to a boundary would consequently represent an interesting future direction.
Another theory concerning our findings centers on the possibility that the present behavioral, modeling, and imaging results simply reflect behavioral and brain differences between responses to imperceptible (zero salience) and perceptible (higher salience) features rather than to differential weighting of category distance and motion coherence. Because our behavioral data indicate that only zero motion coherence and zero category distance are imperceptible, we are able to exclude this possibility. First, our behavioral findings remain significant when these conditions are excluded, indicating that our findings do not depend on a division into perceptible and imperceptible regimes. Second, our imaging findings directly contradict this interpretation. Figure 4B demonstrates a positive parametric effect of motion coherence when the data are collapsed across all values for category distance; four of five of these category distance conditions give rise to above-chance behavior (Fig. 2A). Additionally, if one were concerned about the unlikely possibility that the 0° category distance condition dominates these other four category conditions, Fig. 6 demonstrates that at the ID and 45° conditions (and even at the 90° condition) there remains substantial positive parametric effect, and Fig. 7 demonstrates time courses that vary gradually and parametrically across feature values. Third, our behavioral, modeling, and imaging results all demonstrate a statistically significant interaction between the two features when the zero motion coherence and zero category distance conditions are excluded. Finally, even if one did agree with the idea that subthreshold vs. suprathreshold feature values could explain the switch from positive to negative parametric effect in the distance plots in Fig. 6A, this explanation would not account for the transition from positive to negative parametric effect between the 45° and 90° − ID distances in the coherence plots in Fig. 6B, both of which values are highly perceptible. Moreover, this point of view would not explain our previous work (Kayser et al. 2010a) in which highly salient values of the same feature, either motion coherence or color coherence in that case, demonstrated a negative parametric effect if relevant and a positive parametric effect if not. Thus we do not believe that a dichotomous classification of the stimulus features into perceptible and imperceptible adequately explains these data.
Conclusions
Here we demonstrate that information from perceptual and categorical domains interact to produce choices, albeit with preferential weighting for categorical information. These results highlight the importance of goal-relevant information and its representation within FPN regions, and they are in keeping with the idea that these frontoparietal regions comprise a multiple-demand network that is activated by a variety of tasks (Duncan 2013; Fedorenko et al. 2013) and that represents salience to the extent that it is relevant to goals. Additionally, these findings demonstrate that, although frontal and parietal regions frequently coactivate, their roles in decision making are distinct—as is especially evident in cases in which parametric but ultimately uninformative sensory information is available. Given the diversity of cognitive processes mediated by the FPN, further understanding the mechanisms by which multiple salient stimulus features are represented within it, particularly when categorization is not rule based, may provide another useful window into this important flexibility.
GRANTS
This research was supported by funding from National Eye Institute Grant EY-024554 (A. S. Kayser) and the state of California (A. S. Kayser).
DISCLOSURES
No conflicts of interest, financial or otherwise, are declared by the author(s).
AUTHOR CONTRIBUTIONS
S.S. and A.S.K. conceived and designed research; S.S. performed experiments; S.S. analyzed data; S.S. and A.S.K. interpreted results of experiments; S.S. and A.S.K. prepared figures; S.S. and A.S.K. drafted manuscript; S.S. and A.S.K. edited and revised manuscript; S.S. and A.S.K. approved final version of manuscript.
ACKNOWLEDGMENTS
We thank the subjects for their participation.
REFERENCES
- Anderson DR, Burnham KP, White GC. AIC model selection in overdispersed capture-recapture data. Ecology 75: 1780–1793, 1994. doi: 10.2307/1939637. [DOI] [Google Scholar]
- Ashby FG, Maddox WT. Human category learning 2.0. Ann NY Acad Sci 1224: 147–161, 2011. doi: 10.1111/j.1749-6632.2010.05874.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ball K, Sekuler R. Direction-specific improvement in motion discrimination. Vision Res 27: 953–965, 1987. doi: 10.1016/0042-6989(87)90011-3. [DOI] [PubMed] [Google Scholar]
- Brainard DH. The Psychophysics Toolbox. Spat Vis 10: 433–436, 1997. doi: 10.1163/156856897X00357. [DOI] [PubMed] [Google Scholar]
- Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociol Methods Res 33: 261–304, 2004. doi: 10.1177/0049124104268644. [DOI] [Google Scholar]
- Dale AM. Optimal experimental design for event-related fMRI. Hum Brain Mapp 8: 109–114, 1999. doi:. [DOI] [PMC free article] [PubMed] [Google Scholar]
- de Gardelle V, Summerfield C. Robust averaging during perceptual judgment. Proc Natl Acad Sci USA 108: 13341–13346, 2011. doi: 10.1073/pnas.1104517108. [DOI] [PMC free article] [PubMed] [Google Scholar]
- de Lafuente V, Romo R. Decisions arising from opposing views. Nat Neurosci 6: 792–793, 2003. doi: 10.1038/nn0803-792. [DOI] [PubMed] [Google Scholar]
- de Lafuente V, Romo R. Neural correlate of subjective sensory experience gradually builds up across cortical areas. Proc Natl Acad Sci USA 103: 14266–14271, 2006. doi: 10.1073/pnas.0605826103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Ding L, Gold JI. Caudate encodes multiple computations for perceptual decisions. J Neurosci 30: 15747–15759, 2010. doi: 10.1523/JNEUROSCI.2894-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Duncan J. The structure of cognition: attentional episodes in mind and brain. Neuron 80: 35–50, 2013. doi: 10.1016/j.neuron.2013.09.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fedorenko E, Duncan J, Kanwisher N. Broad domain generality in focal regions of frontal and parietal cortex. Proc Natl Acad Sci USA 110: 16616–16621, 2013. doi: 10.1073/pnas.1315235110. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Freedman DJ, Assad JA. Experience-dependent representation of visual categories in parietal cortex. Nature 443: 85–88, 2006. doi: 10.1038/nature05078. [DOI] [PubMed] [Google Scholar]
- Freedman DJ, Assad JA. A proposed common neural mechanism for categorization and perceptual decisions. Nat Neurosci 14: 143–146, 2011. doi: 10.1038/nn.2740. [DOI] [PubMed] [Google Scholar]
- Freedman DJ, Miller EK. Neural mechanisms of visual categorization: insights from neurophysiology. Neurosci Biobehav Rev 32: 311–329, 2008. doi: 10.1016/j.neubiorev.2007.07.011. [DOI] [PubMed] [Google Scholar]
- Freedman DJ, Riesenhuber M, Poggio T, Miller EK. Categorical representation of visual stimuli in the primate prefrontal cortex. Science 291: 312–316, 2001. doi: 10.1126/science.291.5502.312. [DOI] [PubMed] [Google Scholar]
- Gauthier I, Tarr MJ. Becoming a “Greeble” expert: exploring mechanisms for face recognition. Vision Res 37: 1673–1682, 1997. doi: 10.1016/S0042-6989(96)00286-6. [DOI] [PubMed] [Google Scholar]
- Genovesio A, Wise SP, Passingham RE. Prefrontal-parietal function: from foraging to foresight. Trends Cogn Sci 18: 72–81, 2014. doi: 10.1016/j.tics.2013.11.007. [DOI] [PubMed] [Google Scholar]
- Gold JI, Shadlen MN. The neural basis of decision making. Annu Rev Neurosci 30: 535–574, 2007. doi: 10.1146/annurev.neuro.29.051605.113038. [DOI] [PubMed] [Google Scholar]
- Grinband J, Hirsch J, Ferrera VP. A neural representation of categorization uncertainty in the human brain. Neuron 49: 757–763, 2006. doi: 10.1016/j.neuron.2006.01.032. [DOI] [PubMed] [Google Scholar]
- Hebart MN, Donner TH, Haynes JD. Human visual and parietal cortex encode visual choices independent of motor plans. Neuroimage 63: 1393–1403, 2012. doi: 10.1016/j.neuroimage.2012.08.027. [DOI] [PubMed] [Google Scholar]
- Heekeren HR, Marrett S, Bandettini PA, Ungerleider LG. A general mechanism for perceptual decision-making in the human brain. Nature 431: 859–862, 2004. doi: 10.1038/nature02966. [DOI] [PubMed] [Google Scholar]
- Heekeren HR, Marrett S, Ruff DA, Bandettini PA, Ungerleider LG. Involvement of human left dorsolateral prefrontal cortex in perceptual decision making is independent of response modality. Proc Natl Acad Sci USA 103: 10023–10028, 2006. doi: 10.1073/pnas.0603949103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Heekeren HR, Marrett S, Ungerleider LG. The neural systems that mediate human perceptual decision making. Nat Rev Neurosci 9: 467–479, 2008. doi: 10.1038/nrn2374. [DOI] [PubMed] [Google Scholar]
- Ho TC, Brown S, Serences JT. Domain general mechanisms of perceptual decision making in human cortex. J Neurosci 29: 8675–8687, 2009. doi: 10.1523/JNEUROSCI.5984-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Huettel SA, Song AW, McCarthy G. Decisions under uncertainty: probabilistic context influences activation of prefrontal and parietal cortices. J Neurosci 25: 3304–3311, 2005. doi: 10.1523/JNEUROSCI.5070-04.2005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hurvich CM, Tsai CL. Regression and time-series model selection in small samples. Biometrika 76: 297–307, 1989. doi: 10.1093/biomet/76.2.297. [DOI] [Google Scholar]
- Kayser AS, Buchsbaum BR, Erickson DT, D’Esposito M. The functional anatomy of a perceptual decision in the human brain. J Neurophysiol 103: 1179–1194, 2010b. doi: 10.1152/jn.00364.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kayser AS, Erickson DT, Buchsbaum BR, D’Esposito M. Neural representations of relevant and irrelevant features in perceptual decision making. J Neurosci 30: 15778–15789, 2010a. doi: 10.1523/JNEUROSCI.3163-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kelly SP, O’Connell RG. The neural processes underlying perceptual decision making in humans: recent progress and future directions. J Physiol Paris 109: 27–37, 2015. doi: 10.1016/j.jphysparis.2014.08.003. [DOI] [PubMed] [Google Scholar]
- Lopez-Paniagua D, Seger CA. Coding of level of ambiguity within neural systems mediating choice. Front Neurosci 7: 229, 2013. doi: 10.3389/fnins.2013.00229. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Matthews N, Welch L. Velocity-dependent improvements in single-dot direction discrimination. Percept Psychophys 59: 60–72, 1997. doi: 10.3758/BF03206848. [DOI] [PubMed] [Google Scholar]
- Michael E, de Gardelle V, Nevado-Holgado A, Summerfield C. Unreliable evidence: 2 sources of uncertainty during perceptual choice. Cereb Cortex 25: 937–947, 2015. doi: 10.1093/cercor/bht287. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Palmer J, Huk AC, Shadlen MN. The effect of stimulus strength on the speed and accuracy of a perceptual decision. J Vis 5: 376–404, 2005. doi: 10.1167/5.5.1. [DOI] [PubMed] [Google Scholar]
- Pelli DG. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10: 437–442, 1997. doi: 10.1163/156856897X00366. [DOI] [PubMed] [Google Scholar]
- Philiastides MG, Sajda P. Temporal characterization of the neural correlates of perceptual decision making in the human brain. Cereb Cortex 16: 509–518, 2006. doi: 10.1093/cercor/bhi130. [DOI] [PubMed] [Google Scholar]
- Ratcliff R, McKoon G. The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput 20: 873–922, 2008. doi: 10.1162/neco.2008.12-06-420. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Roitman JD, Shadlen MN. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J Neurosci 22: 9475–9489, 2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Saad ZS, Chen G, Reynolds RC, Christidis PP, Hammett KR, Bellgowan PS, Cox RW. Functional imaging analysis contest (FIAC) analysis according to AFNI and SUMA. Hum Brain Mapp 27: 417–424, 2006. doi: 10.1002/hbm.20247. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schall JD. Neural correlates of decision processes: neural and mental chronometry. Curr Opin Neurobiol 13: 182–186, 2003. doi: 10.1016/S0959-4388(03)00039-4. [DOI] [PubMed] [Google Scholar]
- Scholl CA, Jiang X, Martin JG, Riesenhuber M. Time course of shape and category selectivity revealed by EEG rapid adaptation. J Cogn Neurosci 26: 408–421, 2014. doi: 10.1162/jocn_a_00477. [DOI] [PubMed] [Google Scholar]
- Seger CA, Braunlich K, Wehe HS, Liu Z. Generalization in category learning: the roles of representational and decisional uncertainty. J Neurosci 35: 8802–8812, 2015. doi: 10.1523/JNEUROSCI.0654-15.2015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Seger CA, Miller EK. Category learning in the brain. Annu Rev Neurosci 33: 203–219, 2010. doi: 10.1146/annurev.neuro.051508.135546. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Senkowski D, Saint-Amour D, Höfle M, Foxe JJ. Multisensory interactions in early evoked brain activity follow the principle of inverse effectiveness. Neuroimage 56: 2200–2208, 2011. doi: 10.1016/j.neuroimage.2011.03.075. [DOI] [PubMed] [Google Scholar]
- Stanford TR, Shankar S, Massoglia DP, Costello MG, Salinas E. Perceptual decision making in less than 30 milliseconds. Nat Neurosci 13: 379–385, 2010. doi: 10.1038/nn.2485. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stein BE, Stanford TR, Ramachandran R, Perrault TJ Jr, Rowland BA. Challenges in quantifying multisensory integration: alternative criteria, models, and inverse effectiveness. Exp Brain Res 198: 113–126, 2009. doi: 10.1007/s00221-009-1880-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stevenson RA, Bushmakin M, Kim S, Wallace MT, Puce A, James TW. Inverse effectiveness and multisensory interactions in visual event-related potentials with audiovisual speech. Brain Topogr 25: 308–326, 2012. doi: 10.1007/s10548-012-0220-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Swaminathan SK, Freedman DJ. Preferential encoding of visual categories in parietal cortex compared with prefrontal cortex. Nat Neurosci 15: 315–320, 2012. doi: 10.1038/nn.3016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tosoni A, Galati G, Romani GL, Corbetta M. Sensory-motor mechanisms in human parietal cortex underlie arbitrary visual decisions. Nat Neurosci 11: 1446–1453, 2008. doi: 10.1038/nn.2221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tremel JJ, Wheeler ME. Content-specific evidence accumulation in inferior temporal cortex during perceptual decision-making. Neuroimage 109: 35–49, 2015. doi: 10.1016/j.neuroimage.2014.12.072. [DOI] [PMC free article] [PubMed] [Google Scholar]
- van Atteveldt N, Murray MM, Thut G, Schroeder CE. Multisensory integration: flexible use of general operations. Neuron 81: 1240–1253, 2014. doi: 10.1016/j.neuron.2014.02.044. [DOI] [PMC free article] [PubMed] [Google Scholar]
- van Eeuwijk FA. Multiplicative interaction in generalized linear models. Biometrics 51: 1017–1032, 1995. doi: 10.2307/2533001. [DOI] [Google Scholar]
- Wheeler ME, Woo SG, Ansel T, Tremel JJ, Collier AL, Velanova K, Ploran EJ, Yang T. The strength of gradually accruing probabilistic evidence modulates brain activity during a categorical decision. J Cogn Neurosci 27: 705–719, 2015. doi: 10.1162/jocn_a_00739. [DOI] [PMC free article] [PubMed] [Google Scholar]
- White CN, Mumford JA, Poldrack RA. Perceptual criteria in the human brain. J Neurosci 32: 16716–16724, 2012. doi: 10.1523/JNEUROSCI.1744-12.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
