Skip to main content
PMC home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2012 Nov 21;32(47):16716–16724. doi: 10.1523/JNEUROSCI.1744-12.2012

Perceptual Criteria in the Human Brain

Corey N White 1,✉, Jeanette A Mumford 1, Russell A Poldrack 1,2
PMCID: PMC6621768  PMID: 23175825

Abstract

A critical component of decision making is the ability to adjust criteria for classifying stimuli. fMRI and drift diffusion models were used to explore the neural representations of perceptual criteria in decision making. The specific focus was on the relative engagement of perceptual- and decision-related neural systems in response to adjustments in perceptual criteria. Human participants classified visual stimuli as big or small based on criteria of different sizes, which effectively biased their choices toward one response over the other. A drift diffusion model was fit to the behavioral data to extract estimates of stimulus size, criterion size, and difficulty for each participant and condition. These parameter values were used as modulated regressors to create a highly constrained model for the fMRI analysis that accounted for several components of the decision process. The results show that perceptual criteria values were reflected by activity in left inferior temporal cortex, a region known to represent objects and their physical properties, whereas stimulus size was reflected by activation in occipital cortex. A frontoparietal network of regions, including dorsolateral prefrontal cortex and superior parietal lobule, corresponded to the decision variables resulting from the downstream stimulus–criterion comparison, independent of stimulus type. The results provide novel evidence that perceptual criteria are represented in stimulus space and serve as inputs to be compared with the presented stimulus, recruiting a common network of decision regions shown to be active in other simple decisions. This work advances our understanding of the neural correlates of decision flexibility and adjustments of behavioral bias.

Introduction

Humans flexibly adjust internal criteria to suit their current goals and motivational states. Internal criteria can be thought of as specific values for classification decisions; when deciding whether an object is big or small, the criterial value should be based on the relevant dimension of size. Decisions involving such criteria are used by cognitive scientists to probe different aspects of cognitive processing, including perception (“is this bright or dark?”), memory (“is this old or new?”), and emotion (“is this happy or sad?”), among many others. The goal of this work was to determine how such criteria are represented in the brain and how they relate to other components of the decision process.

The relationship between the stimulus and the internal criterion in simple decisions has been formalized in two influential models, signal detection theory (Green and Swets, 1966) and the drift diffusion model (DDM) (Ratcliff, 1978). Figure 1 illustrates these decision models for a perceptual task in which participants classify lines as big or small relative to a perceptual criterion (dashed line). Signal detection theory assumes that a noisy stimulus is compared with a criterial value along the same dimension to produce the decision. DDMs take this framework further by describing how the decision evidence from the comparison process is sampled and accumulated over time. In this framework, adjusting the criterion effectively biases choices by shifting the decision evidence in favor of one response.

Figure 1.

Figure 1.

Open in a new tab

Example decision scenario in which a person decides whether lines are big or small. The decision depends on the person's criteria for how big the line should be to be labeled big. The left panels show this situation in a signal detection representation, the middle panels show the corresponding drift diffusion representation, and the right panels show the hypothetical choice behavior. The top row shows one condition with relatively small criterion, leading to many “big” responses. The bottom row shows a comparison condition in which the stimulus is the same but the criterion is larger, leading to a bias to respond “small.”

It is unclear precisely how such criteria are represented neurally in relation to other aspects of the decision process. Previous work with learned perceptual criteria showed criteria-related activation in both decision- and perceptual-related neural systems (Li et al., 2009, 2012), although it is not clear which effects are specific to the representation of the criteria and which reflect downstream decision variables. The present study used fMRI and decision models to dissociate criteria values from other decision variables and test whether perceptual criteria are reflected by activation of general decision regions, such as dorsolateral prefrontal cortex (dlPFC); or rather by perceptually related regions like inferior temporal cortex. Participants performed a task in the scanner in which they decided whether perceptual objects were big or small. For each block of trials, a perceptual criterion with one of three different sizes was presented to use for the comparison process (see Fig. 2). A DDM was fit to the behavioral data to extract relevant parameters for the decision components, resulting in a highly constrained model to probe the fMRI data that simultaneously accounted for values of stimulus size, criterion size, difficulty, and motor activation for each condition in the task. By dissociating each component of the decision process, this design allowed investigation of the neural representations of the internal perceptual criteria and the systems involved in the formulation and accumulation of decision evidence resulting from comparing the stimulus to the criterion. This combination of fMRI and cognitive modeling provides a theoretically driven approach to understanding how the brain uses perceptual criteria to adjust the decision process.

Figure 2.

Figure 2.

Open in a new tab

Task details. Left, Stimuli and perceptual criteria used for perceptual discrimination. The pattern of stimulus and criterion sizes was parallel for the dots and lines. Right, Trial sequence for each block of trials.

Materials and Methods

Participants.

Healthy adults were recruited through posted flyers and prescreened. Prescreening criteria included being of age 18–40, having a minimum of eighth grade education, speaking English fluently, being right-handed, having no history of significant medical illness, having no history of major psychotic disorders, having no history of head trauma, having no metal implants, not being pregnant, and not currently using any medication for psychiatric reasons. Exclusion criteria were set a priori: participants were excluded if scans had poor quality or participants had excessive movement (translational displacement, >2.5 mm in any plane). Twenty-six participants (15 females) were scanned and 2 were excluded from analysis: one for poor scan quality and one for excessive motion. Of the remaining, all participants except one had four runs of the task in the scanner; the exception only had three runs because of an error during the last scan.

Stimuli and task.

Two sets of stimuli were used (Fig. 2): vertical lines and patches of dots. Each stimulus type had six different sizes. Line stimuli had lengths of 0.6, 1.0, 1.4, 1.8, 2.2, or 2.6 cm. Noise was added to each presented line stimulus by drawing a value from a uniform distribution spanning −0.15 to 0.15 cm and adding it to the lines. The largest line subtended a visual angle of 4.3°. Dot stimuli were presented in a 10 × 10 array that subtended 5° visual angle, with dot densities of 25, 35, 45, 55, 65, or 75. Each dot in the display was an asterisk symbol (*) with font size 40. Dots were placed randomly in the array for each trial, resulting in trialwise variability in the stimulus similar to the line stimuli. However, the number of dots for each condition did not vary from trial to trial. There were small, medium, and large criteria for each stimulus type (Fig. 2). For lines, the criterial lengths were 1.2 (small), 1.6 (medium), or 2.0 cm (large). For dots, the criterial number of dots was 40 (small), 50 (medium), or 60 (large).

Each stimulus type was presented in a block of 48 trials using the Psychtoolbox (Brainard, 1997) in MATLAB. At the beginning of each block, participants were shown which perceptual criterion (midpoint) to use for that block. To avoid response bias effects outside of those induced by the criterion manipulation, the number of big and small stimuli was equated for each block. For example, when using the small criterion, only two of the line lengths should be labeled small, whereas four should be labeled large. To overcome this asymmetry, each of the small stimuli was presented twice as often as the large stimuli. The opposite was done for blocks with the large perceptual criterion.

Stimulus order was pseudorandom within a block based on the output of efficiency simulations. One thousand random trial orders were simulated for each criterion size, and efficiency was calculated for contrasts based on stimulus size and difficulty (the within-block manipulations). The 30 trial orders with the highest joint efficiency for the contrasts were selected for each criterion size. The most efficient designs tended to involve repetitions of stimulus size, so to maintain psychological validity only those designs with no more than three repetitions of a stimulus were retained. Trial order for each block in the experiment was chosen at random from the remaining possibilities.

Participants first performed a behavioral session in the laboratory to familiarize themselves with the task and learn the criteria. For each trial, a central fixation cue was presented for 200 ms, followed by the stimulus. The dot stimuli were presented in the center of the screen, but the line stimuli were presented just left of center to avoid overlap with the fixation cross. The stimulus remained on screen until a response was given or until the trial timed out. Participants were instructed to press the “z” and “/” keys to indicate whether the stimulus was big or small (button mapping was randomized across participant). They were given up to 1.3 s for each decision. If a response was not given within 1.3 s, the trial ended with no response and the next trial began. The response was followed by 500 ms of feedback. Each trial sequence took 2 s total (Fig. 2).

The behavioral session was broken into four lists lasting 10 min and 34 s each. Lists consisted of three blocks (one of each criterion size) of the line-length judgment and three blocks (one of each criterion size) of the dot judgment. All three blocks for a stimulus type (e.g., lines) were performed before the other stimulus type was tested. The order of stimulus type and criterion size was counterbalanced across runs. At the beginning of each block, the perceptual criterion was presented for 10 s with an example stimulus of that size. There was also a 15 s break when switching from one stimulus type to the other. The first three lists included error feedback after every trial, whereas the final list used no feedback.

The testing session was conducted in the scanner no more than 7 d after the training session. Before entering the scanner, participants first completed a shortened practice list with 24 trials per block (compared with 48), using the same design as above and error feedback. Once in the scanner, four lists of the same composition as the practice session were performed. Responses were collected using a separate button box for each hand with big/small responses mapped to the right or left hand according to the button mapping from the behavioral session. No error feedback was used for the scanning session.

Diffusion model fitting.

The DDM (Ratcliff and Smith, 2004; Ratcliff and McKoon, 2008) was fit to the behavioral data for each participant individually, and separate fits were conducted for line and dot trials. The DDM included the following parameters which were held constant across condition in the fitting process: boundary separation (a), nondecision time (Ter), starting point (z/a), across-trial variability in drift rate (η), across-trial variability in starting point (sz), and across-trial variability in nondecision time (st). Within-trial variability in the diffusion process, s, can act as a scaling parameter and was fixed at 0.1. In a DDM, a separate drift rate is used to estimate the strength of decision evidence for each condition; thus, 18 drift rates (6 stimulus × 3 criteria) were needed for each task, with positive drift rates leading to “big” responses and negative drift rates leading to “small” responses. However, this was accomplished using only six free parameters corresponding to the sizes of the stimulus and criterion for different conditions. Because the decision always depends on the size of the stimulus relative to the criterion, the center of the perceptual space (e.g., 50 dots) could defined as the 0 point with parameters calculated relative to that. Stimulus size was represented by three parameters (s1, s2, s3) indexing the distance from the stimulus to the 0 point (see Fig. 4). For example, if s1 = 0.5, s2 = 0.3, and s3 = 0.1, the stimulus size values for the six stimuli from smallest to largest would be [−0.5 −0.3 −0.1 0.1 0.3 0.5]. The same schema was used to represent the perceptual criterion, using one parameter (r) for each criterion size. In this framework, the drift rate for each of the 18 conditions was calculated as the difference between the stimulus and criterion values (si − ri), with positive values of drift leading to “big” responses and negative values leading to “small” responses. Thus, the DDM was fit to all 18 conditions (6 stimuli × 3 criteria) for a task (e.g., dots or lines) with only s_i allowed to vary across stimulus size and r_i allowed to vary across criterion size. This framework allows all 18 drift rates to be estimated with relatively few free parameters, and also captures the assumption that drift rates are a function of the distance between the stimulus and criterion. Note that this decomposition into stimulus and criteria values was only possible because of the experimental manipulation; because the drift rate is calculated as si − ri, the values of s and r are only constrained by knowledge of the experimental conditions (increasing s has the same effect as decreasing r). By fitting all 18 conditions simultaneously, si and ri become highly constrained relative to each other, providing relevant information about changes in these values across condition. Importantly, because the DDM uses all of the behavioral data when estimating the decision components for each participant, including percentage correct and response time (RT) for correct and error responses, it provides more accurate estimates of decision evidence than obtained with the mean RTs or accuracy values alone (White et al., 2010).

Figure 4.

Figure 4.

Open in a new tab

DDM parameters. A, Estimated values of stimulus size (dashed gray) and criteria size (solid gray) from DDM fits are displayed (see text). The black lines represent the physical values of the criteria presented at the beginning of each block, and the gray lines represent the estimated values used by the participants. The estimated model parameters were regressed onto the objective physical sizes to show how they relate. Values of si correspond to the size of the stimulus relative to the 0 point (e.g., 50 dots in the dot trials), and values of ri correspond to the size of the perceptual criteria. The model fits show that participants effectively shifted their criteria, but did not perfectly match the physical values that were presented. B, Fit quality from the best-fitting DDM parameters is displayed. Predicted versus observed values are plotted for accuracy and the quantiles of the RT distributions. Each point indexes a participant's value for a condition and values along the diagonal show good correspondence between predicted and observed values. The plot shows that the DDM parameters captured the accuracy and RT values successfully, supporting their use in the GLM analysis.

Parameter search was conducted using a SIMPLEX routine to minimize a χ2 value (Ratcliff and Tuerlinckx, 2002). The data entered into the routine were the accuracy values, number of observations, and RT distributions for correct and error responses. RT distribution shape was represented by taking the 0.1, 0.3, 0.5 (median), 0.7, and 0.9 quantiles of the RT distribution. Thus, all the behavioral data were used in estimating the DDM parameters.

fMRI acquisition and analysis.

Imaging data were collected using a General Electric 3T Signa Excite MR scanner. Functional data were collected using a T2*-weighted echo-planar imaging sequence with 90° flip angle (TR, 2.88 s; TE, 30.5 ms; FOV, 22 cm2). Thirty-eight oblique axial slices were collected in interleaved fashion with 0.8 mm spacing. To reduce dropout in orbitofrontal cortex, the slices were tilted at a 10–15° angle off of the anterior-commissure–posterior-commissure line and higher-order shimming was used. A T2-weighted anatomical image with the same prescription as the functional scans was collected with a fast-spin echo sequence (TR, 5100) in addition to a high-resolution T1-weighted anatomical image. A fixed intertrial interval of 2 s was chosen to maximize the number of trials per condition for contrasting the experimental conditions (no null events were included).

DICOM data were converted to NIFTI images and before applying the trial-specific estimation model, and data preprocessing was performed in FEAT, version 5.98, part of FSL (Smith et al., 2004). The following preprocessing operations were applied: image time series were aligned using the MCFLIRT tool (resulting in six motion parameters describing the x/y/z translation and rotation of the brain over time); the skull was removed from the image using the brain extraction tool (BET); spatial smoothing was performed using a Gaussian kernel of FWHM of 5 mm; grand-mean scaling of the entire 4D data set was performed using a single multiplicative factor; and high-pass temporal filtering was performed (Gaussian-weighted least-squares straight line fitting with σ = 50 s). Note that this same high-pass filter was applied to the design matrix when estimating the trial-by-trial estimates.

The general linear model (GLM) included nuisance regressors for RT, error trials, correct trials, line trials, and dot trials. Motion over time was also added to the model by using the six motion parameter time series (described above). The RT regressors were orthogonalized relative to the line and dot trial regressors. Modulated trialwise regressors for stimulus size, criterion size, and difficulty were included based on the parameters from the DDM for each participant. The values of each modulated regressor were mean-centered before inclusion in the GLM. There was a stimulus size and criterion size regressor for each stimulus type (dots and lines), and one regressor for difficulty collapsed across stimulus type, which was calculated as the negative of the absolute value of drift rate for each condition. Each of these regressors was included for correct trials only. Although the same mechanisms are expected to be active for both correct and error trials, errors could arise from variability in perception, criterion representation, and/or decision processing. Inclusion of errors could thus increase variability and decrease power, so only correct trials were included when estimating the neural correlates of the DDM parameters. However, analyses with both correct and error trials were conducted and show similar, albeit weaker, effects as those presented below.

All regressors were entered at the first level of analysis, and all but the motion parameters were convolved with a double-gamma hemodynamic response function. To account for potential slice timing differences, the temporal derivative of each regressor was included in the model. The model was estimated separately for each participant and each run. Functional images were registered to the high resolution inplane scan using the FLIRT linear registration tool of FSL (3 DOF), then the high-resolution structural image using linear registration (6 DOF), and finally the MNI_152 2 mm template using the FNIRT nonlinear registration tool of FSL. Runs were combined within participant using a fixed-effects model and combined at the group level using the FLAME 1 mixed-effects modeling of FSL. All statistical maps were corrected using cluster-based random field theory thresholding (z = 2.3; p = 0.05).

Contrasts were estimated for each regressor of interest relative to baseline. Because of the modulated regressors, results from this analysis reflect neural areas whose activation varies as a function of the corresponding DDM parameter values of stimulus size, criterion size, and difficulty. To assess motor activation, the regressor for stimulus size was also calculated relative to button press by entering the button-mapping information for each participant at the group level. Weighting the stimulus size regressor with a +1 or −1 at the group level, dependent on whether a participant's “big” responses were made with the right or left hand, allowed assessment of response activation dependent on button press but independent of stimulus size.

Results

Behavioral data

Data were analyzed separately for dot and line trials from the scanning session only. RTs and accuracy were recorded for each of 36 conditions (6 stimulus sizes × 3 criterion sizes × 2 stimulus types). Responses faster than 200 ms or slower than 1.3 s were excluded from analysis (<0.6% of the data). The data are presented in Figure 3, with dot trials on the top and line trials on the bottom. The left panels show that the criterion manipulation was effective; the probability of responding “big” shifted as a function of the criterion size. The remaining panels plot accuracy and RT as a function of the size of the stimulus relative to the size of the criterion; thus, a value of +1 on the abscissa refers to trials in which the stimulus was slightly larger than the criterion, whereas a value of −4 refers to trials in which the stimulus was much smaller than the criterion. Consistent with expectations, accuracy was lower and RTs were longer for stimuli that were close in size to the criterion. The data show a slight bias to respond “big” for lines, but little or no bias for dots.

Figure 3.

Figure 3.

Open in a new tab

Behavioral data from scanning sessions. Mean values across participants are shown with 95% confidence intervals for each condition. Dot trials are shown in the top panels and line trials in the bottom panels. The left column shows probability of “big” response across each stimulus size, with each line corresponding to a different criterion value. The shift in responses shows that the criterion manipulation effectively biased behavior. The middle and right columns show mean accuracy and median RTs, respectively, grouped as a function of the stimulus position relative to the decision criterion (note there are 8 possible distances from the criterion even though only 6 stimuli). Trials in which the stimulus and criterion were similar were more difficult, and are reflected by the longer RTs and lower accuracy near 0 in the graphs.

DDM parameters

A DDM was fit to the behavioral data for each stimulus type (dots and lines) for each participant to extract relevant parameters to use in the fMRI analysis. For our purposes, this includes estimating values for the size of the stimulus and the size of the stimulus criterion for each trial (see Materials and Methods). The best fitting parameters, averaged across participants, are shown in Table 1. The perceptual criteria estimates (rS − rL) are plotted in Figure 4 relative to the stimulus size estimates (dashed gray), and the actual criteria that were presented at the beginning of each block (solid black). The criteria estimates in solid gray show that the small and large criterion values were shifted away from the middle of perceptual space, but not as far as they should have been based on the physical properties of the presented criteria. The criteria for dot trials were roughly unbiased, but they were smaller for lines (reflected by the leftward shift from 0), accounting for the bias to respond “big” for lines observed in the behavioral data (Fig. 3).

Table 1.

DDM parameters averaged across participants

DDM parameters a Ter z/a s1 s2 s3 rS rM rL sz st η χ2
Dots 0.12 387 0.49 0.50 0.33 0.10 −0.08 0.01 0.09 0.04 162 0.11 174
Lines 0.12 357 0.53 0.52 0.33 0.11 −0.16 −0.05 0.06 0.03 164 0.12 159
Open in a new tab

For DDM parameters: a, boundary separation (response caution); Ter, nondecision time; z/a, starting point of evidence accumulation (values >0.5 indicate bias for big response); s1, stimulus size for items farthest from middle of perceptual space (1 and 6); s2, stimulus size for items medium distance from middle of perceptual space (2 and 5); s3, stimulus size for items nearest middle of perceptual space (3 and 4); rS, value for small decision criterion; rM, value for medium decision criterion; rL, value for large decision criterion; sz, across-trial variability in starting point; st, across-trial variability in nondecision time; η, across-trial variability in drift rate; χ2, best fitting χ2 value. Note: Table 1 is related to Figure 4.

The DDM parameters provide estimates for the size of the stimuli and the perceptual criteria used by the participants. These values were calculated for each participant and thus do not perfectly correspond to the expected values based on the physical properties of the criteria, but instead reflect individual differences in the internal representations of the criteria. Consequently, the DDM parameters better reflect the values used by participants to adjust their decisions than the raw criteria values presented at the beginning of each block and thus provide more sensitive measures to inform the fMRI analyses below.

fMRI data

Analyses were based on the full general linear model analysis that relied on the DDM parameters. The inclusion of the DDM parameters provided a highly constrained model that simultaneously accounted for stimulus size, criterion size, trial difficulty, and button press. The primary regressors in the model included modulated values for stimulus size (based on the si parameters for dots and lines), stimulus size relative to button press (which was mapped differently across subjects) (see Materials and Methods), criterion size (based on the ri parameters for dots and lines), RT, and trial difficulty (see Materials and Methods for the full model). Thus, stimulus value, difficulty, and RT were controlled for when estimating activation for the criteria. The peak voxel coordinates from the group analysis are shown in Table 2, and the activation maps are shown in Figure 5. All presented contrasts were for the modulated regressor relative to baseline; thus, active voxels show regions whose activation varied with the parameter of interest.

Table 2.

Group level activations from GLM analysis

Group level activations contrast Region X Y Z Voxels Max Z
Stimulus size lines R occ fusiform 8 −76 −2 704 3.83
R occ pole 16 −92 20 378 3.47
Stimulus size dots L occ pole −22 −98 −4 1188 4.64
R occ pole 14 −96 −10 766 4.83
Criterion size lines Temporal pole −34 8 −32 264 3.24
Criterion size dots Temporal fusiform −40 −16 −34 426 4.19
Difficulty
    Positive R IFG 50 10 26 720 4.04
R SPL 28 −56 38 793 3.97
dmPFC 8 22 42 718 4.11
    Negative L IPS −42 −64 34 3164 4.71
R IPS 50 −72 20 1850 4.07
PCC 8 −40 44 1368 4.51
L dlPFC −34 28 38 951 3.87
R IPS 20 −84 20 467 3.46
Motor L c gyrus −44 −20 58 9233 5.81
    Right hand R cerebellum 16 −60 −26 1999 5.69
R cerebellum 22 −70 −52 869 5.99
R c gyrus 42 −12 56 9836 5.63
    Left hand L cerebellum −18 −58 −30 1993 5.84
L cerebellum −14 −74 −50 257 3.65
Open in a new tab

Presented clusters were significant at the p < 0.05 level using cluster-based thresholding. Each contrast was relative to baseline. Voxel values are derived using a standard MNI-152 2 mm template. c, Cingulate; IFG, inferior frontal gyrus; L, left; occ, occipital; R, right.

Figure 5.

Figure 5.

Open in a new tab

Results from whole-brain GLM analyses. All results are for the modulated regressors contrasted with baseline, and thus significant voxels show activity that varies with the regressor of interest. a, Positive activation relative to the size of the presented stimulus (red for dots; green for lines). b, Positive activation relative to the size of the perceptual criterion (red for dots, green for lines). c, Positive (red) and negative (blue) activation relative to trial difficulty (uncertainty), independent of stimulus type. d, Activation relative to motor output independent of stimulus type (yellow for left responses; red for right responses). All statistics were corrected for multiple comparisons using cluster-based thresholding (z = 2.3; p = 0.05).

Perceptual stimuli

Figure 5a shows the regions that demonstrated a positive relationship to stimulus size. Because “big” and “small” responses were mapped to different hands across participants, the motor response was controlled for in these contrasts. For stimulus size, the contrasts of dot size versus baseline and line size versus baseline revealed regions of occipital cortex reflecting the difference in size among the conditions within each block of trials. These regions showed stimulus-related activation independent of which perceptual criterion was being used, and thus were reflective of the size of the perceptual input rather than abstract decision variables. Note that line activation had more activation in the right than left hemisphere because the line stimuli were presented to the left of the central fixation cross. Activation of visual cortex in relation to object size is consistent with the role of this region in processing relevant visual information (Murray et al., 2006), as visual stimuli were presented in the task.

Perceptual criteria

The size of the perceptual criterion was associated with activation in regions of the left inferior temporal cortex (Fig. 5b). Importantly, this activation was only detectable using the DDM parameters; analyses using the physical values instead of the DDM parameters resulted in weaker activation of these regions that did not reach threshold. Because the DDM parameters better capture the values driving the decision behavior (e.g., the slight bias for line trials), they provide a more sensitive measure than the physical values for probing the fMRI data.

In this experimental context, the perceptual criteria are simply internally represented objects (lines or patches of dots) of different sizes; thus, criteria-related activation in inferior temporal cortex might reflect the representation of the physical properties of the criteria. Temporal fusiform gyrus has been shown to be involved in the representation of visual objects and their physical properties (Freedman et al., 2002), containing feature codes for different images corresponding to natural objects in humans (Haxby et al., 2001) and primates (Desimone et al., 1984; Kiani et al., 2007) (for a comparison between humans and primates, see Kriegeskorte et al., 2008). Consistent with this interpretation, previous work has demonstrated that activation in temporal cortex might serve as an input to downstream decision regions [e.g., prefrontal cortex (Heekeren et al., 2004, 2006)]. This suggests that the criterion is represented in stimulus space and used as an input to the decision comparison. Related work found criteria-related activation of occipitotemporal regions thought to be involved in representing visual form [i.e., stimulus properties (Li et al., 2009)], supporting this interpretation. Importantly, unlike their results, there was no significant criteria-related activation in the frontoparietal decision network (e.g., dlPFC) outside of those related to trial difficulty (see below).

Difficulty

Neural systems that are associated with the calculation and formation of the decision should show activation that is affected by the values of both the stimulus and the perceptual criterion. We identified these systems using the difficulty (uncertainty) measure derived by the DDM analysis, which was based on the values of the stimulus and the criterion. We calculated the absolute value of drift (calculated as |si − ri|) for every trial and reversed the order so that higher values reflected greater difficulty. For example, comparing the largest line to the smallest criterion results in the same low difficulty as comparing the smallest line to the largest criterion. Because difficulty is based on the value of the stimulus after it has been compared with the criterion, subsequent activation is reflective of later decision variables that are downstream from the stimulus and criterion activation. Furthermore, the difficulty regressor was calculated for all correct trials, independent of stimulus type, and thus activation is not specific to one type of perceptual input. Regions showing a positive relationship with difficulty are shown in red, and regions showing a negative relationship with difficulty are shown in blue (Fig. 5c). Note that this difficulty effect is separate from an overall effect of response times, which were modeled using an additional trial-based RT regressor.

The difficulty measure can be thought to reflect two important aspects of the decision process that could be performed by distinct neural systems: the calculation of decision evidence from the stimulus/criterion comparison and the accumulation of the corresponding decision evidence to reach the decision. Importantly, these two decision components should have opposite relationship with difficulty. Easy trials are associated with strong decision evidence (from an easy stimulus/criterion comparison) and thus should show greater activation in regions involved in the comparison process. However, easy trials are also associated with shorter duration of evidence accumulation, and thus they should show weaker activation in regions involved in the accumulation of decision evidence (from shorter response duration). In this framework, neural systems involved in comparing the stimulus to the criterion should show a negative relationship with difficulty, whereas systems involved in the accumulation of evidence should show a positive relationship with difficulty.

Consistent with previous research, the difficulty contrast showed activation in a distributed network of frontoparietal regions. A negative relationship with difficulty was found for left dlPFC, posterior cingulate cortex (PCC), and bilateral intraparietal sulcus (IPS). Left dlPFC has been shown in a number of studies to show activation consistent with the comparison process that determines the decision variable. Our results add to a growing set of findings implicating the involvement of dlPFC in the formation of the decision variable across a range of perceptual inputs (for review, see Heekeren et al., 2008). Previous work has also shown IPS activation relative to trial difficulty; however, the relationship was positive, the opposite direction as the present findings (Hare et al., 2011). They found that IPS showed a time course similar to that predicted by a DDM in value-based decisions (Hare et al., 2011), supporting its role in the formation of the decision. However, they did not report any activation that showed a negative relationship with difficulty for direct comparison. The IPS activation in the present study is more lateral and dorsal than in the study by Hare et al. (2011), and we did find nearby parietal activation that showed the same positive relationship with difficulty (see below). Thus, our findings suggest that nearby but distinct regions of parietal cortex might have opposite roles related to task difficulty and/or processing time.

In contrast to the negative relationship presented above, a positive relationship with difficulty was shown for right superior parietal lobule (SPL), dorsomedial PFC (dmPFC), and right PFC. dmPFC has previously been shown to be associated with trial uncertainty in a task similar to the present one (Grinband et al., 2006). The region of superior parietal cortex is similar to that found in the study by Hare et al. (2011) and has also been found in other studies to correlate with decision uncertainty and conflict (Wendelken et al., 2009) and is implicated in the accumulation of decision evidence (Heinen et al., 2006; Vickery and Jiang, 2009). This parietal activation was much stronger, widespread, and bilateral when RTs were not controlled for, potentially accounting for the discrepancy between these findings and the study by Hare et al. (2011) in which response times were not controlled.

Motor output

Motor activation was assessed by calculating the effect of stimulus size relative to the motor response (left vs right hand) independent of the type of stimulus (dots or lines). Because button mapping was counterbalanced across subjects, motor activation was dissociated from stimulus size (see Materials and Methods). As expected, activation was found in motor-related regions including bilateral putamen, thalamus, and motor cortex (Fig. 5d). As with the difficulty measure, this motor activation was independent of stimulus type and thus reflected more abstract motor commands.

Discussion

The ability to flexibly adjust behavior using internal criteria with different values is critical for successful navigation of the environment. Our results provide novel insight into how the brain represents such criteria that can be rapidly adjusted to account for task demands. Activity in left inferior temporal cortex corresponded to the size of the visual object that served as the midpoint, or criterion, for the perceptual decision. This region of cortex has been shown to hold representations of objects and their physical properties (Kriegeskorte et al., 2008). Our findings are consistent with this interpretation and demonstrate how such representations can be used to adjust or bias decision making; the value of this internally represented criterion was used by participants to change the mapping between stimulus and response, meaning the same item could be flexibly mapped to the small or big response. This activation was dissociated from activity in visual cortex corresponding to the size of the perceptual stimulus, from activity in a frontoparietal network of regions involved in general decision processes, and from activity in motor regions involved in the general motor response.

Although stimulus criteria are involved across a range of decisions, our results suggest that they are represented in the brain in a stimulus-specific manner. Rather than abstract, task-general biasing of the decision evidence, the perceptual criteria were reflected by activity related to their physical properties. This finding suggests that the perceptual criterion can be thought of as an internal representation of the perceptual object against which the stimulus is compared. More generally, the loci of stimulus criteria activation would be expected to differ across different types of decisions; for example, memory criteria might be reflected in regions of the medial temporal lobe. Consistent with this, recent work using glass pattern stimuli showed criterion-related activation in regions of occipitotemporal cortex that are known to be involved in visual form processing, which is necessary for discriminating the glass pattern stimuli (Li et al., 2007, 2009, 2012).

Our results also imply that the stimulus criterion is essentially another input to the decision process that recruits the same decision network as other simple decisions. For instance, left dlPFC has been shown across a range of studies to be involved in the comparison process that underlies the formation of decision evidence. One study had participants categorize degraded images as houses or faces and found activation in dlPFC that was related to the difference between house and face-related activity represented in temporal cortex (Heekeren et al., 2004). Thus, dlPFC was involved in the comparison of the house and face information contained in the visual stimulus (Hanes and Schall, 1996; Kim and Shadlen, 1999; Heekeren et al., 2004, 2008; Gold and Shadlen, 2007; Thielscher and Pessoa, 2007; Noppeney et al., 2010; Philiastides et al., 2011), and similar effects have been shown independent of modality (Heekeren et al., 2006). Recent work with repetitive transcranial magnetic stimulation has provided evidence that dlPFC plays a causal role in calculating decision evidence (Philiastides et al., 2011). Our results are consistent with this interpretation as left dlPFC showed activation related to the difference between the external stimulus and the internal criterion, with greater activation of dlPFC for trials with an easier comparison.

The neural systems identified in relation to the difficulty of a trial were highly consistent with previous work in this area. Superior parietal cortex and bilateral dmPFC have all been implicated in the accumulation of decision evidence under uncertainty (Binder et al., 2004; Vickery and Jiang, 2009; Hare et al., 2011), and our results support this interpretation as these regions showed activation based on downstream decision variables rather than the stimulus information itself. Recent work probed the flow of information among these decision regions using dynamic causal modeling and found that dmPFC and bilateral IPS showed activation consistent with transforming the stimulus information into the motor response (Hare et al., 2011). In particular, the value of the two choice options in their study, represented in vmPFC, modulated activity in dmPFC and IPS at the time of the decision. Furthermore, both dmPFC and IPS activation was predictive of the eventual response. Our findings are largely consistent with these results, although we did not observe activation in vmPFC, possibly because the decisions in the present study were not explicitly value based (i.e., no feedback during scanning sessions). We also found distinct regions of parietal cortex that correlated negatively or positively with difficulty. Right SPL showed a positive relationship with difficulty, and this effect was stronger, spatially larger, and bilateral when RTs were not controlled. Thus, our finding of activation in this region is largely consistent with previous work (Hare et al., 2011). In contrast, more dorsal and lateral regions of parietal cortex showed a negative relationship with difficulty, potentially reflecting deactivation of the default mode network. Future work is needed to investigate the relationship between difficulty and these neural systems.

One region we expected to see activation relative to difficulty was right anterior insula (aI). Right aI has been shown to present a time course similar to that predicted by a diffusion process for both button press and eye movement response modalities, suggesting its role in accumulating evidence and mapping the decision to the physical response (Ho et al., 2009; Liu and Pleskac, 2011). Activation in aI has also been shown to correlate with RTs and scale with the amount of differential sensory evidence in the stimulus [which is similar to the difficulty measure presented herein (Thielscher and Pessoa, 2007)]. These findings imply that right aI might serve a role in accumulating the decision variable to guide different motor responses, although aI is also active for a range of different cognitive functions (Kurth et al., 2010). Right aI did show a weak correlation with the difficulty measure in the present study, consistent with the results above. However, when RT was controlled for the aI activation did not pass the statistical threshold. As aI is suggested to be one of the last accumulation regions involved before the response (Ho et al., 2009), it is likely more closely linked with the RT for each trial than the other regions described above.

In addition to disentangling the different components of the decision process, our design also allowed us to probe the flow of stimulus information as it was translated into decision evidence and eventually a response. We were able to dissociate these stages to identify early stimulus and criterion activation, decision-related activation that results from comparing the stimulus to the criterion and accumulating the evidence, and finally motor activation that results from mapping the evidence to a response. The inputs to the decision process consist of the stimulus value and the criterion value, and thus should be sensitive to the type of stimulus being presented but insensitive to the eventual motor response. This corresponds to the stimulus signal in occipital cortex and the criterion signal in inferior temporal cortex. These inputs are fed into the decision process, which involves comparing the inputs and accumulating the corresponding evidence. The neural systems reflecting such decision processes, including regions of the frontoparietal network described above, are independent of the stimulus type and independent of the motor response. Finally, the output of the decision process feeds the motor response, reflected by activation of motor regions that was independent of the stimulus type, but dependent on the button press. These results provide insight into how the brain transforms external information to guide behavior, and are consistent with EEG findings that have decomposed trials into perceptual-, decision-, and motor-related components (Ratcliff et al., 2009).

Internal criteria are a critical component of many decisions. The results presented here demonstrate how such criteria are represented in the brain for simple perceptual decisions. The use of specific experimental manipulations, fMRI, and DDMs to identify the value of such criteria presents a promising technique for separating stimulus processing from decision processing, allowing for a deeper exploration of how these values are used by the brain to formulate the decision evidence that drives the decision process. This approach provides insights into psychological processing that would not be possible based on behavioral evidence alone. Our results suggest that such criteria are reflected by stimulus-specific representations of the item under consideration. Thus, rather than using an general biasing signal in neural systems involved in accumulating decision evidence, an internally represented object can be used to compare against the item under consideration. This process allows for great behavioral flexibility, as decisions can be made based on the comparison of both external and internal information.

Footnotes

This work was supported by The James S. McDonnell Foundation and Texas Emerging Technology Fund.

References

  1. Binder JR, Liebenthal E, Possing ET, Medler DA, Ward BD. Neural correlates of sensory and decision processes in auditory object identification. Nat Neurosci. 2004;7:295–301. doi: 10.1038/nn1198. [DOI] [PubMed] [Google Scholar]
  2. Brainard DH. The Psychophysics Toolbox. Spat Vis. 1997;10:433–436. [PubMed] [Google Scholar]
  3. Desimone R, Albright TD, Gross CG, Bruce C. Stimulus-selective properties of inferior temporal neurons in the macaque. J Neurosci. 1984;4:2051–2062. doi: 10.1523/JNEUROSCI.04-08-02051.1984. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Freedman DJ, Riesenhuber M, Poggio T, Miller EK. Visual categorization and the primate prefrontal cortex: neurophysiology and behavior. J Neurophysiol. 2002;88:929–941. doi: 10.1152/jn.2002.88.2.929. [DOI] [PubMed] [Google Scholar]
  5. Gold JI, Shadlen MN. The neural basis of decision making. Annu Rev Neurosci. 2007;30:535–574. doi: 10.1146/annurev.neuro.29.051605.113038. [DOI] [PubMed] [Google Scholar]
  6. Green DM, Swets JA. Signal detection theory and psychophysics. New York: Wiley; 1966. [Google Scholar]
  7. Grinband J, Hirsch J, Ferrera VP. A neural representation of categorization uncertainty in the human brain. Neuron. 2006;49:757–763. doi: 10.1016/j.neuron.2006.01.032. [DOI] [PubMed] [Google Scholar]
  8. Hanes DP, Schall JD. Neural control of voluntary movement intiation. Science. 1996;274:427–430. doi: 10.1126/science.274.5286.427. [DOI] [PubMed] [Google Scholar]
  9. Hare TA, Schultz W, Camerer CF, O'Doherty JP, Rangel A. Transformation of stimulus value signals into motor commands during simple choice. Proc Natl Acad Sci U S A. 2011;108:18120–18125. doi: 10.1073/pnas.1109322108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 2001;293:2405–2407. doi: 10.1126/science.1063736. [DOI] [PubMed] [Google Scholar]
  11. Heekeren HR, Marrett S, Bandettini PA, Ungerleider LG. A general mechanism for perceptual decision-making in the human brain. Nature. 2004;431:859–862. doi: 10.1038/nature02966. [DOI] [PubMed] [Google Scholar]
  12. Heekeren HR, Marrett S, Ruff DA, Bandettini PA, Ungerleider LG. Involvement of human left dorsolateral prefrontal cortex in perceptual decision making is independent of response modality. Proc Natl Acad Sci U S A. 2006;103:10023–10028. doi: 10.1073/pnas.0603949103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Heekeren HR, Marrett S, Ungerleider LG. The neural systems that mediate human perceptual decision making. Nat Rev Neurosci. 2008;9:467–479. doi: 10.1038/nrn2374. [DOI] [PubMed] [Google Scholar]
  14. Heinen SJ, Rowland J, Lee BT, Wade AR. An oculomotor decision process revealed by functional magnetic resonance imaging. J Neurosci. 2006;26:13515–13522. doi: 10.1523/JNEUROSCI.4243-06.2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Ho TC, Brown S, Serences JT. Domain general mechanisms of perceptual decision making in human cortex. J Neurosci. 2009;29:8675–8687. doi: 10.1523/JNEUROSCI.5984-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Kiani R, Esteky H, Mirpour K, Tanaka K. Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. J Neurophysiol. 2007;97:4296–4309. doi: 10.1152/jn.00024.2007. [DOI] [PubMed] [Google Scholar]
  17. Kim JN, Shadlen MN. Neural correlates of a decision in the dorsolateral prefrontal cortex of the macaque. Nat Neurosci. 1999;2:176–185. doi: 10.1038/5739. [DOI] [PubMed] [Google Scholar]
  18. Kriegeskorte N, Mur M, Bandettini P. Representational similarity analysis—connecting the branches of systems neuroscience. Front Syst Neurosci. 2008;2:4. doi: 10.3389/neuro.06.004.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Kurth F, Zilles K, Fox PT, Laird AR, Eickhoff SB. A link between the systems: functional differentiation and integration within the human insula revealed by meta-analysis. Brain Struct Funct. 2010;214:519–534. doi: 10.1007/s00429-010-0255-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Li S, Ostwald D, Giese M, Kourtzi Z. Flexible coding for categorical decisions in the human brain. J Neurosci. 2007;27:12321–12330. doi: 10.1523/JNEUROSCI.3795-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Li S, Mayhew SD, Kourtzi Z. Learning shapes the representation of behavioral choice in the human brain. Neuron. 2009;62:441–452. doi: 10.1016/j.neuron.2009.03.016. [DOI] [PubMed] [Google Scholar]
  22. Li S, Mayhew SD, Kourtzi Z. Learning shapes spatiotemporal brain patterns for flexible categorical decisions. Cereb Cortex. 2012;22:2322–2335. doi: 10.1093/cercor/bhr309. [DOI] [PubMed] [Google Scholar]
  23. Liu T, Pleskac TJ. Neural correlates of evidence accumulation in a perceptual decision task. J Neurophysiol. 2011;106:2383–2398. doi: 10.1152/jn.00413.2011. [DOI] [PubMed] [Google Scholar]
  24. Murray SO, Boyaci H, Kersten D. The representation of perceived angular size in human primary visual cortex. Nat Neurosci. 2006;9:429–434. doi: 10.1038/nn1641. [DOI] [PubMed] [Google Scholar]
  25. Noppeney U, Ostwald D, Werner S. Perceptual decisions formed by accumulation of audiovisual evidence in prefrontal cortex. J Neurosci. 2010;30:7434–7446. doi: 10.1523/JNEUROSCI.0455-10.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Philiastides MG, Auksztulewicz R, Heekeren HR, Blankenburg F. Causal role of dorsolateral prefrontal cortex in human perceptual decision making. Curr Biol. 2011;21:980–983. doi: 10.1016/j.cub.2011.04.034. [DOI] [PubMed] [Google Scholar]
  27. Ratcliff R. A theory of memory retrieval. Psychol Rev. 1978;85:59–108. [Google Scholar]
  28. Ratcliff R, McKoon G. The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput. 2008;20:873–922. doi: 10.1162/neco.2008.12-06-420. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Ratcliff R, Smith PL. A comparison of sequential sampling models for two-choice reaction time. Psychol Rev. 2004;111:333–367. doi: 10.1037/0033-295X.111.2.333. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Ratcliff R, Tuerlinckx F. Estimation of the parameters of the diffusion model: approaches to dealing with contaminant reaction times and parameter variability. Psychon Bull Rev. 2002;9:438–481. doi: 10.3758/bf03196302. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Ratcliff R, Philiastides MG, Sajda P. Quality of evidence for perceptual decision making is indexed by trial-to-trial variability of the EEG. Proc Natl Acad Sci U S A. 2009;106:6539–6544. doi: 10.1073/pnas.0812589106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Smith SM, Jenkinson M, Woolrich MW, Beckmann CF, Behrens TEJ, Johansen-Berg H, Bannister PR, De Luca M, Drobnjak I, Flitney DE, Niazy R, Saunders J, Vickers J, Zhang Y, De Stefano N, Brady JM, Matthews PM. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage. 2004;23(Suppl 1):S208–S219. doi: 10.1016/j.neuroimage.2004.07.051. [DOI] [PubMed] [Google Scholar]
  33. Thielscher A, Pessoa L. Neural correlates of perceptual choice and decision making during fear-disgust discrimination. J Neurosci. 2007;27:2908–2917. doi: 10.1523/JNEUROSCI.3024-06.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Vickery TJ, Jiang YV. Inferior parietal lobule supports decision making under uncertainty in humans. Cereb Cortex. 2009;19:916–925. doi: 10.1093/cercor/bhn140. [DOI] [PubMed] [Google Scholar]
  35. Wendelken C, Ditterich J, Bunge SA, Carter CS. Stimulus and response conflict processing during perceptual decision making. Cogn Affect Behav Neurosci. 2009;9:434–447. doi: 10.3758/CABN.9.4.434. [DOI] [PubMed] [Google Scholar]
  36. White CN, Ratcliff R, Vasey MW, McKoon G. Using diffusion models to understand clinical disorders. J Math Psychol. 2010;54:39–52. doi: 10.1016/j.jmp.2010.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES