Transformation of stimulus value signals into motor commands during simple choice

Abstract

Decision-making can be broken down into several component processes: assigning values to stimuli under consideration, selecting an option by comparing those values, and initiating motor responses to obtain the reward. Although much is known about the neural encoding of stimulus values and motor commands, little is known about the mechanisms through which stimulus values are compared, and the resulting decision is transmitted to motor systems. We investigated this process using human fMRI in a task where choices were indicated using the left or right hand. We found evidence consistent with the hypothesis that value signals are computed in the ventral medial prefrontal cortex, they are passed to regions of dorsomedial prefrontal cortex and intraparietal sulcus, implementing a comparison process, and the output of the comparator regions modulates activity in motor cortex to implement the choice. These results describe the network through which stimulus values are transformed into actions during a simple choice task.

Implementing a choice necessarily requires taking an action. Consider the problem faced by an individual that has to choose between two stimuli, one placed on the left and obtained through a left hand movement and one placed on the right and obtained through a right hand movement. Theoretical models and a growing body of evidence (1–3) suggest that the brain solves this task by assigning values to the two stimuli, comparing them, and then activating the necessary motor response to implement the choice. It follows that, to solve the choice problem, the brain needs to transform stimulus value signals into motor commands.

Over the last decade, we have learned a considerable amount about the encoding of stimulus value signals at the time of choice. Functional MRI (fMRI) studies have found stimulus value signals in ventromedial prefrontal cortex (vmPFC) at the time of decision-making for primary rewards (4–8), monetary gains and losses (9–14), delayed rewards (15), consumer goods (16, 17), and abstract social rewards (18, 19). Related studies using electrophysiological recordings in nonhuman primates have shown that stimulus values are encoded in the firing rates of individual neurons in the orbitofrontal cortex (20–23). Importantly, these studies have shown that these areas encode stimulus value signals that are independent of the actual choice made, suggesting that they are an input to the choice process, which is outlined in the framework above.

The values assigned to each option must be compared to select the best course of action. Exactly how this comparison occurs in the brain is an area of active research. One important clue comes from computational models of the choice process that have shown that the drift diffusion model (DDM) (24, 25) and some of its variants (26–29) fit the accuracy and reaction time data of simple choice tasks remarkably well. Furthermore, activity resembling the output of the DDM has been found in the lateral intraparietal cortex during perceptual decision-making tasks in nonhuman primates (30, 31) and the dorsal medial prefrontal cortex (dmPFC) during action selection tasks in humans (32).

Here, we seek to identify the network involved in transforming stimulus values into motor commands using fMRI. Our strategy relies on the fact that an area involved in the comparison process and linking value computation to action implementation should satisfy the following three properties. First, its blood oxygen level-dependent (BOLD) signal should reflect the predictions for aggregate activity derived from neural implementations of the DDM. This property is important, because the DDM has been shown to fit the psychometric data in this class of tasks extremely well (24, 25). Second, the region should exhibit increased effective connectivity from areas such as vmPFC that encode stimulus values at the time of choice. This property is important, because the comparator needs to receive the value signals to be able to make choices. Third, the region should exhibit choice-dependent effective connectivity with motor cortex in a way that promotes the observed motor responses: it should enhance activity in the left motor cortex during right actions and activity in the right motor cortex during left actions. Based on the evidence described above and the well-characterized connectivity between the dmPFC and supplemental motor areas (33–35), we hypothesized that dmPFC and intraparietal sulcus (IPS) would satisfy the three properties and thus, provide the link between vmPFC and motor cortex during the transformation of stimulus values into motor commands.

Previous studies have looked at individual aspects of the value transformation network but have never tested for all of the functions necessary to move from valuation to action. A recent fMRI study of human decision-making found that IPS activity was consistent with some of the properties that one would expect from a comparator process, including increased connectivity with vmPFC at the time of choice and greater activity for more difficult choices (36), and a previous study by our own group suggested that activity in dmPFC might reflect, in part, the computations of a comparator process (7). Another study (37) found that activity in vmPFC was stronger in easier trials than more difficult decision trials, which is consistent with the hypothesis that the vmPFC might be involved in the computation of relative stimulus values. Note that, although these papers are important precursors on which we build and their results are consistent with subsets of the results obtained here, none of them address the fundamental goal of fully characterizing the network involved in how value signals are transmitted to putative comparison regions and ultimately, modulate activity in motor cortex to implement the choice. In particular, none of them has examined the connectivity of the entire network to test the predicted intratrial-, choice-, time-, and direction-specific changes in coupling between each region of the network. Here, we test these predictions using dynamic causal modeling (DCM) to examine the modulation of specific connections at a neuronal timescale (inferred from a hemodynamic deconvolution of the BOLD signal) during the periods of stimulus valuation and action preparation. Consistent with our hypotheses, we found that signaling from vmPFC to comparator regions increases at the time of choice, and subsequently, signaling from comparator regions to motor cortex increases during action preparation in a choice-dependent manner.

Results

We tested these hypotheses using a paradigm in which, on every trial, thirsty human subjects were shown pairs of symbols representing various amounts of different liquid rewards on the left and right sides of the screen, and later, they pressed either the left or right thumbs to indicate their choice (Fig. 1A). After another brief delay, the chosen liquid was delivered to the subjects inside the scanner on each trial. The task is closely related to the tasks used in works by Padoa-Schioppa and Assad to investigate the coding of stimulus value in orbitofrontal cortex with nonhuman primates (20, 21, 38).

Fig. 1.

Experimental design and behavior. (A) Subjects were presented with a choice screen offering two different amounts of two different liquids. Colored shapes represented the liquid identity. The number of shapes indicated the amount of liquid being offered. Subjects were instructed to make their choice while the shapes were on the screen, but they could only indicate their choice with a button press (left or right thumb) when the response prompt appeared after a variable delay period. The chosen stimulus was delivered after another variable delay period. (B) Percentage of left choices as a function of value of left minus value of right stimulus. Error bars represent the SEM across subjects.

Behavioral Results.

We estimated the value of each amount of every different juice from the behavioral data using the procedure described in Materials and Methods. The psychometric curve in Fig. 1B shows that these value estimates provide an accurate account of the choice behavior. A mixed effects logistic regression showed that subjects were highly responsive to the relative value of the two juices (t₁₈ = 8.05, P < 0.001).

Estimation of the Neural DDM.

We estimated a simple neural implementation of the DDM. The model is important for our analyses, because it makes predictions about the level of aggregate activity and thus, about the BOLD responses that should be observed in an area involved in the comparison of stimulus values with action selection. As illustrated in Fig. 2A, the standard DDM assumes that (i) a relative value signal measuring the estimated relative value in favor of the left stimulus is computed dynamically through a Gaussian Markovian process, with independent and identically distributed noise and a mean slope of integration proportional to the underlying true value of the left minus right items; (ii) the relative value signal starts at zero; and (iii) a choice is made when either the signal crosses the upper barrier (so that the left option is selected) or the lower barrier (so that the right option is selected). At this level of computational abstraction, the DDM model does not make testable predictions about the level of activity associated with the comparator process that can be used to identify an area involved in these computations using fMRI. To achieve this link, we specified the simplest possible plausible neural implementation of the DDM that matches well with its behavioral predictions (29) (Fig. 2B and Materials and Methods).

Fig. 2.

Theoretical results. (A) Sketch of the basic DDM model of binary choice. (B) Summary of the neural implementation of the DDM. Red denotes activity encoding the relative value of left. Every instant t, this pool of neurons receives input (V_L − V_R) + error_L(t), which is integrated to the previous level of activity a_L(t) without leakage. The pool also receives a constant level of inhibition proportional to the level of activity in the right pool of neurons [denoted in blue; a_R(t)]. The dynamics for the system encoding the relative value of right are analogous. (C) Comparison of the psychometric choice curve for the group (blue) and the one generated by the model using the best-fitting parameters (black). (D) Total predicted activity (arbitrary units) in the comparator units (given by the sum of the instantaneous levels of activity in both pools of neurons up to the response time) as a function of the relative value of the two items and the correctness of the choice. Green, correct; red, incorrect. Error bars denote SE.

An advantage of this model, which we refer to as neural DDM (nDDM), is that it is fully characterized by three free parameters (integration slope, inhibition strength, and integration noise). We estimated the values of these parameters that maximized the match with the group psychometric choice function shown in Fig. 2C. We used the model and best-fitting parameters to compute the expected total level of activity in the comparator region for each trial as a function of the relative values of the left and right items as well as whether the best item was chosen. Fig. 2D depicts the average predicted level of activity that is used below to identify regions associated with value comparison and action selection. This variable measures the total level of predicted activity generated by both pools of neurons, which is the relevant signal for identifying the neural comparator using BOLD fMRI as long as the two pools of neurons are spatially intermingled. Previous studies have used difficulty or reaction times as a marker for putative comparator regions (36, 39–41). Fig. 2D and the analyses discussed below show that, although this assumption is a good approximation to the predictions of the nDDM, it leaves out useful information. This void can be seen from the fact that the predicted activity levels have different curvature and average levels in correct and error trials. As described below and in SI Results, these differences can be exploited to compare the relative fit of the nDDM with difficulty-based regressors.

Stimulus Value Representation.

We estimated a parametric general linear model (GLM) of BOLD activity that allowed us to identify areas in which activity was correlated with various signals of interest. Using this model, we found that activity in vmPFC correlated with the sum of the values during the initial screen depicting the two options (P < 0.05, small-volume corrected) (Fig. 3A and Table S1). Post hoc analyses of this area showed that activity did not differ by stimulus identity [one-way ANOVA for liquid type: F_(3,72) = 0.90, not significant (n.s.)], location (paired t test between left and right values: t₍₁₈₎ = −1.40, n.s.), or choice status (paired t test between chosen and nonchosen value: t₍₁₈₎ = 0.56, n.s.).

Fig. 3.

Valuation and potential comparator regions. (A) Activity in vmPFC correlated with the sum of the stimulus values shown on each trial (P < 0.05, small-volume corrected). (B and C) Activity in bilateral dmPFC (B) and bilateral IPS (C) correlated with the predicted levels of activity generated by the nDDM model at P < 0.05, whole-brain corrected.

Motor Responses.

Next, we used the same GLM to identify activity associated with implementing specific motor responses. Left motor cortex (lMC) and right cerebellum were more active for responses with the right than the left thumb (P < 0.05, corrected) (Fig. S1A and Table S2). Conversely, right motor cortex (rMC) and left cerebellum were more active for responses with the left than right thumb (P < 0.05, corrected) (Fig. S1B and Table S2). Post hoc tests showed that lMC and rMC did not meet the criteria for encoding of action values (details in SI Results).

Comparator Regions Linking Valuation to Action: First Property.

As described in the Introduction, any comparator region linking valuation to action in our experiment should satisfy three key properties. Here, we implement the test for the first property, which requires that BOLD responses at the time of stimulus presentation, when the choice is being made, correlate with the predicted total activity levels estimated from the nDDM, conditional on the relative value (V^best − V^worse) and the quality of choice (best chosen = 0, 1) in each trial (Fig. 2D). We found that activity in the dmPFC, left dorsolateral prefrontal cortex (dlPFC), and bilateral IPS correlated with this regressor during the period from stimulus onset through execution of the motor response (P < 0.05, corrected) (Fig. 3 B and C, Fig. S2, and Table S3).

Given the high correlation between the output of our nDDM and a measure of choice difficulty given by |value left − value right| (r = −0.930) and previous reports of correlations between dmPFC, dlPFC, and IPS with choice difficulty (42, 43), we performed post hoc Bayesian model selection on each region of interest to test whether the predictions from the nDDM or difficulty best explained activity in these regions. Note that, although there is a high correlation between the two measures, the nDDM predicts differences in the signal between correct and error trials that the difficulty measure does not predict. These differences can be exploited to test for the relative ability of each model to fit aggregate neural activity in areas of interest (additional details in SI Results). We used the exceedance probability (EP; the probability that a given model is more likely than any other model in the comparison set given the group data) as our metric for model comparison (44). The EP of the nDDM model was greater than the difficulty model in all four regions (dmPFC EP = 0.99, dlPFC EP = 0.58, lIPS EP = 0.98, and rIPS EP = 0.94), indicating that the nDDM provided a better fit to activity in these areas than the difficulty measure, especially in dmPFC and IPS.

Comparator Regions Linking Valuation to Action: Second and Third Properties.

We next used DCM to investigate if the dmPFC, dlPFC, and IPS also exhibited the two key connectivity properties that an area involved in transforming stimulus values into motor commands should satisfy: (i) increased input at the time of choice from the region of vmPFC involved in computing stimulus values, and (ii) choice-dependent effective connectivity with motor cortex in a way that promotes the observed motor responses.

First, we identified the most likely model using a Bayesian model selection process (Materials and Methods, SI Materials and Methods, and Tables S4–S10). The best model, depicted in Fig. 4, has reciprocal connections between vmPFC, dmPFC, dlPFC, and bilateral IPS and unidirectional connections from dmPFC, dlPFC, and bilateral IPS to lMC and rMC.

Fig. 4.

Tests of functional coupling using DCM. (A) Diagram of the pattern of fixed connections between the seven regions in the most likely DCM model. Note that connections between dlPFC and IPS have been omitted for clarity but can be seen in Tables S1–S3. (B) Changes in connectivity during the stimulus valuation period. (C) Changes in connectivity during the period between stimulus onset and response in trials where subjects chose the left option. (D) Changes in connectivity during the period between stimulus onset and response in trials where subjects chose the right option. Connections in red indicate significant positive coupling coefficients, whereas blue indicates significant negative coefficients. Lines in gray indicate connections with posterior probability less than 90%.

Second, after determining the most likely model, we used Bayesian parameter averaging to estimate the group posterior probabilities of each parameter of the best-fitting model. There are two types of parameters of interest in the model: fixed connection and coupling modulation parameters.

The fixed connection parameters measure the coupling between two areas during the rest periods of the task. There was a high posterior probability (P > 90%) of fixed positive connections between most regions with the exception of vmPFC to dlPFC, dmPFC to lIPS, lIPS and rIPS to rMC, rIPS to lIPS, and rMC to lMC (Table S5). There was also significant negative coupling from lMC to rMC.

The coupling modulation parameters measure how the interactions between regions change during specific phases of each decision trial. These modulation parameters represent our primary measure of interest in the DCM analysis, because they provide a direct test of the two properties of interest of comparator regions that might link valuation to action.

The first set of modulation parameters measures coupling changes during stimulus presentation, which coincides with the initial valuation and comparison of the two stimuli. We found that the coupling from vmPFC to dmPFC [P(|coupling Δ| > 0) = 0.93] and left IPS [P(|coupling Δ| > 0) = 0.97] and right IPS [P(|coupling Δ| > 0) = 0.95] increased during this phase, which provides evidence in favor of the second criterion for these three areas (Fig. 4B and Table S6). Note that this increase in signaling from vmPFC to dmPFC and IPS was independent of the values of the options on each trial. This finding is to be expected, because the comparison process is necessary at all levels of value.

The second set of modulation parameters measures coupling changes during the period from stimulus onset to response for left and right choices separately. These parameters allowed us to estimate the posterior probabilities that the functional coupling strength depended on the identity of the chosen action, and thus, they help to test if IPS and dmPFC satisfy the third desired property. As described in Fig. 4 C and D and Tables S7 and S8, the functional coupling from dmPFC, dlPFC, and IPS to lMC and rMC was dependent on the choice. When the left action was chosen, there was positive modulation of the connections to rMC from dmPFC [P(|coupling Δ| > 0) = 0.99], dlPFC [P(|coupling Δ| > 0) = 0.96], lIPS [P(|coupling Δ| > 0) = 0.92], and rIPS [P(|coupling Δ| > 0) = 0.97]. However, there was negative modulation of the connection to lMC from dmPFC [P(|coupling Δ| > 0) = 0.97], lIPS [P(|coupling Δ| > 0) = 0.97], and rIPS [P(|coupling Δ| > 0) = 0.93]. In contrast, during the period between stimulus presentation and right option selection, there was positive modulation of the coupling to lMC from dmPFC [P(|coupling Δ| > 0) = 0.99], dlPFC [P(|coupling Δ| > 0) = 98%], lIPS, and rIPS [P(|coupling Δ| > 0) = 0.99, whereas there was negative modulation of the connection to rMC from lIPS [P(|coupling Δ| > 0) = 0.96] and rIPS [P(|coupling Δ| > 0) = 0.94]. There was no significant modulation in coupling between lMC and rMC at the time of button press (Table S9). Together, these last findings provide evidence that dmPFC and IPS satisfy the third hypothesized property of a comparator region that links valuation to action.

Discussion

We have used a strategy to characterize the network involved in the transformation of stimulus values into a motor response during simple choice using fMRI. We found that activity in dmPFC and bilateral IPS exhibited three key properties that areas engaged in value comparison and linking choices to actions should satisfy. First, activity in these areas correlated with the predictions of a neural model of choice that approximates the computations of the DDM. This property is an important marker of comparator areas, because the DDM has been shown to fit the psychometric data from binary choice tasks (including ours) extremely well (24, 25). Second, our DCM analysis showed that activity in both areas exhibited increased functional coupling at the time of decision with the region of vmPFC that encoded both stimulus values, regardless of the choice that was eventually made. This increased coupling at the time of decision is expected in areas that use these value signals as inputs to the actual comparison process. Third, dmPFC and IPS exhibited choice-dependent coupling with motor cortex in a way that promoted the observed motor responses: they increased the connectivity with left motor cortex during right actions and with right motor cortex during left actions.

Our results provide a link between the value-based and perceptual decision literatures. Single-unit studies in nonhuman primates have found activity in IPS during saccadic dot motion tasks that resembles the output of a DDM comparator process (30, 31, 45, 46). Similar results have been obtained in human fMRI studies of perceptual decision-making (40, 41). Connections between vmPFC and parietal regions are likely to be polysynaptic. Therefore, future studies should seek to identify the intermediate nodes connecting vmPFC and parietal cortex in decision-making. One intriguing possibility is that the dmPFC and IPS might participate in the decision-making process by computing decision variables in the action and spatial domains, respectively. Unfortunately, in many paradigms, including our paradigm, the two types of decision variables are highly correlated and thus, are difficult to disentangle.

The dmPFC is a natural area to implement a comparison process in simple economic choice, because it is heavily interconnected with both supplementary motor areas and areas of vmPFC thought to be involved in valuation (33–35). Furthermore, neurons in dmPFC have been shown to reflect several different decision variables, making this region ideally qualified to compare different options and promote the best course of action (22).

Previous fMRI studies have shown that the dmPFC, dlPFC, and parietal cortex are more active in more difficult value-based decisions (42, 43). Given the high correlation between difficulty and the levels of activity predicted by the nDDM model, our results are consistent with these findings. However, we found that the nDDM model provides a better fit to activity in dmPFC and IPS, which is consistent with previous studies that have directly compared measures of difficulty and value differences in dmPFC (7).

In summary, our results provide evidence that a neural network, including regions of vmPFC that reflect stimulus value, comparator processes in dmPFC and IPS, and action effectors in motor cortex, mediates the transformation of stimulus values into motor commands at the time of choice. This transformation process is of central importance to decision neuroscience. A critical question for future investigations is to what extent the network identified here is at work in a wide class of decisions (encompassing many different stimulus types and effectors) or if, in contrast, the network linking stimulus valuation, value comparison, and motor action is dependent on the parameters of the task.

Materials and Methods

Participants.

Twenty subjects (four females) participated in the experiment (mean age = 23 y, range = 19–35 y). All subjects were right-handed, were healthy, had normal or corrected to normal vision, had no history of psychiatric diagnoses or neurological or metabolic illnesses, and were not taking medications that interfered with the performance of fMRI. One male subject was excluded from analysis because of irregularities in his pattern of choices. The review board of the California Institute of Technology approved the study.

Stimuli and Task.

Subjects completed a juice decision task in the MRI scanner (SI Materials and Methods). At the beginning of each trial, subjects saw two different flavor amount combinations—one on the left side and one on the right side of the screen (Fig. 1A). Subjects were instructed to make their choice while the left and right options were on the screen. After a variable delay (3–6 s), a response prompt was shown on the screen, and the subject pressed the right thumb to select the right option or the left thumb to select the left option. The chosen liquid was delivered to the subject after another variable delay (3–6 s). There were a total of 120 trials across the four functional runs. SI Materials and Methods has details of liquid reward value calculations.

Neural DDM.

The model assumes that choices are made as follows every trial. There are two pools of neurons, which are spatially intermingled and of equal size, with total instantaneous levels of activity given by a_L(t) and a_R(t), where t indicated elapsed time from the appearance of the stimuli. At the beginning of the choice process, a_L(0) = a_R(0) = 0. A choice is made to the respective action when the level of activity in either of the two populations surpasses a prespecified threshold. The evolution equations for each population are given by (Eq. 1)

and (Eq. 2)

where measures the strength of the inhibitory activity between the two pools, d measures the sensitivity of the integration process to the underlying true values of the stimuli (denoted by v_L and v_R), and the last term, η, measures the measures the amplitude (SD) of Gaussian noise. We assume that the height of the barriers was fixed at ±1. Note that this normalization is without loss of generality, because the DDM is identified only up to relative values of the parameters. The psychometric curve for the best-fitting set of parameters (d = 0.009 ± 0.005, η = 0 ± 0.035, θ= 0.2) is shown in Fig. 2C. Total activity in each trial predicted by the model is referred to as M_out. SI Materials and Methods has model estimation details.

The total activity shown in Fig. 2D was used as a modulator in some of the GLMs of BOLD activity described below to identify areas that might be associated with the comparison process. The logic for using this variable as a marker is described. BOLD activity in any instant in the comparator area should be proportional to the sum of local neural activity. It follows that average bold activity in this area from the time of stimulus onset to the time of decision should be proportional to the average of instantaneous activities. In the analyses below, we cannot modulate activity with duration equal to the reaction time, because our paradigm does not allow us to measure it. Instead, we model the activity of the comparator with an equal duration in all trials. By the previous arguments, average activity during this time should be proportional to the total level of activity.

fMRI Data Acquisition and Analysis.

Data were acquired with a 3T scanner (Trio; Siemens) using an eight-channel phased array head coil (details in SI Materials and Methods). We estimated two GLMs with first-order autoregression for each individual subject (details in SI Materials and Methods). We then computed contrasts of interest at the individual level using linear combinations of the regressors. These contrast coefficients were used in one-sample t tests for second-level group analyses.

We carried out whole-brain corrections for multiple comparisons at the cluster level. Details of the correction for each contrast can be found in Tables S1–S11. Small-volume correction for the vmPFC was conducted within a 10-mm sphere centered on the vmPFC coordinates (x, y, z = −3, 42, −6) from the work by Chib et al. (16).

DCM.

We examined the connectivity between stimulus value and motor response regions on left and right choice trials using DCM (47). The analysis was carried out in several steps.

First, seven activation time courses were extracted from the functional masks in vmPFC, dmPFC, dlPFC, lIPS, rIPS, lMC, and rMC in each subject from a 4-mm sphere centered on the individual subject peak within the group regions of interest identified by the main GLMs and shown in Fig. 3 and Figs. S1 and Figs. S2.

Second, we specified 20 different models of potential connectivity between the seven areas of interest. A full description of the set of models is provided in SI Materials and Methods and Tables S5–S10.

Third, we identified the best model using the Bayesian model selection method developed in the work by Stephan et al. (44) (Fig. S3). Briefly, this technique treats the models as random variables and computes a distribution of the probabilities for all models under consideration (additional details in SI Materials and Methods).

Fourth, we used Bayesian parameter averaging (details in SI Materials and Methods) (48, 49) to estimate the magnitudes and probabilities of each fixed connection (often called intrinsic connections) as well as the magnitudes and effects with which the connections are modulated by different events.