Many hats: intratrial and reward level-dependent BOLD activity in the striatum and premotor cortex

Abstract

Human functional magnetic resonance imaging (fMRI) studies, as well as lesion, drug, and single-cell recording studies in animals, suggest that the striatum plays a key role in associating sensory events with rewarding actions, both by facilitating reward processing and prediction (i.e., reinforcement learning) and by biasing and later updating action selection. Previous human neuroimaging research has failed to dissociate striatal activity associated with reward, stimulus, and response processing, and previous electrophysiological research in nonhuman animals has typically only examined single striatal subregions. Overcoming both these limitations, we isolated blood oxygen level-dependent (BOLD) signal associated with four intratrial processes (stimulus, preparation of response, response, and feedback) in a visuomotor learning task and examined activity associated with each within four striatal subregions (ventral striatum, putamen, head of the caudate nucleus, and body of the caudate) and the lateral premotor cortex. Overall, the striatum and lateral premotor cortex were recruited during all trial components, confirming their importance in all aspects of visuomotor learning. However, the caudate was most active at stimulus and feedback, whereas the putamen peaked in activity at response. Activation in the lateral premotor cortex was, surprisingly, strongest during stimulus and following response as feedback approached. Activity was additionally examined at three reward magnitudes. Reward magnitude affected neural activity only during stimulus in the caudate, putamen, and premotor cortex, whereas the ventral striatum showed reward sensitivity during both stimulus and feedback. Collectively, these results indicate that each striatal region makes a unique contribution to visuomotor learning through functions performed at different points within single trials.

how humans, or other animals, learn to associate visual sensory events with actions has been studied extensively using tasks with a common trial structure (referred to here as visuomotor tasks): the organism is presented with a stimulus, makes a conditional response on the basis of that stimulus, and finally receives instructive feedback or reward. Although tasks following this basic structure are called by various names (instrumental or habit learning in rodents; arbitrary visuomotor, stimulus response, or conditional discrimination in monkeys; and trial-and-error categorization learning in humans), all variations have been shown to rely on the striatum, the input structure of the basal ganglia (for a review see Seger 2008). Evidence for involvement of the striatum in these tasks comes from lesion and electrophysiology work in nonhuman animals (Graybiel 2008; Yin and Knowlton 2006), as well as functional neuroimaging and neuropsychological studies of basal ganglia disorders in humans (Shohamy et al. 2008). In primates, the primary striatal neurons, typically referred to as phasically active neurons (PANs), exhibit multiple bursts of activity within single trials in visuomotor tasks (Haber 2008). Specifically, firing rates increase immediately following the stimulus, during and following response preparation and execution, and immediately following reward. Combined with the fact that not all striatal subregions exhibit identical patterns, these temporally discrete bursts of activity suggest the striatum plays multiple roles in visuomotor learning. Unfortunately, although electrophysiological studies provide excellent within-trial temporal resolution, they are, because of the inherent difficulty in acquiring such recordings, limited to examining one or two striatal subregions at a time. Conversely, although functional magnetic resonance imaging (fMRI) allows for whole brain imaging at millimeter resolution, its limited temporal resolution combined with the sluggish nature of the blood oxygen level-dependent (BOLD) signal complicates identification of striatal intratrial activity patterns.

In the present study we developed a visuomotor learning task that incorporates intratrial periods of temporal “jitter,” a highly optimized presentation sequence, and a rapid acquisition rate [repetition time (TR) = 1 s] to separately estimate, for the first time, the BOLD response associated with stimulus presentation (referred to as “stimulus”), response preparation (“preparation”), response execution (“response”), and feedback receipt and processing (“feedback”). Previous fMRI studies have only successfully dissociated activity during a combined visuomotor processing period from feedback (Aron et al. 2004; Lopez-Paniagua and Seger 2011) or separated reward receipt from reward anticipation (Knutson et al. 2001b). In addition, to identify those areas most sensitive to reward within a trial, we employed three levels of feedback: verbal and verbal plus two levels of monetary reward. We then examined activity in each of the four major subregions of the striatum: the putamen, head of the caudate, body of the caudate, and ventral striatum, as well as the premotor cortex. We based our predictions for activity within these regions on previous electrophysiological and functional imaging research, as reviewed below.

The head of the caudate nucleus has been most strongly associated with reward processing and goal-directed behaviors in human functional neuroimaging studies (O'Doherty et al. 2003; Schönberg et al. 2007; Seger and Cincotta 2005; Tricomi et al. 2004; Tricomi and Fiez 2008; Valentin and O'Doherty 2009) and electrophysiological recordings in primates and rodents (Brasted and Wise 2004; Corbit and Janak 2007; Hadj-Bouziane and Boussaoud 2003; Lau and Glimcher 2007; Williams and Eskandar 2006). However, neuroimaging studies that isolated a combined stimulus and response epoch from a feedback epoch found head recruitment during both epochs (Lopez-Paniagua and Seger 2011). Electrophysiological studies that have targeted the head in primates (or its rodent homolog, the dorsomedial striatum) also have reported distinct bursts of activity at stimulus presentation (Brasted and Wise 2004; Hadj-Bouziane and Boussaoud 2003; Williams and Eskandar 2006; Yamada et al. 2007) as well as during preparation and following response (Brasted and Wise 2004). Head activity at the time of response is generally observed to be less intense than that around stimulus and feedback. Combining these findings, we therefore predicted that the BOLD response in the head should be most prominent during feedback with additional (possibly substantial) activity present during stimulus.

Three separate findings led us to believe that all trial components would exhibit reward-level sensitivity. First, BOLD activity in the head correlates with reward valence and magnitude (Delgado et al. 2003). Second, recent neuronal recordings suggest that, prior to feedback delivery, PAN firing in the head is correlated with the value of selected or past actions (Lau and Glimcher 2007, 2008). Third, Kahnt et al. (2011) showed that BOLD activity in dorsomedial prefrontal cortex, which forms a neural network, or corticostriatal loop, with the head, tracks the variation in value between options at stimulus onset.

Functional neuroimaging studies of the body of the caudate have reported that activity is greater during stimulus and response processing than during feedback (Lopez-Paniagua and Seger 2011) and that this activity is associated with successful learning (Seger and Cincotta 2005; Seger et al. 2010). However, these studies were unable to differentiate between stimulus and response demands. Few electrophysiological studies have targeted this area, although those that have report activity only during stimulus processing (Brown and Desimone 1995). Combined with the knowledge that the body forms a functional loop with visual cortical areas, which may allow the body to assist in processing of visual category information (Ashby et al. 1998), activity in the body was expected solely at stimulus, free of any reward-level sensitivities.

Several laboratories have reported significant tuning of the putamen to presentation of visual stimuli (Brasted and Wise 2004; Buch et al. 2006; Graziano and Gross 1993; Hadj-Bouziane et al. 2003; Kimura 1992), whereas others have not (Barnes et al. 2005; Buch et al. 2006). Given the above, we made no strong a priori prediction about putamen activity at stimulus. Like the body of the caudate, however, functional imaging studies have found putamen activity during category learning is associated with successful learning (Lopez-Paniagua and Seger 2011; Seger and Cincotta 2005; Seger et al. 2010). In neuronal recording studies where the response is delayed by a cue, putamen activity leading up to response is also greater than during any other period and is dependent on correct responding. Incorrect responses do not show significant above-baseline activity (Brasted and Wise 2004; Buch et al. 2006; Hadj-Bouziane et al. 2006), implying a possibly causal relation between putamen activity and response accuracy (Williams and Eskandar 2006). Previous neuroimaging research has also found that putamen is recruited for a number of response-related functions, including selection and/or working memory (Brovelli et al. 2008), state-action value representations derived from reinforcement learning theory (Haruno and Kawato 2006; Seger et al. 2010), and, more recently, signaling of optimal behavioral policy (Li and Daw 2011). The final piece of evidence linking putamen to responding is the anatomic connections between putamen and primary motor cortex, primary somatosensory cortex, and premotor and supplementary motor cortices (Lawrence et al. 1998; Seger 2008). Given the apparent tight coupling between putamen and response, its role during and after feedback is unclear despite three studies that have reported putamen recruitment at reward delivery (Brasted and Wise 2004; Buch et al. 2006; Williams and Eskandar 2006). We therefore predicted activity in the putamen at feedback, as well; however, no reward-level effects were expected.

The function of the ventral striatum differs qualitatively from that of the other regions of the striatum, collectively referred to as the dorsal striatum. Dorsal striatum plays a role in mapping stimuli to appropriate behaviors in rewarding or salient contexts, and thus is contingent on the presented stimuli having behavioral relevance (O'Doherty et al. 2004; Tricomi et al. 2004; Yin et al. 2005; Zink et al. 2006). In contrast, ventral striatum is strongly and reliably recruited by Pavlovian rewarding events, those that require no overt behavioral response (O'Doherty et al. 2003, 2004). The ventral striatum is associated with signaling reward valance and magnitude (Cooper and Knutson 2008; Levita et al. 2009) and/or motivation (Knutson et al. 2001a), as well as salience (Cooper and Knutson 2008; Smith et al. 2011; Zink et al. 2003). This role in valuation is consistent with the ventral striatum's anatomic connections with the ventromedial prefrontal cortex (Frank 2011; Ito and Doya 2011; Seger 2008). Activity in ventral striatum has been further shown to correlate with reward prediction error, as well the transfer of reward value from reinforcer to the stimulus (Fiorillo et al. 2003; D'Ardenne et al. 2008; Rodriguez et al. 2006; Seger et al., 2010), which is driven by phasic dopamine release (Schonberg et al. 2010) from projections of the ventral tegmental area/substantia nigra pars compacta (VTA/SNc) (Hollerman and Schultz 1998; Schonberg et al. 2010). This research has led to substantial recent work investigating the neural correlates of reinforcement learning (Berridge 2007; Dayan and Niv 2008; Dommett et al. 2005; Roesch et al. 2010; for a review see Glimcher 2011).

Because of a focus on reward, relatively few electrophysiological studies have examined the ventral striatum throughout single trials in visuomotor tasks. Kim et al. (2007) found that primate ventral striatal cells were reliably active at the time of reward approach and receipt but were active at stimulus only when there was a history of reward from the previous trials (see also Nicola et al. 2004). They interpret their results as indicating that ventral striatal cells do not code for the behavioral choice on the current trial, but rather code for the reward history from previous trials. This is consistent with Roesch et al.'s (2009, 2010) studies of rodent striatum. On the basis of these studies, we hypothesized that ventral striatal activity would be confined to feedback and stimulus, with possible “leak” in response due to reward anticipation effects (Knutson et al. 2001a), and that stimulus, response, and feedback activity would all include robust reward-level effects.

Given our emphasis on dissociating preparation and response from stimulus and feedback, we also examined activity in the lateral premotor cortex. We focused on the finger area (Haaland et al. 2004) because our subjects made finger press responses and because prior fMRI studies in our laboratory found activity in this region (Lopez-Paniagua and Seger 2011; Seger et al. 2010). Toni et al. (1999), in an early neuroimaging experiment, also implicated dorsal premotor cortex in preparation in a delayed cued movement task. Primate electrophysiological recordings have reported premotor activity during preparation and execution (Cavina-Pratesi et al. 2006). Buch et al. (2006) compared the putamen and dorsal premotor activity during movement preparation: premotor activity peaked just before the response, and putamen peaked just after the response. In addition to response preparation and execution, studies of dorsal and lateral premotor cortex have suggested both play a role in processing visual signals for action planning (Hoshi and Tanji 2006). Electrophysiological studies of stimulus-response tasks have reported strong activity at stimulus and feedback (Brasted and Wise 2004; Buch et al. 2006) in addition to activity directly associated with planning and executing the motor response. We thus looked to confirm that premotor cortex may play an (underappreciated) role in all aspects of visuomotor learning, not just motor planning, and predicted recruitment of the premotor cortex in stimulus and feedback as well as in preparation and response. Reward magnitude effects, however, were not expected.

MATERIALS AND METHODS

Subjects.

Ten subjects (6 women, all right handed, 23–41 yr old) were recruited from the Colorado State University (Fort Collins, CO) and School of Medicine, University of Colorado Denver (Aurora, CO) communities. Subject prescreening excluded anyone who was not fluent in English or who had any history of psychiatric or neurological conditions, brain injury, magnet-incompatible implants, claustrophobia, or knowledge of written Japanese. Subjects gave written informed consent according to a protocol approved by the Colorado State University Institutional Review Board.

Task.

As illustrated in Fig. 1, trials were divided into four parts: stimulus (stimulus and category label presentation), preparation (time between stimulus offset and response cue, signaled by the preparation cue), response (time period including the response cue and postresponse period, cued by paired arrows), and feedback (including both verbal feedback and monetary reward amount if any). Stimuli were eight Japanese kanji characters presented in white on a black background (see Fig. 1 for an example) for 400 ms. Stimulus presentation was immediately followed by the category labels: images of yellow and blue cartoon fishes resting in the bottom corners (400 ms). The location of the category labels were switched randomly from one trial to the next to prevent the subject from learning a motor response (e.g., left or right hand) rather than the abstract category label (blue or yellow). Subjects were instructed to learn to associate each of the eight kanji images with one of the two category labels. Each kanji had only one correct association (blue or yellow), but the response associated with it randomly fluctuated between right and left, leading to four possible combinations (yellow-left, yellow-right, blue-left, and blue-right). Category label presentation was followed by two response screens: a preparation cue and a response cue. The preparation cue consisted of two yellow lines in the center of the screen; it served to provide a delay between the stimulus and response, which we refer to as the preparation period. Previous studies have shown that such delays allow subjects to complete aspects of response planning and preparation that can be dissociated from actual response execution (Gerardin et al. 2004). The response cue consisted of two green arrows presented at the same location as the preparation cue. Responses were made by a button press using the response pads placed under each hand. For example, if the subject believed that the stimulus belonged to the yellow fish category, and the yellow fish icon had appeared on the right-hand side, the subject would press the button with their right index finger on the pad held under their right hand. Subjects had 700 ms to respond following response cue onset. The response was followed by feedback, a fixation cross, and the start of the next trial.

Fig. 1.

Diagrammatic representation of the behavioral task. During the stimulus portion of the trial, the stimulus (Kanji character) was presented for 400 ms, followed immediately by the response mapping cue (blue and yellow fish) for 400 ms, followed by an intratrial jitter period. During the preparation period, subjects viewed 2 horizontal yellow dashes, which stayed on the screen for the entire time, including the intratrial jitter. At response, the yellow dashes were replaced by blue arrows (700 ms), indicating it was time to make the button press responses; this was followed by another intratrial jitter period. Finally, at feedback, subjects viewed 1 of 4 possible displays indicating whether the response was correct or incorrect and, if correct, what the reward received was, if any (incorrect, correct, correct + $0.10, or correct + $0.50). All intratrial jitter periods (indicated by black dashed arrows) extended for a variable amount of time ranging from 1,000 to 6,000 ms and featured a blank (black) screen, excepting the preparation jitter period as described above. Between trials there was a variable intertrial jitter period that extended for a variable amount of time ranging from 1,000 to 7,000 ms and during which subjects viewed a fixation cross. ITI, intertrial interval.

On each trial, subjects were given one of four possible forms of feedback or reward: three positive reward levels and one negative. Positive reward was either verbal with no monetary reward (“correct”) or a small (“correct + $0.10”) or a large (“correct + $0.50”) monetary reward. Negative feedback was always verbal (“incorrect”); there were no monetary penalties. Feedback type depended on the stimulus. Of the eight stimuli, two were always associated with the high monetary reward, two with the low monetary reward, and four with verbal feedback. Stimulus assignment to reward level was counterbalanced across subjects. Previous work using related tasks has demonstrated these levels lead to detectable graded changes in the striatal BOLD response with higher activity for larger monetary rewards (Delgado et al. 2003; Knutson et al. 2001a).

Although combining verbal and monetary rewards might at first seem counterintuitive, we believe it to be justified for two reasons. First, a recent categorization study comparing monetary to verbal rewards showed substantial overlap between these two reward types, with the verbal rewards leading to smaller BOLD responses than monetary rewards (Daniel and Pollmann 2010). Second, all trials included verbal feedback (“correct” or “incorrect”), but money trials also included the requisite amounts, providing a common basis between these two seemingly different reward types.

To achieve adequate separation of the trial into its components, the sluggish BOLD response requires interspersing a variable temporal delay, or “jitter.” However, post-pilot study interviews suggested that when the task featured only long (3–12 s) delays between trial components, the slow nature of the task led subjects to become inattentive. To counter this, trials were divided into two temporal types: long and short. Short trials followed a rapid time course (∼3 s for an entire trial), fast enough to require continuous attention. Long trials allowed for temporal separation of the BOLD signal into discrete trial components by introduction of a period of temporal jitter between each of the intratrial components. The 1- to 6-s jitter periods occurred after stimulus, after presentation of the preparation cue, and after a response was made (see Fig. 1 for a diagram). The jitter times for long trials were randomly sampled from an exponential distribution. Mean jitter time was 2.6 s, resulting in an average trial length of 6.6 s (Serences 2004). Long trials comprised 45% of all trials, with short trials occupying the remaining 55%. Both long and short trials were evenly distributed across the three reward levels for a total of 80 trials per subject for both low and high money conditions and 160 trials for verbal-only trials.

In addition to the intratrial jitter described above, the interval between trials was jittered. This intertrial jitter allowed the feedback-related activity, as well as overall trial-related activity, to be isolated. Intertrial jitter, as well as overall trial order, was optimized using a genetic algorithm (Wager and Nichols 2003) to jointly optimize trial estimation and detection, specifically targeting contrasts between each component and the baseline as well between the three reward levels. Because the genetic algorithm requires fixed trial lengths, we assumed each trial lasted 6.6 s (the mean trial time for long trials) in this optimization. Only “long” trials were included in the region-of-interest (ROI) analyses, because only they allowed for isolation of intratrial activity. Only trials in which the subjects responded correctly were included in the fMRI analyses.

Subjects were pretrained on the task, but not on specific visuomotor relationships. The pretraining task was identical to the scanner task except a single nonmonetary reward level was used (“correct” or “incorrect”). If subjects responded outside the response window (paired arrows in Fig. 1), the trial immediately ceased and was restarted. As a result, subjects learned to respond only during the desired response window. Before training, subjects were told any interruption in the trial was the result of response outside the assigned window. During training, subjects completed 10 trials of 8 different stimuli. The stimuli were kanji characters that did not appear in the scanner task and that were perceptually dissimilar from those that did.

Data acquisition and fMRI preprocessing.

Functional MRI data collection was performed on a GE 3T MR system, using a standard pulse sequence (gradient-recalled echo-planar imaging, SPGR-EPI) with the following functional acquisition parameters: echo time (TE) of 26 ms and a volume sampling rate (TR) of 1,000 ms. To achieve the fast 1-s TR, functional data were collected from only a portion of the brain (19 4-mm thick horizontal slices). The scanning volume was situated in the inferior-to-superior plane to encompass both the ventral striatum and the dorsal lateral premotor cortex. Anatomic images were collected using a T1-weighted SPGR sequence [minimal TR, TE = 3.95 ms, inversion time (TI) = 950 ms, flip angle (FA) = 10°, field of view = 220 mm, coronal matrix = 256 × 256; 166 1.2-mm slices]. The experiment was divided into 4 sections or scans (of about 80 trials each) ranging in length from 12 to 14 min. E-prime (version 2; Psychology Software Tools, Pittsburgh, PA) was used to control both stimulus presentation and behavioral recording. Stimuli were rear-projected onto a screen inside the scanner from a liquid crystal display (LCD) monitor positioned outside the scanner. Responses were collected using two magnet-compatible responses boxes, placed on the subject's thighs, with responses made using the left and right index fingers. Responses during pretraining were made using the arrow keys on a standard QWERTY keyboard.

Functional MRI data were preprocessed using standard methods, including rigid body and elastic (12) subvolume motion correction, slice time normalization (spline fit), temporal filtering to correct for drifts using a high-pass filter that excluded components with a frequency of <4 cycles across the scan, and spatial data smoothing using a Gaussian kernel full width at half-maximum of 4.0 mm. Each subject's high-resolution anatomic image and functional data were normalized into Talairach space (Talairach and Tournoux 1988). The hemodynamic response model (i.e., the “design matrix”) was subjected to the same high-pass filtering as the functional data. BOLD data were subjected to an autoregressive [AR(1)] smooth prior to general linear model calculation to minimize false positives due to nonwhite noise (Smith et al. 2007).

ROI analysis.

We defined the striatal subregions of interest on the basis of their functional interactions within recurrent corticostriatal loops (Lawrence et al. 1998; Seger 2008) to identify regional differences in activity during stimulus, preparation, response, and feedback. These subregions included the head of the caudate, which primarily interacts with lateral and dorsal prefrontal and parietal regions in the “executive” corticostriatal loop; the body of the caudate, which primarily interacts with extrastriatal and inferotemporal visual cortexes in the “visual” corticostriatal loop; the putamen, which primarily interacts with sensorimotor and premotor regions in the “motor” corticostriatal loop; and the ventral striatum (including the nucleus accumbens, ventral caudate, and ventral putamen), which interacts with ventral prefrontal cortex in the “motivational” corticostriatal loop.

Two hand-drawn sets of anatomic ROIs (ventral striatum, head of the caudate, body of the caudate, putamen, and lateral premotor) were created based on the subjects' combined Talairach normalized high-resolution anatomic data (Fig. 2). ROIs were drawn using an all-subject averaged (Talairach normalized) anatomic image. The density of gray in this averaged image allowed visual assessment of intersubject consistency. ROIs for each area were then limited to voxels that appeared to be included for all subjects. To ensure consistency, two sets of these images were initially created, and all analyses were repeated for each. Results were highly consistent between these two tracing sets, so only the results for the first set are discussed here. Additionally, each ROI was overlaid on each subject's individual scan to check for consistency. Lacking the clear anatomic boundaries of the striatal regions, the premotor regions were created by growing an oval region (6 × 3 voxels) centered on coordinates derived from previous human visuomotor learning experiments (i.e., Seger and Cincotta 2005). To ensure adequate separation of the head and body of the caudate ROIs, there was a 2-voxel gap between them.

Fig. 2.

Anatomically derived regions of interest overlaid on an average anatomic image. Dark green, head of the caudate; light green, body of the caudate; red, ventral striatum; light blue, putamen; dark blue, lateral premotor cortex. Numbers refer to the Talairach coordinate of the slice along the Y-axis.

Initially, unilateral ROIs were created, but early analyses showed strong consistency between the two hemispheres. Therefore, all analyses were carried out bilaterally by combining homologous regions in both right and left hemispheres. Bilateral ROIs offer increased power and, just as important for reader comprehension, condense without oversimplifying the observed patterns of activity. Changes in activity across trial components and scans were statistically assessed by pairwise contrasts (i.e., t-tests) inside each ROI using the general linear model. Significance thresholds were post hoc adjusted for multiple comparisons using the Bonferroni method.

RESULTS

Behavioral results.

Behavioral data were divided into four blocks (80 trials each) matching the fMRI data acquisition scans. Response accuracy was averaged across subjects and stimuli for each scan (Fig. 3, right). Above-chance learning was observed in scan 1 (78.8%). By scan 3 learning had plateaued at 96%. Each individual's performance was within 10% of the group mean, except for a single subject who showed somewhat delayed learning (48% accuracy in scan 1 but reaching 97% by scan 4). Despite the observed consistency in learning rate across subjects, it is possible that small individual differences in learning rate confound our trial-level analyses. Overall reaction times were nearly constant across scans (Fig. 3, left). This was expected, because the 700-ms response window was near the ∼400-ms floor observed during pilot studies. On average, subjects failed to respond in 4.4% of the trials. Missed response trials were excluded from both behavioral and fMRI analyses.

Fig. 3.

Average reaction time (left) and accuracy (right) for all subjects across functional MRI data acquisition blocks.

fMRI: whole brain.

We performed two initial whole brain analyses to confirm that our task recruited regions similar to those in previous studies. These analyses used the general linear model as implemented in BrainVoyager 2.03, controlling for intersubject variability by modeling subjects as a random effect (i.e., “RFX”), and employed a statistical threshold of q < 0.01, corrected for multiple comparisons using the false discovery rate (Genovese et al. 2002). Trial components were modeled as a series of single TR (unit) impulses at the onset of each component. Each series was convolved with a canonical hemodynamic response function, i.e., a linear combination of two gamma functions (no derivatives). The first contrast compared all trial components at all reward levels with the implicit baseline (defined in Brain Voyager as encompassing all other time points during the scan). As shown in Fig. 4, this contrast resulted in broad patterns of activity across the basal ganglia (bilateral caudate and putamen), as well as motor and executive cortical regions including lateral premotor cortex, posterior and anterior cingulate cortex (PCC and ACC), bilateral lateral dorsal prefrontal cortex, bilateral inferior parietal lobule, and the anterior insula/inferior frontal gyrus. This pattern of activity is consistent with previous studies of visuomotor learning in categorization and related tasks (Lopez-Paniagua and Seger 2011; Poldrack and Foerde 2008; Seger and Cincotta 2005).

Fig. 4.

Significance maps for all trials compared with the implicit baseline overlaid on a randomly selected subject's anatomic image [q < 0.01, false discovery rate (FDR)]. Numbers refer to the Talairach coordinate of the slice along the Z-axis.

Second, we examined activity for each individual trial component: stimulus, preparation, response, and feedback, compared with the implicit baseline, again collapsing across reward levels (Fig. 5). Overall activity during each of the individual trial components was similar to that during the collapsed contrast discussed in the previous paragraph. Notably, overall activity levels appeared to be lower during preparation than during other trial components. Feedback processing was associated with less motor cortical recruitment than stimulus and response trial components.

Fig. 5.

Significance maps for each trial component (stimulus, preparation, response, and feedback) compared with the implicit baseline (q < 0.001, FDR).

Finally, we examined the effect of reward level, comparing both monetary rewards with verbal feedback only (Fig. 6). Again, we examined this effect separately for each individual trial component. Overall, reward level affected neural recruitment at time of stimulus and feedback. There was no significant reward sensitivity during preparation or response. At stimulus, reward level modulated activity in broad regions of the prefrontal cortex, including lateral, medial, and ventral regions. At feedback, bilateral reward level-modulated activity was observed in a region encompassing the posterior middle temporal gyrus and middle occipital gyrus. When a more a sensitive but nonstatistically generalizable fixed-effects analysis was performed (wherein subjects' predictors are averaged rather than treated as random variables), reward-level sensitivity was displayed during feedback in the ventral striatum, with small clusters (<10 voxels) also appearing the head of the caudate and putamen (not shown).

Fig. 6.

Significance maps for effect of reward level (monetary reward vs. verbal reward) within each trial components (stimulus, preparation, response, and feedback) (q < 0.01, FDR).

ROI analysis.

ROI analyses were performed for each of the bilateral basal ganglia ROIs (ventral striatum, head of the caudate, body of the caudate, and putamen) and lateral premotor ROI described in materials and methods. We performed two contrasts for each ROI and each trial component. The first was against baseline, in which we included all correct trials, collapsed across all reward levels, and compared activity with the implicit baseline condition. The second was the reward level contrast, in which we compared activity for the two combined monetary reward levels with the verbal-only condition. Independent comparisons of the two monetary reward conditions indicated no reliable statistical differences between them.

ROI reward level-independent (main effect) activity.

Across the trial components the ROIs showed distinct patterns of recruitment (Fig. 7). The ventral striatum and head of the caudate were recruited during stimulus, response, and feedback, but not during preparation. Although otherwise similar to the head, the body of the caudate was marginally significant in the preparation period. However, activity in the body of the caudate during preparation was significantly lower than during all other periods [compared with stimulus: t(9) = 2.686, P < 0.0072; response: t(9) = 2.438, P < 0.015; feedback: t(9) = 3.647, P < 0.000027; all comparisons in this section were Bonferroni corrected, as appropriate]. The putamen was recruited across all trial events, but to a significantly greater extent during the response period compared with stimulus [t(9) = 2.67, P < 0.0058], and trended toward Bonferroni-corrected significance (and was significant without correction) compared with both feedback [t(9) = 2.041, P < 0.041] and preparation [t(9) = 2.059, P < 0.039]. Finally, the lateral premotor cortex was also recruited across all trial events but surprisingly was significantly more active during stimulus compared with the preparation [t(9) = 3.57, P < 0.00036] or response periods [t(9) = 2.50, P < 0.012] and in the feedback period compared with preparation [t(9) = 3.53, P < 0.000042].

Fig. 7.

Average percent signal change for each of the 5 regions of interest for each trial component compared with the implicit baseline. The significance level of each predictor (synonymous here with %signal change) is indicated by the shading of each bar (see Predictor Significance key), whereas significant pairwise differences between predictors (assessed by Bonferroni-corrected 2-tailed t-tests) are represented by lines above the bars: *P < 0.05; #P < 0.01; ##P < 0.01. venStr, ventral striatum; lat premotor, lateral premotor cortex.

To examine any learning-related changes, main-effect activity was examined separately for each of the four scan periods (Fig. 8). Stimulus period activity in the head, body, and lateral premotor ROIs significantly and sharply decreased between scans 2 and 3 [respectively: t(9) = 3.30, P < 0.00095, t(9) = 2.76, P < 0.000042, t(9) = 4.13, P < 0.000036], which corresponds to behavioral measures of learning reaching asymptote (Fig. 3). In the ventral striatum, stimulus-related activity only appeared during scan 2. In addition, the lateral premotor ROI showed significantly increased activity in scan 4 compared with scan 2 [t(9) = 3.24, P < 0.0012]. In the preparation period, activity in the body of the caudate and putamen remained fairly consistently elevated across scans. There was no recruitment in any individual scan of the ventral striatum or the head of the caudate in this period, consistent with results across scans shown in Fig. 3, whereas activity in the premotor region was quite varied, increasing in scans 1 and 3, and decreasing in scans 2 and 4. During response, activity in all ROIs increased with learning; activity in scan 3 was always greater than in scan 1 [in ventral striatum: t(9) = 2.39, P < 0.016; head: t(9) = 2.52, P < 0.012; body: t(9) = 2.57, P < 0.010; putamen: t(9) = 2.97, P < 0.0030; premotor: t(9) = 3.081, P < 0.0021]. However, activity in the ventral striatum and body trended toward a decrease in scan 4 compared with scan 3 [respectively: t(9) = 1.99, P < 0.046; t(9) = 1.87, P < 0.061]. Similar to the stimulus period, feedback activity in the head [t(9) = 3.30, P < 0.00095] and the ventral striatum [t(9) = 2.45, P < 0.014] significantly declined from scans 2 to 3 and scans 1 to 3, respectively. Feedback consistently recruited the putamen and premotor cortex across scans, whereas the body slightly increased from scan 1 to 2 (not significant) and then plateaued.

Fig. 8.

Average percent signal change for each region of interest and each trial component for each scan. The significance level of each predictor (synonymous here with %signal change) is indicated by the shading of each bar (see key), whereas significant pairwise differences between predictors (assessed by Bonferroni-corrected 2-tailed t-tests) are represented by lines above the bars: *P < 0.05; #P < 0.01; ##P < 0.01.

ROI reward-level activity.

Collapsing across trial components, all ROIs showed significant reward-level activity, with greater activity for monetary than verbal rewards (not shown). Analysis of the individual trial components across ROIs (Fig. 9) revealed that reward level only affected activity at some points during the trial. No ROIs showed reward level-dependent activity in either the preparation or response periods. All striatal ROIs were recruited during the stimulus period. In addition, the ventral striatum was also recruited during feedback. When collapsed across trial components the premotor cortex showed an overall reward-level effect (not shown); however, when examined across trial components, none reached significance, suggesting a weak sensitivity to reward history (Fig. 9). When reward activity was examined for each responsive ROI and trial component, all but the body showed decreases with learning (Fig. 10). Meanwhile, body activity became significant at scan 2 (all other ROIs were strongest in scan 1) but then declined significantly, becoming negative by scan 4 [t(9) = 2.79, P < 0.0052; Fig. 10].

Fig. 9.

Average percent signal change for the verbal-only condition contrasted to the 2 (combined) monetary conditions for each of the 5 regions of interest and each trial component. The significance level of each predictor (synonymous here with %signal change) is indicated by the shading of each bar (see key).

Fig. 10.

Average percent signal change for the verbal-only condition contrasted to the 2 (combined) monetary conditions, displayed by block. The significance level of each predictor (synonymous here with %signal change) is indicated by the shading of each bar (see key), whereas significant pairwise differences between predictors (assessed by Bonferroni-corrected 2-tailed t-tests) are represented by lines above the bars: *P < 0.05; #P < 0.01; ##P < 0.01. To simplify presentation, only regions that showed trial-component feedback-related effects are shown (i.e., venStr, head, body, and putamen in Fig. 9).

Overall, reward level-dependent effects were weaker for each ROI compared with the reward level-independent effects. However, the limited amount of detectable reward-level activity is likely not due to insufficient power. In the analysis examining the main effect of trial component (see Fig. 7), stimulus-related activity in both the head and body of the caudate was as strong as that observed during feedback. Additionally, activity in the putamen was strongest in the response period and during stimulus in lateral premotor region; therefore, these time periods should be most sensitive to possible reward-level changes.

DISCUSSION

We scanned subjects while they learned to associate eight abstract images with two category labels based on feedback received at the end of each trial. We were able to separately model, and thus isolate, BOLD responses from separate intratrial periods (stimulus, preparation, response, and feedback). We then examined activity in individual regions of interest within the striatum and premotor cortex for each of these trial components to assess how this activity was modulated by reward magnitude and finally examined how this activity changed with learning. Overall, we found a number of notable patterns of activity, which we discuss by striatal subregion.

Caudate nucleus.

First, we will focus on overall activity in the head of the caudate (i.e., activity observed compared with the implicit baseline). The head of the caudate was recruited during stimulus, response, and feedback but not during preparation. Activity during stimulus and feedback was predicted, confirming this region's role in goal-directed behavior and feedback processing. The drop during preparation was also expected (Brasted and Wise 2004). Prior reports suggested activity at response would be significantly lower than during stimulus and feedback. Instead, response activity in the head was as robust as that seen in the other two active periods. Initially we believed this activity could be accounted for by the role of the head in coding for the value of past actions (Lau and Glimcher 2007, 2008); however, as discussed below, this interpretation was not well supported by the reward-level comparisons. The observed response-related activity in the head, we suggest, may be instead due to working memory (Postle and D'Esposito 1999), updating or adjusting response criterion (Kuchinke et al. 2011), or other cognitive demand supported by the head's recurrent connection to dorsomedial prefrontal cortex (Frank 2011).

For the body, we predicted activity to be mostly limited to stimulus; however, the overall pattern of activity in the body was very similar to that in the head: it was recruited at stimulus, preparation, and feedback but significantly less so during preparation. When activity was examined by block (i.e., across learning; Fig. 8), three major differences between the head and body of the caudate emerged. The first was the body's trend toward an increase and then a decrease in activity at response. The second was intermittent activity in the body during preparation that was absent in the head. The third was in the response to feedback across learning: in the head of the caudate activity was significant in early scans but dropped off, whereas in the body activity increased across scans. We have previously found activity in the head decreases or remains stable across scans, whereas activity in the body increases (Lopez-Paniagua and Seger 2011; Seger and Cincotta 2005), and attributed that difference to the body's continuing role in processing of visual category information and mapping visual processing to responses (Ashby and Maddox 2005; Nomura and Reber 2008; Seger and Miller 2010) compared with head's role in feedback processing, which decreases across trials as feedback becomes expected (Lopez-Paniagua and Seger 2011; Seger and Cincotta 2005; Seger et al. 2010).

In the reward-level analyses, which compared the two monetary conditions to verbal feedback, both head and body showed significant effects at stimulus but at no other times during the trial. For the head, these results are informative in two ways. First, they are in contrast to our original prediction that reward level should most strongly impact feedback. Although we did find that the caudate was active during feedback, overall this activity appeared not to be sensitive to the magnitude of the reward. This is difficult to reconcile with prior findings of reward and reward prediction error signal correlations in the head (Rodriguez et al. 2006; Seger et al. 2010). Our results are, however, consistent with theories that striatal neurons encode past reward values at stimulus (Hollerman and Schultz 1998; Kahnt et al. 2011), as well as with motivational accounts of head activity (Delgado et al. 2004). Second, the lack of reward magnitude sensitivity observed during preparation and response is seemingly inconsistent with the results of Lau and Glimcher (2007, 2008), who have argued that the caudate represents the value of specific actions. There are, however, some significant differences between their task and ours. In our task the correct motor responses randomly changed on a trial-by-trial basis (while category label responses remained constant). For subjects to complete the task, they had to map labels to actions, and therefore the action value representations had to be transferred from label to action. One possibility is that PANs in the head signal action values only for direct and consistent reinforcement of motor responses. A simpler possibility is that Lau and Glimcher examined eye movement processing, whereas our subjects responded using finger movements; neural coding in the head (in monkeys) may reflect the value of ocular movements or saccades, but not all movements, which is, in general, consistent with prior findings of effector-specific value representations in human subjects (Gershman et al. 2009).

Putamen.

We observed significant BOLD activity (both overall and as a function of reward level) in the putamen during stimulus, consistent with previous electrophysiological reports of activity in this period. The putamen was also active for all other trial components but to a significantly greater extent during response. In addition, the putamen was recruited to a relatively greater degree than the caudate during preparation. Activity during response and preparation support the putamen's known role in action selection and execution as part of the motor corticostriatal loop (Buch et al., 2006; Gerardin et al. 2004; Grahn et al. 2008; Muranishi et al. 2011; Romo et al. 1992; Ueda and Kimura 2003). There was no effect of reward level during preparation and response despite strong overall activity. This result is consistent with a recent study by Li and Daw (2011), who found that putamen activity was better predicted by a model that directly learns which particular actions are advantageous rather than indirectly learning to choose advantageous actions based on an internal model of the reward values associated with particular states and actions. Previous studies have reported correlations between the putamen BOLD signal and reinforcement learning-derived measures of expected value (Haruno and Kawato 2006; Seger et al. 2010), whereas Li and Daw's (2011) results imply that putamen activity is better modeled as policy selection. One possible resolution to these conflicting reports is suggested by our finding of a reward-level effect at stimulus: the reported value correlations may be due to activity at stimulus, whereas activity at response may drive the policy correlation.

Ventral striatum.

The ventral striatum was recruited overall during stimulus, response, and feedback trial components. Notably, in addition to showing reward-level effects at the time of stimulus like other striatal regions, the ventral striatum was the only region to show reward level-dependent activity at feedback. This result confirms this region's crucial role in feedback processing (D'Ardenne et al. 2008; O'Doherty et al. 2003; Pagnoni et al. 2002; Schönberg et al. 2007; Schultz et al. 1992). The lack of reward-level sensitivity in the preparation and response periods, as well as reward-level effects during feedback, are inconsistent with the hypothesis that the ventral striatum serves a reward anticipatory function (Knutson et al. 2001a, 2001c; for a review see Knutson and Wimmer 2007). It is worth noting, however, that reward anticipation studies have primarily employed the monetary incentive delay (MID) task, which differs markedly from our task, especially with regard to event timing. The delay between stimulus and reward in MID varies between 2–2.5 (Knutson et al. 2001a, 2001b) and 4–4.5 s (Knutson et al. 2000). Our study featured a much larger variability, with delays ranging from 100 ms to 8 s. This variability may have prevented an anticipation signal from developing. If so, it suggests that the ventral striatum may have anticipatory activity only when there is a consistent temporal structure.

Lateral premotor.

The lateral premotor region was active across all trial components and all scans but with overall greatest activity at time of stimulus. This supports theories that the premotor region is involved in processing visual signals for action planning (Hoshi and Tanji 2006). Activity during response is consistent with the motor programming roles of this region. We did not predict the observed activity during feedback, indicating that this region may play an unappreciated role in feedback and reward processing. However, there was no effect of reward magnitude on premotor activity implying that any role in feedback or reward processing is insensitive to reward value.

Conclusion.

Activity as measured by the BOLD signal displayed generally distinct responses to changing task demands for each region of interest, trial component, reward manipulation, and learning trajectory. We speculate that this variability has an unappreciated implication. Previous researchers examining the striatum (including ourselves) often have argued for a variety of exclusive interpretations as to each region's function. For example, the reward anticipation/motivation role of the ventral striatum (Knutson et al. 2001a) has been exclusively contrasted to its alternative role in reward processing via the reward prediction error (D'Ardenne et al. 2008). Although a single function for each striatal region may be parsimonious, it is untenable given the intra-trial variations we observed. If a single region truly implements a singular function (or set of very similar functions), then it should exhibit a consistent pattern of activity reflecting only that function and should otherwise be invariant to changing task demands. In contrast, multiple patterns of activity such as those we report imply multiple computational roles.

Our argument that single striatal regions implement multiple functions is consistent with a number of recent reports identifying separately tuned subpopulations of neurons within the striatal regions we studied. For example, reward-sensitive neurons in the head of the caudate were shown by Lau and Glimcher (2008) to be distinct from neurons active immediately pre- and postresponse. Kubota et al. (2009) found some dorsolateral striatum cells were primarily sensitive to cues, whereas other cells were primarily sensitive to goal attainment. Kimura et al. (1992) identified three independent groups of cells in putamen for three kinds of movements: sensorially guided, internally timed self-initiated, and memory guided. Each striatal region's cortical connections within the corticostriatal loops must certainly affect, and help determine, the roles each striatal region can play.

In conclusion, using a simple deterministic visuomotor learning task, we have isolated for the first time the BOLD response stimulus from that corresponding to response preparation/anticipation, as well from response and also feedback presentation. We further examined how these intratrial components differed across striatal region, how they were affected by reward magnitude, and how they changed with learning. In total, we report a strongly heterogeneous set of activities consistent with the idea that each region may make unique contributions, based on current context and task demands, to each aspect of visuomotor learning. Therefore, the human striatum may wear many (functional) hats whose activities significantly impact BOLD signals in the human brain.

GRANTS

This research was supported by National Institute of Mental Health Grant R01 MH079182 (to C. A. Seger).

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

E.J.P. and C.A.S. conception and design of research; E.J.P. performed experiments; E.J.P. analyzed data; E.J.P. and C.A.S. interpreted results of experiments; E.J.P. prepared figures; E.J.P. and C.A.S. drafted manuscript; E.J.P. and C.A.S. edited and revised manuscript; E.J.P. and C.A.S. approved final version of manuscript.