Performance monitoring by presupplementary and supplementary motor area during an arm movement countermanding task

Abstract

A key component of executive control and decision making is the ability to use the consequences of chosen actions to update and inform the process of future action selection. Evaluative signals, which monitor the outcomes of actions, are critical for this ability. Signals related to the evaluation of actions have been identified in eye movement-related areas of the medial frontal cortex. Here we examined whether such evaluative signals are also present in areas of the medial frontal cortex related to arm movements. To answer this question, we recorded from cells in the supplementary motor area (SMA) and pre-SMA, while monkeys performed an arm movement version of the countermanding paradigm. SMA and pre-SMA have been implicated in the higher-order control of movement selection and execution, although their precise role within the skeletomotor control circuit is unclear. We found evaluative signals that encode information about the expected outcome of the reward, the actual outcome, and the mismatch between actual and intended outcome. These findings suggest that signals that monitor and evaluate movement outcomes are represented throughout the medial frontal cortex, playing a general role across effector systems. These evaluation signals supervise the relationship between intentional motor behavior and reward expectation and could be used to adaptively shape future goal-directed behavior.

goal-directed behavior is based on the selection of actions that will lead to the highest expected reward in any given context (Rangel et al. 2008). However, in an ever-changing environment, task requirements can rapidly and unexpectedly change. The necessary updating of the action-outcome signals, therefore, requires the continuous assessment of both the ongoing actions and the outcomes of these actions (Ridderinkhof et al. 2004). Specifically, this requires the representation of expected reward outcomes after an action has been chosen, and the representation of the actual reward outcome after it is revealed. These two representations, in turn, could be used to compute reward prediction error signals that could guide adjustments in the action-value signals (So and Stuphorn 2012). Note that, in order for such a reinforcement learning scheme to work, it is also necessary to retain information about the type of action that is evaluated, i.e., an eligibility trace (Sutton and Barto 1998). This could be achieved either through a separate working memory trace of the chosen action or through movement-contingency of the evaluation signals.

The medial frontal cortex has long been implicated in monitoring and comparing actual performance with internal goals and standards (Ridderinkhof et al. 2004). Human imaging studies suggest that a large region of the medial wall, including the presupplementary motor area (pre-SMA) and the anterior cingulate cortex, represent signals related to response conflict, decision uncertainty, and response error. In particular, pre-SMA activity has been interpreted as error detection signals (Garavan et al. 2002; Hester et al. 2004; Rubia et al. 2001; Ullsperger and von Cramon 2001, 2003; Ullsperger et al. 2002). Human errors are associated with a negative deflection in their event-related scalp potential (error-related negativity; ERN) recorded over the medial frontal cortex (Falkenstein et al. 1991; Gehring et al. 1993). However, other studies have suggested that the pre-SMA may be involved in representing uncertainty or conflict rather than error, since pre-SMA activation was observed after an incorrect response, even when error feedback was not provided (Garavan et al. 2002; Ullsperger and von Cramon 2001, 2003).

In human studies, mostly negative outcome signals have been described in the pre-SMA. In contrast, single-unit recordings in monkeys predominantly find signals encoding positive outcomes (Akkal et al. 2002; Mars et al. 2005). Activity was found to arise before the end of the last movement and continue through reward delivery. Shima and Tanji (2000) found a similar response, but interpreted it not as an expectation of reward, but as the accomplishment of a motor sequence, which was necessary to obtain reward in their task. Many of these neurons were active in both the movement and postmovement period, suggesting a link between the action and its outcome. Reward expectation signals were also observed in the SMA in the postsaccadic period of an eye movement task (Campos et al. 2005). This burst gradually disappeared if reward was eliminated after a correct response and was not elicited by unexpected rewards. Thus reward expectation may play a role throughout the medial frontal region.

This discrepancy between negative outcome signals that have been described in humans, and positive outcome signals that have been described in nonhuman primates, is most likely related to the use of different behavioral paradigms. In experiments that rely on the monkey's performance on a behavioral task, the animal is usually rewarded for performing the task and has been trained to a high level of performance. This precludes the observation of many negative behavioral outcomes. The stop signal task overcomes this dilemma, since task difficulty can be titrated in such a way so that the subject makes mistakes on about one-half of the critical trials, despite every attempt to perform optimally. Using this task, it was found that the supplementary eye field (SEF) contains both negative and positive outcome evaluation signals, as well as conflict-related signals (Stuphorn et al. 2000). It was recently shown that macaques exhibit an ERN recorded from the cranial surface (Godlove et al. 2011; Schall and Godlove 2012). Local field potentials in the macaque SEF exhibit polarization and timing corresponding precisely to this ERN signal (Emeric et al. 2010), consistent with a current source in medial frontal cortex.

We, therefore, decided to use the stop signal task to test if pre-SMA and SMA, which are neighboring areas to SEF, also contain both positive and negative outcome signals, and to characterize the nature of these signals. Our results showed that evaluative signals are carried by SMA and pre-SMA neurons. The vast majority of these pre-SMA and SMA cells represented three types of evaluative signals: error, reward expectation, and reward signals. In addition, we found a smaller subset of cells that may signal any unexpected outcome independent of its valence. These findings suggest that evaluative signals exist in many areas of the brain, even those thought to have a primarily motor role, and may help link actions with their consequences to help guide future outcomes.

MATERIAL AND METHODS

General.

Two rhesus monkeys were trained to perform the tasks used in this study. All animal care and experimental procedures were approved by Johns Hopkins University Animal Care and Use Committee. During the experimental sessions, each monkey was seated in a primate chair, with its head restrained, facing a video screen. A handle bar was placed in front of the monkey that moved 12 cm in either direction along the horizontal axis. The bar controlled a rectangular cursor on the video screen. The right arm was used for the task. Handle bar position was recorded with the PLEXON MAP system at a sampling rate of 1,000 Hz. Eye movement was monitored with an infrared corneal reflection system (Eye Link) at a sampling rate of 1,000 Hz.

Behavioral tasks.

The countermanding task began with the onset of a yellow center box in the middle of the screen (Fig. 1), instructing the monkey to move the cursor into the box. Following a variable delay (200–400 ms), the center box disappeared and a target box simultaneously appeared 16 visual degrees to the right or left of center, which cued the monkey to move the cursor into the target box within 700 ms to receive a liquid reward (no-stop signal trials). On 25–35% of trials, the center box reappeared after a delay, referred to as the stop signal delay (SSD), and instructed the monkey to withhold movement initiation to receive reward (stop trials). The reward was delivered 400–800 ms after the stop signal onset. In contrast to previously reported countermanding paradigms, if the monkey started to make a movement toward the target, but was able to interrupt this movement before the target was reached, he had the opportunity to return the cursor back to the center for a reward (to receive only one-half reward). Four SSDs were selected through baseline training sessions so the monkey could withhold movement on ∼10, 35, 65, and 90% of the stop trials. The SSDs used for monkey B were 126, 215, 299, and 385 ms and for monkey E were 150, 240, 330, and 420 ms.

Fig. 1.

The sequence of events in the arm countermanding task. The trial begins when the cursor is positioned inside the center box. After a delay, the target box appears to one side of the screen, and the center box disappears, instructing the monkey to move the cursor into the target box. On stop signal trials, the center box reappears after the stop signal delay (SSD), signaling that the monkey should cancel the planned movement.

Behavioral analysis.

The relevant behavioral data for describing the inhibitory process in the countermanding task are 1) the inhibition function and 2) the reaction time distribution for the movements on trials with no-stop signals. The inhibition function plots the proportion of stop signal trials in which the subject generates a movement as a function of the delay between target onset and stop signal onset (the SSD). The probability of erroneous initiation of the movement increases as SSD increases. Performance on the countermanding task can be modeled as a race between a stochastic process that generates the movement (GO process), and a stochastic process that inhibits the movement (STOP process) (Logan and Cowan 1984). The two processes race independently toward their respective thresholds. If the STOP process finishes before the GO process, the movement is not generated (canceled trials). On the other hand, if the GO process finishes before the STOP process, the movement is generated (noncanceled trials). This race model provides an estimate of the stop signal reaction time (SSRT), which is the time needed to cancel the planned movement. We estimated the SSRT using methods detailed previously (Hanes and Schall 1995; Hanes et al. 1998). First, we used the method of integration, based on the assumption that SSRT is constant. We took the mean of the SSRTs calculated individually for each SSD. Second, we used a method based on the mean of the inhibition function. We calculated one SSRT estimate based on the raw behavioral data, i.e., the frequency of noncanceled trials for each SSD. We calculated two more estimates based on different fits of the inhibition function, using a Weibull function and a monotonic spline function. We obtained an overall estimate of SSRT by averaging over the four different estimates. The difficulty of controlling the movement generation can be adjusted parametrically by changing the SSD. As a result, the monkey is fully engaged in the task, but still generates a similar number of noncanceled, corrected, and canceled stop signal trials overall. The proportion of each type of stop trial will vary by SSD, however, which forms the basis of the inhibition function. In our calculations of the inhibition function, noncanceled and corrected trials are counted as incorrect, since the monkey incorrectly makes a movement in both of these trial types.

Identification of arm movements.

The analog data from the handle bar were analyzed to find the beginning and end of arm movements. First, the position signal was smoothed by taking the average of every five data points. To determine movement onset, we first identified locations where there were five changes in handlebar position that were at most 25 ms away from one another. Movement onset was defined as the time at which the first of the five handlebar position changes occurred. We used this method instead of a velocity cutoff, since it allowed us to examine the velocity without requiring a specific speed to be reached. We found that the method very consistently identified the beginning of movement. Movement end was defined as the first point after movement onset, where the position stayed constant for more than 40 ms. We visually inspected the movement start and end times for arm movements that were found using these algorithms to ensure their accuracy.

Single-unit recording.

After training, we placed a square chamber (20 × 20 mm) centered over the midline, 25 mm (monkey B) or 21 mm (monkey E) anterior of the interaural line. Single units were recorded using tungsten microelectrodes with an impedance of 2–4 MΩ. Data were collected using the PLEXON MAP system. Up to four template spikes were identified using principal component analysis, and the time stamps were then collected at a sampling rate of 1,000 Hz. Data were subsequently analyzed offline to ensure only single units were included in consequent analyses.

Spike density functions.

To represent neural activity as a continuous function, we calculated spike density functions by convolving the spike train with a growth-decay exponential function that resembled a postsynaptic potential. Each spike, therefore, exerts influence only forward in time. The equation describes rate (R) as a function of time (t): R(t) = [1 − exp(−t/τ_g)]·exp(−t/τ_d), where τ_g is the time constant for the growth phase of the potential, and τ_d is the time constant for the decay phase. Based on physiological data from excitatory synapses, we used 1 ms for the value of τ_g and 20 ms for the value of τ_d (Sayer et al. 1990).

Detection of evaluative signals.

We analyzed neural activity in the reinforcement period defined as the time 200 ms before movement end to 500 ms after the time reward was received. Student's t-tests were performed on the spike rates in 20-ms intervals throughout the epoch time period, compared with a baseline period that consisted of the 500 ms before target onset. If P values were <0.05 for three or more intervals during the time period, the cell was deemed to have activity significantly different from baseline. The activity of all cells was examined separately for movements to the right and the left. Cells with significant activity in this task epoch were considered to have a possible evaluative role and are the focus of this study. This method was used to identify and analyze movement-related cells, which are described in a previous paper (Scangos and Stuphorn 2010). To ensure that the significant activity during this time epoch was not due to movements of the handle bar that occurred when the monkey moved the handle bar back to the center in preparation for the next trial, we eliminated all cells from the potential evaluative cell pool that were identified to have significant return movement activity (described in Scangos and Stuphorn 2010).

We then performed additional analyses on the remaining cells. These additional tests consisted of a set of comparisons of the neuronal activity in different trial types. To test for significance, we performed t-tests on the spike rates in these two trial types in 60-ms intervals throughout the time period of investigation. This longer time interval was used because the cell numbers were often smaller and the differential responses were often quite brief. If values of P were < 0.05 for three out of four consecutive intervals during the time period, the cell was classified as having significantly different activity in the two conditions. Alternatively, a neuron was also classified as having significantly different activity if it showed a briefer, but more significant, activity difference (P < 0.01) in at least one of two consecutive time bins. The probability of a false positive for three t-tests with P values < 0.05 is very low (P = 0.000125). However, we performed multiple tests, so that the overall significance level of our test was P < 0.01 for each tested neuron. In all cases, we define the time of response onset as the middle time value of the first time bin that showed a significant difference. Following the onset of the significant response, the end of the response was determined by finding the middle time value of the first of two consecutive time bins that showed a P value > 0.05. These criteria were used throughout the cell classifications.

Error cells.

Error cells showed higher activity on noncanceled trials than on no-stop signal trials in the 480 ms following movement end. Cells that also responded to unexpected reward were excluded from the “error” cell classification, as explained in the next paragraph.

Surprise cells.

For some recording sessions, we delivered an unexpected reward in the intertrial time period on 5% of trials. We searched for cells that were active following unexpected reward, by comparing the activity during the 480 ms following the unexpected reward to activity in a 250-ms baseline before the unexpected reward was delivered. Both epochs were part of the intertrial interval. A t-test was applied in 60-ms intervals to compare the activity in the extra reward epoch to the baseline period. Cells with significant activity on noncanceled trials and after unexpected reward were classified as “surprise” cells.

Reward expectation cells.

Reward expectation cells showed an activity pattern that was the converse of error cells, showing higher activity on no-stop signal (correct, rewarded) trials than on noncanceled (error, nonrewarded) trials in the 420 ms following movement end. The period in which the variable delayed reward was given was 400–800 ms following movement end, so this epoch consisted of only prereward activity. To test for significance, we performed t-tests on the spike rates in these two trial types in 60-ms intervals throughout the 480 ms following movement end. Some reward-expectation cells showed additional reward-related activity, as explained in the following section.

Reward cells.

Reward cells had to satisfy three criteria. First, reward cells had to show an increased firing rate in the reward delivery time period (the 480 ms following reward onset) compared with the 250-ms period preceding reward delivery. Activity was tested in 60-ms intervals using t-tests and was deemed to be significantly increased if three consecutive intervals had P < 0.05. This activity difference had to be present on all rewarded trials (no-stop signal, canceled, corrected), independent of preceding action.

Second, reward cells had to show significantly elevated activity on no-stop (correct, rewarded) trials, compared with the activity on noncanceled (error, unrewarded) trials. Again, we applied t-tests in 60-ms intervals from movement end until 480 ms following reward. Activity was considered to be significantly different, if P values were <0.05 for three out of four consecutive intervals during the postreward time period.

Some cells showed significant activity that began before reward delivery and extended through reward delivery. These cells were classified as reward expectation cells. A third criterion was used to determine whether these cells also carried an additional reward signal by searching for cells that responded more to reward delivery than to the anticipation of reward. On rewarded no-stop signal trials, activity in the period immediately following the reward delivery was compared with the activity in the 250 ms prior to reward delivery. Cells which showed elevated activity during postreward compared with prereward, as indicated by three consecutive 60-ms intervals (t-test; P < 0.05), were categorized as carrying reward-related signals. Cells that responded during the “anticipation” period and also showed elevated postreward activity in this test were categorized as carrying both reward-anticipation and reward-related signals.

As described in the Surprise cells section, we also examined activity in response to unexpected reward. For reward cells, we compared the response to unexpected reward with the one to expected reward in the 480 ms following each event. We compared the unexpected and expected reward epochs using a t-test in 60-ms intervals. If there were three consecutive intervals with P < 0.05, we concluded that the neuron significantly favored one type of reward over the other.

Cortical localization.

To determine the locations of the pre-SMA and SMA, we obtained magnetic resonance images for both monkeys (1.5 T) (Chen et al. 2010; Scangos and Stuphorn 2010). A three-dimensional model of the brain was constructed using Brain Voyager (Brain Innovation, Maastricht, The Netherlands) and Rhinoceros (McNeel North America, Seattle, WA). The border of the SMA and pre-SMA was defined by the location of the branch of the arcuate sulcus. Neurons within the region 6 mm posterior to the arcuate branch and within 3.5 mm of the longitudinal fissure were designated as belonging to the SMA. Neurons within the region 5 mm anterior to the arcuate branch and within 3.5 mm of the longitudinal fissure were designated as belonging to the pre-SMA. The neurons were recorded in the medial wall at depths of 6,473–10,618 μm below the surface in monkey B (mean depth: 8,447 μm) and 6,540–8,986 μm below the surface in monkey E (mean depth: 7,534 μm).

RESULTS

Behavior.

The behavioral findings from the countermanding paradigm have been presented previously (Chen et al. 2010; Scangos and Stuphorn 2010). Here we summarize the main findings. Stop signals were adjusted during training so as to obtain a roughly equal overall number of canceled, corrected, and noncanceled trials, ensuring a comparable level of difficulty for each monkey. For all days, the percentage of stop trials ranged from 25 to 30% (go trials = 70–75%). Figure 2A shows the handle bar traces for canceled (top), no-stop signal (middle), and noncanceled trials (bottom) for the recording session on 1 day for monkey B. As shown in the trace, no movement occurred on canceled trials, and a smooth movement toward the target box occurred for both no-stop signal and noncanceled trials. Figure 2, B and C, shows the response time distributions (B), inhibition function (C), and mean proportion of each type of stop trial for each SSD (D) across all recording sessions. The upper row shows these data for monkey B, while the data for monkey E are shown in the lower row. The behavioral data were used to estimate the SSRT. The mean SSRT for monkey B was 139 ms and for monkey E was 138 ms.

Fig. 2.

The behavioral data across recording sessions. A: handlebar traces for monkey B during one recording showing the handlebar position on canceled trials (top), no-stop signal trials (middle), and noncanceled trials (bottom). The response time distributions (B) and the inhibition functions (C) across all recording sessions for monkey B (top) and monkey E (bottom) are shown. D: the mean proportion of each type of stop trial by SSD showing canceled trials in red, corrected trials in blue, and noncanceled trials in green for each monkey. E: the effects of trial history across recording sessions. Left column shows response times for no-stop signal trials one, two, or three trials away from noncanceled trials for monkey B (top) and monkey E (bottom). Right column shows response times for no-stop signal trials one, two, or three trials away from canceled trials. The type of trials to which the response time corresponds is shown in bold (G: no-stop signal; E: noncanceled; Ca: canceled). The dotted line indicates the average response time on no-stop signal trials.

Humans and monkeys show adjustments of saccade response time according to trial history (Emeric et al. 2007; Nelson et al. 2010). In particular, response times increase after successive stop signal trials and decrease after successive no-stop signal trials. More generally, studies have shown response slowing following an error in choice tasks (Rabbitt 1966). This observation has been regarded as evidence of executive control. We have found similar behavioral adjustments in response to trial history in the arm-stop signal task (Chen et al. 2010). Both monkeys adjusted their behavior in response to the outcome of the stop trial, as shown in Fig. 2E. In monkey B, the response time on the following no-stop signal trials was significantly slowed (E-Go: 419 ms), both relative to the noncanceled trials (E: 368 ms; t-test, P < 0.0001) and relative to the last no-stop signal trial before the error (Go-E: 401 ms; t-test, P < 0.0001). Thus monkey B slowed down after an error, presumably in an attempt to improve subsequent performance (Fig. 2E, top left plot). In contrast, monkey E primarily strengthened its response strategy following successes (Go-Ca: 417 ms, Ca-Go: 426 ms; t-test, P < 0.001). Both strategies represent forms of proactive control, where the level of motor readiness is adjusted depending on past trial history (Chen et al. 2010).

Neuronal data set.

The activity of 414 neurons was recorded from the SMA and pre-SMA in the left hemispheres of two rhesus monkeys while they performed an arm movement stop signal task. We recorded the activity of 158 pre-SMA and 197 SMA cells that had activity related to at least one task epoch (see materials and methods). Of the total 414 neurons, 34 (8%) were found to have significantly increased activity after an error was made. These error cells became active on noncanceled trials but were not active on no-stop signal trials, where similar movements were made, but with a different outcome. A second group of cells exhibited activity changes specifically in anticipation of an expected reward and were termed reward expectation cells. We found 77 reward expectation cells out of 414 total recorded cells (19%). A third group of cells, 18 out of the 414 total neurons (4%), showed activity after a reward was actually obtained by the monkey. They were termed reward cells. Finally, a small number of cells were found that responded to any surprising stimulus (4 cells). Together, these 133 cells form the data set for the analyses described in this paper. Table 1 shows a breakdown of the major cell groups in each monkey/brain region. The location of these cells within the recording regions is shown in Fig. 3. When cell types across the two monkeys were combined, no clear topographic pattern was found for the different evaluative or movement cell groups (Fig. 3D).

Table 1.

Number of each type of monitoring cell found in the pre-SMA and SMA, broken down by monkey

	Monkey B		Monkey E
Neuron Groups	Pre-SMA	SMA	Pre-SMA	SMA	Totals
Total recorded	161	135	21	97	414
Reward expectation
L only	4	0	0	8	12
R only	8	1	2	2	13
Bidirectional	19	13	6	14	52
All reward expectation (%)	31 (19)	14 (10)	8 (38)	24 (25)	77 (19)
Error
L only	2	3	0	2	7
R only	8	5	0	1	14
Bidirectional	10	1	1	1	13
All error (%)	20 (12)	9 (7)	1 (5)	4 (4)	34 (8)
Reward
Bidirectional; all reward (%)	7 (4)	4 (3)	0 (0)	7 (7)	18 (4)

L, left; R, right; SMA, supplementary motor area.

Fig. 3.

Localization of recording sites. A: a model of the brain for monkey B constructed from MRI slices showing the pre-supplementary motor area (SMA) region highlighted in blue and the SMA region highlighted in green. Major sulci are outlined in yellow. The black box indicates the location of the recording chamber, and the yellow grid indicates the recording sites. CS, central sulcus; AR, arcuate sulcus; PR, principal sulcus. B: model of the brain showing chamber location for monkey E. C: a magnified version of the recording grid shown in A and B for monkey B (left) and monkey E (right). The horizontal black line indicates the pre-SMA/SMA border. The colored circles indicate the position of the evaluative neurons (error cells: red, reward cells: green, reward expectation cells: blue, surprise cells: pink). The circle size indicates the number of neurons (large: >4 cells, medium: 3–4 cells, small: 1–2 cells). Penetrations that yielded no evaluative neurons are indicated by gray dots. Each box is 0.5 mm × 0.5 mm. D: combined evaluative cell activity across monkeys is shown, together with movement-related activity (black dots).

Error-related activity.

When a stop signal appeared, and the monkeys were not able to inhibit their movement into the target box, an error occurred. The stop signal task allows for the identification of evaluative activity since the same behavior results in a different outcome, depending on the trial context (i.e., an error if a stop signal appeared or a success if no stop signal). The activity of many cells recorded in medial frontal cortex is directionally specific in nature. The neurons were, therefore, analyzed separately for each direction. All of the 34 error-related cells showed activity increases in response to an error (Fig. 4). When the directionality of the response was examined, 38% (13/34) of cells were found to exhibit error activity for movements to both directions (Fig. 4A). The remaining 62% (21/34) of cells were only active when an erroneous movement to a particular direction was made (Fig. 4B).

Fig. 4.

Error-related activity. A: spike density function for an error cell showing a response following both movements to the right (first column) and movements to the left (second column). Error trial activity is shown in green. Activity on no-stop signal trials is shown in black throughout the figure, while activity on noncanceled trials is shown in green. For all figures, the black box above each plot indicates the direction of movement toward the target. Results of t-tests performed on movement-related activity compared with baseline activity in 60-ms intervals are shown below each spike density function. Dotted line indicates P = 0.05 boundary. B: an example of unidirectional error cell activity showing an error response only for movements to the left. C: spike density function showing a transient burst of activity directly following the end of the movement. D: spike density function for an error cell showing tonic activity following an error. For both C and D, activity for movements to the right and left targets are combined. All plots are aligned on the time of movement end.

On average, the error-related activity arose 218 ms after the movement ended, with the majority of cells responding within the first 100 ms after movement end. The fastest error signals occurred even before the end of the movements (21%; 7/34 cells). Error cells showed a range of activity durations, from brief phasic bursts (Fig. 4C) to longer tonic activity (Fig. 4D). The average duration of the error-related activity was 402 ms. There were significantly (χ² test; P = 0.014) more error cells in the pre-SMA (23/158, 15%) than in the SMA (13/197, 7%).

Surprise-related activity.

On a subset of trials (5%), additional liquid reward was delivered during the intertrial interval. This type of “surprise” reward was delivered far from the normal time of reward delivery, but contained the same amount of liquid as a normal expected reward. The reward delivery was randomized and, therefore, unpredictable to the animals. We found a number of neurons that responded to the delivery of the unexpected reward. We will describe most of these cells in the section describing reward-related activity. However, a small set of cells (4/414 total, 1%) was specifically active for behavioral errors as well as unexpected reward. Since error cells, by our definition, should not respond to the appearance of extra juice, these cells were referred to as “surprise cells”, due to their response to any unexpected event, regardless of whether it resulted in a gain or loss of reward. An example of such a cell is shown in Fig. 5. Although we examined all cells for a 500-ms period following unexpected reward, all active cells showed significant activity within 150 ms, with a mean onset of 98 ms. When we compared the activity of these cells following reward delivery for unexpected rewards and rewards earned in the task, all showed significantly higher activity for the unexpected reward. None of these four cells responded significantly to reward.

Fig. 5.

Surprise-related activity. A: spike density function for a surprise cell comparing activity on noncanceled trials (green) and no-stop signal trials (black). Activity is aligned on the time of movement end, and movements to the right and left are combined. B: activity for this same cell aligned on receipt of reward for expected reward trials (black) and unexpected reward (magenta).

Reward expectation-related activity.

The error cells that we found in the pre-SMA and SMA reflected the realization that no reward would follow the inappropriate arm movement, i.e., a loss of reward. We also found anticipatory signals for a gain of reward in 19% (77/414) of our recorded cells (Fig. 6). These neurons showed increased activity in the postmovement period when a reward was expected, but before it was delivered. Reward expectation cells were found slightly more often in the pre-SMA (39/158; 25%) than in the SMA (38/197; 19%), but there was no significant difference in the frequency between the two areas (χ² test; P = 0.22).

Fig. 6.

Reward expectation cells. A: spike density functions for a reward expectation cell with a short burst of activity showing activity on no-stop signal trials in black and noncanceled trials in green. Activity is aligned on the time of movement end. Activity for movements to the right and left are combined. B: spike density functions for a reward expectation cell with sustained activity until reward delivery. C: spike density functions for a reward expectation cell showing a burst of activity before reward, and a second during the reward. D: reward expectation activity for the cell in A, on no-stop signal trials (black) and canceled trials (red). Activity is aligned on time the reward was received.

The reward expectation cells exhibited a range of different response profiles (Fig. 6, A–C). Some cells showed a transient burst of activity in the postmovement period before the reward was received (Fig. 6A), often declining by the time the actual reward was delivered. Others exhibited activity that built up more gradually through the entire trial and remained high up to reward delivery (Fig. 6B). Still others showed an initial burst in expectation of an impending reward, followed by a second burst following the actual delivery of the reward (Fig. 6C). This last type of cells fulfilled both the criteria for reward-expectation signals, as well as the criteria for reward-delivery signals (see below) and, therefore, represented both the expectation and the delivery of reward. The additional reward delivery response of these cells separates them from other reward expectation cells and was found in almost one-third (24/77; 31%) of the cells in our reward expectation group.

We next observed if reward expectation activity existed whether or not a movement was actually made to obtain reward. To do so, we compared activity on canceled trials (no movement made, reward received) and noncanceled trials (error). We found that the majority of cells (66/77; 86%) showed reward expectation activity in the prereward epoch of canceled trials in addition to the corresponding period on no-stop signal trials (Fig. 6D).

In the countermanding task, monkeys can receive reward in the context of three different motor commands: a leftward movement, rightward movement, or the lack of a movement. We examined the reward expectation cells' responses in these three different contexts to further examine the relationship between the action and the expectation of reinforcement for that action. The majority of reward expectation cells showed bilateral responses (52/77; 68%), independent of movement direction (Fig. 7A), while the rest (25/77; 32%) displayed activity specific to one direction (13 right, 12 left) (Fig. 7B).

Fig. 7.

Direction-dependency of reward expectation cells. A: spike density functions for a bidirectional reward expectation cell showing activity on no-stop signal trials in black and noncanceled trials in green. Activity is aligned on the time of movement end. The arrow indicates the time of reward delivery. B: spike density functions for a reward expectation cell with unidirectional activity.

Thus, for most cells, the reward expectation-related activity was not movement contingent. However, in a significant minority of reward expectation cells, the anticipatory signal reflected the type of movement that led to the reward (25/77; 32%). These neurons might be important for the learning and updating of the reward that is expected to follow from specific actions within a task context. This information could provide critical information to learn the value of task-specific behavioral responses.

In the majority of cells, the onset of the anticipatory response was <100 ms after the end of the movement, with a mean latency of 123 ms. Thus the onset of the reward expectation signals was closely tied to the initial moment of certainty regarding reward delivery and appeared around the same time as error-related signals on noncanceled trials. The average duration of the reward expectation signal was 508 ms.

Reward-related activity.

A third, smaller group of cells was found to be responsive exclusively to the reward delivery event itself. An example of such a cell can be seen in Fig. 8. Reward cells respond to the actual receipt of reinforcement; any cell active before the delivery of reward was not classified as a reward cell and was instead classified as a reward expectation cell. As noted before, some reward expectation cells were found to respond to reward delivery as well (see, for example, Fig. 6C). These neurons were not included in our reward cell group. We found 18 cells (18/414; 4%) that responded exclusively to reward, almost all of which (17/18, 94%) responded to reward for both leftward and rightward target-acquiring movements compared with the pretrial baseline (Fig. 8A). We found reward cells with equal frequency in the pre-SMA (7/158; 4%) and in the SMA (11/197; 6%), and there was no significant difference between the two areas (χ² test; P = 0.62). The locations of reward cells found are plotted with other functional types of cells in Fig. 3.

Fig. 8.

Reward cells. A: spike density function for a reward cell showing a response following the receipt of reward (black arrow) for both movements to the right (first column) and left (second column). Activity for no-stop signal trials (black) is compared with activity on noncanceled trials (green). Activity is aligned on the time of movement end. B: activity of the same cell as in A across rewarded trial types aligned on time reward was received. No-stop signal trial activity is shown in black, canceled trial activity is shown in red, and corrected trial activity is shown in blue. Activity for movements to the right and left are combined.

As pointed out in the case of reward expectation signals, in our version of the countermanding task, very different actions could result in the same reward. Although reward cells were identified based on their response on no-stop signal trials, we also examined the response of these cells on canceled and corrected trials. We found that the majority of reward cells showed responses, regardless of the type of trial that led to the reward (13/18, 72%). This is illustrated in Fig. 8B. This finding, in combination with the bidirectional nature of the reward cells, suggests a more general evaluative role for these cells.

As discussed in the case of surprise cells, for a small portion of the trials (∼5%), an unexpected reward was delivered during the intertrial interval. Most of the reward cells (12/16; 75%) that were tested with unexpected reward showed significant activity, both for unexpected reward and for expected reward (Fig. 9A). When we directly compared the responses to unexpected and expected reward, we found cells which preferred an expected reward (3/16; 19%; Fig. 9B). Other cells preferred an unexpected reward (4/16; 25%: Fig. 9C). These were the same cells that also did not show significant activity for unexpected reward. The preference for one type of reward or another might point to a more complex reinforcement signal representation in those cells, possibly including information about what state of the world is expected.

Fig. 9.

Reward cell responses to expected and unexpected reinforcement. A: spike density functions for trials for which expected reward was given (black) vs. when unexpected reward was given (gray), showing a cell that responds similarly to both types of reward. Activity is aligned on the delivery of reward. B: a cell that prefers expected reward. C: a cell that prefers unexpected reward.

Population activity of error, reward expectation, and reward cells.

The relatively large number of reward expectation cells found indicates that these cells play a key role in SMA and pre-SMA function. Despite the diversity in the response profiles of individual reward expectation cells, the average population activity of these cells (Fig. 10) demonstrates clear activity differences that develop around the end of the arm movement and that last during the entire reward latency period (the first 500 ms following the end of movement).

Fig. 10.

Population activity. A: population activity for error cells showing activity on no-stop signal trials in black for rightward movements and gray for leftward movements, and noncanceled trials in dark green for rightward movements and light green for leftward movements. Activity is aligned on the time of movement end. B: population activity for reward expectation cells. C: population activity for reward cells. Arrow indicates average time reward was received.

DISCUSSION

In this study, we used an arm movement version of the stop signal paradigm to search for performance monitoring signals in pre-SMA and SMA. We found a large number (138/414; 33%) of evaluative cells in SMA and pre-SMA, suggesting that these areas play an important role in the monitoring and executive control of skeletomotor movements. The vast majority of these pre-SMA and SMA cells represented three types of evaluative signals: error, reward expectation, and reward signals. This result is very similar to the findings in SEF using an occulomotor version of the stop signal task (Stuphorn et al. 2000). In addition, we found a smaller subset of cells that may signal any unexpected outcome, independent of its valence (surprise cells). This cell type has not been previously described in pre-SMA or SMA.

Comparison of pre-SMA and SMA.

There are marked differences in the connectivity pattern of pre-SMA and SMA (Luppino et al. 1993). The SMA has direct connections to the primary motor area and the spinal cord and is somatotopically organized in a caudal-to-rostral direction (Dum and Strick 1991; He et al. 1993; Luppino et al. 1993). Microstimulation evokes simple body movements (Luppino et al. 1991). In contrast, the pre-SMA does not contain a somatotopic representation of the body and does not connect to the primary motor cortex or the spinal cord. Instead it has strong connections with the dorsolateral prefrontal cortex (Geyer et al. 2000; Luppino et al. 1993). This anatomical difference suggests the pre-SMA might have a more cognitive role, while the SMA might have a more motor-related function.

Given this difference in functional anatomy, one might have expected to find a corresponding difference in the representation of evaluative signals. However, this was not the case. In general, we did not observe any systematic differences between pre-SMA and SMA, either in the overall frequency of neurons with evaluative signals (except in the case of the error cells) or in the relative distribution of specific types of monitoring signals. Thus these two areas seem to play a similar role in outcome evaluation.

This finding can be interpreted in a number of ways. One possibility is that the functional difference is not as severe as implied by the difference in connectivity. This interpretation would fit our previous finding that there are also almost no functional differences between pre-SMA and SMA during movement preparation and suppression (Scangos and Stuphorn 2010). It is also noteworthy that the medial frontal cortical areas that are activated in human imaging studies using a stop signal task typically encompass both pre-SMA and SMA (Aron and Poldrack 2006; Li et al. 2006; Whelan et al. 2012; Zhang and Li 2012).

Another possibility is that the two areas contain similar evaluative signals, but that they are used in different ways in accordance with the different functional role of the two areas in motor control. This interpretation is supported by the fact that a wide range of evaluative signals has been found in a large network of areas encompassing almost the entire frontal cortex, as well as other cortical and subcortical regions (Belova et al. 2007; Bermudez and Schultz 2010; Hayden et al. 2008; Kimura et al. 1996; Matsumoto and Hikosaka 2009b; Matsumoto et al. 2003; So and Stuphorn 2012; Stuphorn et al. 2000; Tremblay and Schultz 1999). Likewise, it was recently demonstrated that positive and negative evaluative signals can be found in nearly all cortical and subcortical structures of the human brain (Vickery et al. 2011). Given this evidence, it is less surprising to find evaluative signals in two neighboring medial frontal regions. Also, it is very likely that the numerous appetitive and aversive signals have differing roles in the various regions of the brain in which they are found.

These two possibilities are not necessarily mutually exclusive. For example, it is possible that pre-SMA and SMA have similar functional roles in the simple form of response inhibition tested with the present behavioral paradigm, but have additional differing roles with respect to other, yet untested, behavioral control processes.

Reward-related activity and orofacial movements.

The recording locations included a region between the arm movement-related neurons in the pre-SMA and SMA that is known to be involved in orofacial movements (Mitz and Wise 1987). It is, therefore, possible that some of the reward-related activity we recorded actually represents orofacial motor signals generated in anticipation of reward and during reward. Since we did not directly record orofacial responses, such as licking, we cannot rule out this possibility. However, it is unlikely to be correct for a number of reasons. First, the neurons recorded in this paper were recorded from an extensive section of medial frontal cortex spanning 6–9 mm. They were not concentrated between two clusters of arm movement-related neurons and were instead intermingled with arm movement activity. Second, the firing pattern of the reward expectation and reward-related cells is unlikely to result from licking movements. Licking behavior occurs before and after reward, but the reward cell activity identified only occurs after reward delivery. We also did not see a rhythmic activity pattern that would be expected from licking. It is, therefore, more likely that the neural activity we recorded represents evaluative activity rather than licking behavior.

Reward expectation signals.

Action selection depends on the representation of action-outcome associations that link particular behavioral responses to their expected consequences. Pre-SMA and the SMA both contain neurons that encode the motivation to perform specific movements (Roesch and Olson 2003; Scangos and Stuphorn 2010). The strength of this motivational signal reflects the amount of reward that is expected to follow from the action and, therefore, encodes an action-value signal. Similar findings have been obtained in the adjacent SEF with respect to saccades (So and Stuphorn 2010; Stuphorn et al. 2010). These action-value signals in the SMA and SEF could guide value-based action selection in the skeletomotor and oculomotor domain, respectively.

Thus we found that reward expectation activity in the pre-SMA and SMA during the movement was often accompanied by significant changes in activity in the postmovement period, when the monkey was waiting to receive reward. This finding is similar to the results of a recent study in pre-SMA and rostral cingulate motor area, which showed that the majority of cells reflecting expected reward in the postmovement period were also active during the movement period (Akkal et al. 2002). Such a response in the postmovement period is similar to previous reports of reward expectation activity in this region and many other regions of the cortex (Akkal et al. 2002; Campos et al. 2005; Mars et al. 2005; Roesch and Olson 2003; Stuphorn et al. 2000). This activity in the delay period represents a working memory signal of the expected outcome after the action has been made to compare it with the actual outcome (So and Stuphorn 2012). Such signals might ultimately support learning and adjustments in future strategy or action selection, but a direct link of outcome-related signals and future adjustments in behavior is still missing in the case of the stop signal task.

Only reward cells encoded the magnitude of the reward.

Our findings show that the evaluative cells across the medial frontal cortex varied in whether or not they reflected the magnitude of reward. The error and reward expectation cells that encoded the expectation of an outcome did not change their responses, depending on the size of the impending reward or the potential loss. In contrast, the reward cells, which encode the actual outcome, do reflect the magnitude of reward. This indicates that reward cells respond to the absolute value of the reinforcement signal, a property that is important for encoding the objective outcome of an action. Given two actions that both result in a positive outcome, an ideal decision-making center must know which resulted in the larger positive outcome. Hence it is important to have a precise input indicating the size of the outcome. In contrast, the reward expectation and error cells seem to encode the relative subjective outcome, i.e., whether it is better or worse than the alternative. This would be more useful for determining whether a mistake or correct action had been executed in a binary evaluative system.

These findings are somewhat in contrast to recent findings in the SEF using an oculomotor gambling task, in which the monkey had to choose between certain and uncertain reward options of varying amount (So and Stuphorn 2012). In this task, the monkey had to wait for the outcome of its choice following the saccade indicating its chosen option. During this delay period, some SEF neurons carried a signal that reflected the subjective expected value of the chosen option, similar to our findings in this study. However, in contrast to our present findings, this expected reward signal was parametrically modulated as a function of subjective value, instead of being modulated in an on-off fashion. Consequently, while the SEF reward expectation signal would be useful for computing reward prediction error-type signals (So and Stuphorn 2012), it is unlikely that the reward expectation signals that we found in this study could be used in this way, since they would not allow for the determination of the degree of mismatch between expected and actual reward.

However, there are important differences in task design between the gambling and the stop signal task that complicate the interpretation of these findings. In the case of the stop signal task, the monkey can be sure that he will receive a reward on correct go trials, since no stop signal was presented. In contrast, in the gambling task, the monkey is not certain whether he will end up with the better or worse outcome. Future experiments that vary the amount of reward gained vs. lost, or that ask the monkey to make choices based on its perceived expectation of reward, could further help elucidate the role of these cells. It might also be useful to test cells both in a stop signal task and in other tasks, such as the gambling task used by So and Stuphorn (2012).

Direction-specificity of evaluative signals.

Although most of the evaluative cells in pre-SMA and SMA were equally active for both leftward and rightward movements, we came across a few cells carrying reward expectation and error signals that responded preferentially for movements to one side or the other. It is important to note that this neuronal activity did not simply reflect motor-related activity evoked by some corrective or abortive response. Our selection criteria ensured that such a relationship could be ruled out. The neurons, therefore, carried a signal that was most likely evaluative in nature.

While the bidirectional nature of most cells would suggest a more abstract evaluative role, these unilaterally active cells seem to be tied more closely to the movement itself. One possible function of these action-dependent evaluation signals could be to help overcome the credit assignment problem (Sutton and Barto 1998). In reinforcement learning, it is poorly understood how an animal identifies which of the many actions it has performed before a particular reward or punishment is responsible for the positive or negative outcome. The pre-SMA and SMA neurons that carry action-dependent evaluation signals retain information about the specific action that produced the expectation to gain or lose reward. Specifically, the cells retain directional information in the error or reward expectation activity. These cells may provide a method for identifying and modifying specific actions. Interestingly, many of the outcome monitoring and evaluation signals found in the gambling task in SEF neurons also carried action-specific information (So and Stuphorn 2012). Similar to our findings here, this information was also implicitly encoded as a contextual modification of the evaluative signal and not explicitly in the form of a working memory signal.

Saliency signals.

Unexpectedly, we also found an additional group of neurons in pre-SMA and SMA that was numerically small, but carried a distinct signal. This group, called surprise cells, was active both for behavioral errors, and in response to unexpected reward. Thus these cells were active for a surprising event, whether positive (extra juice) or negative (a stop signal during movement). Cells with this type of property have not been found previously in SMA or pre-SMA. However, our laboratory recently described very similar salience-encoding neurons in the SEF (So and Stuphorn 2012). Thus surprise signals seem to be present throughout the medial frontal cortex of primates. A similar signal was also reported in the dopaminergic neurons in the midbrain of macaques (Matsumoto and Hikosaka 2009a), the amygdala of macaques (Belova et al. 2007) and rats (Roesch et al. 2010b), as well as in the basal forebrain of rats (Lin and Nicolelis 2008).

The surprise signal may represent an attentional signal cueing the animal that a behaviorally salient event occurred, corresponding to a evaluative signal that has been suggested in animal learning theory (Mackintosh 1975; Pearce and Hall, 1980; Roesch et al. 2010a). In this model, the salience signal controls the amount of attention that is paid to a task event and thus, indirectly, the amount of learning (Pearce and Hall 1980). Signals with the same activity pattern, but a slightly different functional role, have also been proposed in a recent model of medial frontal cortex as an action-outcome predictor (Alexander and Brown 2011). The number of surprise cells was small in our study, and we include them primarily as an observational finding of interest. Further studies will be necessary to explore the importance of these cells to the medial frontal cortex.

Conclusion.

In summary, we have found a number of cells in the SMA and pre-SMA that carry positive and negative evaluation signals. We found three types of cells that encode the expected reward outcome: the actual outcome, and behavioral errors that result in a mismatch of expected and actual outcome. In general, these findings match very well with similar results in SEF, a neighboring oculomotor area. The kinematics and dynamics of eye and arm movements vary greatly and require very different neuronal control systems (De Jong et al. 1990; Robinson 1981). The strong similarity in the evaluative signals found in pre-SMA, SMA, and SEF is, therefore, quite remarkable and suggests that the evaluation of action outcomes is organized along similar functional principles across the medial frontal cortex, independent of the motor system that is monitored. These cells likely link chosen actions with their consequences, to allow for the adjustment of future decision making.

GRANTS

This work was supported by the National Eye Institute through Grant R01-EY019039 to V. Stuphorn.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the author(s).

AUTHOR CONTRIBUTIONS

Author contributions: K.W.S. and V.S. conception and design of research; K.W.S. and R.A. performed experiments; K.W.S. and R.A. analyzed data; K.W.S. and V.S. interpreted results of experiments; K.W.S. and R.A. prepared figures; K.W.S., R.A., and V.S. drafted manuscript; K.W.S. and V.S. edited and revised manuscript; K.W.S. and V.S. approved final version of manuscript.