Neural Correlates of Knowledge: Stable Representation of Stimulus Associations across Variations in Behavioral Performance

Summary

Behavioral responses to a sensory stimulus are often guided by associative memories. These associations remain intact even when other factors determine behavior. The substrates of associative memory should therefore be identifiable by neuronal responses that are independent of behavioral choices. We tested this hypothesis using a paired-associates task in which monkeys learned arbitrary associations between pairs of visual stimuli. We examined the activity of neurons in inferior temporal cortex as the animals prepared to choose a remembered stimulus from a visual display. The activity of some neurons (22%) depended on the monkey’s behavioral choice; but for a novel class of neurons (54%), activity reflected the stimulus that the monkey was instructed to choose, regardless of the behavioral response. These neurons appear to represent memorized stimulus associations that are stable across variations in behavioral performance. In addition, many neurons (74%) were modulated by the spatial arrangement of the stimuli in the display.

Introduction

In many cases, one’s failure to make a “correct” behavioral choice does not imply lack of knowledge. For example, one might see dark clouds on the horizon but nonetheless go out without an umbrella because of a distracting conversation or lack of concern about getting wet. Of course, the established association between dark clouds and rain remains intact even when the umbrella is left behind. The divide between knowledge and choice suggests that it may be possible to access a stable neuronal representation of knowledge that is independent of behavior. By “knowledge” we mean a demonstrably accessible memory of a learned association.

This hypothesis can be tested experimentally using tasks that involve behavioral choices that are guided by memory. One such paradigm is the visual paired-association (PA) task, in which subjects are trained to memorize associations between pairs of visual stimuli. On each behavioral trial, subjects are initially presented with a cue stimulus (e.g., a rain cloud). The cue is followed by presentation of two choice stimuli, from which the subject must identify the stimulus that has been paired with the cue (e.g., an umbrella). To perform correctly the subject must access their knowledge of the stimulus pairing at some time between when the cue is presented and the choice is made.

Using the PA task, previous studies have shown that some neurons in the primate inferior temporal (IT) cortex (Sakai and Miyashita, 1991) and prefrontal cortex (Rainer et al., 1999) exhibit a signal for the paired stimulus after the cue has been presented. This signal clearly conveys information that can enable correct performance. However, because the neuronal signals in these earlier studies were evaluated only on correct trials, for which knowledge and behavior are correlated, it is unclear whether these neurons represent memory of the learned association or the impending behavioral choice.

To distinguish between these possibilities, we recorded neuronal activity in IT cortex as rhesus monkeys performed a visual PA task in which new stimulus pairings were learned by trial and error each day. In this learning variant of the PA task, error trials were more common than they are with fixed, overtrained associations. These errors allowed us to assess whether there were neuronal responses that depended upon which choice stimulus was correct on each trial, apart from which stimulus was selected.

We found that the responses of a small group of neurons depended upon the stimulus chosen by the monkey on each behavioral trial. The activity of these neurons was predictive of the monkey’s behavior and, as was suggested by Chelazzi et al. (1993, 1998), may be a neural correlate of attentional selection. Particularly interesting, however, was a substantial population of neurons for which responses reflected the correct pairing of cue and associated stimulus, regardless of which stimulus the monkey chose. These neurons appear to represent stable knowledge of the learned association, which is one of many factors that govern behavior on a given trial.

Results

General

Two rhesus monkeys (Macaca mulatta) were trained to perform a visual PA task, the details of which are described in the Experimental Procedures. Briefly, each trial began with the appearance of a central fixation target (Figure 1A). Once gaze was directed at this target, the animal was presented with a visual cue stimulus, which determined which of two subsequently presented visual stimuli was to be chosen in order to receive a juice reward. The monkey learned the appropriate pairings of cue and choice stimulus by trial and error. Both animals trained extensively on this task before the electrophysiological recording began so that they were able to learn two new stimulus pairings in one behavioral session. We refer to the four new stimuli used generically as A, B, C, and D. Stimulus A was always paired with B, and C was always paired with D. Depending on which of the four stimuli was the cue stimulus on a given trial, the monkeys either chose between stimuli A and C or between stimuli B and D.

Figure 1. Depiction of Behavioral Task and Experimental Design.

(A) Paired-associates learning task. The boxes show the sequence of events in a behavioral trial, and the interconnecting arrows show different possible sequences. Trials began when the monkey fixated a central target (+ sign) after which a randomly selected cue stimulus was presented. The cue was followed immediately by the decision phase, during which two stimuli (the choice array) were presented in one of two configurations in the visual hemifield contralateral to the neuronal recordings. When the fixation target was extinguished at the end of the 500 ms decision phase, the monkey chose one of the choice array stimuli by looking at it (small arrows). Each cue was paired with a choice array stimulus. The cue thus specified which stimulus should be chosen, i.e., the instructed stimulus. In the example, the ball and airplane were paired so that presentation of the ball as cue instructed selection of the airplane. Likewise, the dinosaur and corn were paired so that presentation of the dinosaur instructed selection of the corn. On other trials (not shown), the airplane or corn was the cue and the choice array consisted of the ball and the dinosaur.

(B) Three factors affecting neuronal activity. The effects of instructed stimulus, chosen stimulus, and stimulus arrangement were assessed. The instructed stimulus is indicated by the burst patterns, which surround the airplane in the top row and the corn in the bottom row. The stimulus chosen by the monkey is indicated by the arrows, which point to the airplane in the left column and to the corn in the right column. Stimulus arrangement refers to the two configurations of the choice array stimuli (namely, airplane on the bottom and airplane on the top), shown in each panel of the figure. The panels along the major diagonal illustrate cases in which the monkeys correctly chose the instructed stimulus.

Behavioral Results

We have previously detailed the behavior of the animals as they performed the PA task (Messinger et al., 2001). For each new set of stimulus pairings, performance was initially at chance level (as expected) and improved modestly over the course of the session as trial-and-error learning proceeded. The duration of the 61 neurophysiological recording sessions varied considerably (480 ± 32 trials [mean ± SEM]; range, 125–1274). We therefore divided each session into quartiles, each with a fourth of the trials. Average performance was essentially at chance in the first quartile (50.8% ± 0.9% correct), but was significantly better than chance (Student’s t test, p < 0.001, n = 61) in each of the subsequent quartiles (53.4%, 55.6%, and 55.4% ± 0.9% correct, respectively). Analysis of variance revealed that performance increased significantly across quartiles (p < 0.001), but did not differ between the two monkeys nor did it exhibit an interaction between quartile and subject (two-way ANOVA). Thus, both animals acquired reliable knowledge of the newly assigned stimulus pairings at an early stage of the recording sessions.

The monkeys responded by making a saccadic eye movement to one of two visual choice stimuli any time after the fixation target was extinguished. Reaction times (RTs) were measured from the disappearance of the fixation target (500 ms after the appearance of the choice stimuli) until saccade initiation. The mean RT across the 61 recording sessions was 140 ± 9 ms. A two-way ANOVA revealed that one monkey’s RT (165 ms, n = 26) was significantly (p < 0.001) slower than the other’s (121 ms, n = 35), but there was neither a significant decrease across quartiles (148, 148, 142, and 131 ms, respectively) nor an interaction between monkey and quartile.

Electrophysiological Results

General Features

In a previous report, we described learning-dependent changes in neuronal responses to the cue stimuli, which developed as animals performed the PA task (Messinger et al., 2001). In the present study, we focused on neuronal activity occurring during the decision phase of each trial, i.e., when the two choice stimuli were on the video display and the monkeys were still required to maintain central fixation (see Figure 1A). We recorded such activity from 94 neuronal units in the inferior temporal (IT) cortices of the two monkeys (41 in one monkey and 53 in the other). During the period from 100 to 500 ms following the appearance of the choice stimuli, the average firing rate of these 94 units across all trial types was 11.4 ± 1.0 spikes/second (sp/s). The average spontaneous rate during the initial period of fixation was 8.0 ± 0.7 sp/s (n = 94).

Experimental Conditions

We investigated the relationship between neuronal activity and three key factors that characterized the choice stimuli—hereafter termed the “choice array.” Figure 1B provides a conceptual illustration of the different experimental conditions defined by these three key factors. The first factor consisted of the stimulus that the animal was required to choose in order to receive a reward. We refer to this as the instructed stimulus, the identity of which was defined on each trial by the preceding cue and the preassigned pairing. The second factor was a function of the monkey’s behavior and consisted of the stimulus that was selected from the choice array. We refer to this as the chosen stimulus. The third factor was the stimulus arrangement. Thus, for example, a choice array containing stimuli A and C could appear in one of two possible spatial arrangements on each trial: A above C, or vice-versa. If a subject’s behavior was informed by knowledge of the correct stimulus pairing, the instructed and chosen stimuli should be identical. Because the subjects’ decisions were not always so informed, we were able to evaluate independently the influence of choice and instruction on neuronal activity.

We evaluated the degree to which each factor accounted for trial-to-trial differences in firing rate during the decision phase of the trial (Figure 1A) using two separate three-way ANOVAs (see Table 1). One ANOVA was for analysis of those trials in which the choice array consisted of stimuli A and C (the AC choice array) and the other was for those trials in which the choice array consisted of stimuli B and D (the BD choice array). Most of the recorded neurons (84%) responded significantly differently to the two choice arrays (Student’s t test, p ≤ 0.05).

Table 1.

Number of Choice Arrays Exhibiting Each Main Effect

	Number of Choice Arrays Exhibiting a Main Effect of:
	Instructed Stimulus	Chosen Stimulus	Stimulus Arrangement
One main effect	23	4	56
Two or three main effects:	43	17	50
Instructed stimulus	–	8	41
Chosen stimulus	8	–	15
Stimulus arrangement	41	15	–
No interactions	45	12	75
One or more interactions:	21	9	31
IS × CS	6	4	9
IS × SA	10	1	12
CS × SA	8	4	17
IS × CS × SA	3	0	2
Significant choice arrays	66	21	106

Entries show how many choice arrays exhibiting each main effect (columns) did so in conjunction with any other main effects (upper half of table) or interaction effects (lower half of the table). Arrays were counted in more than one column if they exhibited more than one main effect. All the upper indented entries reflect six arrays that exhibited all three main effects. For the lower indented tallies, each array was counted once per interaction effect. Not counted are 53 choice arrays that exhibited no main effects, ten of which exhibited interaction effects.

Selectivity for Choice Stimuli

In conjunction with our main analyses, we examined the neuronal responses elicited by the two stimuli in each choice array when presented alone. The mean firing rate response to an individual stimulus was computed from 100 to 500 ms after its presentation as the cue stimulus. Following Chelazzi et al. (1993), we refer to the choice array stimulus that elicited the greater firing rate when presented alone as the good stimulus and the one that elicited the lesser response as the poor stimulus. Neurons responded selectively (Student’s t test, p < 0.05) to the good versus poor stimulus in 145 (77%) of the 188 choice arrays.

Dependence of Neuronal Activity on Instructed Stimulus

More than half of the neurons recorded (54%, 51/94) responded to at least one choice array significantly differently depending on which stimulus the monkey had been instructed to choose on that trial (three-way ANOVA, main effect of instructed stimulus, p ≤ 0.05). Fifteen neurons exhibited this dependence for both choice arrays (see Table 2). Data from one such neuron is shown in Figure 2A and in Figure S1 in the Supplemental Data available with this article online. The four panels show neuronal activity in the form of peristimulus time histograms (PSTHs) averaged across trials that differ in terms of the stimulus that was instructed by the preceding cue (rows) and the stimulus that was chosen by the monkey (columns). Thus, for example, the upper left panel presents data collected on trials in which stimulus B was instructed and correctly chosen. The upper right panel, in contrast, presents data collected on trials in which stimulus B was instructed but the monkey incorrectly chose stimulus D.

Table 2.

Significant Effects Exhibited in Response to the AC and BD Choice Arrays

	Response to One Choice Array Exhibited a Main Effect of:
…and Response to the Other Choice Array Exhibited:	Instructed Stimulus	Chosen Stimulus	Stimulus Arrangement
No main effect	12	7	21
One or more main effects:	54	14	85
Instructed stimulus	30	7	39
Chosen stimulus	7	0	9
Stimulus arrangement	39	9	72
No interactions	47	18	73
One or more interactions:	19	3	33
IS × CS	7	1	9
IS × SA	8	0	13
CS × SA	9	3	17
IS × CS × SA	1	0	0
Significant choice arrays	66	21	106

Each main effect exhibited in response to one choice array was tallied according to whether the same neuron’s response to the other choice array exhibited any main effects (upper half of table) or interaction effects (lower half of table). For the indented values, each effect exhibited by an array was counted separately. When a neuron exhibited the same main effect for both arrays (15 neurons for the Instructed Stimulus effect and 36 neurons for the Stimulus Arrangement effect), the corresponding column reflects the counts for each array. These neurons therefore contributed twice to the totals on the major diagonal of the upper indented matrix. Not counted are the effects exhibited by 32 of the 53 choice arrays whose partner array exhibited no main effect.

Figure 2. Neuronal Activity Dependent on the Instructed Stimulus.

As in Figure 1B, the four panels differ in terms of the instructed stimulus (rows) and chosen stimulus (columns). For clarity, each PSTH shows the average response to the two spatial arrangements of the choice array stimuli. The end of the cue presentation (time < 0), the decision phase (0–500 ms, shaded) and part of the behavioral response period are depicted. Activity from 100–500 ms (dashed lines) was used to calculate the mean firing rate. PSTH color indicates which stimulus was instructed (indicated by burst pattern in the icons). To facilitate comparison, the black PSTHs are reproduced as dotted curves on the same axes as the pink PSTHs.

(A) Responses of an area TE neuron that exhibited the instructed stimulus effect. Activity was 4.7 sp/s higher when stimulus D was instructed (pink curves, bottom row) than when stimulus B was instructed (black curves, top row). This neuron’s firing rate was thus a function of whether stimulus B or D was the correct answer (top versus bottom row) and not whether the monkey ultimately chose stimulus B or D (left versus right column). For the BD choice array, performance was significantly better than chance in the fourth quartile (binomial probability < 0.002) as well as over the session as a whole (p < 0.02).

(B) Average neuronal response to 61 choice arrays exhibiting a main effect of instructed stimulus. The average excludes five cases with a delay period between the cue and the choice array. The four panels use the same plotting conventions adopted in (A) except that stimuli (having been averaged over many different particular images) are identified generically by whether their instruction elicited a greater or lesser neuronal response to the choice array. Neuronal responses to these choice arrays were significantly greater (by 3.8 sp/s, on average) when one of the choice array stimuli (the preferred instructed stimulus, pink curves) was the correct answer than when the other stimulus (the nonpreferred instructed stimulus, solid and dashed black curves) was correct.

The firing rate of the area TE neuron shown in Figure 2A was significantly (three-way ANOVA, p < 0.001) greater when the monkey was instructed to choose stimulus D (bottom row, pink curves) than when the monkey was instructed to choose stimulus B (top row, black curves). This dependence was quantified by the Instructed Stimulus (IS) contrast index. (All three contrast indices defined in the Experimental Procedures range in value from −1 to +1. Index magnitude quantifies the strength of the effect, and index sign identifies which of two arbitrarily designated conditions yielded the larger response.) The IS index was −0.29, indicating a nearly 2-fold difference in response to instruction of stimulus B and D (5.6 versus 10.3 sp/s, respectively). In contrast, neuronal responses to this choice array were not significantly influenced by the monkey’s choice of stimulus (left versus right columns), which was quantified by the Chosen Stimulus (CS) contrast index (CS = −0.09). Activity was thus a function of which stimulus the monkey should have chosen and not which stimulus was actually chosen.

Figure 2B shows the average neuronal responses to those choice arrays that elicited significantly (three-way ANOVA) greater responses when one stimulus was instructed (the preferred instructed stimulus, bottom row) than when the other stimulus was instructed (the nonpreferred instructed stimulus, top row). The mean difference in firing rate for these two instructions was 3.8 ± 0.4 sp/s and the average magnitude of the IS index was 0.20 ± 0.02 (n = 66). Responses to these choice arrays depended (Student’s t test) neither on the arrangement of the nonpreferred and preferred instructed stimuli in the choice array (not shown) nor on which of these stimuli was chosen (left versus right column). Thus, neurons with a main effect of instructed stimulus reliably reflected the demands of the task, rather than the monkeys’ less reliable implementation of those demands.

A significant fraction (62%, 41/66; binomial probability < 0.05) of the choice arrays exhibiting the instructed stimulus effect were more active when the poor (i.e., less effective) stimulus was instructed. The average response to the choice arrays was significantly greater when the instructed stimulus was poor than when it was good (15.1 versus 13.9 sp/s, paired Student’s t test, p < 0.05, n = 66). Twelve (80%) of the 15 neurons that exhibited the instructed stimulus for both choice arrays had consistent preferences for the two arrays, with eight neurons preferring instruction of the poor stimulus in both arrays (see Figure S1) and four preferring instruction of the good stimulus in both arrays.

Responses to the majority (38/66, 58%) of the choice arrays exhibiting a main effect of instructed stimulus were greater when the choice array stimulus paired with the more effective cue stimulus was instructed. Of the neurons that exhibited the instructed stimulus effect for both choice arrays, the majority (8/15) preferred instruction of the paired-associate of the more effective cue stimulus for both arrays. The remaining cells had the opposite preference for both arrays (n = 4) or mixed preferences (n = 3). The tendency for the preferred instruction stimulus and the more effective cue stimulus to be paired is reflected by the cue response (time < 0) in Figure 2B, which was 2.6 sp/s higher when the preferred stimulus was instructed (bottom row) than when the nonpreferred stimulus was instructed (top row). This difference in cue activity was not significant (16.1 versus 13.5 sp/s, paired Student’s t test) but the difference in responses to the subsequent choice array was. The top and bottom curves became significantly different starting 210 ms after choice array onset (see the Experimental Procedures) and maximally different (5.6 sp/s) after 295 ms.

Some IT neurons have been found to exhibit a sustained response to the preceding stimulus during a blank delay in a visual memory task (Miyashita and Chang, 1988), which was abolished when another stimulus was presented (Miller et al., 1996). If some IT neurons in the present study continued to respond to the preceding cue even after the choice array was presented, this sustained response may have contributed to the instructed stimulus effect. Such sustained responses would tend to strengthen the instructed stimulus effect for the 38 choice arrays in which cue activity was greater before the preferred response to the choice array. However, sustained cue responses would diminish the instructed stimulus effect for the 28 choice arrays in which there was less cue activity before the preferred choice array response (see Figures 2A, S1, and S2). There was no significant difference in the strength of the instructed stimulus effect (preferred – nonpreferred instructed stimulus) for these two groups (4.3 versus 3.0 sp/s; Student’s t test, n.s.). Furthermore, the instructed stimulus effect was observed in cases in which the preceding cues would be expected to have minimal influence such as when responses to these cues were nonselective (see Figure S3) and when a delay period (500–700 ms) was added between cue offset and choice array onset. Together, these findings show that the instructed stimulus effect was not simply due to spillover of the cue response into the subsequent epoch.

In summary, many IT neurons responded differently to a choice array containing two visual stimuli depending on which stimulus was the correct one to pick on that trial, regardless of whether this action was carried out.

Dependence of Neuronal Activity on Chosen Stimulus

For some neurons (22%, 21/94), responses to either the AC or the BD choice array (but not to both) depended significantly on which stimulus was chosen at the end of the decision phase (three-way ANOVA, main effect of chosen stimulus, p ≤ 0.05). Data from a neuron exhibiting this effect for the AC choice array are shown in Figure 3A. Activity during the decision phase was correlated with the animal’s stimulus choice. The firing rate was significantly (three-way ANOVA, p < 0.02, CS = −0.05) greater when the monkey subsequently chose stimulus C (32.5 sp/s, right column)—the preferred chosen stimulus—than when the monkey chose stimulus A (29.5 sp/s, left column). This neuron’s response was therefore predictive of which stimulus the monkey would subsequently choose, regardless of whether the choice was correct or incorrect.

Figure 3. Neuronal Activity Dependent on the Chosen Stimulus.

Plotting conventions are the same as for Figure 2, except that PSTH color indicates which stimulus was chosen (indicated by arrows in the icons).

(A) Responses of an area TE neuron that exhibited the chosen stimulus effect. Activity was 3.0 sp/s higher when stimulus C was chosen (right column, pink curves) than when stimulus A was chosen (left column, black curves). Activity was greater whether stimulus C was chosen correctly (bottom right) or incorrectly (top right). As indicated by the dashed horizontal lines, the end of the response to cue stimulus B (top row, left of shading) was significantly (p < 0.001, 24.1 versus 16.0 sp/s) greater than to cue stimulus D (bottom row, left of shading). For the AC choice array, performance was significantly better than chance in the third and fourth quartiles (binomial probability < 0.05) and over the entire session (p < 0.005). Responses to the BD choice array are presented in Figure S4.

(B) Average neuronal response to 17 choice arrays that exhibited a main effect of chosen stimulus. The average excludes four cases with a delay period between the cue and the choice array. Neuronal responses were an average of 1.7 sp/s greater when one of the choice array stimuli (the preferred chosen stimulus, pink curves) was chosen than when the other stimulus (the nonpreferred chosen stimulus, solid and dashed black curves) was chosen. Activity began to reflect which of these stimuli would be chosen ~135 ms after choice array onset (or ~500 ms before saccade initiation). As in (A), the dashed horizontal lines show that the average response to the cue paired with the nonpreferred chosen stimulus (top row, left of shading) was significantly greater than the response to the cue paired with the preferred chosen stimulus (bottom row, left of shading).

Figure 3B presents neuronal responses averaged across those choice arrays that elicited significantly (three-way ANOVA) greater activity when one stimulus was chosen (the preferred chosen stimulus, right column) than when the other stimulus was chosen (the nonpreferred chosen stimulus, left column). For those cells exhibiting the chosen stimulus effect, the magnitude of the response difference yielded by the two choice behaviors averaged 1.7 ± 0.2 sp/s and the magnitude of the CS index was 0.14 ± 0.03 (n = 21). The activity of these neurons was not significantly (Student’s t test) dependent on whether the preferred chosen stimulus was located in the top or bottom position of the choice array (data not shown) or on whether this stimulus was instructed (bottom versus top row).

For the majority (62%, 13/21) of neurons exhibiting a main effect of the chosen stimulus, the preferred chosen stimulus was the one that elicited the greater response when presented alone, i.e., the good stimulus. The tendency for activity to be higher before saccades to a good stimulus supports the findings of Chelazzi et al. (1993) (see the Discussion). Neurons exhibiting such an effect may have been encoding the stimulus that the monkey was planning to look at or a covert shift of spatial attention to the subsequently selected stimulus.

Surprisingly, the identity of the preferred chosen stimulus also depended on neuronal selectivity for the two stimuli that were paired with the choice array stimuli. Specifically, the choice array stimulus whose paired-associate elicited the weaker response as cue was the preferred chosen stimulus in 17 out of 21 (81%) neurons. This pattern suggests that either the identity of the preferred chosen stimulus or the response to its paired-associate was influenced by the learned stimulus pairings. The relation between the preferred chosen stimulus and its paired-associate can be seen from the end of the cue period (time < 0) in Figure 3B. The bottom row shows when the preferred chosen stimulus was instructed and, hence, the cue was the paired-associate of the preferred chosen stimulus. Likewise, the top row shows the average response to the cue paired with the nonpreferred chosen stimulus. The dashed horizontal lines show that the paired-associate of the preferred chosen stimulus elicited significantly less activity than the paired-associate of the nonpreferred chosen stimulus (6.7 versus 11.8 sp/s, paired Student’s t test, p < 0.002, n = 21). Conversely, activity during the decision phase of the trial was significantly greater (10.8 versus 9.8 sp/s, paired Student’s t test, p < 0.05, n = 21) when the monkey chose the stimulus whose paired-associate elicited the weaker response. Thus, activity during the decision phase of the trial not only was predictive of which stimulus the monkey would subsequently pick but also reflected the neuron’s selectivity for the stimulus associated with the chosen stimulus.

We considered the possibility that the chosen stimulus effect was attributable to differences in the time taken to initiate a saccade to the two stimuli in each array. However, there was no difference in the average reaction time associated with selecting the preferred and nonpreferred chosen stimuli for those arrays exhibiting the chosen stimulus effect (140 ms, n = 21). For ten of these arrays, one stimulus was chosen significantly (Student’s t test, p ≤ 0.05) faster than the other (mean reaction times of 111 and 132 ms, respectively). Even in these cases, selection of the preferred and nonpreferred chosen stimuli had essentially the same average reaction time (121 and 122 ms, respectively, n = 10). Therefore, reaction time does not appear to be a causal factor in determining why responses to the choice array were dependent on the monkey’s choice. This is not surprising given that neuronal activity began to reflect which stimulus would be chosen ~135 ms after onset of the choice array (Figure 3B), which was long (~500 ms) before saccade initiation.

In summary, the responses of a subset of IT neurons depended upon which of two simultaneously presented visual stimuli would be the target of an upcoming eye movement. These neurons tended to be more active when the monkeys selected an effective, i.e., good, stimulus. Their activity was also greater when the chosen stimulus was paired with a stimulus that elicited a relatively weak response. The activity of these cells was, therefore, not only predictive of the monkey’s decision but also conveyed information about the stimulus pairings.

Dependence of Neuronal Activity on Arrangement of Choice Stimuli

Almost three-quarters (70/94) of the IT neurons responded significantly differently to one or both choice arrays when the two choice stimuli swapped locations (three-way ANOVA, main effect of stimulus arrangement, p ≤0.05). Data from an area TE neuron exhibiting the stimulus arrangement effect for the AC choice array are shown in Figure 4A. Data collected for the two arrangements of A and C are shown by the two PSTH curves in each of the panels. As can be seen from any of the panels, the firing rate was significantly (three-way ANOVA, p < 0.001) greater when stimulus A was in the lower contralateral quadrant and C was in the upper contralateral quadrant (pink curves), than when their positions were reversed (black curves). The Stimulus Arrangement (SA) index for this neuron was 0.32, indicating a nearly 2-fold difference in response to the two stimulus arrangements (11.4 versus 5.9 sp/s).

Figure 4. Neuronal Activity Dependent on Stimulus Arrangement Plotting conventions are the same as for Figure 2, except that responses to the two spatial arrangements of the choice array are shown by the two color-coded PSTHs in each panel (see icons at right).

(A) Responses of an area TE neuron that exhibited the stimulus arrangement effect. The response of this neuron was 5.5 sp/s greater when stimulus A was below stimulus C (pink curves) than when the same stimuli were in the opposite configuration (black curves). For the AC choice array, performance was significantly better than chance in the fourth quartile (binomial probability < 0.02) and over the entire session (p < 0.05).

(B) Average neuronal responses to 98 choice arrays that exhibited a main effect of stimulus arrangement. The average excludes eight cases with a delay period between the cue and the choice array. Responses to the preferred stimulus arrangement (pink curve) were on average 5.0 sp/s greater than those to the nonpreferred stimulus arrangement (black curve).

Figure 4B presents averaged neuronal responses to those choice arrays for which the two stimulus arrangements elicited significantly different responses (three-way ANOVA). Responses elicited by the two arrangements differed by an average of 5.0 ± 0.4 sp/s, and the average magnitude of the SA index was 0.26 ± 0.02 (n = 106). Average responses to the preferred (pink curve) and nonpreferred (black curve) stimulus arrangements first diverged 80 ms after choice array onset and became significantly different 150 ms after choice array onset. The similarity of the four panels of Figure 4B reveals that cells exhibiting the stimulus arrangement effect were not significantly (Student’s t test) dependent on the location of either the instructed or the chosen stimulus within the array.

We wondered whether the stimulus arrangement effect could be attributed to neurons with receptive fields (RFs) that covered only part of the choice array or, equivalently, were more responsive in one region of the RF than in another. For such a neuron, swapping the positions of the choice array stimuli would amount to changing which stimulus was in the RF. We did not routinely map RF boundaries with precision because previous studies indicate that the RFs of most IT neurons are large enough to have encompassed both choice array stimuli (Desimone and Gross, 1979; Op De Beeck and Vogels, 2000) and that these neurons are equally responsive to a single stimulus at the two locations occupied by our choice array stimuli (Chelazzi et al., 1998; Missal et al., 1999). In addition, we found that the preferred location (i.e., upper or lower contralateral quadrant) of the good stimulus differed between the AC and BD choice arrays for half (47/94) of the recorded neurons and about half (15/36) of those neurons that exhibited a significant effect of stimulus arrangement for both choice arrays (Table 2). Thus, for at least half of the population, RF size or nonuniformity cannot account for the stimulus arrangement effect. More generally, an explanation in terms of RF size is not consistent with the apparent independence of the preferred location of the good stimulus between one choice array and the next.

In summary, the activity of 70 out of 94 (74%) IT neurons was significantly dependent on the spatial arrangement of two visual stimuli presented simultaneously in the contralateral hemifield.

Interaction Effects

We frequently observed significant interactions between the three main factors (stimulus arrangement, chosen stimulus, and instructed stimulus) for one or both of the choice arrays tested (three-way ANOVA, p ≤ 0.05, Table 1). Most common was the interaction between chosen stimulus and stimulus arrangement, which was significant for one or both choice arrays in 28% (26/94) of IT neurons. For 73% (22/30) of the choice arrays exhibiting this interaction, neuronal responses depended significantly on whether the monkey chose the upper or lower choice array stimulus (Student’s t test, p ≤ 0.05), regardless of which stimulus this was. Thus, these neurons were predictive of where the monkey would look, as opposed to what stimulus the monkey would choose.

Nineteen percent (18/94) of IT neurons exhibited a significant interaction between the factors of instructed stimulus and stimulus arrangement in response to at least one of the choice arrays. For 58% (11/19) of the choice arrays exhibiting this interaction, there was a significant difference in response to the choice array when the instructed stimulus was in the upper versus the lower array position (Student’s t test, p ≤ 0.05), regardless of which stimulus was instructed. The activity of these neurons thus depended on where the monkey should have looked.

For 19% percent (18/94) of IT neurons, there was a significant interaction between the factors of instructed stimulus and chosen stimulus, indicating that activity was a function of both which stimulus the monkey was instructed to look at and whether the monkey successfully executed this instruction. For nearly half (8/18) of the choice arrays that exhibited this interaction, neuronal activity before correct choices was significantly different (greater in all but one case) than it was before error choices (Student’s t test, p ≤0.05). This activity might reflect the certainty of the monkey’s choice or, equivalently, its expectation of reward.

Comparison of Variables Influencing Neuronal Activity

The number of neurons exhibiting the main effects of instructed stimulus (54%), chosen stimulus (22%), and stimulus arrangement (74%) differed significantly (χ², df = 2, p < 0.001). ANOVAs are more powerful when they are balanced. We therefore also determined the proportions of significant main effects in the subset of neurons for which each factor was nearly balanced (i.e., neither level of a factor accounted for more than 55% of the AC or the BD choice array trials). The incidence of the main effects, though slightly higher (0%–3%) for these well-balanced neurons, remained significantly different (χ², df = 2, p < 0.001).

The variables of instructed stimulus, chosen stimulus, and stimulus arrangement modulated firing rates to significantly different degrees for both those choice arrays that exhibited each main effect and across all choice arrays (one-way ANOVA, p < 0.001). For those choice arrays exhibiting a significant main effect, post hoc (Tukey) tests revealed that rates were significantly more modulated by the arrangement of the stimuli (5.0 sp/s, p < 0.001) and by which stimulus was instructed (3.8 sp/s, p < 0.05) than by the chosen stimulus (1.7 sp/s). Across all 188 choice arrays, the influence of stimulus arrangement on activity (3.3 ± 0.3 sp/s) was significantly greater than that of both cognitive factors (p < 0.001). The influence of which stimulus was instructed (1.9 ± 0.2 sp/s) was furthermore significantly (p < 0.05) greater than that of which was chosen (1.2 ± 0.1 sp/s). Thus, the responses of IT neurons during the decision phase of the trial represented which stimulus should be chosen in greater number and with greater distinction than they represented which stimulus would be chosen.

Comparison of Effects in Area TE versus Perirhinal Cortex

Our recording sites were located in area TE (53 neurons) and the perirhinal cortex (41 neurons) of both animals (see Experimental Procedures). General response properties—spontaneous activity, responses to single stimuli, responses to choice arrays, proportion of selective neurons—of neurons in the two areas were similar. Area TE neurons exhibited each of the main effects with a frequency similar to that of perirhinal neurons (χ², df = 1, n.s.). For those choice arrays exhibiting each main effect, there was no significant difference (Student’s t test) in the strength of the effects (as measured by the magnitude of the SA, CS, and IS indices) between the two areas.

There were, however, differences in the effect of stimulus choice between perirhinal cortex and area TE. In perirhinal cortex, but not area TE, activity was significantly greater (paired Student’s t test) before the good stimulus was chosen than before the poor stimulus was chosen (13.2 versus 12.9 sp/s, p < 0.05, n = 82) and before the monkey chose a stimulus whose paired-associate was relatively ineffectual (13.2 versus 12.8 sp/s, p < 0.02, n = 82). The preference for selection of the good stimulus is reflected in the correlation between differences in response to the choice array for each stimulus chosen and the differences in response to these stimuli when presented as cue, which was significant for perirhinal cortex (r = 0.28, p < 0.02, n = 82) but not area TE (r = −0.07, n.s., n = 106). Similarly, the preference for selection of the stimulus whose paired-associate elicited a weak response is reflected in the correlation between differences in responses to the choice array for each stimulus chosen and the differences in responses to their paired-associates, which was significantly negative for perirhinal cortex (r = −0.28, p < 0.02), but not area TE (r = 0.01, n.s.). Thus, although activity in both area TE and perirhinal cortex depended on the monkeys’ choices, only in perirhinal cortex was this activity consistently related to how well neurons responded to the selected stimulus and its paired-associate.

Learning Dependence of the Influences on Neuronal Activity

Neuronal activity was recorded as animals acquired each new set of stimulus associations. We previously reported that IT neurons exhibit selectivity changes that reflect this associative learning very rapidly following the presentation of new stimulus pairings (Messinger et al., 2001). Here, we assessed whether the effects of stimulus arrangement, chosen stimulus, or instructed stimulus exhibited any systematic changes during learning. Our expectation was that the stimulus arrangement effect—a dependence on the visual input—should be independent of learning, but that the chosen stimulus and instructed stimulus effects should grow stronger with learning.

To test these predictions, we compared the magnitude of the SA, CS, and IS indices in the first half of the recording session (average of first and second quartiles) with their magnitude in the second half (average of third and fourth quartiles). Between the first and second halves, there was a significant (paired Student’s t test, n = 188) increase in the strength of both the instructed stimulus effect (from 0.16 ± 0.01 to 0.19 ± 0.01, p < 0.05) and the chosen stimulus effect (from 0.12 ± 0.01 to 0.15 ± 0.01, p < 0.05), but no change in the strength of the stimulus arrangement effect (0.23 ± 0.01 in both halves, n.s.). Thus, neuronal activity reflected which stimulus should have been chosen and which stimulus would be chosen to a greater degree as the monkeys learned.

We also compared average performance (see the Behavioral Results) with the average magnitude of the indices across quartiles (|IS| = 0.17, 0.16, 0.19, 0.19; |CS| = 0.12, 0.13, 0.15, 0.15; |SA| = 0.22, 0.25, 0.22, 0.23). The coefficient of correlation (r) between performance and the magnitude of the indices was 0.824 (n.s.) for the instructed stimulus effect, 0.999 (p < 0.001) for the chosen stimulus effect, and −0.067 (n.s.) for the stimulus arrangement effect (n = 4 quartiles). Thus, the degree to which neuronal activity predicted which stimulus would be chosen mirrored the progress of associative learning.

Discussion

We recorded from individual neurons in the inferior temporal cortex and explored activity as monkeys prepared to choose a remembered stimulus from a display. During this phase of the task, the brain is confronted with the simultaneous demands of sensory processing, memory recall, and selection of a target for action. We found subpopulations of neurons that clearly convey each of these sensory/cognitive features. In the following sections we address the functional significance of these neuronal signals.

Dependence of Neuronal Activity on the Instructed Stimulus

For over half of the IT neurons recorded, activity during the decision phase of the trial significantly reflected what the monkeys had been instructed to do on that trial, regardless of the forthcoming behavioral response. Because this signal is independent of behavior, we conclude that it cannot represent a motor plan for the executed saccade, the influence of such a plan on visual processing (Hamker, 2005), or a covert shift of attention to the endpoint of the saccade. Even at the time of the saccade, the activity of neurons exhibiting the instructed stimulus effect did not reflect the executed response (cf. correct and incorrect trials in any row of Figure 2).

These neurons fired at a different rate depending on which stimulus the monkey should have chosen to receive a reward. In this sense, the cells “knew” more than the monkeys exhibited by their behavior. Although behavioral performance is generally accepted to be a veridical assay of knowledge, in fact, behavior is heavily influenced—in the laboratory and in the real world—by other factors such as attentional, motivational, and behavioral states. Thus, behavior may vary, but knowledge endures. The neuronal signals that we have recorded appear to reflect this stability.

What Knowledge Does Instructed Stimulus Activity Represent?

Because there was a one-to-one relationship between each cue and the choice stimulus it instructed, neurons exhibiting the instructed stimulus effect could have been representing either stimulus in the pair or the associative relationship between them. In other stimulus-stimulus association studies, the same possibilities apply to the interpretation of activity between the cue and choice stimuli. Such delay period activity has been shown to contribute to retrospective coding of the cue, prospective coding of the cue’s paired-associate, or both (Erickson and Desimone, 1999; Fuster et al., 2000; Gibson and Maunsell, 1997; Naya et al., 2001; Rainer et al., 1999; Sakai and Miyashita, 1991; Tomita et al., 1999).

Similarly, we find that for cells exhibiting the instructed stimulus effect, neuronal responses to the choice arrays were both correlated with responses to the preceding cue and anticorrelated with responses to the cue’s paired associate. Thus, instructed stimulus activity is a fusion of both retrospective memory of the cue and recall of the association. As we have shown, the retrospective component of this activity cannot be explained simply as a prolonged response to the preceding cue that was superimposed on the response to the choice array. Instead, the cue stimuli transiently and differentially altered the way that these neurons responded to subsequent visual inputs. A possible biophysical basis for this memory trace involves synaptic inputs to a neuron that are activated both by a cue and by the choice array containing the learned paired-associate. Presentation of the cue might potentiate or depress these inputs, thereby influencing the neuron’s response to the subsequent choice array.

Relationship to Other Evidence for Neuronal Instruction Signals

The experiments of Chelazzi et al. (1993, 1998, 2001) and Jagadeesh et al. (2001) are particularly relevant to the current findings. These studies also investigated neuronal activity in IT cortex before monkeys selected a stimulus from a two-item choice array by means of a saccade. Using a delayed match-to-sample task, Chelazzi and colleagues showed that the firing rates of IT and V4 neurons were greater when the monkey chose a good, i.e., effective, stimulus for the recorded neuron, than when it chose a poor stimulus. They interpreted this response modulation as reflecting a covert shift of attention to the chosen stimulus (see below). However, these investigators analyzed neuronal activity only on correct trials, for which the chosen stimulus was also the instructed (matching) stimulus. It is thus impossible to determine if the observed response modulation indeed reflects behavioral choice or, alternatively, stimulus instruction. Our findings suggest that both types of modulation were present in the population sampled by Chelazzi et al.

Using an object discrimination task, Jagadeesh and colleagues also showed that the activity of individual IT neurons correlated with whether the monkey correctly chose a good or a poor stimulus. Furthermore, these investigators found that, on both correct and incorrect trials, activity across the neuronal population was significantly greater when a good stimulus was associated with reward than when a poor stimulus was. Thus, like our main effect of instructed stimulus, activity reflected whether the good stimulus had been instructed (in the sense that its selection would yield reward) even when the monkey chose the poor stimulus. The picture is complicated, however, by their additional finding that, when the poor stimulus was associated with reward, firing rates were dependent upon the chosen stimulus. Hence, there was a significant interaction between which stimulus was instructed and which was chosen. We suggest that the results of Jagadeesh et al. reflect a combination of which stimulus was associated with reward and which was chosen, because a mix of cells exhibiting instruction and choice effects contributed to their population level analysis.

We have extended the important findings of Chelazzi et al. and of Jagadeesh et al. in several critical ways. First and foremost, we have analyzed correct and incorrect trials on a cell-by-cell basis. This led us to the discovery that the activity of many IT neurons depended on which stimulus should be chosen and that the activity of a largely distinct class of neurons depended on which stimulus would be chosen. Secondly, we have shown that responses to the choice array depended on a preceding cue, even though the cue stimulus was not also part of the choice array, i.e., not a physical match. Thirdly, we found that in IT cortex the representation of a stimulus’ value, i.e., its association with reward, can be updated not just over a block of trials but on a trial-by-trial basis via cueing. Fourthly, we found that activity reflecting either instruction or choice can arise long before the behavioral response is made. This timing suggests that the modulation of activity is not tied to the behavior itself, but rather to mnemonic activity and/or an attentional shift.

Neuronal activity analogous to the instructed stimulus effect has been reported in rats trained to associate odor cues with delivery of either a rewarding or an aversive liquid. The rats gradually learned not to drink following the odor associated with the aversive liquid. Neurons in the basolateral amygdala (Schoenbaum et al., 1999) and ventral striatum (Setlow et al., 2003) were found to develop differential responses to the odors at an early stage of learning when the rats were still drinking following both odors. In some cases, neuronal activity became dependent on the odor but not on whether or not the rat drank. These neurons appeared to encode the learned odor-liquid association and not the rat’s overt behavior. This neuronal representation is reminiscent of the instructed stimulus effect, except of course, that the instructed stimulus effect reflects which stimulus is associated with reward, rather than the reward itself.

Dependence of Neuronal Activity on the Chosen Stimulus

For a subset (22%) of the recorded IT neurons, activity was significantly dependent on the stimulus that was actually chosen. This activity was predictive of the target of the upcoming saccade and may reflect a covert shift of attention to the subsequently selected stimulus. IT neurons have been shown to represent attended stimuli at the expense of other RF stimuli (Moran and Desimone, 1985). Consistent with this finding, we found that the majority of the neurons exhibiting a main effect of chosen stimulus were more active before selection of a good stimulus rather than a poor stimulus. Most of these neurons were also more active before selection of a stimulus whose paired-associate elicited a relatively weak response. Thus, neuronal activity depended both on the stimulus that the monkey chose and the stimulus that the monkey had learned was associated with its choice.

As noted above, Jagadeesh et al. and Chelazzi et al. both observed behaviorally dependent neuronal responses that are analogous to our chosen stimulus effect. These studies observed such dependence more frequently than we did (Chelazzi et al., 1998: 45%; Jagadeesh et al., 2001: 53%). We attribute the discrepancy in findings to their inclusion of cells with dependence on the instructed stimulus, which were masquerading as cells exhibiting choice dependence. In support of this conclusion, we found that 65% (61/94) of the neurons that we recorded exhibited a main effect of either chosen stimulus or instructed stimulus.

Dependence of Neuronal Activity on the Stimulus Arrangement

The relative locations of the two choice array stimuli in the contralateral hemifield had a strong influence on the activity of most (74%) of the IT neurons in our population. Similar effects have been reported for IT neurons in other experiments in which two stimuli switched positions (Chelazzi et al., 1998, 1993; Missal et al., 1999; Sato, 1989, 1995; Xiang and Brown, 1999). Perhaps such effects should not be surprising given the known visual selectivity of IT neurons (e.g., Desimone et al., 1984; Gross et al., 1972) because, unlike the other factors influencing neuronal activity that we compared in this study, the two stimulus arrangements actually rendered different retinal images. Importantly, a strong argument can be made (see Results) that the stimulus arrangement effect cannot be accounted for by RFs that fail to span both choice array stimuli or that are more responsive in some regions than others.

Experimental Procedures

Subjects

Two adult male rhesus monkeys (Macaca mulatta) were used in this study. Subjects had normal color vision and no significant refractive error. The monkeys were used in accordance with procedures approved by the Salk Institute Animal Care and Use Committee and in compliance with USDA regulations and NIH guidelines for the care and use of laboratory animals.

Surgical Preparation

Procedures for surgery and wound maintenance have been described in detail elsewhere (e.g., Dobkins and Albright, 1994). Briefly, a head post and a recording cylinder (located at AP +15, ML +20) were affixed to the surface of the skull using stainless steel rails, screws, and dental acrylic. A scleral search coil for measuring eye position was surgically implanted in one eye using a variation of the method of Judge et al. (1980). After surgical recovery and attainment of criterion performance on the visual paired-association task (see below), a craniotomy was performed to allow for dorsoventral electrode passage into IT cortex. All surgical procedures were performed under sterile conditions using isoflurane anesthesia.

Visual Stimuli

All visual stimuli were generated using a high-resolution graphics display controller (1280 × 1024 pixels, 8 bits/pixel) operating in a Pentium-II computer. Stimuli were displayed on a 21 in analog RGB video monitor (Sony GDM-2000TC; 76 Hz, noninterlaced). The computer also controlled behavioral data acquisition and monitored eye position.

The visual stimuli used in these experiments consisted of clip-art figures that were drawn from a 250 item database. The figures were scaled so that the largest dimension spanned 3.4° of visual angle. Mean luminance of each figure was ~10 cd/m². Background luminance of the video monitor was ~0.5 cd/m². Stimuli were viewed in a quiet light-tight room with an ambient luminance of <0.5 cd/m².

Monkeys were seated in a standard primate chair and viewed the display from a distance of 60 cm. Head movements were prevented by bolting the head post to the chair frame. Eye position was monitored continuously (Robinson, 1963).

Behavioral Paradigm

Monkeys were trained to perform a visual paired-association (PA) task. Figure 1A illustrates the sequence of events in each trial of the task. Trials began with the appearance of a fixation target (0.4°) on the video monitor. The monkey had to maintain its gaze within 0.5° of the fixation target for as long as it was present, or the trial was immediately aborted. After 500 ms of fixation, a cue stimulus appeared at the center of gaze. After 1000 ms, the cue disappeared and the choice array appeared. This array consisted of two stimuli that were simultaneously presented at an eccentricity of 5.0°. During electrophysiological recording, these stimuli always appeared in the visual hemifield contralateral to the recorded neurons. One of the choice stimuli was the paired-associate of the cue. The second stimulus was a distractor drawn from the other stimulus pair.

After the onset of the choice array, the fixation target remained on screen for an additional 500 ms, which we termed the decision phase. When the fixation target was extinguished, the monkey was permitted to select one of the stimuli in the choice array by making a saccadic eye movement to the stimulus. A saccade to the paired-associate of the cue stimulus resulted in juice reward and visual feedback (650 ms) illustrating the correct pairing. A saccade to the distractor ended the trial without a reward; a buzzer was sounded and the correct stimulus was presented (150 ms). A correction trial (repeat of the same cue stimulus) was then given after a brief delay.

Four new stimuli were used during each behavioral session and stimulus pairings were arbitrarily assigned. We refer to these stimuli generically as A, B, C, and D, and the assigned pairings were always A with B and C with D. On each trial, one of the four stimuli was randomly selected to be the cue. Following cue presentation, the cue’s paired-associate, i.e., the instructed stimulus, was randomly presented in either the upper or lower visual field quadrant and a particular distractor was presented in the other quadrant. This randomization ensured that saccades to both locations were equally likely to be rewarded and that learning was due to the formation of visual-visual associations rather than visual-motor or visual-place associations. For a given cue stimulus, the same distractor was always presented so that the temporal associations formed between the cue and its paired-associate would be comparable to those between the cue and the distractor. Therefore, the temporal associations formed between successively presented stimuli could not be used to learn the stimulus pairings and the monkeys instead had to rely on the conditional associations formed on the basis of which choices were rewarded.

Training of the Paired-Associates Task

The monkeys received several months of training in the PA task prior to the electrophysiological recording phase of the experiment. For each behavioral session, we selected a set of four physically dissimilar stimuli that lacked any prior semantic relationships. To facilitate mastery of the PA task requirements, each set of four stimuli was initially used for several consecutive behavioral sessions. As learning rates improved, new stimuli were introduced with greater frequency until the animal could reliably learn new pairings within a single session. During each session, animals generally performed 500–1500 trials.

Electrophysiological Phase

For each IT neuron isolated, a set of four novel stimuli was chosen for learning in the PA task. In addition to the constraints applied during the training phase, stimulus selection was based on the pattern of neuronal responses to a set of 8–16 pictures presented at the center of gaze in a simple fixation task. The four PA stimuli chosen from this set were those eliciting the greatest neuronal response, the smallest response, and two intermediate-level responses. These four stimuli were randomly grouped into two pairs. Behavioral and neuronal responses were recorded simultaneously as the animals learned the desired stimulus pairings.

Electrophysiological Procedures

An insulated tungsten microelectrode (0.5–1.5 MΩ impedance) was lowered into the brain during each recording session, using a hydraulic micropositioner (David Kopf Instruments). IT cortex was identified by comparing neuronal activity encountered as the electrode was lowered with high-resolution MR brain sections and confirmed on the basis of physiological properties such as visual stimulus selectivity and receptive field characteristics. Once the electrode was in IT cortex, the brain was allowed to settle. We then isolated a neuron, assessed its stimulus selectivity, and began the PA task. Extracellular action potentials were amplified, filtered, and recorded by standard methods.

Postmortem histology is unavailable because the monkeys used in this study are currently involved in other experiments. We therefore used sulcal landmarks to estimate whether the recorded neurons were located in area TE or the perirhinal cortex. Area TE was taken to include the ventral bank of the superior temporal sulcus, the middle temporal gyrus, and both the lateral bank and floor of the anterior middle temporal sulcus. Perirhinal cortex was taken to include the medial bank of the anterior middle temporal sulcus and the inferior temporal gyrus up to the lip of the rhinal sulcus.

Data Analysis

We focused on neuronal activity associated with the decision phase (Figure 1A), during which the choice array was present and the monkeys were fixating. In particular, we investigated how responses to choice arrays depended on: (1) the stimulus that the monkey had been instructed to pick by the preceding cue stimulus, (2) the stimulus chosen by the monkey, and (3) the spatial arrangement of the stimuli in the choice array. We refer to these factors respectively as the instructed stimulus, the chosen stimulus, and the stimulus arrangement. We quantified the absolute and relative influences of these three factors by calculating the amount of variance in the neuronal response to a particular choice array accounted for by each factor. This calculation was based on a three-way analysis of variance (ANOVA).

We also quantified the degree to which these variables modulated neuronal firing rates using three contrast indices: the Instructed Stimulus (IS) index, the Chosen Stimulus (CS) index, and the Stimulus Arrangement (SA) index. For responses to the AC choice array, the indices were defined as follows:

$$\text{IS}=({\text{A}}_{\text{Instructed}}-{\text{C}}_{\text{Instructed}})/({\text{A}}_{\text{Instructed}}+{\text{C}}_{\text{Instructed}})$$ (1)

$$\text{CS}=({\text{A}}_{\text{Chosen}}-{\text{C}}_{\text{Chosen}})\cdot ({\text{A}}_{\text{Chosen}}+{\text{C}}_{\text{Chosen}})$$ (2)

$$\text{SA}=({\text{A}}_{\text{Down}}-{\text{A}}_{\text{Up}})/({\text{A}}_{\text{Down}}+{\text{A}}_{\text{Up}})$$ (3)

where each term corresponds to the mean firing rate from 100 to 500 ms after choice array onset, averaged across the trials that satisfied the condition described. Thus, for example, “A_Instructed” is the firing rate on those trials for which stimulus A was the correct answer. Similarly, “A_Down” is the firing rate on those trials for which stimulus A was in the lower choice array position and stimulus C was in the upper position. Corresponding definitions were used to analyze responses to the BD choice array.

Peristimulus time histograms (PSTHs) were generated by convolving average spiking activity across like trials with a Gaussian of unit area and a standard deviation of 12.5 ms. PSTHs were considered significantly different at the first of at least five consecutive 10 ms bins with a significant difference in activity (one-tailed Student’s t test, p ≤0.05). All other group comparisons were two-tailed tests, and the threshold for significance was α = 0.05. All errors are standard deviations of the mean.

Supplemental Data

Supplemental data include four figures and can be found with this article online at http://www.neuron.org/cgi/content/full/48/2/359/DC1/.