Monkey prefrontal neurons during Sternberg task performance: full contents of working memory or most recent item?

It has long been thought that items held in working memory are encoded by delay-period activity in the dorsolateral prefrontal cortex. Here we describe evidence contrary to that view. In monkeys performing a serial multi-item working memory task, dorsolateral prefrontal neurons encode almost exclusively the identity of the sample presented most recently. Information about earlier samples must be encoded outside the prefrontal cortex or represented within the prefrontal cortex in a cryptic code.

Abstract

To explore the brain mechanisms underlying multi-item working memory, we monitored the activity of neurons in the dorsolateral prefrontal cortex while macaque monkeys performed spatial and chromatic versions of a Sternberg working-memory task. Each trial required holding three sequentially presented samples in working memory so as to identify a subsequent probe matching one of them. The monkeys were able to recall all three samples at levels well above chance, exhibiting modest load and recency effects. Prefrontal neurons signaled the identity of each sample during the delay period immediately following its presentation. However, as each new sample was presented, the representation of antecedent samples became weak and shifted to an anomalous code. A linear classifier operating on the basis of population activity during the final delay period was able to perform at approximately the level of the monkeys on trials requiring recall of the third sample but showed a falloff in performance on trials requiring recall of the first or second sample much steeper than observed in the monkeys. We conclude that delay-period activity in the prefrontal cortex robustly represented only the most recent item. The monkeys apparently based performance of this classic working-memory task on some storage mechanism in addition to the prefrontal delay-period firing rate. Possibilities include delay-period activity in areas outside the prefrontal cortex and changes within the prefrontal cortex not manifest at the level of the firing rate.

NEW & NOTEWORTHY It has long been thought that items held in working memory are encoded by delay-period activity in the dorsolateral prefrontal cortex. Here we describe evidence contrary to that view. In monkeys performing a serial multi-item working memory task, dorsolateral prefrontal neurons encode almost exclusively the identity of the sample presented most recently. Information about earlier samples must be encoded outside the prefrontal cortex or represented within the prefrontal cortex in a cryptic code.

the maintenance of information in working memory is widely thought to depend on persistent neuronal activity in the dorsolateral prefrontal cortex (Constantinidis and Wang 2004; Funahashi 2013; Riley and Constantinidis 2016; Wimmer et al. 2014). A key foundation for this view is the classic observation that prefrontal neurons in monkeys encode planned actions and expected objects in delayed-response and delayed-match-to-sample tasks (Funahashi et al. 1989; Miller et al. 1996). Because these tasks require monkeys to remember only one item per trial, they do not, however, allow distinguishing between two quite different possibilities: first that dorsolateral prefrontal neurons represent the entire contents of working memory and second that they represent only the item most recently presented.

To make this distinction necessitates analyzing neuronal activity over the course of a trial in which successive items are added to working memory. Prior studies based on this approach have used tasks requiring monkeys to remember two successive items and their order (Funahashi et al. 1997; Inoue and Mikami 2006; Rigotti et al. 2013; Siegel et al. 2009; Warden and Miller 2007, 2010) or to remember the order in which three items, always the same, were presented (Ninokura et al. 2003, 2004). Reports based on these studies indicate that dorsolateral prefrontal neurons encode information about multiple items. However, interpretation is complicated by the requirement to remember the order as well as the identity of the items. Several studies have used a design in which multiple items are presented simultaneously rather than sequentially (Buschman et al. 2011; Lara and Wallis 2014; Matsushima and Tanaka 2014). The resulting reports indicate that prefrontal neurons encode the identities either of multiple items (Buschman et al. 2011; Matsushima and Tanaka 2014) or of none (Lara and Wallis 2014). However, because the items were presented simultaneously, these studies do not cast light on the issue of whether the most recently presented item receives preferential representation.

The most straightforward approach to resolving this issue is to record neuronal activity during the performance of a Sternberg working-memory task. In such a task, multiple samples are presented sequentially followed by a probe that the subject must identify as matching or not matching one of the preceding samples (Sternberg 1966). The Sternberg task is easily adapted for use in monkeys (Sands and Wright 1982; Sands and Wright 1980a, 1980b; Wittig and Richmond 2014; Wright et al. 1985). Accordingly, we set out to determine whether neuronal activity in the monkey prefrontal cortex during performance of a Sternberg task preferentially encodes the identity of the sample presented most recently or, alternatively, represents all samples currently in the memory store.

MATERIALS AND METHODS

Subjects

All procedures were in accordance with guidelines set forth by the United States Public Health Service Guide for the Care and Use of Laboratory Animals and were approved by the Carnegie Mellon University Institutional Animal Care and Use Committee. The experiments were carried out on two adult male rhesus monkeys (Macaca mulatta) weighing 8–9 kg: Ro (henceforth M1) and Be (henceforth M2). Each monkey was surgically equipped with an acrylic cranial implant allowing head restraint and scleral search coils for eye-position monitoring. After behavioral training was completed, a recording chamber with a bore of ~2 cm was implanted with its base flush to the dura overlying the dorsolateral prefrontal cortex in the right (M1) or left (M2) hemisphere.

Tasks

The monkeys sat with head fixed facing a 17-in. LCD monitor in a dark room at a distance of 56 cm. All aspects of the experiment, including monitoring eye position, displaying stimuli, and delivering reward, were under online control using National Institute of Mental Health (NIMH) Cortex software. Eye position was monitored using scleral search coils with a field driver and signal processing filter (Riverbend Technologies). The outputs of the search-coil were led to the computer via an A/D converter and monitored online through NIMH Cortex. The computer sampled and stored horizontal and vertical readings at a rate of 1 kHz. Daily intake of fluids was regulated to maintain motivation to perform the task.

Both monkeys were trained to perform a spatial Sternberg task (Fig. 1A). In this task, there were six possible samples, each a white disk 1.3° in diameter at an eccentricity of 5°. The six samples were located so as to form a hexagonal array with two items directly to the right and left of fixation. M1 also mastered a color variant of the task (Fig. 1B). The six samples in this task were 1.3° disks, red, yellow, green, cyan, blue, and magenta, respectively, centered at fixation. Training occupied ~1 yr for each monkey.

Fig. 1.

Sequence of events in a trial. Sequence is shown for (A) the spatial and (B) the color version of the cued-recall Sternberg task. Dotted circle indicates gaze direction. The samples were either all different (3-item trials) or all the same (1-item trials). The monkey was required to respond with a saccade (arrow) to that probe that matched 1 of the preceding samples. In the examples shown here, the match probe corresponded to sample 1.

The monkey initiated each trial by fixating a gray central spot subtending 0.3°. During the ensuing phase of the trial leading up to presentation of the choice array, the monkey was required to maintain gaze within a 3° window centered on this spot. Deviation of gaze outside the window resulted in immediate termination of the trial without reward. The choice array consisted of three items: a match probe and two nonmatch probes. The monkey selected one of the three probes by making a saccade to it. Selection of a nonmatch probe resulted in immediate termination of the trial without reward. Following selection of the match probe, the nonmatch probes disappeared. This served as immediate positive feedback. After a 400- to 550-ms fixation on the match probe, the monkey received reward (0.1 ml of liquid).

In each run, the monkey was required to complete successfully four trials conforming to each of 36 conditions. The conditions represented all possible combinations of sample 1 identity (6 possibilities) and sample 2 identity (6 possibilities). If sample 1 and 2 were different, then sample 3 was chosen at random from the remaining four items. If samples 1 and 2 were the same, then sample 3 was chosen to be identical to them. The single match probe in the choice array was chosen at random from the three samples presented earlier in the trial. The two nonmatch probes in the choice array were chosen at random from the three items not presented as samples earlier in the trial. One run consisted of 144 correct trials. The order in which the conditions were imposed was random except for the requirement that each block of 36 successfully completed trials must consist of 1 trial conforming to each condition. The monkeys typically completed one run of each learned task during a daily recording session.

Behavior

The key aim of behavioral analysis was to characterize the impact on performance of two factors that varied across trials: 1) memory load and 2) the ordinal position of the sample corresponding to the match probe presented at the end of the trial. The key measures of performance were the percent correct score and the reaction time on correct trials. All measurements were based on trials in which the monkey made a saccade to one of the three probes and maintained fixation on it until termination of the trial.

Memory load.

Memory load was either low (on 24 trials per run in which samples 1, 2, and 3 had the same identity) or high (on 120 trials per run in which samples 1, 2, and 3 had different identities). For each run, we computed the mean of the dependent variable (percent correct or reaction time) for each of 12 conditions obtained by crossing memory load (2 levels) with match probe identity (6 levels). Then, combining the results across runs, we carried out an ANOVA with memory load and match probe identity as factors. We included match probe identity as a factor solely in order that any variance dependent on it would be factored out in the analysis of the dependence of performance on memory load.

Ordinal position.

On high-load trials, the match probe could correspond to either the first, the second or the third sample presented during the preceding sequence. For each run, we computed the mean of the dependent variable (percent correct or reaction time) for each of 18 conditions obtained by crossing ordinal position (3 levels) with match probe identity (6 levels). Then, combining the results across runs, we carried out an ANOVA with ordinal position and match probe identity as factors. We included match probe identity as a factor solely in order that any variance dependent on it would be factored out in the analysis of the dependence of performance on ordinal position. If the full ANOVA revealed a significant main effect of ordinal position, then we carried out three post hoc ANOVAs with the factor of ordinal position confined to two levels: 1 vs. 2, 1 vs. 3, and 2 vs. 3.

Recording

The recording chamber accommodated a grid allowing alignment and replication of electrode tracks with 1-mm spacing (Crist et al. 1988). During each recording session, a tungsten electrode with an initial impedance of ~5 MΩ at 1 kHz (Frederick Haer) was advanced through the dura into the cortex at a selected grid location. Spike waveforms (with 40-kHz resolution) and event markers (with 1-kHz resolution) were stored on a computer running Plexon software (Plexon). Spike sorting was accomplished off-line by use of the Plexon Off-line Sorter program. After neuronal data collection was completed, the location of the recording sites was established by analysis of the alignment between fiducial markers in the chamber and brain landmarks visible in structural images obtained by magnetic resonance imaging.

Histograms

To construct population histograms, we first split high-load trials into odd-numbered and even-numbered groups on the basis of their position in the 144-trial sequence making up a run. For each neuron, for each ordinal position, using data from odd-numbered trials, we identified the “best” and “worst” samples: those associated with strongest and weakest firing 100–600 ms after sample onset during the relevant epoch. Then, using data from even-numbered trials, we computed the mean across neurons of the firing rate in each 1-ms bin on “best sample” and “worst sample” trials. We smoothed the resulting histograms with a 10-ms Gaussian kernel.

Spatial Tuning

To characterize the spatial tuning of each neuron, we constructed an angular bias vector, using a method developed previously to characterize orientation tuning (Smith et al. 2002). We represented response strength with the sample at the six locations as a set of six response vectors, (θ_n, R_n), where θ_n was the angular direction from fixation to the sample, R_n was mean firing rate with the sample at that location, and n was an index in the range 1–6. We characterized the angular preference of the neuron as the direction (θ) of the vector obtained by summing the six response vectors. We characterized tuning sharpness by normalizing the magnitude of the summed vector to the sum of the magnitudes of the six response vectors. The resulting measure ranged from 0 for a neuron firing at the same rate when the sample was at all six locations to 1 for a neuron firing only when the stimulus was at one location. To determine whether the angular bias vectors of a population of neurons were distributed uniformly around the clock, we employed the Moore variant of the Rayleigh Test (Moore 1980).

Correlation Analysis

The aim of this step of analysis was to measure the consistency with which neurons represented the six locations in the spatial task or the six colors in the color task. We first describe how we analyzed consistency within a given delay period. For each neuron, for odd and even trials independently, we computed the mean firing rate associated with each of the six stimuli during the epoch 100–600 ms after stimulus onset. We then corrected each firing rate by subtracting from it the mean of all six rates. This step yielded six pairs of values, with a mean of zero, representing firing rates associated with the six stimuli on odd and even trials, respectively. We then carried out a correlation analysis on all 6 × n points obtained from n neurons. The Pearson’s correlation coefficient reflected the degree to which variations in firing rate dependent on stimulus selectivity were consistent across odd and even trials within the same delay period. To analyze consistency between different delay periods, we compared odd trials in the earlier delay period to even trials in the later delay period and vice versa. We then took the average of the two resulting Pearson’s correlation coefficients.

Decoding

The aim of this step of analysis was to measure the level of accuracy achieved on the task by a correlation-based linear classifier (Meyers et al. 2008) operating on population activity during delay 3. This classifier has the advantage of basing classification on the pattern of population activity without regard to gain, which might change over the course of the trial. For sample 1, sample 2, and sample 3 independently, we decoded sample identity over 10,000 iterations. On each iteration, we removed one trial from the database, created a classifier on the basis of the remaining trials, and used the classifier to identify the sample from the left-out trial.

Because the neurons had not been recorded simultaneously, we had to construct the trial to be left out on each iteration. We randomly selected an identity for the sample under consideration. In the spatial task, this was one of six possible locations; in the color task, it was one of six possible colors. Then, we constructed a multineuronal pseudotrial by randomly selecting from each neuron’s database a single trial in which the sample under consideration had the stipulated identity.

We constructed the classifier from trials remaining in the database. The classifier consisted of six classification vectors representing mean population activity during delay 3 on trials in which the sample under consideration had each of the six different possible identities. Each vector had a dimensionality equal to the number of neurons in the database. We computed the Pearson correlation between each classification vector and the vector representing population activity on the left-out trial. Classification of the sample from the left-out trial was based on which classification vector yielded the strongest correlation.

To assess the level of significance with which classifier accuracy exceeded the level expected by chance, we carried out 1,000 iterations of a bootstrap procedure. On each iteration, before extracting one trial from the database and constructing a classifier on the basis of the remaining trials, we randomly shuffled across trials the identity of the sample under consideration. We identified the 95% confidence bounds on the bootstrap distribution of percent-correct scores. If the percent correct achieved by the classifier lay above the upper bound, we categorized the outcome as significant.

To determine whether the classifiers could support task performance at the level achieved by the monkeys, we converted the classifier percent-correct score, C, to a task-based percent-correct score, T, according to the formula T = (4C + 100)/5. This formula accommodates contingencies arising when the classifier signals an incorrect identity. With a probability of 2/5, the erroneously signaled identity will belong to one of the nonmatch probes, leading to a wrong choice. With a probability of 3/5, it will belong to none of the probes. In this case, guesswork will lead to a wrong choice on 2/3 of trials and a correct choice on 1/3 of trials.

RESULTS

Two monkeys (M1 and M2) performed a cued-recall task that required holding three sequentially presented spatial samples in working memory (Fig. 1A). On each trial, following successive presentation of the three samples, three probes appeared simultaneously. Only one of the probes matched a preceding sample. The monkey had to execute a saccade directly to the match probe to receive reward. One of the monkeys (M1) also performed a color variant of the task (Fig. 1B), but the other monkey was unable to master this variant even after extensive training. All key results, both behavioral and neuronal, were demonstrable in the spatial task. We include data from the color task so as to demonstrate that all key measures from the spatial domain generalized to the color domain in the monkey able to master both tasks.

We based the analysis of behavioral performance on all runs during which we collected neuronal data: 84 runs of the spatial task (41 in M1 and 43 in M2) and 45 runs of the color task in M1 (details in Behavior). To determine whether performance was affected by memory load, we compared performance on 120 high-load trials per run (in which the 3 samples differed from each other) to performance on 24 low-load trials per run (in which the 3 samples were the same). The monkeys performed significantly above chance under both loads but the percent correct was significantly lower and the reaction time was significantly longer under the high-load than under the low-load condition (Fig. 2). This observation agrees with previous reports of load effects in monkeys (Elmore et al. 2011; Heyselaar et al. 2011; Lara and Wallis 2012). To determine whether the ability of the monkeys to recall a sample depended on its ordinal position in the trial sequence, we analyzed data from the high-load trials. We found that the monkeys performed significantly above chance regardless of whether they were required to select a match probe corresponding to the first, second, or third sample but that performance did vary as a function of the ordinal position of the sample. The percent correct increased and the reaction time decreased across trials in which the probe matched the first, second, or third sample (Fig. 2). We conclude that the monkeys exhibited a recency effect.

Fig. 2.

Behavioral performance. A and B: spatial task: data from monkey 1 and 2 (M1 and M2) combined. C and D: color task: M1 only. S1, S2, S3, samples 1, 2, and 3. Dashed lines indicate levels of performance expected by chance. *P < 0.05, **P < 0.0005, ***P < 0.00001, statistically significant differences. Quantification and breakdown by monkey are in Table 1.

Table 1.

Quantification and breakdown by monkey of results shown in Fig. 2 concerning load and recency effects

	M1 Spatial	M2 Spatial	M1 Color
Load effect (low-high)
%Correct: delta	25%	25%	19%
%Correct: P	<0.0001	<0.0001	<0.0001
RT: delta	−46 ms	−20 ms	−19 ms
RT: P	<0.0001	<0.0001	<0.0001
Recency effect (S3-S1)
%Correct: delta	8%	9%	8%
%Correct: P	<0.0001	<0.0001	<0.0001
RT: delta	−5 ms	−13 ms	−5 ms
RT: P	<0.0001	<0.0001	<0.0001

Effect size (delta: the difference between conditions) and effect significance (P) are provided for measures based on accuracy (%Correct) and speed (RT) in both monkeys (M1 and M2) and for both tasks (spatial and color). Load effect: higher accuracy and greater speed under low load than under high load. Recency effect: higher accuracy and greater speed for recall of sample 3 (S₃) than for recall of sample 1 (S₁).

We recorded from neurons in the dorsolateral prefrontal cortex in the right hemisphere of M1 and the left hemisphere of M2 during the same runs on which the preceding behavioral analysis was based (details in Recording). In each monkey, exploratory probes revealed a hotspot of task-related activity. In subsequent data collection sessions, we focused on this region. MR images collected at the conclusion of the period of neuronal data collection revealed that the recording zone was located ventral to the posterior principal sulcus in each monkey (white symbols in Fig. 3). We collected data from 254 neurons (122 in M1 and 132 in M2) in the context of the spatial task and 132 neurons (in M1) in the context of the color task.

Fig. 3.

Structural MR images tangential to the surface of the cortex underlying the recording chambers. White dots indicate recording sites at which task-related neurons were encountered. Black dots indicate recording sites at which no task-related neurons were encountered. Images from the 2 monkeys abut at the interhemispheric midline. AS, arcuate sulcus. PS, principal sulcus.

We first asked whether, in high-load trials, neurons responded selectively to each sample immediately following its presentation. For each neuron, for samples 1, 2, and 3 independently, we identified the best and worst identities on odd-numbered trials, identities associated with strongest and weakest firing during the delay period 100–600 ms following sample onset. We then constructed histograms representing the mean population firing rate on best sample and worst-sample even-numbered trials. Neurons responded more strongly to the best sample than to the worst sample during each delay period (Fig. 4, continuous vs. dashed curve). To confirm this observation, we assessed the Pearson correlation between odd and even trials with regard to activity dependent on the identity of the most recently presented sample (details in Correlation Analysis). The correlation was positive and significant during each delay period (see Fig. 6, “selectivity”).

Fig. 4.

Neurons were selective for the identity of a sample immediately following its presentation but selective activity did not persist into subsequent task epochs. Population histograms of represent activity from the spatial task (A) and the color task (B) during trials in which the “best” sample, that most preferred by the neuron during the indicated trial epoch (solid curve), and the “worst” sample, that least preferred by the neuron during the indicated trial epoch (dashed curve), were presented at ordinal position 1 (left), ordinal position 2 (middle), or ordinal position 3 (right). “Best” and “worst” samples were identified independently for each ordinal position on the basis of trials not included in the histograms.

Fig. 6.

Neurons responded selectively to the samples; the pattern of selectivity was consistent across delay periods; but selective activity elicited by each sample did not persist beyond the immediately ensuing delay period. A: spatial task. B: color task. The height of each bar indicates the strength of correlation between sample selectivity measured on odd trials under one testing condition and on even trials under another testing condition. The 2 testing conditions are identified under each bar according to the convention S_xD_y, where x indicates the ordinal number of the sample and y indicates the ordinal number of the delay period during which firing was measured. “Selectivity”: encoding of S_nD_n on odd trials was well correlated with encoding of S_nD_n on even trials. “Consistency”: encoding of S_mD_m on odd trials was well correlated with encoding of S_nD_n on even trials. “Persistence”: encoding of S_mD_m on odd trials was poorly correlated with encoding of S_mD_n on even trials. All positive correlations reflecting selectivity (black bars) and consistency (gray bars) were statistically significant (P < 0.0001) whereas those associated with persistence (white bars) were not. Quantification and breakdown by monkey are in Table 2.

These analyses establish that neurons responded to the samples with consistent selectivity but do not capture the nature of their tuning for sample identity. We were able to address this issue in the spatial task because the locations at which samples were presented were distributed at regular intervals around the clock. For each neuron during each high-load delay period, we constructed an angular bias vector pointing in its estimated preferred direction and possessing a magnitude proportional to the sharpness of its spatial tuning (details in Spatial Tuning). The magnitude could range from zero (if a neuron showed no net spatial bias in its responses) to one (if it responded to a sample at only one location). The results are presented in Fig. 5. Preferred directions were distributed around the clock, but there was a tendency toward overrepresentation of locations contralateral to the recording hemisphere. This tendency attained statistical significance during each delay period in each monkey (P < 0.05, R > 0.25, Moore-Rayleigh test). To quantify tuning, we utilized data from low-load conditions. This approach eliminated the potential for interference from samples presented earlier in the trial and allowed collapsing data across all three delay periods. The results are shown in Fig. 5, rightmost column. The direction of the mean angular bias vector was 192 and 344° in M1 and M2, respectively. The average vector magnitude was 0.15 in M1 and 0.25 in M2. On the assumption that tuning conformed to a wrapped normal distribution, these values correspond to standard deviations of 149 and 112°, respectively. We conclude that the neurons in our sample exhibited broad spatial tuning and tended to prefer locations opposite the recording hemisphere.

Fig. 5.

Neuronal tuning for sample location in the spatial task. In each polar plot, each vector represents the properties of one neuron. The vector points in the estimated preferred direction of the neuron and its magnitude indicates the sharpness of the neuron's spatial bias. The measure of sharpness is the magnitude of the summed response vector normalized to the sum of the magnitudes of the response vectors (details in Spatial Tuning). Results from M1 (top row) and M2 (bottom row) are based on firing during delay periods 1, 2, and 3 on high-load trials and on data from low-load trials collapsed across all 3 delay periods.

We next asked whether the same neurons responded with the same pattern of selectivity to the presentation of each successive sample. To answer this question, we assessed the Pearson correlation between each pair of delay periods with regard to activity dependent on the identity of the most recently presented sample (details in Correlation Analysis). For example, we analyzed the correlation between the pattern of selectivity for sample 1 during delay 1 on odd trials and the pattern of selectivity for sample 2 during delay 2 on even trials. The correlation was positive and significant in all comparisons (Fig. 6, “consistency”). Thus neurons responding relatively strongly to a stimulus at a certain location or of a certain color when it was presented as sample 1 also responded relatively strongly to it when it was presented as sample 2 or sample 3.

Table 2.

Quantification and breakdown by monkey of results shown in Fig. 6 concerning selectivity, consistency, and persistence

	M1 Spatial	M2 Spatial	M1 Color
Selectivity
S1D1-S1D1	0.20	0.39	0.37
S2D2-S2D2	0.31	0.47	0.46
S3D3-S3D3	0.35	0.37	0.37
Consistency
S1D1-S2D2	0.23	0.45	0.38
S1D1-S3D3	0.19	0.41	0.29
S2D2-S3D3	0.36	0.38	0.33
Persistence
S1D1-S1D2	0.14	0.04	−0.02
S1D1-S1D3	−0.03	−0.12	−0.04
S2D2-S2D3	0.12	0.04	0.04

Each Pearson’s correlation coefficient reflects the correlation between sample selectivity measured on odd trials under one condition and on even trials under another condition. The 2 conditions are identified according to the convention S_xD_y where x indicates the ordinal number of the sample and y indicates the ordinal number of the delay period during which firing was measured.

We next asked whether firing representing a given sample persisted beyond the delay period immediately following its presentation. The population histograms revealed little if any persistence (Fig. 4). To confirm this observation, we assessed the Pearson correlation between each pair of delay periods with regard to activity dependent on the identity of a given sample (details in Correlation Analysis). For example, we analyzed the correlation between the activity dependent on the identity of sample 1 during delay 1 on odd trials and activity dependent on the identity of sample 1 during delay 2 on even trials. The correlations were neither consistently nor significantly positive (Fig. 6, “persistence”) indicating that activity selective for a given sample did not outlast the delay period immediately following its presentation.

Neuronal activity signaling the identity of an early sample during a late delay period might have taken an anomalous form not detectable by the measure of persistence described in the previous paragraph. For example, other neurons might have signaled the identity of the sample or the same neurons might have signaled its identity using a different code. To allow for these possibilities, we analyzed sample selectivity by an approach that made no assumptions with regard to consistency of coding across delay periods. For each neuron, during each delay period, we carried out an ANOVA with firing rate as the dependent variable. During delay periods 1, 2, and 3, the ANOVAs were based, respectively, on one, two, and three factors corresponding to the identities of all samples presented up to the corresponding point in the trial. We describe the main effects here and defer comment on interaction effects to a later paragraph. During each delay period, more neurons than expected by chance exhibited significant main effects (α = 0.05) of sample identity. This was true for all three samples (Fig. 7). However, neurons representing the immediately preceding sample markedly and significantly (α = 0.05, χ²-test) outnumbered those encoding earlier samples.

Fig. 7.

Neurons significantly selective for the most recent sample markedly outnumbered neurons selective for earlier samples. The height of each bar represents the percentage of neurons exhibiting significant selectivity for the indicated sample (S1, S2, and S3) during the indicated delay period in the spatial task (A) and the color task (B). Dashed lines indicate percentage expected by chance. *P < 0.05, ***P < 0.00001, statistically significant differences (χ²-test with Yates correction). Quantification and breakdown by monkey are in Table 3.

Table 3.

Quantification and breakdown by monkey of results shown in Fig. 7 concerning the frequency with which neurons exhibited selectivity for particular samples during particular delay periods

	M1 Spatial	M2 Spatial	M1 Color
S1D1	29 (24%)	63 (48%)	47 (36%)
S1D2	18 (15%)	22 (17%)	9 (7%)
S2D2	41 (34%)	80 (61%)	46 (35%)
S1D3	15 (12%)	24 (18%)	9 (7%)
S2D3	20 (16%)	32 (24%)	19 (14%)
S3D3	38 (31%)	80 (61%)	35 (27%)
P Values
S1D2 vs. S2D2: P	0.001	<0.0001	<0.0001
S1D3 vs. S2D3: P	0.47	0.29	0.072
	0.00064	<0.0001	<0.0001
	0.011	<0.0001	0.022

Counts (percentages) of neurons exhibiting significant selectivity are provided for each sample in each relevant delay period in each monkey (M1, M2) and for each task (spatial, color). Conditions are identified according to the convention S_xD_y where x indicates the ordinal number of the sample and y indicates the ordinal number of the delay period during which firing was measured. Counts obtained under different conditions were compared by use of a χ²-test with Yates correction yielding the indicated P values.

To determine whether neuronal activity during delay 3, as revealed by the preceding analysis, could have supported the monkeys’ level of accuracy in reporting sample identity, we assessed the performance of a linear pattern classifier operating on population activity combined across the two monkeys (details in Decoding). We combined data from the two monkeys so as to maximize decoding accuracy. This approach was justified by the fact that in each monkey considered individually decoding accuracy climbed steadily without reaching an asymptote as the number of neurons included in the analysis increased. For sample 1, sample 2, and sample 3 independently, we classified sample identity using an iterative correlation-based leave-one-out design (Meyers et al. 2008). On each iteration, we removed one trial from the database, created a classifier on the basis of the remaining trials, and used the classifier to identify the sample from the left-out trial. The classifier compared the population vector from the left-out trial to six classification vectors representing mean population responses elicited by samples with the six possible identities and based classification on the comparison yielding the highest correlation. In preliminary analyses, we found that classification performance was indistinguishable from chance forsamples 1 and 2. This might have been due to interference from firing dependent on the identity of sample 3. We proceeded to analyze classification accuracy after preconditioning the data so as to filter out activity representing samples other than the one under consideration. For example, in classifying sample 1, the classifier operated on the residuals remaining after regression of each neuron’s firing rate on sample 2 and sample 3 identity. Classifiers trained and tested in this manner were able to identify samples 1, 2, and 3 at rates significantly above chance (outside 95% confidence bounds established by a 1,000-iteration bootstrap shuffle test). We converted each classifier percent-correct score, C, to a task-based percent-correct score, T, according to the formula T = (4C + 100)/5. This formula allows for a guess-based success rate of 1/3 in the event that the output of the classifier matches none of the three presented probes (details in Decoding). It ensures that if the success rate of the classifier were 1/6, as expected by chance, then the success rate on the task would be 1/3, also as expected by chance. In the spatial task, the classifier success rates were 39, 45, and 66% for samples 1, 2, and 3, respectively. In the color task, the corresponding success rates were 35, 42, and 61%. Although the success rates for samples 1 and 2 were modestly above chance, it is critical to note that this outcome was attainable only by 1) allowing neurons to encode sample identity during the final delay period in a code different from the code used during the initial delay period, and 2) preconditioning the data by removing the strong neural signal representing the most recent sample. Even under this arguably nonbiological implementation, the information carried by the neuronal population fell short of the information available to the monkeys. Classifier performance was commensurate with the performance of the monkeys in the case of sample 3 (Fig. 8, white bars) but fell markedly short of monkey performance for samples 1 and 2 (Fig. 8, black and gray bars).

Fig. 8.

A linear classifier trained to decode sample identity from population activity during delay 3 performed at approximately the level of the monkeys on trials in which the match probe corresponded to sample 3. However its performance, unlike the performance of the monkeys, dropped precipitously toward chance on trials in which the match probe corresponded to sample 1 or 2. A and B: spatial task. C and D: color task. Dashed lines indicate levels of performance expected by chance. Quantification and breakdown by monkey in Table 4.

Table 4.

Quantification and breakdown by monkey of results shown in Fig. 8 concerning the accuracy with which each sample could be classified on the basis of population activity during delay 3

	M1 + M2 Spatial	M1 Spatial	M2 Spatial	M1 Color
Sample 1	66%	47%	64%	61%
Sample 2	45%	43%	43%	42%
Sample 3	39%	38%	36%	35%

Each value indicates the percent correct score achieved through use of an optimal strategy based on the output of the classifier. This is given by the formula t = (4C + 100)/5, where C is the percent correct of the classifier and T is the projected task-based percent correct.

Finally, we considered the possibility that neurons combined information about multiple samples using a nonlinear code inaccessible to analyses described above. For example, a given neuron might fire if and only if the first sample had a certain location or color and the second sample had a particular other location or color. The number of possible combinations was too large and the number of trials was too small to allow meaningful assessment of this possibility during delay 3. However, we were able to test the idea in an analysis based on delay 2. The 2 samples presented before delay 2 conformed with equal frequency to 30 different sequences. Their influences combined nonlinearly in some neurons as indicated by the occurrence of interaction effects in an ANOVA with sample 1 identity and sample 2 identity as factors. However, interaction effects were much less common than main effects, occurring in 12% as compared with 52% of neurons in the spatial task and 11% as compared with 39% of neurons in the color task. To determine whether nonlinear effects conveyed significant useful information, we compared the performance of a “simple” classifier based solely on main effects to the performance of a “complex” classifier allowing for interaction effects. Both classifiers operated on population activity during delay 2. The simple classifier compared the population vector from the left-out trial to six classification vectors representing mean population responses elicited by samples with the six possible identities. It did so for sample 1 and sample 2 during independent runs. The complex classifier compared the population vector from the left-out trial to 30 classification vectors based on the remaining trials. Each classification vector represented the mean population response elicited by a single sequence out of the 30 possible sequences. The classifier was rated as having correctly identified sample 1 if sample 1 in the output sequence matched sample 1 from the left-out trial and likewise for sample 2. The simple classifier significantly outperformed the complex classifier for each sample in each task (with the difference in performance exceeding 95% confidence bounds established by a 1,000-iteration bootstrap shuffle test). In the spatial task, the rate of correct classification was 28 vs. 22% for sample 1 and 49 vs. 34% for sample 2. In the color task, the rate of correct classification was 22 vs. 20% for sample 1 and 56 vs. 34% for sample 2. In conclusion, nonlinear interactions were rare at the single-neuron level and we were unable to leverage them for increased classification performance at the population level.

DISCUSSION

The key finding of this study is that prefrontal delay-period activity robustly encoded the identity only of the most recently presented sample although the monkeys were able to recall all three samples at a level well above chance. This implies that the monkeys based their performance on some store other than the prefrontal delay-period firing rate. This finding is relevant to a longstanding and ongoing debate concerning whether delay-period activity in the prefrontal cortex serves (Riley and Constantinidis 2016) or does not serve (D’Esposito and Postle 2015; Sreenivasan et al. 2014) as the ultimate neural substrate for working memory. It adds weight to the argument against identifying prefrontal delay-period activity as the working memory store.

Several prior reports have claimed that prefrontal neurons robustly represent the identities of multiple items held simultaneously in working memory. The earliest such reports provided no information on signal strength and so do not allow for good comparison (Funahashi et al. 1997; Inoue and Mikami 2006). More recent reports based on a task in which two samples were presented sequentially have described strong representations of both samples (Rigotti et al. 2013; Siegel et al. 2009; Warden and Miller 2007, 2010). In considering how to reconcile our findings with these reports, we consider two factors: recording location and task design. 1) Recording location: it is conceivable that the weak representation of early samples in our study was specific to the cortical region from which we recorded. Previous reports describing the representation of multiple items focused on cortex ventral to the principal sulcus, as did ours, but were confined to (Funahashi et al. 1997) or included (Inoue and Mikami 2006; Warden and Miller 2007, 2010) more anterior sites. We doubt this explanation for two reasons. First, none of the previous studies revealed any systematic anterior-posterior functional gradient. Second, in a preliminary exploration of more anterior sites (Fig. 3, black dots), we discovered an absence of rather than a qualitative difference of task-related activity. 2) Task design: the failure of neurons to represent early samples in our study may have been related to differences between the classic Sternberg design that we used and other designs employed in earlier studies. There were three key differences. First, we required memory for three items, not just two. Distributing resources across three items may have differentially weakened the representation of the earlier ones. Second, we adopted a design requiring memory only for the identities of the samples. Previous studies required memory for their order. This may have encouraged remembering them as a chunk. Third, we employed short delay periods. The delay periods in previous studies were at least a second as compared with half a second in our study. During longer delay periods there is time for multiple items to be represented one at a time in alternation. For instance, if the first item must be reported first, then its representation may rebound toward the end of the delay period (Siegel et al. 2009; Warden and Miller 2010).

One point of convergence between this study and earlier studies is the observation of anomalous coding whereby neurons selective for a sample late in the trial are not the same as neurons selective for it immediately after its presentation. Early studies did not address this issue (Funahashi et al. 1997; Inoue and Mikami 2006), but more recent studies have asked whether coding is consistent across phases of the trial and have found that it is not (Warden and Miller 2007, 2010). Anomalous coding in these studies might have arisen from the requirement to remember the order of the samples as well as their identity, but its occurrence in our study cannot be so explained. The occurrence of anomalous coding is incompatible with the classic “fixed selectivity” scheme (Sreenivasan et al. 2014) in which the firing of a prefrontal neuron corresponds the representation of a particular item in working memory.

A few prior reports based on recording from monkey prefrontal cortex during multi-item working memory have indicated, like our study, that prefrontal neurons do not robustly represent the identities of multiple items. Lara and Wallis (2014), recording in prefrontal cortex during performance of a task in which several visual samples were presented simultaneously, found that few neurons encoded the identities of the samples. Matsushima and Tanaka (2014), recording from prefrontal cortex in a task requiring monkeys to remember either one or two locations, found that a location was represented more robustly under the single-item than under the double-item protocol. Wise and colleagues, recording in prefrontal cortex during performance of a task requiring monkeys to attend to a visual target and hold a location in working memory simultaneously, found that neuronal activity encoded primarily the location of the attended target (Lebedev et al. 2004; Messinger et al. 2009). Our use of a Sternberg design has allowed us to make a distinction, impossible in the former studies, between neuronal activity representing the most recent sample and neuronal activity representing earlier samples.

It is possible that prefrontal neurons mediate multi-item working memory but encode the memory contents in an arcane form not accessible by measuring mean firing rate during the delay period. Some have suggested that the identity of a sample held in working memory is represented by a distinctive pattern of cross-neuronal synchrony (Salazar et al. 2012). This idea is consonant with the observation that pairs of prefrontal neurons exhibit synchronous firing and that the occurrence and precise pattern of synchrony may depend on task context (Buschman et al. 2012; Constantinidis et al. 2001; Pipa and Munk 2011; Sakurai and Takahashi 2006). There is, however, no current evidence for encoding of the identities of multiple items by synchronous activity in the prefrontal cortex. Others have suggested that representations of multiple items are kept separate by rapid temporal interleaving (Lee et al. 2005; Siegel et al. 2009). Signals carried in this way would, however, be evident at the level of activity averaged across the delay period.

It is also possible that monkeys performing multi-item working-memory tasks rely on a passive mechanism whereby information is stored in synaptic or cellular changes not manifest at the level of delay-period activity. They might, for example, make use of a mechanism similar to the one employed for long-term memory. The ability of monkeys and humans to recognize hundreds of distinctive images after brief sequential exposure is a form of long-term memory that clearly must rely on passive storage (Sands and Wright 1980a, 1980b; Wright et al. 1985). Standard working memory tasks are designed to prevent performance based on this form of storage. They do so, on the assumption that passive traces are persistent, by making repeated use, on successive trials, of images from a small set. Changes persisting beyond an individual trial would produce marked proactive interference under these circumstances. Monkeys do make errors based on proactive interference but they are not so frequent as to interfere catastrophically with performance (Wittig and Richmond 2014). A passive mechanism with a short time constant or with reset capability would be required to mediate accuracy at the level typically observed. The argument that this is a viable mechanism has been put forward in several recent reports and reviews (Brady et al. 2011; Fusi 2008; Mongillo et al. 2008; Nee and Jonides 2013). Reliance on passive storage could provide a parsimonious explanation for the ability of the monkeys to remember samples 1 and 2 despite the fact that the prefrontal representation of those samples was weak and anomalous.

Finally, it is possible that monkeys performing multi-item working-memory tasks depend on delay-period activity in posterior cortical areas (Pasternak and Greenlee 2005). Single-neuron recording studies have provided limited support for this idea by showing that parietal neurons carry categorical color signals (Buschman et al. 2011) while prefrontal neurons fail to carry parametric color signals (Lara and Wallis 2014) in chromatic working memory tasks. The idea is further supported by human functional imaging studies showing that persistent neural activity in low-order sensory areas encodes the identity of multiple items (Emrich et al. 2013). However, in the context of the Sternberg task (LaRocque et al. 2013; Lewis-Peacock et al. 2012; Lewis-Peacock and Postle 2012), and in other related contexts (Peters et al. 2009, 2012), it appears that persistent activity preferentially encodes the identity of the single item currently at the focus of attention (Nee and Jonides 2008; Öztekin et al. 2010).

The fact that prefrontal metabolic activity in humans grows stronger as items are added to working memory (Rottschy et al. 2012) might seem to constitute evidence for the representation of multiple items by prefrontal cortex. Load-related effects have been demonstrated in the context of visual working memory tasks utilizing both simultaneous and serial presentation of samples. In the simultaneous design, cortical activation is enhanced on trials when more samples have been presented both at the level of the EEG (Delvenne et al. 2011; Ikkai et al. 2010; McCollough et al. 2007; Palva et al. 2010; Vogel and Machizawa 2004) and at the level of the blood oxygen level-dependent (BOLD) signal (Magen et al. 2009; Meyers et al. 2012; Rypma et al. 1999; von Allmen et al. 2013). In the sequential design, measures based on EEG (Jensen and Tesche 2002), magnetoencephalographic (MEG) (Tesche and Karhu 2000), functional (f)MRI (Altamura et al. 2007; Landau et al. 2004), and intracranial local field potential (Axmacher et al. 2010; Axmacher et al. 2007; Haenschel et al. 2007; Haenschel et al. 2009; Howard et al. 2003; Meltzer et al. 2008; van Vugt et al. 2010) increase monotonically as samples are presented successively. Load-related activity might arise if adding more items to working memory recruited more neurons to represent them. However, it could also reflect the ramping up of executive control processes brought more forcefully into play under conditions of greater difficulty. To resolve this issue would require decoding information stored in the prefrontal cortex by use of a classifier based on the voxel-based pattern of activation. This approach is difficult to apply to the prefrontal cortex (Riggall and Postle 2012) although it may allow decoding in some cases (Jerde et al. 2012; Lee et al. 2013).

The fact that prefrontal injury in humans sometimes results in impaired working memory might be taken as evidence for its serving as a storage site. However, other interpretations are possible. An extensive meta-analysis of the neuropsychological literature on working memory (D’Esposito and Postle 1999) makes a critical distinction between delayed-response tasks and span tasks. The authors note that there is strong convergent evidence, from monkey and human studies, for an impairment of delayed-response performance following dorsolateral prefrontal injury but that there is little evidence for reduced span. Delayed-response tasks tax the ability of subjects to retain a single instruction or memorandum across a delay whereas span tasks measure multi-item capacity.

We conclude with the suggestion that delay-period activity in the prefrontal cortex, rather than encoding the full contents of working memory, encodes the identity of whatever item is at the current focus of attention. There is a clear distinction, founded on human behavioral studies, between these two constructs. In serial-presentation multi-item working-memory tasks, the item most recently presented is accessible by default with a unique degree of speed (McElree 2001; McElree and Dosher 1989; Wickelgren et al. 1980), there is a temporal cost to selecting for attention any other item from the working memory store (Garavan 1998; Oberauer 2002), and only one item from working memory can be selected at a time (Makovski and Jiang 2007). A related observation is that only one template at a time can guide visual search (Houtkamp and Roelfsema 2009). The distinction between the contents of working memory and the current focus of attention has been addressed in several recent reviews (Cowan 2011; Larocque et al. 2014; McElree 2006; Nee and Jonides 2013; Oberauer and Hein 2012; Olivers et al. 2011). It is not to be confused with the idea that internal attention mediates maintenance of multiple items in working memory (Chun 2011; Chun et al. 2011; Kiyonaga and Egner 2013). The idea that prefrontal neuronal activity encodes the item at the current focus of attention is an explicit and testable alternative to the idea that it encodes the full contents of working memory.

GRANTS

This study was supported by National Institutes of Health (NIH) Grants R01-EY-024912, P50-MH-103204, K08-MH-080329, and R00-EY-018894 and the Pennsylvania Department of Health’s Commonwealth Universal Research Enhancement Program. Technical support was provided by NIH Grants P30-EY-008098 and P41-RR-03631.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

R.O.K. and C.R.O. conceived and designed research; R.O.K. performed experiments; R.O.K. and M.A.S. analyzed data; R.O.K., M.A.S., and C.R.O. interpreted results of experiments; R.O.K. and C.R.O. prepared figures; R.O.K. and C.R.O. drafted manuscript; R.O.K., M.A.S., and C.R.O. edited and revised manuscript; R.O.K., M.A.S., and C.R.O. approved final version of manuscript.