Sharp emergence of feature-selective sustained activity along the dorsal visual pathway

Abstract

Sustained activity encoding visual working memory representations has been observed in several cortical areas of primates. Where along the visual pathways this activity emerges remains unknown. Here we show in macaques that sustained spiking activity encoding memorized visual motion directions is absent in direction-selective neurons in early visual area middle temporal (MT). However, it is robustly present immediately downstream, in multimodal association area medial superior temporal (MST), and in the lateral prefrontal cortex (LPFC). This sharp emergence of sustained activity along the dorsal pathway suggests a functional boundary between early visual areas, encoding sensory inputs, and downstream association areas, additionally encoding mnemonic representations. Moreover, local field potential oscillations in MT encoded the memorized directions and, in the low frequencies, were phase-coherent with LPFC spikes. This suggests that LPFC sustained activity modulates synaptic activity in MT, a putative top-down mechanism by which memory signals influence stimulus processing in early visual cortex.

Humans and other primates can temporarily encode, store, and keep within the focus of attention, visual information unavailable to the eyes. This ability, known as working memory, allows past experiences to influence our current thoughts and behaviors, and is crucial to cognitive processes such as abstract thinking, decision-making, and action planning¹. During behavioral tasks requiring temporarily memorizing the locations or features of visual objects, such as delayed match-to-sample, neurons in high-order association areas such as the lateral prefrontal (LPFC)^2–3, posterior parietal⁴, and inferotemporal⁵ cortices of macaques, show elevated and sustained spiking activity that encodes the memorized information. This sustained activity in the absence of visual stimulation is considered by many as the neural correlate of working memory^2–6.

It has been recently proposed that because neurons in high-order association areas may lack fine selectivity for single visual features (e.g. direction of motion, color, or orientation), working memory representations of such features must be encoded by feature-selective neurons in early visual cortex^7,8. However, electrophysiological studies in monkeys have reported contradictory evidence. During the delay period of delayed match-to-sample and similar tasks, the activity of early visual neurons has been found to be either weak and transient^9–11, absent¹², or observed while visual stimulation remains inside the neurons’ receptive field¹³. A puzzling finding in functional magnetic resonance imaging (fMRI) studies in humans is that during the same tasks, blood-oxygenation-level-dependent (BOLD) signals in early visual areas remain at baseline levels, yet the memorized features can be decoded from these signals using pattern classification analysis techniques^8,14,15. Currently, it remains highly controversial whether sustained activity encoding memorized visual features is present in early visual areas or, alternatively, it emerges further downstream along the visual pathways.

To examine this issue, we trained monkeys in a delayed match-to-sample task that required memorizing the motion direction of a visual stimulus, and recorded the activity of direction-selective neurons in three serially-connected areas along the dorsal visual pathway: middle temporal (MT) – an early visual area involved in motion processing¹⁶; its immediate downstream neighbor, medial superior temporal (MST)^17,18 – a multimodal association area that integrates visual motion signals from MT with vestibular inputs¹⁹; and LPFC – a high-order association area further downstream that has been classically associated with working memory coding³. We found that sustained spiking activity encoding the memorized motion directions was absent in MT neurons, but robustly present in MST and LPFC.

RESULTS

Behavioral task

We trained two macaque monkeys to perform a delayed match-to-sample task (Fig. 2a). In each trial, a sample random dot stimulus with motion in one of four directions was presented for 1 s. After a variable delay period (1.2 to 2 s), two test stimuli were sequentially presented for 0.59 s, one of which matched the sample motion direction. To obtain a juice reward, the monkeys had to release a button when the matching test was presented. A behaviorally-irrelevant stimulus – which served experimental purposes unrelated to the results reported here – was presented in the opposite hemifield simultaneously with the tests. Because the location of the tests and irrelevant stimulus were randomly swapped from trial to trial, the animals could not predict the tests’ location. Importantly the task required the animals to memorize the sample direction throughout the delay period. In all experimental sessions, the performance of both animals was well above the 50% chance level (mean of 79% for monkey M, 76% for monkey S).

Figure 2. Firing rate across task periods for example neurons in MT, MST, and LPFC.

(a) Visual display during all task periods. (b–g) Mean firing rate (± standard error) over time in trials with each of the four sample directions (color-coded arrows) for neuron examples in MT (a,b), MST (c,d), and LPFC (e,f). Each neuron’s preferred direction is shown in red. Gray area shows corresponding auROC over time (right axis label). In (b), the test stimuli, but not the sample, were placed inside the neuron’s receptive field, and colors during the test period represent test directions.

Sustained activity is present in MST and LPFC but not MT

We recorded the responses of 631 neurons in two monkeys, M and S: 112 in MT (57 in M and 55 in S), 247 in MST (145 in M and 102 in S), and 272 in LPFC (118 in M and 154 in S). MT and MST neurons were identified by their motion direction tuning properties, anatomical location, and receptive field size and position (Fig. 1b, Supplementary Fig. 1). LPFC neurons were recorded from the cortical surface around the posterior end of the principal sulcus corresponding to areas 8 and 46²⁰ (Fig. 1a).

Figure 1. Anatomical location of recorded neurons.

(a) Left, cortical surface showing LPFC (red). Right, location of LPFC recording sites with respect to arcuate (AS) and principal (PS) sulci. (b) Left, coronal MRI section showing MT (green) and MST (blue). Right, MRI section parallel to electrode trajectories (yellow and white lines) for each monkey, showing the location of all recorded neurons in MT (green) and MST (blue) projected onto that section. Bottom, close-up of recorded region. Black lines show gray/white matter boundaries. STS, superior temporal sulcus.

We first examined whether the sample direction was encoded by individual neurons in each of the three areas. A neuron was considered motion direction-selective if its firing rate significantly varied as a function of the sample direction (two-factor ANOVA, see Online Methods) during the sample period (sensory-selectivity), or the delay period, while the animals memorized the sample direction (delay-selectivity). We found that neurons in MT were sensory-selective but not delay-selective (Fig. 2b,c). Surprisingly, immediately downstream from MT, in MST, we found neurons with both sensory- and delay-selectivity (Fig. 2d,e). In LPFC, neurons also showed both types of selectivity (Fig. 2f,g). Interestingly, several MST and LPFC neurons showed strong delay-selectivity but weak or no sensory-selectivity (Fig. 2e,g), suggesting that they better represent a memorized direction than the direction of a present stimulus.

Direction selectivity was found in a total of 109 neurons in MT (97%), 218 in MST (88%), and 86 in LPFC (32%). From these neurons, 100% in MT, 93% in MST, and 70% in LPFC showed sensory-selectivity, while 8% in MT, 36% in MST, and 55% in LPFC showed delay-selectivity (Fig. 3g). Both monkeys showed similar results (Supplementary Fig. 2a,b). All percentages, except the percentage of delay-selective MT neurons, were significantly higher than expected by chance, (permutation tests, P < 0.05). A one-factor ANOVA comparing mean firing rates over each entire task period between sample directions yielded similar results (Supplementary Fig. 2c). All analyses hereafter were performed on sensory-selective and/or delay-selective neurons.

Figure 3. Direction discriminability in MT, MST and LPFC.

(a,c,e) Time periods with significant direction discriminability (auROC) above (blue) and below (red) 0.5 for all neurons in MT (a), MST (c) and LPFC (e). (b,d,f) Average auROC across MT (b), MST (d) and LPFC (f) neurons over time as a function of the percentage of averaged neurons (organized from maximum to minimum auROC in each time bin). Sample and delay periods used for analysis are indicated. (g) Percentage of selective neurons with sensory-selectivity (upright solid bars), and delay-selectivity (inverted hashed bars). Because these percentages are computed from selective neurons, the bar overlaps represent neurons with both sensory- and delay-selectivity. (h–i) Mean auROC (h) and percent bins with significant auROC (i) during the sample and delay among all (light bars) and the top 10% (dark bars) of the neurons selective during each corresponding period. In g–i, dashed white lines show the values expected by chance. Error bars denote standard error of the mean.

To measure how well neurons discriminated between motion directions (discriminability), we computed the area under the receiver operating characteristics curve (auROC) between activity in trials with the sample moving in the neuron’s preferred and least-preferred directions (Fig. 2b–g; Online Methods)²¹. Upon sample onset, discriminability increased in all three areas, but did so more rapidly and reached higher values in MT, followed by MST, and then by LPFC (Fig. 3b,d,f). During the sample period, the discriminability strength (i.e. mean auROC; Fig. 3h) and duration (i.e. time that the auROC remained above chance; Fig. 3a,b,c,i; Supplementary Fig. 3a,b), were higher in MT than MST (strength: unpaired t test, t = 6.16, P ≪ 0.001; duration: unpaired t test, t = 3.34, P ≪ 0.001, t-tests), and higher in MST than LPFC (strength: unpaired t test, t = 5.99, P ≪ 0.001; duration: unpaired t test, t = 5.55, P < 0.01, t-tests). This indicates that the representation of motion direction of a present stimulus is strongest in MT, weaker downstream in MST, and weakest in LPFC.

Immediately after the sample offset, discriminability in MT neurons quickly dropped and remained at chance throughout the delay (Fig. 3a,b). In contrast, in many MST and LPFC neurons, it remained high (Fig. 3c–f). Remarkably, delay period discriminability in MST was as strong as (Fig. 3h; unpaired t test, t = −1.51, P = 0.13) and lasted significantly longer than (Fig. 3i, Supplementary Fig. 3c,d; unpaired t test, t = 2.20, P = 0.03, t-test) in LPFC, persisting throughout the entire delay in 22% of MST and 11% of LPFC delay-selective neurons. There was a small but significant decrease in discriminability by the end of the longest delay (2 s) in LPFC (0.036 decrease, P = 0.04, paired sample t test, t = 1.90) but not MST (0.014 decrease, P = 0.08, paired-sample t test, t = 1.42). These results indicate that sustained activity in MST encoded the memorized direction more robustly over time than in LPFC.

In contrast, in the few MT neurons classified as delay-selective, discriminability was only weak and brief (Fig. 3h,i; Supplementary Fig. 3c,d), and was not significantly different in strength (unpaired t-test, t = 0.76, P = 0.45) or duration (unpaired t-test, t = 1.04, P = 0.30) from that of neurons in which the test stimuli, rather than the sample, were placed inside the receptive field (Fig. 2c). Likewise, out of 12 MT neurons recorded with the sample and test at the same location inside the receptive field, none was delay-selective, and their mean discriminability was 0.54, as for all other MT neurons. In contrast, out of 30 MST neurons recorded with the sample and test inside the receptive field, 50% were delay-selective; their mean discriminability was 0.64, the same value as for all other delay-selective MST neurons. Thus, sustained activity encoding the memorized motion directions was present in MST and LPFC, but not in MT.

To estimate how strongly the activity of neuronal populations in each area encoded the sample direction, we performed linear discriminant analysis (LDA; Online Methods). In contrast to ROC analysis, which measured the ability of each neuron to discriminate between two of the four sample directions (preferred and least-preferred), LDA estimates the ability of an entire population of recorded neurons to discriminate among the four directions (decoding accuracy).

In both MST and LPFC, population decoding accuracy increased after sample onset, remaining well above chance and relatively stable throughout the entire delay period (Fig. 4). Interestingly, decoding accuracy was significantly higher in MST than LPFC not only during the sample presentation, but also throughout all of the delay (Bonferroni-corrected t-test), indicating that the memorized directions were more strongly represented by the population of neurons in MST than in LPFC. In MT, decoding accuracy during the sample period was higher than in MST and LPFC. However, after the sample offset, it quickly dropped and remained at chance for most of the delay period. Therefore, the memorized directions were robustly encoded in the population activity in MST and LPFC, but not in MT.

Figure 4. Direction decoding accuracy for the populations of MT, MST and LPFC neurons.

Mean (± standard error) sample direction decoding accuracy (percent of correctly decoded trials) over time for the neuronal populations in MT (green), MST (blue) and LPFC (red). Dashed lines, decoding accuracy expected by chance (see Online Methods). Horizontal color bars show periods with significant decoding accuracy for each area.

Interestingly, decoding accuracy in MT showed a brief and weak rebound after the sample offset, which coincided with the occurrence of a transient selectivity inversion in some neurons (lower responses to the preferred direction than to the least-preferred direction; Fig. 3a). This effect has been reported following visual stimulation in tasks with no memory requirements or during anesthesia, and is likely a consequence of rapid neural adaptation^22,23. Periods of inverted selectivity were also present in some MST neurons, but were observed throughout the entire delay (Fig. 3c). In LPFC, this phenomenon was rarely observed (Fig. 3e).

Sustained activity is linked to task performance

A hypothesis derived from our results is that the direction-selective sustained activity of MST and LPFC neurons played a role in the animals’ performance of the working memory task. This hypothesis predicts that delay period discriminability should be reduced in error relative to correct trials, and that trial-to-trial variations in sustained activity should correlate with the animals’ performance.

For the example MST and LPFC neurons in Fig. 5a and b, delay activity in trials with the sample in the neurons’ preferred direction was higher in correct than error trials. As a consequence, the difference in activity between preferred and least-preferred sample trials was larger in correct than in error trials. Across neurons and in both areas, the difference in auROC between correct and error trials was significantly higher than 0, indicating that discriminability during the delay period was reduced in error trials relative to correct trials (one-sample t tests; MST: t = 2.12, P = 0.02, LPFC: t = 5.37, P < 0.001). Interestingly, this reduction was significantly larger in LPFC than MST (Fig. 5c–e; unpaired t test, t = 1.79, P = 0.038). This suggests that the animals’ performance was linked to how strongly MST and LPFC neurons encoded the memorized direction, and that this link was stronger in LPFC than in MST.

Figure 5. Relationship between task performance and delay period direction discriminability of MST and LPFC neurons.

(a,b) Mean firing rate of example MST (a) and LPFC (b) neurons over time in correct (green) and error (orange) trials with preferred (solid lines) and least-preferred (dashed lines) sample direction. Colored areas show difference in activity between preferred and least-preferred sample trials. (c,d) Delay period auROC of delay-selective neurons in MST (c, blue dots) and LPFC (d, red dots) in correct (horizontal axis) vs. error trials (vertical axis). Gray, identity line. (e) Mean difference (± standard error) in delay period auROC between correct and error trials (ΔauROC = auROC_correct − auROC_error) across MST and LPFC neurons. *, P < 0.05.

We then examined whether trial-to-trial variations in sustained activity correlated with the animals’ performance and whether this relationship depended on each neuron’s direction preference with respect to the memorized sample direction. We compared the distributions of firing rates during the delay period between correct and error trials using choice probability analysis²⁴. This was done independently for trials with the sample moving in the neurons’ preferred, least-preferred, and intermediate directions (Online Methods). Choice probability was significantly higher than expected by chance in 25% of LPFC neurons and 14% of MST neurons, and significantly lower in 0% of LPFC neurons and 4% of MST neurons (Fig. 6a,b; permutation test, P < 0.05). Across all neurons, mean choice probability during the delay period in preferred-sample trials was significantly higher than chance in both areas (one-sample t tests; MST: t = 3.53, P < 0.001; LPFC: t = 4.76, P < 0.001), and was significantly higher in LPFC than in MST (Fig. 6a–d; unpaired t test, t = 2.10, P = 0.02). In both areas, mean choice probability across significant neurons remained significantly above chance throughout most of the delay period (Supplementary Fig. 4). These results show that variations in the neurons’ sustained activity correlated with the animals’ performance in both areas, but more so in LPFC than MST. Lastly, in LPFC but not MST, mean choice probability decreased significantly as a function of the difference between the neurons’ preferred direction and the memorized direction (Fig. 6c–d; repeated-measures ANOVA; LPFC: F = 4.78, P = 0.01; MST: F = 2.41, P = 0.09). This suggests that the link of each LPFC neuron to behavior depends on the similarity between its preferred feature and the memorized feature.

Figure 6. Choice probability of delay activity in MST and LPFC neurons.

(a,b) Frequency histogram of delay period choice probability among delay-selective MST (a) and LPFC (b) neurons in preferred-sample trials. Vertical black dashed lines show chance choice probability. Color tones represent neurons with choice probability significantly above (darker tones), significantly below (lighter tones) and not significantly different from (middle tones) that expected by chance. (c,d) Mean choice probability (± standard error) among delay-selective MST (c) and LPFC (d) neurons as a function of sample direction with respect to each neuron’s preferred direction. *, P < 0.05; ns, non-significant.

LFP power in MT encodes memorized directions

Given the existence of feedback projections from MST and LPFC to MT^18,25, it may appear surprising that the sustained activity of MST and LPFC neurons does not cause firing rate increases in MT neurons during the delay period. One possible explanation is that during this period, feedback signals from MST and/or LPFC modulate synaptic activity in MT, sufficiently to cause changes in local field potentials (LFP) that reflect the memorized directions, but not strongly enough to cause increases in neuronal firing.

Indeed, in many MT recording sites, such as the example in Fig. 7a, power in several LFP frequencies differed in response to different memorized sample directions. This effect was not due to residual activity (spikes or LFPs) caused by the sample, since it was observed not only in sites recorded with the sample inside the receptive field, but also in sites recorded with the sample outside, such as that shown in Fig. 7a. For each frequency band²⁶ (Online Methods), we measured the delay period direction discriminability of the LFP power in each MT site (i.e., the auROC between power in preferred and least-preferred sample trials). In all bands, the percentage of sites for which the auROC was significantly higher than expected by chance ranged between 14% and 22% (Fig. 7b; permutation test, P < 0.05). These values were significantly higher than the percentage of false positives (population permutation test, P < 0.05). The mean auROC across significant sites in all frequency bands ranged between 0.64 and 0.67, and were all significantly above the values expected by chance (permutation test, P < 0.05, Fig. 7c). These results indicate that in MT, the amplitude of LFP oscillations reflects the memorized directions, despite the absence of direction-selective sustained spiking activity.

Figure 7. Direction discriminability of LFP power in MT during working memory.

(a) Mean normalized LFP spectrogram of an example recording site in MT during trials with preferred (top) and least-preferred (bottom) sample directions. Black horizontal bars and dashed lines delineate sample period. Frequency band ranges are color-coded. (b,c) For each frequency band, percent LFP sites in MT for which the LFP power auROC in the delay period was significantly higher than expected by chance (b), and mean LFP auROC (± standard error) among selective sites (c).

We conducted the same analysis in MST and LPFC. In both areas, the percentage of sites with delay period direction discriminability in the LFP power was higher than the percentage of false positives for all frequency bands. These values, as well as the mean auROC across significant sites, were similar to those in MT (Supplementary Fig. 5). Thus, in MST and LPFC, both sustained spiking activity and LFP power encoded the memorized motion directions.

Spike-field synchrony between LPFC and MT during the delay

To examine whether the LFP activity observed in MT during the delay period was driven by feedback signals from LPFC, we measured phase coherence between simultaneously-recorded spikes from LPFC neurons and LFPs from MT recording sites. For the example pair (LPFC neuron – MT site) shown in Fig. 8a, coherence was above that expected by chance in the low frequencies (randomized surrogates test, P < 0.01), particularly in the beta band (12–25 Hz). Significant coherence was observed in 12.5% (14 of 112) of the recorded LPFC-MT pairs. For all of these pairs, coherence was high and reached significance in the low frequency bands (theta, alpha, and beta; Fig. 8b,c), peaking in the range of 7–15 Hz. The same analysis was repeated on pairs showing significant coherence after shuffling the trial order, which destroyed the trial-to-trial simultaneity of LFP and spike signals. The coherence peak in the lower bands was completely absent (Supplementary Fig. 6), showing that such peak was due to the real-time interaction of LFP and spike signals, and was not an artifact of any other property of these signals.

Figure 8. Spike-field synchrony between LPFC and MT during working memory.

(a) Spike-field phase coherence (shaded area) of example LPFC-MT pair as a function of LFP frequency; mean coherence among randomized surrogates (black line) and confidence limit of P < 0.01 computed from surrogates (red line). (b) Spike-field coherence as a function of LFP frequency for all significantly coherent pairs, sorted by frequency at peak coherence. (c) Among coherent pairs, percent reaching significant coherence at each frequency. Inset, phase of coherence for all coherent pairs in the beta band (in degrees). Arrow indicates mean phase across pairs. Frequency bands are color-coded.

In the theta, alpha, and beta bands, the phases of coherence among significantly coherent pairs were non-uniformly distributed along the LFP oscillatory cycle (Rayleigh test, P < 0.05), showing the maximal concentration in the beta band (concentration parameter κ = 1.64), mostly within the rising phase of the cycle (Fig. 8c inset; mean phase = 51°, significantly different from 0°; one-sample test for circular data, P = 0.005). Thus, the phase relationship between LPFC spikes and LFP oscillations in MT was consistent across pairs.

Lastly, we examined whether there was a relationship between LPFC-MT synchrony and the animals’ task performance. Indeed, among synchronous pairs, the percentage with significant coherence was decreased in error relative to correct trials by 54% in the theta, 64% in the alpha, and 65% in the beta band (Fig. 8c). Our results show that during the delay period, the spiking activity of a proportion of LPFC neurons was synchronized with low-frequency LFP oscillations in MT, and that lower task performance was associated with reduced synchrony. Coupling of low-frequency oscillatory activity between cortical areas during similar tasks has been previously observed between prefrontal and ventral visual cortex²⁷ and between prefrontal and parietal areas²⁸.

DISCUSSION

Emergence of feature-selective sustained activity

Our study demonstrates that while monkeys memorized the motion direction of a stimulus, sustained spiking activity encoding the memorized direction was absent in early visual area MT. Together with previous studies reporting weak, transient, or no coding of memorized features in early visual neurons during similar tasks^9–12, our results argue against the hypothesis that neurons in early visual cortex are involved in the coding of working memory representations of single visual features. Our observation that MT neurons robustly encoded the sample direction exclusively during the sample period suggests that spiking activity in early visual areas mainly encodes the current visual input.

Given the direct feedforward and feedback connectivity between MT and MST¹⁸, it was surprising that feature-selective sustained activity was robustly present in MST despite being absent in MT. Our results indicate that along the dorsal visual pathway, this sustained activity sharply emerges as a de novo property of MST neurons. Moreover, we found that some neurons had sensory- but not delay-selectivity, others had delay- but not sensory-selectivity, and others had both (Fig. 3g). This indicates that information about present and past stimuli remains segregated in some neurons and coexists in others, and suggests that along the dorsal visual pathway, the transformation of sensory representations of motion direction into mnemonic representations occurs within the MST circuitry. Such transformation may also involve LPFC, since neurons in this area also showed heterogeneity of coding.

The observation that the memorized motion direction was encoded by neurons in LPFC is further evidence that, in addition to locations² and complex objects³, these neurons also encode mnemonic representations of single visual features. working memory coding of visual motion speed gradients²⁹ and vibrotactile stimuli³⁰ has also been observed. On the other hand, our observation seems at odds with results from human functional imaging experiments showing the inability to decode the contents of working memory using multivoxel pattern classification analysis of BOLD signals recorded from LPFC¹⁵. This may be explained by considering that LPFC lacks a topographical organization of feature coding at a scale that can be detected with the coarse resolution of functional imaging.

Consistent with our results, a previous study¹⁰ found no sustained activity in MT neurons during a delayed match-to-sample task for motion direction, and observed only brief and transient activity mostly restricted to the first few hundred milliseconds after sample offset. This is likely due to residuals of the response to the sample, given the observations that sensory responses are followed by a brief period of decay towards baseline^31,32 and by rapid neural adaptation^22,23,9, even during tasks that do not require memory or during anesthesia^22,23,31,32.

The same study¹⁰ reported that no LPFC neurons showed sustained activity persistently encoding the memorized directions. They proposed that in LPFC, as well as in MT, brief and transient instances of weak coding in individual neurons during the delay period could be integrated, yielding a population code that reliably represents the memorized directions over time. However, such mechanism might not be necessary, given our finding that in both MST and LPFC, a substantial number of neurons robustly encoded the memorized directions throughout the entire delay. Moreover, population decoding accuracy in MT dropped to chance during the delay period, suggesting that a representation of the memorized directions is absent even when integrating the activity of multiple MT neurons (Fig. 4).

Another study reported that the firing rate of neurons in early visual area V1 was modulated by the memorized location of a motion-defined figure previously presented on a moving background texture¹³. This effect, however, is different from the sustained activity reported here in the absence of visual stimulation, since it required the background texture to constantly stimulate the neuron’s receptive field during the delay period.

One issue that has been a matter of debate is whether the sustained activity of LPFC neurons reflects the allocation of attention rather than the maintenance of memories³³. The sustained activity we found in MST and LPFC may reflect either the maintenance of mnemonic representations of motion direction or the allocation of attention to (or monitoring of) such representations²⁰. These two different functions have been considered as components of the broader construct of working memory¹. Our behavioral paradigm was not designed to discriminate between them. Further electrophysiological studies using retro-cueing³⁴ tasks are needed to clarify this issue.

Role of MST and LPFC sustained activity in working memory

A puzzling finding in our study was that neurons in both MST and LPFC redundantly encoded memorized motion directions. Redundancy has also been observed between parietal and prefrontal cortices during memory for visuospatial locations³⁵ and visual feature categories³⁶, and between inferotemporal and prefrontal cortices during memory for complex objects³. One likely explanation for this redundancy is that the maintenance of working memory representations results from the coordinated activity of an interconnected network of brain areas, each of them playing a different and complementary role. For example, evidence of different roles of the lateral intraparietal (LIP) area and LPFC in the filtering of distracting information during working memory tasks was recently reported³⁷. Delay activity in LPFC neurons is more robust to distractors than in LIP. We did not test the effect of distracting information during the delay period on MST and LPFC neurons, but it is likely that activity in MST is more sensitive to distractor interference than in LPFC.

One possible scenario is that MST maintains a robust representation of the sample direction, which can be “read out” by LPFC and integrated with signals encoding reward value and the allocation of attention in order to produce a meaningful behavioral response. Consistent with this hypothesis, LPFC neurons encode the behavioral choice in a delayed match-to-sample task²⁹. In our study, representations of the memorized direction in MST neurons lasted longer than, and at the population level were stronger than, in LPFC, challenging the view of LPFC as the major contributor to working memory maintenance³. However, variations in sustained activity were more predictive of task performance in LPFC than MST, consistent with the notion that activity becomes more closely linked to behavior further downstream along the chain of visuo-motor processing³⁰. However, it may also reflect the influence of attention and other variables in choice probability measurements, which may be heterogeneous across different brain areas³⁸.

Modulation of LFPs in area MT

Despite the absence of sustained spiking activity in MT, we found that the memorized direction was encoded in the amplitude of LFP oscillations. Given that LFPs represent the overall synaptic activity around the recorded site³⁹ (due to feed-forward, feedback, and local neuronal interactions), and given the absence of feed-forward visual inputs into MT neurons during the delay period of our task, our results suggest that this effect may be due to feedback signals from MST or LPFC²⁵, sufficient to modulate MT synaptic activity, but insufficient to increase firing rates. Dissociations between LFP and spiking activity are now established^12,40. If functionally relevant, the information reflected in the LFPs of MT must exit the area and hence must be observable in spiking activity. Since it is not observed at the single-unit level, it might be manifested in spike synchrony⁴¹.

Supporting this hypothesis, during the delay period, spikes of LPFC neurons were phase-coherent with low-frequency LFP oscillations in MT. Moreover, in agreement with a previous report²⁷, when LPFC-MT synchrony was reduced, the animals were more likely to subsequently produce an incorrect response. This synchrony may represent a top-down mechanism by which sustained activity in higher-order areas selectively modulates the responses of early visual neurons to incoming sensory inputs, biasing visual perception^42–45. Supporting this hypothesis, during a delayed-match-to-sample task for motion direction, MT neuron responses to the test stimulus are influenced by the direction of the remembered sample⁴². Whether the same phenomenon is present in our data will require an analysis of the test responses as a function of the sample direction.

Importantly, coding of the memorized direction in the LFPs from MT may explain why the contents of visual working memory can be decoded from BOLD signals recorded in human early visual cortex using multivoxel pattern classification analysis^8,14,15. Given that the amplitude of BOLD signals correlates with LFP amplitude in the absence of changes in neuronal firing rates⁴⁰, these fMRI studies have likely measured a BOLD correlate of the LFP modulation we observed in early visual cortex^8,46.

Implications for mechanisms and models of working memory

Current models of working memory networks propose that recurrent excitatory connections in high-order association areas such as LPFC underlie sustained spiking activity^47,48,49. Our results suggest that such mechanisms may operate as early as in MST. In contrast, in early visual areas such as MT, strong feedforward inputs from upstream areas, relatively weak feedback inputs from downstream areas, and stronger lateral inhibitory interactions between neurons⁵⁰, may cause spiking activity to increase vigorously in response to retinal inputs but rapidly decay in their absence. Alternatively, it may be that the cortical architecture characteristic of LPFC is not present in MST, and sustained activity in MST arises via LPFC feedback. However, visual responses to the sample arise in MST earlier than in LPFC, and persist throughout the delay period, suggesting that sustained activity may be intrinsically generated in MST rather than inherited from LPFC, and may instead be transferred from MST to LPFC through feedforward inputs.

Our results indicate that the properties of the cortical architecture in MST that allow neurons to generate sustained activity during working memory are absent immediately upstream, in MT. This suggests a sharp transition in cortical architecture between MT and MST, which may also be present at similar processing stages along other sensory processing streams^5,30. We propose that the boundary between the two architectures may be important for the brain to distinguish representations of current sensory experiences from those imagined or memorized, a function that is impaired in schizophrenia and other hallucinatory mental disorders.

ONLINE METHODS

Animals

Two adult male rhesus monkeys (Macaca mulatta), 10 and 11 years old and weighing 8 and 9 kg, participated in the experiments. The monkeys were rewarded with fruit juice for correctly performing each task trial (300 to 600 ml daily). At the end of each training and recording session, they also received fruits additionally to their daily food ration. We measured their body weights daily to ensure stable health conditions. All animal procedures complied with the Canadian Council of Animal Care guidelines and were approved by the McGill University Animal Care Committee.

Visual Stimuli

Visual stimuli were generated using custom-made software on an Apple G4 computer, and were back-projected onto a screen using a NEC WT610 video projector (1024 × 760 pixels resolution, 85 Hz refresh rate). The monkeys were positioned 57 cm away from the screen. Sample and test stimuli were composed of random dots moving linearly with 100% coherence within a virtual circular aperture (13 cd/m² dot luminance contrast, 0.17° dot size, density of 4 dots/degree²). High motion coherence ensured that encoding of the sample direction in working memory was not ambiguous. The dots’ speed was matched to the preferred speed of the recorded neuron, between 2 and 32 degrees/s. The motion directions of the sample and tests were chosen from a set of four orthogonal directions and aligned so that one of them matched the neuron’s preferred direction.

Behavioral task

We trained monkeys to perform a delayed match-to-sample task (Fig. 2a). During each trial, the monkey maintained gaze on a white fixation square (size 0.25° × 0.25°) at the center of the screen and pressed a button to initiate the trial. After 470 ms, a sample stimulus with motion in one of four orthogonal directions was presented for 1000 ms. The sample was then removed and after a delay period of variable duration (from 1200 to 2000 ms), two test stimuli were sequentially presented for 590 ms each, with 590 ms in between. To receive a juice reward, the monkey was required to release the button during the presentation of a test with the same motion direction as the sample. This occurred in half of the trials for each test. If the monkey failed to do so, the trial was immediately terminated without a reward.

A set of four directions, from which the sample and test directions were chosen, was used for each recorded neuron (often differing from neuron to neuron within the same day). Thus, the monkeys could neither use long-term memory representations of these directions nor simply learn four fixed categories across sessions to solve the task. In trials in which the first test did not match the sample, the direction of this stimulus was randomly chosen from the other three directions in the set. A behaviorally irrelevant stimulus with 0% coherent motion and lower luminance contrast was presented simultaneously with the two tests on the opposite hemifield. The test and irrelevant stimulus locations were randomly swapped from trial to trial, preventing the animal from predicting the test location prior to its presentation. The irrelevant stimulus also served other experimental purposes unrelated to the results presented here. The variable delay prevented monkeys from anticipating the timing of the test onset.

Eye Positions

We sampled eye position signals at a frequency of 200 Hz using a video-based eye tracker system (Eye Link 1000, SR Research, Kanata, Ontario, Canada). Monkeys were allowed to start a trial only when their gaze position fell within 1° from the fixation point center. The trial was terminated without a reward if gaze position moved outside this area before the end of the trial.

Surgical preparation of the monkeys

Monkeys were implanted with titanium head posts that stabilized the head during recordings, and with two circular Cilux recording chambers 20 mm in diameter (Crist Instruments, TX, USA). One chamber was positioned on top of a circular craniotomy of the frontal bone that provided access to the right lateral prefrontal cortex (LPFC), specifically the region anterior to the arcuate sulcus, posterior and around the principal sulcus (centered at 30 mm anterior and 17 mm lateral in stereotactic coordinates). The other chamber was implanted on top of a craniotomy of the occipital bone, with its border 2mm anterior to the occipital ridge and 2mm lateral of the sagittal suture. The angle between the chamber’s vertical axis and the horizontal plane was 20°. Electrode trajectories reached areas MT and MST in the right hemisphere according to MRI reconstructions (Fig. 1).

Anatomical localization of recording sites

An MRI scan was conducted on each monkey before the surgery in order to guide the positioning of the chamber. After chamber implantation, a plastic grid (Crist Instruments, TX) was positioned on top of each recording chamber and glass capillaries filled with mineral oil were positioned parallel to electrode trajectories at five different positions in the grid. An additional MRI was conducted to precisely locate the brain areas of interest with respect to the electrode trajectories. LPFC neurons were recorded by placing the electrode tip in positions around the principal sulcus, anterior to the arcuate sulcus. To record from MT, we positioned the electrode tip ventral and posterior to the superior temporal sulcus. To record from MST, the electrode tip was placed dorsal and anterior to the superior temporal sulcus. A reconstruction of the recording sites shows a clear segregation in the locations of the recorded MT and MST neurons, separated by the superior temporal sulcus (Fig. 1b). The recorded MST neurons were located within the area described as MSTd.

Electrophysiological recordings

During each recording session, we made transdural penetrations with standard epoxy-insulated extracellular tungsten electrodes (FHC Inc, Bowdoin, ME, USA; shank diameter = 500 μm in LPFC and shank diameter = 125 μm in MT and MST; impedance = 2–4 MΩ at 1 kHz). For LPFC recordings, a blunt guide tube positioned 5–10 mm from the recording electrode(s) served as the reference. For MT and MST recordings, a guide tube was lowered through the posterior craniotomy until penetrating the dura, and the electrode was the lowered through the guide tube until the desired depth. During each session, we recorded with one electrode placed in MT or MST and/or with one to four electrodes simultaneously in LPFC (separated by at least 2 mm). A Plexon data acquisition system (MAP) was used to record, store and sort spike and LFP data (Plexon Inc., Dallas, TX, US A)²¹. LFP signals were band-pass filtered between 0.7 and 170 Hz and sampled at 1kHz.

Characterizing spatial and motion tuning

The spatial and motion tuning properties of MT and MST neurons were characterized during trials in which the monkey responded to a contrast change in the fixation point while random dot stimuli with different locations, sizes, linear and spiral motion directions and speeds were presented in the visual periphery⁴⁴. MT neurons were identified based on their linear motion direction selectivity, receptive fields size and laterality (contralateral to the recorded hemisphere). MST neurons were identified based on their linear and spiral motion direction selectivity and receptive field size and position (considerably larger than for MT neurons and often spanning both hemifields). The distributions of receptive field sizes of MT and MST neurons were clearly segregated (Supplementary Fig. 1), and the percentage of neurons with receptive fields including ipsilateral regions was far higher in MST (56%) than in MT (12%).

We positioned the sample inside the MT or MST neuron’s receptive field and chose four orthogonal sample directions with one of them matching the neuron’s preferred direction. As a control, 70 MT neurons were recorded with one of the two locations of the test stimuli, but not the sample location, inside the receptive field. 12 MT and 30 MST neurons were recorded with the sample and tests at the same location inside the receptive field.

Because the activity of LPFC neurons is highly task-dependent, we did not use the mapping task to characterize their response properties. Instead, we recorded while the monkey performed the delayed match-to-sample task immediately after isolating a neuron. It has been shown that most LPFC neurons have large receptive fields and show space-independent responses to visual motion¹⁰. We chose the sample position and directions to match the MT neuron’s receptive field properties.

Data analysis

All analyses (except error trial analyses) were conducted in data recorded in correctly performed trials. Results obtained from both monkeys were qualitatively similar. Therefore, all neurons from both monkeys were pooled together for analysis. We analyzed neuronal activity during the sample and delay periods. However, one possible confounder in our paradigm is the fact that sensory neurons show residuals of the sensory response during the first few hundred milliseconds after stimulus offset. Two factors contribute to this phenomenon: first, a stimulus response takes a brief period of decay after stimulus offset while it returns to baseline firing rate^31,32; second, in many neurons, sensory stimulation causes a brief period of neuronal adaptation (see discussion). In order to avoid such confounder, all analyses of delay period activity excluded data from the first 240 ms after the sample offset. Similar percentages of delay-selective neurons were obtained by excluding the first 480 ms instead of 240 ms of the delay period (Supplementary Fig. 2d). Thus, the presence of sustained activity in MST and LPFC, but not in MT, was not dependent on the precise criterion of time exclusion from the delay period.

Given that the duration of the delay period varied across trials between 1200 and 2000 ms, we only analyzed delay period activity until 1200 ms after the sample offset. To assess whether each neuron encoded the sample direction over time, we tested for significant differences in firing rates between trials with different sample directions across time bins of 120 ms using a mixed between-within two-factor analysis of variance (ANOVA) with sample direction as a “between-subjects” factor and time bin as a “within-subjects” factor. Each neuron was classified as direction-selective if it had a significant (P < 0.05) main effect of direction in at least one of two ANOVAs: one using time bins from the sample presentation period (sensory-selectivity) and the other using time bins from the delay period (delay-selectivity).

To determine the percentage of sensory-selective and delay-selective neurons that would be expected by chance, we randomly shuffled, for each neuron, the sample direction labels of all trials. We then performed the same two-factor ANOVA to obtain a surrogate percentage of sensory-selective and delay-selective neurons. The analysis was repeated 500 times to obtain 500 surrogate values of percentage of selective neurons. Percentages were considered significantly higher than chance if they were ranked within the top 95^th percentile among all 500 surrogate values.

To test whether in some neurons, especially in area MT, the memorized direction could be represented very weakly in each individual time bin but more strongly in the average response across the entire delay, we conducted a one-factor ANOVA using the mean firing rate across each entire period (baseline, sample and delay). However, the percentages of neurons with a significant main effect of motion direction during the sample period (sensory-selectivity) and delay period (delay-selectivity) were similar to those obtained with the two-factor ANOVA (Supplementary Fig. 2c).

Receiver operating characteristics (ROC) analysis

To quantify neurons’ ability to discriminate between sample motion directions, we performed Receiver Operating Characteristics (ROC) analysis. For each neuron, we computed the area under the ROC curve (auROC) to measure the separability of the distributions of firing rates between all possible pairs of sample directions. The auROC was computed across a sliding time window of 200 ms shifted by increments of 40ms. auROC values between 0 and 0.5 were rectified to their corresponding values in the range between 0.5 and 1. Among all pairs of sample directions, the two directions for which the mean auROC across the sample and delay periods was highest were chosen as preferred and least preferred directions. The auROC computed between these two sample directions was used to measure each neuron’s direction discriminability and to compute population averages (Figs. 2 and 3).

To test whether auROC values were significantly higher than expected by chance, we performed a permutation test in which we shuffled preferred-sample and least preferred-sample trial labels and computed the auROC between the shuffled trials. This procedure was repeated 500 times. An auROC was considered significant if it reached or exceeded the 99^th percentile of the distribution of the 500 shuffled surrogates. This was performed through all steps of the sliding time window to detect periods of significant direction discriminability. These periods were further separated into those in which the unrectified auROC values were above and below 0.5 (Fig. 3a,c,e).

For each neuron, we added the duration of all significant bins to obtain the total duration of discriminability in each task period (Fig. 3i, Supplementary Fig. 3a,c); to obtain the maximum duration of consecutive discriminability in each task period, we identified the time segment with the maximum number of consecutive significant bins served to measure the duration of consecutive discriminability (Supplementary Fig. 3b,d). For each task period, the average of the 500 surrogate auROC values for each neuron was used to compute the mean auROC across neurons that would be expected by chance (Fig. 3h).

auROC in correct vs. error trials

For each delay-selective MST and LPFC neuron, the aforementioned ROC analysis was repeated on error trials, as well as on a randomly chosen down-sampled set of correct trials that matched the number of error trials. We then computed the mean auROC across the delay period independently for correct and error trials (Fig. 5c,d), and then computed the difference in auROC between them, ΔauROC (auROC_correct − auROC_incorrect; Fig. 5e). We tested whether the mean Δ auROC across neurons was significantly different from 0 using a one-sample t-test. We compared mean Δ auROC between MST and LPFC using a two-sample t-test.

Population decoding analysis

To estimate how strongly the activity of the population of recorded neurons in each area encoded the sample direction over time, we performed linear discriminant analysis (LDA). For each area, we generated a pseudo-population of neurons. This procedure is commonly used to estimate the coding ability of neuronal populations. It is important to note that because neurons were not recorded simultaneously, the resulting estimates of decoding accuracy are approximations of the coding ability of each population.

LDA was performed on the firing rates of neurons across a sliding window of 200 ms in steps of 40 ms using a leave-one-out cross-validation method to decode the sample direction. We excluded neurons with less than 30 trials per sample direction. The number of neurons used for the analysis from each area was randomly down-sampled to match that of the area with the lowest number. Before training, we z-scored all firing rates and performed feature pre-processing on the training set of trials at each time window, consisting of an ANOVA to select neurons with a significant main effect of sample direction on the firing rates in that window.

For each brain area, we randomly selected 30 trials per condition from each neuron. Trial simultaneity among neurons was then randomly assigned between trials of the same condition. Decoding accuracy was computed as the percentage of trials for which the sample direction was correctly decoded. The above procedure was repeated 50 times, randomly reassigning trial simultaneity each time. At each time window, we then computed mean decoding accuracy and standard error across all repetitions (Fig. 4). To compute chance levels of decoding accuracy, we performed the above procedure after randomly shuffling sample direction labels between all trials. At each time window, significance of decoding accuracy was tested by comparing the distribution of decoding accuracy values with those expected by chance using a t-test (Bonferroni-corrected for multiple time window comparisons). Decoding accuracy was compared between MST and LPFC with a t-test between the distributions of accuracy values of the two areas at each time window (Bonferroni-corrected for multiple time window comparisons). A comparable analysis using support vector machine instead of LDA yielded similar results.

Choice probability

For each neuron, we computed choice probability as the auROC between the mean firing rates during the delay in correct and error trials, separately for trials with the sample in the preferred, least-preferred, and intermediate directions (averaging across these latter directions). One-sample t-tests (one-tailed) were used to test whether mean choice probability across neurons in each area was significantly higher than 0.5 in trials with preferred, intermediate, or least-preferred sample directions (Fig. 6c,d).

To test whether each neuron’s choice probability was significantly higher or lower than expected by chance, the behavioral outcomes (correct or error) of all trials were randomly reassigned and a surrogate choice probability value was computed. We repeated this 500 times to obtain 500 surrogates. If the neuron’s choice probability was in the top or bottom 2.5 percentile (α= 0.05), it was considered significant. For each neuron with significant choice probability, we computed the time course of choice probability across a sliding window of 200 ms in steps of 40 ms. At each time step, we then computed the mean and standard error of choice probability across these neurons (Supplementary Fig. 4).

To test whether the relationship between the memorized sample direction and the neurons’ direction preference had a significant effect on choice probability, we performed a one-factor repeated-measures ANOVA with choice probability as the dependent variable and sample direction as a factor with three levels (sample in preferred, intermediate or least preferred directions), testing the simple main effect of sample direction on choice probability across neurons.

Local field potentials

Spectral analyses of LFP data were performed using multitaper methods in the Chronux Toolbox for Matlab (number of tapers k = 3, time-bandwidth product TW = 2, frequency range of 1 – 140 Hz, see http://www.chronux.org). We divided the LFP frequency spectrum into bands: theta (θ, 4 – 8 Hz), alpha (α, 8 – 12 Hz), beta (β, 12 – 25 Hz), low gamma (γ_L, 25 – 55 Hz), and high gamma (γ_H, 65 – 135 Hz)²⁶. Increasing the high gamma range to 250 Hz did not change any of the results. Recording sites with less than 15 trials per sample direction were excluded from analysis. For each recording site, single trial power spectrograms were computed over the entire delay period (240–1200 ms after sample offset). LFP power was normalized by dividing it by the mean power during the fixation period preceding the sample presentation. We excluded trials with mean normalized power in any band exceed 3 standard deviations of the power distribution across trials. These were rare, likely caused by occasional movements of the animal. Within each frequency band, ROC analysis was performed using the mean normalized power during the delay period in individual trials. The two sample directions yielding the highest auROC were selected as preferred and least preferred directions, and were used to measure the direction discriminability of LFP power (Fig. 7). To graphically display the time course of LFP power for the example LFP site in Fig. 7a, the power spectrogram was computed across a sliding window of 240 ms in steps of 40 ms.

For each band, we tested for significant discriminability of each LFP site by comparing the real auROC against 500 shuffled surrogate values (permutation test, α = 0.05; Fig. 7b). To obtain the false positive rate of significant discriminability (i.e. the percentage of significant sites expected by chance) for each area and frequency band, we repeated the above analysis replacing the auROC of each site with a randomized surrogate, and computed the percentage of significant sites. We repeated this process 500 times and averaged the resulting surrogate percentages to obtain a mean percentage expected by chance (Fig. 7b and Supplementary Fig. 5a,b, white dashed lines). For each brain area and frequency band, the percentage of sites with direction discriminability was considered significant if it exceeded the 95^th percentile of the surrogate values. To compute the mean auROC among significant sites expected by chance, we averaged all surrogate auROC values for each site, and computed the mean across all significant sites (Fig. 7c and Supplementary Fig. 5c,d, white dashed lines).

Spike-field phase coherence

Spike-field phase coherence analysis was performed using multitaper methods in the Chronux toolbox and the same parameters described above. For each trial, phase coherence between the LFPs from an MT site and simultaneously recorded spikes from each LPFC neuron during the delay period was computed as a function of LFP frequency, averaged then across all trials (Fig. 8a,b). Spike-field pairs with less than 50 trials were excluded from analysis. We computed surrogates of coherence by randomly shuffling trial labels of LFP data while keeping the same labels for the spike data. This procedure was repeated to obtain 500 surrogates of coherence. The mean and 99^th percentile of all surrogates as a function of LFP frequency was computed using a sliding 5-Hz window in steps of 1 Hz. Coherence was considered significant in frequencies reaching the 99^th percentile of the surrogates’ distribution (Fig. 8a,c). Each LPFC-MT pair was classified as significantly coherent if coherence reached significance in a range of at least 5 Hz.

To correct for multiple comparisons resulting from testing significance using a sliding frequency window, we chose an alpha level that would yield a low incidence of false positives. We measured this incidence by repeating the significance test on surrogate values and computing the percentage of significant pairs. Using an alpha of 0.01, the incidence of false positives was 0%.

For pairs with significant coherence, we also performed phase coherence analysis using data from error trials. We computed the percentage of pairs with significant coherence as a function of frequency and then averaged the percentages across all frequencies within each frequency band independently for correct and error trials. We computed the percent reduction in the percentage of significant neurons as 100*(C−E)/C, where C and E are the percentages of significant neurons in correct and error trials, respectively. We obtained similar results when repeating this procedure after down-sampling the number of correct trials to match the number of error trials and equating the mean firing rates in correct and error trials by probabilistically removing spikes.

For each frequency band, we applied the Rayleigh test of uniformity to test whether the phases of coherence among all coherent pairs were non-uniformly distributed along the oscillatory LFP cycle, and measured their concentration with the concentration parameter kappa (κ).