Representation of Multiple, Independent Categories in the Primate Prefrontal Cortex

Summary

Neural correlates of visual categories have been previously identified in the prefrontal cortex (PFC). However, whether individual neurons can represent multiple categories is unknown. Varying degrees of generalization vs. specialization of neurons in the PFC have been theorized. We recorded from lateral PFC neural activity while monkeys switched between two different and independent categorical distinctions (Cats vs. Dogs, Sports Cars vs. Sedans). We found that many PFC neurons reflected both categorical distinctions. In fact, these multitasking neurons had the strongest category effects. This stands in contrast to our lab’s recent report that monkeys switching between competing categorical distinctions (applied to the same stimulus set) showed independent representations. We suggest that cognitive demands determine whether PFC neurons function as category “multi-taskers”.

Introduction

The ability to categorize or group stimuli based on common features is a fundamental principle of cognition (Gluck et al., 2008; Rosch, 1973). Categorization gives our perceptions meaning by allowing us to group items by function. Without this ability to extract useful information while ignoring irrelevant details one could become overwhelmed with individual stimuli at the expense of a “big picture”. This is an experience observed clinically in many people with neuropsychiatric disorders, including autism and schizophrenia (Bolte et al., 2007; Kuperberg et al., 2008; Scherf et al., 2008; Uhlhaas and Mishara, 2007). For instance, when asked to categorize a dog, one often has a general sense of what a dog entails. However, a person with autism may instead picture each individual dog they have seen (Grandin, 2006).

The prefrontal cortex (PFC) is the brain area most central to higher-order cognition and implicated in neuropsychiatric disorders (for reviews, see Bonelli and Cummings, 2007; Miller and Cohen, 2001; Stuss and Knight, 2002). Using a novel behavioral paradigm, Freedman et al. (2001) first identified neural correlates of visual categories in the primate PFC. Categories were parameterized using a morphing system to blend between different “cat” and “dog” prototypes, which created images of varying physical similarity (Figure 1A). Importantly, images with close visual features could be in opposite categories while images with a greater difference in physical similarity could be in the same category. This allowed testing for a hallmark of perceptual categorization: a sharp transition across a discrete category boundary such that stimuli from the same category are treated more similarly than stimuli directly across the boundary (Miller et al., 2003; Wyttenbach et al., 1996). The paradigm thus enabled “visually selective” neurons that responded to similar physical features of the stimuli to be dissociated from those neurons that actually categorize the stimuli on a more abstract level. Freedman et al. found that approximately 1/3 of randomly selected lateral PFC neurons were involved in categorization.

Figure 1. Stimulus Set & Behavioral Task.

A: Morphing allowed parameterization of sample images. An example morph line between Cat prototype c1 and Dog prototype d2 displays images at the morph steps used for recording. Intermediate images were a mix of the two prototypes. Those images comprised of greater than 50% of one category (marked by the ‘Category Boundary’) where to be classified as a member of that category. B: Stimuli came from two independent category sets, Animals and Cars. The Animal category set was divided into “Cats” vs. “Dogs” and the Car category set had “Sports Cars” and “Sedans” categories. Both sets were comprised of four prototype images (two from each category as shown) as well as images along four between category morph lines. C: The delayed match to category task required monkeys to respond to whether a test stimulus matched the category of the sample stimulus. During the sample and delay periods, the monkeys must hold in memory the category of the sample stimulus but the outcome of the trial is unknown.

This large proportion of neurons raises a critical question: Do these neurons act as cognitive “generalists” or “specialists”? Generalist or adaptive theories predict many PFC neurons are highly adaptable by task demands, and may multiplex different types of information across different contexts (Duncan, 2001; Duncan and Miller, 2002; Miller and Cohen, 2001). Thus each neuron could multitask and represent multiple category distinctions, explaining why so many neurons could be involved in representing the cat and dog categories. By contrast, there is a more specialist or localist view of PFC function that posits highly specialized properties for each PFC neuron (Goldman-Rakic, 1996a, b; Romanski, 2004; Wilson et al., 1993). While this idea could also explain the Freedman et al. results since it allows for long-term plasticity (and thus the high proportion of PFC “cat vs. dog” neurons by the long-term training on the category task), these two theories predict different outcomes on how the PFC would represent multiple category schemes. That is, generalist theories predict individual PFC neurons could encode more than one category scheme while specialist models predict individual neurons would only encode a single category scheme, with different schemes being encoded in largely distinct neural populations. Whether or not PFC neurons are generalists or specialists is unresolved because virtually all neurophysiologists train monkeys on a single cognitive problem. In this study, we addressed this question by investigating how the PFC encodes multiple, independent categories in monkeys trained to randomly alternate between performing two category problems.

We employed the same Cat-and-Dog morph stimuli used in previous studies (Freedman et al., 2001, 2002, 2003, 2006; Roy et al., 2010), but also trained monkeys on a set of Car morph stimuli that were categorized as either “Sports Cars” or “Sedans” (Figure 1B). Monkeys performed a delayed match to category task (Figure 1C) with samples coming randomly from either category set while we recorded from the lateral PFC. Thus, on any given trial a sample image could come from either the Animal category set (Cat or Dog) or from the Car category set (Sports Car or Sedan). Test images, to which the monkeys determined whether the sample image was a category match or non-match, always came from the same category set as the sample image so that both category sets remained independent. We then examined how PFC neurons represented these multiple, independent categories to determine if they function as category generalists or category specialists.

Results

Behavior

Both monkeys were proficient on the categorization task (Figure 2A and B). They correctly categorized images of one category as belonging to that category on most of the trials (>80% correct), while seldom incorrectly classifying images that were on the opposite side of the category boundary. If errors were made, these were usually on the morphs closest to the category boundary (i.e., those images made with 60% of one category and 40% of the opposite category). For both category sets, we saw the behavioral hallmark of perceptual categorization: a much greater distinction between than within categories, with a sharp change in behavior across the category boundary. While both monkeys easily categorized images from both category sets (Animals and Cars), slightly more errors were made on the Car images than the Animal images suggesting that the Car set was more difficult (t-test, p < 0.01). This was also suggested by the significantly shorter behavioral reaction times for the Animal images than for Car images (mean reaction times for match trials, monkey Ti: Animals = 242ms, Cars = 278ms, t-test, p < 0.01; monkey Lu: Animals = 281ms, Cars = 378ms, t-test, p < 0.01).

Figure 2. Behavior & Single Neuron Example.

A & B: Performance of both monkeys on the delayed match to category task with multiple, independent category distinctions across all recording sessions. Monkeys were able to categorize both Animals and Cars exceptionally well, and displayed a hallmark step function in behavior at the category boundary. Error bars represent standard error of the mean. C: A single PFC neurons showed distinct firing for stimuli of one category (e.g., Sedans) vs. the other category (e.g., Sports Cars). Note how all morph percentages on either side of the category boundary (50%) grouped together (e.g., blue vs. red lines), despite the fact that sample images near the boundary line (60%/40%, dark lines) were closer in physical similarity. Thus, this neuron responded to the category membership of the stimuli rather than their visual properties. This individual PFC neuron multitasked, categorizing both Animals (Cats vs. Dogs) and Cars (Sedans vs. Sports Cars) during the late delay interval.

Neural activity to independent category sets

PFC neurons are sensitive to category membership

We focused on neural activity during the sample and delay intervals, which is when the monkeys had to categorize the sample stimulus and retain that information in short-term memory. We first identified a population of neurons with potential neural correlates of the category distinction seen in the monkeys’ behavior. A t-test was performed on each neuron’s firing rate to all sample images from one category vs. all images from the other (i.e., all Cat vs. Dog images or all Sports Car vs. Sedan images). We will refer to these neurons as “category sensitive” because the t-test identifies neurons that could potentially show category effects (in the next section, we will show that they do). Table 1 shows the breakdown of neuronal sensitivity to the Animal and Car category distinctions by trial interval (sample and delay). Over 1/3 of randomly recorded neurons in the lateral PFC showed a significant difference in the sample and/or delay intervals for one category set or the other (Animals = 37%, 167 of 455; Cars = 38%, 173 of 455). Of these category sensitive neurons (236 in total), almost half showed a significant difference in average activity for both category distinctions (Animals and Cars, 44%, 104 of 236). Most of these “multitasking” neurons (84/104 or 81%) showed significant category sensitivity for both category schemes during the same task intervals. That is, at least one interval overlapped such that a neuron was selective for both category schemes (on different trials) during that interval. Only about a quarter of the category sensitive neurons showed significant differences for either one of the category schemes but not for the other (Animals = 27%, 63 of 236; Cars = 29%, 69 of 236). These neurons were category “specialists” (i.e., they categorized Animals only or Cars only). These results are summarized in Table 2. We next present analyses to show that the population of neurons identified as category sensitive did indeed show the hallmarks of perceptual categories. As in our previous studies, there was no obvious clustering or organization of neurons with category effects across our recording sites.

Table 1.

	Intervals Where Category Sensitive				Number of Neurons
	Animals		Cars
	Sample	Delay	Sample	Delay
Multi-Sensitive (Generalists)							104
Intervals Overlap						84
	1	1	1	1	13
	1	1	1	0	13
	1	1	0	1	8
	1	0	1	1	11
	0	1	1	1	19
	1	0	1	0	1
	0	1	0	1	19
Intervals Don’t Overlap						20
	1	0	0	1	5
	0	1	1	0	15
Animal-Only Sensitive (Specialists)							63
	1	0	0	0	15
	0	1	0	0	39
	1	1	0	0	9
Car-Only Sensitive (Specialists)							69
	0	0	1	0	17
	0	0	0	1	37
	0	0	1	1	15
Non-Sensitive							219
	0	0	0	0	219
Total							455

Table 1 groups neurons based on whether each neuron was category sensitive (see text) in the sample and/or delay intervals (t-test at p < 0.01). Each row of the table specifies whether neurons showed significance category sensitivity by category set and task interval (1 = significant for that category set/interval, 0 = non-significant for that category set/interval). For example, the first row identifies that 13 neurons were sensitive for all four tested category set/interval combinations (i.e., a 1 is shown for Animals/Sample, Animals/Delay, Cars/Sample, and Cars/Delay). The table is sub-divided to show neurons that were: category sensitive to both category distinctions in the same or different intervals, sensitive to the Animal distinction only, Car sensitive only, or non-sensitive. Multi-sensitive neurons were also distinguished based on whether neuronal selectivity for Animals and Cars overlapped (was significant in the same interval).

Table 2.

Independent Categories (Animals vs. Cars)

	Animal Sensitive	Not Animal Sensitive	Total
Car Sensitive	104	69	173
Not Car Sensitive	63	219	282
Total	167	288	455

Table 2 lists the count of all recorded neurons (455) based on whether each neuron was category sensitive (t-tests, p < 0.01) for both the Animal (Cats vs. Dogs) and Car (Sedans vs. Sports cars) category sets, only one set or the other, or neither set. The last row and column show the sums and specify neuronal selectivity for one category set independent of selectivity for the alternative category set.

PFC neural activity shows the hallmarks of perceptual categorization

A multitasking neuron that was significantly category sensitive (t-test, as above) for both category schemes is shown in Figure 2C. When activity was sorted by the morph level of the sample images, firing rates grouped together based on the category of the images rather than their physical appearance. That is, this neuron’s firing rate (especially near the end of the memory delay), clusters according to category membership and there is a sharp difference in firing rate between images right across the category boundary (60%–40% images, dark lines) even though those images are relatively similar in appearance. Note how this category selectivity occurs for both category sets, distinguishing Cats from Dogs as well as Sedans from Sports Cars.

The same effect can be seen across the population of category sensitive neurons (Figure 3). The average activity for neurons that showed category sensitivity (according to a t-test, as above) across a given interval (sample, delay, or test intervals) is shown in each panel of Figure 3 (except for the fixation interval which shows the average activity of neurons that were category sensitive during either the sample, delay, or test intervals). A preferred category was defined as the category having the higher firing rate for each interval. This includes all neurons that had a statistically significant difference in firing rate between categories. “Preferred” was just an arbitrary designation; individual PFC neurons could respond with either an enhanced or suppressed firing rate. During the sample, delay, and test periods firing rates are significantly different across the category boundary, but not different within categories (i.e., all bars are statistically similar on either side of the boundary line but different across it, t-tests, p < 0.05). During fixation (baseline), there is no significant difference in firing across trials of the preferred vs. non-preferred category – this shows that selecting a preferred category by choosing the category set with the highest firing rate during each interval does not artificially induce a category effect. Thus, the neural activity of the PFC population shows the same sharp transition across a discrete category boundary as was seen in the behavior. The Car category distinction elicited a significantly greater difference in activity in averaged normalized firing rates during the sample and test intervals (t-tests, fixation: p = 0.37, sample: p = 0.02, delay: p = 0.09, test: p = 0.005). This may be related to the monkeys finding the Cars distinction a little more difficult (see above). This is consistent with observations of sharper neural tuning and higher firing rates in the area V4 with increased difficulty in visual discriminations (Spitzer et al., 1988).

Figure 3. Category Selectivity by Morph Level.

Normalized neuronal firing of category sensitive neurons sorted by the percentage of each neuron’s preferred category that made up the sample image. During fixation (baseline) no category effect is present, but during the sample, delay, and test periods there is a significant difference in firing across the category boundary (but not within). The same hallmark step function as seen in the monkeys’ behavior is seen in the neural population activity. Error bars indicate standard error of the mean. Asterisks indicate that bars were significantly different (t-tests, p < 0.05) from each bar on the opposite side of the category boundary. Bars on the same side of category boundary were never significantly different.

Next, we took our analysis one step further and compared average population activity for individual sample images by computing a correlation matrix (Freedman and Miller, 2008; Hegde and Van Essen, 2006; Roy et al., 2010). The correlation values reflect the degree of similarity of the level of neural activity between all possible pairs of images. Correlations were computed between the activities of all category-sensitive neurons (t-test, as above) to all possible pairs of sample images; the color of each square indicates the correlation coefficient (r) for a single pairing. To simplify the presentation of the results, Figure 4 shows data from the delay interval (when monkeys had to remember the sample category). Identical effects were also seen during the sample and test intervals (Figure S1). The correlation matrix is organized with all the images lined up along each axis in the same order for a given category scheme such that images 1–10 were always from one category (e.g., “Cats”) and images 11–20 were always from the other category (e.g., “Dogs”) – this arrangement is depicted in the figure key. The results showed high correlations (warm colors) when both images of a pair were from the same category whereas negative correlations (cool colors) were seen when images were from different categories (Figure 4A & B). Note the sharp difference between correlations across the category boundary (between images 10 and 11), indicating the sharp difference in activity to physically similar images that belong to different categories. As expected, the highest correlations were when the pairs of images were from the same category and prototype, i.e., they were from the same category and physically similar (see figure key). When non-category sensitive neurons were tested, no category effect was seen (Figure 4C).

Figure 4. Category Selectivity Across All Images.

Correlations values (r) were computed for all neurons’ mean responses to all possible pairings of sample images from the same category set (images 1–20, either Animals or Cars). Figure Key: Correlation values were then used to define the color of each square in a matrix representing these image pairings. The matrix was arranged such that images 1–10 came from one category (e.g., “Sports Cars”) and images 11–20 came from the opposite category (e.g., “Sedans”). The matrix was further subdivided such that every five images came from the same prototype. A: The average activity of PFC Animal sensitive neurons to images from the same category was highly correlated (similar) - as seen in the warm-colored squares, whereas correlations to images from different categories was negative - deep blue squares. This was true for both the Animal category distinction (left panel) as well as for the Car category distinction (right panel), despite the fact that Car sensitivity was not a factor in selecting the neurons. Thus, Animal sensitive neurons multitask and also convey information about the Car category. B: Car sensitive neurons display strong selectivity to the Car category distinction as well as sensitivity to the Animal category distinction. Again, activity is strongest for the expected distinction (Cars), but clearly evident for the non-selected distinction (Animals) due to multitasking neurons. C: Non-category sensitive neurons had low correlation between the sample images and these correlations were non-significant across categories (between images 1–10 and 11–20). P values indicate significance between categories as determined by permutation tests (see text).

The correlation matrices were tested for significance by permutation tests. We first computed the actual difference in average correlation between values computed from images of the same category vs. all values from images of different categories. We did not use values from the same prototype (see figure legend) in order to ensure that we were testing category effects per se (i.e., we excluded same prototype images because they also looked alike). A null distribution was then computed by randomly shuffling these values (i.e., randomly assigning each values to either the “within” or “between” category groups) and computing the difference for this randomized data. We repeated this 10,000 times. The percentage of times the theoretical difference exceeded the true difference was the permuted p-value. Identical results (i.e., the same groups were significant/non-significant in all cases) were obtained by performing t-tests across these groups. The resulting p values are shown in Figure 4; they indicate significantly higher correlations of activity to images within than between categories.

Individual PFC neurons multitask (generalize) and encode multiple categories

The t-test for category sensitivity (above) revealed that 44% of PFC category-sensitive neurons were category multi-taskers, i.e., showed a significant difference in average activity to both the Animal and Car category distinctions. This large degree of neuronal overlap in category effects can be seen in the population correlation matrices in Figure 4. Even when neurons were selected solely based on their category sensitivity to the Animal category distinction (Figure 4A, left panel, i.e., regardless of whether or not they were sensitive to the Car category distinction), category effects could still be seen when the correlation matrix was computed for these neurons using the Car distinction (Figure 4A, right panel). Likewise, when we computed the correlation matrix using neurons that showed significant category sensitivity for Cars (Figure 4B, right panel), category effects were still seen for the Animal category distinction (Figure 4B, left panel). Therefore, a large percentage of PFC neurons must have shown effects for both category schemes or else the not-selected-for category distinction would not show an effect. We further demonstrated this by re-computing the correlation matrices, but this time by selecting neurons by whether they were sensitive to both category sets (Animals and Cars, t-tests as above, both p < 0.01), sensitive for Animals but not Cars (t-test for Animals, p < 0.01; t-test for Cars, p > 0.01), sensitive for Cars but not Animals (t-test for Cars, p < 0.01; t-test for Animals, p > 0.01), or sensitive to neither category (both t-tests, p > 0.01, Figure S2). In this case, category effects were seen only where expected: Animal-only sensitive neurons only showed category effects for Animals and not Cars, Car-only sensitive neurons showed category effects for Cars and not Animals, etc (Figure S2). These results confirm the finding that PFC neurons can multitask categories and, in addition, they demonstrate that the category sensitivity t-tests used to select neurons for analysis faithfully identify category effects at the population level.

To further explore this multitasking as well as to examine the temporal dynamics of category selectivity, we calculated the information that each PFC neuron carried about each category distinction as a function of time within the trial (Figure 5). Neural information about a category distinction was quantified as the area under the receiver operating characteristic (ROC) curve for each category contrast (e.g., “Cat” vs. “Dog”). We used rectified ROC values (indicating the level of category sensitivity, but not which category was preferred). Higher ROC values (approaching 1) indicate a larger degree of difference in activity to sample images of the different categories (orange/white colors) while lower ROC values (near 0.5) indicate no or weak category sensitivity (black colors). In Figure 5A, each row corresponds to a single neuron. For the left panel, all the neurons identified as being significantly category sensitive for Animals were sorted by the time of maximum ROC values (167 neurons, t-test, as above, Figure 5A, left panel, rows 1–167 below the dashed white line). The neurons that did not show a significant Cats vs. Dogs sensitivity were randomly sorted (Figure 5A, left panel, rows 168–455 above the dashed white line). Note the variability in the timing of category-related activity. Some neurons are category sensitive only transiently (short orange bands), while others maintain their selectivity over a long duration (long orange bands). This selectivity can peak early in the sample period for some neurons while other neurons have peaks of selectivity ranging throughout the delay period and into the test interval. The onset of category selectivity varied similarly. Note the great deal of variability in the temporal dynamics of the category signals across neurons. In general, the highest ROC values for the Cat vs. Dog category of the sample stimulus were around the time of the test stimulus, when the category match/non-match decision had to be made. This is consistent with prior reports (Freedman et al., 2002).

Figure 5. ROCs for the Recorded PFC Population.

A: ROC values for each of the recorded 455 PFC neurons are shown, with the same neuron depicted in the same row on the left and right panels. Bright orange colors indicate high category sensitivity. Neurons identified previously as sensitive to the Cats vs. Dogs category distinction (via t-test) are sorted on max ROC (left panel, below the dashed white line). The same neurons have the highest ROC values for the Sports Cars vs. Sedans category distinction (right panel). B: Data as in panel A now realigned on the max ROC for Car sensitive neurons. Again, the majority of selectivity for both category distinctions is in the lower portion of the panels below the dashed white line, even though the Animal ROCs are aligned to match the Car sensitivity. Thus, the majority of PFC category sensitive neurons multi-task and encode both category distinctions.

To determine the degree of overlap of category effects in individual neurons, we next plotted the ROC values for the Car category distinction (Figure 5A, right), but retaining their sorting by the time of the maximum ROC for the Animal category distinction. In other words, corresponding rows in the right and left panels of Fig 5A plot ROC values for the same neuron. With the neurons aligned, we see that the majority of the highest ROC values for the Car distinction (bright orange color) in the right panel is also in the lower third of the graph (below the dashed line), indicating that many of the neurons that distinguished between Cats and Dogs also distinguished between Sedans and Sports Cars. There are some neurons with high Car ROC values in the upper two thirds of the plot (i.e., neurons that were only sensitive to the Car distinction, but not the Animal distinction), but they are fewer in number than the neurons that showed relatively high ROC values for both distinctions. Interestingly, the same neurons often showed markedly different time-courses of category effects for Animals and for Cars. Note that while the Animal ROCs are ordered according to the time of max ROC, the same neurons show no such ordering for their Car ROC values. A given neuron could show a short latency for one category but long latency for the other category. A correlation analysis confirmed that the time to max ROC was not significantly correlated between the Animal and Car category distinctions (r = −0.08, p = 0.41), indicating that the temporal dynamics for the two categories schemes were unrelated.

Figure 5B shows the same comparison, only this time the neurons are sorted according to the peak ROC value for the Car category distinction. Car-sensitive neurons (173 neurons, t-test, as above) were sorted by the time of the peak ROC below the dashed line of the right panel (rows 1–173); those that were not significantly Car-sensitive were randomly sorted, as above (row 174–455, Figure 5B, right panel). The left panel of Figure 5B displays the ROC values of the same neurons as in the right panel, but for the Animal category distinction (sorted according to the maximum ROC of the Car distinction). Again, we see the majority of higher ROC values (orange color) is in the bottom third of each graph, confirming the large number of PFC neurons that reflected both the Animal and Car category distinctions.

Is the likelihood of a PFC neuron being a category multitasker higher than expected by chance? To address this question, we tested the null hypothesis that a neuron’s category sensitivity to either category distinction had no effect on whether it was also category sensitive for the alternate category scheme. A chi square analysis indicated that neural selectivity for one category scheme is not independent of selectivity for the other (χ² = 65.86, df = 1, p < 0.01). In other words, if a neuron was sensitive for one of the independent categories, there was a greater probability than expected by chance that it would also be sensitive to the second category scheme. Thus, PFC neurons that participate in categorization have a greater than chance probability of participating in multiple category schemes.

Category multitasking neurons have the strongest category sensitivity

To compare the strength of category selectivity of individual neurons across category distinctions, we plotted for each neuron the ROC values for the Animal category distinction against its ROC value for the Car category distinction. Because of the great deal of variability in temporal dynamics of category effects across individual PFC neurons (see Figure 5), we used the ROC value computed across a 500ms window centered at the time of the maximum ROC from either the sample or delay periods. We color coded the ROC values according to whether the neurons were classified as category sensitive to both categories, Animals only, Cars only, or neither category (t-tests, as above). We plotted non-rectified ROC values ranging from 0 to 1 to capture the category preference of each neuron. Values near 0.5 indicate low category sensitivity whereas ROC values approaching 1 indicate greater activity for either Cats or Sports Cars while ROC values approaching 0 indicate greater activity for Dogs or Sedans. As expected, category-sensitive neurons had higher mean ROCs than non-sensitive neurons for both the Animal and Car distinctions (t-test, p < 0.01). Data points were evenly distributed across the four quadrants of the graph, indicating an equal neuronal preference for all category combinations (i.e., there was no significant difference in the number of points in the quadrants, chi square test, χ²=1.43, df = 1, p = 0.23). As expected, the neurons that were identified by t-test as Animal-only sensitive spread more horizontally (they had weak ROC values near 0.5 for the Car category distinction and thus tended to cluster near 0.5 values for the Car distinction) and Car-only sensitive neurons tended to be spread more vertically (they had weak ROC values for the Animal category distinction). However, note that the neurons that were significantly sensitive to both category distinctions were furthest from the center of the graph, indicating that they tended to show the strongest category ROC values. We confirmed that the neurons sensitive to both Animals and Cars had higher ROC values than neurons sensitive to either Animals or Cars alone by computing t-tests on their ROC values for the Animal distinction and for the Car distinction. In both cases, the neurons sensitive to both category distinctions had higher ROC values for each distinction than the neurons sensitive to one or the other distinction only (p < 0.01). Thus, many PFC neurons were sensitive to both category distinctions and they showed the strongest category effects.

Category match/non-match effects

To solve the task, monkeys had to determine whether the category of the test stimulus matched that of the sample. This was reflected in the activity of many PFC neurons: 190 of 455 neurons (42%) showed a significant difference in activity to the test stimulus depending on whether or not it was a category match or non-match (t-test, p < 0.01). Of these, 104 (55%) showed match enhancement (a higher firing rate to match trials than non-match trials) whereas 86 (45%) showed match suppression (a lower firing rate to match trials than non-match trials). As with the category information, the temporal dynamics of the match vs. non-match information varied across neurons (Figure S3).

Discussion

We recorded from lateral PFC neural activity while monkeys randomly categorized images from two independent category sets (Animals and Cars). We found that many PFC neurons were sensitive to both category distinctions and, indeed, they tended to be the neurons with the strongest category sensitivity. These results suggest that PFC neurons can “multitask” independent categories. This adaptability allows the brain to re-utilize the same pool of neurons for different tasks. Without it, storage capacity might be severely limited.

Does this mean that prefrontal neurons always multitask? It might depend on cognitive demands. Our lab (Roy et al., 2010) recently trained two monkeys to flexibly re-categorize the same images under two different schemes, a situation analogous to categorizing an airplane as either a “flying object” or a “method of transportation”. Monkeys categorized the same Animal stimuli under two orthogonal category schemes using identical procedures to those used in our study. One scheme divided the image set into Cat-like and Dog-like morphs (Animals, the same scheme as used above and shown in Figure 1B and Figure 7A) while the other scheme grouped together different pairs of Cat and Dog prototypes to form a new animal category set (termed “Animals 2” in Figure 7B). Monkeys were trained to switch between these two orthogonal, Animal category schemes (a cue told them which scheme to follow on a given trial). Recordings were again made from the lateral PFC as in the present study. In contrast to the present results, Roy et al. (2010) found that there was little overlap in category representation across neurons: Only 24% of the neurons showed category sensitivity for both category schemes vs. the 44% found in our study (Figure 7, Tables 2 and 3). This difference in the proportion of the overlap of category representations between our studies was significant (p = 0.0001, permutation tests, see experimental procedures). This suggests that relatively few PFC neurons multitask two different category distinctions when those categories are in conflict (i.e., when images from the same stimulus set must be categorized in different ways and thus could be confused with one another). By contrast, in our study the two different category distinctions were independent and not in conflict (i.e., they could not be confused). The larger degree of separate representations of the two competing categories in Roy et al. (2010) may allow for easier inhibition of the competing category through distinct neural populations.

Figure 7. Multitasking of Neurons for Independent and Competing Category Schemes.

A: In the current study, when monkeys categorized two independent category sets (Animals and Cars), 44% of category sensitive neurons showed category effects for both category sets. B: This is in contrast to another study from our laboratory (Roy et al., 2010) in which the same images were categorized under two different (orthogonal) category sets: Animals (“Cats vs. Dogs”) or Animals2 (a new category distinction based on two unique combinations of a cat and dog prototype). The Cat and Dog categories were the same used in this study. In the Roy et al. (2010) experiment with orthogonal category sets, fewer PFC neurons (24%) showed category sensitivity for both category distinctions.

Table 3.

Related/Competing Categories (Animals vs. Animals 2) from Roy et al. (2010)

	Animal Sensitive	Not Animal Sensitive	Total
Animal 2 Sensitive	49	67	116
Not Animal 2 Sensitive	90	330	420
Total	139	397	536

Table 3 lists how many of the 536 neurons reported by Roy et al. (2010) were category sensitive (t-tests, p < 0.01) for both the Animal and Animal2 category sets, only one set or the other, or neither set. The last row and column show the sums and specify neuronal selectivity for one category set independent of selectivity for the alternative category set. The Animal category set was same “Cat” vs. “Dog” distinction used in this study while the Animal2 category set refers to an alternate, orthogonal grouping of the same images into two different categories (see “Animals2” in Figure 7B and Roy et al., 2010).

It is important to note that our results are opposite to those predicted by a “sensory” or bottom-up driven view of neural representation in the PFC. If the physical appearance of the stimuli alone were determining how information is distributed among PFC neurons, then we would expect the most overlap in representation when the categories are competing (as in Roy et al., 2010) because the exact same images are being categorized. Instead, we found a greater degree of overlap when monkeys were categorizing independent image sets that were physically distinct from each other. Thus, unlike sensory cortex, especially primary sensory cortex, it appears that cognitive demands of the task are more influential to how information is distributed across PFC neurons than bottom-up sensory inputs.

Our observation of category multitasking in the PFC also stands in contrast to that seen in the medial temporal lobe (MTL) of humans, which contains neurons that respond only to images from a single category, such as animals (Kreiman et al., 2000) or “Jennifer Aniston” (Quiroga et al., 2005). MTL coding is sparse; relatively few neurons seem to represent a given category and single MTL neurons seem highly selective for only one category. By contrast, we found that in the monkey PFC many neurons are engaged during categorization and many of them are category multi-taskers. This could be due to the different functions of the PFC and MTL. Sparse encoding in the MTL may be better suited for long term memory storage whereas multitasking may be better suited for the cognitive flexibility for which the PFC is critical.

A variety of studies point to a central role of the PFC in categorization. The human PFC is activated during categorization (Aron et al., 2006; Vogels et al., 2002). When new categories are learned via trial and error, they may be acquired via interactions in corticostrial loops between the PFC and basal ganglia (Buschman and Miller, 2008; Seger, 2006, 2008; Seger et al., 2010). The PFC may also orchestrate communication between the basal ganglia (BG) and the medial temporal lobe (MTL) during category learning (Poldrack and Rodriguez, 2004). Further, there seem to be contrasts between the PFC and the inferior temporal cortex (ITC), which provides the PFC with highly processed visual information. While ITC neurons reflect visual category membership (Kiani et al., 2007; Vogels, 1999a, b), category effects are stronger in the PFC because the ITC has more detailed visual information (Freedman and Miller, 2008; Freedman et al., 2003; Meyers et al., 2008). Indeed, ITC neurons have been shown to have altered shape tuning properties after category learning (De Baene et al., 2008; Freedman et al., 2006; Jiang et al., 2007) and emphasize the visual features that are diagnostic of categories (Sigala and Logothetis, 2002), rather than explicit representations of category per se.

We have shown that PFC neurons generalize to multiple visual categories when these categories are independent and not in conflict. If PFC neurons are true “cognitive generalists” they might be able to categorize stimuli across multiple modalities. The ventrolateral PFC has been shown to be engaged by auditory categorization (Cohen et al., 2006; Gifford et al., 2005; Lee et al., 2009; Russ et al., 2007) and have neural correlates of somatosensory (Machens et al., 2005; Romo et al., 1999) and gustatory (Lara et al., 2009) working memory. But it remains to be seen whether category multimodality is apparent within individual PFC neurons.

In summary, our results support the idea that PFC neurons can function as cognitive generalists because more category sensitive neurons in this study multitasked and distinguished between Cat vs. Dog as well as Sports Car vs. Sedan than categorized either distinction alone. However, the fact that category representations in the PFC neurons are more independent of one another when multiple category distinctions are not independent suggests there are times when relative specialization is advantageous. We suggest that specialization may occur with high cognitive demands, such as when categories are in conflict. Prefrontal cortex neurons may multitask by default, but may make representations more independent when there are task demands for which this is advantageous. In other words, harder problems may require more specialized neural architecture.

Experimental Procedures

Subjects

Data were collected from two macaque monkeys (Macaca mulatta) that were cared for in accordance with National Institutes of Health guidelines and the policies of the Massachusetts Institute of Technology Committee on Animal Care. One monkey, male ‘ti’, was trained from naivety for this study. He first learned to perform the delayed match to category task with the Animals (Cat vs. Dog) category set, then subsequently learned to perform the same task with the Car morphs. The second monkey, female ‘lu’, has participated in several prior categorization studies. She also first learned the Animal category set, but as part of a more difficult task where the category boundary was flexible and cued at the beginning of each trial (Roy et al., 2010). She subsequently learned the Car categorization.

Stimuli

Two independent category sets

Stimuli were from one of two independent category sets (Figure 1B). Each set consisted of four prototype images (two prototypes from each category) and morphs between those prototypes (Figure 1A). The first category set (Animals) consisted of Cats vs. Dogs. The second category set (Cars) consisted of Sports Cars vs. Sedans.

Stimuli are morphs between prototypes

Numerous morph images were generated by varying the percent composition of the prototype images using the vector differences between corresponding points (Beymer and Poggio, 1996; Freedman et al., 2001, 2002; Jiang et al., 2007; Shelton, 2000). Stimuli from different categories differed along multiple features and were smoothly morphed (i.e., without sudden appearance of any feature). Each category set (Animals and Cars) had a fixed category boundary at 50%. Thus, an image was considered a member of a category if it contained more than a 50% contribution from a prototype in that category. During training of both category schemes, the image set consisted of hundreds of images generated from combinations of the four prototypes. For the recording sessions, we generated 20 images from six levels of combinations (100:0, 80:20, 60:40, 40:60, 20:80, and 0:100) of each pair of prototypes across the category boundary to be used as the sample images. The prototype images of the four morph lines that span between the prototypes across the category boundary are shown in Figure 1B, with an example of all images along a single morph line depicted in Figure 1A. All images within a category set had identical color, shading, orientation and scale. This process was repeated for both category sets, so that the final recording set contained 40 possible sample images which were repeatedly presented to obtain sufficient statistical power for neuronal analysis. In order to prevent memorization during the recording sessions, these sample images were randomly paired with hundreds of test images that were a minimum of 80% morph from one category. These sample – test pairs were randomly selected and were changed daily.

Behavioral Task

Monkeys performed a delayed match to category test (Figure 1C). They initiated the trial by grabbing a response bar. This caused the onset of a white fixation square. Monkeys were required to maintain fixation within 2 degrees of this square during the course of the entire trial. After a 1000ms period of fixation, one of 40 possible sample images (20 from either category set) was presented for 600ms. This was followed by a 1000ms delay period where the monkeys held in mind the category of the sample image. Subsequently, a test image was presented. This test image could be a category match (an image from the same category as the sample) or a category non-match (an image from the opposite category as the sample). For example, if the sample image was a cat (any image > 50% Cat), a match trial would have a test image that was also a Cat image whereas a non-match trial would have a Dog image during the test phase. On match trials monkeys were required to release the bar within 600ms of the test image presentation to indicate the category match. On non-match trials, monkeys were to continue holding the bar and after 600ms the test image turned off. This later case was followed by a second delay period (600ms) and the subsequent presentation of a second test image that was always a category match to the sample image. Thus, monkeys were required to make a response on every trial and could receive a reward on every trial. Note how the motor response is dissociated from the category of the sample stimulus (i.e. there is no one-to-one mapping for sample category and motor response). Reward amounts were kept constant throughout each recording session. Trials of each category set were randomly interleaved and occurred at similar frequency, as did trials of each type (i.e., match vs. non-match). Test stimuli were always from the same category set as the sample (e.g., if the sample was an Animal the test stimulus was also an Animal), but could either be a category match (e.g., Cat - Cat) or a category non-match (e.g., Cat - Dog).

Electrophysiological Recordings

Eye movements were recording using an infrared eye tracking system (Iscan, Burlington, MA) at a sampling rate of 240 Hz. Neural recordings were made using individual, epoxy-coated tungsten electrodes (FHC Inc., Bowdoin, ME). Up to 16 of these electrodes were lowered through the dura each day using in-house screw microdrives. Electrodes were either driven independently or in pairs. Recording wells were positioned over the lateral PFC. Electrodes were lowered into the cortical cell layer (i.e., when neurons or hash were easily identifiable) and allowed to settle. Electrodes were adjusted to obtain neurons on one or both electrodes on each microdrive, but no prescreening of neurons took place. This resulted in an unbiased sample of lateral PFC neurons, rather than simply those neurons that may be task related. Waveforms were amplified, digitized and then stored for offline sorting. Principal components analysis was subsequently used to sort the waveforms into individual neurons (Offline Sorter, Plexon Inc., Dallas, TX). We included all well isolated neurons that were held for a minimum of 500 correct trials in our analysis. This resulted in a total of 455 lateral prefrontal cortex neurons (358 from monkey ‘ti’ and 97 from monkey ‘lu’).

Data Analysis

Analysis Intervals

Data were analyzed over three time intervals throughout the trial. The fixation interval included the last 500ms before the onset of the sample image. During this time the eyes were stable and no images were on the screen – this served as a baseline measure of neural activity. The sample interval was analyzed from 100 to 600 ms after sample onset and represented the time when the sample image was present (adjusted for the visual delay to the PFC). The delay interval was analyzed from 300 to 1100ms after the sample offset and captured the period when no image was physically present on the screen but the monkey was remembering the category (presumably) of the sample image (again adjusted for the PFC neural delay). The use of these analysis intervals provided a baseline and two periods when the only information present was the sample image (and its category). The timing of these analysis intervals is consistent with our previous studies (Freedman et al., 2001, 2002, 2003; Roy et al., 2010). For some analyses we also included the test interval defined as a 500ms window starting at the time of test stimulus onset.

Normalization

For Figure 3, neural activity was normalized by first subtracting from the measured firing rate the minimum firing rate during that interval and then dividing by the difference of the maximum and minimum firing rates. This method maximized the dynamic range of each neuron in each time interval and is consistent with that used in our previous studies (Freedman et al., 2001, 2002, 2003). All results were similar when firing rate was replaced with number of spikes.

Permutation Tests

In this experiment with two independent category sets and a similar experiment with related (competing) category sets (Roy et al., 2010) we randomly sampled lateral PFC neurons and tested their category selectivity. In both cases, we found neurons in the PFC selective for one or multiple category sets. We then used permutation tests to examine if the number of neurons selective for both category sets was the same or different across experiments. We first calculated the actual difference in the percentage of selective neurons in the two experiments. Then we created a distribution containing all neurons recorded in both experiments. For each permutation we shuffled this distribution, randomly assigned each neuron to an “experiment”, and computed a theoretical difference in the percentage of selective neurons. We repeated this 10,000 times to estimate the distribution of population differences under the null hypothesis that there’s no difference between the two experiments. A p value was then determined by summing the number of theoretical differences that were less than the actual percentage difference, dividing by the number of repeats, and subtracting this value from one. We used a similar procedure to compute p values for the correlation plots, again repeated 10,000 times.

Supplementary Material

01

Figure S1 - Category Effects Across All Images by Task Interval

A: Data from the baseline (fixation) interval for all the neurons with category sensitivity in any task interval (i.e., the neurons plotted in 4B-D). B-D: Correlation matrices plotted for those neurons identified as category sensitive (by t-test) for the sample, delay, and test intervals. High correlations (warm colors) are seen when comparing images from the same category, whereas negative correlations (cool colors) are seen when comparing images from opposite categories. Note the sharp change across the category boundary (between images 10 and 11). P values indicate significance between categories as determined by permutation tests (see text).

Figure S2 - Category Effects Across All Images For Category “Generalists” and “Specialists”

Correlation matrices plotted for those neurons identified as category sensitive (by t-test) for both categories (A), Animals only (B) or Cars only(C). Non-sensitive neurons show no category effect for either category distinction (D). P values indicate significance between categories as determined by permutation tests (see text).

Figure S3 - ROC for Category Match vs. Non-match

ROC values for neural activity to the category match vs. non-match status of the test stimulus for each of the 455 PFC neurons. They are aligned by the time of each neuron’s maximum ROC value. Bright orange colors indicate a strong difference between match and non-match. Note how the temporal dynamics of category match/non-match effects vary in duration, time of maximal selectivity, and time of selectivity onset across individual neurons.

Figure 6. Mean ROCs for each Category Scheme.

Mean ROC values for all recorded neurons to both category distinctions. Data points are color coded based on significant category sensitivity via t-test. Values closer to the origin indicate weaker category sensitivity for the given distinction. Data points were equally distributed in all four quadrants, indicating equal neuronal preference for all categories. Data points furthest from the origin in all four directions were from those multitasking neurons sensitive to both the Animal and Car category schemes (yellow circles).