First-pass processing of value cues in the ventral visual pathway

SUMMARY

Real-world value often depends on subtle, continuously variable visual cues, specific to particular object categories, like the tailoring of a suit, the condition of an automobile, or the construction of a house. Here, we used microelectrode recording in behaving monkeys to test two possible mechanisms for category-specific value cue processing: (I) Previous findings suggest that prefrontal cortex (PFC) identifies object categories. Based on category identity, PFC could use top-down attentional modulation to enhance visual processing of category-specific value cues, providing signals to PFC for calculating value. (II) A faster mechanism would be first-pass visual processing of category-specific value cues, immediately providing the necessary visual information to PFC. This, however, would require learned mechanisms for processing the appropriate cues in a given object category. To test these hypotheses, we trained monkeys to discriminate value in four letter-like stimulus categories. Each category had a different, continuously variable shape cue that signified value (liquid reward amount) as well as other cues that were irrelevant. Monkeys chose between stimuli of different reward values. Consistent with the first-pass hypothesis, we found early signals for category-specific value cues in area TE (the final stage in monkey ventral visual pathway), beginning 81 ms after stimulus onset, essentially at the start of TE responses. Task-related activity emerged in lateral PFC approximately 40 ms later, and consisted mainly of category-invariant value tuning. Our results show that, for familiar, behaviorally relevant object categories, high-level ventral pathway cortex can implement rapid, first-pass processing of category-specific value cues.

eTOC blurb

Real world value judgments often depend on subtle variations in object appearance. Sasikumar et al. show that neurons in high-level cortex can become sensitive to these variations following extensive training with specific object categories. This provides a fast mechanism for value judgments in familiar object categories.

INTRODUCTION

Microelectrode recording studies in behaving macaque monkeys and other species have elucidated how value is processed and represented in prefrontal cortex [1–13]. In most of these studies, value levels are arbitrarily assigned to a small number of discrete, easily distinguished stimuli through extensive training. In real life, however, object values often depend on subtle, continuously varying visual information that signifies edibility, drinkability, age, material, manufacture, strength, health, and other dimensions of object quality. Judging this kind of real-world value requires detailed visual processing usually ascribed to the ventral pathway of visual cortex [14–18].

This processing would necessarily focus on the specific visual cues that determine value in a given object category. (By “cues”, we mean shape features and other visual properties.) For example, the monetary values of racehorses and show dogs depend on conformation standards (size, shape, and relative position of body parts) that vary between breeds. Focusing on the correct value cues could depend on top-down modulation of processing in visual cortex. This would entail category recognition followed by attentional modulation. Population activity in later stages of ventral pathway visual cortex (TE in monkeys) carries sufficient shape information to discriminate object categories [19] (e.g. the sharp beard, eyebrows, flopped ears, and short docked tail of a wire fox terrier; Figure 1A, blue). TE projects to lateral prefrontal cortex (LPFC), where explicit signals for familiar object categories can emerge (e.g. “wire fox terrier”) [20–22] (Figure 1B). Following category identification, top-down attentional modulation [23–27] could be invoked to select for category-specific value cues (e.g. length of snout, curvature of neck, short back; Figure 1A, yellow). Visual information about value cues would then be available to PFC for calculating value. This top-down mechanism might be particularly important for judging value in new or less familiar object categories.

Figure 1. Proposed mechanisms for category-specific value processing.

(A) Here, the hypothetical category is “fox terrier”, for which category diagnostic shape cues might include the sharp beard, eyebrows, flopped ears, and short docked tail (blue). Desirable shape qualities (value cues) for a fox terrier include a long snout, a curved neck, and a short back (yellow). (B) Top-down. This strategy would concatenate known neural mechanisms of visual processing, category recognition, and top-down modulation. Category-diagnostic visual information from TE, such as the full beard and short tail, could provide inputs necessary for categorical recognition in PFC. Category recognition could invoke top-down control mechanisms for selective processing of snout length, neck curvature, and back length. The resulting category-specific information about fox terrier conformation could then pass to PFC and/or other structures involved in calculating value. A long snout, curved neck, and short back would produce a high value signal. (C) First-pass. This strategy would depend on learning-induced changes in neural tuning to process all the necessary information during the first pass of visual information through TE. Specifically, neurons that respond to category-diagnostic cues (beard, tail) would also be tuned for snout length, neck curvature, back length, and or other value-diagnostic shape dimensions. Based on such information from TE, value information could be generated immediately in PFC.

However, for extremely familiar categories, it would be simpler and faster to process visual value cues in the first pass of visual information through TE (Figure 1C). This could make value information available to PFC simultaneously with category information. It would demand that acquired knowledge of category-specific value cues be incorporated into category-specific tuning. That is, TE neurons tuned for category-diagnostic characteristics (Figure 1A, blue) would also be tuned for category-specific value cues (Figure 1A, yellow). This might require extensive experience with the object category (e.g. through training as a show dog judge). It is important to note that, under both hypotheses, we are not suggesting that TE represents value itself, only that it provides signals for visual cues necessary for downstream computation of value.

Here, we present evidence that the primate brain can implement this first-pass strategy (Figure 1C) after behavioral experience with value (liquid reward) variations signified by shape cue variations in familiar object categories. We found that, in monkeys performing an object value comparison task, TE neurons were sensitive to category-specific value cues from the very onset of responses. In contrast, neurons in LPFC did not express task-related information until approximately 40 ms later. While value cue signals in TE were category-specific, most LPFC neurons did not differentiate between categories; instead they signaled category-invariant value. These results demonstrate the possibility of first-pass visual value cue processing. Conceivably, this is an efficient mechanism by which experts can quickly perceive the value of real-world objects.

RESULTS

We trained two monkeys to interpret reward value in four letter-like object categories (Figure 2A, rows I–IV). Like letters and numerals, these categories were defined by their medial axis structure—the relative orientation, position, and connectivity of elongated limbs. Letters and numerals are categories in the sense that their shapes vary widely with font, handwriting, slant, and orientation, but they remain identifiable by their medial axis configurations in all cases. (This type of “category” is distinct from categories defined by boundaries in continuous spaces or by arbitrary learned associations.) For example, all the stimuli in category I had a long stroke to which two shorter orthogonal strokes were attached on one side, as in the Greek letter pi. (It has been shown that many TE neurons have clear tuning for medial axis structure [28].) Also like letters and numerals, the precise shape (curvature, length, thickness) and orientation of these stimulus categories could vary. In each category, liquid reward amount (symbolized by blue dots) was proportional to the magnitude of a specific shape cue (illustrated on the left under “Relevant”). In categories I and II, the cue was the distance between two limbs. (The relationship between distance and reward was reversed between the two categories.) In categories III and IV, the cue was curvature at the junctions between limbs. (The curvature/reward relationship was also reversed between categories.) The variations in distance and curvature were essentially continuous (9 steps). Importantly, each category also varied in ways that were irrelevant to value (right, “Irrelevant”). In categories I and II, curvature variations occurred but were irrelevant, and in categories III and IV, distance variations occurred but were irrelevant. Thus, both limb distance and curvature served as positive cues, negative cues, and irrelevant cues in different categories, ensuring that the monkeys were required to utilize the cues in a category-specific manner (and making category identification essential to the task). In addition to relevant and irrelevant shape variations, orientation also varied randomly, producing an effectively infinite range of stimuli. As a result, it would have been impossible to learn individual stimulus-reward pairings.

Figure 2. Behavioral task.

(A) Stimuli comprised four categories (rows). In each category, liquid reward value (symbolized by blue dots) was proportional to either the distance between limbs or the curvature at limb junctions. These were the value-relevant cues, shown on the left. In each category, the other cue varied but was irrelevant, as shown on the right. (B) Task sequence. After the appearance of a 0.2° diameter fixation point, the monkey was required to fixate within a 1° radius window throughout the trial. After initiation of fixation, the task stages were: stimulus 1 presentation (600 ms), delay (600 ms), stimulus 2 presentation (600 ms), delay (600 ms), simultaneous offset of fixation point and onset of saccade targets. The monkey was required to saccade to a target within 1 s to receive the liquid reward amount indicated by either stimulus 1 (blue target) or stimulus 2 (yellow target). Stimuli were generated randomly but balanced for category and reward level. Targets appeared on opposite sides of the fixation point at a randomized angle. (C) Behavioral performance. Probability of choosing the reward indicated by stimulus 1 (see color scale) was averaged across 90 training sessions for the first monkey and 75 training sessions for the second monkey. Reward values are shown in arbitrary units, which equated to a variable amount across sessions, ranging from 15 to 20 μl. The range of values for stimulus 2 was actually larger (0–12 units, data not shown) to prevent early decisions based on maximum or minimum values for stimulus 1 (3–9 units). Thus, the maximum reward ranged from 0.18 to 0.24 ml. See also Figure S1.

The task (Figure 2B) was a sequential value comparison task. It began with a 500 ms fixation period, followed by presentation of object 1, which was pseudo-randomly drawn from the domain of 4 categories × 9 reward cue levels × 9 irrelevant cue levels × orientation. After a delay, a second pseudo-randomly drawn stimulus was displayed. After another delay, two targets were displayed on opposite sides of fixation at a random orientation. Saccading to the blue target triggered delivery of the reward amount associated with object 1; saccading to the yellow target triggered the reward associated with object 2. (The positions of the blue and yellow targets were randomized to prevent early formation of a motor plan.) The chosen object was also redisplayed at the chosen target location to help reinforce shape/reward learning. One monkey was trained for 9 months prior to recording experiments, the other for 12 months. (TE was studied first in both, so additional training occurred prior to LPFC experiments.) For both monkeys (Figure S1), choice probability was a steep sigmoidal function of reward value difference, over a 3-fold range of absolute reward values (Figure 2C; e.g. dashed rectangle), demonstrating accurate decoding of value based on correct application of the shape variation rules according to category.

TE neurons tuned for category-specific value cues

Figure 3A exemplifies category-specific tuning for reward value cues in a single TE neuron. In each of the four categories, stimuli shown during the experiment are arranged from top left to bottom right in order of reward value. Background color indicates the neuron’s response to each stimulus (averaged across the 600 ms presentation period and 5 repetitions, which included presentations as both object 1 and object 2; see scale at right). This neuron was most responsive to category II, in which stimuli with large limb distances, signifying low reward amounts, evoked responses around 60 spikes/s. Thus, average responsiveness decreased as a function of decreasing limb distance, which signified increasing value. In contrast, for the second example TE neuron (Figure 3B), category IV stimuli evoked the strongest responses, specifically in the range where contour curvature was broad, which signified large reward amounts. Thus, in this case, average responsiveness increased as a function of decreasing curvature and increasing value. Given the comparative nature of the task, visual cues associated with both higher and lower rewards were potentially relevant. Temporal profiles of value cue tuning in TE are exemplified in Figure S2A–C.

Figure 3. Responses of example TE neurons.

(A) TE neuron tuned for large limb distance in category II. Each stimulus used to study this neuron is represented by a white icon. Background color indicates the time-averaged response rate to the stimulus (see scale bar) based on five repetitions including presentation as both object 1 and object 2. Stimuli are grouped into categories and arranged from lowest reward value (upper left) to highest (lower right). The shape cue indicating reward value is diagrammed beside each category. This neuron’s tuning provided potentially useful information about low reward values in category II. (B) TE neuron tuned for broad curvature in category IV. This tuning provided potentially useful information about high reward values in category IV. See also Figures S2 and S4.

In these examples, tuning for limb distance and curvature was specific to categories in which those cues were relevant to reward value. Responses were weaker and uncorrelated in categories where the same cues varied but were irrelevant. This reflects a strong trend in the population of 155 TE neurons studied here (Figure 4). For each neuron and in each category, we calculated the correlation (r) of response rate with both the relevant cue (either limb distance or curvature) and the irrelevant cue. We used a t-test (p < 0.05) to measure significance of correlation. The probability distributions of significant correlations are plotted in black for relevant cues and gray for irrelevant cues. Significant correlations were approximately two-fold more common for value-relevant cues compared to irrelevant cues.

Figure 4. Probability distributions for value cue tuning in TE.

Tuning was measured by the linear correlation (r) between value cue level and response. Results are shown as absolute r values. Only significant (p < 0.05, t-test) r values are plotted. Frequencies of tuning for limb distance and curvature are combined. The frequency scale (y axis) shows fraction of tests out of 310 (2 categories X 155 neurons) for both relevant and irrelevant cues in each bin. Tuning was stronger in categories where the cue was relevant (black) in comparison to categories where the cue was irrelevant (gray).

LPFC neurons tuned for category-invariant value

This category-specific tuning for value cues in TE contrasted with category-invariant tuning, apparently for value itself, in LPFC. Figure 5 shows the stimulus responses of an individual LPFC neuron. In all four categories, responses were strongest for the large reward range of shape variations, even though large rewards were associated with different or opposing shape cues in each category. This response pattern is consistent with previously described value-related responses in LPFC [1–6]. The temporal profiles of value tuning in LPFC are exemplified in Figure S2D.

Figure 5. Responses of example LPFC neuron.

In each category, this neuron responded more strongly to high value stimuli. In this earlier experiment, category IV stimuli had a simpler shape, and limb thicknesses varied, but the value cues were the same. See also Figure S2.

The example neurons illustrate a consistent difference between category-specific value cue tuning in TE and category-invariant value tuning in LPFC. To visualize this difference across the neural populations, we again used correlation (r) between response rates and reward value in individual categories. We plotted each neuron in Figure 6 according to its extreme r values across the four categories. Maximum absolute r (r_max, the r value with the largest magnitude, whether positive or negative) is represented on the horizontal axis. The most different r value (obtained using the same model) across the remaining three categories (r_min) is represented on the vertical axis. Neurons with category-invariant tuning for value itself should appear near the upper right, showing a consistent positive or negative value relationship across all four categories (similar r_max and r_min). Only 5/155 TE neurons showed a consistent, significant (t-test, p < 0.05) relationship across all four categories (filled circles at upper right in Figure 6A). Most TE neurons had only weak, non-significant r_min tuning (Figure 6A, open circles) or significant r_min tuning in the opposite direction for another category (filled circles at lower right in Figure 6A), meaning responses to high value cues in one category and low value cues in another category. This is predictable because tuning for either limb distance or curvature would produce opposite relationships to value in the two categories where that cue was relevant, because the value relationship for each cue was reversed across the two categories in which it was relevant. An example of this is shown in Figure S3A. Signals from these neurons would still be useful for calculating value in combination with information about category identity. Category tuning was strong in TE, providing a potential source of category identity information, while category tuning in LPFC was weak (Figure S3B).

Figure 6. Category specificity of value/value cue tuning in TE (A) and LPFC (B).

For each neuron, correlation between response strength and value/value cue level (r) was calculated for each of the four categories using linear regression. For each neuron, the largest absolute r value (r_max) out of the four categories is plotted on the x axis. The same linear model was then used to predict responses in the other three categories, in order to test for consistency of value tuning across categories. The most different r value (r_min, correlation between predicted and observed responses) among the other three categories is plotted on the y axis. Thus, positive values on the vertical axis represent consistent value tuning across all four categories. Filled circles indicate significant r_min (p < 0.05). See also Figure S3.

In contrast to TE, task-related tuning in LPFC was consistently related to value across categories (Figure 6B). All LPFC neurons with r_max above 0.5 had similar values across all four categories, since even the most different value (r_min) was very similar (near the diagonal) and often still significant (filled circles, 20/125). Thus, this subpopulation in LPFC carried category-invariant value signals that could be used to solve the task by comparing object 1 value with object 2 value, regardless of category, as required by the task. Because the task design included reversals of cue to value relationships between categories, this consistent value tuning observed in LPFC is necessarily independent of shape. In contrast, the common reversals in value relationships between categories in TE demonstrates that TE tuning is based on shape, and distinguishable from tuning for value per se (see next section).

First-pass value cue tuning in TE

The results presented above demonstrate category-specific value cue tuning in TE and category-invariant tuning for value itself in LPFC. The time courses of these signals (Figure 7) are critical for understanding their roles and interactions in the value comparison task. For each neuron in the TE and LPFC populations, we calculated response/value correlations (r) in each category across time in 1 ms bins (after smoothing and averaging response profiles). For this temporal analysis, we identified neurons for which r was significant over a continuous period of at least 50 ms in at least one category (55/155 TE neurons, 42/125 LPFC neurons; t-test, p < 0.05, corrected for four comparisons). Significant response/value correlations were roughly evenly split between positive and negative (26/55 TE neurons and 25/42 LPFC neurons had negative r values). Thus, value-related tuning in TE and LPFC cannot be explained as simple attentional enhancement of responses to more rewarding stimuli. Negative reward tuning could be useful for the reward comparisons required by the task.

Figure 7. Temporal profiles of TE and LPFC tuning.

To perform these analyses, spike trains were smoothed by convolution with an asymmetrical product of exponents filter to prevent backward biasing of response energy in time [62–64]. Tuning was evaluated for each 1 ms time bin. (A) Temporal tuning profiles for individual neurons with significant (p < 0.05) value-related tuning in at least one category for a continuous period of at least 50 ms. For each neuron, brightness corresponds to tuning strength (response/value correlation) in the category where its tuning strength was greatest (i.e., the category with maximum time-averaged response/value correlation, r_max, as in Figure 6, horizontal axis). Thus, brightness is analogous to curve height in a peri-stimulus time histogram. Color indicates consistency of value tuning across categories, in terms of minimum cross-predicted correlation among the other three categories (r_min, as in Figure 6, y axis). Neurons are rank-ordered from top to bottom within each subpopulation (TE and LPFC) according to value-tuning consistency. (B) Temporal profiles for category-specific value cue tuning in TE (green, averaged across 50 TE neurons for which r_min had a different sign or was not significant, t-test, p < 0.05) and category-invariant value tuning in LPFC (red, averaged across 20 neurons for which r_min had the same sign and was significant). See also Figures S4, S5 and S6.

To visualize tuning across time, we used the same r_max and r_min measures shown in time-averaged form in Figure 6. In Figure 7, each neuron is plotted as a strip against time (horizontal axis). Brightness of the strip represents time-varying tuning strength (correlation between responses and value cue level) in the neuron’s strongest tuning category (based on r_max, Figure 6, horizontal axis). Color represents r_min, the most different cross-predicted r value from among the remaining three categories. Red represents cases where r_max and r_min have the same sign, signifying consistent value tuning across all four categories, as in the top halves of the Figure 6 plots. Orange represents cases where r_min is near 0. Green represents cases where r_max and r_min have opposite signs, as in the bottom halves of the Figure 6 plots (see scale bar at right), reflecting an inconsistent relationship to value across multiple categories. In agreement with Figure 6, most (50/55) TE neurons (Figure 7A, top) are in the green to orange range, reflecting category-specific value cue tuning and no strong, consistent tuning for value itself across all four categories. Of these neurons, 44/50 exhibited significant (p < 0.05, corrected for four comparisons) cross-prediction of shape-tuning models across categories, confirming the shape-based nature of their task related tuning (Figure S4). Many LPFC neurons (Figure 7A, bottom), in contrast, are in the red range, consistent with the category-invariant tuning seen in Figure 6. None of the LPFC neurons in this analysis exhibited significant cross-prediction of shape-tuning models.

The temporal profiles of task-related signals were consistent with the first-pass mechanism, and inconsistent with the top-down mechanism. Category-specific tuning for value cues in TE emerged early, at the start of TE responses. This is apparent in individual time courses (Figure 7A) as well as in average tuning across 50 TE neurons (Figure 7B, green curve; this average excludes the 5 TE neurons with category-invariant tuning). Average value cue tuning in TE became significant (continuously greater than pre-stimulus baseline tuning, one-sided t-test, p < 0.05) 81 ms after stimulus onset, essentially at the start of IT responses, which significantly exceeded baseline (pre-stimulus) response levels 78 ms after stimulus onset. Average tuning for value-relevant cues was stronger than tuning for irrelevant cues (see Figure 4) throughout the response period, from 77 ms onward, and this difference became significant at 92 ms after stimulus onset. (Temporal profiles for all types of tuning in both area are presented in Figure S5.)

The immediate emergence of category-specific value cue information in TE can only be explained in terms of first-pass processing based on learning about what defines value in a given category. Our task design provided no information about value, category, or relevant value cue prior to stimulus onset, so early TE responses could not have been affected by pre-existing top-down signals. Any top-down modulation would necessarily depend on information derived from the object itself (as required by our task design), and thus could only affect later processing given the much longer visual latencies in PFC, OFC, and parietal cortex [5, 29–32]. A fast, alternative route for visual processing through the amygdala has been proposed, but even the fastest amygdala responses, to low spatial frequency components of fearful faces, do not emerge until ~70 ms after stimulus onset [33], and responses to other stimuli tested so far (to our knowledge) have latencies on the order of 100 ms or more [33, 34].

Task-related information emerged substantially later in LPFC. Average category-invariant value tuning in LPFC (Figure 7B, red curve) became significant 122 ms after stimulus onset, a delay of 40 ms relative to category-specific value cue tuning in TE. This delay is apparent in individual time courses for LPFC neurons. It is not associated with lower firing rates in LPFC neurons (Figure S6). It is also apparent for the small number of IT neurons with category-invariant value tuning, suggesting that such tuning depended on top-down feedback from LPFC or other value-processing regions. The 40 ms delay is consistent with the first-pass model in Figure 1, where category-specific value cue information from TE supports the subsequent emergence of category-invariant value signals in LPFC and/or other brain regions that compute generalized value.

DISCUSSION

Our results demonstrate that first-pass visual processing of category-specific value cues (Figure 1C) can occur after experience with a small number of object categories. The much stronger tuning for shape differences that defined value in a given category, the immediate onset of tuning for those differences in TE at about 80 ms after stimulus onset, essentially coincident with response onset, and the subsequent emergence of general, category-invariant tuning for value itself in LPFC 40 ms later, are all strongly consistent with a first-pass mechanism. (Potential sources of top-down modulation in prefrontal and parietal cortex do not reflect even simple visual differences until ~125 ms after onset [32]. Visual response latencies in orbitofrontal cortex are also later, with median values in the 120 ms range or higher [5, 29–31].) Our results are incompatible with the proposed top-down mechanism (Figure 1B), based on previous demonstrations of category representation and attentional control in LPFC [20–27], because TE tuning for value cues was too early for top-down control and category information was not observed in LPFC (Figure S3B).

This rapid feed-forward mechanism in TE for value cue signals could be important for highly familiar, behaviorally relevant objects in the real world. It could speed time-critical behavioral responses during foraging, courtship, and predator evasion. It could also free up cognitive processing resources for higher-level operations like comparison and decision-making. Analysis of fine visual detail depends on the ventral pathway, making it the logical site for initial processing of subtle value cues. Ventral pathway value cue processing is also consistent with the immediate, automatic nature of value perception for familiar, important objects.

The response patterns we observed reflect changes in shape processing based on learning about shape-reward relationships. This is consistent with previous demonstrations of other types of learning effects in TE [35–49]. In particular, Tanaka and colleagues [50,51] found that about 20% of TE neurons showed significant positive or negative differences in responses to rewarded and unrewarded stimuli in a go/no-go visual discrimination task, although variance due to reward contingency was a small fraction of variance due to stimulus selectivity in most cases. The average onset latency for reward dependency was near 200 ms, indicating a delayed, top-down effect unrelated to the first-pass effects described here.

In addition, Kaskan et al. [52], used fMRI in monkeys to show that learning produced stronger responses to high-value images in TE, LPFC, and medial frontal cortex, results that are in accordance with human imaging studies [53–59]. Consistent with our results, Nelissen et al. [60] used fMRI in monkeys to show that VLPFC and OFC are differentially responsive to value-related shapes during learning, but post-learning this difference is only observed in visual cortex. Livingstone and colleagues have shown that training on relative object value can reorganize ventral pathway cortex in young monkeys [61]. They demonstrate that large functional modules, distinguishable with fMRI, can develop for arbitrary groups of stimuli through behavioral experience, and need not reflect evolutionary hard wiring. This does not necessarily show that value information per se is processed in ventral pathway, but it does show that extensive experience with stimuli in a value comparison task can lead to highly specialized, dedicated processing mechanisms.

In our experiment, we did not see evidence for a mechanism in which category identification in LPFC leads to top-down enhancement of visual value cue processing in TE (Figure 1B). Task-relevant information did not appear in LPFC until 40 ms after category-specific value cue tuning appeared in TE. Moreover, the task-relevant information in LPFC was category-invariant value tuning rather than category identity signals. While many TE neurons (53/155) exhibited strong tuning (P < 0.01, F test) for category identity (Figure S3B), we found no instances of tuning for category identity at that statistical threshold in our sample of 125 LPFC neurons.

This contrasts with a well-known study by Freedman and colleagues [20] showing that, in a categorical delayed-match-to-sample task (DMS), LPFC manifested clear, binary signals for 2-alternative category identity, even though visual variation was continuous across the stimulus space and the category boundary was arbitrary. In the same experiment, TE neurons did not express strong signals for category identity, although they did carry task-relevant shape information [21]. There are two salient differences between the experiments that could explain this discrepancy. First, in our experiment, the four categories had distinct differences in medial axis shape, which is represented strongly in TE even without training [28]. In Freedman et al. [20], the category boundary was an arbitrary, subtle division in a continuous space of cat- and dog-like shapes. TE neurons carry information about fine shape differences [41], but might be unable to exhibit the precipitous drop-offs in response at arbitrary boundaries because they represent structure, not abstract meaning.

Second, category identity was only an intermediate logical value in our task design. It was important only for defining the correct shape cue and its positive or negative relationship to value. There was no need for explicit cognitive representation of category identity or storage of category identity in short-term memory, as there was in Freedman et al. [20]. Once category identity had been used to interpret shape cues correctly, it became irrelevant to the task, since only value (liquid reward) needed to be stored in memory and compared at the end of the task. In Freedman et al. [20], in contrast, identity was the final variable to be compared to determine the behavioral response. Thus, on both stimulus design and task design grounds, the two experiments predispose different results for neural representation of object categories. It is also possible, however, that the discrepancies are explained by different recording locations within large areas (TE and LPFC) in which compartmentalization remains uncertain.

STAR Methods

CONTACT FOR REAGENT AND RESOURCE SHARING

Requests for resources should be directed to and will be fulfilled by the Lead Contact, Dr. Charles E. Connor (connor@jhu.edu).

EXPERIMENTAL MODEL AND SUBJECT DETAILS

Two adult female rhesus macaques (Macaca mulatta) weighing 6.0 and 4.0 kg were used in this study. These animals were not involved in previous procedures. They were group housed prior to training but singly housed during training and experiments. All procedures were approved by the Johns Hopkins Animal Care and Use Committee and conformed to US National Institutes of Heath and US Department of Agriculture guidelines.

METHOD DETAILS

Behavioral task and stimuli

Both monkeys were trained to perform an object value comparison task. Throughout each behavioral trial, monkeys were required to maintain fixation on a small red spot within 1° of visual angle. Eye position was monitored with an infrared eye tracker (ISCAN). Object stimuli were rendered in OpenGL and presented on a computer monitor in white against a gray background, centered at fixation. The maximum stimulus dimension range was 5–6.5° of visual angle.

The task required monkeys to estimate and compare the reward values of two object stimuli randomly drawn from four categories. The four categories were defined by their medial axis shapes (Figure 2A), independent of orientation. In other words, orientation of all stimuli varied randomly (at 45° intervals) and monkeys were required to ignore and generalize across these orientation changes. Each category-defining shape was a specific configuration of connected limbs. The use of common constituent parts across categories ensured that category recognition required global shape perception.

Variable distance between limbs and curvature at limb junctions served as both value-defining cues and irrelevant cues in different categories, and each category had both relevant and irrelevant feature variations. Relevant and irrelevant features varied randomly, but with constraints to ensure even sampling of cue values and even sampling of reward value differences between the two stimuli in each trial. The categories of the two stimuli in each trial also varied randomly (but were constrained for equal sampling of the four categories), and were always unknown to the animal prior to stimulus onset. Finally, stimulus orientations varied randomly.

Neural recording

We recorded electrical spike times of 155 well-isolated single units in area TE in the two monkeys, 101 from the 6.0 kg female (monkey 1, right hemisphere) and 54 from the 4.0 kg female (monkey 2, left hemisphere). We recorded spike times of 125 well-isolated single units in LPFC in the same two monkeys, 100 from monkey 1, left hemisphere, and 25 from monkey 2, right hemisphere. Recordings were made with polyamide-coated, 125 μm diameter, 2–4 MΩ tungsten electrodes (AM Systems, Frederick Haer, Microprobe). Electrical activity was amplified and filtered and single units were discriminated with multiple time-amplitude windows on a Tucker-Davis Technologies recording system. Spike times were digitized at a temporal resolution of 25 KHz.

Electrodes were introduced through a 25-gauge guide tube using a custom microdrive system mounted in an acrylic cap attached with orthopedic bone screws. Electrode positioning was based on anatomical MRI images for each monkey. TE neurons were recorded from the ventral bank of the superior temporal sulcus and the lateral convexity of the inferior temporal gyrus, over an anterior-posterior range of 12–20 mm anterior to the external auditory meatus. LPFC neurons were recorded anterior to the arcuate sulcus, 0–3 mm dorsal or ventral to the principal sulcus.

QUANTIFICATION AND STATISTICAL ANALYSIS

Neural tuning for value cues and absolute value

Response rates for each stimulus were averaged across the 600 ms presentation window and across 5 repetitions. Value-related tuning was characterized with linear regression models fit to individual stimulus categories, with value cue level as the independent variable and response rate as the dependent variable. Statistical significance of value-related tuning (correlation between response rate and value cue level) was measured with t-tests corrected for 4 comparisons. Significance of correlation was measured with t-tests (p < 0.05) corrected for 4 comparisons. Significant tuning for value cues could occur in one or more categories, potentially with opposite relationships to absolute value. Tuning for absolute value was defined as significant tuning in all four categories with the same sign with respect to absolute value (r > 0 in all four categories, or r < 0 in all four categories). Magnitude of absolute value tuning is expressed throughout the paper as tuning strength (r) in the category with the smallest absolute r value.

Temporal response profiles

For temporal analyses, spike trains were smoothed with an asymmetric Gaussian function (15 ms SD causal side, 5 ms SD acausal side) [62–64] and averaged across all repetitions of each stimulus. This procedure yields a robust estimate of instantaneous response rate that avoids backward bias in time by means of the primarily causal weighting in the smoothing filter. To determine when population average tuning curves became significantly different from baseline level (averaged over the period from −50 to 0 ms relative to stimulus onset), we performed a one-sided t-test in each 1 ms time bin, and found the first point at which significance exceeded p < 0.05 and remained above that level. The same procedure was used to determine the time at which average TE response levels became significantly different from baseline.

Shape tuning models

We quantified shape tuning with linear/nonlinear models based on L- and T-shaped medial axis fragments, using procedures described in previous publications [28, 64] and explained here. The shapes and positions of these fragments defined category identity and value cue levels in the behavioral task. Models were based on 2–4 multi-dimensional Gaussian tuning functions (i.e. products of 1-dimensional Gaussians) for L- or T-fragment object-centered position in polar coordinates, orientation, and curvature. Curvature was squashed to a range from −1 (acute concave) through 0.0 (flat) to 1 (acute convex) with the following function:

$$c^{\prime }=\frac{2.0}{1+e^{-a·c}}-1.0$$

where c is curvature (radians/cm; at the screen distance of 50 cm, 1 cm subtended 1.15° of visual angle), c′ is squashed curvature, and a = 0.05. Thus, each L- or T- fragment was represented as a point in a 4-dimensional space, and each stimulus was represented as a set of such points corresponding to its constituent elements. The 2–4 range of component Gaussian tuning functions corresponds to the model complexities providing best cross-validated fits to IT response patterns in previous analyses, based on the Bayesian Information Criterion and/or cross-prediction between independent datasets [28,64].

For a given stimulus, the response predicted by each Gaussian tuning function was the sum of function values at the points corresponding to that stimulus’ L- and T- fragments (i.e. inner product between Gaussian and contour element points):

$$\begin{gathered} \mathrm{R}=\mathrm{A}·\sum _{\mathrm{c}=1}^{\#\text{fragments}}\mathit{gaussian}({\mathrm{k}}_{\mathrm{c}},{\mathrm{o}}_{\mathrm{c}},{\text{rp}}_{\mathrm{c}},{\text{ap}}_{\mathrm{c}}) \\ =\mathrm{A}·\sum _{\mathrm{c}=1}^{\#\text{fragments}}{e}^{-\left(\frac{{(k_c-{\mu }_k)}^2}{2{\sigma }_k^2}+\frac{{(o_c-{\mu }_o)}^2}{2{\sigma }_o^2}+\frac{{(r{p}_c-{\mu }_{rp})}^2}{2{\sigma }_{rp}^2}+\frac{{(a{p}_c-{\mu }_{ap})}^2}{2{\sigma }_{ap}^2}\right)} \end{gathered}$$

where k_c, o_c, rp_c, and ap_c are the (fixed) curvature, orientation, relative position, and absolute position values for each contour element in the stimulus, the μ’s and σ’s are the fitted Gaussian centers and standard deviations on each of these 4 dimensions, and A is the fitted Gaussian amplitude (positive/excitatory or negative/inhibitory).

For each neuron, we fitted models based on combinations of 2–4 Gaussian component functions. A model’s predicted response to each stimulus was a weighted combination of the individual Gaussian function responses (the linear component) and products of Gaussian responses of the same sign (the nonlinear component). All possible pairwise product terms were tested in different model-fitting procedures. By varying weights of linear and nonlinear terms, our models could range continuously from linear summation across medial axis fragments to nonlinear selectivity for fragment combinations. In equation form, the predicted response was:

$$\mathrm{R}=\text{gain}·{\lfloor \sum _{\mathrm{s}=1}^{\#\text{Gaussians}}\left({\mathrm{w}}_{\mathrm{s}}{\mathrm{R}}_{\mathrm{s}}\right)+{\mathrm{w}}_{{\text{NL}}_{+}}\prod _{\mathrm{s}=1}^2\left({\mathrm{R}}_{\mathrm{s}}\right)-{\mathrm{w}}_{{\text{NL}}_{-}}\prod _{\mathrm{s}=1}^2\left({\mathrm{R}}_{\mathrm{s}}\right)\rfloor }^{+}+{\mathrm{b}}_0$$

where R_s is the unweighted response predicted by each Gaussian alone, w_s is the fitted linear weight (amplitude) for each Gaussian, w_NL+ and w_NL− are the fitted weights for the excitatory and inhibitory nonlinear terms, b₀ is the baseline neuronal firing rate, and ⌊·⌋⁺ represents rectification of the predicted response at 0 spikes/s.

These models were fitted to each neuron’s response data using an iterative nonlinear least-squares algorithm (MATLAB, lsqnonlin function) to minimize the sum of squared differences between observed and predicted responses. For each neuron, we used a stepwise regression procedure to determine the optimum number of Gaussian tuning functions to explain responses without over fitting. Fits were validated by cross-prediction of responses in one held-out category by models fit to the remaining 3 categories. Significance of cross-prediction was measured with a t-test for correlation between predicted and observed responses, at a statistical threshold of 0.05, corrected for 4 comparisons to 0.01. Addition of Gaussian tuning functions to the model was arrested at the level where cross-prediction was highest, and further addition produced over-fitting of the training categories and lower cross-prediction of the held out category.

DATA AND SOFTWARE AVAILABILITY

Data from these experiments and analyses can be obtained by request to the Lead Contact.

KEY RESOURCES TABLE.

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Experimental Models: Organisms/Strains
Rhesus macaques (Macaca mulatta)	Johns Hopkins Univ	N/A
Software and Algorithms
MATLAB	MathWorks	http://mathworks.com
OpenGL	Khronos Group	https://OpenGL.org
Other
Tungsten microelectrodes	FHC	https://www.fh-co.com
	Microprobes	https://www.microprobes.com
	A-M Systems	https://www.a-msystems.com

HIGHLIGHTS.

• Visual neurons respond early to shape variation cues that indicate object value

• These responses are specific to value cues relevant to particular object categories

• These responses precede absolute value signals in prefrontal cortex

• This suggests a rapid processing mechanism for value in familiar object categories