Shape encoding consistency across colors in primate V4

Abstract

Neurons in primate cortical area V4 are sensitive to the form and color of visual stimuli. To determine whether form selectivity remains consistent across colors, we studied the responses of single V4 neurons in awake monkeys to a set of two-dimensional shapes presented in two different colors. For each neuron, we chose two colors that were visually distinct and that evoked reliable and different responses. Across neurons, the correlation coefficient between responses in the two colors ranged from −0.03 to 0.93 (median 0.54). Neurons with highly consistent shape responses, i.e., high correlation coefficients, showed greater dispersion in their responses to the different shapes, i.e., greater shape selectivity, and also tended to have less eccentric receptive field locations; among shape-selective neurons, shape consistency ranged from 0.16 to 0.93 (median 0.63). Consistency of shape responses was independent of the physical difference between the stimulus colors used and the strength of neuronal color tuning. Finally, we found that our measurement of shape response consistency was strongly influenced by the number of stimulus repeats: consistency estimates based on fewer than 10 repeats were substantially underestimated. In conclusion, our results suggest that neurons that are likely to contribute to shape perception and discrimination exhibit shape responses that are largely consistent across colors, facilitating the use of simpler algorithms for decoding shape information from V4 neuronal populations.

the ventral visual pathway is the main locus of form and color processing in the primate brain (Felleman and Van Essen 1991; Ungerleider and Mishkin 1982). Lesion studies at various stages along this pathway have demonstrated impairment on form and color discrimination tasks (Aggleton and Mishkin 1990; Cowey and Gross 1970; De Weerd et al. 1996; Dean 1976; Gross et al. 1971; Heywood and Cowey 1987; Heywood et al. 1995; Huxlin et al. 2000; Iwai and Mishkin 1969; Merigan 1996; Merigan et al. 1993; Walsh et al. 1992a, 1992b), and single-cell neurophysiological studies have demonstrated selectivity for various attributes of stimulus form and color (Baizer et al. 1977; Brincat and Connor 2004; De Valois et al. 1979; Desimone and Schein 1987; Desimone et al. 1984; Gallant et al. 1993; Gouras and Kruger 1979; Gross et al. 1972; Hegde and Van Essen 2000; Hubel and Livingstone 1990; Hubel and Weisel 1968; Ito and Komatsu 2004; Kobatake and Tanaka 1994; Komatsu et al. 1992; Movshon et al. 1978a, 1978b; Pasupathy and Connor 1999, 2001; Perrett et al. 1982; Peterhans and von der Heydt 1991; Schein and Desimone 1990; Tanaka et al. 1991; Thorell et al. 1984; Zeki 1973, 1977). However, we do not know how neuronal selectivity for form and color interact. This is because most studies that investigate form encoding present all visual stimuli in a single color (for example, see Desimone and Schein 1987; Gallant et al. 1993; Pasupathy and Connor 1999, 2001), and studies that investigate color encoding do so with a single shape (for example, see Schein and Desimone 1990; Komatsu et al. 1992). This experimental strategy of characterizing stimulus dimensions one at a time is dictated largely by practical constraints of recording stability and experimental duration, but an implicit assumption of this approach is that tuning for shape (color) is consistent across colors (shapes), i.e., shape and color tuning are separable. Such tuning separability is attractive from a decoding standpoint, because if tuning for color and shape are separable, a simple basis function decoding approach can be used to decode both the shape and color of the stimuli from a population of neurons with sufficient overlap and coverage in the shape × color space. If, however, tuning were not separable, experimentalists would need to arrive at an efficient way to jointly study shape and color tuning, and the brain would need to implement appropriate algorithms to decode color and shape information from such populations of neurons.

To assess tuning separability, one would ideally want to study the responses to many shapes presented in many different colors. This, however, is highly impractical in experiments involving awake fixating primates; for example, recording the neuronal response to 10 repeats of 100 shapes presented in 10 different colors takes ∼5 h, not including the initial receptive field characterization. Therefore, rather than take on this difficult endeavor, as a first step to examining the separability of shape and color tuning of neurons, we compared responses to 56–136 shapes presented in 2 different colors. We targeted neurons in extrastriate cortical area V4, an intermediate stage along the ventral visual pathway. We found that many V4 neurons showed strong consistency in shape preferences across colors, in keeping with previous results from inferior temporal (IT) cortex (Komatsu and Ideura 1993). Our results suggest that neurons that are likely to contribute to shape discrimination (those with less eccentric receptive fields and those that exhibited greater dispersion of responses across shapes) showed greater consistency of shape responses across color. From a practical standpoint, our results empirically suggest that greater than 10 stimulus repeats are essential to obtain a good estimate of the true consistency in shape responses across colors for the experimental and analytical methods used here.

MATERIALS AND METHODS

Animal preparation for the experiments described in this study, including implants, surgeries, and behavioral training, conformed to National Institutes of Health guidelines and was approved by the Institutional Animal Care and Use Committee at the University of Washington. All methods were previously described in detail by Bushnell et al. (2011a), but briefly, two rhesus monkeys (Macaca mulatta) were surgically implanted with custom-built head posts and recording chambers attached to the skull with orthopedic screws. A craniotomy was performed in a subsequent surgery. The recording chamber targeted area V4, and electrode penetrations were largely in the prelunate gyrus and occasionally in the adjoining sulcal banks.

Animals were trained to fixate a 0.1° white dot within 0.5–0.75° of visual angle while seated in front of a computer monitor at a distance of 57 cm. Eye position was monitored using a 1-kHz infrared eye-tracking system (Eyelink 1000; SR Research, Osgode, ON, Canada). Each trial began with the presentation of a fixation spot at the center of the screen. Once fixation was acquired, 4–6 stimuli were presented in succession, each for 300 ms, separated by interstimulus intervals of 200 ms. Stimulus onset and offset times were based on photodiode detection of synchronized pulses in the lower left corner of the monitor.

During each recording session, a single dura-puncturing microelectrode (FHC Instruments, Bowdoin, ME), was lowered into the cortex with an eight-channel acute microdrive system (Gray Matter Research, Bozeman, MT) and single units were identified using a spike-sorting system (Plexon Systems, Dallas, TX).

Stimuli were presented on a cathode ray tube monitor (40.6 × 30.5 cm; 97-Hz frame rate; 1,600 × 1,200 pixels) calibrated with a spectrophotoradiometer (PR650; PhotoResearch, Chatsworth, CA) against an achromatic background of mean luminance = 5.4 cd/m². For each isolated unit, an initial qualitative preferred stimulus (shape, color, orientation) and a rough receptive field (RF) location were identified using a variety of shapes under the experimenter's control. This was followed by an automated RF mapping procedure that presented the initial preferred stimulus in a densely sampled grid; the refined RF center was based on a two-dimensional Gaussian fit to the data. To characterize color preferences, for many neurons (35/60), we measured the responses to the preferred shape presented in 25 chromaticities at 4 different luminance values (2.7, 5.4, 8.1, and 12.1 cd/m²). The 25 chromaticities provided a roughly uniform sampling of the CIE color space (see Bushnell et al. 2011b). For the remaining neurons (25/60), detailed color characterization was bypassed in the interest of time.

To assess consistency of shape preferences across colors, we studied responses of 60 V4 neurons that 1) were well isolated, 2) were clearly responsive to visual stimuli, 3) had well-defined RFs during preliminary characterization, and 4) remained well-isolated for 4 or more repeats in the experiment described below. We studied the responses of each such neuron to 7–17 shapes, each presented at 8 orientations at 45° intervals, in two different colors, for a total of 56–136 shapes. The tested shapes (Fig. 1) were designed to explore a range of convex and concave contour features known to be effective at driving responses of a large fraction of V4 neurons (Bushnell et al. 2011a; Pasupathy and Connor 1999, 2001). Eighteen of the 60 cells were studied using the 17 shapes shown in the top row of Fig. 1. Data from the remaining 42 cells were collected as part of a separate study on partial occlusion, and the shapes used (which were also subsets of shapes in Fig. 1) were dictated by the parameters of that study (Bushnell et al. 2011a). Stimuli were sized such that all parts of all stimuli were entirely within the estimated RF area (estimated RF diameter = 1.0° + 0.625 × RF eccentricity, based on data from Gattass et al. 1988) of the cell under study. In every case, the stimulus size so-chosen fit within the RF boundary measured by our fine grid-mapping procedure (see above).

Fig. 1.

Shape stimuli. A subset (7 to 17) of the 34 stimuli shown was used to characterize the shape preferences of neurons. These shapes were designed to explore a range of convex and concave contour features known to drive selective V4 responses (Bushnell et al. 2011a; Pasupathy and Connor 2001). All shapes were presented in 8 orientations at 45° intervals. Stimuli were sized such that all parts of all stimuli were entirely within the receptive field (RF) of the cell. Because shapes were centered at a radial distance of 0.25 × RF radius from the center of the RF (which was the axis of rotation), the different rotations sampled different positions within the RF. This positioning strategy, which seems arbitrary here, was dictated by the requirements of a previous study (Bushnell et al. 2011a) where it is described in detail; it does not impact the present results or conclusions.

The two colors of the shape stimuli were customized for each cell such that both evoked moderate to strong responses and were visually distinct from each other. Color choices were based on either detailed color characterization (35 cells; see above) or preliminary characterization by the experimenter (25 cells). The distance in CIE space between the chosen colors (color distance) essentially reflected the width of color tuning, because we chose two colors that were farthest apart in CIE space and still evoked responses from the cell. Thus the chosen colors were closer together for cells with narrow color tuning and farther apart for cells with broader tuning. Of the 60 cells, 52 were tested with 2 colors that differed in their chromaticity but were at the same luminance contrast relative to the background. Of the remaining 8 cells, 7 were tested with 2 colors that differed in both their chromaticity and luminance contrast, and one with achromatic stimuli at two different luminance contrasts.

For each stimulus presentation, we counted spikes in a 300-ms window shifted by 50 ms to account for average latency of V4 neurons (Bushnell et al. 2011a). We then averaged these counts across stimulus repeats and divided by the counting window duration (300 ms) to obtain the spike rate. Across neurons, the number of stimulus repeats ranged from 4 to 20 (median 10).

RESULTS

To assess whether shape preferences of V4 neurons are consistent across colors, we studied the responses of 60 single neurons in 2 monkeys to a variety of complex shapes (see Fig. 1) presented in 2 different colors (see materials and methods). Results for an example neuron are shown in Fig. 2. In preliminary (qualitative) color characterization, this neuron did not show strong color tuning, and we chose green and magenta, both of which evoked strong responses, as the stimulus colors. We studied responses of this neuron to 14 different complex shapes (rows in Fig. 2, A and B) presented at 8 different orientations (columns) in green (Fig. 2A) and magenta (Fig. 2B). Many shapes evoked strong responses from this cell, and the shape preferences across the two colors were quite consistent as demonstrated by the scatter plot in Fig. 2C. We quantified the consistency in shape responses by calculating the linear correlation coefficient, r_C, between responses to shapes in the two colors. The value of r_C ranges from −1, for perfectly inconsistent responses, to 1, for perfect consistency. For this example neuron, r_C was 0.87. In addition to the strong correlation, it is also clear from the scatter plot (Fig. 2C) that the neuron responded more strongly to color 1 (points lie mainly below the diagonal); a two-way ANOVA revealed main effects for both color and shape and an interaction between the two factors (P < 0.01). A strong correlation in responses across colors and a significant interaction between color and shape (as exhibited by this cell) are not at odds with each other; a simple scaling of shape responses by color, for example, could produce the observed result.

Fig. 2.

Responses of an example V4 neuron with highly consistent shape responses across colors. A and B: responses to 14 shapes (rows) presented at 8 orientations (columns) in green (A) and magenta (B). The background gray level of each icon represents the average response, per scale bar, across N = 20 repeats to the superimposed shape. Qualitatively, this neuron did not show strong color preferences, but 2-way ANOVA revealed significant (P < 0.01) main effects for color, shape, and their interaction on the responses of the cell, which were marginally stronger to shapes in green (A). Shape preferences were consistent across colors: in both colors, responses were strong to several shapes with a concavity to the right or lower right of the shape. The RF of this neuron was centered at 8.2° from the fixation point. C: scatter plot of responses shown in A (X-axis) and B (Y-axis). Points lie below the line of slope = 1 (dashed line). The correlation coefficient between responses in the 2 colors (r_C) was 0.87.

Two additional examples are shown in Fig. 3. Figure 3, A–C, shows an example neuron that exhibited a moderate color preference during preliminary color characterization, responding preferentially to blue and more weakly to red. A two-way ANOVA revealed main effects for both color and shape and an interaction between the two factors (P < 0.01). Only a few shapes evoked strong responses from this cell, but the preferred shapes showed some consistency across the two colors, yielding a response correlation (r_C) of 0.58. The third example, in Fig. 3, D and E, underwent a detailed color characterization with 25 colors that sampled the CIE space, presented at 4 different luminance contrasts relative to the background (see materials and methods). The four panels in Fig. 3D show chromatic responses at each of the four luminance contrasts tested (−50%, 0%, 50%, and 125%). Within each panel, the various chromaticities and the corresponding neuronal responses are indicated by the position and surface color of the small squares, respectively. Responses of this neuron were strongest to blue and magenta at positive luminance contrasts. For the main experiment, green and blue at 125% luminance contrast (labels 1 and 2 in Fig. 3D) were chosen as the stimulus colors, and the responses to shapes in these two colors are shown in Fig. 3E. Here too, the shape responses showed a moderate level of consistency across the two colors, yielding a r_C of 0.58.

Fig. 3.

Responses of 2 example V4 neurons showing moderately consistent shape responses across colors. A and B: responses of a neuron to 14 shapes (rows) presented at 8 orientations (columns) in blue (A) and red (B). All other details are as in Fig. 2. This neuron exhibited moderate color tuning; responses were stronger to shapes in blue (A) than in red (B). Shape preferences were largely consistent: in both colors, responses were strongest to shapes with a sharp convexity pointing up. The RF of this neuron was centered at 2.6° from the fixation point. C: scatter plot of responses shown in A (X-axis) and B (Y-axis). Responses (based on N = 20 repeats) were largely below the diagonal line, reflecting stronger responses to shapes in blue. Correlation coefficient between responses in the 2 colors was 0.58. D and E: responses of a neuron that underwent detailed color characterization. D: each panel shows the neuron's chromatic responses at one of the luminance contrasts tested (−50%, 0%, 50%, and 125%). The X- and Y-axes of each panel represent CIE x- and y-coordinates respectively. Surface color indicates average response strength across 6 repeats, per scale bar. Responses at −50% contrast were uniformly weak. At 0% and positive contrasts, neurons showed a moderate level of color selectivity, responding strongest to stimuli in blue and magenta. For the highest luminance contrast panel (125%), labels 1 and 2 identify the 2 chosen stimulus colors (green and blue, respectively); responses were moderate for green and strong for blue. E: scatter plot of responses (based on N = 16 repeats) to shapes presented in green and blue (1 and 2 in D) along the X- and Y-axes, respectively. This neuron responded preferentially to a few shapes, but the responses in the 2 colors were largely consistent, yielding a correlation coefficient of 0.58.

Figure 4A shows a histogram of shape consistency values for the 60 cells in our data set. Consistency values ranged from −0.03 to 0.93 (mean 0.49; median 0.54). Thus many V4 neurons showed shape preferences that were strongly consistent across the two colors tested, but many others showed weak or no consistency. The lack of consistency in shape responses could result if a neuron's shape tuning is color dependent, i.e., a neuron prefers one shape in color 1 but a different one in color 2. However, poor shape consistency could also result if the neuron in question is simply not shape selective. Such a neuron will respond similarly to a variety of shapes, and consistency across colors will be close to zero because the range of responses across shapes will simply reflect the inherent noise in the neuron's response. To identify poorly shape-selective neurons in our data set, we quantified the strength of shape selectivity by calculating shape-response dispersion. This metric, which provides a measure of the shape information capacity of a neuron, was quantified as the ratio of the variance to the mean response across shapes in a given color. If n_s represents number of stimuli and r_i is the mean response to the ith stimulus, then shape dispersion is equal to Var/μ, where

$$\begin{array}{c}\mu =\frac{1}{n_s}{\displaystyle \sum _{i=1}^{n_s}{r}_i}\\ \text{var}=\frac{1}{n_s}{\displaystyle \sum _{i=1}^{n_s}{\left(r_i-\mu \right)}^2}\end{array}$$

Fig. 4.

Population results. A: histogram of shape consistency (r_C) across 60 V4 neurons. Shape consistency was quantified by the correlation coefficient between shape responses in 2 different colors. Across the population, shape consistency values varied widely, ranging from −0.03 to 0.93 (median 0.54). B: scatter plot of shape dispersion (X-axis) vs. shape consistency (Y-axis). Shape dispersion was quantified as the ratio of the variance to the mean response across all shapes. For each cell, represented by a pair of dots connected by a line, the open and filled symbols represent shape dispersion based on shapes in the preferred and nonpreferred colors, respectively. There was a significant positive correlation (r = 0.54 for preferred color and r = 0.55 for nonpreferred color; P < 0.0001) between shape dispersion and Fisher-transformed shape consistency values. The vertical dashed line denotes shape dispersion = 1.5, the cutoff between poorly and strongly selective neurons. C: scatter plot of F-test statistic (X-axis) vs. shape consistency (Y-axis). The 1-way ANOVA F-test statistic measures across-shape variability vs. within-shape variability. Note broken X-axis. There was a strong and significant positive correlation between F value and Fisher-transformed shape consistency (r = 0.74 and 0.63 for preferred and nonpreferred colors, respectively; P < 0.01) D: histogram of shape consistency values for shape-selective and unselective cells classified on the basis of whether they exhibited significant shape selectivity as based on a 1-way ANOVA (P < 0.05). Cells with significant selectivity to shape presented in both colors were classified as shape selective. Shape-selective cells (44/60) had shape consistency values that ranged from 0.16 to 0.93 (median 0.63).

A neuron that shows strong shape selectivity will be associated with large shape dispersion because it will exhibit a large dynamic range in its responses across shapes and, thus, a large variance-to-mean ratio across shapes; a neuron with weak shape selectivity will exhibit similar responses across shapes and a small variance-to-mean ratio. Therefore, V4 neurons that exhibit poor consistency due to color dependence of shape selectivity will be associated with large shape-response dispersion, whereas those that are unselective to shape will be associated with low dispersion values.

Figure 4B summarizes the relationship between shape dispersion (X-axis) and shape consistency (Y-axis) across our data set. Data from each cell are denoted by a pair of dots connected by a line segment; each dot denotes shape dispersion based on stimuli in one color (see Fig. 4 legend for further details). Across our sample of V4 neurons there was a wide range in dispersion values, but one trend is evident: shape-selective neurons, i.e., neurons with high shape dispersion, were associated with high shape consistency values. To illustrate this point more clearly, we divided the population of neurons into two groups: those with shape dispersion values >1.5 (strongly selective = 22) and those with dispersion values ≤1.5 (weakly selective = 38). Mean shape consistency for the strongly selective and weakly selective groups was 0.71 and 0.36, respectively, and the Fisher-transformed mean value for the strongly selective group was significantly larger than that for the weakly selective group (t-test, P < 0.01). In fact, among strongly selective neurons, shape consistency was >0.5 for 19/22 neurons. Note that in Fig. 4B, whereas neurons with high dispersion values are associated with high consistency, the converse is not true, i.e., neurons that exhibit highly consistent responses across colors (for example, r_C > 0.5) are not always associated with high shape dispersion values. Those neurons with high consistency and low shape dispersion values (top left corner of plot, Fig. 4B) were cells that typically exhibited a small dynamic range in their response across shapes even though such responses were highly consistent. Because shape dispersion does not take the noise variance of the neurons into account, we also quantified shape selectivity using the one-way ANOVA F-test statistic (Fig. 4C), which measures the ratio of across-shape variability to within-shape variability. Results were very similar to those in Fig. 4B: there was a strong positive correlation between Fisher-transformed shape consistency and the F value (r = 0.74 and 0.63 for preferred and nonpreferred colors, respectively). Figure 4D shows the histogram of shape consistency values for cells divided into two groups on the basis of whether they exhibited significant shape selectivity for both colors as based on the one-way ANOVA (P < 0.05). Among the shape-selective group (44/60), shape consistency ranged from 0.16 to 0.93 (mean 0.61; median 0.63). These results suggest that cells with low dispersion and a lack of shape selectivity may have low consistency values because the variance across shape is largely due to noise. Interestingly, we also found that the RF eccentricity of individual neurons showed a weak but significant negative correlation with shape dispersion (r = −0.32; P < 0.02) and shape consistency (r = −0.42; P < 0.01). These results are consistent with the idea that cells more likely to contribute to shape coding, i.e., those with less eccentric RFs and high shape-response dispersion, have more consistent shape tuning across colors.

Poor consistency in some cells could be due to an underestimation of the correlation metric r_C as computed on the basis of finite stimulus repeats. If we denote the true responses to shapes in the two colors by vectors X and Y, respectively, then the true shape consistency = Corr(X, Y). However, when we study shape responses with a finite number of repeats N, assuming additive noise, mean estimates will be X + Z_X and Y + Z_Y, where Z_X and Z_Y denote noise in the estimates of X and Y, respectively. The measured shape consistency, r_C, is therefore Corr(X + Z_X, Y + Z_Y). If Z_X and Z_Y are independent of the true means and of each other, then it is easy to see that Corr(X + Z_X, Y + Z_Y) = Cov(X, Y)/Var(X + Z_X)Var(Y + Z_Y) ≤ Corr(X, Y). As N increases and approaches infinity, the standard errors in the estimates of the mean responses decrease, i.e., Z_X and Z_Y decrease, and Corr(X + Z_X, Y + Z_Y) will increase and approach the true correlation. In our experiments, due to practical constraints, N ranged from 4 to 20 across cells. To empirically assess the size of the underestimate caused by the finite repeats, Fig. 5 plots the minimum number of repeats across stimuli vs. shape tuning consistency. Because we used a pseudorandom stimulus protocol (see materials and methods), a minimum number of repeats equal to N implies that data collection was terminated during the (N + 1)th run and the mean responses for the different stimuli were based on N or N + 1 repeats. Across our data set, there was a significant positive correlation between N and Fisher-transformed shape consistency (r = 0.58; P < 0.0001). Thus, as expected, shape responses were more consistent when we obtained more stimulus repeats, and shape consistency values based on four repeats (r_C4) were less than those based on all available repeats for any given cell (Fig. 5B). To compute shape consistency based on four repeats, for each cell we randomly chose four repeats for each color and shape, computed shape dispersion based on the chosen trials, and repeated this bootstrap procedure 100 times. The average across the 100 repetitions is reported as r_C4 in Fig. 5. The difference between r_C and r_C4 (Y-axis, Fig. 5C), which quantifies the change in shape consistency as number of repeats increases from four to N, provides a measure of the underestimate in shape consistency as a function of N (X-axis, Fig. 5C). As N increases, r_C − r_C4 also increases and plateaus somewhat at r_C − r_C4 ≈ 0.3 for N > 10. This can also be visualized as a plateau in mean shape consistency (red crosses in Fig. 5A) at ∼0.7 for N > 10. Thus r_C for N < 10 repeats appears to be severely underestimated. On the basis of this empirical evidence, we propose that for the inherent noise and dynamic range of V4 neurons, and for the present experimental and analytical conditions, N > 10 is essential to avoid biases in correlation metrics based on mean responses. Figure 5D shows shape dispersion vs. shape consistency (same as Fig. 4B) for only those cells studied with >10 repeats (n = 22 cells). All except three cells exhibited strong consistency (r_C > 0.5) in their shape responses; the three cells with values close to 0.3 were associated with low shape dispersion and exhibited weak tuning for shape. This matched what we found by eye: we could not find any examples of color-dependent shape tuning. Cells that exhibited strong shape selectivity showed consistent selectivity in both colors; other cells showed a mush of responses with no discernible shape tuning.

Fig. 5.

Effect of stimulus repeats on shape consistency and shape dispersion. A: scatter plot of minimum number of stimulus repetitions vs. shape consistency. For a cell, if minimum number of repeats (X-axis) is N, then mean shape responses would be based on N or N + 1 repeats, because data collection was terminated during the (N + 1)th repeat. Red crosses represent the mean shape consistency at the corresponding number of repeats. Overall, there was a trend of increasing shape consistency with increasing number of stimulus repeats, and shape consistency appears to plateau somewhat for N > 10. Fisher-transformed shape consistency and minimum number of repeats were positively correlated (r = 0.56, P < 0.0001). B: scatter plot of shape consistency values based on all stimulus repeats (r_C; X-axis) vs. 4 stimulus repeats (r_C4, Y-axis). We randomly chose 4 repeats without replacement and computed the mean responses and shape consistency based on those repeats. The whole procedure was repeated 100 times; the average consistency across these 100 repetitions is reported as r_C4. In all except 1 case, shape consistency based on 4 repeats was less than that based on more stimulus repeats. Cells for which data collection was terminated on the 5th run, i.e., minimum number of repeats = 4, are shown as open symbols. These values do not lie on the diagonal, because although the Y-axis was based on mean responses derived from 4 repeats, the X-axis measure was based on mean shape responses derived from 4 or 5 repeats. C: bar graph of the mean difference in shape consistency values based on all stimulus repeats (r_C) and 4 repeats (r_C4) as a function of minimum number of repeats. X-axis is the same as in A. Y-axis is the difference between X- and Y-axes in B with cells grouped on the basis of the number of repeats. The difference r_C − r_C4 increases with increasing stimulus repeats, implying that there was a systematic underestimate in shape consistency for small number of stimulus repeats. D: shape-response dispersion (X-axis) vs. shape consistency (Y-axis) for cells studied with >10 repeats (n = 22 cells). This plot is similar to Fig. 4B, except that it includes only those cells studied with >10 repeats. As before, there was a strong relationship between shape dispersion and shape consistency. E: shape-response dispersion (X-axis) vs. shape consistency based on 4 stimulus repeats. Y-axis is the same as in B. As with r_C4 computation described in B, shape dispersion for each cell was the average across 100 repetitions of a bootstrap procedure, each producing a shape dispersion value based on 4 randomly chosen trial repeats. This confirms that the relationship between shape dispersion and shape consistency is not due to the dependence of both measures on the number of stimulus repeats.

As with shape consistency, the number of stimulus repeats also influences shape dispersion for the same reason discussed above. Therefore, for all cells, we recalculated shape consistency and shape-response dispersion using N = 4 and the bootstrap procedure described above. The average shape consistency and shape dispersion values across 100 iterations for each cell are shown in Fig. 5E. These results are very similar to those reported in Fig. 4, confirming that the relationship between shape tuning and shape consistency was not trivially due to differences in N across cells.

We next investigated how the chosen stimulus colors and the color-selective properties of neurons influenced the observed shape consistency values. Specifically, it is possible that the spread of shape consistency values was simply a function of the colors that were chosen for any given cell: neurons studied with similar colors (closer in the CIE space, and thus evoking similar responses) may have been associated with higher consistency values than those studied with more disparate colors. Figure 6A, which plots the distance between the tested colors in CIE space vs. shape consistency for each cell, demonstrates that this was not the case. Across the 60 cells, color distances ranged from 0 (for 1 cell that was studied with achromatic stimuli at 2 different luminance contrasts, see materials and methods) to 0.57 (median 0.19, mean 0.21). Both low and high color distance values were associated with high shape consistency values, and the correlation between the Fisher-transformed shape consistency values and color distance was not significantly different from zero. This suggests that cells studied with colors closer together in the CIE space were not especially associated with higher consistency values, confirming that the high consistency observed is not an artifactual result of studying some cells with similar chromaticities. It is also possible that the spread of consistency values was a function of the color-selective properties of neurons: cells that do not receive chromatic inputs, which are therefore not color selective, may exhibit more consistency than cells that are color selective. We used two methods to address this question. For all 60 neurons, we conducted a two-way ANOVA with color and shape as factors, to investigate whether the two colors differentially modulated the responses of the neurons. Fifty-one of the 60 cells showed a main effect of color and/or an interaction between color and shape (44/60 cells exhibited a significant interaction term, P < 0.05); many of these color-selective neurons exhibited highly consistent responses (filled bars, Fig. 6B). In fact, cells that were not color selective tended to exhibit weaker consistency in shape responses. Second, for the 35 cells that underwent detailed color characterization, we assessed the strength of color selectivity by computing color-response dispersion. Color-response dispersion was computed on the basis of detailed color characterization conducted with a single preferred shape of the cell (see materials and methods) and thus was fully independent of the data used to compute shape consistency and shape-response dispersion. Analogous to shape-response dispersion, color-response dispersion was quantified as the ratio of the variance to the mean of the responses across all 25 chromaticities at the same luminance contrast as the colors in which the stimuli were presented (see materials and methods). In the few cases where the stimulus colors were at different luminance contrasts, we computed color-response dispersion at both luminance contrasts (see Fig. 6 legend). Cells that show weak or no color selectivity will respond similarly to all colors and have small color-response dispersion values (far left of X-axis, Fig. 6C), whereas strongly color-selective neurons will exhibit a large dispersion of color responses. Across our data set, color-response dispersion ranged from 0.22 to 3.8 (X-axis, Fig. 6C). However, unlike shape-response dispersion, there was no clear trend in the relationship between color-response dispersion and shape consistency. In other words, weak color selectivity (low color-response dispersion) did not imply high shape-response consistency. Altogether, Fig. 6, B and C, indicates that highly consistent shape responses across colors observed in our data set cannot be simply attributed to the lack of color selectivity, and thus chromatic inputs, in those cells. Even in the presence of chromatic inputs, it is possible that shape selectivity is informed only by luminance inputs to cells and that chromatic inputs simply provide a shape-independent modulation of the overall response. Under this hypothesis, cells studied with two colors at the same luminance contrast but different chromatic contrasts (52/60 cells in our database) will exhibit high consistency, but cells studied with chromaticities equiluminant with the background, i.e., 0% luminance contrast, will not exhibit any shape selectivity or consistency. Three cells in our present data set were studied with colors that were equiluminant with the background; all three exhibited significant shape selectivity (ANOVA, P < 0.01) and moderate levels of shape consistency (r_C = 0.25, 0.40, 0.69), arguing against the above hypothesis. These results are also consistent with our recent finding that many V4 neurons exhibit selectivity for shapes defined solely by chromatic contrasts (Bushnell et al. 2011b).

Fig. 6.

Relationship between shape consistency, stimulus color choices, and color selectivity. A: relationship between color distance between the chosen colors and shape consistency. Color distance (X-axis) is quantified by the Euclidean distance in CIE space between the 2 colors used to study each cell. Data from 1 cell, studied with shapes presented at 2 different achromatic luminance contrasts (and no color contrast), is shown at X = 0. No systematic relationship is evident between shape consistency and the chosen colors. B: histogram of shape consistency values for color-selective and unselective cells (analogous to Fig. 4D). Cells were classified on the basis of whether or not a 2-way ANOVA revealed a significant (P < 0.05) main effect of color and/or an interaction between color and shape responses. All except 9 cells (open bars) were significantly modulated by color. Cells that were not color selective tended to exhibit weaker consistency in shape responses across colors. C: scatter plot of color-response dispersion (X-axis) vs. shape consistency. Color-response dispersion was quantified by the ratio of the variance to mean across 25 colors presented at the same luminance contrast as the colors used to study shape responses. Each filled circle corresponds to data from a cell that was studied with 2 colors that differed only in their chromaticity values. Cells studied with colors that differed in both chromaticity and luminance are indicated by open circles. Four of these cells are associated with 2 dispersion values (2 open circles connected by a line) at each of the 2 luminance contrasts tested. Three neurons represented by a single open circle were studied with maximum luminance achromatic stimulus (no other colors were studied at this luminance contrast, and thus no color dispersion can be calculated) and another color at one of the four standard luminance contrasts (see materials and methods). No systematic relationship is evident between shape consistency and color-response dispersion. D: scatter plot of shape-response dispersion vs. color-response dispersion for the 35 cells that underwent detailed color characterization. X-axis plots shape-response dispersion in the preferred color (same as open symbols along X-axis in Fig. 4B). Y-axis is the same as X-axis in C. No systematic relationship is evident between shape- and color-response dispersion; the strength of color selectivity exhibited by a cell did not predict the strength of its shape selectivity.

Finally, we asked how color modulates shape responses, whether it provides an additive modulation, multiplicative modulation, or both. The response R of a neuron that shows highly consistent shape responses across colors could be written as

$$R=f\left(c\right)\times g\left(s\right)+h\left(c\right)$$

where c is the color of the stimulus and s is the shape. In the above equation, f represents the gain and h is the additive modulation of the shape response by the stimulus color. If different colors just provided an additive modulation, f(c) = 1 for all c. If, on the other hand, color only provided a gain modulation, h(c) = 0 for all c. If the responses to shapes in the two colors, c₁ and c₂, are denoted by R₁ and R₂, then

$$R_1=f\left(c_1\right)g\left(s\right)+h\left(c_1\right)$$ (1)

$$R_2=f\left(c_2\right)g\left(s\right)+h\left(c_2\right)$$ (2)

and, substituting for g(s) from Eq. 2,

$$R_1=\frac{f\left(c_1\right)}{f\left(c_2\right)}\times \left[R_2-h\left(c_2\right)\right]+h\left(c_1\right)$$ (3)

Thus the slope of the best-fit line between responses in the two colors, R₁ and R₂, provides a measure of f(c₁)/f(c₂): a slope of 1 implies f(c₁) = f(c₂) and, thus, no gain modulation. For each cell, we performed a linear regression between the shape responses in the two colors. Of the 60 cells, 47 had a significant linear regression fit (F-test, P < 0.05). The slopes for these 47 neurons ranged from 0.13 to 0.91 (c₂ was the preferred color in the equations above), and the 95% confidence limits of 44 cells did not include 1.0. This suggests that f(c₁)/f(c₂) is different from 1 for many cells and that color does provide a gain modulation of shape responses. Drawing conclusions about the strength of additive modulation from the intercept [= h(c₁) − s·h(c₂), where s denotes the slope] is less straightforward. If h(c) was zero for all c, then the intercept would be zero as well. The intercept would also be zero if the ratio of the additive modulations [h(c₁)/h(c₂)] was equal to the ratio of gain modulations [f(c₁)/f(c₂)]. A positive intercept would mean that the ratio of additive modulation is greater than the ratio of gains, i.e., h(c₁)/h(c₂) > f(c₁)/f(c₂), whereas a negative intercept would mean the reverse. For our data, the intercept was not significantly different from zero for 8/47 cells (P < 0.05) and was 2.67 spikes/s, on average. These results suggest that there is an additive modulation in addition to the gain modulation by color signals. Further experiments with more colors are needed to confirm these findings and more precisely characterize the modulation of V4 shape responses by color.

DISCUSSION

To assess consistency in shape responses across colors, we studied the responses of V4 neurons to a set of shapes presented in two different colors. Our results indicate that V4 neurons show different levels of consistency: whereas many neurons show strongly consistent shape preference across colors, others do not. Neurons that show poor consistency tend to have more eccentric receptive fields and show less dynamic range in their responses across shapes. These neurons are less shape selective and thus may be less likely to contribute to shape processing. Finally, our results also suggest that, given the noise level of V4 neurons and the experimental and analytical procedures used in this study, more than 10 repeats per stimulus are necessary to obtain a good estimate of the consistency in shape responses across colors.

Previous results.

Past studies in area V4 have demonstrated selectivity for both stimulus form (Desimone and Schein 1987; Gallant et al. 1993; Kobatake and Tanaka 1994; Pasupathy and Connor 1999, 2001) and color (Bushnell et al. 2011b; Conway et al. 2007; Kotake et al. 2009; Kusunoki et al. 2006; Schein and Desimone 1990; Zeki 1973). However, because characterizations of tuning have typically been performed along the dimension of interest (shape or color) while holding the other (color or shape) constant, past V4 results do not reveal whether selectivity for form and color interact at the single-neuron level, i.e., whether shape preferences are the same regardless of the color of the stimulus. Our findings of consistency in shape selectivity across colors in V4 are in keeping with previous results from IT cortex which suggest that selectivities for color and shape are largely independent of each other (Komatsu and Ideura 1993) and that shape preferences are consistent across contrast reversals (Kovacs et al. 2003). One previous study (McMahon and Olson 2009) demonstrated that influences of color and shape sum additively for most neurons in IT cortex and that nonlinear interactions between these stimulus attributes were rare. Although we also found some additive influence in V4, the primary effect across the majority of neurons was a multiplicative interaction between color and shape. One possible reason for this difference may be the number of stimuli: whereas we used ≥56 shape stimuli in each of 2 colors, they used only 2 stimuli in 2 colors. A neuron that appears linear across a narrow range (of 4 stimuli) may not appear so when a larger range of stimuli are considered. Further experiments in IT cortex with a larger stimulus set are required to determine whether this difference is due to the number of stimuli or if it is, in fact, a difference in how V4 and IT cortex combine color and shape information. Our discovery of multiplicative combination of color and shape information in V4 is consistent with a previous demonstration that tuning for shape and texture, also a surface property like color, in IT cortex can be well described by a multiplicative model (Koteles et al. 2008). Finally, Komatsu and Ideura (1993) also demonstrated that the occurrence of color and shape selectivity in a single IT neuron were independent of each other. In other words, whether or not a cell was shape selective had no bearing on whether it was color selective. Across the 35 cells that underwent detailed shape and color characterization, we found that the dispersion in shape responses was not significantly correlated with the dispersion in color responses (Fig. 6D; r = 0.14; P = 0.46). This suggests that the strengths of color and shape selectivity in V4 might also be independent of each other, in keeping with IT studies.

Implications for decoding.

Primates can recognize a wide variety of shapes regardless of their position, size, viewing angle, presence of occluders, etc. This is a computationally challenging problem, and past studies have hypothesized that such invariant recognition can be achieved by a population of neurons that exhibit shape tuning that is invariant of other stimulus attributes (Ito et al. 1995; Logothetis et al. 1995; Vogels and Orban 1996). Neurons in IT cortex have been shown to exhibit shape preferences that are consistent across position in the visual field even though response magnitude itself may depend on position (DiCarlo and Maunsell 2003; Ito et al. 1995; Logothetis et al. 1995; Schwartz et al. 1983; Tovee et al. 1994; Zoccolan 2007). Preserving the rank order of shape preferences is sufficient to successfully decode stimulus identity using linear decoding methods (Li et al. 2009). If rank order is not preserved, i.e., shape tuning is not consistent across position or other stimulus attributes, stimulus identity may still be decoded, but this would require nonlinear/computationally intensive algorithms.

Our results demonstrate that many neurons in area V4, especially those cells that show higher values of dispersion in shape responses, exhibit consistent shape preferences across colors, i.e., the rank order of shape preference is preserved across colors. Thus these neurons exhibit a crucial property for supporting invariant recognition. For example, the shape and color of a stimulus can be successfully decoded by using a simple basis function decoding procedure from the activity of a group of V4 neurons that show consistent shape tuning across colors. Basis function decoding is simple (the identity of a stimulus is given by the sum of the tuning functions weighted by the response of the corresponding neuron to the stimulus in question) and could be easily implemented across neural populations (Zhang et al. 1998). The size of the population required for successful recognition depends on the width of tuning functions, which requires additional experiments to determine. In addition to decoding simplicity, the consistency of shape selectivity across colors has an enormous practical advantage because it allows experimentalists to quantify detailed shape × color tuning functions of V4 neurons without having to quantify shape tuning functions in every color. For example, one could measure detailed shape tuning in one color and then quantify the multiplicative and additive modulations provided by the different colors with a much reduced shape stimulus set (n could be as small as 2). Thus one could reconstruct the entire shape × color tuning function for n shapes and m colors with as few as n + 2m measurements; such an approach does not suffer from the combinatorial explosion (n × m stimuli) caused by a detailed shape characterizations in every color.

Finally, cortical neurons tend to be noisy, and characterizations of tuning properties are based on mean responses derived from multiple stimulus repeats. Given the practical constraints of holding well-isolated single units, there is an inherent trade-off between number of different stimuli and number of repetitions of each stimulus. In studies that attempt to characterize tuning over a large space of shape or color, the number of stimulus repeats is often held low in the interest of presenting a larger variety of stimuli. Our results indicate that this strategy could have a deleterious effect: as expected, large standard errors in mean responses caused by low repeats could produce biased estimates of the properties being characterized. In our case, a low number of repeats implied a systematic negative bias in the shape consistency that was most pronounced when the number of repeats was <10. In conclusion, our results caution physiologists to consider the biases induced by low stimulus repeats and the trade-off between number of stimuli and the number of repeats of a given stimulus when designing their studies.

GRANTS

This work was supported by National Eye Institute Grant R01 EY018839, the Whitehall Foundation, University of Washington Vision Core Grant P30 EY01730, and the National Center for Research Resources and the Office of Research Infrastructure Programs through Grant RR00166.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

Author contributions: B.N.B. and A.P. performed experiments; A.P. conception and design of research; A.P. analyzed data; A.P. interpreted results of experiments; A.P. prepared figures; A.P. drafted manuscript; A.P. edited and revised manuscript; A.P. approved final version of manuscript.