Nonlinear analysis of macaque V1 color tuning reveals cardinal directions for cortical color processing

Abstract

Understanding color vision requires knowing how signals from the three classes of cone photoreceptor are combined in the cortex. We recorded from single V1 neurons in awake monkeys while an automated, closed-loop system identified stimuli that differed in cone contrast but evoked the same response. We found that isoresponse surfaces for about half the neurons were planar, consistent with linear processing. The remaining isoresponse surfaces were nonplanar. Some were cup-shaped, indicating sensitivity to a narrow region of color space. Others were ellipsoidal, indicating sensitivity to all color directions. The major and minor axes of these nonlinear surfaces were often aligned to a set of three color directions that were previously identified in perceptual experiments. These results demonstrate that many V1 neurons combine cone signals nonlinearly and provide a new framework within which to decipher color processing in V1.

Introduction

Color vision begins with the transduction of light into neural signals by the three classes of cone photoreceptors and ends with the processing of these signals in the cerebral cortex. Historically, quantitative studies of color processing in the visual system have estimated the strength of cone inputs to downstream neurons by assuming that cone inputs are combined linearly. This approximation has been valuable for understanding color processing in subcortical structures, but less so in the cortex.

When stimulated with coarse spatial patterns and characterized with linear models, neurons in the retina and lateral geniculate nucleus (LGN) segregate naturally into discrete clusters on the basis of their cone inputs ^1-5. These clusters explain a body of psychophysical observations and their identification was a critical step in our current understanding of the elemental color computations performed by these structures ^6-9. The same methods applied to neurons in the primary visual cortex (V1) do not reveal discrete clusters, but instead suggest heterogenous combinations of cone inputs that are not related to color perception in any obvious way ^10-13. However, nonlinearities in the color tuning of V1 neurons are well documented ^{10, 12, 14-17}, suggesting the alternative possibility that V1 neurons combine cone signals in systematic, nonlinear ways with an organization that appears disordered only because of the inadequacy of linear methods.

To understand the organization of cone signal processing in visual cortex, we introduce a new technique for analyzing nonlinear signal combination and apply it to V1 neurons in awake, fixating monkeys. Roughly half of the recorded neurons combined cone signals nonlinearly. Analysis of these nonlinear combinations revealed an unexpected relationship to color directions previously identified as perceptually and physiologically important ^{2, 3, 7, 18-20}. These results are consistent with a simple hierarchical model whereby signals from linear neurons tuned to a small set of color directions combine via simple nonlinear operations to create a diversity of color tuning in V1.

Results

We recorded from 118 V1 neurons in two monkeys (61 from Monkey K and 57 from Monkey S). For each neuron, we used an automated, closed-loop system to find an isoresponse surface: a collection of points in cone contrast space that evoked the same firing rate. Stimuli were drifting Gabor patterns, and firing rates were measured from an estimated response latency until the end of each stimulus presentation (see Methods).

Fig. 1 shows examples of isoresponse contours (in 2-D) for three hypothetical V1 neurons. The neuron in Fig. 1a combines cone signals linearly, so its isoresponse contours are lines and would be planes in 3-D color space. The neuron in Fig. 1b combines cone signals that have been put through a compressive nonlinearity, so its isoresponse contours are concave. The neuron in Fig. 1c combines cone signals that have been put through an expansive nonlinearity, so its isoresponse contours are convex.

Figure 1.

Predicted color tuning under three models of cone signal combination. Upper panels show models as box-and-arrow diagrams. Lower panels show neural responses and isoresponse contours as a function of inputs from two cone types. A: Isoresponse contours are lines for neurons that combine cone signals linearly. Output nonlinearities affect the spacing between the lines but do not bend them. B: Compressive nonlinearities prior to linear cone signal combination produce concave isoresponse contours. C: Expansive nonlinearities prior to linear cone signal combination produce convex isoresponse contours.

Distinguishing these hypothetical tuning functions using traditional methods can be challenging. A conventional experimental approach is to measure responses to a small set of predetermined stimuli. This is analogous to holding an opaque mask with a few holes (each representing a stimulus) over the lower panels in Fig. 1. Depending on the locations and number of holes, the three tuning functions in Fig. 1 can appear identical. An alternative approach that we used in this study is to measure the shapes of isoresponse surfaces.

Fig. 2 shows data from three representative V1 neurons that resemble the hypothetical examples in Fig. 1. Each data point in Fig. 2 represents a stimulus that evoked the target firing rate, which for example neuron 1 was 5 spikes per s. The isoresponse surface of neuron 1 is well described by a pair of planes, as shown in Fig. 2a,b. A quadratic fit to these data (Fig. 2c,d) was not a significant improvement over the planar fit (F-test, p > 0.01). The color tuning of this neuron is therefore reasonably well described by a linear combination of cone signals.

Figure 2.

Data from three example neurons (two projections for each). Dots indicate staircases terminations. Gray lines indicate staircases that exceeded the monitor gamut. For neuron 1, planar and quadratic fits are shown in separate panels (A&B and C&D, respectively) to facilitate visualization. For neurons 2 and 3, best fitting planes (black outlines) and quadratic surfaces (green) are superimposed in individual panels. Axes have been scaled to show the spread in the data points. Here and throughout, units are in cone contrast vector lengths: $c=\sqrt{{\left(\frac{\Delta L}{L}\right)}^2+{\left(\frac{\Delta M}{M}\right)}^2+{\left(\frac{\Delta S}{S}\right)}^2}$. Data points and fits are plotted symmetrically about the origin to reflect the fact that each Gabor stimulus modulated symmetrically through the white point.

The orientation of isoresponse planes can be specified by the unique direction orthogonal to them ³. This is the color direction of maximal neural sensitivity; less contrast is needed to reach the target firing rate in this direction than any other. Directions of minimal sensitivity are parallel to the planes; no amount of contrast in these directions is sufficient to reach the target firing rate. The orientation of the planes in Fig. 2a,b shows sensitivity to in-phase modulations of the L- and M-cones, so this neuron can be classified as an LM non-opponent cell.

Isoresponse surfaces from the second neuron (Fig. 2e,f) are poorly described as planes. Data points in color directions near L−M+S lie closer to the origin than the fitted plane whereas data points in other color directions lie beyond the fitted plane. This pattern indicates that neuron 2 does not combine cone signals linearly (F-test, p < 0.01). Instead, it is highly sensitive to modulations near L−M+S and relatively insensitive to modulations in other color directions.

Neuron 3 responded in every color direction tested. The data points lie close to the surface of an ellipsoid (Fig. 2g,h), and as a result the best fitting plane provides little relevant information about color tuning. A preferred color direction is difficult to define for this neuron because its direction of maximal sensitivity (the shortest axis of the ellipsoid) depends on the color space in which the data are represented ²¹. A linear transformation of cone contrast space can convert an isoresponse ellipsoid into a sphere, which has no long nor short axes. This concern does not extend to neuron 1; a plane remains a plane after linear transformations.

Analysis of cone weights

The color tuning of V1 neurons has traditionally been quantified with cone weights. Implicit in this characterization is the idea that cone signals are combined linearly. As shown above, some V1 neurons combine cone signals nonlinearly, so summarizing the color tuning of these neurons with cone weights may be misleading. Nevertheless, to obtain a first-order description of color tuning that can be compared with previous studies, we calculated cone weights for every neuron in our data set. Cone weights were calculated from the orientation of planar fits to the data from each neuron. Later, we extend this analysis to non-planar surfaces.

Fig. 3 shows the distribution of normalized cone weights across the population of V1 neurons that we studied. This analysis is best-suited to neurons with planar isoresponse surfaces, a criterion that was poorly met by many of the neurons we sampled. The planar model was rejected for 64/118 neurons (F-test, p ≤ 0.01, blue and yellow symbols in Fig. 3). Leave-one out cross-validation (see Methods) confirmed that quadratic fits provided better predictions than planes for 94/118 neurons, and this difference was statistically significant for 39 (Wilcoxon signed-rank test on prediction errors, p ≤ 0.01). We conclude that many V1 neurons combine cone signals nonlinearly. A description of color tuning in terms of cone weights therefore has limited utility in V1. As a particularly salient example, consider cone weights for the 33 neurons whose isoresponse surfaces were ellipsoidal (yellow symbols). These cone weights are broadly distributed because planar fits to their data have essentially random orientations.

Figure 3.

A: Normalized cone weights derived from the orientations of planar fits to staircase terminations. The coefficients of a line orthogonal to the fitted planes are the unnormalized cone weights. Cone weights were normalized by dividing each by the sum of their absolute values ¹⁰. Filled and unfilled symbols represent neurons with positive and negative S-cone weights, respectively. Black symbols represent neurons with planar (p > 0.01, F-test) isoresponse surfaces. Yellow and blue points represent neurons with ellipsoidal and hyperbolic isoresponse surfaces, respectively (p ≤ 0.01). Example neurons from Figure 2 are indicated by red numerals and arrows.

Predicting conventional color tuning measurements

During an initial characterization procedure (see Methods), each neuron was probed with 9 colored gratings. Fig. 4a–c show grating responses for the example neurons in Fig. 2. Each grating is represented as a disk whose position represents its L-, M-, and S-cone contrasts and whose size represents the response it evoked. Small disks tend to be close to the origin and large disks tend to be far from the origin because neural responses generally increase with stimulus contrast. Planar (blue) and quadratic isoresponse surfaces (green) obtained using the closed-loop procedure are superimposed on the data obtained using drifting gratings. As their name implies, isoresponse surfaces are expected to pass through equally-sized disks.

Figure 4.

Isoresponse surface fits, grating responses, and predictions of grating responses for the three example neurons from Figure 2. A, B and C: Disks indicate the coordinates of grating stimuli in color space. Each disk is plotted twice: once on either side of the origin to reflect the fact that each grating modulates symmetrically through this point. Disk size represents the mean response to each grating (6 repeats). Blue and green surfaces are planar and quadratic fits to staircase terminations, respectively. D, E, and F: Responses to grating stimuli (abscissa) and predictions obtained from Equation 1, using a planar description of the isoresponse surface (ordinate). G, H, and I: as above, but response predictions are based on quadratic isoresponse surfaces. r values are Pearson’s correlation coefficients.

We predicted neuronal responses to these gratings from each neuron’s best fitting planar and quadratic isoresponse surfaces using the formula:

$$\mathit{fr}=c\frac{r}{\widehat{r}},$$ (1)

where fr is the firing rate, c is the target firing rate used in the isoresponse measurement, r is the distance from the origin to the grating in cone contrast space, and $\widehat{r}$ is the distance from the origin to the isoresponse surface in the same color direction. This analysis assumes that stimuli on the isoresponse surface evoke the target firing rate, and that stimuli off the surface evoke a response that is proportional to the distance from the origin in units of distance to the isoresponse surface. In other words, it assumes that the neuron has a linear contrast-response function whose slope varies with color direction.

Fig. 4a–c illustrate the method. Gratings that elicited weak responses from neuron 1 (small disks in Fig. 4a) lie between the origin and the isoresponse surface whereas gratings that elicit robust responses are located distal to the surface. Distance from the origin, with respect to distance from the isoresponse plane, is thus a good predictor of the responses to the grating stimuli.

Fig. 4d shows the relationship between the responses of neuron 1 to the colored gratings and those predicted from a planar description of its isoresponse surface. The close agreement between the data and predictions (Pearson’s r = 0.88) shows that color tuning estimated from the planar isoresponse surface is consistent with color tuning measured conventionally. Predictions based on the quadratic isoresponse surface were not superior (Fig. 4g; r = 0.87).Thus, for some neurons, a planar description of the isoresponse surface is sufficient to predict responses to stimuli that lie off of the surface.

Example neuron 2 had unusually tight color tuning. It responded at 20 spikes per s to the L−M+S grating and below 11 spikes per s to the other 8. The isoresponse surface for this neuron formed a cup that held the L−M+S grating and excluded the others (Fig. 4b). The correlation between actual and predicted responses was 0.71 when predictions were based on the planar isoresponse surface and 0.94 when predictions were based on the quadratic surface (compare Fig. 4e,h). To compare the quality of linear and quadratic predictions statistically, we randomly swapped “linear” and “quadratic” labels on the response predictions and recalculated correlation coefficients (2⁹ = 512 permuted data sets). The correlation coefficient was significantly larger when based on the quadratic surface than when based on the planar surface (p < 0.05).

Example neuron 3 had a convex isoresponse surface and responded strongly to most of the colored gratings (Fig. 4c). Response predictions based on a planar isoresponse surface were poorly correlated with the measured responses evoked by the gratings (Fig. 4f; r = 0.07). In contrast, the quadratic surface yielded significantly better predictions (Fig. 4i; r = 0.71; p < 0.05 by permutation test on the difference in correlation coefficients).

Across the population, correlation coefficients between actual and predicted responses were skewed toward positive values for predictions based on either type isoresponse surface (i.e., planar or quadratic; Fig. 5). Nevertheless, the quadratic model was superior. The median correlation for predictions based on quadratic surfaces (r = 0.79) was significantly greater than the median correlation based on planar surfaces (r = 0.61; Wilcoxon test: p < 0.0001). The superior predictive power of the quadratic model was confirmed by two additional analyses in which prediction accuracy was quantified with either Spearman’s rank correlation coefficient, which is invariant to monotonic nonlinearties (e.g. contrast-response functions), or with mean-squared error, which is sensitive to systematic biases in offset and scale of the predictions (median r_Spearman for quadratic fit: 0.73; median r_Spearman for planar fit: 0.65; Wilcoxon test: p < 0.001; median MSE for planar fit: 156, median MSE quadratic fit: 120; Wilcoxon test: p < 0.05).

Figure 5.

Scatterplot of correlation coefficients between actual and predicted responses to 9 colored gratings. Predictions were based on a planar description of the isoresponse surface (abscissa) or a quadratic description (ordinate). Yellow symbols represent neurons with ellipsoidal isoresponse surfaces. The histogram shows differences between correlation coefficients, and the triangle indicates the mean.

Analysis of quadratic isoresponse surface shape

Quadratic isoresponse surfaces can be divided into three mutually exclusive categories. Ellipsoids are 3-D generalizations of ellipses (e.g. Fig. 6a). Hyperboloids of 1 sheet look like hourglasses: they are narrow in the middle and flared at the top and bottom (e.g., Fig. 6b). Hyperboloids of 2 sheets look like bowls facing away from the origin (e.g., Fig. 6c). We determined the category of each fitted surface from the signs of the eigenvalues of the matrix of the fitted coefficients see Methods and ²². Regardless of category, quadratic surfaces more accurately predicted responses to the colored gratings than planar surfaces did (Wilcoxon tests, ellipsoids: p < 0.01, all hyperboloids together: p < 0.0005, hyperboloids of 1 sheet: p = 0.06, hyperboloids of 2 sheets: p < 0.0001). This demonstrates that the superiority of the quadratic predictions shown in Fig. 5 was not dominated by any single shape acting alone.

Figure 6.

Distribution of principal axes of quadratic isoresponse surfaces. The geometry of principal axes can be seen in the 3-D rendered surfaces on the left side of each row. Each point in the normalized cone weight space represents the direction of one principal axis. Points are color coded according to the axis type they represent. Arrows in A point to the three principal axes from example neuron 3 from Figure 2. The arrow in C points to a single axis for example neuron 2. For this neuron, the other two axes did not differ sufficiently in length to be plotted.

Aspect ratio and orientation of quadratic surfaces

An isoresponse plane has a single direction orthogonal to it, so the color tuning of a linear neuron can be described with a single preferred color direction or equivalently a single point in Fig. 3. A quadratic isoresponse surface does not have a single direction orthogonal to it but can be characterized similarly by virtue of its three principal axes. A quadratic surface’s principal axes describe its orientation, and its distention along these axes describe its aspect ratio.

Fig. 6a shows the orientation of the long, medium, and short axes of isoresponse ellipsoids in cone contrast space. The orientation of each axis can be described with two numbers, for example, normalized L- and M-cone weights, which facilitates comparison with Fig. 3. Each ellipsoid comprises three orthogonal axes. Pairs of axes that differ in length by less than a factor of 5 have not been plotted because their directions are numerically unstable (see Methods). Thus each neuron is represented by a maximum of three points in Fig. 6.

Each ellipsoid had a long axis that was nearly parallel to the S-cone axis. This demonstrates that neurons with ellipsoidal isoresponse surfaces are less sensitive to S-cone modulations than to contrast-matched modulations of the L-cones, the M-cones, or any linear combination thereof. The medium and short axes correspond to color directions of moderate and high sensitivity, respectively. These axes are mathematically constrained to be orthogonal to the long axis, thus the medium and short axes lie close to the LM plane, and the orange and purple symbols in Fig. 6a lie near the oblique edges of the bounding triangle. The fact that the symbols tend to cluster near the midpoints of these edges, and not in the corners, is not a trivial consequence our procedure (Supplementary Fig. 1). Instead, this result indicates that medium and short axes tended to be oriented in the L+M and L−M color directions, not the L- and M-cone isolating directions.

A hyperboloid of 1 sheet, unlike an ellipsoid, has one axis that never intersects the surface. This axis was usually oriented near the S-cone isolating direction or the L−M direction; it was rarely oriented in the L+M direction (Fig 6b). The second axis corresponds to the widest part of the hyperboloid. This axis was not oriented consistently across our dataset, but like the first axis, had a tendency to be near the S- or L−M directions. The third is the narrowest axis, which tended to be oriented in the L−M or L+M directions and never in the S-cone direction. These data are consistent with the relative insensitivity of V1 neurons to S-cone modulation and further demonstrates a bias for S-, L−M and L+M color directions.

A hyperboloid of 2 sheets has a single axis that intersects the surface. This axis was in the L+M direction for most neurons but was closer to the L−M direction for others (Fig. 6c). The remaining two axes do not intersect the surface and instead indicate directions of steep and shallow surface curvature. These axes tended to be in the L−M and S-cone directions, respectively.

Across the neurons we tested, isoresponse surfaces tended to be aligned to the L+M, L−M, and S-cone axes. Not every neuron conformed to this pattern, but the consistency of the trend shows that these axes provide a convenient basis for describing quadratic isoresponse surfaces in cone contrast space.

Isoresponse surface shape and modulation ratio

Complex cells can be thought of as squaring and adding the outputs of simple cells to achieve near-invariance to spatial phase and luminance contrast polarity ^23-25. Similarly, neurons with ellipsoidal isoresponse surfaces can be thought of as squaring and adding chromatic signals to achieve near-invariance to color direction (the equation for an ellipsoid is ax²+by²+cz² = 1). These observations motivated us to ask whether neurons with ellipsoidal isoresponse surfaces might be complex cells as defined by invariance to spatial phase. A positive result would be consistent with the idea that some complex cells combine cone inputs from a variety of sources with different spectral sensitivities ²⁶, and would be inconsistent with models that describe the color tuning of complex cells with a single, rectified linear mechanism e.g. ¹⁰. To test this hypothesis, we grouped neurons by isoresponse surface shape and computed their modulation ratios (i.e. the amplitude of the modulated response to a drifting grating divided by the sustained response). Modulation ratios were calculated from responses to gratings of a preferred orientation, spatial frequency, and size (and various colors) that evoked a response ≥ 10 spikes per s. Small modulation ratios indicate insensitivity to spatial phase and thus a “complex” classification ²⁷.

Modulation ratios were significantly different across isoresponse surface shapes (1-way ANOVA, p < 0.05; Fig. 7). Neurons with ellipsoidal isoresponse surfaces had smaller modulation ratios (geometric mean = 0.46) than neurons with planar isoresponse surfaces (geometric mean = 0.71), or hyperbolic isoresponse surfaces whether considered as a single pool (geometric mean = 0.67) or broken down into 1-sheet (geometric mean = 0.60) and 2-sheet (geometric mean = 0.92) varieties. The only post-hoc comparison that reached statistical significance was between ellipsoids and planes (Tukey-Kramer test, p < 0.05). We conclude that many complex cells, but relatively few simple cells, are sensitive to modulations in all color directions.

Figure 7.

F1/F0 modulation ratios separated by isoresponse surface shape. Neurons with significantly non-planar isoresponse surfaces (F-test, p ≤ 0.01) are grouped into ellipsoidal, 1-sheet hyperbolic, and 2-sheet hyperbolic categories. The remaining cells are assigned to the plane category. Thirteen neurons that responded at < 10 spikes per s during the modulation ratio measurement were omitted from this analysis.

Discussion

To investigate how individual V1 neurons combine cone signals, we used an automated system to identify a set of stimuli in 3-D color space that evoked the same spike rate. For roughly half of the neurons studied, these stimuli lay on planes, a result consistent with a linear combination of cone contrast signals. For the other half, quadratic surfaces were better fits. Quadratic surfaces were often oriented along the L+M, L−M, and S axes of cone contrast space, suggesting a natural coordinate frame for measuring and describing the color tuning of V1 neurons. Below, we discuss the implications of our results for the measurements required to characterize color tuning in V1, and we advance a simple model that describes how curved isoresponse surfaces can be constructed through simple nonlinear operations on signals from linear neurons.

Sufficient statistics for color tuning

How many numbers are required to characterize the color tuning of a V1 neuron? A linear neuron has a preferred color direction that can be described with two numbers: for example, a pair of angles in 3-D color space. This parsimony allows color tuning to be represented as a point in a 2-D space like the one in Fig. 3. Our results show that more than two numbers are required to describe the color tuning of many V1 neurons without losing information.

Quadratic surfaces, which require six parameters, described the isoresponse surfaces of V1 neurons better than planes did. Nevertheless, quadratic surfaces still provide an incomplete description of color tuning: they describe only a single 2-D level surface of a 3-D function. To predict firing rates off of this surface, we assumed a linear contrast-response function. This model is a special case of a broader class that is separable in color direction and contrast.

To test this broader class of models, we measured isoresponse surfaces for 31 neurons at two different target firing rates. If color-direction and contrast make separable contributions to V1 responses, we would expect isoresponse surfaces at different firing rates to be scaled versions of each other. This was not the case; isoresponse surfaces changed shape, not just scale, with target firing rate (Supplementary Fig. 2 see also ¹⁴).

Color tuning may be separable with respect to a different set of axes. We found that the principal axes of quadratic isoresponse surfaces tended to align with the L+M, L−M, and S-cone axes. In the coordinate system of the principal axes, only 3 parameters are needed to describe a quadratic surface because the interaction coefficients in Equation 4 are 0. Exploiting this fact may facilitate efficient measurements of color tuning functions with a small set of strategically placed stimuli. An important future direction is to find a class of model that describes the tuning of V1 neurons in a full 3-D color space and to design a stimulus selection procedure to estimate the parameters of the model efficiently.

Effects of spatiotemporal parameters

A complete description of how a V1 neuron processes cone signals requires a description of how this processing depends on the spatiotemporal parameters of the stimulus. Obtaining a complete spatiotemporal-chromatic description of a neuron is difficult because the space of all possible stimuli is too large to probe finely enough. If the chromatic tuning of a V1 neuron were independent of its spatiotemporal tuning, we could measure each separately and arrive at a complete characterization. This is not the case ^{28, 29}. Consequently, the surfaces we measured depend on the spatiotemporal stimulus parameters that we used. In this study, we probed only a single 3-D slice from the space of all possible stimuli, but in principle our technique generalizes to higher dimensional spaces. Parametric manipulations of spatial, temporal, and chromatic aspects of a stimulus to obtain a more complete description of cone signal processing in V1 remains an important goal.

Hierarchical model

Even if V1 neurons naturally segregate into a finite number of types based on color tuning, this clustering would be difficult to uncover with linear analyses. We found that isoresponse surfaces in cone contrast space could be described with a small set of quadratic shapes (ellipsoids and hyperboloids) whose principal axes were oriented similarly across neurons. This suggests that color processing in V1 may be more orderly than previously recognized.

By analogy with Hubel and Wiesel’s ³⁰ classic model of simple and complex cells, signals from linear neurons (with planar isoresponse surfaces) can be combined nonlinearly to create higher-order neurons with curved isoresponse surfaces. V1 neurons with concave isoresponse surfaces, which are likely the Type 3 neurons of Hanazawa et al. ¹⁶, may represent a biologically realistic logical AND gate. As shown in Fig. 1b, neurons that respond exclusively to the conjunction of “A-and-B” can be constructed by taking the output of linear neurons tuned for A and B (or linear combinations thereof) and transforming their outputs by a compressive nonlinearity prior to signal summation. Biophysically, concave isoresponse surfaces can also be created by tuning excitation to stimulus direction A and shunting inhibition to stimulus direction B^{31, 32}. Such suppressive influences could manifest as 1- or 2-sheet hyperboloids, depending on whether the suppression was tightly (1-sheet) or broadly (2-sheet) tuned. The fact that hyperboloids of 1- and 2-sheets can be transformed into one another by flipping the signs on the coefficients (a through f in Equation 4), is consistent with the idea that these surfaces might reflect a common functional form, differing only in a criterion.

Neurons with convex isoresponse surfaces have broad color tuning, and are likely the Type 4 neurons of Hanazawa et al. ¹⁶ and the universal cells of Yoshioka et al. ³³. These neurons can be thought of as computing a logical OR: they respond whenever any of several color channels is active. This tuning can be constructed by transforming cone inputs (or linear combinations thereof) through an expansive nonlinearity prior to summation, as shown in Fig. 1c. If this nonlinearity is quadratic, isoresponse contours will be ellipsoidal, consistent with the energy model of complex cells ²⁴ and with many surfaces observed in this study.

The idea that a neuron can achieve invariance by pooling responses from heterogeneously tuned subunits, was advanced by Hubel and Weisel ³⁰ and refined by Movshon, Thompson, and Tolhurst ³⁴ to explain the position/phase invariance of complex cells. A relationship between ellipsoidal isoresponse surfaces and spatial phase invariance is not trivial; the majority of neurons that had ellipsoidal isoresponse surfaces were complex cells. One possibility is that a common mechanism produces invariance to both spatial phase and color direction, and that some of the apparent complexity of color tuning in V1 may be due to nonlinearities that have been well-studied in the spatial domain. A similar speculation was made by Lee et al. on the spatial and chromatic nonlinearities of magnocellular LGN neurons ³⁵.

Some of the nonlinearities we observed in V1 were presumably inherited from the LGN. Parvocellular and S-cone-dominated LGN neurons exhibit relatively mild nonlinearities that have been characterized as gain controls and might cause isoresponse surfaces to bend away from the origin at high contrast ^36-38. Magnocellular neurons respond at the fundamental frequency of luminance modulations and at twice the frequency of out-of-phase L- and M-cone modulations ^{35, 39, 40}. Isoresponse surfaces for magnocellular neurons are thus expected to be cylinders parallel to the S-cone axis. One possibility is that magnocellular input contributed to the combined L+M and L−M sensitivity that we observed in many V1 neurons.

Role in vision

V1 neurons with concave or convex isoresponse surfaces presumably play different roles in color vision since the former population is tightly tuned for color direction and the latter largely discards this information. Neurons with concave isoresponse surfaces carry color information that can be easily decoded by downstream circuitry: a relatively small subset will be activated by any given light. Sparse activity amid the silence of other tightly tuned neurons may provide a reliable code for color and allows for efficient discrimination of lights with natural (heavy-tailed) cone contrast distributions ⁴¹. In contrast, neurons with convex isoresponse surfaces respond to modulations in all color directions. A sufficiently complex downstream decoder might be able to determine hue from a population of these neurons, but we speculate they may be more important for computations requiring sensitivity to contrast in many color directions (e.g, detecting the presence of a stimulus, or estimating orientation or direction of motion). Importantly, however, both groups of neurons are responsive to isoluminant modulations and by that definition would be considered “color cells”. To determine which neurons are involved in which visual computations, a fruitful approach is to probe relationships between neural responses and behavior on color psychophysical tasks.

Methods

Two monkeys (M. mulatta) participated in the experiments. All procedures conformed to the guidelines provided by the NIH and the University of Washington Animal Care and Use Committee. Each monkey was surgically implanted with a titanium headpost, a monocular scleral search coil, and a recording chamber over area V1 (Crist Instruments).

During experiments, monkeys sat in a primate chair 1 m from a cathode ray tube monitor (Sony Trinitron) in an otherwise dark room. Neural signals were recorded with extracellular tungsten microelectrodes of 1-2 MΩ (Frederick Haer) and digitized at 40 kHz. Spikes were isolated online on the basis of waveform timing and amplitude criteria and saved to disk for offline analysis.

Three personal computers were used for data collection: First, a Dell Dimension 4800 monitored eye movements and controlled event timing using the REX software package (NIH). Second, an Apple Mac Pro displayed visual stimuli via custom software based on the PsychophysicsToolbox ⁴² for Matlab (MathWorks). Third, a Dell Precision T3400 acquired data and permitted online spike sorting (Plexon, Inc.). A Matlab process on this computer had real-time access to spike and event times and sent the results of online analyses to the REX computer via a UDP socket.

Stimuli and monitor calibration

Stimuli were Gabor patches (sigma = 0.4°) drifting at 3 Hz in a direction and with a spatial frequency tailored to each neuron (see Initial neuronal characterization). Contrast increased linearly over the first half-cycle, remained constant for one cycle, and decreased linearly over the second half-cycle (full duration = 667 ms). The space/time average chromaticity and luminance of each Gabor stimulus was identical to the background (x = 0.3, y = 0.3, Y = 90 cd/m²). Emission spectra and voltage-intensity relationships of each monitor phosphor were measured with a PR-650 SpectraColorimeter (PhotoResearch). The depth of each color channel was increased from 8 to 14 bits using a Bits++ video signal processor (Cambridge Research) at the expense of spatial resolution; each pixel was twice as wide as it was tall.

Behavior

Monkeys were rewarded for maintaining fixation on a 0.2 × 0.2° black square in the center of the monitor. Trials were aborted if the eye position left a 1 × 1° electronic window centered on the fixation point. During the initial characterization procedure the interstimulus interval was 1 s. During the isoresponse surface measurement, the first stimulus appeared 1.5 s after the monkey acquired fixation. Subsequent stimuli appeared with an interstimulus interval of 640 ms until the monkey broke fixation (average of 4 stimuli per trial).

Initial neuronal characterization

Each neuron was probed with circularly apertured sinusoidal gratings that drifted at 3 Hz for 1 s. The orientation, spatial frequency, and diameter of the grating were adjusted manually by the experimenter and then by the computer, which optimized them automatically in the order given above. If the preferred spatial frequency differed from the initial guess by ≥ 1 octave, orientation tuning was remeasured at the new spatial frequency, and the preferred spatial frequency was remeasured at the new preferred orientation. By default, the initial characterization procedure was performed using achromatic gratings at 40% contrast. 35/118 neurons that responded poorly to achromatic gratings were characterized with chromatic gratings (L−M, S, L−M−S, or L−M+S). After a set of “preferred” spatial parameters were found, nine colored gratings in L-, M-, S-cone contrast space were presented in a pseudorandomly interleaved order (Table 1).

Table 1.

Grating cone contrasts (%)

L-cone	M-cone	S-cone
9 (9)	9 (9)	0 (0)
7 (9)	−9 (−9)	0 (0)
1 (0)	3 (0)	64 (64)
12 (13)	−1 (0)	0 (0)
1 (0)	13 (13)	0 (0)
6 (6)	−5 (−6)	−46 (−45)
5 (6)	−4 (−6)	46 (45)
7 (6)	8 (6)	45 (45)
6 (6)	6 (6)	−45 (−45)

Grating cone contrasts calculated using the Stockman, MacLeod, Johnson 10° cone fundamentals (large font). Stimuli were constructed on the basis of the 2° fundamentals (cone contrasts in parentheses), but are represented in the 10° cone contrast space for analysis in this report.

Isoresponse surface measurement

Following the initial characterization procedure, we used a closed-loop system to find a set of chromatically distinct stimuli that elicited similar responses ⁴³. This approach is a logical extension of the “action spectra” measurements which were a cornerstone of early studies of color processing in V1 ^44-47. To measure an action spectrum, an experimenter increased the intensity of a narrow-band light until a criterion response was obtained. Our measurements differ from the classic ones because we probed a 3-D color space under automated computer control.

At the beginning of each isoresponse surface measurement, the experimenter set a spike counting window and target firing rate manually. The spike counting window began at the response latency (estimated from a continually updated peri-stimulus time histogram) and ended when the stimulus disappeared. The target firing rate was chosen on the basis of baseline and stimulus-evoked firing rate histograms. Target firing rates exceeded the baseline firing rate but fell below the maximum evoked response (Supplementary Fig. 3).

Isoresponse surface measurements proceeded in a series of iterations, each of which consisted of two phases. In the first phase, contrast in several randomly interleaved color directions was titrated to evoke the target firing rate from the cell. In the second phase, a new set of color directions was selected and the next iteration began.

Phase 1: Contrast titration

The goal of this procedure was to find a contrast in a specified color direction that evoked the target firing rate as nearly as possible. After each stimulus presentation, the evoked response was measured. If it exceeded the target firing rate, the contrast of the stimulus was reduced on the next presentation. Otherwise, contrast was increased.

This process continued until a reversal occurred. A reversal is a response that exceeded the target firing rate having fallen below it on the previous presentation, or a response that fell below having exceeded it on the previous presentation. After each reversal, the magnitude of the contrast adjustment was reduced by a factor of 0.66 in some experiments or 0.5 in others. After 7 reversals, the procedure halted. The contrast at the last reversal was defined as the staircase termination. Presentations of stimuli in 3 to 10 color directions were randomly interleaved to mitigate changes in adaptation.

Phase 2: Color direction selection

The color direction selection algorithm is most easily explained geometrically in the color space shown in Supplementary Fig. 4. Termination points of each of the three initial staircases (usually conducted in the L+M, L−M, and S directions) for a hypothetical V1 neuron are shown in Supplementary Fig. 5a. Color directions probed in the second iteration of staircases were selected by connecting the initial 6 staircase terminations to create four symmetric pairs of triangles (Supplementary Fig. 5b). The second round of staircases was conducted along the four lines connecting the midpoint of each triangle to the origin (Supplementary Fig. 5c). The contrast for each staircase began at the midpoint of its parent triangle.

Sometimes the contrast requested by the staircase algorithm was outside of the monitor gamut. The terminations of such staircases was set arbitrarily to the edge or, more often, a point 50–70% of the distance from the origin to the edge of the monitor gamut. These terminations were not used in the surface fits except to penalize surfaces that did not leave the gamut (as described in Model fitting and statistics,).

The outcome of second-round staircases was used to select color directions probed in the third round. Each of the eight original triangles shown in Supplementary Fig. 5b was divided into three sub-triangles by connecting the terminations of the second-round staircases with each pair of vertices. This is shown for one triangle in Supplementary Fig. 5d. New staircases were then conducted through the midpoint of each of the subtriangles. This recursive procedure was used in every subsequent round of the experiment; staircases were conducted through the midpoint of triangles which were then divided into three subtriangles, the midpoints of which were then probed with new staircases.

Some subtriangles were not probed. If a staircase terminated within a specified distance from its parent triangle (usually ± 30% of the distance from the origin to the triangle), the parent triangle was not subdivided and no additional staircases were passed through it. This strategy prevents sampling regions of color space over which the isoresponse surface is flat. Similarly, triangles were not probed if all three vertices were outside the monitor gamut. This strategy prevents sampling regions of color space over which the neuron is unresponsive. This procedure is based on the assumptions that isoresponse surfaces are smooth and that contrast response functions are monotonic. In the absence of noise, it yields a nonparametric estimate of the isoresponse surface that is composed of a mesh of interlocking triangles that are small in areas where the isresponse surface is curved and large in areas where it is flat.

Model fitting and statistics

Staircase terminations were fit with a pair of symmetric planes that can be described by the equation:

$$\mid \mathit{ax}+\mathit{by}+\mathit{cz}\mid =1$$ (2)

where a, b, and c are fitted parameters that determine the position and orientation of the planes. To improve numerical stability in the fitting algorithm without biasing the end result, fitting was performed in a “whitened” cone contrast space such that x, y, and z were different linear combinations of L-, M-, and S-cone contrasts ⁴⁸. Fitted parameters were then unwhitened (transformed back to cone contrast space) for surface rendering and cone weight calculation.

Measurement error was radial because our dependent variables, the staircase terminations, were measured along fixed color directions. We fit models by minimizing squared radial error:

$$\mathit{SSE}=\sum _i{\left(\log \left(r_i\right)-\log \left({\widehat{r}}_i\right)\right)}^2$$ (3)

where r_i is the distance from the origin to the i^th data point and ${\widehat{r}}_i$ is the distance from the origin to the i is the distance from the origin to the fitted surface in the same direction. The log transformation assumes multiplicative errors, consistent with the scaling of contrast steps during the contrast titration procedure. It also renders the fits invariant to linear transformations of the color space: linear transformations scale each r_i and ${\widehat{r}}_i$ by the same factor which cancels in the subtraction. Staircases that exceeded the monitor gamut did not contribute to the error unless they penetrated the fitted surface. In other words, staircases that should have hit the surface prior to exiting the monitor gamut increased the error, but staircases terminating at the edge of the gamut prior to hitting the surface did not. Fitting was performed using a generic function minimization algorithm (fminsearch in Matlab) seeded with initial guesses from a grid search.

Staircase terminations were also fit with the quadratic equation:

$${\mathit{ax}}^2+{\mathit{by}}^2+{\mathit{cz}}^2+2\mathit{dxy}+2\mathit{exz}+2\mathit{fyz}=1$$ (4)

which can describe an ellipsoid, a 1-sheet hyperboloid, or a 2-sheet hyperboloid depending on the choices of the six parameters: a, b, c, d, e, and f. The category of shape is invariant to linear transformations of the color space. Surfaces described by this equation are symmetric about the origin, as required by the Gabor stimulus we used, which modulates symmetrically about the background white point. Additional linear terms (e.g. gx+hy+iz) would have allowed the surfaces to be asymmetric with respect to the origin and were therefore not included.

The eigenvectors of the matrix

$$\left(\begin{array}{ccc}\hfill a\hfill & \hfill d\hfill & \hfill e\hfill \\ \hfill d\hfill & \hfill b\hfill & \hfill f\hfill \\ \hfill e\hfill & \hfill f\hfill & \hfill c\hfill \end{array}\right)$$

are the principal axes of the quadratic surface and the eigenvalues are the lengths of these axes ²². Pairs of axes that have the same length have non-unique directions. Pairs of axes that are similar in length have numerically unstable directions. In our data, 75 of 236 axis pairs were sufficiently different in length (≥ a factor of 5) to permit a meaningful analysis of axis orientation.

Because of its three extra parameters, the quadratic model always fit the data better that the planar model. To compare the quality of the model fits, we used an F-test based on the test statistic:

$$F=\frac{\left({\mathit{SSE}}_{\text{plane}}-{\mathit{SSE}}_{\text{quad}}\right){/}_3}{{\mathit{SSE}}_{\text{quad}}{/}_{n-6}}$$ (5)

where n is the number of staircases that terminated inside the monitor gamut. We also compared the ability of planar and quadratic models to predict the position of individual data points intentionally omitted from the fitting process. We conducted this analysis n times per neuron, leaving out a different data point each time and fitting the planar and quadratic models to the remaining n-1 data points. Prediction accuracy was quantified as median squared error.