Cone selectivity derived from the responses of the retinal cone mosaic to natural scenes

Abstract

To achieve color vision, the brain has to process signals of the cones in the retinal photoreceptor mosaic in a cone-type-specific way. We investigated the possibility that cone-type-specific wiring is an adaptation to the statistics of the cone signals. We analyzed estimates of cone responses to natural scenes and found that there is sufficient information in the higher order statistics of L- and M-cone responses to distinguish between cones of different types, enabling unsupervised learning of cone-type specificity. This was not the case for a fourth cone type with spectral sensitivity between L and M cones, suggesting an explanation for the lack of strong tetrachromacy in heterozygous carriers of color deficiencies.

Introduction

Two chromatic pathways originating in the retina subserve color vision in humans and other trichromatic primates (Mollon, 1989). One pathway compares intensity in the short-wavelength part of the spectrum, as conveyed by S cones, with intensity in the longer wavelength part, as conveyed by L and M cones, in trichromatic primates. The second, phylogenetically younger chromatic pathway encodes spectral composition within the longer wavelength part of the spectrum by comparing the responses of L and M cones.

S cones differ from L and M cones in morphology, neurochemistry, and spatial arrangement in the retina (Ahnelt, Keri, & Kolb, 1990; Calkins, Tsukamoto, & Sterling, 1998; Hendry & Calkins, 1998; Kouyama & Marshak, 1992). These differences could underlie the developmental formation of cone-specific circuitry with respect to S versus L and M cones and thus lead to S–LM cone-opponent processing as realized in this ancient color pathway (Dacey & Lee, 1994; Martin, White, Goodchild, Wilder, & Sefton, 1997).

In contrast to the differences between S cones and L or M cones, it is unclear whether there is any difference between L and M cones apart from their photopigments (Dacey, 2000; Hendry & Calkins, 1998), which determine the spectral sensitivities of the cones and, consequently, the responses to chromatic stimuli. Moreover, there is evidence that the cone opsin is the only feature that defines cone type in L and M cones (Smallwood, Wang, & Nathans, 2002; Smallwood et al. 2003; Wang et al., 1999). Because the visual system is able to distinguish between L and M cones, the question of how this is achieved arises. Two fundamentally different mechanisms are conceivable.

One possibility would be that the different cone opsins expressed lead to further changes at the molecular or cellular level (Mollon, 1999), based on which postreceptoral neurons could connect specifically to the respective cone types. So far, however, there is no clear evidence for such changes as a consequence of cone pigment expression in L and M cones.

Alternatively, appropriate wiring could be learned in an unsupervised way, based on the receptor responses, which are different in L and M cones, depending on the wavelength composition of the stimuli. Learning cone-specific wiring based on cone responses would not require differences between L and M cones other than the differences in photopigments. The distinction between L and M cones would be achieved as a result of the different statistics of their signals. Neural plasticity has been proposed as a mechanism to achieve cone-specific wiring (Boycott & Wässle, 1999; Nathans, 1999), and the mammalian visual system is known to adapt during development to the properties of visual input (Blakemore & Cooper, 1970). But are the responses of L and M to natural stimuli sufficiently different to distinguish between these cone types?

The properties of the cone photoreceptor mosaic pose several problems for the distinction between L and M cones on the basis of their responses. First, the spectral sensitivities of L and M cones overlap strongly (Figure 1C), leading to highly similar responses. Second, as a consequence of the fact that only one cone exists at each location in the retinal mosaic, cone responses confound information about spatial variation and about spectral composition: A difference in the responses of two cones may be caused by a difference in intensity or spectral composition in the retinal image at the locations of these cones, or by a difference in the spectral sensitivities of the cones, or by both. Thus, spatial variability further obscures the small differences between L- and M-cone responses, as illustrated in Figures 1D–1G. Consequently, the joint distributions of the responses of neighboring cones seem virtually indistinguishable with respect to cone type (Figures 1F and 1G).

Figure 1.

Cone responses to natural scenes confound spatial and chromatic information. (A) Example of natural scene image. (B) Simulated cone mosaic to sample the images. Blue, S cones; green, M cones; red, L cones. (C) Human cone spectral sensitivities (Stockman & Sharpe, 2000). (D–G) Effects of photoreceptor location and spectral sensitivity. Plots show joint distributions of the response estimates of two cones in the mosaic patch to 1,500 random samples of the images from the Párraga data set, assuming a nonlinear cone response function (see Methods section). Insets indicate the types and relative positions of the two cones for each plot. (D) Two L cones sharing the same retinal location. (E) Effect of differences in spectral sensitivity: hypothetical case of L and M cones sharing the same retinal location (correlation coefficient r = .995). (F) Effect of differences in spatial location: two neighboring L cones (r = .923). (G) Combined effect of location and spectral sensitivity: neighboring L and M cones (r = .910). A requirement for the learning of cone-specific wiring is the ability to determine whether two cones in the mosaic are of the same or of different type, that is, discrimination between the distributions in Panels F and G.

As further illustration, let us consider three cones in a row at adjacent positions: one L, one M, and another L cone. According to estimates of cone responses to natural scenes (see Methods section), the responses of the first L cone will correlate stronger with the neighboring M cone (r = .910) than with the L cone further away (r = .826). Thus, mechanisms of plasticity that simply wire together neurons according to signal correlations can be expected to fail to differentiate between L and M cones.

To investigate the possibility of unsupervised learning of cone selectivity, we had to take into account the constraints imposed by the cone mosaic arrangement (see also Doi, Inui, Lee, Wachtler, & Sejnowski, 2003). We analyzed estimates of L-, M-, and S-cone responses to images of natural scenes, spatially sampling the images with a cone mosaic patch that mimics the arrangement of cones in the human retina (Figure 1B). Data with responses of all three cone types at each image point (LMS images), as used in several studies of cone response statistics (Caywood, Willmore, & Tolhurst, 2004; Ruderman, Cronin, & Chiao, 1998; Wachtler, Lee, & Sejnowski, 2001), would contain an unrealistic amount of information and would therefore not be appropriate for the analysis.

In a previous article (Doi et al., 2003), we had analyzed cone mosaic responses to natural scenes and found that spatiochromatic receptive field properties derived by decorrelation methods showed striking similarities to properties of neurons in the LGN and cortex. The results suggested the possibility that the statistics of L- and M-cone responses might be sufficient to distinguish between these cone types. Here, we investigate this possibility by analysis of cone mosaic responses using different data sets and methods. The results greatly support the possibility that color selectivity in the visual system is the result of a learning process.

Methods

Data sets

Cone response estimates were derived from 62 natural scenes taken in Kyoto Botanical Gardens with a carefully calibrated 1,280 × 1,000 pixel 3CCD digital still camera system (HC-2500, Fujifilm, Japan). Pixel size corresponded to 2.3′ of visual angle. The linear RGB data were transformed into LMS cone excitation estimates (Stockman & Sharpe, 2000) by a 3 × 3 matrix that yielded minimal estimation error (Doi et al., 2003). The data were further transformed using the nonlinear cone response function (Baylor, Nunn, & Schnapf, 1987) r_cone = 1 − exp(−kr), where r is the linear cone excitation, r_cone is the estimated cone response, and the parameter k was determined such that the median of r_cone was 0.5.

A second set of cone response estimates was obtained from eight hyperspectral images of the data set of Párraga, Brelstaff, Troscianko, and Moorehead (1998). Images were 256 × 256 pixels, with a pixel size of 3.3′. We used the same eight outdoor images as in previous studies of LMS images (Lee, Wachtler, & Sejnowski, 2002; Wachtler et al., 2001). LMS images were obtained by calculating the inner product between the image pixel spectra and the Stockman and Sharpe (2000) human cone spectral sensitivities. As in our previous studies with LMS images derived from this data set (Lee et al., 2002; Wachtler et al., 2001), we used the logarithm of the cone responses. For comparison, we also performed an analysis with the nonlinear cone response function described above.

The LMS (cone response) images were sampled by a simulated hexagonal mosaic of 217 cone photoreceptors (Figure 1B). Each cone corresponded to a 2 × 2 pixel image patch, with successive rows of cones shifted by 1 pixel to approximate a hexagonal arrangement. S cones were assigned to cone positions on a regular hexagonal grid with a spacing of four cones. L and M cones were assigned randomly to the remaining photoreceptor positions with probabilities of 2/3 and 1/3, respectively. Thus, L:M:S cone ratios of approximately 10:5:1 were achieved. The obtained mosaic patches approximate the arrangement of cones in the human retina (Roorda & Williams, 1999). From the two data sets, we derived 507,904 and 460,800 samples, respectively, that were used for analysis.

Analysis

We consider representations of cone responses by a linear model visual system, u = Wx, where x = (x₁, …, x_n)^T is the vector of cone responses, with T denoting the transpose; W = [w₁,…, w_n]^T is the square matrix of filters, whose rows correspond to the receptive fields of model neurons; and u = (u₁,…,u_n)^T is the neural representation (responses of model neurons). Thus, each neuron’s responses are modeled as a weighted sum of cone responses, with weights given by the receptive field of the neuron. Note that each of the n dimensions of x and W corresponds to a unique spatial location because only a single cone exists at each position in the mosaic. This is in contrast to the analysis of LMS images (Caywood et al., 2004; Lee et al., 2002; Wachtler et al., 2001), where, for each pixel, there are three components, corresponding to L, M, and S responses.

We consider unsupervised learning of receptive fields, that is, of the matrix W. Unsupervised methods find transformations of the data that achieve some objective with respect to the properties of the data, for example, decorrelation of the signals. The matrix W is a decorrelating matrix if the covariance matrix of the output vectors is diagonal, 〈uu^T〉 = D, where 〈⋯〉 denotes the average and D is a diagonal matrix. In general, there is no unique solution for W because UW is also a decorrelation matrix for any diagonal matrix U. There are various ways to constrain the solution. We applied three commonly used decorrelation methods to investigate the ability of achieving cone specificity by decorrelation: principal component analysis (PCA), zero-phase whitening (ZCA), and independent component analysis (ICA). Details of the algorithms used can be found in our previous article (Doi et al., 2003) and in Bell and Sejnowski (1997) and Lee, Girolami, and Sejnowski (1999). Here, we briefly summarize these methods.

Principal component analysis

Principal component analysis is a covariance-based decorrelation method with an orthogonality constraint on the transform. The decorrelation matrix W_P consist of the eigenvectors of the input covariance matrix. The transformation captures as much variance as possible with a limited number of outputs and thus achieves an efficient representation in the sense of minimal mean squared errors of the reconstruction. Principal components were calculated using the Matlab princomp function.

Zero-phase whitening

The matrix W_Z is called a whitening matrix if its covariance matrix 〈uu^T〉 is the identity matrix, 〈uu^T〉 = 〈(W_Zx)(W_Zx)^T〉 = I. W_Z decorrelates the data x and also normalizes variance. Here, we assume that the whitening matrix is symmetric, ${\mathbf{W}}_{\mathbf{Z}}={\mathbf{W}}_{\mathbf{Z}}^T$, resulting in ZCA filters (Bell & Sejnowski, 1997). Accordingly, we use the abbreviation ZCA for this method. Like PCA, ZCA is covariance based and removes second-order dependencies from the data. In contrast to PCA, ZCA does not constrain the matrix to be orthogonal but symmetric. ZCA can be achieved with biologically plausible mechanisms, and it has been proposed as a model for visual processing in the retina (Atick, 1992; Bell & Sejnowski, 1997; Doi et al., 2003). W_Z was derived by the square root of the inverse of the input covariance matrix (Bell & Sejnowski, 1997).

Independent component analysis

Independent component analysis is a decorrelation method that removes not only second-order but also higher order redundancy, thus achieving maximal statistical independence of the outputs. ICA is performed by adapting the parameters of a linear transformation of sequentially presented data such that mutual information between the outputs is minimized (for review see Girolami, 1999; Hyvärinen, Karhunen, & Oja, 2001; Lee, 1998).

We used the extended infomax learning rule with natural gradient extension (Lee et al., 1999). In this scheme, the learning algorithm for updating the filter matrix W_I is ΔW_I ∝ [I − φ(u)u^T]W_I, where φ(u) = −∂logp(u)/∂u and ΔW_I is the change of W_I at each learning step. ΔW_I will converge to zero as the adaptation process completes. Note that φ(u) requires a probability density model p(u_i). The extended infomax algorithm takes into account both super-Gaussian and sub-Gaussian densities (Lee, 1998). Thus, our experiments do not constrain the coefficients to have a sparse distribution, unlike some methods used in previous studies (Bell & Sejnowski, 1997; Olshausen & Field, 1996). For comparison, we also used another variant of the ICA algorithm where the probability density is adjusted during the learning process to match the distribution of the estimated sources (Lee & Lewicki, 2000), which had been used for analysis of LMS images (Lee et al., 2002).

Cone-type specificity

Cone-type-specific units were first classified by visual inspection. A criterion for cone-type specificity was that cone weights depended on cone type, not only on spatial location. This was the case when the weights of neighboring cones had the same sign for cones of the same type but an opposite sign when the cones were of different types or when, within a subregion of the receptive field, all cones of one type had consistently larger or smaller weights than did neighboring cones of a different type. We classified receptive fields that showed consistent differences in connection strengths between S cones and both L and M cones, or had receptive field subregions fulfilling this criterion, as S–LM specific. Likewise, receptive fields with consistently different connections from L and M cones, or with such subfields, were classified as L–M specific.

To quantify cone-type specificity, we used two different measures. The results for both measures agreed with the visual classification. As in our previous study (Doi et al., 2003), we derived a color selectivity index according to Hanazawa, Komatsu, and Murakami (2000). First, the size and orientation of the achromatic bar stimulus that yielded the largest response were determined. Then, responses to bars of the same geometry but different isoluminant colors were determined (Doi et al., 2003). Colors were isoluminant and systematically distributed on the CIE 1931 chromaticity diagram, corresponding to the stimuli used by Hanazawa et al. From the responses, we calculated color selectivity index as 1 − (minimum response)/(maximum response). The higher this index, the stronger the discrimination between different colors. An index higher than 1 indicates color opponency, that is, excitation by some colors and inhibition by others. All units classified as S–LM specific or L–M specific had color selectivities above 1. Units with color selectivity index below 1 were classified as achromatic.

This method yields a measure comparable with physiological studies, but it is an indirect measure and can depend on the choice of stimuli. Because we had the full receptive field structure in terms of the cone weights, we derived a more general measure of cone-type specificity with respect to L and M cones. For each unit, we obtained an overall weight distribution for each of the cone types separately by convolving the discrete cone weight pattern with a two-dimensional Gaussian kernel G,

$$l(x,y)={\displaystyle \sum _i{L}_{i}G(x-x_i,y-y_i)},$$

and

$$m(x,y)={\displaystyle \sum _j{M}_{j}G(x-x_j,y-y_j)},$$

where L_i and M_j are the cone weights and (x_i, y_i) and (x_j, y_j) are the respective positions of the cones. The width of the Gaussian was 1 cone spacing. Thus, we obtained continuous spatial distributions corresponding to smoothed versions of the spatial weight patterns of each cone type. These distributions can be regarded as receptive fields for cone-isolating stimuli. The receptive field plots shown in the insets of Figures 2 and 3 and in Figure 4B were obtained by combining such cone-isolating receptive fields. The purpose of this procedure was to spread each cone weight within a small region so that the weights of neighboring cones could be compared. An L–M selectivity index was calculated as the integral of the difference between the weight density distributions of L and M cones normalized by the sum of absolute cone weights,

$$c_{\mathit{\text{LM}}}=\frac{{\displaystyle \int \left|l(x,y)-m(x,y)\right|\mathit{\text{dxdy}}}}{{\displaystyle \sum _i\left|L_i\right|}+{\displaystyle \sum _j\left|M_j\right|}}.$$

Figure 2.

Spatiochromatic receptive fields derived by PCA and ZCA. Circles represent connections from cones at respective positions in the mosaic. Circle size corresponds to the strength of connection; cone types are denoted by colors (red, green, and blue for L, M, and S cones, respectively), where light colors indicate excitation (positive connections) and dark colors with black outlines indicate inhibition (negative connections). Insets show the overall spatiochromatic receptive field structures, obtained by blurring each cone weight to generalize from the particular arrangement of cone types in the mosaic. (A) Orthogonal decorrelation (PCA). Receptive fields have global structure with increasing spatial frequencies. The plot at the far right shows one of three receptive fields with S–M cone specificity. No receptive fields with L versus M specificity were found. (B) Zero-phase decorrelation (ZCA). Receptive fields have localized, circular center-surround structure with a center of a single cone. All cone types contribute to the surround with equal sign, which depends on distance from the center.

Figure 3.

Spatiochromatic receptive fields derived by ICA. Left column: two receptive fields sensitive to luminance; cone input strengths depend on spatial location with similar strength for L and M cones. Middle column: Two receptive fields showing S–M specificity; within receptive field subregions, S cones have opposite signs of L and M cones. Right column: Two receptive fields showing L–M specificity; within receptive field subregions, L and M cones have opposite signs. Symbols and insets as in Figure 2.

Figure 4.

Cone-type specificity of receptive field structure. (A) Signs of weights of the L–M-specific unit on the lower right panel of Figure 3. Left: Positive weights are indicated by black dots; negative weights, by white dots for both L and M cones. Right: Positive weights are indicated by black dots; negative weights, by white dots for M cones and vice versa for L cones. This yields homogeneous subfields, indicating +L − M and −L + M opponency. (B) Receptive fields corresponding to the cone weight pattern imposed on the original mosaic (leftmost receptive field plot, “learned”) and on four mosaic patches where the type of each L and M cone had been reassigned randomly with the same probabilities as in the original mosaic (“randomized mosaics”). In the latter cases, the resulting receptive field structures are weak and irregular, indicating that the learned weight pattern is not only spatial but also cone-type specific.

This index measures the differences between neighboring cones of different types across the receptive field. A predominantly spatial (achromatic) receptive field structure with no systematic differences between L and M cones will yield a small value. If there are systematic differences between neighboring L and M cones, that is, the receptive field structure is L–M selective, the index will approach a value of 1. For example, a perfectly L–M-selective receptive field, where all L-cone weights have the same sign and all M-cone weights have the opposite sign, will yield an index value of 1.

Results

Principal component analysis

Receptive fields corresponding to three of the principal components of the cone mosaic responses are shown in Figure 2A. The vast majority (214 of 217) of receptive fields showed no cone specificity: Their receptive field structure consisted of inputs from all cones, and the input strengths were spatially modulated, with no apparent specificity for either cone type. The connection from each cone to these receptive fields depended on the position of the cone, but not on cone type. Only three receptive fields (1.4%) showed S–LM specificity, with the spatial structure of these receptive fields being different for S cones from that for L or M cones (see Figure 2A, right panel). In agreement with our previous analysis (Doi et al., 2003), there were no L–M-specific receptive fields. To quantify this observation, we calculated the L–M selectivity index and found that it was less than 0.5 for all PCA units. This finding contrasts with results of previous studies (Buchsbaum & Gottschalk, 1983; Ruderman et al., 1998) that did not take into account properties of the cone mosaic and indicates that, under realistic conditions, it is not possible to achieve color opponency by decorrelation under the orthogonality constraint, as with PCA.

Zero-phase whitening

As reported previously (Doi et al., 2003), the receptive fields derived by decorrelation with the symmetry constraint (ZCA) show a concentric center-surround organization, similar to receptive fields of neurons in the retina (Boycott & Wässle, 1999) and LGN (Wiesel & Hubel, 1966). The receptive field center corresponds to a single cone (Figure 2B). The surround is “mixed”, drawing from both L and M cones with like sign. A mixed surround is consistent with current anatomical evidence for the organization of retinal receptive fields (Boycott & Wässle, 1999). Single-cone-center receptive fields with mixed surround preserve the specificity of the center cone but do not establish cone-specific wiring because the processing of cone signals depends solely on spatial relations, not on cone type.

In the ZCA receptive fields, both cone types contributed to the surround. But there was a slight bias toward larger negative values in cones with the type of the center cone, indicating weak cone specificity in the surround. However, the L–M selectivity index was less than 0.6 for all ZCA units. Values above 0.5 occurred for three units where a center M cone was surrounded exclusively by L cones (in the mosaic shown in Figure 1B, the M cone at Position 8 from the left in the fourth row from top and the two M cones at the far right of rows 3 and 5 from bottom), indicating that even in these cases, the index values could be explained by a purely spatial weight pattern.

The cone-type specificity of the surround has been a controversial issue in the literature. The ZCA model is in accordance with receptive field estimates derived from the results of detection experiments (McKee & Westheimer, 1970); physiological results, however, often have found cone-type opponency between the center and the surround (Buzás, Blessing, Szmajda, & Martin, 2006; Lee, Kremers, & Yeh, 1998; Reid & Shapley, 1992).

Independent component analysis

In contrast to the previous methods, ICA attempts to remove both second-order and higher order statistical dependencies from the data. Statistical analysis of the cone response estimates shows that, indeed, they are far from Gaussian. In particular, response differences of neighboring cones have high kurtosis that depends on whether the cones are of same or different types. (For the data in Figures 1E and 1F, the kurtosis values of the differences of cone responses are 4.2 and 4.7, respectively, compared with 0 for Gaussian data.)

Many of the resulting ICA filters showed a spatial receptive field structure that was localized and oriented, similar to the achromatic edge filters found in previous studies using natural images (Bell & Sejnowski, 1997; Olshausen & Field, 1996; Wachtler et al., 2001). Most receptive fields showed no overt specificity for L or M cones. Several receptive fields, however, exhibited strong L–M selectivity, in agreement with previous results (Doi et al., 2003). The right column of Figure 3 shows the receptive fields of the two units with the highest color selectivity and L–M selectivity index values. The color selectivities of these units for isoluminant color stimuli were 1.64 and 1.70. This indicates that these filters were highly color selective, discriminating between different isoluminant colors. For these units, the weights of L and M cones had opposite signs within receptive field subregions, with the exception of a few single cones. Correspondingly, the values of the L–M selectivity index for these units (0.75 and 0.80) were much higher than for the other units (Figure 5C). Figure 4A illustrates the consistent cone-specific pattern of cone weight for one of the L–M-specific units. The positive and negative signs of cone weights are indicated by black and white dots, respectively. When the sign-to-color assignment is the same for L and M cones, the receptive field shows an irregular pepper-and-salt pattern. When the assignment for L cones is opposite to that for M cones, the receptive field shows two almost homogeneous subfields with opposite sign. Similarly, when the same spatial weight pattern was applied to mosaic patches with different realizations of the random L–M mosaic assignment, the resulting receptive fields were weak and irregular (Figure 4B). This indicates that the receptive field structure was not independent of cone type, but that the distinction between cone types had been learned based on the cone responses.

Figure 5.

L–M selectivity index distributions. (A)–(C) Histograms of L–M selectivity index for units obtained from the 3CCD camera image data set: (A) PCA, (B) ZCA, and (C) ICA, with the nonlinear cone-response function and the extended infomax algorithm; arrows indicate the values for the two red–green units shown in Figure 3. (D) Units obtained from the hyperspectral images data set with the log nonlinearity and the ICA algorithm with adaptive probability density model.

The ICA results were robust, and the occurrence of L–M-specific units did not depend on the arrangement of cones in the mosaic, on a specific algorithm, or on the data set. Likewise, prewhitening the data (Doi et al., 2003) or not did not affect the results. The data set by Párraga et al. (1998) contains images of outdoor scenes and flowering plants, which typically have higher red–green contrast than outdoor images do. Using the outdoor images yielded similar results to those of the previous analysis. Using images with higher red–green contrast likewise yielded qualitatively similar results, but overall, there was a tendency for M cones to have lower weights, indicating a slight shift in the chromatic tuning of the units. We tested the role of the nonlinearity by repeating the analysis with the second data set with either the exponential cone response function or with the logarithm. In both cases, L–M-specific units emerged, indicating that the non-linearity is not critical in this respect. Figures 5C and 5D show a comparison of the distributions of L–M selectivity index values obtained with different data sets, different nonlinearities, and different ICA algorithms. There are quantitative differences, but qualitatively, the distributions are similar, and in particular, in both cases, there are units with high L–M selectivity index.

The relatively low number of overtly L–M-specific components may be the result of certain constraints of the method and the nature of the cone mosaic, where even wiring by purely spatial mechanisms leads to receptive fields with a spatiochromatic structure. In the cone mosaic, L and M cones are organized in a fixed, random pattern. At each location, only one cone exists. Any wiring pattern will therefore generate a spatiochromatic receptive field. We analyzed the ICA filters with respect to the achromatic receptive field structure, considering pure luminance inputs that stimulate L and M cones in the same way, and with respect to chromatic receptive fields, considering chromatic inputs that stimulate L and M cones oppositely. Figure 6 shows a unit that was classified as achromatic and shows an oriented, Gabor-like receptive field structure for achromatic stimuli, that is, selectivity for achromatic edges with a certain orientation. However, this unit is also sensitive to L–M opponent stimuli. Its receptive field has small subregions where neighboring L and M cones have opposite signs, making it selective for red–green edges with an orientation that, however, is different from the preferred orientation for achromatic stimuli. Similar mismatch of preferred orientation for chromatic and achromatic stimuli was previously reported for ICA filters derived from LMS images (Caywood et al., 2004).

Figure 6.

Different receptive field structure for chromatic and achromatic stimuli. As a consequence of the mosaic arrangement, each receptive field has a spatiochromatic structure. The receptive field structure for chromatic stimuli may be different from that for achromatic stimuli. Left: Input weights as in previous figures. Middle: Smoothed receptive field (as in insets of previous figures) for achromatic stimuli. L and M cones contribute with like signs, as would be the case for pure luminance stimuli. Right: Smoothed receptive field for chromatic (L–M opponent) stimuli. L and M cones contribute with different signs, as would be the case for isoluminant chromatic stimuli. The unit is selective for oriented achromatic and chromatic stimuli, but the preferred orientation is different for achromatic than for chromatic edges.

Most of the units classified as achromatic had, in addition to their localized and oriented receptive fields for luminance stimuli, a localized and oriented chromatic receptive field (Doi, 2003). The preferred orientation of the chromatic receptive field was often different from that of the achromatic receptive field. However, the cone-specific subfields were small, and it cannot be excluded that they appeared by chance (cf. Young & Marrocco, 1989). Nevertheless, these properties enable representation of chromatic (red–green) image structure, explaining how the inputs could be represented with the small number of overtly L–M opponent units. In contrast, the two L–M opponent units had achromatic receptive fields with a seemingly random structure (Figure 4), confirming that these receptive fields were determined primarily by cone specificity rather than spatial structure. They encode low-spatial-frequency chromatic information (Figure 3), which is probably incompatible with the simultaneous encoding of luminance pattern information.

To investigate the spatiochromatic constraints of the mosaic arrangement, we blurred the images prior to sampling, using a kernel with a half-width corresponding to the cone spacing. In this case, we obtained a higher number of L–M-specific units (8/217 = 3.7%). Thus, with reduced spatial information in the images, more components became available for representation of chromatic information, confirming that the low number of L–M-specific units is a consequence of the specific properties of our methods.

Tetrachromatic cone mosaic

In the human population, there is a relatively high number of potentially tetrachromatic heterozygous carriers of X-linked color deficiencies, but so far, no clear cases of perceptual (gstrongh) tetrachromacy have been found (Jordan & Mollon, 1993; Nagy, MacLeod, Heyneman, & Eisner, 1981). In the study by Jordan and Mollon (1993), only one 1 of 31 carriers showed weak evidence of tetrachromacy. We investigated the possibility of learning of cone selectivity with four cone pigments. We repeated our experiments using a mosaic with four cone types, replacing randomly, with a probability of .5, L cones in the mosaic (Figure 1A) by cones of the fourth type. The spectral sensitivities for the fourth cone type had a peak wavelength between the L- and M-cone peak wavelengths of 530 and 560 nm. The shape was determined based on the template of Lamb (1995), taking into account prereceptoral absorption (Stockman, MacLeod, & Johnson, 1993). Cone response estimates were obtained by sampling from hyperspectral images (Párraga et al., 1998).

Three different spectral absorptions for the fourth cone with peak wavelengths of 538, 545, and 552 nm, respectively, were used in different experiments. Cone-type selectivity for the fourth cone type was never observed, although these cones contributed to color-selective receptive fields. Their contribution was of similar magnitude as for L and M cones, and almost all of their weights within a subfield had the same sign (Figure 7).

Figure 7.

ICA results for tetrachromatic cone mosaic. Cone-type-selective unit from a mosaic where L cones (Figure 1) had been replaced, with a probability of .5, by a fourth cone type. Color codes for S, M, and L cones as in previous figures. Cones of the fourth type are plotted in light and dark yellow for positive and negative input sign, respectively.

To investigate whether the lack of opponency resulted from an inability to distinguish between the cone types at all due to the high similarity between L or M cones and the fourth cone type, we repeated the experiments with a trichromatic mosaic, using the spectral sensitivity with 545-nm peak wavelength instead of the L cones. The analysis yielded cone-type-specific units similar to those obtained with the L, M, S mosaic, indicating that the similarity of the spectral sensitivities per se was not the cause for the lack of tetrachromacy. In summary, introducing the fourth cone type did not lead to a further dimension of opponency but rather modified the L–M opponency, effectively resulting in larger variability of the spectral tuning of these opponent receptive fields. These results suggest that an explanation for the lack of tetrachromatic perception in female humans may be that, given the human retinal structure and cone pigments, the information in the cone signals may not be sufficient for learning of cone specificity with respect to the fourth cone type.

Discussion

Our results demonstrate that it is possible to learn the individual types and positions of L and M cones in the retinal mosaic by analyzing their responses to natural visual input. Achieving color vision by learning would have enabled the visual system to immediately make use of a trichromatic retina after a gene duplication introduced the third cone type in primate evolution.

The comparison of different methods suggests that such a learning process may be supported by information in the higher order statistics of cone responses because methods based on second-order statistics failed to produce cone-opponent receptive fields. However, the structures of the ZCA filters (Figure 2B) agree well with the center-surround receptive field organization in the fovea of the mammalian retina (Wässle & Boycott, 1991). Thus, it could be speculated that the visual system separates removal of first- and second-order statistics from that of higher order statistics.

Single-cone-center receptive fields, as in midget retinal ganglion cells (Wässle & Boycott, 1991), have been shown to achieve some degree of color opponency (Lennie, Haake, & Williams, 1991; Paulus & Kröger-Paulus, 1983), even if the receptive field surround mixes L and M cones. Therefore, this wiring scheme has often been proposed as the solution to the problem of cone-specific processing. It explains color selectivity of retinal ganglion cells and LGN cells without assuming cone-type-specific wiring of L and M cones. However, the single-cone-center scheme merely postpones the problem of distinction between L and M cones to the next stages of visual processing. At some point, the brain has to combine visual signals in a way that takes account of cone type to achieve an explicit representation of color.

Nevertheless, it is conceivable that the decorrelation achieved by the center-surround processing makes it easier for subsequent visual stages to learn cone selectivity. This possibility is considered in Figure 8, where midget cell responses are estimated under the assumption of a single-cone center and a mixed surround. For simplicity, the surround was simulated as the average of a pure L and a pure M surround. Thus, in terms of cone composition, the surrounds are homogeneous and identical for all cells. In reality, the random cone compositions of the surrounds would introduce further variability in the responses, that is, larger scatter in the data points of the distributions. Thus, our simulation overestimates the discriminability between the response distributions.

Figure 8.

Effect of center-surround receptive field structure. Responses of simulated midget cells with single-cone centers and mixed surrounds. Joint distributions for center cone types and positions corresponding to Figures 1F and 1G, as indicated by insets. Mixed surrounds were idealized as averages of L and M cone surrounds; thus, variability in the distributions is lower than expected with realistic surrounds. Compared to cone responses, correlations are strongly reduced and differences between distributions are larger (A: r = .25; B: r = .10).

As Figure 8 shows, the opponency achieved by single-cone-center receptive fields with a random surround reduces the correlations between responses of neighboring units. The differences between correlations of responses of units with centers of the same cone type (r = .25) and correlations of responses of units with different center cone type (r = .10) are somewhat larger than the corresponding differences for cone responses (Figure 1). However, the distributions for L- and for M-center receptive fields are still very similar, suggesting that the contribution of purely spatial processing for the learning of cone selectivity is limited. Note, however, that these considerations require that the spatial processing preserves the cone-type information, such as in single-cone-center receptive fields. If the processing mixes cone responses indiscriminately, in both the center and the surround, then information about cone type would be lost, and cone-type specificity might be impossible to achieve.

Signals from L and M cones need to be processed adequately, that is, in a cone-type-selective manner, to achieve trichromatic color vision (Dacey, 2000; Mollon, 1989). However, it is not clear by what mechanism the visual system accomplishes this task (review, Dacey & Packer, 2003). Lack of evidence for alternative models has been taken to support the notion that L–M chromatic signals are carried together with spatial information by the midget pathway via the parvocellular layers of the LGN to the visual cortex (Boycott & Wässle, 1999). Plasticity in the processing of visual signals has been proposed to separate spatial and chromatic information in the midget pathway (Boycott & Wässle, 1999; Martin, Lee, White, Solomon, & Rüttiger, 2001; Nathans, 1999). Experience plays a role in the development of visual processing, such as orientation selectivity (Blakemore & Cooper, 1970). It is quite feasible that learning processes, which could affect connectivity in the retina (Calkins, Schein, Tsukamoto, & Sterling, 1994; Martin et al., 2001) and at later stages of the visual system, may also be important in the development of color vision (Brenner, Schelvis, & Nuboer, 1985). Our results with slightly increased image blur indicate that learning of color selectivity would be facilitated in infants, where spatial vision is not yet fully developed (Teller, 1997). Furthermore, the possibility that specificity for L versus M cones is acquired by learning would be consistent with the otherwise puzzling finding that experimental evidence supports at least some degree of cone-type specificity in the parvocellular pathway (Lee et al., 1998; Martin et al., 2001; Reid & Shapley, 1992, 2002), while a corresponding anatomical or molecular difference in the properties of L and M cones has not been identified (Boycott & Wässle, 1999; Calkins & Sterling, 1999; Dacey & Packer, 2003; Hendry & Calkins, 1998).

With our analysis, we could show that cone-type selectivity can be learned in principle, but the comparability of the specific properties of our results with color processing in the visual system is limited. Our analysis yielded very small numbers of overtly cone-specific units. This discrepancy probably reflects the properties of our method, which is constrained to linear transformations and non-overcomplete basis sets; that is, the number of output units is limited to the number of inputs. Achromatic contrasts are typically higher than chromatic contrasts in natural scenes (Ruderman et al., 1998), and thus, the algorithm will preferentially find units that represent achromatic information. A requirement for the representation of color images with only few cone-specific basis functions is that chromatic variation can be encoded by the other receptive fields. This is indeed the case, because, due to the mosaic arrangement with random placement of L and M cones, receptive fields will inevitably have a spatiochromatic structure (Figure 6).

Our analysis does not lead to strong conclusions with respect to the stage of the visual system where this learning may occur. It seems feasible that the observed cone specificity in retinal ganglion cells and the LGN (Buzás et al., 2006; Calkins et al., 1994; Martin et al., 2001; Reid & Shapley, 1992) is a result of synaptic learning processes at these early stages. Receptive fields in the retina change dynamically to adapt to the stimulus statistics (Hosoya, Baccus, & Meister, 2005). Further learning may occur in the cortex, where a different representation is achieved (De Valois & De Valois, 1993; Lennie, Krauskopf, & Sclar, 1990; Wachtler, Sejnowski, & Albright, 2003), possibly taking advantage of the higher number of neurons available for coding (Olshausen & Field, 1997). Our ICA results show closer resemblance to cortical receptive fields (Conway, Hubel, & Livingstone, 2002) than to the center-surround structure of receptive fields in the retina and LGN (Reid & Shapley, 2002). Likewise, the simultaneous encoding of color and orientation (Doi et al., 2003) may be reminiscent of findings in cortical neurons (Johnson, Hawken, & Shapley, 2001). However, a strict comparison at this level would be inappropriate. Our linear model of the visual system is an idealization, and the specific receptive field structures may reflect properties imposed by the method.

We used a method that attempts to minimize redundancy. Redundancy reduction has been proposed as one goal of sensory coding (Barlow, 1961), and codes with minimal redundancy have been found to share the properties of neurons in the early visual system (Bell & Sejnowski, 1997; Olshausen & Field, 1996). So far, no models exist for implementing a successful learning algorithm for redundancy reduction, such as the one we used in our analysis, in a biological system. Current ICA learning rules are nonlocal and require full feedback from the output to the adapting stage. However, local learning rules for information maximization have been proposed (Girolami, 1999; Linsker, 1997), suggesting that there are biologically feasible ways to extract the information necessary to learn cone specificity. The goal of this study, however, was not to model how the visual system might achieve color selectivity, but rather to investigate whether there is sufficient information in cone responses to learn overt cone-type specificity in an unsupervised way. The respective receptive fields found by our analysis are suitable for the representation of color and were obtained without prior knowledge of the types and positions of the cones in the retinal mosaic.

In color vision, both discrimination and identification of chromatic stimuli are important goals (Hillis & Brainard, 2005). We did not consider the question whether the learned representations specifically support discrimination of chromatic stimuli. This would have required assumptions about noise and properties of postreceptoral processing (von der Twer & MacLeod, 2001). To optimize discrimination of chromatic stimuli, it is necessary that the response dimensions corresponding to the different cone types are identified, that is, that the cone types can be distinguished and cone-type-specific processing is achieved. We could show that this is possible by unsupervised mechanisms.

The visual system might have access to additional information to support learning of cone-specific wiring. The consequences of perceptual decisions probably provide for strong reinforcement signals and could potentially play a role in the development of color vision. We cannot conclude that cone-specific processing is based solely on unsupervised learning. Mechanisms of supervised or reinforcement learning may facilitate the development of cone-type specificity. However, our results show that, in principle, the information in cone responses is sufficient and further information is not necessary.

Because unsupervised learning of cone types is based on differences in spectral sensitivities, the success of this strategy depends on these differences. If the spectral sensitivities of L and M cones were more similar, we would expect it to be harder or even impossible to achieve a differentiation. The same would be expected for an additional cone pigment with a spectral sensitivity near L or M cones. The failure to distinguish a fourth cone type with spectral sensitivity between L and M cones offers an explanation for the lack of finding tetrachromacy in human heterozygous carriers of color deficiencies. A fourth cone type would carry little additional information relative to L and M cones (D.H. Foster, personal communication, 2005; see also Foster et al. (2006)), which may be insufficient to enable differentiation by a learning mechanism. If cone signals are processed indiscriminately of cone type, the different spectral sensitivities lead to variations in responses that decrease the signal-to-noise ratio, for both chromatic and achromatic stimuli. Thus, increasing the number of cone types without appropriate cone-type-specific processing would not improve color vision.

In dichromatic monkey species, many heterozygous females with three cone types exhibit behavioral trichromacy (Tovée, Bowmaker, & Mollon, 1992). Our analysis shows that trichromacy may be the result of a learning process. For tetrachromacy, our results do not strictly exclude that learning of selectivity for four cone types in humans is possible, because the failure may be due to our specific method. But the negative results suggest that the reason for the lack of tetrachromacy in humans may lie in the difficulty to learn cone-selective wiring with an additional intermediate cone type, because the statistics of natural scenes result in cone responses that are too similar. Consequently, we expect that the heterozygous human carriers who show some signs of tetrachromacy (Jameson, Highnote, & Wasserman, 2001; Jordan & Mollon, 1993) have a fourth cone type with a larger difference in spectral sensitivity with respect to L and M cones than those who do not.

In conclusion, our results show that the higher order statistics of cone responses to natural scenes provide sufficient information for the learning of color-selective receptive fields. Investigating experimentally whether the visual system employs a learning strategy to achieve color vision and how it is realized remains a challenge for future research. It is conceivable that biologically realistic forms of plasticity could perform the same function as our algorithms. Thus, our results strongly support attempts to look for plasticity in the visual system as basis of color selectivity.