A tour of contemporary color vision research

Abstract

The study of color vision encompasses many disciplines, including art, biochemistry, biophysics, brain imaging, cognitive neuroscience, color preferences, colorimetry, computer modelling, design, electrophysiology, language and cognition, molecular genetics, neuroscience, physiological optics, psychophysics and physiological optics. Coupled with the elusive nature of the subjective experience of color, this wide range of disciplines makes the study of color as challenging as it is fascinating. This overview of the special issue Color: Cone Opponency and Beyond outlines the state of the science of color, and points to some of the many questions that remain to be answered in this exciting field.

The study of color is perhaps the oldest discipline of psychology. For the ancient Greeks, colors took fifth place after the classical elements of fire, air, water, and earth (see page 28 of Kuehni & Schwarz, 2008). Helmholtz’s observation that colors can be created by superposing three primaries (von Helmholtz, 1867), and Maxwell’s measurements of the proportions of primary lights (positive and negative) required to make the match (Maxwell, 1860), paved the way to our first scientific understanding of the psychological representation of color. Maxwell’s work led to an objective set of rules that govern color mixture (Brainard & Stockman, 2010). Eventually, with direct measurements of the three classes of cone photoreceptors in the normal eye (Bowmaker & Dartnall, 1980; Dartnall, Bowmaker, & Mollon, 1983; Schnapf, Kraft, & Baylor, 1987; Roorda & Williams, 1999), and with advances afforded by use of molecular genetics (Nathans, Thomas, & Hogness, 1986), the trichromatic theory of color (Young, 1802; von Helmholtz, 1852; Maxwell, 1855) was firmly established.

Much of the spectral information contained in the patterns of photons arriving at the eye is lost: the complex spectral power distribution entering the eye is reduced to a trichromatic code by the univariant responses of the three classes of cone photoreceptor (Mitchell & Rushton, 1971). This code, relayed by the relative responses in the three cone classes, is the retinal basis for color vision. We can now predict, with reasonable precision, how the three classes of cones react to any given physical stimulus. Yet many mysteries about how that code leads to the perception of color remain.

The trichromatic code is transformed at the first postreceptoral synapse between the photoreceptor and bipolar cells aided by lateral feedback (and perhaps feedforward) signals between cones from horizontal cells (Verweij, Hornstein, & Schnapf, 2003). It is at this and subsequent stages that cone-opponency—in which signals are compared across different cone types—comes into play. In this special issue, Segal & Perlman (2018) study the cone-opponent properties of horizontal cells in turtle retina. They show that the strength and specificity of horizontal cell feedback is increased by presence of the known neuromodulator nitric oxide, perhaps as part of a mechanism for sharpening spectral discrimination as light levels increase.

It is important to keep in mind that cone-opponency – subtractive interactions between cones of different types – is distinct from color-opponency, which relates to color appearance and the phenomenological opponency of colors first described systematically by Hering (1878, 1920) (and reviewed by Mollon & Jordan, 1997; Shevell & Martin, 2017). Cone-opponency can, in principle, occur without any color association (e.g., Stromeyer, Kronauer, Ryu, Chaparro, & Eskew, 1995; Stockman, Henning, Anwar, Starba, & Rider, 2018). The same caution applies to the nomenclature of the cones themselves. There has been a long tradition of applying color names to cone types (red, green, blue), but cones do not correspond to color percepts. For example, the sensitivity peak of the so-called red cone is not in the part of the spectrum we call red, but in the yellow-green. For these reasons, color scientists favor the cone names “L”, “M”, and “S”, referring to the location in the spectrum where each cone type has its peak (long, near 566 nm; middle, near 541 nm; and short, near 441 nm). The color categories red, green, and blue are the output of the visual system, not the input; relating these categorical responses to natural image statistics is considered by Milojevic, Ennis, Toscani, and Gegenfurtner (2018). They showed that observers mainly use the average color of a leaf photograph to assign it to a color category and use unique yellow as a decision boundary.

The two most important classes of opponent interactions in the retina occur between L-cones and M-cones, and between the S-cones and some combination of the L- and M-cones. The L-M signal is thought to be encoded by cone-opponent “midget” bipolar cells, which have chromatic-opponency in the fovea by virtue of having single L-cone or M-cone inputs in the centers of their receptive fields, but perhaps mixed L-cone and M-cone inhibitory surrounds derived from horizontal cells. Midget cells project to the parvocellular layers of the lateral geniculate nucleus (LGN) (reviewed by Solomon & Lennie, 2007; Lee, Martin, & Grünert, 2010). In this special issue, Lee, Cooper and Cao (2018) investigated the spatial structure of cone-opponent receptive fields in macaque parafoveal retina and report responses of midget ganglion cells to cone-isolating and red-green chromatic stimuli. Despite wide variation in the relative size of center and surround in midget cells, they found most cells show good functional segregation of L-cone and M-cone inputs (Verweij et al., 2003; Crook, Manookin, Packer, & Dacey, 2011).

The other main class of opponent interactions is between S cones and some combination of L and M; this opponency is thought to be encoded by S-cone bipolar cells and the distinctive bistratified “blue-yellow” ganglion cells (Kouyama & Marshak, 1992; Dacey & Packer, 2003) that project to the koniocellular layers of the LGN (reviewed by Hendry & Reid, 2000; Martin & Lee, 2014). Connections between S-cones and midget OFF bipolar cells (Field et al., 2010) may form the basis for a similar but oppositely-signed “yellow-blue” signal. In this issue, Eiber, Pietersen, Zeater, Solomon, and Martin (2018) study opponent interactions in blue-on and blue-off cells in marmoset LGN and show that both the timing and strength of opponent S and M/L cone inputs are balanced, making these cells highly selective for color over brightness changes. This result adds to other evidence that, unlike the midget-parvocellular system (which likely serves both high-acuity vision and red-green color vision), the bistratified-koniocellular system is dedicated to detecting color contrast.

How color is encoded in the cortex is still a major scientific question. Cataloguing the physiological properties of cortical cells has been an enormous challenge, in part because of the massive expansion in the number of neurons found at successive stages of visual processing. There are about one million retinal ganglion cells in each eye, and a comparable number of cells in each LGN, but over 100 million cells in each hemisphere in V1, the first cortical target of the LGN (Curcio & Allen, 1990; Leuba & Kraftsik, 1994). Across all studies, we have far fewer recordings of V1 color properties as a proportion of the population than we have in retina and LGN, and the fraction drops precipitously for subsequent cortical areas (V2, V3, V4, and inferior temporal cortex). It is tempting to bundle what little we know about the cortex into a tidy story. But given how little data there are, the story is likely to be wrong. With that caveat, we will attempt a summary of what we think we know about the physiology of color vision in the cortex, and then return to the challenge of linking physiology with psychology.

The first single-cell recordings in V1 came to the surprising conclusion that in contrast to the LGN, few neurons in V1 appeared to be overtly color tuned (Wiesel & Hubel, 1966; Hubel & Wiesel, 1968). There is now a consensus that the population of cortical neurons in V1 (and almost every other cortical area studied) have tunings that span a broad range of color directions (reviewed by Gegenfurtner, 2003; Solomon & Lennie, 2007). The meaning of this multiplicity of tunings with regards to color vision is not at all clear and may not even be relevant to color perception (Eskew, 2009). A cortical neuron that receives equal input from L, M, and S cones will respond best to achromatic modulations; any degree of imbalance of those three inputs will cause the cell to respond best to some color direction other than the achromatic one. Consequently, the mere fact that, across the population, individual cortical cells are tuned to many color directions may simply reflect the near-impossibility of precisely regulating the relative strengths and signs of the cone inputs from retina to LGN to cortex. Meanwhile, among the population of V1 cells is a small subset of neurons that show striking cone-opponency. These cells were originally overlooked because the fraction seemed too small to be relevant (Lennie, Krauskopf, & Sclar, 1990). But subsequent work has pinned down the spatial and temporal structure of the cone inputs of these cells and shown many of them to have the elusive “double-opponent” structure first proposed by Edwin Land in his color constancy models (Conway, 2001). Others of the cone-opponent cells in V1 appear to be “single-opponent”, with simpler, and smaller receptive fields that look much like parvocellular cells (Johnson, Hawken, & Shapley, 2004; Shapley & Hawken, 2011). Thus there appear to be a range of color-responsive neurons in V1. This diversity is now widely acknowledged, and improvements in theoretical and empirical methods of assessing color tuning, as described in this issue by Weller and Horwitz (2018), are helping bridge the gap between physiology and functional role. The diversity of color responses at various stages of the visual cortex suggests that color plays a key role in many aspects of visual behavior. The importance of color to visual computations is curiously recovered by an analysis of the layers in convolutional networks, which shows surprising richness of color-tuned “neurons”, and provides clues for how color information may be used at different stages of cortical processing (Rafegasa & Vanrell, 2018).

Because of technical limitations, a given study can only measure single-cell responses for a tiny fraction of the neurons in a given area. The challenge has been how to relate the findings across studies. The anatomical boundary that separates V1 from the rest of the cerebral cortex is clear to the naked eye, which greatly promoted the study of V1. By contrast, physiological studies outside of V1 were long hampered from a lack of anatomical or functional markers to demarcate the areas (gross anatomical landmarks are not very useful because they vary across individuals). Headway was made following the serendipitous discovery of subcompartments within V1 and V2 using a stain for the enzyme cytochrome oxidase (Wong-Riley, 1979). In V1, the stain uncovers blobs (or patches), each about 200 μm across (Horton & Hubel, 1981), and in V2 it uncovers a system of thin and thick stripes. Color-tuned neurons are often located within cytochrome oxidase blobs of V1 (Livingstone & Hubel, 1984), and the thin stripes of V2 (summarized in Gegenfurtner, 2003). But the colocalization is not precise: many color-tuned neurons are found outside cytochrome-oxidase regions, and many cells within cytochrome-oxidase regions are not obviously color tuned (Leventhal, Thompson, Liu, Zhou, & Ault, 1995). The cartoon that color is coded by blobs and thin stripes is clearly an over-simplification that might be misleading. For example, there is a strong projection linking blob cells with area MT (Boyd & Casagrande, 1999); MT is a region that is universally recognized as specialized for motion processing, with limited role in color. And cats have clear cytochrome oxidase blobs (Boyd & Matsubara, 1996), yet poor color vision. Cytochrome oxidase blobs might be more associated with computations of surfaces, rather than color specifically.

Pockets of color-tuned neurons outside of V1 and V2 were discovered first by Zeki (1973) in V4, and then by Komatsu (1998) in inferior temporal cortex (IT), a very large swath of tissue implicated in high-level object vision. However, progress was again hampered by the lack of consistent functional or anatomical markers, and the tendency to extrapolate the function of an entire brain region based on recordings from a very tiny fraction of the population of cells housed there. The debate about the role of V4 in color is especially instructive: following Zeki’s initial work on V4, other studies failed to find clear evidence that V4 was any more involved in color processing than V1 or V2 (Schein & Desimone, 1990). As we describe below, this controversy was mostly resolved by experiments that combine fMRI and micro-electrode recording.

The advent of fMRI, and its application in non-human primates as a tool for guiding micro-electrode recording, has revolutionized the study of color circuits outside of V1 and V2. The landmarks provided by fMRI relate directly to the computation under study (color), unlike histo-chemical markers such as cytochrome-oxidase. Moreover, the combination of fMRI and micro-electrode recording bridges the gap between gross cortical organization and microscopic cellular resolution. Results using this approach reveal that area V4 has a mesoscale organization comprising millimeter-sized color-biased regions, dubbed globs, which are massively enriched for color-tuned neurons, interspersed by inter-glob regions that show much lower rates of color tuning (Conway, Moeller, & Tsao, 2007). These observations have been confirmed by optical-imaging-guided microelectrode recording (Tanigawa, Lu, & Roe, 2010). In retrospect, it seems likely that Zeki happened to target globs, while the later studies targeted inter-globs. FMRI also shows at least four large color-biased domains within inferior temporal cortex (Lafer-Sousa & Conway, 2013), one of which corresponds to the region discovered by Komatsu. These and other fMRI studies of color provide insight into the fundamental organizing principles of inferior temporal cortex, and also present clues to the neural basis for the role of color in visual cognition (Bannert & Bartels, 2017). We are now at the threshold of making questions of color cognition scientifically tractable at the circuit level (Koida & Komatsu, 2007).

A central question in color vision research is how different types of psychophysical color measurements map onto different stages of the neural color processing stream. This is a difficult question to answer, since psychophysical responses typically represent the output of the entire neural system. Nevertheless, some psychophysical measurements can be reasonably linked to particular stages of processing. For example, color matching and the laws of color matching (e.g., Krantz, 1975) can be straightforwardly linked to the univariant properties and spectral sensitivities of the three cone types and color matching functions should be a linear combination of the three cone spectral sensitivities (König & Dieterici, 1886; Brindley, 1960). Moreover, color discrimination experiments can, in principle, be related to the cone-opponent stages. For example, under some conditions an incremental L-cone light becomes more detectable if combined with a decremental M-cone light (Stromeyer, Cole, & Kronauer, 1985; Cole, Stromeyer, & Kronauer, 1990), which implies a cone-opponent coding. Moreover, the spectral sensitivities of these detection mechanisms are consistent with cone-opponency (i.e., L-M) rather than red-green color opponency (Mollon & Cavonius, 1987). The simple picture of L-M, S-(L+M) and L +M as the main second-stage mechanisms (Krauskopf, Williams, & Heeley, 1982) has become more complicated by the identification of additional psychophysical mechanisms (Krauskopf, Williams, Mandler, & Brown, 1986; Stoughton, Lafer-Sousa, Gagin, & Conway, 2012; Hansen & Gegenfurtner, 2013; Shepard, Swanson, McCarthy, & Eskew, 2016). Nevertheless, in one recent paper, Shepard, Lahlaf & Eskew (2017) were able to link six distinct, categorical color percepts to specific detection mechanisms.

Although two different stimuli that cause the same triplet of activation in the three classes of cones will appear the same in whatever context they are presented, we cannot predict the color appearance of that triplet, even though we know a fair amount about some of the intermediate stages between cone quantal catch and ultimate appearance. The transformation of cone to cone-opponent signals in the retina was foreshadowed by Hering (1878, 1920), who noted that color pairs such as red versus green, and blue versus yellow are phenomenologically opposed in perception (there are no reddish greens). He postulated that color is represented by the responses of three different mechanisms each driven by paired color-opponent processes (red/green, blue/yellow and white/black). Hurvich and Jameson (1955) elaborated this observation using a psychophysical procedure known as hue cancellation, from which they inferred spectral sensitivities of red-green and blue-yellow color-opponent mechanisms. However, the spectral sensitivities of these “valence” mechanisms, as they called them, are inconsistent with physiological cone-opponent measurements in the retina and LGN, and with the color discrimination measurements described above (e.g., Guth, 1991; De Valois & De Valois, 1993; Webster, Miyahara, Malkoc, & Raker, 2000; Stockman & Brainard, 2010; Shevell & Martin, 2017). This inconsistency is also uncovered by measurements of the unique red, green, yellow and blue colors, which do not appear to relate in a direct way to cone-opponent mechanisms (Wuerger, Atkinson, & Cropper, 2005; Wool, et al., 2015). An important property of the color-opponent mechanisms (as opposed to cone-opponent detection mechanisms) is that even the red-green mechanism has a pronounced S-cone input. Another important property is that – despite the simple textbook picture – the two blue and yellow ‘poles’ of the blue-yellow color-opponent mechanisms do not receive symmetric input from the cones (Wuerger et al., 2005). Thus the loci of unique red and green (where the blue-yellow response is zero) are not colinear in color space (Dimmick & Hubbard, 1939b, 1939a; Burns, Elsner, Pokorny, & Smith, 1984; Wool et al., 2015). Witzel and Gegenfurtner (2018) investigated categorical perception for unique hues in order to test for the relationship between color appearance, color discrimination, and low-level (second-stage) mechanisms. Their findings underscore the point that the four chromatic unique hues cannot be related in a simple way to cone-opponent stages of processing. In terms of psychophysical detection mechanisms, the S+ and S− mechanisms have different contrast response properties as demonstrated in this issue by Gabree, Shepard and Eskew (2018).

A great deal is still uncertain about other aspects of color appearance. Although we have a conviction that colors are ordered, we have little understanding of the rules that govern the geometry of the space that represents color appearance. Colors can be defined by lightness (value), color purity (saturation), and hue, but these properties interact. There is still no clear empirical or theoretical way of relating these dimensions to each other. To address this fundamental question, Schiller, Valsecchi and Gegenfurtner (2018) evaluated different measures of color saturation. CIELUV, CIELAB, and CIECAM02 provided the best metrics of perception. Many other aspects of color remain unexplained. One of the most pressing questions concerns the relationship of color and shape processing. This question has been addressed by evaluating how luminance edges influence color perception, such as in the watercolor effect, where colors appear to spread from luminance boundaries into nearby regions (Pinna, Brelstaff, & Spillmann, 2001). Coia and Crognale (2018) show that contour adaptation reduces the spreading of edge induced colors, which provides clues to the neural mechanisms that integrate color and shape information.

Although we can measure neural responses to modulations in cone activity, we do not know what stimuli to use to relate the neural activity to computational mechanisms (Sanada, Namima, & Komatsu, 2016). There is even uncertainty at Marr-Level 1, regarding the computational goal of color: why do we have color vision at all? Some evidence suggests that color may aid in storing and remembering objects (Gegenfurtner & Rieger, 2000), although to a broad first-order approximation color provides little benefit for object recognition. Indeed, most studies of object vision use achromatic images. Perhaps the role of color extends beyond object recognition. Science has largely overlooked one of the most important facts of all about color: color is attractive. Our desire for color has occasionally fueled wars and continues to drive technological developments such as those for high-gamut displays and LEDs. Color preferences are well documented, but their origin remains a mystery. In order to address this broader question, Schloss, Lessard, Racey, and Hurlbert (2018) modelled color preferences using a variety of color space metrics. They found that the best model was one specified in CIELAB space that included 1st and 2nd harmonics of hue. These results are interesting, in part because they relate to quantitative analysis of color-tuning responses at mid-tier stages of visual processing. This analysis shows that the V4 Complex encodes a uniform representation of color space best captured by CIELAB, providing a link between physiology and perception (Bohon, Hermann, Hansen, & Conway, 2016), and now possibly preferences.

A plethora of other colorful topics graces this issue. Rucker, Henriksen, Yanase, and Taylor (2018) studied the role of temporal contrast and blue light on emmetropization (the critical period of eye growth that results in a focused retinal image). Dore, Dumani, Wyatt, and Shepherd (2018), in a multi-faceted investigation, examine associations between global and local shape perception, colored backgrounds, color discrimination, and non-verbal IQ. Koenderink, van Doorn and Gegenfurtner (2018) investigated color weights in photometry by going beyond the simple heterochromatic methods, the results of which comply with the CIE luminance, to tasks that involve mid-level and high-level aspects of perception. These studies all help fill in gaps about the psychology of color, and they will guide physiological investigations aimed at understanding the neural mechanisms that support color behavior.

One theme that emerges from the collection of the papers in this special issue is that studying color is not easy. Why is it so challenging to unlock color’s secrets? One reason is that color is a ‘proper sensible’ (Aristotle, 350 B.C.E.), meaning that we only have access to color through our eyes. We can learn something about color through language, and blind people can acquire a rather sophisticated knowledge of color relationships by listening to others describe color (Shepard & Cooper, 1992). However, unlike object shape, which can be known by both touch and sight, color can only be experienced through vision. This aspect of color introduces a challenge because at a fundamental level color is a private experience. The subjective nature of color can hinder scientific inquiry, but it adds much to its allure. And through the study of color we hope to glean insight into fundamental aspects of human experience.