The Verriest Lecture: Adventures in blue and yellow

Abstract

Conventional models of color vision assume that blue and yellow (along with red and green) are the fundamental building blocks of color appearance, yet how these hues are represented in the brain and whether and why they might be special are questions that remain shrouded in mystery. Many studies have explored the visual encoding of color categories, from the statistics of the environment to neural processing to perceptual experience. Blue and yellow are tied to salient features of the natural color world, and these features have likely shaped several important aspects of color vision. However, it remains less certain that these dimensions are encoded as primary or “unique” in the visual representation of color. There are also striking differences between blue and yellow percepts that may reflect high-level inferences about the world, and specifically about the colors of light and surfaces. Moreover, while the stimuli labeled as blue or yellow or other basic categories show a remarkable degree of constancy within the observer, they all vary independently of each other across observers. This pattern of variation again suggests that blue and yellow and red and green are not a primary or unitary dimension of color appearance, and instead suggest a representation in which different hues reflect qualitatively different categories rather than quantitative differences within an underlying low-dimensional “color space.”

1. Introduction

Modern color science remains strongly rooted in two foundational theories of color vision. The trichromatic theory of Young and Helmholtz provides an account of the initial absorption of light by the three classes of cone photoreceptors, and goes a long way in explaining how the visual system samples the spectrum and which spectra can be distinguished [1]. Hering’s theory of opponent processes instead focused on color appearance, and has been used to explain how signals from the cones are combined within postreceptoral mechanisms to represent the perceptual experience of color in terms of underlying red-green and blue-yellow dimensions [2]. However, as we learn increasingly more about the intricacies of the visual system both theories face challenges [3]. The genetics of the cone opsins have revealed much richer variation in the photopigments than expected, and have raised exciting new questions about how different gene variants are expressed and how these relate to the behavioral capacities and perceptions of the observer [4, 5]. An example is the ongoing research on the potential for female tetrachromacy [6, 7]. Nevertheless, the principle of univariance and the properties of color matching inspired by trichromatic theory remain solid cornerstones of color science and colorimetry [8, 9].

The opponent process theory has not fared as well. The representation of chromatic information by contrasting signals (e.g. from the cones) is a fundamental insight that has seen clear and repeated confirmation in psychophysics and physiology. Yet the specific comparisons that are made, how many are used, and how they give rise to our conscious experience of color are increasingly shrouded in mystery. In particular there is now little consensus over the nature of the Hering primaries of red-green and blue-yellow, or whether these are primary at all. For example, apart from their presumed special appearance, the unique hues do not clearly behave as if they are unique or special in many tasks (e.g. [10–14]), though there are tantalizing exceptions [15]. This debate has had the refreshing consequence of reinvigorating color science, and has given rise to many intriguing new ideas and theories about how the brain builds a representation of color. This essay is not intended to be a comprehensive review of these theories, and instead narrowly focuses on my own ideas and how they have evolved from a career chasing the elusive shades of blue and yellow.

2. Blue and yellow in the brain

Hering’s theory held that the perception of any color depended on the responses in three opponent processes signaling red vs. green, blue vs. yellow, or bright vs. dark sensations [2]. For the two chromatic processes the opponent pairs were mutually exclusive, so that a color could not be red and green at the same time. Stimuli that isolated one of the processes should give rise to the pure undiluted sensation of one of the primaries. These percepts correspond to the unique hues of red, green, blue, and yellow. For example, a unique yellow is a yellow that has neither a reddish nor greenish tinge. All other hues correspond to different combinations of the unique hues. For example, orange reflects the combined attributes of red and yellow, while purple is a combination of red and blue. A critical assumption was thus that some hues are unique because they reflect unique or special states in the brain. Measurements of the strength of red-green and blue-yellow responses at different wavelengths could map out the spectral sensitivities of the opponent chromatic response functions [16]. These could in turn be fit with the cone fundamentals to estimate how the cone signals are combined to form the opponent dimensions [17].

Color-opponent theory was revived by the classic psychophysical studies of Jameson and and Hurvich [16], but gained solid grounding when a plausible mechanism for generating an opponent-like representation was identified in physiological recordings from neurons in the fish and primate visual system [18, 19]. These showed that cells were excited by lights at some wavelengths but inhibited at others, and thus had precisely the responses that could in theory encode mutually-exclusive responses within a single opponent mechanism. However, these studies also noted important inconsistencies between the physiology and perception. In an early study comparing cell responses in the primate lateral geniculate to the color percepts reported by humans, De Valois, Jacobs, and Abramov [20] pointed to differences at wavelengths shorter than unique blue, which appear violet (i.e. reddish-blue). However, the recorded cell population did not exhibit the expected change in response polarity at these wavelengths, predicting that the shorter wavelengths should instead appear greenish-blue.

This discrepancy was brought to the fore by the seminal studies by Krauskopf and colleagues. Krauskopf, Williams, and Heeley [21] used a habituation or contrast adaptation paradigm to identify the chromatic mechanisms mediating threshold color changes and their potential cone inputs. This revealed the “cardinal mechanisms” of color opponency and showed that they corresponded to two channels comparing signals in the two longer-wavelength cones (LvsM) or to the short-wave cones compared to the longer-wave cones (SvsLM). In subsequent analyses of the LGN responses to chromatic stimuli, Derrington, Krauskopf, and Lennie [22] further showed that these were the cone inputs to the two main classes of color-opponent cells, which correspond to cells in the parvocellular (LvsM) and, as discovered later [23], the koniocellular (SvsLM) layers of the LGN, and to the midget and bistratified ganglion cell classes of the retina [24]. These studies, along with the chromaticity diagram developed earlier by MacLeod and Boynton [25] which also defined stimuli in terms of the cardinal axes, had an enormous impact on the field. It is now routine to design experiments and interpret results in terms of the cardinal or cone-opponent axes, with the advantage that these can be more directly related to the cone signals and the principal color pathways at early postreceptoral stages.

Like the work of De Valois et al. [20], one of the key findings of Krauskopf et al. [21] was that the cardinal axes are not aligned with the perceptual-opponent axes. This is illustrated in Figure 1, which plots the stimuli chosen for the unique hues or binary hues (e.g. orange or purple) for a number of color-normal observers [10] (see also [26, 27]). The SvsLM axis varies from purple to yellow-green, while the LvsM axis from red to blue-green. In contrast, unique yellow and blue lie roughly midway between the cardinal axes, and thus are not stimuli that isolate either axis. It is unfortunate that despite this, it is common in the field to refer to the LvsM and SvsLM axes as red-green and blue-yellow, just as it was once common to refer to the LMS cones as RGB. As Figure 1 shows, the colors along the SvsLM axis are in fact as red-green as they are blue-yellow, while the +M pole of the LvsM axis is close to an equal mixture of blue and green. Three of the four cardinal axis poles therefore correspond roughly to the categorical boundaries between adjacent unique hues [10]. Recent analyses have found that these axes also represent the categorical boundaries for color discrimination in infants, who might thus base their discriminations on the relative polarity of the cardinal axis mechanisms [28, 29]. However in adults the interobserver differences in the unique hues substantially exceed the range of variability in the stimuli that isolate the cardinal axes [27, 30].

Figure 1.

Color categories relative to the cardinal axes. Symbols show the stimuli chosen as unique (Red, Green, Blue, Yellow) and binary (Purple, Orange, Yellow-Green, Blue-Green) hues by individual color-normal observers, plotted by their angle in the LvsM and SvsLM space (after [10]).

The mismatch between the stimuli that appear perceptually pure and the stimuli that align with the cone-opponent pathways in the retina and geniculate, prompted questions about the stages of the visual system where the unique hues become more explicitly represented. One possibility is that the corresponding substrate arises further along the visual stream, because of further recombinations of the cone-opponent signals, and models of the potential transformations were subsequently proposed [31]. A number of attempts have also been made to identify downstream neural signatures of the unique hues or the perceptual organization of color [32–34]. An alternative theory is that the cardinal mechanisms reflect pathways involved in other functions of color vision, such as scene segmentation, and that color percepts are carried by a different neural circuit also arising in the retina but potentially overlooked because this pathway is smaller [35]. Several lines of evidence have been put forth in support of this model [36] including the remarkable recent achievement of stimulating single photoreceptors with adaptive optics [37]. These studies suggest that the majority of L and M cones signal achromatic percepts, with only a small proportion producing red or green sensations.

However, further work by Krauskopf et al. [38] pointed to a second fundamental problem with the Hering theory. This study used a variety of techniques to demonstrate the presence of additional “higher-order” color mechanisms tuned to directions intermediate to the cardinal axes. Webster and Mollon [39, 40] subsequently applied a similar color contrast adaptation paradigm to examine suprathreshold color appearance. Adapting to a given color axis strongly altered the perceived hue and saturation of a subsequent test stimulus, and these aftereffects were selective for any arbitrary angle in color space, implicating mechanisms tuned to these directions that contribute to color percepts (Figure 2). This could conceivably reflect tuning changes in two mechanisms rather than gain changes in multiple mechanisms [41, 42]. However paralleling these psychophysical results, recordings of chromatic sensitivity in cells at different cortical levels reveal a much broader range of preferred color directions, suggesting that the relatively discrete representation in the LGN gives way to a more uniform tiling of color space in the cortex [43, 44]. Moreover, there is abundant evidence for higher-order color mechanisms mediating visual performance in a variety of behavioral tasks that do not involve adaptation [45], though the extent to which they are manifest for specific tasks remains debated [46].

Figure 2.

Multiple chromatic channels. Asymmetric color matches (circles and triangles) to the same set of test stimuli (squares) after adaptation to chromatic modulations along different axes within the LvsM and SvsLM plane. The adaptation leads to selective losses in perceived contrast (saturation) along the adapting axis, and biases perceived hues away from the adapt axis and toward an orthogonal axis (after [39]).

The presence of these additional opponent mechanisms is difficult to reconcile with the tenet that unique hues look unique because they are the responses of the isolated opponent processes, because if the chromatic plane is represented by multiple mechanisms tuned to different color angles but with overlapping sensitivity, then there is no stimulus direction that can isolate a single mechanism [39, 40]. Instead, hue may be represented in ways analogous to spatial orientation, by the population responses of channels tuned to different directions around the (color) clock [40, 47, 48]. Such models have recently been proposed to account for color coding at higher cortical stages [49]. There are not clear signs that a discrete two-channel representation re-emerges at higher stages, and instead the chromatic tuning of cells becomes substantially narrower, further challenging the notion that a single mechanism encodes the “redness” or “greenness” of hues across different stimulus angles [43, 49]. This multiple-channel representation could yet be reconciled with opponent process theory if, for example, the representation reflected other aspects of color vision such as its role in spatial vision. However, the adaptation effects again suggest that color appearance is mediated by multiple and potentially a continuous distribution of channels, at least at the visual level affected by the adaptation.

3. Discounting the observer

Yet another obstacle in seeking an explanation for the unique hues solely from the structure of color coding in the brain is that the visual system varies widely from one individual to the next. Even in observers with normal color vision, there are large differences in spectral and neural sensitivity [50, 51]. Without correcting for these differences, very different stimuli would be required to produce the same neural states in different observers. This is also a problem within the observer, because sensitivity varies markedly over time (e.g. as we age) and over space (e.g. at different retinal locations). However, a number of mechanisms act to compensate for the optical and neural idiosyncrasies of the observer in order to maintain stable color percepts, or color constancy. Constancy is typically studied in the context of discounting the illuminant in color perception [52]. However, an equally important aspect of color constancy is discounting the observer, and this occurs for many aspects of color vision.

Compensating for white.

Variations in spectral sensitivity arise at the earliest stages of vision, from differences in the density of the lens and macular screening pigments, the optical density and spectral peaks of the cones, and in the relative number of different cone types [50, 51]. These differences have important effects on threshold sensitivity and color matching, but surprisingly little effect on color appearance. For example, white settings remain very similar in the fovea and near periphery despite the fact that the foveal cones are screened by macular pigment [53, 54]. Similarly, the lens of the eye progressively yellows with age, limiting the shorter wavelength light reaching the receptors. Yet achromatic settings again remain very stable with aging, so that what appears white is continuously adjusted to discount the sensitivity change [55, 56]. Delahunt et al. [57] tracked these adjustments in patients before and after cataract surgery. When the cataractous lens is first replaced, the world appears very blue and bright. Yet over time the stimulus that appeared white approached the settings before surgery, so that they tended to see the world in the same way yet through their different eyes. By assessing this change on a dark background (in order to remove an influence of the immediate context), the study revealed that these adjustments developed over surprisingly long timescales of weeks or months. Long-term adjustments in color appearance have also been found in a number of other studies [58].

Equating the perception of white at two retinal loci could occur at many stages in the visual system, from a peripheral sensitivity calibration to a high-level learned inference that both loci should be representing the same world [53, 59]. To explore the site of the calibration, Webster and Leonard [53] compared achromatic settings in the fovea and periphery after short-term chromatic adaptation, to identify the stimulus that did not produce a color aftereffect. Chromatic adaptation adjusts sensitivity at a very early retinal level and potentially within the photoreceptors, because the gain changes are cone-specific [60] and have a spatial resolution comparable to single receptors [61] (though the afterimages have been identified with the rebound signals generated in the ganglion cells [62]). If the long-term normalization for white in the fovea and periphery also occurred at this early level, then the short-term chromatic adaptation at the two loci should be the same for the same external stimulus, but for different retinal spectra because of the differences in macular screening (i.e. the stimulus hat does not produce a color aftereffect should be the same stimulus that is perceived as achromatic). Conversely, if the normalization depended on processes later in the visual pathway, then chromatic adaptation at the two loci should be similar when the cone excitations are similar, but the stimuli differ (i.e. later compensation predicts that the neutral stimulus for chromatic adaptation may not “look” neutral). Chromatic adaptation at the two sites was indistinguishable – and for both the stimulus that was perceived as achromatic (the “perceptual norm”) was close to the adapting stimulus that did not induce an aftereffect (the “sensitivity norm”) [53] (Figure 3). Thus these results suggest that the intrinsic discounting for spectral sensitivity differences across the retina may already be present as early as the receptors, and may represent a long-term adaptation or memory for the average spectrum the cone is exposed to. The short-term fluctuations in sensitivity with light or chromatic adaptation as conventionally measured and defined thus appear to ride on top of this long-term equilibrium.

Figure 3.

Perceptual norms vs. sensitivity norms. Plots show chromatic adaptation in the fovea (circles) and 8-deg periphery (triangles). Shifts in the achromatic settings are plotted as a function of the chromaticity of the adapting stimulus along the LvsM (left) or SvsLM (right) axis. Aftereffects are similar at the two loci for the same adapting chromaticity, with a null in the aftereffect close to the stimulus level that was judged as achromatic under neutral (dark) adaptation (dashed lines labeled “percep norm”). Arrows indicate the adapting stimuli that did not alter the neutral achromatic settings (“resp norm”). Lines labeled “no compensation” show the vertical difference in the adapting curves at the two loci that would be predicted if the adaptation were instead controlled by equivalent spectra at the level of the receptors, after filtering by the macular pigment (based on estimated differences in macular pigment screening at the two loci) (after [53]).

There are two further implications of this result. First, the achromatic point is part of the family of unique hues and perhaps the most important one, for it is the equilibrium point for both opponent processes. That this percept (as measured under neutral or dark adaptation) also corresponds to the equilibrium point for chromatic adaptation reveals a close connection between “perceptual norms” – the stimuli that are experienced as neutral; and “sensitivity norms” – the stimuli that produced balanced or neutral responses in the visual system. Unlike the other unique hues, white is therefore a case where we arguably can link a special color experience to a special state in the underlying neural code [53]. The second implication is that norms are not unique to color but are common and central to the norm-based codes thought to underlie the representation of many perceptual dimensions [63]. For example, common models of face perception assume an individual is represented by how they deviate from a neutral or prototypical identity [64], analogous to the role of “gray” in color coding. Thus these models closely parallel models of color appearance, and notably face and color perception are affected by adaptation in very similar ways [65]. Similarly, in blur aftereffects, the adapting stimulus that does not bias subjective focus is the same stimulus that appears in focus to the observer, suggesting that the perceived focus reflects how spatial vision is normalized to discount the retinal image blur specific to the individual’s optical aberrations [66, 67].

Compensating for hue.

Settings for unique hues also remain stable despite changes in spectral sensitivity. For example, unique yellow is largely unaffected by the large differences in the ratio of L and M cones [68–70]. Hue percepts also remain similar between the fovea and near periphery; and while not complete, the constancy is better than predicted by adjustments only to the average color (von Kries adaptation [71]), implicating an additional correction [72, 73]. The loci of the unique hues also remain stable across the lifespan [56, 74], and again better than would be expected from adjusting only to the changes in mean spectral sensitivity [72]. These adjustments have their limits, because color in the periphery varies in ways that can be tied to the variations in retinal physiology [75, 76], and the hue percepts of color-anomalous observers show only partial compensation for their anomalous pigments [77]. However, the variations in appearance are generally less than physiological constraints alone would suggest.

A further surprising compensation is for the effects of saturation on hue. We were led to examine this question by the suggestion of Mollon and Jordan [78] that there should be greater consensus in the unique hues for natural broadband stimuli than narrowband, since the compensations observers learn for the natural stimuli should cause the hues seen for narrowband stimuli to diverge. Mizokami et al. [79] tested this by matching the hues of Gaussian spectra with narrow or broad bandwidths. The spectral filtering of the eye should bias the cone excitations for broader spectra, requiring a shift in the stimulus peak to match the cone excitations for the narrow spectra. Yet constant hues instead corresponded to constant peaks, and thus different relative cone responses, for the different bandwidths (Figure 4). This again suggests that the visual system ties constant hues to a constant property of the physical stimulus (e.g. the estimated peak of the spectrum) rather than to constant properties of the neural response. (In a related vein, Knoblauch and Shevell showed that a single cone class need not signal a fixed hue [80].) The Gaussian prediction is only approximate, and in particular poorly fits the hue matches at longer wavelengths [81]. However, it provides a novel perspective on a classic color phenomenon – the Abney Effect – in which hue does change as saturation is varied [82]. The difference is that in this case the desaturation is produced by adding a fixed background stimulus, rather than varying the stimulus bandwidth. The Abney effect has been attributed to a nonlinearity in color mechanisms, without a functional benefit. However, the observed hue shifts roughly follow the prediction that the visual system uses the limited information from the cones to approximate the spectrum as a Gaussian and compensate for this – and in this sense applies the right correction for the wrong stimulus. Mizokami and Webster [83] also showed that this is a reasonable inference for the visual system to embody, because a Gaussian fits natural spectra as well as conventional linear models with the same number of parameters.

Figure 4.

Compensation of hue for saturation. Left: Because of filtering by the spectral sensitivity of the eye, as the bandwidth of a Gaussian spectrum is increased, the peak wavelength of the spectrum must be shifted to maintain the same relative cone excitations (dashed lines). However, the wavelength for unique blue settings for different observers remains constant for the narrow (filled symbol) or broad bandwidth (unfilled symbols). Right: Desaturating a narrowband light by instead adding a fixed background spectrum leads to a fixed filtering pattern by the visual system. In this case (mis)applying the Gaussian correction requires a shift in the dominant wavelength (unfilled lines) that roughly predicts the curved hue loci in the Abney Effect (filled dashed lines) (after [79]; Abney Effect data from [82]).

Compensating for contrast.

A final example of compensating for sensitivity limits is in the perception of color contrast. Many authors have noted that because of the overlapping sensitivities of the cones, the differences (which define chromatic contrast) are much smaller than the sums (which define luminance). Yet sensitivity to chromatic contrast is correspondingly higher, offsetting this physiological difference [84]. Moreover, the cone contrasts at which luminance and color or different chromatic axes appear equally strong are roughly consistent with the contrasts from natural scenes, suggesting that the visual system tends to balance or “sphere” the responses for natural color gamuts [85]. An intriguing extension of these ideas is to the color percepts of anomalous trichromats. These individuals retain three cone types but with a smaller separation in the peaks of their longer-wave cones (with the anomalous pigment shifted toward the peak of the normal L or M cone). The difference signals from their longwave pairs are consequently much smaller, yet in principle these observers could also amplify postreceptoral responses to compensate for the loss [86]. Studies of color salience in fact suggest that at least some anomalous observers experience reddish-greenish differences as more similar to normal than their weakened cone contrasts predict [87, 88].

4. Blue and yellow in the world

An alternative to tying the unique hues to a unique state in the brain is that they might reflect unique properties of the environment. For example, on average the stimuli that appear pure blue and yellow lie close to the daylight locus [89]. Thus these hues may appear special because they correspond to salient and potentially learned characteristics of the world, regardless of how they are represented in the neural architecture (though as discussed below, there are large individual differences in the blue and yellow loci which are difficult to attribute simply to differences in the observers’ light environments). Similarly, red has been associated with ripe fruit [90] or blood [91]. In this case the percept tends to align with the LvsM axis and is thus a unique hue that does have a potential unique neural signature, and one which has been implicated in the evolution of LvsM dimension of trichromacy [90–92]. However as noted above variability in unique red also exceeds the inter-observer variability in the angle of the LvsM axis. The unique hues have also been tied to special properties in the reflectance functions of natural spectra as sampled by the cones [93, 94], though this idea has been challenged [95]. Finally, cross-cultural studies continue to explore the role that communication may play in shaping color categories [96–100].

Compensating color perception for the sensitivity limits of the observer also emphasizes that the salient properties of color perception are set by properties of the world. One mechanism mediating these adjustments is adaptation, which regulates sensitivity according to the stimuli we are exposed to. Adaptation is a fundamental process in all sensory systems and adjusts to many of the stimulus attributes we experience [63]. Adaptation to color involves a number of distinct mechanisms, but two prominent adjustments are adaptation to the average of the stimulus (primarily retinal) and to the variance or contrast in the stimulus (primarily cortical) [101, 102]. The latter are the adaptation effects probed by Krauskopf et al. [21].

Based on these mechanisms, white or gray may simply correspond to the average spectral stimulus we are exposed to, since light adaptation in the photoreceptors will over time equate their sensitivity for the average [103]. Unique yellow has also been accounted for by an adaptation to the average spectrum that equates the relative responses in the L and M cones [77, 104]. However, what constitutes an average in both cases is unclear. Natural images can be roughly partitioned between earth and sky. Terrestrial colors are yellowish and the achromatic average thus requires some contribution from the sky, yet we presumably spend more time looking at the ground [105–107]. If adaptation is local, then this can account for the compensation for sensitivity variations across the visual field, but only if different retinal locations are exposed to the same average stimulus. It is instead possible that the upper and lower visual fields are adapted somewhat differently because of the differences in their spectral diet [105].

Adaptation to contrast can similarly explain how the visual system might compensate for the sensitivity differences for luminance and chromatic contrast. If chromatic signals are inherently weaker, then the gain of chromatic channels may adjust to offset this [107]. Again the same principle predicts that color-anomalous observers might further amplify the weakened chromatic contrasts afforded by their cones [86]. An additional sign of the general effects of adaptation on color appearance is the bias in visual sensitivity for blue and yellow (Figure 5). The cardinal axes have been postulated to represent an efficient representation of chromatic information, on the assumption that they are aligned with the principal axes of variation in natural color distributions. While this is approximately true for scenes dominated by lush foliage [108], most natural color gamuts tend to have wider variation along bluish-yellowish axes [109], a bias that is also present in paintings [110, 111]. For these distributions, color coding in the retina and geniculate therefore seems surprisingly inefficient, because the signals carried by the LvsM and SvsLM pathways are often strongly correlated [109]. As noted natural daylight also varies along the blue-yellow axis. Thus the visual system may be habitually exposed to stronger contrasts along the negative blue-yellow axis of the cardinal-axis space than for other color directions, predicting larger sensitivity losses for this axis [109]. This stimulus bias shows up as reduced sensitivity for blue-yellow contrasts in a variety of tasks [112–114]. Notably, it is also a common feature of uniform color systems or color spaces. That is, in these spaces stronger bluish-yellowish stimulus contrasts are required to produce perceptually equivalent changes [115]. The close correspondences between color in the environment and color coding – mediated by adaptation – suggest that uniform color spaces could in fact be developed by starting with natural image statistics and then asking how the visual system would adjust to these. Smet et al. [116] developed a model of color vision based on these principles. The model predicts perceptual color differences nearly as well as current uniform color metrics which are instead derived by fitting empirical measurements of color discrimination.

Figure 5.

Blue-yellow bias in the world and perception. Left: The color gamuts of natural scenes tend to show greater variation along the bluish-yellowish axis of the LvsM and SvsLM space. Right: Samples from the Munsell Color Palette projected into the cardinal axis space show that equal perceptual steps require larger chromatic contrast changes along the bluish-yellowish-orange axes (after [117]).

An important consequence of these adaptation effects is that they adjust the visual system to match the world, and this again works to discount the properties of the observer. Simple mechanisms like this can thus allow our color percepts to remain stable despite spatial or temporal variations within the individual. At the same time, they should tend to equate some aspects of color experience across observers. That is – old and young adults, and potentially normal and anomalous trichromats – may share more similar color percepts than expected from their optical and neural differences simply because they are adapted to a similar color environment [63].

However, the same processes that drive observers toward convergent percepts when they are exposed to the same environment should lead to divergent percepts when the environment changes. The color statistics of images show large changes across environments, both in the mean color and the color gamut. For example, wet environments tend to be dominated by green foliage while dry environments are more yellow, and there are corresponding swings in the dominant axis of the distributions [109]. Substantial variations also occur over time because of seasonal changes [106]. Thus individuals living in different environments should be adapted and experience color differently, and their percepts might even cycle with the seasons [118].

To explore these effects, we developed a model to simulate how adaptation should alter color appearance across different environments [86, 119, 120]. The model assumed that the photoreceptors adapt to the mean of the color gamut while cortical mechanisms selectively adapt to the contrasts, and that adaptation adjusts the gains of the mechanisms so that within each, the average response in the current environment equals the response to a reference environment. The algorithm is simple and not intended as an actual model of human vision because it does not, for example, incorporate spatial structure or the fact that scenes are not uniformly sampled. It can nevertheless provide intuitions about how colors in a scene should appear to observers immersed in a given environment. A further advantage is that the model can simulate color appearance under theoretically complete and thus potentially very long-term adaptation (Figure 6). These “adapted images” can then be used in psychophysical studies to ask what an observer can see or do within an environment that they could not do before adapting [119]. For example, one consequence of adaptation is to reduce the contrast or salience of the dominant or expected characteristics of the scene, while increasing the relative contrast or salience of more novel colors. This predicts that observers should become more efficient at searching for novel colors or statistical outliers in a color distribution if they are first adapted to the distribution, a performance improvement that has also been found in empirical measurements of adaptation and visual search [114].

Figure 6.

Swapping swatches [111]. Top: Starry Night by Munch and Man with a Hoe by Millet (digital images courtesy of Getty’s Open Content Program). Bottom: The palettes in each image have been “adapted” in a model visual system so that the average response of each color mechanism is equated for the average response of the same mechanism to the alternate original image.

5. Relationships between color categories

While these adaptation effects thus play an important role in shaping whether two observers experience color in similar or different ways, there may nevertheless be more fundamental factors contributing to individual differences in color appearance. As Figure 1 illustrates, the stimuli that color-normal observers select for the unique or binary hues shows marked variability [27, 121], to the point where one observer’s best example of orange could be close to another’s yellow or red [10]. Differences in unique hues and focal colors have also been found across different linguistic groups [122]. However the differences among individuals within each language are often far greater than the mean differences between populations [122–124].

A striking feature of these individual differences is that the focal choices for different color categories are uncorrelated [27, 123]. Thus while blue and yellow or red and green may be phenomenal opposites, how two observers differ in their unique blue choices does not predict how they will differ for yellow. Binary hues like orange and purple also vary across observers in ways that cannot be predicted from the settings for the unique hue primaries for which they are presumed to be composed [10]. This independence is inconsistent with the biases predicted by variations in the observers’ state of adaptation or their spectral sensitivity, which as noted above largely fails to account for normal variations in color appearance. The effects of adaptation and sensitivity variations also tend to be spectrally broad, and thus would predict correlated influences across different hues [27]. Thus the nature of these inter-observer variations in color categories remain enigmatic.

Emery et al. [125, 126] finely sampled individual differences in color appearance using a hue scaling task, in which a given hue is decomposed into the perceived proportion of the red-green or blue-yellow components [127]. For example a balanced purple can be described as 50% blue and 50% red. Settings were collected for 26 observers who scaled 36 stimuli spanning the LvsM and SvsLM plane, and factor analysis was then used to assess the underlying sources of the individual differences. A factor analysis or principal components analysis extracts the latent dimensions that best account for the pattern of correlations among the observed set of variables. These analyses are well suited to variables that reflect quantitative stimulus dimensions like wavelength or spatial frequency, and can potentially identify the precise mechanisms controlling the responses and how much they vary [50, 128].

The hue scaling resulted in a pattern of 7–8 factors that each accounted for the variations only over a fairly narrow range of hues [126] (Figure 7). Moreover, none of the factors exhibited a biphasic pattern that would be expected if they represented a color-opponent dimension. These results are surprising given that the task required the observers to decompose each hue into its red-green and blue-yellow components, and moreover could only use one color of each opponent pair to describe the hue. Yet these response dimensions did not manifest as the perceptual dimensions controlling the responses. Instead, the pattern could reflect a population code for color, in which each chromatic stimulus is represented by the distribution of activity across mechanisms tuned to different directions in the chromatic plane. The variations across observers would then suggest that different hues are decoded independently, perhaps because we learn the population responses corresponding to different color categories like red and orange independently. Why there are 7–8 factors and not 4 or 16 is unclear, but it is tempting to speculate that this is because English speakers tend to describe the hue circle in terms of 8 categories corresponding to the unique and binary hues. When observers completed a second task where they not only scaled the hues but also had to label them by one of 8 terms, the analysis revealed some factors that accounted for both tasks, while others that were specific to scaling or naming [125]. This indicates that there may not be a strict correspondence between how the stimuli were perceived and verbally labeled. In any case, the fact that multiple, narrowly tuned factors emerged for both tasks suggests that the different hue categories or regions exist somewhat free-floating, rather than derived from and bound to an underlying color-opponent structure.

Figure 7.

Individual differences in hue scaling. Left: Perceived hue angle (blue-yellow / red-green) as a function of the stimulus angle in the cone-opponent space, for 26 observers. Right: Variations across the observers revealed 7 systematic factors narrowly tuned to different stimulus angles (after [129]).

6. Numbering by color

Further evidence that the unique hues or other color categories differ not only quantitatively but also conceptually comes from studies where observers are asked to infer the metrical relationships between different colors. Specifically, to what extent do we represent color in a way that we can readily perceive the coordinate relationships between colors based on the three-dimensional representation imposed by the cones? On the one hand, similarity judgments appear intuitive and can be used to recover the two-dimensional structure of hue variations. In fact this provided the initial validation of multidimensional scaling [130]. However, that color similarities can be accounted for by distances in a representational color “space” does not indicate how an individual can judge the relationships between coordinates within the space. For example, Wuerger et al. showed that such judgments are not consistent with a Euclidean representation [131]. Recently, Ennis and Zaidi [132] reported that similarities based on perceived midpoints between different color pairs could reflect an affine structure, but only if observers were explicitly instructed to judge the color differences in terms of the red-green and blue-yellow opponent axes. If left to their own devices they instead showed much less evidence for an underlying geometric representation.

We examined the ability to infer the metrical relationships between colors in an ensemble coding task. Many studies have suggested that the visual system can reliably extract the summary statistics of a distribution of stimuli, such as their mean or variance, and that the mean estimate is often more robust than for the individual items [133]. This ensemble coding has now been demonstrated for a wide range of stimuli, including low-level features like motion or orientation and high-level features like facial expressions. Color is an example where the mean provides especially useful information, for example to the color of the illuminant. Several studies have now examined ensemble coding for color [134–137]. Sensitivity to the mean of a color distribution is high if the differences between the colors is not too large (e.g. within or between adjacent color categories), though it can be systematically biased, for example toward the more saturated elements [138, 139]. However, for very different colors the task becomes difficult. For example, Figure 8 shows results from an unpublished study where subjects were shown one hue and asked to choose the complementary hue, such that the average would be gray. We originally approached this with the goal of exploring which “space” the observer used to estimate the average, for what is complementary in one representation (e.g. the cardinal axes) is different from another (e.g. perceptual opponency). However, we instead found that observers without formal training in color vision found the task meaningless, for they had no perceptual intuition that the average of two saturated hues should be achromatic. In related work MacLeod, Pallett, and Krizay [140] found that green appeared no more different from red than a blue or yellow.

Figure 8.

Inferring color relationships. Settings for an individual observer asked to adjust the hue of a test stimulus (circles) so that it was complementary to a reference stimulus.

In a second task, we displayed a color distribution with a fixed gamut along the LvsM and SvsLM axes and asked observers to adjust the mean to be achromatic. In this case, increasing the variance along either axis reduced sensitivity along either axis [141]. For example, balancing the colors along the LvsM axis was harder when we increased the variance along the SvsLM axis or vice versa. This nonselective interference is in contrast to the strong selectivity for these axes in tasks like noise masking [142], but is consistent with the possibility that increasing the differences between the colors made it harder to judge their relationships, because the colors are encoded as qualitatively different categories rather than as points in a metrical space. Note that this may be very different from other visual attributes. For example one can readily judge the average trajectory of a field of moving dots [143], and this feels like an intuitively easy judgment, like estimating the wind direction in a snow flurry. For color it does not, and observers may instead have to indirectly infer the average (e.g. from the relative saturation of the colors). This raises important general questions about ensemble coding and which visual dimensions in fact allow an explicit representation of summary statistics, and how this depends on the nature of the visual representations for different attributes.

7. Blue-yellow asymmetries

Thus far we have considered blue and yellow as if they are – if not opposites – at least equals in the perception of color. However, a number of recent studies have pointed to important differences in how the visual system responds to blue vs. yellow hues and to possible differences in the inferences about color associated with these hues. In measures of scenes shown under different illuminants, observers are less sensitive to an illuminant change along a bluish axis than along a yellow or red or green axis [144–147]. This suggests greater constancy for blue illuminants, since their variation is less likely to visibly alter scene color.

Winkler et al. [148] compared color percepts in images shown with blue or yellow tints. The study was inspired by a comment from Lothar Spillmann to John Werner that the photographic negative of a person appeared achromatic, yet the afterimage from staring at the negative seemed strongly colored (personal communication). Since the afterimage is the complement, why should it appear more saturated? Winkler et al. found that the asymmetry was due to the yellowish skin tones appearing more colorful than the colorimetrically equivalent blue tones. However, this did not reflect knowledge about the specific object in the image, because the blue-yellow differences remained when the pixels were scrambled. The differences also persisted for uniform color patches. If the colors were shown as increments or decrements on a background, then much higher chromatic contrast had to be added in the blue than the yellow direction in order for the patch to be described as a color and not white. The asymmetries were specific to the blue-yellow axis, with no difference in the relative strength of reddish vs. greenish hues. Winkler et al. also showed that these biases occurred for many different bluish objects. For example silver coins and steel pots appear strongly golden or copper when the chromatic contrast was inverted (Figure 9). They also occur for the viral image of #thedress, which is seen by different people as either blue-black or white-gold. Inverting the image color causes the stripes to appear yellow with high consensus [148, 149].

Figure 9.

Blue light and yellow objects. When the original bluish hue of the coins is amplified the light appears bluer while the coins tend to continue to appear silver. In the complementary images the coins instead appear to become more golden while the lighting remains neutral (after [148]).

A potential explanation for these effects is that individuals do not vary in their intrinsic sensitivity to blue and yellow, but tend to attribute bluish tints to the illuminant while yellowish tints to the object [148]. Consistent with this, when the chromatic contrast of an image was amplified, in blue-tinted images the lighting appeared to be changing while in the yellow complements the object color instead seemed to change. Discounting blue as the illuminant color may decrease its effective contrast, and Retter et al. [150] showed that this reduced contrast is measureable as an asymmetry in the EEG response to an alternation in the original and inverted dress image. In fact how observers classified the dress could be predicted with more than 80% accuracy based on the topography of the EEG responses. The different attributions might also account for why yellows are more salient than blues in a visual search task [12].

These asymmetries may reflect general inferences about material and lighting. In natural scenes shadows tend to be blue from the diffuse skylight, and the blue chromaticity of shadows can be strongly discounted [151]. Conversely, recent analyses suggest that objects tend to be warm colors [152]. Thus the visual system may have a prior to ascribe blue to the lighting and yellow to the object. It is possible that this is why (to me) the world itself appears to change color dramatically at sunset – when the sun’s warmer rays are misattributed to the objects they illuminate (Figure 10).

Figure 10.

A tower in Tallinn Estonia, shortly after the 2019 ICVS conference in Riga. The orange hue is from the light of the setting sun. (See also [153])