A psychophysical dissection of the brain sites involved in color-generating comparisons

Abstract

We have used simple psychophysical methods to determine the sites of color-generating mechanisms in the brain. In our first experiment, subjects viewed an abstract multicolored “Mondrian” display through one eye and an isolated patch from the display through the other. With normal binocular/monocular viewing, the patch has a different color when viewed on its own (void mode) or as part of the Mondrian display (natural mode) [Land, E. H. (1974) Proc. R. Inst. G. B. 49, 23–58]. When the two stimuli were viewed dichoptically, with the patch occupying the position that it would occupy in the Mondrian complex under normal viewing, the patch always appeared in its void color. In a second experiment, when subjects viewed multicolored displays through a different narrow-band filter placed over each eye, the information from the two eyes was combined to result in new colors, which were not seen through either of the two eyes alone. Taken together, these results dissect color-generating mechanisms into two stages, located at different sites of the brain: The first occurs before the appearance of binocular neurons in the cortex and compares wavelength information across space, whereas the second occurs after the convergence of the input from the two eyes and synthetically combines the results of the first.

The wavelength composition of light reflected from objects changes continually during the day, without altering their color (1, 2, 3). The brain's ability to “discount the illuminant” (2) and thus assign a (constant) color to a surface can be demonstrated by Land's (1) psychophysical experiments, which show that this capacity is critically dependent on the brain's ability to compare the wavelength composition of the light reflected from a patch with that reflected from its surrounds. It follows that any given patch of a multicolored display will have two colors, the one that it acquires when viewed in context, i.e., together with the rest of the display (natural color), and the one it has when viewed on its own (void color). Land called his theory the “Retinex theory of color vision,” because he did not know where in the visual pathway between the retina and the cortex these color-generating interactions occur. Studies with a single “split-brain” patient, who was unable to perceive the natural color of a patch presented to one hemifield when the rest of the Mondrian was presented to the other, suggest a cortical site for such interactions (4). In anesthetized monkeys, cells selective for the color (as opposed to the wavelength) of the stimulus are found in area V4, but not in areas earlier along the color-specialized pathway (5, 6). Lesions in area V4 impair color constancy, but not wavelength discrimination, in both monkey (7) and man (8, 9). Furthermore, recent imaging studies (10) show that a role for area V4 in color constancy is very likely. On the other hand, a subcortical mechanism for color-generating interactions also has been suggested by the color-induction results of experiments in a single brain-injured patient with hemianopia (11). A subcortical mechanism could also be hypothesized through the opponent center-surround organization of the receptive fields of lower order visual neurons, although their wavelength opponency is not fully compatible with a Retinex-type “lightness-generation” mechanism (12).

Theoretically, the Land (1) computational system posits two processes for color perception. The first consists of calculating a “lightness record” of a scene for light of a given waveband, by comparing the (different) amounts of light of that waveband reflected by the various areas of the scene (i.e., a global-spatial comparison). In the long-wave lightness record, for example, each area is assigned a relative brightness value (with respect to the other areas, irrespective of the illumination) under illumination with that particular waveband. The second process consists of combining the lightness records generated by the long, middle, and short wavebands, and thus constructing the colors of the various areas (assigning the color white, for example, to an area with the highest relative brightness in all three records). This conceptual division of color generation into two broad operations, one spatial and one synthetic, inspired us to use dichoptic viewing to investigate which color mechanisms are situated where in the brain. By using a simple psychophysical procedure, we have been able to distinguish in broad anatomical terms between the two different stages of the (Land) operations leading to color perception, and to relate them to different parts of the color pathway. We are not unaware of the fact that there has been some dispute as to whether the phenomena described by Land could not have been predicted from the already known literature (see ref. 13, for example). But the perceptual reality of his spectacular demonstrations is beyond any doubt, and his theoretical proposition can more or less explain very basic color vision mechanisms (whether known before him or not) in a simple and fairly accurate way. It is for this reason that we have chosen the Retinex theory perspective (and terminology) for our paper. It must be noted, however, that the validity of our experiments does not depend on choosing one or other of the (several) theoretical frameworks of color vision.

Experimental Procedures

Five male subjects, aged between 25 and 50 years, all with normal color vision, viewed a multicolored Mondrian display (12 × 14 cm) presented at a viewing distance of 20–50 cm on a computer screen (Sony Multiscan 20 SE II with a 120-Hz refresh rate) against a black background in a completely dark room. For any given experiment, an area of the Mondrian was chosen (the nominated area) and a copy of that area (the target) was made. The target and the Mondrian could be presented separately to the two eyes, using a stereoscopic viewer (CrystalEyes 2; Stereographics, San Rafael, CA). The left and right eye stimuli were interchanged on the computer screen at a rate of 7 ms, and liquid crystal goggles alternately occluded the two eyes at a rate synchronized to that of the stimulus-presenting monitor. Under dichoptic stimulation, the nominated area was replaced by a black patch of the same shape and size on the Mondrian (presented to one eye), and the target (presented alone to the other eye) was accurately superimposed on that “empty” part of the Mondrian. Fig. 1 shows the appearance of the Mondrian and the patches used, under natural viewing conditions; because of color constancy, these (natural) colors did not change much with changes in illumination (1). The appearance of each patch under the void viewing condition can be inferred by Table 1, constructed by using a SpectraScan spectrometer (Micron Techniques, Chatsworth, CA). Although the natural/void color differences when using a computer screen are not as spectacular as when using a paper Mondrian with projectors, they are strong enough for subjects to be able to clearly distinguish between the two colors (see results in Fig. 1 and also http://www.vislab.ucl.ac.uk./WEB/imgexpland.html for a demonstration).

Figure 1.

(Left) The appearance of the Mondrian stimulus in our experiments with the five patches used as targets. The natural colors seen here change little with changes in illumination (color constancy); the void color of each patch under each particular illumination used is described in Table 1. (Right) Normal versus dichoptic presentation results. Each of the Mondrian/target pairs was presented a total of 100 times, 50 times separating Mondrian and target between the two eyes and 50 times presenting both to the same eye (see Experimental Procedures). With the exception of a few trials in which subjects were uncertain (most probably because of an instant lack of concentration), they always reported seeing color A (natural) in normal and color B (void) in dichoptic presentations; the opposite was never the case.

Table 1.

The Commission Internationale de l'Eclairage (CIE) coordinates of the five Mondrian patches used

Patch no.	CIE (x, y, Y)
1	0.376, 0.415, 3.47
2	0.459, 0.477, 5.78
3	0.344, 0.348, 9.47
4	0.423, 0.452, 15.0
5	0.597, 0.354, 5.46

Shown are the CIE chromaticity coordinates of the five different Mondrian patches used in this study, describing their color appearance under void viewing conditions. x and y, chromaticity coordinates; Y, luminance specified as cd/m². 

At the beginning of the experiment, each target was presented under both the natural and the void viewing conditions, and subjects were asked to name the two (different) colors as “color A” and “color B,” respectively. Each subject viewed each Mondrian/target pair 20 times, by using 10 normal (Mondrian and target presented to the same random eye, nothing to the other eye) and 10 dichoptic (Mondrian minus target in one eye, target alone in the other eye) presentations, in a mixed order. To eliminate any binocular rivalry, stimuli were flashed on the screen for 82 ms. Such brief presentation times are enough for color constancy mechanisms (14) but not for eye-specific rivalrous networks to develop (15). Thus, because of the fusion of the two monocular stimuli, subjects could not distinguish between the two different presentation types. After each presentation, subjects had to reply whether the perceived color of the target was color A or color B, or choose the “not sure” option if not totally confident. The experiment was also repeated with the subjects viewing the stimuli through narrow-band filters, so that the effect of the surrounds on the appearance of the target in dichoptic stimulation could be investigated at single wavebands as well.

In the second experiment, the subjects used a 644-nm narrow-band filter for one eye and a 504-nm narrow-band filter for the other (Barr & Stroud, London; 8–10 nm at half height) to view multicolored stimuli. They were asked to close their eyes before the stimuli appeared on the screen (to avoid the possibility of any memory effects) and then to open them separately and report on the colors they could see with each eye alone, before opening both eyes together, and then report again on the colors present on the screen. Brief presentations (82 ms) were used to eliminate any binocular rivalry and thus to avoid dominance and suppression phases of one or the other stimulus across different parts of the visual field. The Mondrian stimulus was again that of Fig. 1. Any pair of green/red gelatin filters can be used to get an idea of our two monocular stimuli, whereas periods of fusion between the dominance and suppression periods of binocular rivalry (under prolonged viewing) can reveal the appearance of our dichoptic percept as well.

Results

Spatial Interactions and the Calculation of Lightness Records.

In our first experiment, subjects viewed a multicolored Mondrian display through one eye and an isolated patch from it through the other, both illuminated with the same amounts of long-, middle-, and short-wave light (see Fig. 2). Under normal binocular (or monocular) viewing, the patch appears to have different colors when viewed alone against a black background (void) and when viewed together with the rest of the Mondrian (natural). Under our dichoptic viewing conditions, we were interested to see whether the patch would be perceived as having its void color, or whether the presence of the surrounding Mondrian would result in the perception of its natural color. As seen in Fig. 1, subjects always perceived the natural color (color A) under normal presentations, and the void color (color B) under dichoptic presentations. In the latter case, therefore, although the patch was perceived as an integral part of the Mondrian, it nevertheless appeared in its void rather than in its natural color. A direct distinction between the two colors could also be made by slightly changing the experiment and leaving the original nominated patch in place and superimposing the target (presented to the other eye) over a different location of the Mondrian. Furthermore, one could directly compare the colors that result from dichoptic and normal viewing (in the original superposition experiment) by looking at the computer screen with and without the goggles, respectively. Whatever the method, by using dichoptic presentations, although the target area and the Mondrian were perceived together and in the same part of visual scene as if the target was indeed another patch of the Mondrian, the target area always appeared in its void color. If the original nominated patch was also present, this patch and the target had unambiguously different colors, although they were both perceived as part of the same Mondrian and despite the fact that they were reflecting the same triplet of energies at the same time.

Figure 2.

Normal versus dichoptic presentations of a Mondrian and a single target area derived from it. The visual brain has been schematically divided into three parts: two monocular, where neurons are driven by one of the two eyes, and one binocular, where neurons are driven by both eyes. Under normal binocular viewing, light information from both the Mondrian and the single patch (target) are available to the monocularly and to the binocularly driven parts of the brain. This scenario is not the case with dichoptic presentation, where each monocular channel contains input from either the Mondrian or the target, with the two inputs coming together after the emergence of binocularly driven neurons in the cortex. (This figure is an illustrative diagram and does not contain the original stimuli used.)

The result of the spatial (comparative) interactions is what Land (1) calls a “lightness record,” i.e., a record of the relative brightnesses of the areas within a scene illuminated by light of a particular wavelength (irrespective of the exact amount of illumination). Our result suggests that the calculation of this relative brightness is done at the monocular level, i.e., the surrounding Mondrian has no effect on how bright a patch appears if the former is presented to one eye and the latter to the other. Fig. 3 demonstrates this phenomenon under illumination with monochromatic middle-wave light. The Mondrian appears as areas of different brightness in a greenish “wash.” The isolated patch to the right of the figure looks brighter than does the identical Mondrian patch on the left. If we use dichoptic fusion to superimpose the left and right parts of the figure, the two (identical) patches still appear different, although they have the same spectral return to the eye and are perceived on a single common background. It is therefore not the perceived background but the background at the monocular level that determines the brightness. In our dichoptic presentations, each lightness record is calculated as if there were no surround (void condition), although the surround is present and clearly perceived by the observer. It therefore is not surprising that, when the long-, middle-, and short-wave lights (and, thus, the lightness records) are combined, surround influences in the perception of the resulting colors are not possible under our dichoptic stimulation, and so the color perceived under such conditions is the void one.

Figure 3.

Natural versus void viewing conditions and the result of dichoptic presentation, under illumination with middle-wave light only. When patch no. 2 is isolated from the rest of the Mondrian (Right), it appears brighter than when viewed as part of the Mondrian (Left). This demonstration of our dichoptic presentations is not as good as are the true laboratory conditions, but it still works. By using a reduction tube, make sure your right eye can see only the isolated patch on the right part of the figure, whereas your left eye can see only the Mondrian on the left part of the figure. Observe the difference in brightness between the two identical patches. Now cross your eyes so that the isolated patch is superimposed on the Mondrian and compare again. Although the two no. 2 patches are identical and perceived on a common Mondrian background, they still appear very different.

Lightness Record Combination and the Generation of Colors.

In his early studies, Land (1) projected a long- and a short-wave record of the same scene by using two different projection lights, and he observed the appearance of colors that were not present when either of the two projectors was used alone, which could not be explained directly as a result of mixing different amounts of the two projection lights used (16). We wanted to see whether this synthetic combination of different lightness records takes place at the early, monocularly driven parts of the visual brain or is a function of the higher visual areas. We also wanted to devise an experiment in which color stimuli from the two eyes do combine, to verify that our first result is specific to target/surround interactions, rather than simply the consequence of separating the stimuli between the two eyes. We have repeated Land's experiment almost exactly, where a long- and a short-wave record are combined, to give rise to a full gamut of colors (rather than simple wavelength mixing); the difference in our case is that one record is presented to one eye and the other record to the other. Therefore, if this combination takes place in our experiment as well, the involvement of visual areas beyond the monocular part of the brain is strongly suggested. Fig. 4 illustrates our second experiment, where subjects viewed multicolored stimuli through a 644-nm narrow-band filter over one eye and a 504-nm narrow-band filter over the other. With the tricolor stimulus on the left, the red and yellow areas were indistinguishable bright patches when viewed through the eye with the 644-nm filter, whereas the green area appeared as a dark patch. Viewed through the eye with the 504-nm filter, the yellow and green areas were bright and indistinguishable, whereas the red was very dark. However, when the scene was viewed through both eyes, all three colors were perceived clearly and separate from each other. With the Mondrian, different areas had different brightnesses within a reddish or greenish “wash” when viewed monocularly through the 644-nm and 504-nm filters, respectively. No colors could be perceived by either of the two eyes alone, just as in Land's two-projector experiments, no colors are seen by switching on either of the two projectors alone (16). With both eyes open, however, the full gamut of the Mondrian colors was generated (see Figs. 1 and 4), colors that were absent from the images viewed through each eye alone. In this case, unlike our first experiment, the interaction of stimuli between the two eyes is possible, and thus the brain mechanisms involved must be situated beyond the monocularly driven part of striate cortex.

Figure 4.

Dichoptic presentation of two different lightness records of multicolored stimuli, using 644-nm and 504-nm narrow-band filters in front of each eye. In this experiment, the two lightness records of the stimuli are combined after the convergence of the two monocularly driven parts of the visual system. Therefore, any interaction between the input and the two eyes must take place at higher, binocularly driven visual areas.

Discussion

From our first experiment, we conclude that the visual system is unable to combine information about the wavelength composition of one part of the visual field and its surround when the two are viewed separately through the two eyes. The relevant spatial-interaction mechanisms must operate at an early stage of the color system, before the information from the two eyes is combined in visual areas with binocularly driven neurons (see Fig. 2). All of the spatial-comparison interactions thus must be completed at the stage of V1 monocular neurons or earlier. If the spatial integration took place in area V4, for example, dichoptic stimulation should result in the perception of the same color as normal binocular vision would, because this area contains a vast majority of cells driven equally well by both eyes and is therefore provided with information concerning the wavelength compositions of light reflected from both the target area and the Mondrian. If, on the other hand, the necessary spatial comparisons take place at the retina (or the lateral geniculate nucleus or the monocularly driven part of V1), the appearance of the void color should be expected in our experimental setup, because, at this level, the Mondrian information from one eye is inaccessible to neurons processing the target stimulus through the other eye. Our result also can account for neurophysiological results from the cat, which show that most V1 and some lateral geniculate nucleus neurons modulate their firing rate in correlation with the perceived brightness of achromatic stimuli within their receptive field (17–19); the influence of the surround on the responses of these cells is not attributable to feedback from higher areas, as postulated, but is a consequence of the fact that relative brightness is calculated at the monocular level.

Our aim was to investigate the anatomical correlates of the way in which the total visual image influences the color of the various areas that compose it. The method of dichoptically separating center from surround has been used previously, in psychophysical experiments investigating the appearance of achromatic stimuli fused between the two eyes (see below). However, the influence of a background presented to one eye on the brightness of a stimulus presented to the other depends on many subtle factors (20). The design of most previous studies does not clearly distinguish between these factors, and therefore no simple conclusion concerning dichoptic center-surround influence can be drawn. Some studies suggest that the effect of a contralateral background on patch brightness is small (3, 20–22), whereas others imply that monocular mechanisms alone cannot account for induced brightness/blackness, and the involvement of more central mechanisms is necessary (23, 24). A latter study (25) tries to separate between the physical contrast effect of the immediate surround, assigning it to monocular mechanisms, and the assimilation effect of a ring surrounding the surround, assigning it to central mechanisms. How are these two effects related to the calculation of lightness records? Assimilation, although not negligible, has been shown to have a much weaker effect than does physical contrast in determining the brightness of a stimulus (26, 27). Furthermore, the effect of assimilation is a perceptual one: A stimulus surround can be made to look brighter because of the presence of a darker external ring (change in physical contrast), and this brightness also is transferred to the center (change in perceived brightness). But, with respect to the center and surround, there is no change in the illumination of either, i.e., the illumination ratio-taking lightness mechanisms will not be affected by the surround's extra perceived brightness. Contrast effects, on the other hand, are attributable to a difference in the amount of light reflected from two areas, and therefore are the determinant of the lightness calculations. However, all the psychophysical studies mentioned above have concentrated on local contrast, i.e., the effect of the immediate surround on the brightness of an area. But in the real world, colors do not change markedly when objects move in space against a variegated background. Similarly, in the Mondrian experiments, the appearance of a patch does not depend on which part of the Mondrian it is placed on, and thus the Retinex method compares the light reflected from one area of the Mondrian with the light reflected from all of the other areas of the Mondrian, not just from the immediate surrounds (28). The position of each area in a lightness record of a scene is the result of a global rather than a local comparison calculation in the brain, and this is the way our experiment was designed to attack the problem. Furthermore, the striking and unambiguous difference between the appearance of the void and natural colors in our experiment is both simpler and clearer than in previous studies involving detailed brightness-matching tasks of different grays (see first paragraph of Methods in ref. 20).

We, of course, do not suggest that the perceived brightness of an area is completely determined at the monocular level. As mentioned above, we were specifically interested in the wavelength ratio-taking mechanisms responsible for the calculation of an area's position in the lightness records of a scene, and thus for the determination of its color. The involvement of higher-order mechanisms in determining the apparent brightness of an area has been demonstrated in experiments where simple modifications in the organization of a scene that should have little effect on low-level mechanisms can greatly alter the brightness of a stimulus (29). Furthermore, a series of recent elegant psychophysical experiments suggest that the luminance profile of the whole scene is associated with the most probable real-world illumination source, which is in turn used as a guide for the brain to assign brightness values to the individual elements (30–33). It therefore seems very plausible that top-down mechanisms can alter the brightness of a stimulus, the end result being a combination between this and earlier low-level mechanisms. We tend to believe that low-level mechanisms are nevertheless stronger. For example, a gray patch appearing to be directly illuminated by the light source of a scene still appears lighter than an identical (isoluminant) patch appearing to be in shade, if the immediate surround of the former is much darker than that of the later (see figure 2 in ref. 32). More experiments are necessary to determine the relative strength and any possible interactions between high- and low-level mechanisms.

Although essential, the lightness-generating process is but one step in the construction of color. If the final part of color generation takes place later in the (binocularly driven) cortex, it should be possible to demonstrate psychophysically the interocular interaction of information within the color system. This idea was the aim of our second experiment, in which two different lightness records of the same scene were presented dichoptically to subjects, resulting in the formation of new colors that did not exist in either of the two monocular stimuli. Our result is in agreement with previous studies, showing that dichoptic color interactions are possible under some conditions (for a review, see ref. 34). The colors produced in our interocular interactions were not limited to reds, greens, and yellows, just as the colors seen on Land's two-projector demonstrations were not only those of the two projection lights used and of their mixtures (1, 16, 35). What Land has described under normal binocular vision and we have observed with dichoptic stimulation is a color-generating rather than a wavelength-mixing mechanism. Therefore, unlike the generation of lightness records, which takes place in the monocularly driven part of the brain, the generation of color by synthetically bringing these records together occurs later on, after the eye-specific information in the cortex is lost. This finding could suggest one possible role for area V4 in generating colors. The fact that V4 has large receptive fields that make possible interactions of large parts of the visual space (12, 36) might intuitively lead one to suppose that it could be the center for the spatial comparisons. But the generation of lightness records can be equally well achieved by a network of local comparisons (14). Our results are consistent with the latter hypothesis, because they demonstrate that spatial-wavelength interactions take place before the appearance of binocular cells in V1. On the other hand, a role for higher areas (such as V4) in using the results of these interactions to generate color is in agreement with, and in fact suggested by, our second experiment. Thus, the use of dichoptic viewing has allowed us to determine that color-generating interactions are widely distributed within the color system of the brain, occurring at both its pre- and postbinocular stages.