Optimal sampling of visual information for lightness judgments

Abstract

The variable resolution and limited processing capacity of the human visual system requires us to sample the world with eye movements and attentive processes. Here we show that where observers look can strongly modulate their reports of simple surface attributes, such as lightness. When observers matched the color of natural objects they based their judgments on the brightest parts of the objects; at the same time, they tended to fixate points with above-average luminance. When we forced participants to fixate a specific point on the object using a gaze-contingent display setup, the matched lightness was higher when observers fixated bright regions. This finding indicates a causal link between the luminance of the fixated region and the lightness match for the whole object. Simulations with rendered physical lighting show that higher values in an object’s luminance distribution are particularly informative about reflectance. This sampling strategy is an efficient and simple heuristic for the visual system to achieve accurate and invariant judgments of lightness.

Judging the lightness of visual stimuli has been studied for centuries, since the original investigations by Weber (1) and Fechner (2). The light reaching the eye is the product of the illumination and the reflectance of the object, and also depends on the scene geometry (3). However, only the proportion of reflected light is an invariant property of the object and thus of great importance for vision. There are several well-established factors that support lightness constancy in the face of these challenges. On the one hand, lateral inhibition between retinal neurons filters out shallow intensity gradients, which are mostly caused by illumination effects (4, 5). On the other hand, more complex factors also have an effect on lightness perception, such as object shape (6–9) or the interpretation of transparent surfaces (10, 11). However, eye movements have been almost completely neglected so far, even though a general influence of viewing behavior has been shown for some color constancy tasks (12–15). This finding is surprising because the visual system needs to sample the local properties of objects and this is accomplished by moving the eyes and the focus of spatial attention around. Because visual acuity, luminance sensitivity, contrast sensitivity, and color sensitivity change with retinal eccentricity (16–18), our visual system has to stitch together its representation of the world from many small samples to analyze the visual scene in detail. Peripheral vision is not only characterized by poor resolution, but also the appearance of basic visual features—like spatial frequency, luminance, or chromatic saturation—is distorted in the periphery of the visual field (19–22). Eye movements may then be used to select relevant information, even for stimuli that are above threshold in peripheral vision. We investigated whether the distribution of fixations on an object has an effect on its apparent lightness. For surfaces made of a single material, the reflected light varies with the illuminant and its interactions with the surface’s geometry, whereas the reflectance is a property of the material. To judge the lightness of an object, defined as its apparent reflectance (23), the visual system has to select a single value from a whole distribution of local luminance values across the whole object. We therefore tested the hypothesis of a link between the local information sampled from individual fixations and the apparent lightness of an object.

First, we show that observers tend to take heavily into account the brighter parts of objects when they are asked to match the color or lightness of these objects. Second, we show that observers tend to fixate on the brighter parts of the objects as they make their match. Third, we show that this link between fixations and lightness perception is causal. When we forced the observers to look at particular points on the objects, their lightness impressions changed according to the luminance of the fixated regions. Fourth, we show that eye fixations and attention both contribute to this effect. Fifth, we show that the brighter parts of objects are particularly diagnostic of the object’s reflectance.

Results

Observers had to adjust the color of a small patch of light to match the color of one of several real objects presented to them, as illustrated in Fig. 1 A and B. The luminance adjusted by the observers was significantly higher than the mean luminance of the light being reflected from the object into the eye (t₅ = 11.6084, P < 0.001). In fact, the matches closely correspond to the brightest parts of the objects. This finding indicates that the brighter regions of the objects are weighted more heavily. Fig. 2 shows the observers’ average lightness matches together with samples of the same object (a paper cone) under different illumination conditions. The match is quite similar to the brightest parts of the cone and a piece of paper cut out from the cone, such that it is oriented perpendicular to the light source, maximizing its luminance. The piece of paper cut out from the cone, mounted at the location of the matching box, appears much darker than the cone, because the surface of the computer monitor was oriented nearly parallel to the light source. Observers are known to be far from perfect in taking such geometrical aspects into account (24). The match is also much brighter than the mean luminance across the whole object, which raises the question as to why observers match the brightest parts of the objects, and whether this strategy is of advantage.

Fig. 1.

(A) Photograph of the matching setup. The image shows the paper cone together with the matching disk on the upper right of the cathode ray tube (CRT)-monitor. (B) Real objects we used. From left to right, green wool ball, green wool cylinder (same wool), green candle, red candle, yellow candle, and orange paper cone. (C) Spectroradiometric measurements. Spectral images obtained by measuring the objects superimposed on a 22 × 34-cell grid. For reproduction purposes, spectral data have been transformed to RGB (red-green-blue) values.

Fig. 2.

Object lightness distribution and lightness (L*) matches. Gray circles represent the mean object matches from each of the six observers. Black crosses represent the mean matches for each object averaged across the observers. The gray bars represent the mean ±1 SD and the black vertical lines represent the range of the distribution of L* within the objects. The colored squares to the right show the orange paper cone under several different illumination conditions. From top to bottom: paper oriented perpendicular to light source, mean match, maximum within the object, average within object, paper mounted on the CRT screen.

Role of Eye Fixations.

We explored a possible role of eye movements by measuring the fixation locations on the objects (Fig. 3A). Fig. 3B shows histograms of the luminance distributions across the objects together with a luminance histogram of the fixated regions. Even though the object luminance histograms are typically skewed toward darker values, the luminance distributions associated with fixations are centered on values higher than the median. Binomial tests performed for each object revealed that the proportions of fixations on brighter points than the median were significantly higher than chance (all Ps < 0.001). To investigate the relationship between fixations and matches, we computed the absolute difference between the matched luminance and the measured luminance at each point. We then described the relationship between these distances and the probability for each cell to be fixated with a cumulative Gaussian function. Goodness-of-fit analyses (mean χ² = 5.78, critical χ² = 12.59) revealed that our choice was appropriate. If the observers rely on what they fixate, the probability for a cell to be fixated should decrease when the luminance of that cell is very different from the matched luminance. Fig. 3C represents an example of the relationship between the fixation probability and the distance between the cell’s measured luminance and the matched luminance. The closer the luminance of a region was to the matched luminance, the higher its chance of being fixated. The slopes of these functions were indeed all negative. This result is consistent with the hypothesis that the local information of each fixation influences the apparent overall lightness.

Fig. 3.

(A) Example of fixations on an object (orange paper cone) during the color-matching task. Fixations falling outside the object area are not shown. (B) Relative frequencies of fixations and lightness (L*) for the six objects. The red histogram depicts the L* distribution within the object area, with the vertical dashed line indicating its median. The blue histogram depicts the distribution of the L* values associated with fixations (pooled across all observers). (C) Probability of fixation as a function of distance from the matched color for all spectral matrix cells. This example refers to one object and one observer. The example represents the probability for a cell to be fixated at least once, as a function of the difference between the matched and the spectrally measured L* for the given cell. Symbols represent the proportion of fixated points in ten bins.

Comparable results were obtained when the observers, instead of matching the hue, saturation, and lightness of a colored cone, matched only the lightness of an achromatic cone (Fig. S1). Changing the direction of the light source changed the distribution of the fixations, showing that the luminance distributions of the objects were actually driving the fixation behavior (Text S1 and Fig. S2).

Causal Role of Fixations.

This finding about the impact of changing direction raises the question whether the relationship between fixated regions of the objects and matched lightness is causal in nature: that is, whether where we look determines what we see. We investigated this question by forcing our observers to look at particular bright or dark regions of the objects. This process was achieved by presenting digitized photographs of some of the objects on a computer screen and monitoring eye movements while the observers performed the matching task. In this gaze-contingent setup, we erased the object image from the screen whenever observers tried to fixate a part of the object outside of the predefined fixation region. Whenever this happened, a dot was shown indicating the desired fixation position. The object image was also displayed when the observers fixated on the area of the matching box, so that the parafoveal information was available, as it was in the first two experiments.

We selected the forced bright (LF, light fixation) and dark fixation (DF) points based on the spatial distribution of luminance within the objects (average L* difference = 21.09). To control for a possible effect of the direction of the illumination gradient, images were presented in two versions, illuminated from the left or from the right. The matches were significantly brighter in condition LF, as shown in Fig. 4A. t-Tests revealed a significant effect of the fixation condition in both gradient conditions (t₃ = 4.10, P < 0.05; t₃ = 8.72, P < 0.05). To exclude a potential role of local adaptation, we also performed this experiment under conditions where viewing and matching was done in different retinal regions. We obtained identical results (Text S2 and Fig. S3).

Fig. 4.

(A) Lightness matches in the LF and in the DF condition: means and SEs of the matches. (Left) Data for images with a light gradient from the left side to the right; (Right) data for images with the opposite gradient. Black vertical bars represent the SEs. The four pictures at the bottom represent examples of the stimuli with the white dot indicating the fixated area. (B) Dark gray and light gray circles represent the mean object matches from each of the six observers, respectively, in the DF and in the LF condition. Dark and light crosses represent the mean matches for each object averaged across the observers, respectively, in the DF and LF condition. The gray bars represent the mean ±1 SD and the black vertical lines represent the range of the distribution of L* within the objects. Matches are always within the ranges.

Reflectance Versus Luminance.

One could argue that observers were not actually judging lightness, the apparent reflectance, but local brightness, the apparent luminance (23). This theory seems unlikely based on the results shown in Fig. 4B, indicating that the luminance difference between the observers’ matches under the LF and DF conditions was only 18% of the actual luminance difference between the fixated bright and dark regions. Furthermore, the matches had a considerably higher luminance than the fixated regions. To bring even more clarity to this issue, we designed a control experiment where we explicitly asked our observers to perform a lightness match (25). Pictures of the objects were presented on a computer screen and the observers used real grayscale paper chips illuminated by a light bulb to select their matches (shown in Fig. S4). The observers had to “pick the chip made of the same material as the object.” The presence of two light fields and the explicit paper-matching task are designed to induce the observers to perform the task in terms of lightness (25). Observers chose chips with higher reflectance when they were asked to fixate the bright regions (Text S3 and Fig. S5). Under such viewing conditions, photometric luminance is typically not perceptually accessible and perceived brightness heavily depends on the context (26, 27). We confirmed this result by asking three observers to sort the chips in terms of the brightness and the lightness of their paint. Observers were not able to distinguish between brightness and lightness (Text S3) and in the cases where luminance and reflectance were dissociated, the observers’ judgments were determined by reflectance rather than luminance (Fig. S5). Therefore, we are highly confident that our observers based their judgments on lightness in these experiments.

These results point to a direct causal link between the way an object is sampled through eye movements and its perceived overall lightness. Observers tend to produce estimates of the global lightness of objects that are above the physical average of the light intensities reaching the eyes and close to the brightest object regions. Interestingly, we observed that the matched luminance was more similar to that of the most frequently fixated regions of the objects, as our observers tended to fixate points with above-average luminance. Furthermore, the observers matched a higher lightness when they were forced to fixate a brighter region of the image compared with when they fixated a darker region. We can thus conclude that the way we perceive the lightness of objects is driven by the way we look at them.

Eye Movements and Attention.

An important question when considering our results is whether it is specifically related to eye movements or whether other selection processes could contribute as well. Although eye movements may well be the most important selection mechanism in the visual system (28), visual attention can be directed covertly at locations outside of the fixated region (29) and can affect the appearance of objects (30). We therefore performed an experiment where we decoupled the fixated and attended regions of the object. Our goal was to determine the contributions of eye fixation and attention to perceived lightness separately. We asked our observers to covertly attend to the fixated light or dark region or to another region of opposite luminance polarity. We found main effects of both attention (F_1,11 = 6.41, P < 0.05, η² = 0.06) and fixation position (F_1,11 = 7.54, P < 0.05, η² = 0.23), as shown in Fig. 5. There was no significant interaction (F_1,11 = 0.78, P = 0.4). This finding suggests that part of the forced-fixation effect we observed in the previous experiments might be attributed to attention rather than fixation per se. Because attention is normally tightly coupled with saccadic eye movements (31–33), we can assume that in our study the observers attended to the areas they were fixating. However, in principle the visual system could support the various sampling strategies by means other than eye-movement control.

Fig. 5.

Role of attention. Plot of the means and SEs of the points of subjective equal lightness obtained when observers compared a test disk and the objects' images presented on a computer screen. Gray bars represent the data in the light and black bars in the DF condition. The two bars on the left represent the data when attention was directed to the dark point, whereas the bars on the right represent the data obtained when attention was directed to the light point. The points of subjective equal lightness were on average higher when the observers fixated on the light point (light gray bars) and when the observers attended to the light region (bars on the right).

Optimal Lightness Estimates.

These results raise the question as to why the observers choose to preferentially sample the brightest points of the objects, and whether there is any advantage to this strategy. The brightest parts of the object frequently correspond to salient image features, and luminance contrast is regarded as an important feature for attracting human gaze (34). This way, the viewing strategy could simply be a side effect of more general oculomotor behavior. However, it could also be an indicator for a particularly useful strategy that observers have learned and that might even have awarded these image regions a high priority for attracting gaze. If this was the case, the fixated regions should yield the best estimate of the object’s reflectance. In other words, these regions should be reliable estimates by being maximally immune to the effects of the viewing conditions on luminance. At the same time, the estimate should be valid by being sensitive to changes in the object’s reflectance.

In general, the luminance of diffusely reflecting surfaces is proportional to the cosine of the angle between the surface normal and the direction of the incident light. This finding implies that differences in surface reflectance have the largest impact on luminance when they occur in regions that are oriented perpendicular to the light source (i.e., in the brightest parts of the objects). Moreover, variations in surface orientation have the smallest effect on luminance when the surface normal is almost perpendicular to the light. We used a physics-based rendering simulation to explore how well the reflectance of real-world objects can be estimated in a natural light field.

Using the software RADIANCE interfaced with a MATLAB toolbox (35), we rendered a set of six virtual models of objects (shown in Fig. 6A) under 100 random viewing and eight illumination conditions. Each view was rendered with one of six values of reflectance. We performed two separate analyses for each percentile of the luminance distribution, which we used as our potential estimates of lightness. Our goal was to establish which parts of the luminance distribution are most invariant to object rotation and which parts are most diagnostic of the reflectance changes.

Fig. 6.

Results of the physically based rendering simulation. (A) Set of 3D models of objects used in the simulation. (B) Single percentile distributions. The histograms represent the two distributions of the radiances (output of the rendering software), 100 different orientations for a certain percentile. These distributions are clearly overlapping. (C) ROC curve. The ROC curve is plotted for the two distributions of B. The AUC is close to 1, indicating high discriminability. (D) Classification performances for each percentile. The highest percentiles’ aggregated discrimination performance is higher than the performance of the discrimination based on the object average luminance (dashed black line). (E) SD for each percentile. Extreme percentiles are more stable.

A good estimate of lightness needs to be valid, namely sensitive to changes in reflectance. Fig. 6 B–D shows the results of a receiver operating characteristic (ROC)-analysis (36) that indicates to what degree an ideal observer can identify a change in reflectance in the presence of variations in scene geometry. The reflectance values have been chosen to lead to partially overlapping estimate distributions for each pair (Fig. 6B). The area under each ROC curve is a measure of criterion independent diagnosticity (Fig. 6C). We computed the cumulative area under the curve (AUC) (37) as an aggregated index of discriminability for each percentile (Fig. 6D). This analysis shows that the discriminability increases with the luminance of the object region that is compared. Performing the same analysis on the average luminance yields worse discrimination performance compared with the higher percentiles (Fig. 6D). A good estimate should also be reliable. Perceived lightness should stay constant when an object is rotated or the viewpoint is changed. The variance of each estimate, shown in Fig. 6E, is a good measure for reliability. The distribution has an inverted U-shape with minima for the darkest and the brightest object regions. Both the minimum and the maximum luminance would be reliable estimates of lightness. However, reliability alone is not sufficient for a good estimate, because a constant incorrect estimate is of course perfectly reliable, but entirely useless. Taken together, the results of both analyses indicate that the luminances of both the dark and the bright regions of objects are comparatively invariant under different views, but only the most illuminated regions are also diagnostic of the object’s reflectance.

Discussion

Our results have interesting consequences for our thinking about the early visual system. First, the findings show that eye movements, together with attention, play a decisive role not only for guiding action (38) or when high acuity foveal vision is required (39), but even for very simple visual tasks involving judgments of an object’s most basic visual properties. Second, the findings illustrate that the visual system might sometimes use simple and unexpected strategies to optimize its solutions to perceptual problems (40, 41). In our case, the lightness of the object is better estimated by the brighter regions of the object and oversampling the brighter parts is thus a good heuristic to estimate the lightness of the object. The advantage of this heuristic is that no knowledge about any high level visual aspects, such as geometry or shape, is required.

The visual system has been proposed to use several principles to resolve the ambiguity arising from the problem that local changes in the light reaching the eye could have different physical causes. Other than changes in reflectance and illumination, the geometric layout of the scene and relationships between occluding, transparent, and connecting surfaces can have dramatic effects (7, 11, 42). The visual system can only partially discount these factors to achieve lightness constancy, for example through the use of stereo cues (8, 42, 43) or relatively simple heuristics for interpreting luminance changes (10, 44). In our case, objects were uniformly colored and not occluded. The luminance gradients could only be caused by the interaction of the surface orientation and the light field. In this case the visual system is still faced with the problem of computing one level of lightness from a nonuniform pattern of brightness. One way the system could use to solve this task is to produce separate representations of the illuminant on the surface and its albedo (10). However, there is evidence that lightness judgments are, at least to some degree, directly based on brightness (24, 26). Along these lines, we propose that the brightness of the most illuminated region within the object is the source from which our visual system extracts a global impression of lightness. The use of the simple heuristic described here agrees with the finding that observers are impaired when they have to judge the lightness of a flat object, because they are unable to discount the effect of surface orientation (24). Flat surfaces show a reduced gradient of illumination generated by variations in the angle of the incident light, and the assumption that some portion of the object is always optimally illuminated does not hold. In such a situation, the heuristic we propose would produce a lightness estimate highly contaminated by object orientation, as observed (24).

Our results are not just of academic interest. Whenever surfaces lead to nonuniform distributions of light, by their texture or by the interactions of object reflectance, geometry, and illumination, eye movements might play a major role. Matching the color and lightness of food, hair, teeth, or textiles (45–47) are just a few examples. Every time we look at these objects we have the compelling impression that we can appreciate their global chromatic properties at a glance. We demonstrate that this impression is largely illusory, as our visual system reconstructs this global percept from local samples, smartly directing eye movements and attention to the areas where the most reliable information is to be found.

Methods

Stimuli.

In the first experiment, six real objects were used (Fig. 1B), made of different materials (wool, wax, and paper) and differing in shape and color. Each object was made from a uniform material, to ensure a uniform surface reflectance. All colors were within the range reproducible on our computer monitor. In the remaining experiments we used pictures of the objects of approximately the same size and chromatic properties as the real objects.

Spectroradiometric Measurements.

To ensure the alignment of spectral data and eye-tracking data, we superimposed the objects on a fixed reference grid. We measured the spectrum on a 22 × 34-cell grid. Cell size was 1.5° of visual angle, matching the spatial reliability of the eye tracking system. We used a Photo Research PR650 spectroradiometer with a spatial resolution of 1° of visual angle, measuring from the observer’s point of view.

Monitor.

The objects were placed in front of the CRT monitor where the matching disk was also displayed. The matching disk was placed in the upper right corner unless noted otherwise. The monitor was calibrated with the PR650 spectroradiometer.

Matching Task.

Unless noted otherwise, observers were instructed to “match the color of the object” by adjusting the color of a 5° visual angle disk. Adjustments were done in CIE L*C*h* color space. This space is basically the cylindrical representation of the CIE L*a*b* (48) color space with and . Observers could adjust the hue (h*), saturation (C*), and lightness (L*); they also could change the step size. In all of the experiments the observers provided six matches that were averaged for the analysis.

Eye-Movement Recording.

Gaze position signals were recorded with a head-mounted, video-based eye tracker (EyeLink II; SR Research) and were sampled at 500 Hz. Observers viewed the display binocularly but only the right eye was sampled. The eye tracker was calibrated at the beginning of each session. At the beginning of each trial the calibration was checked. If the error was more than 1.5° of visual angle a new calibration was performed; otherwise, a simple drift correction was applied. A calibration was accepted only if the validation procedure revealed a mean error smaller than 0.4° of visual angle.

Attention Experiment.

To investigate the potential role of covert spatial attention, we repeated the forced fixation experiment with gray-scale images. Visual covert attention was either coherently located at fixation or shifted to the point with opposite luminance. Observers performed a demanding rapid serial visual presentation (RSVP) letter-detection task at the attended location, which extended throughout the presentation of the object (3 s). Observers reported how many a characters appeared in the attended location, while keeping fixation. After the stimulus disappeared, they first reported whether the lightness of a comparison disk was higher or lower than that of the cone. The lightness of the disk was controlled adaptively. Subsequently, observers reported the number of targets. The RSVP was adjusted individually to ensure 50% performance in central and peripheral viewing, respectively. For each observer we computed points of subjectively equal lightness for all four conditions (all combinations of dark/light fixation and dark/light attention) by fitting cumulative Gaussians to the lightness judgment data.

Simulation.

For each object, we obtained 800 views combining eight equally spaced rotations of the light field on the equator plane and 100 random rotations of the object in 3D. We used an illumination map captured photographically from a real world scene (49) and three couples of reflectances to compare in the ROC analysis: {0.1670, 0.2330}, {0.4670, 0.5330}, {0.7670, 0.8330}. Only the reflectance and radiance at a wavelength of 570 nm were rendered and analyzed. Surfaces have been specified as perfectly Lambertian. For each object and each point of view, the luminance distributions were analyzed separately. Each distribution includes 100 object rotation samples. We computed the mean radiance of the pixels in each percentile. For each of the three pairs of reflectances, for each of the six objects, for each of the eight light field rotations, and for each percentile we performed a ROC analysis. We thus had 144 AUCs for each percentile. We aggregated these data in a cumulative AUC (37): namely, for each percentile we traced the function relating each possible value of AUC to the proportion of the classification performances above this value. The integral of this curve, called cumulative AUC, is an index of the classification performance (37). With this method we were able to aggregate data from several ROC curves, obtaining a single index of discriminability for each percentile.

Observers.

Six naive observers took part in the free-viewing experiment, four in the forced-fixation experiment, and 12 in the covert-attention experiment. All of the observers were paid undergraduate students. All observers provided written informed consent. The study was approved by the local ethics committee LEK FB06 at Giessen University (2009-0008). All observers had normal or corrected-to-normal vision and normal color vision as tested with Ishihara plates (50).