The statistical structure of natural light patterns determines perceived light intensity

Abstract

The same target luminance in different contexts can elicit markedly different perceptions of brightness, a fact that has long puzzled vision scientists. Here we test the proposal that the visual system encodes not luminance as such but rather the statistical relationship of a particular luminance to all possible luminance values experienced in natural contexts during evolution. This statistical conception of vision was validated by using a database of natural scenes in which we could determine the probability distribution functions of co-occurring target and contextual luminance values. The distribution functions obtained in this way predict target brightness in response to a variety of challenging stimuli, thus explaining these otherwise puzzling percepts. That brightness is determined by the statistics of natural light patterns implies that the relevant neural circuitry is specifically organized to generate these probabilistic responses.

The perception elicited by the luminance of a visual target, generally called brightness, is arguably the most basic quality of vision. A central puzzle in understanding how such percepts are generated by the visual system is that brightness does not correspond in any simple way to luminance. Thus, the same amount of light arising from a given region in a scene can elicit dramatically different brightness percepts when presented in different contexts (1, 2) (Fig. 1).

Fig. 1.

The influence of spatial patterns of luminance on the apparent brightness of a target [the targets (T) in each stimulus are equiluminant and are indicated in Right Insets]. (A) Standard simultaneous brightness contrast effect. The central square in the dark surround appears brighter than the equiluminant square in the light surround. (B) White's illusion. Although the gray rectangles in the left stimulus are all equiluminant, the ones surrounded by the generally lighter context (left side of the stimulus) appear brighter than those surrounded by the generally darker context (right side of the stimulus). When, however, the luminance of the target rectangles is the lowest (center stimulus) or highest (right stimulus) value in the presentation, the targets in the generally lighter context appear somewhat less bright than ones in the generally darker context (called the “inverted White's effect”). (C) Wertheimer–Benary illusion. The triangle embedded in the arm of the black cross appears brighter than the one that abuts the corner of the cross. The slightly different brightness of the equiluminant triangles is maintained whether the presentation is upside down (Center) or reflected along the diagonal (Right). (D) The intertwined cross illusion. The target in the left stimulus appears substantially brighter than the equiluminant target in the right stimulus. (E) The inverted T illusion. The inverted T shape in the left stimulus appears somewhat brighter than the equiluminant target in the right stimulus.

A variety of explanations have been suggested since the basis for such phenomena was first debated by Helmholtz, Hering, Mach, and others. Although lateral inhibition in early visual processing has often been proposed to account for these “illusions” (1), this mechanism cannot explain instances in which similar overall contexts produce different brightness effects (compare Fig. 1 A with Figs. 1 B and E; see also Fig. 1C). This failure has led to several more recent suggestions, including complex filtering and neural network models (3, 4), the idea that brightness depends on detecting edges and junctions that promote the grouping of various luminances into interpretable spatial arrangements (5–11), and the proposal that brightness is “resynthesized” from 3D scene properties “inferred” from the stimulus (12–14). None of these approaches, however, can explain the full the range of brightness phenomena illustrated in Fig. 1 (1).

Here we examine a different concept of the way brightness is generated by the visual system. A growing body of evidence has shown that the visual system uses the statistics of stimulus features in natural environments to generate the visual percepts of the physical world (15); if so, the visual system must incorporate these statistics as a central feature of processing relevant to brightness and other visual qualia (2). Accordingly, we suppose that the perceived brightness elicited by the luminance of a target in any given context is based on the value of the target luminance in the probability distribution function of the possible values that co-occur with that contextual luminance experienced during evolution. In particular, whenever the target luminance in a given context corresponds to a higher value in the probability distribution function of the possible luminance values in that context, the brightness of the target will be greater than the brightness elicited by the same luminance in contexts in which that luminance has a lower value in the probability distribution function.

A large set of images of natural scenes (16) was used to approximate the range of visual stimuli experienced by humans in natural environments. From this database, we obtained the probability distribution functions of target luminance in contextual luminance patterns similar to those in Fig. 1. The predictions of brightness made on the basis of these probability distributions explain the full range of these phenomena, strongly supporting the hypothesis that brightness percepts are based on instantiation in the visual processing circuitry of the statistical structures of light patterns experienced in natural environments.

Materials and Methods

Statistical Framework. Natural environments comprise objects of different sizes at various distances that are physically related to each other and the observer in a variety of ways (17, 18). When the light arising from objects is projected onto an image plane, these complex relationships are transformed into 2D patterns of light intensity with highly structured statistics. As a result, the luminance at any location in a pattern of light arising from natural scenes will have a characteristic distribution. A corollary is that in such scenes the probability distribution of the luminance of, say, the central target in a standard simultaneous brightness contrast stimulus (Fig. 2A) will depend on the surrounding luminance values (Fig. 2B).

Fig. 2.

Statistical framework for understanding the generation of brightness percepts. (A) The brightness elicited by a given target luminance in any context depends on the frequency of occurrence of that luminance relative to all of the possible target luminance values experienced in that context in natural environments. This concept is illustrated here by using the standard simultaneous brightness contrast stimulus in Fig. 1 A. The series of squares with different luminance values indicate all of the possible occurrences of luminance in the target (T) in the two different contexts; the symbol (∋) indicates the relationship of a particular occurrence of luminance to the all possible occurrences of target luminance values experienced in the two contexts in natural environments. (B) This statistical relationship can be derived from the probability distribution density of target luminance values co-occurring with the luminance pattern of the two contexts of interest. The red and blue curves indicate these probability densities of the luminance of the targets in A, obtained by sampling the natural image database. The size of the sampling configuration was 1° × 1° (see supporting information). In this example, the most likely luminance values of the targets in the distributions are the same as the mean luminance of the corresponding surrounds. (C) The brightness elicited by the luminance of the targets in A is based on the percentile of that luminance in the probability distribution functions (i.e., the integrals of the probability densities in B) for the two different contexts, which are indicated by the icons.

Fig. 2C illustrates the supposition that, for any context, the visual system generates the brightness of a target according to the value of its luminance in the probability distribution function of the possible target luminance experienced in that context. This value is referred to subsequently as the percentile of the target luminance among all possible luminance values that co-occur with the contextual luminance pattern in the natural human environment. In formal terms, this supposition means that the visual system generates brightness percepts according to the relationship Brightness = AΦ(P)+A₀, where A and A₀ are constants, and Φ(P) is a monotonically increasing function of the probability distribution function P.

By definition, then, the percentile of target luminance for the lowest luminance value within any contextual light pattern is 0% and corresponds to the perception of maximum darkness; the percentile for the highest luminance within any contextual pattern is 100% and corresponds to the maximum perceivable brightness. In any given context, a higher luminance will always have a higher percentile and will always elicit a perception of greater brightness compared to any luminance that has a lower percentile. Because the relation Brightness = AΦ(P)+A₀ is based not on a particular luminance within the context in question but rather on the entire distribution of possible luminance values experienced in that context, the context-dependent relationship between brightness and luminance is highly nonlinear (see Fig. 2C). In consequence, the same physical difference between two luminance values will often signify different percentile differences and thus perceived differences in brightness. Furthermore, because the percentiles change more rapidly as the target luminance approaches the luminance of the surround, one would expect greater changes of brightness, an expectation that corresponds to the well known “crispening” effect in perception (19).

Finally, because the same value of target luminance will often correspond to different percentiles in the probability distribution functions of target luminance in different contexts, two targets having the same luminance can elicit different brightness percepts, the higher percentile always corresponding to a brighter percept. Thus, in the standard simultaneous brightness contrast stimulus in Fig. 1 A, the target (T) in Fig. 2 A Left appears brighter than the equiluminant target in Fig. 2 A Right.

Obtaining Conditional Probability Distribution Functions. The relevant probability distribution functions were obtained by sampling a database of natural scenes (ref. 16; http://hlab.phys.rug.nl) with target-surround configurations that had the same local geometry as the stimuli in Fig. 1. As a first step, these configurations were superimposed on the images to find light patterns in which the luminance values of both the surround and target regions were approximately homogeneous (see supporting information, which is published on the PNAS web site, for further details); for those configurations in which the surround comprised more than one region of the same luminance (see Fig. 1), we also required that the relevant sampled regions meet this criterion. The sampling configurations were moved in steps of one pixel to screen the full image, in this way obtaining a large number of samples that met the stipulated criteria. The mean luminance values of the target and the surrounding regions in the samples were then calculated and their occurrences tallied in the form of histograms. Given the specific surround luminance values similar to those shown in Fig. 1, the probability distribution functions of the target luminance were obtained from the histograms.

Results

White's Illusion. White's illusion (Fig. 1B), which has no generally accepted explanation, presents a particular challenge for any explanation of brightness (20–24). The equiluminant rectangular areas surrounded by predominantly more luminant regions in the stimulus appear brighter than areas of identical luminance surrounded by less luminant regions (Fig. 1B Left). The especially perplexing characteristic of this percept is that the effect is opposite that elicited by standard simultaneous brightness contrast stimuli (Fig. 1 A). Even more puzzling, the effect reverses when the luminance of the rectangular targets is either the lowest or highest value in the stimulus (21, 22) (Fig. 1B Center and Right).

The explanation for White's illusion provided by the statistical framework outlined above is illustrated in Fig. 3. When presented separately, as in Fig. 3A, the components of White's stimulus elicit much the same effect as in the usual presentation. By sampling the images of natural visual environments using configurations based on these components (Fig. 3B) (see supporting information), we obtained the probability distribution functions of the luminance of a rectangular target (T) embedded in the two different configurations of surrounding luminance in White's stimulus. As shown in Fig. 3C, when the target in the intermediate range of luminance values (i.e., between the luminance values at the two crossover points) abuts two dark rectangles laterally (Fig. 3B Left), the percentile of the target luminance (red line) is higher than the percentile when the target abuts the two light rectangles (Fig. 3B Right; blue line in Fig. 3C). If, as we suppose, the percentile in the probability distribution function of target luminance within any specific context determines the brightness perceived, the target with an intermediate luminance in Fig. 3B Left should appear brighter than the equiluminant target in Fig. 3B Right. Moreover, because the difference between the percentiles of the same luminance in the two different contexts is relatively large for this standard set of luminance values in White's stimulus, the brightness percepts elicited should be quite different, as they are. Finally, when all of the luminance values in the stimulus are limited to a very narrow range (e.g., from 0 to 100 cd/m² or from 1,000 to 1,100 cd/m²), when the sampling configurations are orientated vertically, or when the aspect ratio of the sampling configurations is changed (e.g., from 1:2 to 1:5), the probability distribution functions derived from the database are not much different. These further results are consistent with the observations that White's stimulus elicits much the same effect when presented at a wide range of overall luminance levels, in a vertical orientation, or with different aspect ratios (21).

Fig. 3.

Statistical explanation of White's illusion. (A) The usual presentation of White's illusion; boxed areas indicate the basic components of the stimulus, which elicit about the same effect as the usual presentation. (B) The sampling configurations used to obtain the probability distribution functions of target luminance (the red and blue rectangles), given a pattern of surrounds with luminance values L_u and L_v (size of the sampling configuration in this example was 0.6°[H]×0.3°[V]). (C) The probability distribution functions of the luminance of the targets in these contexts (red curve: L_u = 145, L_v = 105; blue curve: L_u = 105, L_v = 145). Here and in Figs. 4 and 5, surround luminance values were chosen in the middle range of the values in the database to ensure sufficient samples to fairly assess variations in natural luminance. For the middle luminance values lying within the two crossover points (at ≈105 and 145), the red curve is above the blue curve; as a result, the luminance configurations in B generate White's illusion [as indicated (Insets)]. For other luminance values of the target, the blue curve is above the red curve; as a result, the luminance configurations in B generate the inverted White's effect. (D) Examples from the database illustrating the most likely luminance value of the target in B, given the contextual luminance indicated by the icon (Left). Because the most likely target luminance is similar to that of the relevant part of the surround, the target does not “pop out” of the scene here or in Figs. 4 and 5.

An aspect of White's illusion that has been particularly difficult to explain is the so-called “inverted White's effect”: when the target luminance is either the lowest or the highest value in the stimulus, the effect is actually opposite the usual percept (21, 22) (see Fig. 1B). The explanation for this further anomaly is also evident in Fig. 3C. When the target luminance is the lowest value in the presentation (see Fig. 3C Insets), the blue curve is above the red curve. As a result, a relatively dark target surrounded by more light area should now appear darker, as it does (see also Fig. 1B Center). By the same token, when the target luminance is the highest value in the stimulus (see Fig. 3C Insets), the blue curve is also above the red curve. Accordingly, the relatively light target surrounded by more dark area should appear lighter, as it does (see also Fig. 1B Right). Thus the statistical structure of natural light patterns predicts not only White's illusion but the inverted White's effect as well. Notice further that the two crossover points of the blue and red curves shift to the right when the contextual luminances increase and to the left when they decrease; thus the inverted effect will be apparent, although altered in magnitude, for any luminance values of the surrounding areas.

Once the probability distribution function of target luminance had been obtained, we could also determine the most likely target luminance in natural environments, given the contextual luminance patterns in the basic components of the standard White's stimulus. Fig. 3D shows examples from the database corresponding to the most often encountered target luminance in the contextual luminance patterns illustrated in Fig. 3D Left. When the upper and lower bars are relatively light and bars that abut the target laterally dark (upper row), the most likely luminance of the target is relatively low and similar to that of middle bars. Thus, when the target luminance is higher than the luminance most frequently experienced at that location in that context, the target should appear brighter (because the target has a higher percentile than the most probable luminance on the red curve in Fig. 3C). Conversely, when the upper and lower bars are relatively dark and middle bars light (lower row), the most probable luminance of the target is similar to that of middle bars and is relatively high. Accordingly, when the target has a luminance that is lower than the luminance most often experienced at that location in that context, it should appear darker (because the target now has a lower percentile than the most probable luminance on the blue curve). The basis for all of the effects elicited by White's stimulus is thus the characteristic co-occurrence of luminance in natural environments.

These characteristic natural statistics (which we found to be scale invariant in all of the analyses reported here) appear to explain all of the peculiar phenomenology of White's effect. The results, however, should not be taken to imply that the visual system needs to group luminance patches into particular spatial arrangements to generate brightness, as has often been suggested (5–11). It should also be apparent from Fig. 3D that, in agreement with other recent evidence (23, 24), the natural luminance patterns that give rise to the probability distribution functions in Fig. 3C are rarely configurations that have straight edges, well-defined junctions, and/or occlusions, all of which have been suggested to be essential in this and other brightness illusions. We should further emphasize that the behavior of the probability distribution functions in Figs. 2, 3, 4, 5, 6 depends on the occurrences of all the possible luminance values in the relevant contexts, regardless of whether the test region is of higher or lower luminance than the surrounding region in any particular occurrence. This behavior cannot therefore be derived from the most probable target luminance in the relevant contexts or from processing any particular stimulus with Gaussian, Laplacian, or Gabor filters. It is also worth pointing out that the brightness of any region in the stimuli in Fig. 1 is determined in the same way; the reason the major effect is on the target rather than the surround is simply that the contexts of the surround (i.e., the rest of page or computer screen) do not shift the distribution of the luminance values of the surrounds very much. These several points are further considered in the Discussion.

Fig. 4.

Statistical explanation of the Wertheimer–Benary illusion. (A) The usual presentation of the Wertheimer–Benary stimulus. As in White's stimulus, the components of the stimulus (boxed areas) elicit about the same effect as the usual presentation. (B) Configurations used to sample the database (size = 0.4° × 0.4°). Due to the shapes of the local contexts, the geometries of the two sampling configurations in this case necessarily differ (see supporting information). (C) The probability distribution functions of target luminance, given the surrounding luminances in B. The red curve corresponds to the conditions shown in B Left (L_u = 205, L_v = 45) and the blue curve to the conditions shown in B Right (L_u = 45, L_v = 205). (D) Examples from the database illustrating the most likely luminance value of the targets in B, given the contextual luminance indicated by the icon (Left).

Fig. 5.

Statistical explanation of the intertwined cross illusion. (A) Configurations used to sample the database (size = 0.6° × 0.6°). (B) The probability distribution functions of target luminance for the configurations in A. The red curve corresponds to the condition shown in A Left (L_u = 75, L_v = 125, L_w = 100) and the blue curve to the condition shown in A Right (L_u = 175, L_v = 125, L_w = 150). (C) Examples from the database illustrating the most likely luminance value of the target in A, given the contextual luminance indicated by the icon (Left).

The Wertheimer–Benary Illusion. In the Wertheimer–Benary illusion (Fig. 1C), equiluminant gray triangles, which unlike the targets in White's illusion have similar local contexts, appear differently bright, the triangle in the corner of the cross looking slightly darker than the triangle embedded in arm of the cross. Like White's illusion, the Wertheimer–Benary illusion has no satisfactory explanation.

The explanation of the Wertheimer–Benary illusion in the statistical framework considered here is illustrated in Fig. 4. Fig. 4A shows that, when presented separately, the basic components of the Wertheimer–Benary stimulus elicit much the same effect as in the usual presentation. By sampling the images of natural environments using configurations based on these components (Fig. 4B), we obtained the probability distribution functions of target luminance in these contexts. As shown in Fig. 4C, when the triangular patch is embedded in a dark bar with its base facing a lighter area, the percentile of the luminance of the triangular patch (red line) is always higher than the percentile when the triangular patch abuts a dark corner with its base facing a similar light background (blue line). Accordingly, the same gray patch should always appear brighter in the former context than in the latter, as is the case. Moreover, because the typical difference between the percentiles here is less than the difference in White's illusion at comparable surrounding luminance values, the Wertheimer–Benary effect should not be as strong as White's illusion, as is also the case. The probability distribution functions obtained after changing the triangles to rectangles, rotating the configurations in Fig. 4B by 180°, or reflecting the configurations along the diagonal of the cross (see Fig. 1C Center and Right) were much the same as those shown in Fig. 4C. These several observations accord with the fact that the Wertheimer–Benary effect is little changed by such manipulations.

As in the analysis of White's stimulus, we could also examine the most frequently encountered target luminance values in natural environments, given the contextual luminance patterns in the Wertheimer–Benary illusion. Fig. 4D illustrates the most likely target luminance encountered in natural scenes in which the contextual luminance values are similar to the basic components of the Wertheimer–Benary stimulus. When the triangular patch is embedded in a dark vertical bar with its base abutting a light area, a relatively low luminance similar to that of the dark vertical bar is likely to coincide with the position of the triangle. When, however, the triangular patch lies in the corner of the dark cross with its base abutting a light background, a higher luminance similar to that of the light background is likely in that location. Thus when the two triangles in these configurations are presented as equiluminant gray patches, as in the Wertheimer–Benary illusion, the lower triangle, which occupies a lower percentile on the blue curve, should appear darker, as it does.

More Complex Brightness Illusions. The brightness percepts elicited by other more complex luminance patterns are equally well explained by this statistical framework. For example, the brightness difference of two equiluminant targets is much enhanced by the specially configured, intertwined contexts in Fig. 1D (10). Fig. 5 shows the statistical basis of this further phenomenon. When a rectangular area is surrounded by a pattern of luminance configured as in Fig. 5A Left, the percentile of any luminance of that target is far greater (red curve in Fig. 5B) than the percentile for the same target luminance when the surrounding luminance pattern is configured as in Fig. 5A Right (blue curve in Fig. 5B). Accordingly, the target in Fig. 5A Left should appear brighter than the equiluminant target in Fig. 5A Right. Moreover, because the difference between the percentiles in the two contexts is much larger than the difference for the Wertheimer–Benary illusion with comparable surrounding luminance values, the difference in brightness elicited by the same target luminance in the two contexts should be much greater, as it is. Fig. 5C, as Figs. 3D and 4D, shows examples from natural image database that correspond to the most likely target luminance, given the configurations in Fig. 5C Left. This framework can also explain some especially subtle brightness effects such as Fig. 1E that are otherwise extraordinarily difficult to rationalize (see supporting information).

Discussion

These results show that the probability distributions of naturally co-occurring luminance values can account for the brightness percepts generated by a variety of stimuli whose consequences have been difficult to explain in other ways.

The Statistical Nature of Perception. Although studies of brightness perception now span more than 100 years, this phenomenon has never been considered in statistical terms; indeed, there has been very little analysis at the “computational theory level” (25). Here, we show that brightness percepts encode not luminance as such but rather the statistical relationship between the luminance in an area within a particular contextual light pattern and all possible occurrences of luminance in that context experienced by humans in natural environments.

The statistical basis for this aspect of visual perception is quite different from traditional approaches to rationalizing brightness. In the “relational approach” (26), an idea that evolved from the late 19th century debate between Helmholtz, Hering, and others, brightness percepts are “recovered” by the visual system from explicitly coded luminance contrasts and gradients. Another idea that has recently gained ground is that brightness depends on intermediate-level visual processes that detect edges, gradients, and junctions, which are then grouped into specific spatial layouts to allow an appropriate interpretation of the scene (5–11). Finally, the brightness elicited by a given luminance has also been considered to be “resynthesized” by processing at several levels of the visual system that is based on inferences about the possible arrangements of surfaces in 3D, their material properties, and their illumination (12–14). These various approaches, however, cannot account for the range of phenomena illustrated in Fig. 1 (as well as other related effects) (1). As a result, the debate initiated by Helmholtz, Hering, Mach, and others remains current.

The common deficiency of these various ways of thinking about brightness is their failure to relate the statistics of light patterns experienced in the course of evolution to what the corresponding brightness percepts need to signify (namely, the relationship of a particular occurrence of luminance to all possible occurrences of luminance in a given context). Because light patterns on the retina are the only information the visual system receives, basing brightness percepts on the statistics of natural light patterns allows visual animals to deal optimally with all possible natural occurrences of luminance, using the full range of perceivable brightness to represent the physical world. This statistical concept of perception and its neural mechanisms (see below) has deep roots (27, 28) and has recently gained considerable support (15, 29–33).

Segmentation and Grouping. The use of a set of spatial configurations specific to the phenomena in Fig. 1 to predict brightness percepts on a probabilistic basis obviously does not address how the brain computes and represents the relevant statistics, or how it relates them to perception. A prevalent intuition in many studies has been that to generate perceptions of brightness, the visual system must first detect edges and junctions and then appropriately group various luminance patches (5–11). The results we report here do not support this view. Segmentation and grouping neither address the statistical nature of perception nor provide the means to compute these statistics from a set of natural stimuli. By the same token, given a particular stimulus, the percentile of any luminance in that stimulus in the relevant probability distribution function is determined. Thus “knowledge” about background and foreground or edges generated by reflectance or illumination is irrelevant to a determination of the percentile of the luminance values in the relevant probability functions. Accordingly, these functions, which predict perception, cannot be derived from segmentation and grouping. Indeed, because such concepts, like brightness, are meaningful only in a probabilistic sense, the statistics that generate brightness are the basis for segmentation and grouping, not the other way around.

Neural Instantiation of Natural Statistics. What sort of neural mechanisms, then, could incorporate these statistics of natural light patterns and relate them to brightness percepts? Although the answer is not known, the present results suggest that the circuitry at all levels of the visual system instantiates the statistical structures of light patterns in natural environments.

In this conception, the center-surround organization of the receptive fields of retinal ganglion cells (34) provides the initial basis for representing the necessary statistics. A further speculation would be that neural circuitry at the level of the visual cortex is organized to instantiate the statistics of luminance patterns with arbitrary target and context shapes and sizes within an appropriate range. These statistical structures at the cortical level would be functionally similar to the adaptive deformable templates that have been used successfully in computational studies of pattern recognition (35). Given the statistical regularity of natural visual environments, the number of templates needed for this task is necessarily limited. When a visual stimulus is presented, the luminance at and around any location would drive the system toward one of the instantiated statistical structures, perhaps in the way that associative memory is generated by attractor dynamics (36). As a result, the neuronal response at each location would signify the percentile of the target luminance in the probability distribution function pertinent to a given context.

A good deal of physiological evidence accords with this general concept of visual brain function. For example, neuronal responses are strongly modulated by context (37), and many perceptual qualities have neuronal correlates in the primary visual cortex (38–40). Finally, other evidence supports the idea that neuronal responses are closely related to the statistical characteristics of naturally occurring stimuli (41–43).

Despite the rudimentary nature of these speculations about the way the visual system elaborates percepts, the strength of the evidence here that brightness is generated on the basis of statistics of natural light patterns as they pertain to consequent behavior implies that the relevant visual circuitry will eventually need to be understood in these terms.