Visual perception

Abstract

How do humans interpret visual input to estimate the properties of a surface? In the case of estimation of gloss and lightness, it seems that neural discrimination of simple image statistics plays a large part.

How do you tell the difference between peaches and nectarines, or between unfinished and polished wood? There are many visual attributes that help us to distinguish different surface materials, including lightness, colour and texture. The salient attribute shared by nectarines and finished wood is a mirror-like (specular) component of its reflectance, which is perceived as gloss or shininess. Motoyoshi and colleagues (page XXX of this issue)¹ have made a surprising discovery concerning surface perception: a simple characteristic of image statistics — the distribution of luminance values in an image, the ‘skew’ — is highly correlated with judgements of gloss and lightness. The principle can be illustrated by the manipulation of a picture in which a nectarine has been visually transformed to look more like a peach by removing a highlight (Fig. 1).

Figure 1. Highlights from the fruit bowl.

a, This photograph is a composite of two images, with most of the composite being a photo with highlights from a bright light source coming from the upper left. But the central fruit, which is in fact a nectarine, comes from a second photo in which the glossy highlights were removed by putting a polarizing filter on the light source and a crossed polarizer on the camera. So this normally glossy nectarine looks more like a matte peach. b, An unmanipulated image of the same collection of fruit with all highlights present. c, Motoyoshi and colleagues¹ show how the visual system can estimate gloss, or lack of it, from the amount of positive skew in the distribution of luminance in an image. The luminance histogram of the nectarine in the center of the fruit bowl is more positively skewed with the highlight than without it.

The authors took calibrated photographs of stucco-like materials varying in albedo (the amount of black pigment in the material) and gloss (the amount of clear acrylic media coating), and found that as gloss was increased, or as albedo was reduced for a glossy surface, the luminance distribution became positively skewed (see Fig. 2a of the paper on page 000). In other words, images of glossy materials are predominantly dark, with occasional bright highlights (Fig. 1c). They found that human visual judgements of glossiness and lightness were correlated with histogram skew for the stucco images, as well as for photographs of other natural materials. More importantly, simply skewing the histogram of a photograph of a material caused the surface to appear glossier and darker. Finally, they found that if observers adapted to an image with positive skew, a subsequently viewed surface appeared less glossy (with the opposite result for adaptation to negative skew), indicating that humans extract something like luminance skew from images.

This finding is consistent with other work showing that humans are sensitive to image statistics for a variety of judgments. In addition to gloss, perceived surface roughness and translucency also depend on image statistics^2–4. Skew is an example of one statistic derived from the luminance histogram. But humans are sensitive to at least three statistics of the histogram^5,6. Perceived brightness and contrast correspond roughly to the mean and variance of luminance^7,8. Although in early work^9,10 luminance statistics were found to be insufficient to account for the discriminability of texture patterns, more recent studies indicate that humans are sensitive to the statistics of responses of bandpass filters (e.g., the responses of simple cells in primary visual cortex) to images both for texture discrimination¹¹ and appearance^12,13.

How might the visual system compute statistics such as histogram skew? The initial coding involves spatial linear filtering, which is carried out by various parts of the visual system: the centre-surround receptive fields of ganglion cells in the retina; cells in the lateral geniculate nucleus region of the brain; and the orientation-tuned receptive fields of simple cells in the primary visual cortex. Histogram statistics, and skew in particular, could be recovered from the cells with centre-surround receptive fields, for which darkness and brightness information are separately represented by Off and On channels. Motoyoshi and colleagues¹ simulated such a model. Alternatively, such statistics might be recovered from the responses of orientation-selective simple cells in primary visual cortex⁵.

Why should positive histogram skew result both in increased perception of gloss and an apparent darkening of the surface? Many perceptual capabilities are described in terms of ‘discounting’. For example, colour constancy refers to the ability, albeit incomplete, of observers to estimate surface colour independent of the spectral power distribution of the illuminant, thus discounting the illuminant in the interpretation of the retinal signal¹⁴. When a histogram is positively skewed, apparent glossiness is increased. Thus, pixels in the positive tail of the luminance distribution are interpreted as highlights (mirror reflections of the illuminant), and then discounted in interpreting surface lightness¹⁵. Lightness then becomes a function of the remaining, darker pixel values. This explains why increase in perceived glossiness is often associated with decreased lightness.

However, there is another possible explanation of the correlation of image skew with judgements of both gloss and lightness. Parameters of a luminance histogram (mean, variance, skew and so on) are convenient mathematically, but may not correspond precisely to the computations used in making perceptual judgments. In fact, luminance variance is not the form of nonlinearity used by humans for estimates of image contrast⁷. If the impact of different luminance levels on judgments of glossiness were directly measured, one might find that a different nonlinearity (other than skew) is computed, such as the ‘blackshot mechanism’^5,6 that was, by design, orthogonal to the computation of mean luminance, and hence should not correlate with judgements of lightness. It remains to be seen how one can determine the perceptually relevant quantity for estimation of gloss.

Histogram statistics are not the whole story for the perception of lightness, contrast or gloss. Perceived lightness and contrast of a surface depend in a complex way on the surrounding surfaces¹⁶. For an image to appear glossy, it has to first look like a surface. As Motoyoshi et al. point out, the mere presence of a positively skewed histogram is not enough. If an image is modified by randomly permuting its pixels, or by giving random phase values to its sine wave components, the resulting image may have positively skewed luminance statistics, but will not look like a surface, so the rare, bright pixels will not look like highlights.

For a surface to appear glossy, not only must it include a specular reflectance, but the surroundings must result in a pattern of illumination consistent with the statistics of natural scenes^17,18. There are many physical dimensions of gloss that affect the perception of surface material. The first, studied by Motoyoshi and colleagues, is the percentage of ambient light that is reflected in the mirror direction. A second is the degree to which the specular reflection is point-like or blurred (for example in the case of polished versus brushed metal). Its effect on perception has not been studied systematically. But although histogram skew does not explain everything about the perception of surface material, or even of gloss, it is a major step towards a theory of the perception of surface materials.