Color and material perception: Achievements and challenges

Abstract

There is a large literature characterizing human perception of the lightness and color of matte surfaces arranged in coplanar arrays. In the past ten years researchers have begun to examine perception of lightness and color using wider ranges of stimuli intended to better approximate the conditions of everyday viewing. One emerging line of research concerns perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or coplanar and with geometrically complex illumination. A second concerns the perception of material surface properties other than color and lightness, such as gloss or roughness. This special issue features papers that address the rich set of questions and approaches that have emerged from these new research directions. Here, we briefly describe the articles in the issue and their relation to previous work.

Introduction

The classic perceptual correlates of object surface properties are lightness and color. Although there are notable exceptions (e.g., Gilchrist, 1977; Hochberg & Beck, 1954; Kardos, 1934; Mach, 1886/1959; for review, see Gilchrist, 2006), the preponderance of what we know about perception of these attributes comes from the study of simplified stimuli, flat–matte surfaces placed on a single plane under diffuse illumination. After more than a century, researchers have developed an accurate if still incomplete outline of how the human visual system assigns lightness and color descriptors in such “flat–matte” scenes (for reviews, see Brainard, 2009; Shevell & Kingdom, 2008).

Over the past decade, research has increasingly begun to focus on two rich generalizations of the classic “flat–matte” paradigm (for reviews, see Adelson, 2008; Maloney, Gerhard, Boyaci, & Doerschner, 2011). The first generalization concerns the perception of lightness and color in scenes that approximate the three-dimensional environment we live in, with objects that need not be matte or flat and geometrically complex illumination. The second generalization concerns perception of material surface properties other than lightness and color, such as gloss or roughness. Both lines of move us toward the goal of understanding object surface perception as it operates in natural viewing. At the same time, the research presents novel challenges as we try to formulate theories that account for how the perception of lightness, color, gloss, roughness, and other material properties interact with one another as well as with object shape, object pose, and illumination geometry.

In 2004, we edited a special issue in the Journal of Vision entitled “Perception of color and material properties in complex scenes” (Brainard & Maloney, 2004). This special issue is its sequel. Here, we briefly describe the articles in this issue. We have organized the overview according to broad themes. We have also set the current work in the context of its antecedents, with particular emphasis on work published since 2004. Overall, the current special issue illustrates the rich set of questions and promising approaches that are driving current interest in this maturing area of inquiry.

Characterizing, estimating, and discriminating the light field

One of the most important themes in color and material perception is the role of scene illumination. In flat–matte scenes, the placement and directionality of light sources was little emphasized (Boyaci, Doerschner, Snyder, & Maloney, 2006). In everyday scenes, the light field need not be simple (Adelson & Bergen, 1991; Gershun, 1939) and material properties such as gloss or roughness are easier to assess in scenes with geometrically rich light fields (Fleming, Dror, & Adelson, 2003).

Earlier in the decade, several papers (Koenderink, Pont, van Doorn, Kappers, & Todd, 2007; Pont & Koenderink, 2007; Rutherford & Brainard, 2002) directly assessed how observers judge the light field and changes in the light field (see also Foster & Nascimento, 1994). In addition, Dror, Leung, Adelson, and Willsky (2001) and Dror, Willsky, and Adelson (2004) measured the statistical properties of lighting in natural scenes while Doerschner, Boyaci, and Maloney (2007) tested whether human matte surface color perception can compensate for the potential complexity of light fields in natural scenes.

In the current issue, three papers continue this theme. Schofield, Rock, Sun, Jian, and Georgson (2010) examine how human observers discriminate between changes in scene lighting and scene contents. They propose a special role for mechanisms associated with second-order spatial vision in discriminating illumination changes from reflectance changes. Gerhard and Maloney (2010a) measure how observers discriminate between changes in scene illumination and reflectance in three-dimensional scenes, under stimulus conditions where discrimination based on local illumination and monocular cues is not possible. They demonstrate that detection of a lighting change leads to enhanced ability to detect a simultaneous change in surface reflectance. Gerhard and Maloney (2010b) propose a model of light change detection, based on earlier work by Pentland (1982), and show that this model accounts for their data in detail.

Complex light fields and surface color/lightness perception

In parallel with direct assessment of the perception of the light field, the past decade has seen a slew of papers that study how the visual system achieves color and lightness constancy in the context of spatially complex light fields (e.g., Bloj & Hurlbert, 2002; Bloj, Kersten, & Hurlbert, 1999; Bloj et al., 2004; Boyaci, Doerschner, & Maloney, 2004; Boyaci et al., 2006; Boyaci, Maloney, & Hersh, 2003; Hedrich, Bloj, & Ruppertsberg, 2009; Kraft, Maloney, & Brainard, 2002; Ripamonti et al., 2004; Robilotto & Zaidi, 2004; Snyder, Doerschner, & Maloney, 2005; Todd, Norman, & Mingolla, 2004; Werner, 2006; Yang & Maloney, 2001; Yang & Shevell, 2002; Zaidi & Bostic, 2008). Important earlier work includes Gilchrist (1977, 1980), Hochberg and Beck (1954), Ikeda, Shinoda, and Mizokami (1998), and Pessoa, Mingolla, and Arend (1996).

Along these lines in the current issue, Radonjić, Todorović, and Gilchrist (2010) examine surface lightness perception in three-dimensional scenes with directional lighting and show how grouping principles such as adjacency and surroundedness can help organize the empirical phenomena. Olkkonen, Witzel, Hansen, and Gegenfurtner (2010) study color categorization for real surfaces and daylight illuminants. An entire room with controlled illumination served as the laboratory. They find that color categorization was little changed by marked changes in daylight illumination.

Surface material perception: Gloss, roughness

A very active area of research is the assessment of lighting and environmental conditions that affect the perception of material properties such as gloss and roughness, and how these properties interact with the perception of color and lightness. Important early work includes Beck and Prazdny (1981) and Nishida and Shinya (1998). Pellacini, Ferwerda, and Greenberg (2000) used scaling methods to determine the perceptual dimensions of gloss perception using stimuli rendered via computer graphics, as did Obein, Knoblauch, and Viénot (2004) using stimuli consisting of real illuminated objects.

Fleming et al. (2003) measured asymmetric matching performance with stimuli rendered under different real-world illuminants. Berzhanskaya, Swaminathan, Beck, and Mingolla (2005) studied how the perception of gloss propagates from highlights across the surface of three-dimensional objects.

Motoyoshi, Nishida, Sharan, and Adelson (2007; Sharan, Li, Motoyoshi, Nishida, & Adelson, 2008; for review, see Adelson, 2008) proposed a model of material perception based on statistical moments of the luminance pixel and sub-band histograms of images, most notably histogram skewness. Anderson and Kim (2009), however, questioned whether histogram skewness provided significant explanatory power in the absence of explicit consideration of surface geometry.

Ho, Landy, and Maloney (2006; Ho, Maloney, & Landy, 2007) used forced-choice methods to examine how texture (roughness) perception is affected by scene illumination and observer viewpoint. Emrith, Chantler, Green, Maloney, and Clarke (2010) used scaling methods to investigate perception of roughness.

Most recently, Doerschner, Boyaci, and Maloney (2010) tested whether human observers have self-consistent perception of gloss and show that they do when the percepts are assessed using forced-choice, but not asymmetric matching, methods.

Surface material perception is represented by a number of papers in this issue; Kim and Anderson (2010) extend their work on the limits of the explanatory power of simple luminance histogram statistics. Wijntjes and Pont (2010) systematically investigate interactions between relief height (as reported by Ho, Landy, & Maloney, 2008) and histogram skewness (Motoyoshi et al., 2007) and develop a model of the physical factors that determine perceived glossiness. They reject the skewness hypothesis in favor of a novel hypothesis they develop.

Sakano and Ando (2010) examine the effect of binocular disparity and head movement and find that both factors enhance perceived glossiness. Wendt, Faul, Ekroll, and Mausfeld (2010) likewise explore the factors that affect gloss and lightness constancy. They find that motion, disparity, and color all improve the constancy of gloss matches across changes of object shape, but that only motion and color improve the constancy of corresponding lightness matches. Doerschner et al. (2010) examine gloss perception in scenes with high-dynamic ranges of illumination and report a gloss illusion.

Interactions

There are several studies that examined interactions among different material properties. Ho et al. (2008) used a conjoint measurement design and rendered stimuli to show that surface roughness affected observers' judgments of surface glossiness and vice versa. The degree of contamination was small. Xiao and Brainard (2008), using rendered stimuli, assessed how the presence of specular highlights affects the color appearance of three-dimensional objects and showed that the visual system stabilizes color appearance with respect to material variation. Yoonessi and Zaidi (2010) examined the role of color in recognizing material changes.

In this issue, Giesel and Gegenfurtner (2010) systematically investigate color perception for real objects made of different materials varying in roughness and gloss from smooth and glossy to matte and corrugated. They show that hue is quite stable across their manipulations, but that other attributes interact. Olkkonen and Brainard (2010) study how changes in real-world illumination affect perceived glossiness and lightness with emphasis on testing independence principles. They show, for example, that the effect of geometric changes in the light field on perceived glossiness is independent of the diffuse reflectance component of the surfaces.

Novel themes

A number of papers in the current issue introduce novel themes. Wolfe and Myers (2010) examine visual search performance when targets and distractors are characterized by surface material. They find that, although it may be easy to discriminate “fur” or “stone,” searching for a patch of fur among the stones is difficult and time-consuming. Visual search based on material differences is inefficient. Motoyoshi (2010) examines how the relationship between highlights and shading triggers perception of translucency and transparency. Marín-Franch and Foster (2010) develop mathematical and experimental methods to assess how many perceptually distinct surfaces are present in natural scenes. Ged, Obein, Silvestri, Le Rohellec, and Viénot (2010) measure how perceived gloss relates to physical properties of actual surfaces.

Goddard, Solomon, and Colin (2010) study the adaptable neural mechanisms responsible for surface color constancy. Boyaci, Fang, Murray, and Kersten (2010) also consider mechanism. They report behavioral results showing how lightness across occlusion depends on spatially distant image features and show (using brain imaging) that human early visual cortex responds strongly to occlusion-dependent lightness variations. They conclude that early cortical processing of lightness is affected by three-dimensional scene interpretation.

Another emerging theme, not represented in this issue, is the introduction of non-homogeneous reflectance into the objects whose color or other material properties are being judged (Ho et al., 2008; Hurlbert, Vurro, & Ling, 2008; Olkkonen, Hansen, & Gegenfurtner, 2008; Robilotto & Zaidi, 2006). Most real-world objects contain some degree of such non-homogeneity, and we expect this line to grow in importance over the next few years.