Adaptation and visual coding

Abstract

Visual coding is a highly dynamic process and continuously adapting to the current viewing context. The perceptual changes that result from adaptation to recently viewed stimuli remain a powerful and popular tool for analyzing sensory mechanisms and plasticity. Over the last decade, the footprints of this adaptation have been tracked to both higher and lower levels of the visual pathway and over a wider range of timescales, revealing that visual processing is much more adaptable than previously thought. This work has also revealed that the pattern of aftereffects is similar across many stimulus dimensions, pointing to common coding principles in which adaptation plays a central role. However, why visual coding adapts has yet to be fully answered.

Introduction

Figure 1a shows a popular illustration of color after-effects known as the “lilac chaser” introduced by Jeremy Hinton (http://en.wikipedia.org/wiki/Lilac_chaser). As you fixate the cross, the lilac spots fade to the point of disappearing, and the greenish afterimage as each spot is briefly removed becomes the most conspicuous color. The fading in part reflects a loss in sensitivity to the steadily presented stimulus and a renormalization that sets the white point according to the local average chromaticity. In the extreme, this removes the steady-state signal from visual processing, leaving you aware only of the novel transient and how it deviates from your (gray) expectation. A number of different factors contribute to the illusion (e.g., Troxler fading and filling in for stabilized images), but foremost it is a powerful demonstration of the effects and consequences of visual adaptation for perception. One theme of this review is that these effects are constantly regulating your perception and visual experience. There is an underappreciated aspect of this illusion. The lilac hue you notice when first looking at the stimulus is also striking because it is an aftereffect of the state of adaptation you brought to the display.

Figure 1.

Examples of visual aftereffects. (a) The Lilac Chaser illusion (http://en.wikipedia.org/wiki/Lilac_chaser). With fixation on the cross, the lilac dots fade revealing a strong greenish afterimage when each dot is briefly removed. (b) Adaptation effects analogous to color are found for faces. When fixating the central face, the distortions in the peripheral faces become less apparent, leading to a strong perceived distortion when each face is briefly replaced with the undistorted center face.

Figure 1b shows a variant of the illusion in which the color spots have been replaced with distorted images of a face (Winkler, McDermott, Caplovitz, & Webster, in press). (In the spirit of this 10th anniversary issue, we used an image of Andrew Watson, the founding editor of the Journal of Vision. However, we confirmed that the effects also work with an image of Al Ahumada.) As you stare at the original image in the center, the distortions within the peripheral faces again fade from view, and the transient change to the undistorted face appears the most distinctive because of how it deviates from each adapting face. Here again, there are a number of processes underlying these effects (including different levels of adaptation and possible foveal capture of the appearance of faces in the periphery). Yet, the basic aftereffects for faces are very similar to the aftereffects for color (Webster & MacLeod, 2011). A second theme is that the effects and consequences of adaptation are pervasive throughout visual coding and may involve many common design principles.

To explore these principles, I have focused on developments in the study of adaptation roughly over the last 10 years spanned by the Journal of Vision. A search of the journal returned 183 articles that included “adaptation” in the title or abstract. These ranged from mechanisms of light adaptation in the photoreceptors to the calibration of eye movements. The last decade has seen a number of important advances in the understanding and application of adaptation, and different perspectives on these developments emphasizing different levels of analysis can be found in a number of other recent reviews (Clifford et al., 2007; Clifford & Rhodes, 2005; Demb, 2008; Kohn, 2007; Rieke & Rudd, 2009; Wark, Lundstrom, & Fairhall, 2007). In the present case, the aim is to consider when adaptation is manifest in vision and its implications for visual coding.

Adaptation and visual plasticity

As the examples in Figure 1 illustrate, looking at a pattern for a short time typically results in a loss in sensitivity to the pattern and a bias in the appearance of other patterns. Visual adaptation is operationally defined in terms of these brief exposures and aftereffects (Thompson & Burr, 2009). However, the visual system exhibits an enormous variety of dynamic and experience-dependent adjustments, and it is difficult to give a functional definition of adaptation that can safely distinguish it from other forms of plasticity. Even within the sensitivity adjustments that are normally described as light adaptation, there are many different mechanisms at play (Rieke & Rudd, 2009; Stockman, Langendorfer, Smithson, & Sharpe, 2006). Visual coding adjusts not only to the recent past but also nearly instantaneously to changes in the spatial context. It is not clear to what extent these spatial and temporal adjustments should be treated as functionally distinct (Schwartz, Hsu, & Dayan, 2007), but they are often synonymously labeled with terms like adaptation or gain control. What constitutes a “brief” sensitivity change is also unclear. As discussed below, adaptation occurs over multiple timescales, but when these involve a switch in the possible function or mechanism [e.g., from intra-cellular response changes (Sanchez-Vives, Nowak, & McCormick, 2000) to a change in synaptic weights (Massey & Bashir, 2007) to the growth of new neural circuits (Wandell & Smirnakis, 2009)], is still poorly understood. As aftereffects become extended to increasingly abstract levels (Dils & Boroditsky, 2010), it is also blurs the distinction between visual and conceptual adjustments.

A long-standing issue is the relationship between adaptation and learning. Perceptual learning usually produces improvements in discrimination with long-term training on a perceptual judgment (Lu, Yu, Watanabe, Sagi, & Levi, 2009) whereas adaptation is typically characterized as a more immediate loss in sensitivity when we are exposed to a stimulus. Learning might thus be distinguished from adaptation because it primarily reflects changes in performance rather than appearance, facilitation rather than suppression, a longer time course (Teich & Qian, 2003), and changes in how the visual system interprets the available neural signals rather than in the strength of those signals (Series, Stocker, & Simoncelli, 2009). However, none of these distinctions is definitive. Like adaptation, learning paradigms can also change the appearance of patterns (Haijiang, Saunders, Stone, & Backus, 2006). Conversely, like learning, adaptation can facilitate some discriminations (e.g. Clifford, Wyatt, Arnold, Smith, & Wenderoth, 2001; Kristjansson, 2011; McDermott, Malkoc, Mulligan, & Webster, 2010), most obviously in the case of light adaptation (Barlow, 1972). Moreover, some aftereffects described as adaptation can show remarkably long persistence, and as discussed below, there is likely to be a whole regime of adaptation-like adjustments operating over much longer timescales than are typically studied. In fact, the processes of adaptation itself might exhibit forms of learning (Yehezkel, Sagi, Sterkin, Belkin, & Polat, 2010).

Similar problems arise in trying to parse adaptation from other forms of plasticity such as priming (e.g., Oruc & Barton, 2010) or even attention. Many perceptual aftereffects can be modulated by attention, especially at higher levels of visual coding, and adaptation and attention could reflect complementary neural modulations (Barlow, 1997; Boynton, 2004, Pestilli, Viera, & Carrasco, 2007; Rezec, Krekelberg, & Dobkins, 2004). On the other hand, one powerful application of adaptation has been to study the mechanisms of attention and awareness. After-effects in visible stimuli can be induced by adapting stimuli that are themselves invisible because they vary too finely in space or time to be resolved (Shady, MacLeod, & Fisher, 2004; Vul & MacLeod, 2006) or are suppressed by crowding (He, Cavanagh, & Intriligator, 1996) or rivalry (Blake & Fox, 1974). These studies have played an important role in understanding the sites and nature of conscious perception (Blake, Tadin, Sobel, Raissian, & Chong, 2006; He & MacLeod, 2001).

Adaptation as a tool

In many studies of visual aftereffects, the adaptation is the tool rather than the topic. That is, the focus is on using the response changes to reveal the processes encoding different perceptual attributes and not on the nature of adaptation itself. This is in contrast to studies of other forms of short-term visual plasticity such as perceptual learning. Interest in perceptual learning has exploded over the last decade, with much of this recent effort directed at elucidating the conditions under which learning occurs and the mechanisms mediating it (Lu et al., 2009; Sagi, in press; Sasaki, Nanez, & Watanabe, 2010). Analyses drawn from this work have been successfully applied to dissecting the processes of adaptation (e.g., Dao, Lu, & Dosher, 2006), but more often adaptation has remained the scalpel. One reason for this difference is that adaptation provides a venerable and very rapid procedure for altering the visual system and then probing the resulting changes in visual coding. Most aftereffects are selective, producing the largest changes in sensitivity to the adapting pattern and weaker response changes as the difference between the adapt and test stimuli increase. Psychophysical measures of these tuning functions have been one of the primary sources of evidence for the basic channel structure of the visual system (Graham, 1989).

A development with far-reaching impact has been the extension of this tool to functional imaging in the technique of fMRI adaptation (Grill-Spector, Henson, & Martin, 2006; Grill-Spector & Malach, 2001; Krekelberg, Boynton, & van Wezel, 2006; Weigelt, Muckli, & Kohler, 2008). The BOLD response declines with repeated presentations of the same stimulus, and thus the selectivity of the response can be measured by determining how the stimulus must change to release the suppression. This has allowed fMRI to be used at spatial scales much finer than a voxel. Studies of fMRI adaptation have demonstrated selective response changes paralleling many of the classic behavioral aftereffects for dimensions such as orientation (Boynton & Finney, 2003; Fang, Murray, Kersten, & He, 2005; Tootell et al., 1998), color (Engel & Furmanski, 2001; Wade & Wandell, 2002), contrast (Gardner et al., 2005), and motion (Huk, Ress, & Heeger, 2001). The paradigm has also been extensively used to study more complex stimuli like objects (e.g., Kourtzi & DiCarlo, 2006) and faces (Andrews & Ewbank, 2004; Grill-Spector et al., 1998; Loffler, Yourganov, Wilkinson, & Wilson, 2005).

However, like different measures of plasticity, the relationship between different techniques and scales (from single unit to behavior) for measuring adaptation is again difficult to resolve. For example, some accounts of repetition effects in fMRI have emphasized the close connections to priming rather than adaptation (Schacter, Wig, & Stevens, 2007) or to behavioral habituation (Turk-Browne, Scholl, & Chun, 2008), which has itself been both associated with and distinguished from adaptation. Moreover, for any single technique, the interpretation of results is complicated by the fact that changes in the selectivity or form of the aftereffect could arise either from differences in the nature of adaptation or the underlying mechanisms. Finally, a recurring problem is that the pattern of aftereffects can usually be accounted for by very different models. For example, the selectivity of pattern aftereffects have been attributed to both independent response changes within mechanisms or to interactions between mechanisms, and both accounts can sometimes fit the data equally well (Atick, Li, & Redlich, 1993; Webster & Mollon, 1994; Zaidi & Shapiro, 1993). There continue to be important questions raised about what a selective adaptation effect actually represents (Benton & Burgess, 2008; Hegde, 2009; Mur, Ruff, Bodurka, Bandettini, & Kriegeskorte, 2010).

Early stages of adaptation

An emerging insight is that the processes of adaptation are much more widespread than previously considered. One example of this is that early visual levels have been found to demonstrate a surprisingly richer array of sensitivity adjustments. Many of the classic visual aftereffects measured for stimulus dimensions such as orientation, spatial frequency, or direction of movement show strong interocular transfer and pattern selectivity, implying that they depend at least in part on sensitivity changes at cortical levels. This was consistent with evidence suggesting that geniculate cells showed little adaptation to contrast (Derrington, Krauskopf, & Lennie, 1984; Ohzawa, Sclar, & Freeman, 1982) and suggested that adaptation in the retina was primarily confined to adjusting to the average luminance and color of the stimulus. However, it is now apparent that the retina adapts not only to the mean light level but also to contrast, an effect that has been observed across a number of species (Baccus & Meister, 2002; Brown & Masland, 2001; Chander & Chichilnisky, 2001; Rieke, 2001; Smirnakis, Berry, Warland, Bialek, & Meister, 1997). The contrast adjustments include not only a very rapid contrast gain control (Shapley & Enroth-Cugell, 1984) but also a more sluggish sensitivity change that adjusts to the stimulus contrasts over several seconds and thus has a comparable time course to the contrast aftereffects measured psychophysically (Baccus & Meister, 2002). The extent to which these early contrast adjustments contribute to behavioral contrast aftereffects is not certain. In primate retina, contrast adaptation is confined largely to the magnocellular pathway and is most strongly manifest at high temporal frequencies and at lower test contrasts, which may explain why earlier studies failed to observe the adaptation (Camp, Tailby, & Solomon, 2009; Solomon, Peirce, Dhruv, & Lennie, 2004). Nevertheless, as Solomon et al. noted, this adaptation is consistent with the loss of pattern selectivity of contrast aftereffects observed psychophysically at higher temporal frequencies (Kelly & Burbeck, 1987) and also with the aftereffects following adaptation to stimuli that are flickering at frequencies too high to be resolved (Shady et al., 2004).

Studies of retinal function continue to reveal a diverse range of mechanisms and computations that show that much more image analysis takes place within the earliest stages of vision than previously supposed (Gollisch & Meister, 2010). Thus, it is perhaps not surprising that the retina can also exhibit more complex forms of adaptation. For example, the dynamics of the adaptation to luminance and contrast are not fixed but adjust to the rate at which these levels vary in the stimulus (Wark, Fairhall, & Rieke, 2009). Thus, the adaptation can be controlled by higher order properties of the stimulus history, even though the cells do not directly adapt to these properties. These tunable dynamics may allow the retina to adapt more rapidly when there is better evidence that the state of the world has changed (Wark et al., 2009). Contrast adaptation within the retina can also show selectivity for spatiotemporal patterns to which the cells are not directly selective. For example, ganglion cells can selectively adapt to oriented patterns (Hosoya, Baccus, & Meister, 2005) or movement (Olveczky, Baccus, & Meister, 2007). A possible site for both contrast adaptation and pattern-selective aftereffects is in the synaptic transmission from bipolars to ganglion cells, which might allow the adaptation to alter the pattern of inputs across the bipolars and thus the spatial and temporal properties of the sensitivity change (Gollisch & Meister, 2010).

Adaptation and high-level aftereffects

A second important extension of adaptation has been to higher levels of visual coding, to explore the aftereffects of much more abstract perceptual attributes. For example, adaptation not only affects the perceived tilt or curvature of lines but can also separately influence higher order shape properties like aspect ratio (Suzuki & Cavanagh, 1998) or three-dimensional viewpoint (Fang & He, 2005). Moreover, these figural aftereffects can be modulated by the direction of gaze, suggesting that the adaptation in part depends on non-retinocentric coordinate frames (Nishida, Motoyoshi, Andersen, & Shimojo, 2003). Similarly, adaptation not only can affect perceived color but may also include aftereffects driven by the inferred surface reflectance (independent of the illuminant; Goddard, Solomon, & Clifford, 2010), as well as material qualities such as whether they appear glossy or matte (Motoyoshi, Nishida, Sharan, & Adelson, 2007), and again can be contingent on extraretinal cues like gaze direction (Bompas & O’Regan, 2006; Richters & Eskew, 2009). Adaptation can also influence the perceived layout and affordances of visual scenes, such as how open or navigable they appear (Greene & Oliva, 2010).

A number of dissociable forms of motion aftereffects have been identified based on the types of stimuli that drive them (e.g., static vs. dynamic or translating vs. expanding) or over which they transfer (e.g., to new retinal locations; Mather, Pavan, Campana, & Casco, 2008). High-level motion aftereffects can also be induced by attentional tracking (Culham, Verstraten, Ashida, & Cavanagh, 2000) and, moreover, by imagining motion or viewing still photographs that depict movement (Winawer, Huk, & Boroditsky, 2008, 2010). (Notably, implied motion from stationary images also results in activity in motion-sensitive cortical areas (Kourtzi & Kanwisher, 2000; Krekelberg, Dannenberg, Hoffmann, Bremmer, & Ross, 2003).) Further, adaptation not only affects the perceived direction of motion but can also influence complex inferences based on the motion. For example, biological motion in point-light walkers can convey a strong impression of characteristics such as gender, and these characteristics can, in turn, be strongly biased by prior adaptation to the pattern of motion characterizing a particular gender (Jordan, Fallah, & Stoner, 2006; Troje, Sadr, Geyer, & Nakayama, 2006). Thus, adaptation effects in motion appear to arise at multiple processing levels and may potentially arise as an intrinsic component of many of the computations underlying the visual analysis of movement.

A large number of studies examining high-level after-effects have examined adaptation and face perception (Webster & MacLeod, 2011). After viewing a configurally distorted face (e.g., with eyes close together), an undistorted face will appear distorted in the opposite direction (i.e., the eyes will appear too far apart; Webster & MacLin, 1999; see Figure 1b). These aftereffects can occur for diverse representations of the face from photo-realistic to schematic (Anderson & Wilson, 2005) to silhouettes (Davidenko, Witthoft, & Winawer, 2008) and can be driven by both the shape and textural information in the face (Jiang, Blanz, & O’Toole, 2006; O’Neil & Webster, 2011). Moreover, they occur for and strongly influence the perception of many of the natural dimensions that are important for judgments of faces including identity, gender and ethnicity, expression, age, and attractiveness (Hsu & Young, 2004; Leopold, O’Toole, Vetter, & Blanz, 2001; O’Neil & Webster, 2011; Rhodes, Jeffery, Watson, Clifford, & Nakayama, 2003; Schweinberger et al., 2010; Webster, Kaping, Mizokami, & Duhamel, 2004). Adaptation, therefore, provides a potential tool for understanding how the perceived characteristics of faces are encoded and represented in the visual system.

However, to what extent does adaptation to a high-level attribute reflect response changes at high levels in the visual stream? This issue is complex because sensitivity regulation may occur at multiple (and possibly all) levels, and thus higher levels may inherit the response changes from earlier stages. For example, cells in MT show contrast adaptation that is selective for different subregions within their receptive field, consistent with sensitivity changes that are passed on from V1 (Kohn & Movshon, 2003). Yet, response changes first arising in MT may underlie the adaptation for other types of moving stimuli (Priebe, Churchland, & Lisberger, 2002). Similarly, face aftereffects (e.g., for expression) can be induced from local adaptation to simple shapes (e.g., curved lines), which are themselves unlikely to be represented as a face (Xu, Dayan, Lipkin, & Qian, 2008). Such results illustrate the point that adaptation to a visual stimulus will necessarily induce sensitivity changes beginning as early as the receptors. Thus, a question in understanding examples of high-level behavioral aftereffects is not whether more complex stimuli induce adaptation at peripheral sites, but whether there is evidence for additional stages of adaptation that are linked to the explicit coding of the attribute.

A number of strategies have been developed to isolate these high-level aftereffects. For example, higher order shape aftereffects can be distinguished from conventional figural aftereffects because they occur even with very brief presentations, show stronger transfer across different retinal locations, and show less dependence on stimulus contrast (Suzuki, 2005). The latter effects are consistent with larger receptive fields and greater contrast invariance of extrastriate areas. Similarly, face aftereffects show strong transfer across changes in position and image size (Afraz & Cavanagh, 2008; Leopold et al., 2001; Zhao & Chubb, 2001) and also across changes in orientation. For example, Watson and Clifford (2003) found that adaptation to a distorted face depended on the axis of distortion within the object (e.g., a horizontal constriction of the face) regardless of the axis within the visual frame (e.g., whether the face was tilted clockwise or counterclockwise). Face aftereffects have also been found to depend more strongly on the perceptual category of the stimuli rather than their structural similarity. Thus, in a morph between male and female faces, separate contingent aftereffects can be generated for distortions that fall on different sides of the identity or gender boundary (i.e., when one face appears male and the other female) but not for equivalent physical differences within each category (Bestelmeyer et al., 2008; Rotshtein, Henson, Treves, Driver, & Dolan, 2005). The adaptation is also more dependent on awareness compared to simpler stimuli, so that the adaptation is largely abolished when the face is masked by binocular suppression (Moradi, Koch, & Shimojo, 2005). Like motion aftereffects, aftereffects for faces have been reported following visualization (DeBruine, Welling, & Jones, 2010; Ganis & Schendan, 2008; Ryu, Borrmann, & Chaudhuri, 2008) and can also transfer across different representations of implied attributes. For example, adaptation to female or male headless bodies induces a gender aftereffect in the perception of a face (Ghuman, McDaniel, & Martin, 2010; though a similar transfer has not been found between the gender cues carried by faces and hands (Kovacs et al., 2006)). On the other hand, it is unlikely that face aftereffects can always arise at a conceptual level, for adaptation to facial expressions does not occur for non-facial images or words that convey an emotion (Fox & Barton, 2007). Taken together, such results strongly suggest that aftereffects for high-level attributes do at least partly reflect response changes at high levels of visual processing.

Thus far, studies of these high-level visual aftereffects have pointed to two important conclusions. The first is that most aspects of visual perception are adaptable. Thus, adaptation appears to be a central and inherent mechanism in visual processing at all stages and might warrant the status of a general law (Helson, 1964). Second, the basic pattern of high-level aftereffects shows a number of striking similarities to the aftereffects that have conventionally been measured for simpler stimulus attributes. For example, the buildup and decay of face aftereffects follows the same time course as contrast adaptation in gratings (Leopold, Rhodes, Muller, & Jeffery, 2005) and, as discussed below, may normalize the representation of faces in the same way that adaptation sets the reference level for stimuli like color (Webster & MacLeod, 2011). This suggests that how the visual system adapts and represents different perceptual dimensions may often draw on common coding strategies. This similarity has been noted previously in the close parallels between adaptation aftereffects for dimensions such as motion, orientation, and color (Clifford, 2002) but may potentially generalize to many visual attributes.

As noted above, adaptation is commonly used as a tool for measuring the number and selectivity of the channels encoding different stimulus dimensions. Can it be applied in the same way to characterize the “channels” mediating high-level percepts such as the characteristics of a face? For example, a number of studies have examined the interactions between different facial dimensions by testing whether aftereffects along one dimension (e.g., gender) are contingent on the differences along a second dimension (e.g., ethnicity; Jaquet & Rhodes, 2008; Jaquet, Rhodes, & Hayward, 2008; Little, DeBruine, & Jones, 2005; Little, DeBruine, Jones, & Waitt, 2008; Ng, Ciaramitaro, Anstis, Boynton, & Fine, 2006; Yamashita, Hardy, De Valois, & Webster, 2005). These have generally found some degree of selectivity paralleling the stronger selective aftereffects found for simpler stimulus dimensions such as color and orientation and size. However, it remains unclear whether this technique will yield the same insights for face coding, for it is difficult to determine whether the dimensions measured in the aftereffect are really the dimensions along which the visual response is altered. Thus, while revealing in many ways, high-level aftereffects for stimuli like faces have yet to clarify what information the visual system actually samples to represent an individual face.

Adaptation and the natural visual environment

The high-level aftereffects examined in these investigations have also necessarily extended the study of adaptation to stimuli that are more ecologically relevant and characteristic of the types of images that observers encounter in everyday viewing, and imply that many aspects of natural vision are probably routinely regulated by adaptation. Moreover, they suggest that the natural visual world probably does vary enough in ways that can influence the states of adaptation. Thus, how we perceive color or faces or scenes may strongly depend on the particular environments we are adapted to.

The statistical structure of natural scenes has been a subject of intense interest over the last two decades and has resulted in powerful insights into the basic design and of the visual system (Geisler, 2008; Simoncelli & Olshausen, 2001). These statistics also provide important clues about how the visual system may be adapted (Wainwright, 1999; Webster & Miyahara, 1997; Webster & Mollon, 1997). The range of luminance levels and contrasts at different points within a scene can be far greater than the dynamic range over which neurons can signal differences, and thus retinal mechanisms of light and contrast adaptation must rapidly adjust as the scenes are sampled with eye movements (Rieke & Rudd, 2009). Moreover, how different mechanisms contribute to these adjustments depends fundamentally on the lighting context. Variations in mean luminance and contrast are uncorrelated within natural scenes. This predicts that the processes of light and contrast adaptation should operate independently, which has been observed both in single cells (Mante, Frazor, Bonin, Geisler, & Carandini, 2005) and behaviorally (Webster & Wilson, 2000). Similarly, knowledge of the statistics of natural color signals have been central to understanding adaptation and the mechanisms of color constancy (Foster, 2011; MacLeod, 2003; Smithson, 2005).

An important aspect of natural stimuli is that their statistics are highly constrained. Thus, some aspects of the world present the visual system with consistent patterns of stimulation, and these may define the characteristic operating states of vision. For example, natural images have more energy at low spatial and temporal frequencies (Field, 1987; Field & Brady, 1997; Tolhurst, Tadmor, & Chao, 1992; Van Hateren, 1993). Adaptation to this structure selectively reduces sensitivity to low spatial frequencies, resulting in marked changes in the shape of the contrast sensitivity function (CSF; Bex, Solomon, & Dakin, 2009; Webster & Miyahara, 1997) or in the tuning functions of individual cortical cells (Sharpee et al., 2006), though notably, the same adaptation does not bias the suprathreshold perception of image focus in space or time (Bilson, Mizokami, & Webster, 2005; Webster, Georgeson, & Webster, 2002)). Similar structure is found for the spatial variations in the chromatic contrast of scenes (Parraga, Troscianko, & Tolhurst, 2002), and adaptation to this chromatic structure can cause the normally low-pass chromatic CSF to become nearly band pass (Webster, Mizokami, Svec, & Elliott, 2006). Conventional measures of contrast sensitivity, based on adaptation and presentation on a uniform gray field, thus may fail to capture important features of visual sensitivity under the viewing conditions we normally operate under.

In the case of color, natural scenes may have less consistent properties. For example, both the average color and how color contrasts are distributed around the average can vary across environments or within the same environment over time (e.g., in different seasons). These variations are large enough so that specific environments may induce very different states of chromatic and contrast adaptation (Webster, Mizokami, & Webster, 2007; Webster & Mollon, 1997). In more lush environments, color variations tend to be roughly independent in terms of the L vs. M cone (parvocellular) and S vs. LM (koniocellular) dimensions of early post-receptoral color coding, so that these dimensions are optimal for the initial opponent coding of color (Ruderman, Cronin, & Chiao, 1998). Yet, more arid scenes tend to have a blue–yellow bias in their colors (an axis intermediate to the two cardinal geniculate axes).

To the extent that we know the distribution of stimuli and how the visual system adapts to them, we can simulate how color appearance should change across these common natural variations in the color environment (Juricevic & Webster, 2009). This is illustrated in Figure 2, which simulates the predicted perceptual shifts when the observer is adapted to a lush or arid environment. The simulation is based on sampling the distribution of colors from the same location in India in two different seasons, and then using an empirically motivated model of color vision based on adaptation to the mean luminance and color (within the cones) and to the luminance and chromatic contrasts (within channels tuned to different color and luminance axes). Adaptation is modeled as a gain change in each channel so that the response to the prevailing environment is matched to the response in a reference environment, and the image colors are then rendered from the adapted channels’ outputs. Note that the predicted effects of adaptation are again to reduce sensitivity to the prevailing colors in the scenes and thus enhance the salience of novel colors. In a dry environment, the greens stand out, while yellows are more conspicuous in an observer adapted to green. Analogous effects are predicted for other dimensions of variation in the visual environment. For example, the faces that appear distinctive or conspicuous may also depend on how face perception is adapted to the individual’s specific social environment (Webster & MacLeod, 2011).

Figure 2.

Simulations of the changes in color appearance predicted in observers adapted to the color characteristics of different environments. Top images show roughly the same scene in two different seasons. Bottom images depict how the scenes might appear to an observer completely adapted to the color distributions from either wet or arid seasons.

Are the color worlds we live in also restricted enough in ways that might hold us in dominant states of adaptation? There are a number of signs that sensitivity to blue–yellow variations is reduced (relative to a complementary lilac–green direction that produces the same component signals along the L vs. M and S vs. LM geniculate axes; Goddard, Mannion, McDonald, Solomon, & Clifford, 2010; Juricevic, Land, Wilkins, & Webster, 2010; McDermott et al., 2010). Notably, a similar bias is also built into a number of uniform color spaces (such as CIE Luv or Lab). These are designed to represent equivalent perceptual differences by equivalent metric distances in the space and, again, predict less sensitivity to the geniculate signals along bluish-yellowish directions. Such results are puzzling in the context of the early encoding of color but are not unexpected if cortical color coding is under long-term adaptation to the prominent blue–yellow bias of many natural environments.

Adaptation and variations in the observer

Discussions of visual adaptation have primarily focused on how the visual system adjusts to changes in the environment. Yet, similar calibration problems occur when it is the observer that changes. These changes are often gradual and involve many forms of plasticity such as the experience-dependent neural changes that have been extensively studied in visual development. However, the visual system continues to change throughout the life span and thus must be continuously recalibrated to maintain the match between visual coding and the visual environment. Moreover, changes in the observer can also occur suddenly, because of injury or disease or a new pair of glasses. These variations will again alter the match between visual coding and the environment and thus should again trigger changes in the state of adaptation. Finally, calibrations for variations within the observer are also important for correcting variations across space (e.g., with retinal eccentricity) or across different stimulus attributes (e.g., so that coding for one dimension, such as size, is consistent across other dimensions, such as orientation).

Most studies of adaptation to normal variations in the sensitivity of observers have examined the influence of pre-neural factors such as the optics of the eye or the filtering effects of the lens and macular pigment. The crystalline lens progressively yellows with age and thus increasingly screens the short-wavelength light reaching the retina. If color vision was not readjusted for this transmittance change, then the stimulus that appeared white in our youth should appear progressively yellower as we age. Instead, achromatic settings and hue judgments remain very stable across the life span, and this stability could be accomplished if the spectral screening changes give rise to corresponding neural sensitivity adjustments through adaptation (Hardy, Frederick, Kay, & Werner, 2005; Schefrin & Werner, 1990; Werner & Schefrin, 1993). This is illustrated in Figure 3, which shows how the scene in the upper left of Figure 2 would appear when filtered through the eyes of an observer with the average lens pigment density of someone in their 70s (Webster, Juricevic, & McDermott, 2010). The assumption of simple cone-specific gain changes (known as von Kries adaptation) is sufficient to correct for most (but not all) of the lens filtering, so that the adapted image remains very similar for the two observers despite enormous differences in spectral sensitivity. Analogous effects are seen in the compensation of color appearance for the spatial variations in macular pigment screening, which selectively filters the short-wavelength light reaching the foveal receptors. Despite this, the stimulus perceived as white remains very similar between the fovea and nearby periphery (Beer, Wortman, Horwitz, & MacLeod, 2005; Webster & Leonard, 2008).

Figure 3.

Simulations of the changes in color appearance predicted as observers adapt to age-related changes in lens pigment density. (Left) The upper left image from Figure 2 as filtered through the eyes of an observer with the average lens density of a 70 year old (relative to the reference lens density of a 12 year old). (Right) The filtered image after adapting to the mean color shift by rescaling the cone sensitivities for the mean spectral change. Independent gain changes in the cones compensate for most (but not all) of the predicted effect of the lens screening on color appearance.

An important source of variation in spatial sensitivity is from optical aberrations of the eye. A number of studies have examined how the visual system adapts to blur in the retinal image. Viewing a blurred or sharpened image leads to rapid aftereffects in the level of physical blur that appears in focus (Webster et al., 2002). With longer exposures, adaptation to defocus also leads to improvements in visual acuity (Mon-Williams, Tresilian, Strang, Kochhar, & Wann, 1998; Pesudovs & Brennan, 1993). Both short- and long-term adaptation can result from the actual patterns of blur resulting from the eye’s optics, including low-order aberrations of defocus and astigmatism as well as high-order aberrations (Artal et al., 2004; Sawides et al., 2010). Moreover, there are again analogous effects across the visual field, so that the perception of image focus in the periphery is maintained despite the marked losses in spatial resolution (Galvin, O’Shea, Squire, & Govan, 1997). These results thus suggest that adaptation may be important for matching spatial vision to the optical quality of the eye.

A common consequence of these adjustments is that they tend to compensate or correct visual appearance by discounting the spectral or spatial sensitivity of the observer, so that the physical stimulus that appears white or in focus does not depend on the observer’s specific sensitivity limits. In this sense, the adaptation is helping to maintain perceptual constancy (Walraven & Werner, 1991). This has a number of important implications for visual experience, because it predicts that the world will “look” less different than predicted by the interobserver differences in threshold sensitivity. For example, a number of studies have simulated an image as seen through the eyes of an infant or a color-deficient observer or different species (Rowe & Jacobs, 2004; Teller, 1997; Vienot, Brettel, Ott, Ben M’Barek, & Mollon, 1995; Vorobyev, Marshall, Osorio, Hempell De Ibarra, & Menzel, 2001). These simulations typically filter the images for the differences in spectral and spatial contrast sensitivities and thus depict the information that is lost to the observer. However, this often may not capture how the information that they can see is actually “perceived.” As long as these individuals adapt in similar ways, (and retain at least some sensitivity to the stimulus dimension), then much of the sensitivity limits may be discounted from the suprathreshold percept, and they may be no more likely to experience the world as blurred or tinted through their visual systems than you do through yours. A related effect is that the compensatory adjustments of adaptation may tend to mask sensitivity losses with progressive disease, so that observers may be less aware of a developing visual impairment. Finally, these compensations highlight an important asymmetry of visual adaptation—that it is the observer that is adjusted to match the world (Clifford & Rhodes, 2005) and thus that variations in the environment may be more important than variations in visual sensitivity for controlling some aspects of perception. To draw on color again, whether two trichromatic observers experience the same stimulus as white may depend much more on whether they are exposed to the same color environment than whether their eyes filter the environment in the same way.

Very little is known about how visual adaptation itself varies across observers, though there can be pronounced individual differences in visual aftereffects (Vera-Diaz, Woods, & Peli, 2010). Further, surprisingly few studies have examined how the form and integrity of adaptation processes change across the life span. Factors influencing the development of light adaptation, contrast sensitivity, and contrast gain control have been well characterized in infant vision (Brown & Lindsey, 2009), and pattern-selective adaptation has been demonstrated (e.g., with VEPs) within the first few weeks (Suter et al., 1994). Yet, it is not clear whether there are significant developmental changes in the mechanisms of cortical adaptation. Notably, on the one hand, the suppressive effects of adaptation may be disadvantageous during the initial pruning of connections during post-natal development (Daw, 2003). Yet, on the other hand, it is arguable that the main work of sensitivity regulation should be done during infancy to initially calibrate the visual system, so that it is possible that much of the adaptation observed during adulthood is merely fine tuning. Similarly, compensation for age-related losses requires that the processes of adaptation remain largely functional in the senescent visual system. The rate of dark adaptation is slowed in older eyes because of changes in the rates of photopigment regeneration (Jackson, Owsley, & McGwin, 1999). Senescent changes have also been found in post-receptoral sites of chromatic adaptation (Werner, Bayer, Schwarz, Zrenner, & Paulus, 2010), as well as in high-level shape aftereffects (Rivest, Kim, Intriligator, & Sharpe, 2004). In contrast, a study of cortical adaptation effects on perceived blur found little age-related decline in adaptation strength (Elliott, Hardy, Webster, & Werner, 2007). Overall, some aspects of adaptation may, therefore, remain largely robust despite the profound neural changes that accompany normal aging, perhaps because adaptation remains critically important for compensating for these changes.

Adaptation and visual codes

Conventional models of visual aftereffects tended to focus on multiple channel accounts in which the stimulus dimension (e.g., orientation) is represented by a bank of overlapping filters, each narrowly tuned to a particular level (e.g., vertical or horizontal) along the continuum (Graham, 1989), and each obeying the principle of univariance, so that changes in the stimulus affect the size of the response but not the form of the response (Naka & Rushton, 1966). Adaptation could reduce the responses in the channels that were most sensitive to the stimulus and thus bias the pattern of activity to a test stimulus presented subsequently (Figure 4a). This predicts “repulsion” aftereffects in which test levels slightly above or below the adapting level appear biased away from the adapting level. These aftereffects are characteristic of the appearance changes observed following adaptation to a single spatial frequency (Blakemore & Sutton, 1969) or a single contrast axis in color space (Webster & Mollon, 1994) and have also been reported recently for effects of perceived eye gaze (Calder, Jenkins, Cassel, & Clifford, 2008), body orientation (Lawson, Clifford, & Calder, 2009), and viewpoint (Fang & He, 2005).

Figure 4.

Adaptation and visual channels. Curves show the sensitivity of a set of channels before (dashed gray) or after (solid) adaptation. (a) Narrowly tuned channels and narrowband stimuli. Adaptation reduces sensitivity within the subset of channels sensitive to the adapting level, biasing the modal response to flanking stimuli away from the response to the adaptor. Response changes are similar for different adapting levels and thus there is no narrowband stimulus corresponding to a unique norm. (b) Broadly tuned channels and narrowband stimuli. Adaptation reduces sensitivity in the more strongly stimulated channel, shifting the balance point or norm (n) toward the adapting level. (c) Narrowly tuned channels and broadband stimuli. A unique norm occurs when the balance of activity is equal across the channels. Adaptation to a biased stimulus distribution renormalizes the balance, shifting the norm toward the adapting stimulus. (d) An opponent mechanism, in which the norm corresponds to the null between excitation and inhibition. Adaptation at a pre-opponent site can change the balance of inputs resulting in a mean shift in the norm toward the adapt level. Adaptation to contrast, at the opponent site, instead alters sensitivity without shifting the null point.

However, many visual aftereffects instead follow a different pattern, in which the visual responses are renormalized so that the adapting stimulus itself appears more neutral. For example, as we continue to look at them, colors fade toward gray, distinctive faces appear more average (Robbins, McKone, & Edwards, 2007), and a blurred or sharpened image appears more focused (Elliott, Georgeson, & Webster, 2011). With adaptation, curved and tilted lines also tend to appear straighter and more closely aligned to vertical and horizontal (Gibson, 1933; Gibson & Radner, 1937). The canonical example of norm-based coding is color vision, in which hue and saturation are judged relative to a white point that is normalized by adaptation. Norm-based coding has also been extensively studied in the context of face perception, in which individual faces might be represented by how they differ from an average or prototype that is again calibrated by adaptation (Leopold, Bondar, & Giese, 2006; Leopold et al., 2001; Loffler et al., 2005; Rhodes & Jeffery, 2006; Rhodes et al., 2005; Robbins et al., 2007). In both color and faces, there is a special asymmetry in the adaptation—the norm can be strongly biased by exposure to a novel stimulus (e.g., a particular hue or distinctive face), while the norm itself induces little aftereffect, since it merely reinforces the current adaptation level (Webster & MacLin, 1999).

A renormalization aftereffect is predicted when the stimulus dimension is encoded by broadly rather than narrowly tuned channels (Figure 4b) or when the stimulus spectrum itself is broad (Figure 4c; as is typical of many natural stimuli). As an example of the former case, color is sampled by only three broadly tuned mechanisms, and thus, adaptation even to a narrowband wavelength will normalize the responses across the three cones. As an example of the latter, even if there are multiple narrow spatial frequency channels that generate a repulsion aftereffect when adapting to a single grating, natural stimuli like edges have broad amplitude spectra, and thus, adaptation to these should instead normalize the responses across a broad range of channels. In fact, multiple channel accounts are also a form of local renormalization within the channels, so that the two accounts differ only in whether the channels are narrow or broad relative to the stimulus. However, they have led to very different assumptions about how the visual system represents information. In multiple channel models, the value of the stimulus is usually assumed to be represented by the central tendency of the responses and thus by which channels respond, and there is no stimulus level that is special. In norm-based accounts, stimulus levels are represented by the response deviation from the norm, which has a unique neutral status.

Before considering the implications of norms, it worth noting that both types of models and aftereffects follow from simple assumptions about how individual mechanisms adapt to match their mean response to the prevailing stimulus. Adapting to a stronger stimulus results in a correspondingly lower sensitivity. If each channel adjusts independently, then this adaptation will work to level the playing field across an array of channels for the prevailing stimulus distribution. This illustrates that norms do not have to be pre-built in the system (and probably cannot be), because the balanced responses necessarily arise through the processes of adaptation. By this account, norms are in fact synonymous with adaptation states.

However, this alone does not explain why norms look special. A further assumption of norm-based codes is that the visual system monitors the relative responses. In color vision, this representation is built into opponent coding, in which the neural signals directly correspond to the relative activity in different classes of cones (Figure 4d). At the level of the receptors, “white” is implicitly represented as balanced activity across the receptors, while at the opponent level, it is instead directly coded as null in the opponent responses. Opponent-like processes have been implicated in a number of coding dimensions beyond color (Morgan & Regan, 1987; Regan & Beverley, 1985; Zemany, Stromeyer, Chaparro, & Kronauer, 1998) and are widely invoked to account for face aftereffects (Webster & MacLeod, 2011), but it remains unknown to what extent the many stimulus dimensions that have a clear perceptual norm in fact have an explicit opponent representation paralleling color vision. For color, sensitivity adjustments at opponent sites can be readily demonstrated by adapting to stimuli that vary around the mean, which selectively reduce the perceived variations along the adapting axis (Krauskopf, Williams, & Heeley, 1982). This is not surprising because this is, after all, contrast adaptation. However, what is surprising is that there is so little evidence for analogous contrast adaptation effects along many other stimulus dimensions. For example, adapting to an individual face produces strong aftereffects, while adapting to a change in the range of faces has little effect (Spetch, Cheng, & Clifford, 2004).

An assumption behind these considerations is that perceptual norms appear neutral and unique because they reflect a unique null or balance in the underlying neural responses. However, an alternative is that they are simply a learned property of the environment and thus might map on to visual responses in arbitrary ways. This issue is central to current discussions in color vision, for cells with the specific spectral sensitivities required to predict the red–green and blue–yellow nulls of color appearance have yet to be clearly identified (Conway et al., 2010; Mollon, 2006). For the achromatic norm (and for the mean of other stimulus dimensions), to the extent that adaptation has balanced the channel responses, this balance should correspond to the average spectral stimulus. Yet, individuals vary reliably in their chosen white points (Werner & Schefrin, 1993), and these variations could again reflect differences in learned criteria rather than how their sensitivities are normalized.

Adaptation provides a simple test for distinguishing these possibilities, by measuring which stimulus level does not produce an aftereffect. This level corresponds to the neural norm at the sites of the sensitivity change, because it is the stimulus level that does not alter the balance of responses within the channels. Webster and Leonard (2008) applied this test to examine how color vision was normalized in the fovea and near periphery to compensate for differences in macular pigment screening. The neutral adapting levels were very similar at the two loci and also very close to the perceptual norms (the white point chosen while the observer was instead dark adapted) for individual subjects. This suggests that color coding was already calibrated for the spectral sensitivity differences before the sites of chromatic adaptation, which is largely receptoral (MacLeod, Williams, & Makous, 1992; Stockman et al., 2006). This could occur if the intrinsic gain of the cones is set by long-term adaptation to the local average spectral stimulus (Walraven & Werner, 1991). Moreover, the results imply that individual differences in the perception of white probably do depend largely on differences in the adaptation states of observers, perhaps because of differences in their individual color environments. Consistent with this, these individual differences were substantially reduced when observers were instead adapted to a common stimulus during the experiment.

While such results suggest that there might be a close relationship between perceptual norms and the normalized responses of visual mechanisms, how the visual system normalizes is clearly more complex than the models of Figure 4 suggest. For example, tilt aftereffects are several times stronger than predicted by the changes in the perceived tilt of the adapting stimulus (Muller, Schillinger, Do, & Leopold, 2009). MacLeod and Beer (2005) observed a potentially related effect in light adaptation. The measured gain changes were different depending on whether the adapting stimuli had completely faded or were still visible.

A further complication that the models in Figure 4 fail to capture is that adaptation may change not only the sensitivities of the channels but also their tuning functions (Clifford, Wenderoth, & Spehar, 2000). Adaptation in single cells instead often alters the neuron’s selectivity, and these changes may be very different in different cortical areas (Kohn, 2007). In V1, adaptation to the cell’s preferred stimulus produces the largest response change at this level, while adapting to a stimulus to one side of the cell’s preferred stimulus results in a selective depression of the response on the adapted side, leading to a repulsive shift (Dragoi, Sharma, & Sur, 2000; Movshon & Lennie, 1979). The cells also adjust their tuning to the ensemble statistics such as the amplitude spectrum of images (Sharpee et al., 2006). In MT, exposure to the preferred stimulus instead leads to a sharpening of the tuning function and to attractive shifts toward a flanking adaptor (Kohn & Movshon, 2004; Krekelberg, van Wezel, & Albright, 2006). Kohn and Movshon noted that these attractive shifts are consistent with the negative after-effects of motion adaptation. Moreover, adaptive interactions across cells have been suggested as a possible basis for contingent aftereffects like the McCollough effect (Barlow, 1990b). Individual cells can show adaptation to stimulus contingencies (Carandini, Barlow, O’Keefe, Poirson, & Movshon, 1997; Hosoya et al., 2005). However, in general, it remains unclear how the selective aftereffects within cells translate to the adaptation effects measured behaviorally. Finally, an important issue is whether adaptation alters only the response of the channel or also the “labels” those responses carry—the information that allows the responses of different channels to be distinguished (Watson & Robson, 1981).

A number of these factors were recently formally explored by Series et al. (2009). The aftereffects of adaptation on the appearance and discrimination of motion and contrast could be generally well predicted from gain changes in channels whose responses were interpreted by an unadapted decoder (i.e., that was unaware of the changes in the adapted state of the channels). Morgan, Chubb, and Solomon (2006) also found that almost all of the response changes underlying motion aftereffects could be accounted for by sensitivity changes, with only a very small potential contribution from a “relabeling” of the channel outputs. Thus, simple channel models of the type illustrated in Figure 4 may capture the essence of perceptual aftereffects. As a specific example, color contrast adaptation may primarily arise at early cortical levels and even at early layers within V1 (Tailby, Solomon, Dhruv, & Lennie, 2008). Cells tuned for color in V1 vary in how linear they are (Conway et al., 2010), but color contrast adaptation effects can be closely predicted by gain changes within a set of mechanisms with linear tuning functions (Webster & Mollon, 1994). Thus, there is little overt evidence for a change in the underlying spectral sensitivities with contrast adaptation. Yet, as we have noted, the resulting changes in perceived hue are functionally very similar to the aftereffects of orientation and motion (Clifford, 2002; Clifford et al., 2000), where the effects of adaptation on the cell tuning may be more complex.

The timescales of adaptation

As noted at the outset, adaptation is normally measured and defined by changes in sensitivity over brief timescales ranging from milliseconds to minutes. Multiple rates of adaptation have been identified over this interval (Kohn, 2007; Wark et al., 2007). However, there is increasing evidence for sensitivity adjustments that operate over much longer times, from hours to weeks or perhaps even years. A number of examples of these long-term adjustments have been described for color vision. A classic example is the McCollough effect, in which the color aftereffect is contingent on the spatial orientation of the patterns (McCollough-Howard & Webster, 2011). McCollough effects are known to show unusually long persistence. A recent study found that the aftereffects show two independent time courses, a brief decay over seconds similar to the dynamics measured for contrast adaptation and an essentially static bias that requires exposure to a new pattern to reset (Vul, Krizay, & MacLeod, 2008). Long-term color aftereffects can also be induced when observers are exposed for a few hours a day to color-biased environments created by changing the spectrum of the room lighting, wearing tinted contacts, or viewing colored edges on a display (Belmore & Shevell, 2008, 2010; Eisner & Enoch, 1982; Neitz, Carroll, Yamauchi, Neitz, & Williams, 2002). Exposure to a red context shifts the wavelength that appears unique yellow, and these aftereffects can persist for days. When “color filters” are removed in cataract surgery, the changes in color appearance follow a very slow time course and do not completely renormalize even months after the surgery (Delahunt, Webster, Ma, & Werner, 2004). Long-term aftereffects have also been found on color salience. Observers trained to search for a red target among green distracters showed a very long-lasting sensitization to red in an attentional motion task (Tseng, Gobell, & Sperling, 2004).

Long-term adaptation has also been found for dimensions beyond color. Recently, Zhang, Bao, Kwon, He, and Engel (2009) deprived observers of specific orientations by having them view the world through a virtual reality display that could filter the images in real time. Exposure to these altered environments led to a relatively long-lasting improvement in sensitivity to stimuli presented at the missing orientation. In another recent study, Kwon, Legge, Fang, Cheong, and He (2009) had observers wear goggles that reduced image contrast. After 4 hours, observers became more sensitive to contrast as measured by both lower discrimination thresholds and steeper BOLD responses to image contrast in early cortical areas. Importantly, these changes were better characterized by changes in response gain (a multiplicative scaling) than by the contrast gain (roughly additive shift) that characterizes short-term contrast adaptation (Foley & Chen, 1997; Georgeson, 1985). This raises the intriguing possibility that the functional form of adaptation might change at different timescales. Both these studies are further notable in revealing improvements in contrast sensitivity when observers are adapted to lower contrasts. Most effects of contrast adaptation are usually characterized as a loss of contrast sensitivity. Yet, this loss is relative to the sensitivity measured on a uniform (zero-contrast) field. The improvements are instead predicted from the fact that in normal viewing we are adapted to the range of contrast in the natural environment and that this is the natural reference state that changes in adaptation should be assessed relative to.

It is instructive to consider that long-term states of adaptation are also apparent from how the visual system “recovers” when the nominally adapting stimulus is removed. For example, if light adaptation were controlled only by the immediate preceding context, then the colors signaled by the cones should diverge as they become uncalibrated during dark adaptation. Yet, white settings remain stable in the dark and at different retinal locations (Webster & Leonard, 2008; Werner & Schefrin, 1993). This indicates that the baseline sensitivity of the visual system is already calibrated for some longer term estimate of the environment, and that the brief aftereffects typically studied in adaptation ride atop longer term sensitivity adjustments that define the “pre-adapt” settings. These underlying adaptation states may be important for establishing the “intrinsic” sensitivity of the visual system that is then tuned by short-term adaptation to the immediate “extrinsic” context. However, very little is currently known about adaptation at these longer timescales or what visual attributes they adjust to.

The functions of adaptation

Perhaps the biggest lingering mystery in adaptation is the purpose that it actually serves in perception. A wide range of potential functions has been proposed. The most common involve coding efficiency (Clifford et al., 2007; Wainwright, 1999; Wark et al., 2007). Natural signals have distributions that are strongly peaked around a mean level, yet both the mean and the variance can vary widely and must be encoded by neurons with very limited dynamic range. This can be optimized by centering the neural response around the current average to prevent response saturation and to devote the best discriminative power (where the response function is steepest) to the most common signals in the environment (Laughlin, 1981). In turn, contrast adaptation can adjust the gain of the neural response so that its dynamic range is matched to the range of levels in the stimulus. An example where this is clearly critical is in the scaling of sensitivity to luminance and chromatic contrast. Because of the overlap of the cone sensitivities, the signals available for color (based on the differences in the cone responses) are many times smaller than the signals for luminance (based on the overall cone responses). Yet, the world does not appear deficient in color compared to lightness, and this is likely because the contrast sensitivity of chromatic channels is amplified to match the stimulus gamut (von der Twer & MacLeod, 2001). In turn, this balance across channels maximizes the information carried by each channel. Adaptation might also increase efficiency by decorrelating the responses across channels (Barlow, 1990b), though the evidence for this form of adjustment is weaker.

A second related function is to allow the visual system to build a predictive code where the mean expected stimulus is represented only implicitly so that responses instead signal the errors (Srinivasan, Laughlin, & Dubs, 1982). This is effectively a norm-based code. It has the advantage of increasing metabolic efficiency (Lennie, 2003) and may also aid the visual system in detecting novelty (Barlow, 1990a; Gardner et al., 2005; Ranganath & Rainer, 2003).

A third class of potential benefits involves error correction (Andrews, 1967) and constancy. As noted above, adaptation may play an important role in compensating for variations in sensitivity within the observer, by discounting the signals (e.g., spectral filtering or blurring of the retinal image) that arise from the observer. It also plays a central role in discounting variations in the stimulus. For example, adaptation to the mean chromaticity is thought to be an important first stage in adjusting to the illuminant to promote color constancy (Smithson & Zaidi, 2004). It is possible that adaptation is similarly important for removing extraneous information in other dimensions. For example, face recognition might benefit from adapting out the average configural properties of one’s social group so that the percepts are more directly tied to the individual’s unique (within the group) attributes (Webster & MacLeod, 2011).

The Lilac Chaser illustrates many of these predicted effects. At each spot, adaptation adjusts the mean response to match the average level of the stimulus. This removes the “predictable” colors, leaving the full resources of the visual system to represent the errors in the prediction (the afterimage). These novel stimuli become highly salient, and in an effect not shown, sensitivity to color differences becomes best for variations around the adapting level. At the same time, the adaptation is performing a form of error or constancy correction by filtering out the differences in the average color at different retinal locations. Note that under normal viewing conditions, color differences tied to a fixed retinal area would normally reflect sensitivity differences within the observer.

Figure 5 provides a further illustration, by simulating how adaptation might adjust to more extreme “natural” environments, such as the surface of Mars or an underwater scene. The upper panels depict how these appear to an individual adapted to a terrestrial environment on Earth, while the lower images depict how they should appear to individuals who are instead visually acclimated to either new environment. Again, this adaptation was simulated for each scene by adjusting the mean and contrast responses of a set of mechanisms so that the average response within each channel was the same as the response for a typical natural outdoor environment on Earth (Juricevic & Webster, 2009). The pairs of images thus again differ only in the state of adaptation, and it is clear that information is available to the adapted observer that is hidden in the eye of an observer matched to the wrong environment. Note that a similarly extreme example can be seen in Figure 3, where the mismatched image instead corresponds to an observer who is not adapted for variations in their own sensitivity.

Figure 5.

Simulations of changes in color appearance with adaptation to more extreme environmental variations. Upper images show the surface of Mars or an underwater scene as perceived by an observer adapted to a natural land environment on Earth. Lower images depict how the same scenes are perceived if observers are adapted within each new environment so that the responses in color mechanisms are matched to their responses under adaptation to the terrestrial Earth distribution.

One concern with these functional accounts is that so many different advantages appear to be gained from the same few tricks. However, there are costs to adapting. One is that gain changes can amplify not only signals but also the noise. If the limiting noise is prior to the response change, then the adjustments will not improve sensitivity (MacLeod, 2003; Rieke & Rudd, 2009). Moreover, adapting may be deleterious if it is driven by the noise rather than an actual change in the environment. For this reason, the timescales and processes of adaptation may reflect a complex inferential process of estimating when there is sufficient evidence to readapt (Wark et al., 2009). A second problem is what has been called the “coding catastrophe” (Schwartz et al., 2007). If downstream processes do not know that the state of adaptation has changed, then they may misinterpret changes in the pattern of activity, attributing it to the stimulus. This leads to the perceptual biases in aftereffects. The same calibrations that can promote constancy by removing extraneous information will thus lead to constancy failures when relevant information changes. This problem has been noted in color constancy in the context of the gray world assumption. Adapting to the average chromaticity in a scene will discount the illuminant if the average scene reflectance is gray but will adjust the appearance of object colors in the wrong way if the scene itself is biased, and thus resolving this ambiguity requires adjustments to the higher order color statistics of scenes (Golz & MacLeod, 2002). A final set of problems is that the same adjustments do not optimize all potential goals. For example, the adaptive changes that maximize efficiency are not necessarily the adjustments required to maintain constancy (Webster & Mollon, 1995). In addition, increasing the salience of novel stimuli can also be at odds with maintaining their appearance (McDermott et al., 2010).

A larger concern is that despite the many potential benefits, these are not often clearly manifest in measures of visual performance. The effects for constancy (for example, for changes in observers with aging) are in fact robust but have generated less interest than they possibly should, perhaps because the behavioral abilities they allow are not as clearly evident. Most studies have instead focused on sensitivity and discrimination, and as many previous reviews have noted, improvements in pattern discrimination are surprisingly weak in relation to the pronounced changes that adaptation induces in the pattern’s appearance (Clifford et al., 2007). One possible reason for this is that the coding problems the visual system faces may be very different at different stages. The retina must cope with a staggering array of light levels, and at this peripheral stage, it may be much more important to match the available dynamic range to the current stimulus (Rieke & Rudd, 2009). There is little question that adapting to the average light level improves vision. Yet, some visual attributes may vary too little to swamp the system’s dynamic range. For example, as we noted, the visual system does not clearly show short-term adjustments to the variance along many stimulus dimensions, and this may be because this variance is not large enough to saturate the responses. Moreover, at progressively higher levels, the opportunities increase for noise to intrude before the adaptation site, so that the benefits of gain changes for sensitivity are reduced. An interesting example is in face perception, which like many other patterns shows only mixed evidence for improved discrimination with adaptation (Ng, Boynton, & Fine, 2008; Rhodes, Maloney, Turner, & Ewing, 2007; Rhodes, Watson, Jeffery, & Clifford, 2010). The distribution of faces is again peaked around the average, and like luminance or chromatic variations, this might predict a sigmoidal response to optimally encode the stimulus differences (Bartlett, 2007). Yet, discrimination and adaptation have instead been found to be consistent with a largely linear response function for facial configurations, even over ranges larger than faces naturally vary (Susilo, McKone, & Edwards, 2010). Thus, at this level, adaptation may be less important for preventing response saturation, (though potentially still important for normalizing and balancing the responses across the coding dimensions). Nevertheless, adaptation to faces leads to appearance changes that closely parallel the changes in color arising from adaptation in the photoreceptors (Webster & MacLeod, 2011). Resolving the mystery of why the visual system adapts may thus depend on understanding why the adaptation is prevalent throughout the visual system and often in such similar ways, even though the constraints on sensory processing may be very different.