Visual Adaptation

Abstract

Sensory systems continuously mold themselves to the widely varying contexts in which they must operate. Studies of these adaptations have played a long and central role in vision science. In part this is because the specific adaptations remain a powerful tool for dissecting vision, by exposing the mechanisms that are adapting. That is, “if it adapts, it's there.” Many insights about vision have come from using adaptation in this way, as a method. A second important trend has been the realization that the processes of adaptation are themselves essential to how vision works, and thus are likely to operate at all levels. That is, “if it's there, it adapts.” This has focused interest on the mechanisms of adaptation as the target rather than the probe. Together both approaches have led to an emerging insight of adaptation as a fundamental and ubiquitous coding strategy impacting all aspects of how we see.

Introduction

The visual system adapts to change in many ways, over timescales ranging from milliseconds to millennia, and each adjustment may recruit a diverse array of mechanisms or subserve multiple functions. This makes it difficult to disentangle different forms of plasticity, such as adaptation versus learning (e.g. (McGovern et al., 2012, Harris et al., 2012)). Visual adaptation is typically defined operationally, as a brief and temporary change in sensitivity or perception when exposed to a new stimulus, and by the lingering aftereffects when the stimulus is removed (Webster, 2011). (See illustrations in online supplementary material.) A hallmark of these changes is that they are selective, reducing sensitivity for stimuli similar to the adaptor but not for patterns sufficiently different. Characterizing these selective changes reveals the coding strategies in the visual system, and equally importantly, how these codes are calibrated. A central insight from adaptation is that these codes appear to operate in functionally similar ways across diverse stimulus domains (Figure 1), pointing to a common repeating principle thatis itself not selective, and instead reaches the status of a universal law (Helson, 1964). This review emphasizes the pervasiveness of adaptation in perception and neural coding, how these adjustments operate within normal viewing, and what they suggest about both visual representations and visual experience.

Figure 1.

Examples of similar perceptual aftereffects across different stimulus domains. Left panel: adaptation to a mean color, blur, or faces produces roughly global shifts in the appearance of stimuli relative to a norm (e.g. gray, focused, undistorted), so that the adapting stimulus appears more neutral, with no aftereffect when adapting to the norm (n). This norm-based coding pattern and its adaptation can arise in mechanisms broadly tuned to the stimulus dimension, or if the stimulus itself is broad rather than punctate (e.g. representing a bias in the stimulus spectrum). Right panel: adaptation to color contrast, spatial frequency, or viewpoint instead biases the appearance of other stimuli away from the adapting level, with no change in the perceived level (e.g. size) of the adapt level itself, and produces similar aftereffects after adapting to any level across the dimension. This pattern implicates coding in multiple more narrowly-tuned channels.

The scope of visual aftereffects

If adaptation is an intrinsic feature of neural processing, then we should expect to see its signs throughout the visual stream, reflecting at each stage a plasticity for the kinds of information the neural circuits are designed to represent. That is, the types and patterns of visual aftereffects should be nearly as rich and complex as the gamut of our perceptions. This does not seem far from the truth. Aftereffects were originally described for seemingly “low-level” features of form, color and motion. However, as these are probed with more diverse stimuli, it is increasingly evident that each engages and adapts multiple processes and levels of the visual system. Adaptation to color includes a variety of distinct adjustments including sensitivity changes in the cones, “second-site” calibrations in post-receptoral pathways, and adjustments not only to the mean chromaticity but to the variance or contrast (Webster, 1996). These contrast adjustments are more selective for some axes of color space but can be tuned to any arbitrary direction, and led to the discovery of both cardinal and higher-order color mechanisms (Krauskopf et al., 1986). Adaptation to color can also be contingent on the form or motion of the pattern, as in the McCollough effect (in which different color aftereffects are generated for different orientations of the adapting pattern) (McCollough-Howard and Webster, 2011). Some of these adaptations contribute to color constancy (Foster, 2011), while others may reflect post-constancy stages, after surface color has been disambiguated from the illuminant (Goddard et al., 2010b).

Spatial adaptations such as tilt aftereffects can be induced with both real and subjective contours, with asymmetries between them suggesting adaptation at different cortical sites (Paradiso et al., 1989). The aftereffects also depend on whether the same edges appear to belong to the same or different objects (von der Heydt et al., 2005), and show varying dependence on whether the edges occupy the same position in space or on the retina (e.g. (Knapen et al., 2010)). Distinct shape aftereffects have also been found by briefly flashing adapt and test stimuli (Suzuki and Cavanagh, 1998). These are more susceptible to attention and less to low-level features compared to conventional tilt-aftereffects, again consistent with response changes at higher visual levels. High- and low-level shape aftereffects are also affected in different ways by perceptual grouping (He et al., 2012). Similarly, for motion, different aftereffects occur for static and dynamic test stimuli, for patterns defined by first-order (luminance) or second-order (e.g. contrast) motion, or for global motion such as an expansion or rotation (Mather et al., 2008). They can also be induced by adapting to static images implying motion (Winawer et al., 2008), and both motion and tilt aftereffects can be generated by mental imagery (Mohr et al., 2011, Winawer et al., 2010). These aftereffects again point to adaptation at multiple levels tied to processing different aspects and representations of movement.

Increasingly adaptation is also being used to probe “high-level” percepts based on more abstract and complex stimulus attributes. For example, observers adapt not only to surface color but to material properties such as glossy versus matte (Motoyoshi et al., 2007); to the perceived affordances of scenes such as how open or navigable they seem (Greene and Oliva, 2010); to the specific gaits implied by biological motion (Troje et al., 2006, Jordan et al., 2006); and to the causal structure (Rolfs et al., 2013) and synchrony (Roseboom et al., 2015) of events. Strong aftereffects are also induced in the perceived 3D orientation or viewpoint of objects (Fang and He, 2005) or bodies (Lawson et al., 2009) and in the direction of gaze (Jenkins et al., 2006). Among the most widely-studied class of high-level aftereffects is adaptation to faces (Webster and MacLeod, 2011). The perceived characteristics of a face can be strongly biased by the types faces seen previously. These adaptations occur for most of the dimensions we classify faces along, including identity, gender, expression, or ethnicity (Leopold et al., 2001, Webster et al., 2004). As with the different examples of form and motion aftereffects, many of these adaptations are assumed to reflect higher levels of visual coding, in part because they exhibit properties that cannot be accounted for by adaptation to low level features, an issue we return to below. However, regardless of their neural locus, such after effects reveal the pervasiveness of adaptation in affecting how we perceive nearly all aspects of the world.

Can we adapt to anything? Despite the rich varieties of visual aftereffects, there are also limits to how the visual system can adapt. One obvious limit is set by the selectivity of the adaptation. If patterns produce the same net activity within the sites controlling the adaptation, then they will not induce different states of adaptation even if the stimulus differences are distinguishable. In other cases, the visual system may not directly encode information in a way that can be adapted. For example, McCollough effects cannot be induced for any arbitrary pairing of color and form, even when the contingencies are readily apparent (McCollough-Howard and Webster, 2011). These limits are important because they suggest that the response changes with adaptation do in fact depend on and thus can help reveal the nature of the neural representations. The implication is that adaptation is probably manifest in every way we see, but in ways strongly constrained by how we see.

Neural Mechanisms

Paralleling the spread of perceptual aftereffects, physiological studies have revealed that adaptation is more extensive than previously expected. Originally the retina was thought to adapt only to the average light level or color. An array of sophisticated mechanisms combine to control these adjustments, switching between different classes of photoreceptors, gain changes in the receptors, and adaptation to the pooled receptor signals within retinal circuits (Rieke and Rudd, 2009). However, the retina of many species also adapts to stimulus contrast (Demb, 2008). This includes both a rapid contrast gain control as well as slower adjustments over many seconds. In the primate retina and geniculate this slow component is restricted to the magnocellular pathway (Solomon et al., 2004). A common principle underlying these different forms of adaptation might be gain changes driven by the mean level of the adapting stimulus (Kastner and Baccus, 2014). For contrast this could arise by altering the input (e.g. rectifying the contrast) so that variations around the mean are converted into a signal that varies in the mean level. Instead of simply filtering the image, the retina may also be built to detect complex features of the stimulus (Gollisch and Meister, 2010), and consistent with this, ganglion cells can also adapt to surprising properties such as differential motion or orientation. A potential site of these pattern-selective adjustments may be the synaptic inputs from bipolar cells (Gollisch and Meister, 2010).

Contrast and pattern-selective adaptation have been extensively studied in cortical cells (Kohn, 2007, Solomon and Kohn, 2014). Adaptation affects both the response gain of the cell and its tuning (Ohzawa et al., 1982, Movshon and Lennie, 1979) and depends on both intrinsic activity (Sanchez-Vives et al., 2000) and synaptic changes (Abbott et al., 1997). These selective aftereffects emphasize that for understanding the consequences of adaptation, the cell itself cannot be treated as an isolated functional unit. An important development of this idea is to consider how adaptation alters the network of interactions within the cortex (Solomon and Kohn, 2014). The response of cells depends on both the direct input they receive and a divisive scaling drawn from a broad pool that provides a suppressive surround beyond the classical receptive field and functions to normalize the neural responses (Carandini and Heeger, 2011). Adaptation that primarily targets this pool (e.g. by adapting to features to which the cell's receptive field is not sensitive) can reduce the normalization signal and thus lead to disinhibition and enhanced responses. The interplay of these two components can predict many of the complex characteristics of adaptation measured in individual cells in the retina and cortex (Solomon and Kohn, 2014).

Neural adaptation has also been probed at higher cortical levels along both the dorsal and ventral streams. For example, a number of studies have examined the consequences of motion-selective adaptation in MT, and of adaptation to objects in IT (Kohn, 2007, Solomon and Kohn, 2014). These have revealed potentially strong parallels between perceptual aftereffects and how the adaptation alters neural responses. For example, a recent study found that the effects of adaptation on the responses of face-selective cells in the human medial temporal lobe corresponded closely with the perceptual biases that adaptation induced in the observer (Quian Quiroga et al., 2014).

A further powerful approach for exploring the neural correlates of perceptual adaptation has been the technique of fMR-adaptation (Grill-Spector and Malach, 2001). The repeated presentation of a stimulus results in a decline in neural responses as measured by fMRI. If a change in the stimulus leads to a release from this suppression, then this implies that the underlying neural mechanisms are selective for the stimulus change, and thus that the stimuli are encoded by distinct neural populations. These adaptation effects have now been widely applied to examine the nature of visual representations (Malach, 2012, Weigelt et al., 2008), and have been especially important in overcoming the spatial sampling limits of traditional fMRI. How the response changes indexed by fMR-adaptation are related to changes in the actual neural activity remains uncertain (Krekelberg et al., 2006, Grill-Spector et al., 2006). Moreover, whether it is specifically a signature of perceptual adaptation is also unresolved, for it has also been interpreted as a correlate of priming or expectation (e.g. (Larsson and Smith, 2012)). However, the logic of the approach closely parallels the rationale behind perceptual adaptation, and the neural tuning properties it has revealed at progressive stages of cortical processing are in many ways in line with the visual representations inferred from behavioral studies.

Propagation of adaptation through the visual stream

Because adaptation is affecting neural processing at multiple levels, the signals available at any level will depend on how responses are adapted at other levels. Later stages will therefore inherit sensitivity changes arising at earlier levels. For example, the contrasts available to retinal and cortical mechanisms – and how these mechanisms adapt to them –depend on how the visual system is adjusted to the average light and color. The visual stream flows in many directions, and includes reciprocal connections and feedback throughout the hierarchy. Thus the potential also exists for earlier stages to inherit response changes arising at higher levels. Consequently, the effects of a sensitivity change at one site can be manifest and manipulated at many.

Consider the fate of color afterimages, which begin with the first steps of seeing but then percolate through the system to reach our awareness. After viewing a red patch, a gray patch appears greenish. The sensitivity changes generating the aftereffect originate in the cone receptors. However, the afterimage lasts several seconds. This is too long to depend on the cones, and instead parallels the sluggish after-discharge in ganglion cells (Zaidi et al., 2012). On a uniform field the color of the afterimage appears labile and diluted. Yet when an outline is added, the hue becomes strikingly vivid and stable (van Lier et al., 2009). This spatial context triggers cortical filling-in processes that spread the perceived color between the delineated image regions, even to locations at which the retina was not originally adapted. The strength of the afterimage can also be modulated by attention (Suzuki and Grabowecky, 2003) or interocular suppression (Tsuchiya and Koch, 2005) during adaptation, while the context also affects its perceived size (Sperandio et al., 2012).

Successive stages of adaptation can account for several aspects of how adaptation alters neural responses at different levels. As noted, these response changes can include both changes in gain and in tuning. Gain changes can arise from adaptation at both the level of the cell and in the inputs it inherits, while biases in selectivity can arise from changes in the distribution of inputs (with both also affected by the network of normalizing interactions within each stage) (Solomon and Kohn, 2014). For example, contrast adaptation alters sensitivity in the LGN but alters receptive field position in V1, because of the bias in the geniculate inputs to the receptive field (Dhruv and Carandini, 2014). Contrast adaptation in the LGN may in turn be partly inherited from the retina (Solomon et al., 2004). Motion-sensitive cells in MT show changes in motion tuning and localized adaptation in different regions of the receptive field, consistent with inheriting inputs adapted in V1, as well as adaptation arising directly in MT (Kohn, 2007). On the other hand, MT and V1 appear to adapt in similar ways when probed with similar stimuli (Patterson et al., 2014). Such results suggest that at each stage sensitivity changes are both generated and passed on to subsequent levels.

These serial effects have important implications for designing and interpreting perceptual experiments. For example, how can we know whether aftereffects for the abstract attributes of a face reflect adaptation within mechanisms that directly represent those attributes, rather than the low-level features of the stimulus? The appearance of a face can be biased by adaptation to non-face stimulus properties such as local curvature (Xu et al., 2008) or orientation gradients (Dickinson et al., 2010). Again such effects are not unexpected, because the response changes they directly generate are likely to be carried forward in the processing stream to alter subsequent codes from which they are assembled. However, a number of strategies have been devised to circumvent these feed-forward effects. One is to create “adaptation metamers,” stimuli that should induce the same response changes at a given stage of the visual system. This technique is routinely used to study pattern aftereffects, for example by moving the stimulus during adaptation so that the aftereffect is not simply owing to the “pattern” of local light adaptation in the retina. The time-averaged light level can also be controlled by counter phasing the pattern contrast. However, this has recently been shown to generate powerful aftereffects specific to the contour (Anstis, 2013). In a striking counter-example to the border-enhancement of color afterimages, adapting these contours out reduces the spatial context for filling-in, and can render the grayscales (but not the color) of an actual image invisible.

A further strategy for isolating different levels of adaptation is to take advantage of known differences in neural representations at different stages. For example, interocular transfer was important in establishing a cortical locus of pattern aftereffects. Many of these show partial transfer consistent with adaptation in both monocular and binocular mechanisms (Blake et al., 1981). Eye-specificity is progressively lost at higher levels, and some aftereffects exhibit complete transfer, implicating later sites (Raymond, 1993, Nishida et al., 1994). Higher visual levels are also characterized by greater invariance. We can recognize a face regardless of size, position or viewpoint, or global distortions such as stretching the image, and adaptation to faces partially transfers across these low-level image changes, as well as higher-order attributes such as how faces are categorized (Webster and MacLeod, 2011). This suggests that high-level aftereffects do depend at least in part on adaptation at the levels at which the attributes become explicitly encoded.

A fundamental issue for understanding the consequences of inheritance in adaptation and perception is whether the visual system is “aware” of the state of adaptation it is in (Series et al., 2009). In some cases the signals generated by adaptation can be actively suppressed as if they are recognized as illusory. For example, the same contour processes that exaggerate color afterimages can reduce or hide them when the spatial context is misaligned, and may explain why normally one has to look closely to see the afterimage signals our eyes are constantly generating (Powell et al., 2012). However, for most aftereffects the perceptual consequences appear equivalent to attributing the adaptation-induced biases to changes in the stimulus rather than the observer. That is, subsequent levels process the inputs without accounting for the states of adaptation that gave rise to them, giving rise to the “coding catastrophe” that underlies the perceptual biases in visual aftereffects (Schwartz et al., 2007).

Timescales

Just as the sites and types of visual aftereffects have expanded, so too have their timescales. These dynamics are critically important for understanding what kinds of information the visual system is tracking and calibrating for, yet to a large extent they remain poorly understood. A number of factors influence the rates at which adaptation should adjust. One is the timescales over which visual coding must operate. Light levels drift not only from day to night but vary dramatically even within individual scenes, where there can be a 1000-fold range of luminance and contrast. To be able to see within areas in both shadow and sun requires rapid and local adjustments each time we change fixation (Rieke and Rudd, 2009). On the other hand, if adaptation is too rapid then scenes would quickly fade and information would be lost about the broader context. For example, adapting to the average illumination in the scene is important for lightness and color constancy. For spatially local mechanisms this average requires pooling information over time across multiple surfaces and fixations, and thus requires a longer memory.

A second factor is that adaptation should adjust sensitivity quickly when the world actually changes while responding slowly enough to maintain sensitivity when the changes are instead noise. Light adaptation is faster at higher light levels where less pooling is needed to estimate the signal. The rate of adaptation in retinal ganglion cells also adjusts depending on the time required to estimate a change in the mean luminance or contrast, with faster recalibrations when there is better evidence for a change (Wark et al., 2009).

Finally, the world and the observer both themselves change over many different timescales, and the dynamics of the adaptation should be matched to appropriately track these (Kording et al., 2007). Changes that are fleeting should be compensated by more rapid but transient response changes than when the changes are gradual but persistent. These multiplexed timescales have again been most clearly revealed in analyses of visuomotor adaptation (Shadmehr et al., 2010, Wolpert et al., 2011). For example, the timescales of muscle fatigue are very different from those of muscle damage or development. An optimal observer should estimate the source of the error to set the timescale of the adaptation, and these estimates depend on the temporal pattern of the errors. Models of this kind predict a number of properties of the dynamics of visuomotor and saccadic adaptation, and similar strategies are likely involved in calibrating many aspects of visual coding.

Traditional studies of adaptation have focused on brief intervals ranging from seconds to minutes. Over this range the strength and duration of aftereffects increases as a power law of the adapting duration (Greenlee et al., 1991). These dynamics appear similar for stimuli as different as gratings and faces, again suggesting common mechanisms (Leopold et al., 2005). Rather than passively decaying, the duration of visual aftereffects depends in part on re-exposure to stimuli. Interposing an interval between the adapt and test stimuli can result in “storage” of the adaptation such that stronger aftereffects persist at longer durations. In some cases this can result in effectively permanent changes if an appropriate de-adapting stimulus is not experienced (Vul et al., 2008).

Recent work has extended adaptation to both shorter and longer timescales. Motion aftereffects can be elicited by exposures as brief as 25 ms, paralleling the rapid adjustments seen in single cells (Glasser et al., 2011). Results of this kind are important in showing that pattern adaptation is not merely a consequence of over-stimulation, but rather is a process reactivated by each glance. A number of recent studies have also explored adaptation to longer durations, spanning hours or days. These have revealed longer-term aftereffects that may emerge only after sustained periods of adaptation. For example, long-term exposure to a colored bias induces aftereffects that are much more persistent than short-term chromatic adaptation (Neitz et al., 2002, Belmore and Shevell, 2010, Eisner and Enoch, 1982), and color vision in cataract patients requires months to readapt after surgery (Delahunt et al., 2004). Similarly, while brief adaptation to blur or contrast is sufficient to induce a strong perceptual aftereffect, hours of exposure are required to see increases in acuity and sensitivity (Mon-Williams et al., 1998, Kwon et al., 2009, Zhang et al., 2009).

An important question is whether these longer-term changes are merely stronger adaptation or whether they tap mechanisms tuned to different timescales. One way this has been tested is by pitting long and short term adaptation against each other. The aftereffect of adapting to a prolonged stimulus (e.g. a clockwise tilt) can be extinguished by brief adaptation to the opposite stimulus (e.g. counter clockwise). However, this brief de-adaptation is itself short-lived, so that as it decays the aftereffect to the longer stimulus re-emerges. This “spontaneous recovery” indicates that there are at least two distinct mechanisms adapting at different rates (Shadmehr et al., 2010). These effects have now been demonstrated for a wide range of stimuli (Mesik et al., 2013, Vul et al., 2008). Distinct components of adaptation have also been inferred from non-monotonic effects of continuous adaptation over days (Haak et al., 2014) or even over minutes (Chopin and Mamassian, 2012), though the latter has also been accounted for by a single adjustment (Maus et al., 2013).

A further important issue is whether mechanisms tuned to different timescales are inducing the same response changes but tracking different rates, or whether they are adjusting visual coding in qualitatively different ways. Kwon et al. adapted observers for 4 hours by wearing goggles that reduced contrast (Kwon et al., 2009). After this period contrast sensitivity improved, and the pattern of adaptation shifted from resembling the contrast gain typical of short timescales to a change in response gain. This shift could reflect a shift in strategy of contrast coding from one driven by the current mean contrast at short timescales to the overall range of contrast at longer durations. It is possible that these differences also reflect different timecourses and consequences of adaptation within cortical cells and their suppressive surrounds. Over much briefer durations, the pattern of adaptation in single cells varies in qualitatively different ways as adaptation rebalances the relative responsivity of the cells and their gain control (Patterson et al., 2013).

Adapting to the environment

Traditionally, the effects of adaptation have been studied by exposing observers to highly artificial stimuli and viewing contexts. J.J. Gibson, who first documented the tilt-afereffect, in later years dismissed such aftereffects as irrelevant to natural vision, arguing that they arose only when observers were engaged in unnatural tasks (Gibson, 1986). Yet this missed the point that the processes of adaptation are always engaged. Many now consider it evident that adaptation is an essential part of natural viewing, and that adaptation itself can only be understood within the context of the visual worlds we normally see within. This realization has been driven by the growing understanding of the importance of natural image statistics for all aspects of perception (Simoncelli and Olshausen, 2001, Geisler, 2008).

The characteristics of the environment constrain the mechanisms and consequences of adaptation in many ways. First, how the world varies shapes how vision is designed to cope with those variations. One of the greatest natural challenges is how to see over the enormous range of light levels during the course of a day. The magnitude and dynamics of these changes shaped many design features of the retina (Rieke and Rudd, 2009). Large changes in mean luminance and contrast occur even within different parts of the same scene. Analyses show that these image statistics vary independently, and predict light and contrast adjustments that operate independently (Mante et al., 2005). Natural image statistics also point to the most efficient representations of images and how they vary, and again predict how neural responses should vary to encode them (Wainwright, 1999, Wark et al., 2007).

How vision is adapted to the natural world also determines the natural operating states of the visual system. It is common to use a “standard observer” to characterize spatial or spectral sensitivity. However this observer is relevant only to the stimulus context in which it is embedded, and is often based on states of adaptation we rarely encounter outside the lab. For example, natural images have a characteristic 1/f amplitude spectrum. Adaptation to this structure selectively peduces sensitivity at lower frequencies, so that the effective contrast sensitivity function is more bandpass (Webster and Miyahara, 1997, Bex et al., 2009), even for chromatic contrast (which is normally taken to be lowpass) (Webster et al., 2006). The relative sensitivity to luminance and chromatic contrast also reflects adaptation to the world. Because the cones have overlapping spectral sensitivities, the cone contrasts for color are much smaller than for luminance. Yet post-receptoral gains are scaled to offset this imbalance (MacLeod, 2003), such that the relative salience of luminance and color is matched to the range of luminance and chromatic variations in natural scenes (McDermott and Webster, 2012). This also predicts the relative scaling for different color dimensions. Many natural scenes have a blue-yellow bias, and adaptation to the natural color gamut leaves us less sensitive to this blue-yellow variation (Webster and Mollon, 1997, Goddard et al., 2010a).

Substantial visual differences also exist within and between different environments, and these are important for understanding how much the states of adaptation and thus perception might vary between individuals or in the same person over time. For example, colors cycle with the seasons, and vary widely across ecosystems, so that color perception itself should vary with time or location (Webster and Mollon, 1997), (Figure 2). Similarly, social environments vary systematically in facial attributes such as ethnicity or age. We are better at discriminating small differences in color around the mean color we are adapted to, and a plausible account of the “other race effect” is that we are similarly tuned through adaptation to the average faces we are exposed to (Webster and MacLeod, 2011).

Figure 2.

Simulations of color adaptation to a change in environment or change in observer. Top left and middle: roughly the same scene in two seasons. Bottom left and middle: adapting to the color statistics in each season biases color appearance by toning down the dominant hues and increasing the salience of novel hues (e.g. increasing the perceived saturation of greens in the arid scene). Thus the same observer codes color differently in the two environments. Top right: the middle, arid scene viewed through the lens of an older observer. Bottom right: adapting to the spectral changes introduced by the lens removes most of the color bias. Thus the two different observers code color similarly when adapted to the same environment.

While the natural world shaped the evolutionary design of our visual system, in many ways it no longer corresponds to the carpentered visual worlds many of us now occupy. This raises intriguing questions about how processes of adaptation that were built to operate within natural scenes might function within the new and sometimes arbitrary visual environments we are increasingly exposed to. Cultural practices vary widely and thus expose individuals to very different visual diets. Thus the “cultural relativity” of some perceptual judgments could actually reflect “universal” processes of adaptation operating within different contexts (Webster et al., 2005). Within a culture, styles and aesthetics evolve continuously, and the dynamics of how these become incorporated into a society may similarly depend in part on how members visually adapt to these changes (Carbon, 2011). Individuals also increasingly occupy specialized and unique visual niches, and technology has created or made it possible to explore a diverse array of new visual worlds. For example, radiologists spend hours at a time inspecting medical images that have their own characteristic statistics, and adaptation to these statistics may influence how these images are perceived or interpreted (Kompaniez et al., 2013). Fully adapting to new environments may require long periods. However, to the extent that the effects of adaptation can be predicted, algorithms could be developed to adapt the images rather than the observer so that they are optimized for the calibration the observer walks in with (Webster, 2014).

Adaptation and compensation

If the states of adaptation vary as the same observer moves among different worlds, what of different observers immersed in the same world? In this case the same processes should adjust each to the common prevailing environment in spite of possible inherent differences in their visual systems. This side of the calibration – adjusting to variations in the observer rather than the environment – highlights the role of adaptation in compensating or “error correcting” vision for the vagaries of the individual's visual system. In many instances this may also be the more important side, for the visual system may often vary more than the properties of the world. For example, most of the blur in the retinal image arises from the optics of the eye rather than the scene itself.

There are many examples of these compensatory adjustments. Sensory-motor control is constantly recalibrated to maintain the coordination of visual and proprioceptive signals and to adjust to physiological changes such as injury or fatigue (Shadmehr et al., 2010, Wolpert et al., 2011). In color vision, individuals vary widely in their spectral sensitivities because of differences in the density of screening pigments or in the sensitivity and relative numbers of cones. Yet for the most part these differences fail to predict individual differences in color percepts. Thus the stimulus that appears white shows little change with age despite the steady yellowing of the lens with age (Werner and Schefrin, 1993) (Figure 2), and what looks yellow is unaffected by the enormous inter-observer differences in L/M cone ratios (Brainard et al., 2000). Similarly, many aspects of spatial vision are adjusted to discount for variations in spatial sensitivity. Adaptation to a blurred or sharpened image recalibrates the stimulus that appears in focus (Webster et al., 2002). Individuals vary widely in the magnitude and pattern of optical aberrations, and thus in the amount and type of blur they are routinely exposed to. Yet they tend to agree on the physical stimulus that appears in focus. This occurs because each is adapted to discount their own native blur from their perception (Artal et al., 2004, Sawides et al., 2011, Radhakrishnan et al., 2015).

The same adjustments are also critical for calibrating for sensitivity differences within the observer. Sensitivity varies markedly across the visual field, yet the appearance of the world degrades much more gracefully. Color percepts between the fovea and near periphery remain very similar despite large changes in factors such as macular pigment screening (Webster et al., 2010); and perceived focus is adjusted for the declines in spatial resolution with increasing eccentricity (Galvin et al., 1997). Processes like adaptation also correct for sensitivity differences at a common locus. For example, the band pass tuning of spatial contrast sensitivity at threshold gives way at visible contrasts levels to a scaling that is instead independent of spatial frequency (Georgeson and Sullivan, 1975).

How far can these processes go to compensate perception for the idiosyncrasies of the observer, or the environment? We can partly answer this for ourselves by asking how gray or focused the world appears through our own eyes, or when we compare these judgments across the visual field. Formal tests suggest adaptation provides nearly complete compensation for some properties but is limited for others. One constraint is that, as noted, adaptation is less able to overcome sensitivity limits than appearance, because thresholds depend on less malleable constraints like sampling and noise. For appearances, adaptation can factor out many sensitivity variations but not all. Perceived focus is largely corrected for the optical aberrations of the eye, but for higher-order aberrations is more effectively adjusted to the magnitude than the specific blur pattern (Sawides et al., 2012). Compensation of color appearance for lens and macular pigment is better than predicted by adapting to the average color alone – suggesting it goes beyond independent gain changes in the cones – but residual errors remain (Bompas et al., 2013, O'Neil and Webster, 2014). An interesting example of the limits of adaptation for color perception is anomalous trichromats. Their longer-wave cones have very similar spectral sensitivities and thus the differences they convey are much weaker. Yet many seem to experience stronger color percepts than their reduced sensitivity predicts (Neitz et al., 2002, Regan and Mollon, 1997, Boehm et al., 2014). Again this could occur if post-receptoral mechanisms amplify their gain to match the weakened inputs. On the other hand, color salience remains weaker and hue loci are altered relative to trichromats, suggesting that this compensation is incomplete.

The extent to which adaptation normalizes visual coding also depends on how much the processes of adaptation themselves vary. Little is known about individual differences in adaptation, though observers consistently differ in the magnitude and pattern of aftereffects (Elliott et al., 2012, Vera-Diaz et al., 2010), and some of these differences may reflect polymorphisms in the genes coding neurotrophins (Barton et al., 2014). Adaptation also varies over the lifespan. The kinetics of light adaptation vary with development and aging (Owsley, 2011, Brown and Lindsey, 2009). Less is known about cortical adaptation. Pattern selective adaptation can be seen with evoked potentials as early as 3 weeks of age (Suter et al., 1994), but whether it shows developmental changes has not been well documented, and it is difficult to distinguish changes in plasticity from changes in other factors such as stimulus selectivity. In aging, declines in neural tuning linked to decreased inhibition have been associated with senescent changes in some cortical aftereffects (Wilson et al., 2011). Yet in adaptation to blur or chromatic contrast, the strength remains similar or even stronger in older observers (Elliott et al., 2012, Elliott et al., 2007). This suggests that many aspects of adaptation remain stable across the lifespan, and this stability might be essential to stabilize visual perception for the many optical and neural changes that do occur.

Adaptation, channels, and norms

As noted, the popularity of adaptation studies continues to be driven primarily by its use as a tool for probing vision. New examples are constantly reported where a mechanism is inferred by demonstrating selective adaptation to the stimulus it is presumed to encode. Such measurements have played a central role in defining the number and tuning of visual “channels” encoding different stimulus dimensions (Graham, 1989). It is because of this that adaptation is repeatedly described as “the psychologist's electrode.”

But at what level of the system is this electrode recording? This issue is murky, because the relationship between psychophysically defined channels and single cells is poorly understood. If an adaptation effect is selective, then clearly the stimulus differences must be represented in the underlying neural code. Yet this places little constraint on how or whether the neural code is “tuned” for those differences. In some cases there is striking agreement between the tuning functions derived from adaptation and individual neurons. How color appearance is adapted to uniform fields corresponds closely to the spectral sensitivities of the cones, and the bandwidths of aftereffects for orientation, spatial frequency, or color contrast approximate the average spatial tuning of V1 cells. In such cases adaptation may hold promise for tapping deep into the neural architecture. On the other hand, it remains challenging to reconcile visual aftereffects with the enormous heterogeneity of cell responses found at many levels of the system (Tailby et al., 2008, Webster and Mollon, 1994). Moreover, a number of studies have pointed out that this problem is inherently ill-posed, in that the same aftereffects could arise from very different channel structures (Hegde, 2009, Ross et al., 2014, Mur et al., 2010). This is further complicated by the fact that it is sometimes hard to disentangle properties of channels from properties of the stimulus. For example, whether the channels appear broadly or narrowly tuned depends in part on whether the stimulus itself is best conceived as broad or punctate (Elliott et al., 2011). This difference may seem evident for a single wavelength vs. a light spectrum, but is less certain for many of the complex attributes now probed with adaptation. For example, it remains unclear whether blur or a face is represented as an explicit feature or the envelope of many.

Problems of this kind suggest we cannot use adaptation to unambiguously decipher the architecture of the visual code. Yet in other ways the channel structure it reveals is profoundly important and simple, because it directly measures the tuning characteristics of the mechanisms mediating how the visual system responds to and recalibrates for changes in the stimulus context. That is, the channels that adaptation measures are by definition the functional ways in which vision can selectively adapt. What do these indicate about visual representations? An important distinction has been made between two general classes of visual codes in which a stimulus dimension is represented either by its absolute or relative value (Valentine et al., 2015). In the former, known as exemplar-based or multi-channel coding, different levels of the dimension are represented by different channels, each tuned to and labeled for a narrow range. Thus the stimulus is encoded by which channels are most responsive. In the latter or norm-based code, the stimulus is encoded by the relative activity of broadly-tuned mechanisms, and thus the stimulus level is conveyed by how responsive the channels are. The differences between these models thus amount to whether the channels themselves are narrowly or broadly tuned, but they make very different predictions about the pattern of visual aftereffects (Webster, 2011, Webster and MacLeod, 2011).

In multi-channel codes, adaptation has more localized effects by reducing sensitivity in the channels tuned to the adapting stimulus, thus pushing away the appearance of nearby stimuli. In norm-based codes, the response changes are more global, and tend to recenter the appearance of all stimulus levels so that the adapting stimulus appears more neutral (Figure 1). Both types of aftereffects are prevalent, and for many stimuli both patterns occur (Storrs and Arnold, 2012). For example, the tilt aftereffect includes both a repulsion and renormalization of perceived orientation that intriguingly unfold at different rates (Muller et al., 2009). In both models an important consequence of adaptation is to normalize responses across mechanisms. In multi-channel models this amounts to equating sensitivity across channels so that their average responses are similar within the current context. This is seen in the recalibration of cone sensitivities so that the response to the mean color is equivalent to gray, and in how populations of cortical cells adjust to compensate for a bias in the distribution of orientations (Benucci et al., 2013). In norm-based codes it reflects the rebalancing of sensitivities to neutralize the intensity of the response. This could occur within a pair of mechanisms that signal opposing levels of the stimulus, in which the norm is represented implicitly by equal responses within the mechanisms; or alternatively, could reflect an actual opponent mechanism in which the norm is represented by an explicit null point (e.g. the null in a color-opponent cell). These norms are important because they reflect the responses states that the visual system is constantly readjusting to achieve. As a consequence, visual coding is always relative because the response is always contingent on the current norm.

The norms established by adaptation have further significance because they represent unique and special states in the neural code – in which there is no bias in the neural responses. Are there perceptual correlates of these states? The subjective experience of many stimulus dimensions varies relative to a point that itself appears psychologically neutral. Thus colors are more saturated the more they differ from gray, and a face's identity becomes better articulated the farther it deviates from the prototype, which itself appears indistinct. The stimuli that define these perceptual norms are likely to correspond to the stimuli the visual system is currently adapted to, and evidence for this is that the stimulus that appears subjectively neutral to an observer is the same stimulus that does not produce a visual aftereffect (Sawides et al., 2011, Webster and Leonard, 2008, Radhakrishnan et al., 2015). This argues against the possibility that perceptual norms are arbitrarily shaped by criterion (e.g. corresponding simply to how we label what we are used to seeing), and that they are instead direct manifestations of how sensitivity is calibrated. The problem of whether special percepts reflect special neural states is unresolved, and continues to plague fields like color vision where a neural basis for pure hue sensations has yet to be found. Yet in the case of pure gray, or norms more generally, there may be a simple and direct link between our subjective experience of what appears neutral and what is objectively neutral in the underlying neural code, both set by the state of adaptation. Moreover, this adaptation is again inherently asymmetric – the mind is fit to the world. Thus the norms in our percepts and neural coding are both set by the norms in the environment, all linked through the fact that we are adapted to the environment. We still have no way of revealing the subjective mental experience of another, but at least some aspects of the problem of other minds is arguably traceable through adaptation to objective properties of the physical world. Whether two individuals experience the same stimulus as phenomenologically neutral – as a “gray” percept – may depend on the extent to which they are adapted to the same or different worlds (Webster et al., 2005).

The functions of adaptation

The presence of adaptation throughout visual processing provides both insights and challenges for understanding the potential roles it plays. On the one hand, this prevalence highlights that adaptation is fundamental to neural computations at all stages, pointing to a common purpose. On the other, the nature and demands of visual coding change at different stages, suggesting that why the code adapts may also change across levels or tasks.

Many disparate roles have been posited for adaptation. One set emphasizes efficient coding and optimizing information (Wark et al., 2007, Wainwright, 1999, Rieke and Rudd, 2009, Clifford et al., 2007, Stocker and Simoncelli, 2006). Neurons have limited dynamic range but must potentially encode an enormous range of stimulus levels. By adjusting this range to the ambient level, adaptation prevents response saturation and balances sensitivity across mechanisms, maximizing the information they can carry. Efficiency might also be enhanced by reaollocating resources (Gepshtein et al., 2013) or removing redundancies between mechanisms (Barlow, 1990b). These information theoretic approaches have been highly successful in predicting the properties of both visual coding and adaptation at early levels of the visual system, where initially both the range of stimulus levels and the need for compact codes is especially important. They also account well for the marked effects of adaptation on visual sensitivity – without light adaptation we would literally be blind most of the time. It is likely that the same principles are necessary to understand how higher-level representations are calibrated, especially for first establishing the sensitivities. That is, adaptation may be essential for initially adjusting the operating curve of each mechanism for the range of stimuli it is likely to receive. But it is less clear why higher visual codes require continuous readjustments. Few visual attributes vary over the range that light levels do, and it has been difficult to demonstrate impressive improvements in sensitivity or discrimination with pattern adaptation.

A second set of functional accounts focuses on the changes that adaptation produces in appearance rather than performance. It is an important and striking fact that very strong visual aftereffects occur even when there is no sign of a change in sensitivity. What role might these play? One account is that they reflect error correction (Andrews, 1967). If the world looks tilted or too yellow then a possible cause is that the neural code is biased, and this bias can be corrected by adaptation. This function is implied by how adaptation compensates for properties of the observer, or enables constancy when it is the stimulus (e.g. the lighting) that changes (Foster, 2011). In such cases, adaptation acts to filter out uninteresting variations in the context (e.g. the observer's sensitivity), to provide stable percepts of the informative properties.

A final putative set of roles involves building predictions about the world (Srinivasan et al., 1982, Chopin and Mamassian, 2012). Adaptation tends to null out the stimuli we are exposed to. These stimuli represent our current expectations about the world, and an efficient representation is to encode these expectations implicitly, as a null response. As noted, these nulls also correspond to norms. Predictive coding saves metabolic resources and allows the system to use its full capacity to signal only the errors, or unexpected stimuli, and may help make these errors more conspicuous (Barlow, 1990a, Gardner et al., 2005, Ranganath and Rainer, 2003, Boehnke et al., 2011). Supporting this, adaptation has been shown to enhance salience and visual search for novel stimuli (McDermott et al., 2010, Wissig et al., 2013). This also suggests that in building the current prediction, adaptation plays a central role in determining what captures our attention. Much of what we notice about the world may be a visual aftereffect, driven by the world we are currently adapted to (Barlow, 1990a, Webster et al., 2005).

In some cases these functions may be incompatible. For example, different adjustments may be required to optimize detection versus identification (Hillis and Brainard, 2007) or to promote constancy versus efficiency (Webster and Mollon, 1995) or salience (McDermott et al., 2010). But a definitive test between different functional accounts remains lacking, and it is probably a futile exercise to try to choose between them. All of them make sense, and all of them seem necessary. Moreover, for most if not all there is good evidence that these goals are in fact achieved. A more fruitful approach for understanding the universality of adaptation may be to instead note that most if not all of these benefits can be realized by a single shared adjustment - normalizing neural responses for the mean stimulus level (Webster, 2014). Like other canonical computations (Carandini and Heeger, 2011), this adjustment probably represents a universal principle that is woven into the fabric of any neuron's response, and thus affects everything we see.