Eye smarter than scientists believed: Neural computations in circuits of the retina

Summary

We rely on our visual system to cope with the vast barrage of incoming light patterns and to extract features from the scene that are relevant to our well-being. The necessary reduction of visual information already begins in the eye. In this review, we summarize recent progress in understanding the computations performed in the vertebrate retina and how they are implemented by the neural circuitry. A new picture emerges from these findings that helps resolve a vexing paradox between the retina’s structure and function. Whereas the conventional wisdom treats the eye as a simple pre-filter for visual images, it now appears that the retina solves a diverse set of specific tasks, and provides the results explicitly to downstream brain areas.

Introduction

The retina is a neural circuit of marvelous anatomical complexity that famously fascinated Cajal (Cajal, 1893) and has since drawn many researchers in its spell. Every technical advance in microscopy, imaging, or cell labeling has further reinforced the message of intricacy and precision in the retina’s wiring. At last count, the network is composed of at least 50 clearly distinct cell types (Masland, 2001) (Figure 1A). They differ widely in shape, from very local neurons tens of micrometers in size to some whose processes span clear across the eye. The neurons are arranged in 3 cellular layers and interconnected in the intervening 2 synaptic layers (Figure 1B–C). On a finer scale, one finds finer structure: Within the inner plexiform layer one can distinguish at least 10 thin sublayers, and the processes of a given cell type are often restricted to just one of these (Wu et al., 2000; Wässle, 2004). On the single-neuron scale one finds even greater specificity; for example, the blue ON bipolar cell connects only to blue cones, even though these form a tiny minority among photoreceptors (Dacey and Packer, 2003; Haverkamp et al., 2005). The retina’s output is conveyed to the brain by many different ganglion cell types, numbering about 15 in mammalian retinas. The population from each type tiles the visual field and thus conveys a complete but processed visual image (Wässle, 2004). Much of this intricate and specific structural organization is conserved from mouse to man, indicating that it serves a continuing computational purpose common to many animals.

Figure 1. Retinal circuitry.

A: Diversity of retinal cell types. For all five classes of retinal neurons – photoreceptors (P), horizontal cells (H), bipolar cells (B), amacrine cells (A), and ganglion cells (G) – a number of types can be identified according to morphological characteristics and dendritic stratification patterns. The image shows the major cell types of a typical mammalian retina. (Reprinted with permission from Masland, 2001.)

B: Specificity of retinal wiring. Double immunostaining for calbindin (red) and calretinin (green) in a vertical section of mouse retina (Haverkamp and Wässle, 2000) visualizes some of the structure and complexity of the retinal network. The staining labels horizontal cells, certain amacrine cells in the inner nuclear layer (INL), and some ganglion cells in the ganglion cell layer (GCL). The interposed outer and inner plexiform layers (OPL and IPL) are the sites of massive and often very specific synaptic contacts between the various cell types. For example, the labeled amacrine cells and ganglion cells extend their dendrites into three distinct thin strata of the IPL, which underscores the specificity of retinal microcircuits. (Reprinted with permission from Haverkamp and Wässle, 2000; see also Wässle, 2004.)

C: Schematic drawing of connections between the basic cell classes. The neurons in the retina are connected through chemical synapses that are either sign-preserving (excitatory, closed circles) or sign-inverting (inhibitory, open circles). In addition, one finds a considerable amount of electrical coupling between cells via gap junctions within all cell classes (not shown) and across some types of cells (marked by resistor symbol). The input into the network is incident light, which hyperpolarizes the photoreceptors. The connections from photoreceptors to bipolar cells are of either sign, producing both OFF-type and ON-type bipolars. Horizontal cells provide negative feedback and lateral inhibition to photoreceptors and bipolar cells. Bipolar cells are reciprocally connected to amacrine cells with chemical synapses and, for some types, through electrical gap junctions. Ganglion cells represent the output layer of the retina; their axons form the optic nerve. They collect excitation from bipolar cells and mostly inhibition from amacrine cells. In addition, ganglion cells and amacrine cells can be electrically coupled. This general connectivity sets the framework for any specific retinal microcircuit.

But what is that purpose? Most vision researchers will argue that the retina’s principal function is to convey the visual image through the optic nerve to the brain, where the cortex can bring a great deal of clever circuitry to bear on it. They may acknowledge that light adaptation is an important retinal function, akin to an automatic gain control. On further thought, the retina also implements some lateral inhibition – embodied by the center-surround antagonism in the ganglion cell’s receptive field – to sharpen the image in space and also in time. This picture of the retina as a simple spatio-temporal pre-filter is espoused almost universally by textbooks and review articles (including one by a present author: Meister and Berry, 1999). And it is adopted, at least implicitly, by virtually all neuroscientists who work in visual areas beyond the retina, where the truly sophisticated, heavy-duty computations are thought to take place. Yet the paradox is clear: To implement simple functions, like light adaptation and image sharpening, there is no need for 50 neuron types with fantastically intricate network structure. In fact, the retina of the horseshoe crab accomplishes all this already within the layer of photoreceptors (Ratliff and Hartline, 1959; Fuortes and Hodgkin, 1964). What are the other 49 cell types doing in the vertebrate retina?

There is a distinct possibility that we haven’t yet understood what the retina is for. What if it is not merely a sharpening filter for a cable to the visual cortex? Perhaps each of the many ganglion cell types already computes something rather specific about the visual scene. Each type would then need a dedicated neural circuit to extract the visual feature of interest. In this picture, the downstream areas in the brain receive not a generic pixel representation of the image, but a highly processed set of extracted features. Indeed, there is a well-known example of this kind of processing: the direction-selective ganglion cell. These neurons respond strongly to moving stimuli, such as traveling spots or bars, but they greatly prefer one direction of motion over the others (Barlow et al., 1964; Taylor and Vaney, 2003; Demb, 2007). The phenomenology is remarkable; for example, only a tiny movement of the spot through 1/10 of the cell’s receptive field is needed to elicit the direction-selective response. Now there is something distinctly different about this direction-selective processing compared to a center-surround pre-filter. First, it computes a specific feature of the visual input, namely the direction of movement within the receptive field. Second, as a result of this specificity, a good amount of stimulus information is discarded; for example, the cell does not fire at all for certain directions, regardless of the details of the moving object. Third, the result of the computation is represented explicitly in the response of the cell. Firing vs. not-firing indicates whether the spot moves one way or the other. Finally, no “higher processing” is needed to extract the information. For example, a downstream neuron could obtain the exact angle of the spot’s trajectory simply by pooling the firing of various direction-selective ganglion cells in a weighted summation.

Here we explore the notion that this kind of processing, namely the selective computation of specific stimulus features, is not the exception but the rule in retinal function. In doing so, we will repeatedly encounter the above-mentioned characteristics of task specificity, selective encoding of information, and explicit straightforward representation. At the core of these abilities lie certain strongly nonlinear processing steps, and identifying these key nonlinearities gets to the heart of the retina’s computations. The popular concept of linear spatio-temporal pre-filtering may well apply to particular kinds of retinal ganglion cells under certain conditions; but in other cases, the classic center-surround receptive field reflects a crude average of the ganglion cell’s behavior under stimuli that fail to probe its function properly. As we will see, it helps to work with visual stimuli that somehow reflect the actual challenges the visual system faces in its natural environment. In a search for general computational abilities of the vertebrate retina, we will focus on visual tasks relevant to all species: detecting light at low intensity; dealing with image motion caused by objects in the scene or the movement of the observer; and adapting to changing visual environments. Because of the generic nature of these tasks, we will freely discuss results obtained from different animal models.

Light detection

The most straightforward task of the visual system is the detection of dim lights. Human observers can sense a flash of light even at very low intensities that lead to only a handful of successful photon absorptions in the retina (Hecht et al., 1941; Sakitt, 1972). Correspondingly, rod photoreceptors display small responses to single photon absorptions, ~1 mV in amplitude (Baylor et al., 1979; Schneeweis and Schnapf, 1995), and retinal ganglion cells can indeed signal these events to the brain (Barlow et al., 1971). A ganglion cell typically collects inputs from many hundreds of rods (Sterling et al., 1988). Thus the computational challenge for the retina lies in separating the small single-photon signal in one or a few rods from the continuous electrical noise that is present in all photoreceptors. Indeed the problem arises already at the first stage of convergence, where the rod bipolar cell collects the outputs from tens of rods via graded synapses (Freed et al., 1987; Tsukamoto et al., 2001). How the retina sorts the sparse signal from the ubiquitous noise is beginning to be understood.

The light-independent fluctuations in the rod’s membrane potential are of two kinds: One, called “discrete noise”, results from spontaneous thermal isomerization of the photopigment (Baylor et al., 1980). These events are identical in all respects to authentic single-photon signals, and thus cannot be separated out. In fact, human visual sensitivity at absolute threshold is likely limited by this noise source (Barlow, 1956; Baylor, 1987). The other kind, called “continuous noise”, arises from spontaneous activations within the chemical transduction machinery downstream of photon absorption (Baylor et al., 1980). As a result, it has a different frequency spectrum from the single-photon signal. This spectral difference could support a separation of signal from noise by the method of temporal filtering: enhancing the frequencies that primarily contain light signals and suppressing the others. One can predict the optimal filter function for this task (Bialek and Owen, 1990), and indeed a transformation of the predicted type is observed in the transmission from rods to bipolar and horizontal cells of the salamander retina (Armstrong-Gold and Rieke, 2003). The filter appears to be implemented presynaptically in the rod through a combination of electrical coupling between rods and calcium dynamics at the transmitter release sites.

The temporal filtering improves the signal-to-noise ratio for photon events coming from a single photoreceptor. But still this signal is threatened to be swamped by noise from other photoreceptors. If the rod bipolar cell combined activity linearly from all its presynaptic rods, a single photon event would easily be lost in the accumulated noise. Yet, the bipolar cell produces clear depolarizing potentials in response to single-photon stimulation (Ashmore and Falk, 1976; Field and Rieke, 2002). It had therefore been proposed that the bipolar cell sums rod signals in a very nonlinear fashion (Baylor et al., 1984; van Rossum and Smith, 1998).

This strategy has indeed been confirmed by studies on mouse retina (Field and Rieke, 2002): The output of each rod photoreceptor is first thresholded before summation by the rod bipolar cell. Rod signals below the threshold level are simply discarded, and this affects some of the bona fide light responses as well: around 50% (Berntson et al., 2004) to 75% (Field and Rieke, 2002) of single-photon events do not pass the synapse. But this loss in signal is more than compensated by the reduction in noise. In fact, the observed threshold is positioned nearly optimally to maximize the signal-to-noise ratio in the rod bipolar cells (Field and Rieke, 2002). As discussed above, it is essential that the thresholding take place before the summation, to avoid the summing of many noise signals. Indeed, the mechanism that generates the threshold is found to be local to individual synapses between rods and bipolar cells. In darkness, the synapse is in a state of saturation so that some minimal level of presynaptic activity is required before a postsynaptic depolarization occurs (Sampath and Rieke, 2004). This aspect appears to be unique to the rod-bipolar pathway, which is specialized for detection at very low light levels. OFF bipolar cells, which are distinct from rod bipolars but also receive input from rod photoreceptors, respond to light stimuli in an approximately linear fashion (Field and Rieke, 2002).

Schematically, the proposed neural circuit that achieves the separation of dim light stimuli from noise thus consists of the following elements (Figure 2A): Signals from individual rods are first temporally filtered, then rectified by a threshold mechanism, and finally summed over many rods. This processing sequence appears to be a useful basic circuit design; we will encounter it repeatedly in the examples that follow. Here, it implements a computation that results in a selective neuronal response only if sufficient evidence is encountered that a photon event has occurred.

Figure 2. Computations performed by the retina and their underlying microcircuits.

A: Detection of dim light flashes in the rod-to-rod bipolar pathway. Left: Rod bipolar cells pool over many rod photoreceptors, which show distinct responses to single-photon activation embedded in noise. Bipolar cell potentials are not swamped by the accumulated noise in all rods, but instead show distinct activations from single photons, as shown by the voltage traces from a simple model simulation. Right: The important elements of the corresponding retinal microcircuitry. Each photoreceptor output is sent through a band-pass temporal filter followed by a thresholding operation before summation by the rod bipolar cell (Field and Rieke, 2002). Notation for this and all circuit diagrams: triangle = neuron; rectangle = temporal filter function; oval = instantaneous rectifier; closed/open circle = sign-preserving/inverting synapse.

B: Sensitivity to texture motion. Left: Y-type ganglion cells show activation when a fine grating shifts in either direction over the receptive field (circle), even though the average illumination remains constant. Right: The underlying microcircuit. Each shift of the grating excites some bipolar cells and inhibits others. The bipolar cells have biphasic dynamics (see impulse response in inset) and thus respond transiently. Only the depolarized bipolar cells communicate to the ganglion cell, because of rectification in synaptic transmission. Thus the ganglion cell fires transiently on every shift (Hochstein and Shapley, 1976).

C: Detection of differential motion. Left: An object-motion-sensitive ganglion cell remains silent under global motion of the entire image, but fires when the image patch in its receptive field moves differently from the background. Right: The circuitry behind this computation is based on similar elements as the Y-cell (panel B). Rectification of bipolar cell signals in the receptive field center creates sensitivity to motion. Polyaxonal amacrine cells in the periphery are excited by the same motion-sensitive circuit and send inhibitory inputs to the center. If motion in the periphery is synchronous with that in the center, the excitatory transients will coincide with the inhibitory ones, and firing is suppressed (Ölveczky et al., 2003; Baccus et al., 2008).

D: Detection of approaching motion. Left: A certain type of retinal ganglion cell responds strongly to the visual pattern of an approaching dark object, as indicated by the schematic spike train below, but only weakly to lateral object motion. Right: The circuit that generates this approach sensitivity is composed of excitation from OFF bipolar cells and inhibition from amacrine cells that are activated by ON bipolar cells, at least partly via gap junction coupling. Importantly, these inputs are organized in subfields whose signals are nonlinearly rectified before integration by the ganglion cell (Münch et al., 2009).

E: Rapid encoding of spatial structures with spike latencies. Left: Specific retinal ganglion cells encode the structure of a new image by their spike latencies. Cells with receptive fields (circles) in a dark region fire early, those in a bright region fire late. Cells whose receptive fields contain both dark and bright produce intermediate latencies and thus encode the boundary in their synchronous firing. Right: The responses result from a circuit that combines synaptic inputs from both ON and OFF bipolar cells whose signals are individually rectified. The timing differences in the responses follow from a delay (Δt) in the ON pathway (Gollisch and Meister, 2008a).

F: Switching circuit. Left: A control signal selectively gates one of two potential input signals. Right: In the retina, such a control signal is driven by certain wide-field amacrine cells (A₁), which are activated during rapid image shifts in the periphery. Their activation leads to a suppression of OFF bipolar signals and, through a putative local amacrine cell (A₂), to disinhibition of ON bipolar signals (Geffen et al., 2007).

Motion detection and discrimination

Beyond mere light detection, the visual system must interpret the many spatio-temporal patterns in photoreceptor activation on the retina. Among the myriad possible input patterns, only a minuscule minority is ultimately of behavioral interest. A dominant feature in the retinal input is image motion, which has two kinds of sources: The first results because movement of body, head, or eye of the observer induces global optic flow on the retina. This image motion largely represents “noise” for the purpose of visually guided behavior, and though the brain has dedicated the entire vestibulo-optic reflex pathway to reducing it, a significant global image jitter remains at all times. The second kind results from the motion of objects within the scene: For most intents, this is the behaviorally relevant “signal”. We will see that the retina contributes already to sorting this important signal from the morass of distracting noise, just as it did in the context of photon detection.

Texture motion

When a textured image patch moves across the retina, it causes an increase of light intensity at some points and a decrease at others. To be sensitive to this motion signal, a circuit would want to detect and integrate many such local changes. The so-called “Y-type” ganglion cells seem to signal the result of such a computation. These neurons fire when the texture moves, and the activity is largely independent of the direction of motion or the spatial layout of the moving pattern. Such ganglion cells have been identified in many species (Enroth-Cugell and Robson, 1966; Hochstein and Shapley, 1976; Caldwell and Daw, 1978; Kaplan and Shapley, 1982; Demb et al., 1999; Petrusca et al., 2007).

A paradoxical feature of the Y-cell is its sensitivity to high spatial frequencies, for example gratings that are much finer than the receptive field size. If the neuron simply pooled intensity signals from points throughout its receptive field, the positive and negative changes induced by grating motion should neatly cancel out and produce zero response. Instead, the Y-cell pools inputs from smaller subregions in the receptive field whose signals are individually rectified, which endows the Y-cell with its characteristic nonlinear response properties (Figure 2B). There is good evidence now that the subfields correspond to individual bipolar cells: These interneurons match the size of the subfields (Demb et al., 1999; Dacey et al., 2000), and their synaptic input to ganglion cells can indeed show strong rectification (Demb et al., 2001).

This circuit explains qualitatively how the Y-cell responds to moving textures regardless of the direction or the spatial pattern. Small features of the texture activate different subfields as they move around. The subfields have strongly transient responses, embodied in the biphasic shape of their impulse response function (Figure 2B). This makes the subfields sensitive to local changes, but not to static patterns. The nonlinear rectification then allows accumulation of signals from many activated subfields while preventing cancellation from other subfields that experience non-preferred stimulus changes. A time-varying velocity of the image pattern leads to a time-varying firing rate, and the simple Y-cell circuit model (Figure 2B) can predict this output quantitatively (Victor and Shapley, 1979; Enroth-Cugell and Freeman, 1987; Ölveczky et al., 2003). The transformation is dominated by the spatio-temporal receptive fields of bipolar cells (Baccus et al., 2008). While this circuit encodes the velocity signal reliably, it discards information about the spatial layout of the moving pattern. As a consequence, when different Y-cells experience the same motion trajectory, but with different spatial patterns at their receptive field locations, they can show the same activity profile and thereby signal a common origin of their activation. We will recognize this as an essential aspect in the retina’s scheme for segregating moving objects.

Object motion

To detect that an object moves within the observed scene, it is not sufficient, as one first might think, to measure the motion signal at the location of the object. The reason is that the visual system faces incessant motion signals that result from movements of the eye. Even when we try to fix our gaze on a static scene, minute drift and tremor in eye position provide a persistent source of image motion, whose trajectory is shared by all locations on the retina (Martinez-Conde et al., 2004). Object motion thus manifests itself on the retina in the difference between the motion trajectory of a local patch and that of the background. Neurons that detect this differential motion signal have been found in various parts of the visual system (Hammond and Smith, 1982; Frost and Nakayama, 1983; Born and Tootell, 1992), and they are particularly well studied in the retina. Here, object-motion-sensitive (OMS) ganglion cells were discovered that respond selectively to differential motion (Lettvin et al., 1959; Ölveczky et al., 2003). These neurons remain silent when the image moves across the retina rigidly, but fire vigorously when a local patch on the receptive field center moves with a trajectory different from the background (Figure 2C).

The circuits that implement the OMS computation have been probed in some detail (Ölveczky et al., 2003; Ölveczky et al., 2007; Baccus et al., 2008). In the receptive field center, the OMS cell pools over rectified bipolar signals (Figure 2C), as discussed for Y-cells. The resulting excitation is antagonized by inhibitory signals from similar Y-like motion detectors in a broad surrounding region. This inhibitory motion detector has been identified with a polyaxonal amacrine cell (Baccus et al., 2008), and it seems to act primarily via presynaptic inhibition of the bipolar cell terminals.

To detect differential motion, the circuit functions as follows: The motion in the center alone produces a sequence of excitatory inputs from bipolar cells to the ganglion cell. The time course of these inputs reflects the central motion trajectory. Analogously, the trajectory of the background motion results in a sequence of inhibitory signals in the polyaxonal amacrine cells. When the two trajectories are identical (global motion), the inhibition quenches the excitatory inputs at the bipolar cell terminals, and the ganglion cell remains silent. When the center motion differs from the background (differential motion), the excitatory signals do not coincide with inhibition and therefore reach the ganglion cell and cause it to spike, leading to an explicit representation of the local motion component.

It is interesting to note that the circuit distinguishes between global and differential motion by selectively suppressing a specific, yet very common visual signal: the coherent motion in the center and periphery of the receptive field. The only relevant stimulus feature for the comparison is the speed of image motion; by virtue of the Y-type circuitry (Figure 2B), both the direction of motion and the image pattern are ignored. This allows all parts of the background region to produce a coherent inhibitory signal regardless of their local pattern. Similarly, it allows the OMS cell to detect movement of an object regardless of its specific content (Ölveczky et al., 2003). In recent work, an OMS ganglion cell has been identified in the mouse retina that projects strongly to the superior colliculus, where it may direct orienting movements towards the site of object motion (Y Zhang and M. Meister, unpublished). Thus, these neurons likely serve an alarm role that triggers further inspection of the moving object with other visual mechanisms.

Approaching motion

Objects moving vertically or horizontally in the visual field lead to translation on the retina. But what about motion in the third dimension of depth? An approaching object would produce an image patch that gradually expands on the retina, with no net displacement. Recently, a ganglion cell type was described that is indeed selective for this stimulus feature (Münch et al., 2009). Identification of this cell type in the mouse retina was facilitated by genetic labeling with a fluorescent marker. These ganglion cells showed OFF-type responses. They were driven strongly by an expanding dark spot, even if it was accompanied by a global brightening of the scene. Yet they remained silent during lateral motion of a dark spot.

The circuit that achieves the approach-specific responses is based on excitatory inputs into the ganglion cell through the OFF pathway and inhibitory inputs through the ON pathway (Figure 2D). When a dark object approaches, the ganglion cell receives strong excitation and no inhibition and therefore responds vigorously. If the object moves laterally, on the other hand, excitation from its leading edge is balanced by inhibition from the trailing edge, and the ganglion cell therefore remains silent. Inhibition thus serves to suppress responses to the non-preferred motion signal, similar to the strategy of the OMS cell circuit. In contrast to the OMS cells, however, it is essential that the inhibition act postsynaptically rather than presynaptically at bipolar terminals, since signals from different parts of the object must be combined.

Again, nonlinear processing constitutes an important step in the circuit model: Excitation and inhibition are organized in small subfields whose signals must be rectified in order to account for the ganglion cell’s approach sensitivity even during global brightening. Interestingly, the study conjectures that the direct inhibitory pathway to the ganglion cell passes from ON cone bipolar cells through electrical junctions to the inhibitory AII amacrine cells. Presumably the speed of the electrical synapse ensures that this pathway keeps up with the excitatory pathway that has one less interneuron. Note that the AII amacrine also serves an entirely different function during scotopic vision, namely to feed rod signals into the cone bipolar cells (Bloomfield and Dacheux, 2001). This is an interesting example of a single cell type that serves quite different roles, even signaling in opposite directions, and it will be interesting to look for further examples of such functional promiscuity.

Detecting an approaching object is naturally of high behavioral relevance, and nervous systems have developed various mechanisms to deal with this challenge. Indeed, a different type of approach-sensitive response has been found in the frog retina. Here, retinal ganglion cells called dimming detectors engage in highly synchronized oscillations during global dimming or for large expanding dark patches (Ishikane et al., 1999). The oscillations are generated in a retinal circuit presynaptic to the ganglion cells, presumably through negative feedback involving amacrine cells, because they can be abolished by pharmacological blockage of GABA_A receptors (Arai et al., 2004). Little else is known about how the retina generates these responses, but it was possible to directly link them to a specific visually guided behavior: the frog’s escape from an approaching dark object. Frogs whose retinal oscillations are suppressed pharmacologically no longer perform this escape response (Ishikane et al., 2005).

Anticipation

There is great survival value in being able to anticipate the future. Indeed, because of delays in our sensory pathways, the brain really experiences the past, and even knowing the present requires a measure of prediction. We will see how the retina can contribute to this process.

Motion extrapolation

Detecting a moving object and its motion direction are generally not sufficient; to catch fleeing prey or to avoid an approaching predator an animal needs to track the object location precisely. This becomes a challenge because of the rather large response delays introduced by the process of phototransduction. Even in bright daylight conditions, the cone photoreceptor responds with a time to peak of several tens of milliseconds (Baylor et al., 1974; Schnapf et al., 1990). Thus, the neural signals in the photoreceptors already lag behind the actual object motion. During the phototransduction delay, a well-served tennis ball flies ~2 m, a distance many times larger than the receiving player’s racket. Clearly the visual system must somehow compensate for response delays introduced by the retina. It has been shown that the retina itself already contributes to this computation. It undoes the delay by relying on the fact that objects typically move in a smooth fashion, which allows an extrapolation of the motion trajectory.

When the image of an object moves on the retina, it creates a wave of neural activity among the ganglion cells. One should expect that this wave lags behind the object image because of the delay in phototransduction. Instead, experiments show that the activity in the ganglion cell layer moves at the true location of the object or even along its leading edge (Berry et al., 1999). Effectively, the retinal network computes the anticipated object location and thereby cancels the phototransduction delay. Surprisingly, this complex computation comes about through the interplay of rather generic features of retinal circuitry. The main contributors are the spatio-temporal receptive field of the ganglion cell and a dynamic gain-control mechanism. Because the receptive fields are extended in space, the object already activates ganglion cells that lie some distance ahead in its motion path. The spatial receptive field alone would predict an equally extended zone of activation behind the object. However, firing of those ganglion cells is suppressed first by the biphasic temporal receptive field, and second by a dynamic gain control mechanism, which itself gets activated by the response to the object (Shapley and Victor, 1978; Victor, 1987; Baccus and Meister, 2002). The gain control lets the ganglion cell be sensitive upon first entry of the object into the receptive field, but then shuts down the response. It is an essential component for the anticipatory response to motion stimuli (Berry et al., 1999) and gives this computation a highly nonlinear flavor.

By this algorithm, the retina performs a crude extrapolation of the object’s trajectory, such that its current location is represented in the population activity of the ganglion cells. Clearly the correction is only approximate: A perfect extrapolation would compensate for the photoreceptor delay irrespective of the speed of the object, whereas the retina’s method has a characteristic spatial and temporal scale, and thus functions only over some range of speeds. Similarly, the gain control depends on stimulus contrast, and thus extrapolation fails for low contrast objects (Berry et al., 1999). Of course, most animals’ visual performance, including that of expert tennis players, also degrades at high speeds and low contrasts.

In the salamander retina, the retina’s algorithm works well for objects with the size and speed of small insects that form the animal’s preferred prey. In this regime, the moving insect is represented at the retinal output by a blob of firing ganglion cells, whose center of mass is precisely aligned with the current location of the target (A. Leonardo and M. Meister, unpublished data). Thus downstream circuits can read out the target position by a simple “population vector” average (Georgopoulos et al., 1986), in which each ganglion cell’s firing rate is weighted by the vector representing its receptive field center (see also Figure 4). Of course this deceptively simple representation is the result of highly nonlinear operations that extrapolate the motion trajectory based on delayed data from the photoreceptors. The underlying computation reveals itself when the object makes a sharp turn: The neural image among the ganglion cells continues straight for a few tens of milliseconds, then turns and catches up with the new trajectory. When the object executes not a turn but a complete reversal of its trajectory, an additional response feature emerges: Many ganglion cells near the reversal point fire a brief synchronized burst of spikes (Schwartz et al., 2007b). This signal may be read out by downstream processing stages to identify an error in the retina’s prediction. What circuit mechanisms underlie this phenomenon is still unclear.

Figure 4. Linear readout of a neural computation.

A sensory neural network such as the retina responds to a stimulus, here the movement of an object in the visual field, by producing spike patterns in the output neurons. The network is postulated to solve a computational task, here compensating for the phototransduction delay and extrapolating a motion trajectory to the current object location. To claim that the network has solved this computation, we stipulate that the computational result should be obtainable as a linear combination of the output neurons’ activities. In the present case, this readout is provided by binning the spike trains in short observation windows and weighting each neuron’s spike count, n_i(t), by the position vector of its receptive field center, x⃗_i. Summation of these weighted responses over all neurons yields successive representations of the current object position, $\overrightarrow{x}(t)={\displaystyle \sum _i}{\overrightarrow{x}}_i·n_i(t)/{\displaystyle \sum _i}{n}_i(t)$.

Omitted stimulus response

A somewhat different form of anticipation can be observed when the visual system is exposed to a periodic stimulus, such as a regular series of flashes. The activated visual neurons typically become entrained into a periodic response. If the stimulus sequence is interrupted, for example by omitting just one of the flashes, some neurons generate a pulse of activity at the time corresponding to the missing stimulus (Bullock et al., 1990; Bullock et al., 1994). This phenomenon, termed the “omitted stimulus response”, is quite widespread, and has been noted in the brains of many species, including humans (McAnany and Alexander, 2009). Qualitatively it suggests the build-up of an anticipation for the next stimulus, and the large response reflects surprise at the missing element in the sequence. Unlike in the case of moving objects, where the anticipation involves lateral processing in space, here the information about the stimulus sequence must be propagated forward purely in time.

Recently, the omitted stimulus response has come under greater scrutiny in the retina, where the superior experimental access allows a focused search for the underlying neural mechanisms (Schwartz et al., 2007a; Schwartz and Berry, 2008). The response requires the convergence of excitatory signals from the ON and OFF pathways to the ganglion cells. Remarkably it does not require any inhibition mediated by amacrine cells (Schwartz and Berry, 2008; Werner et al., 2008). In one model, the periodic stimulus is extrapolated in time by a tunable oscillator: The flash sequence sets up a resonance within the retina that locks onto the stimulus frequency. When the sequence ends, the internal oscillation continues and causes the large omitted stimulus response (Schwartz and Berry, 2008). A competing model suggests that the combination of excitatory inputs from ON and OFF bipolar cells by itself is sufficient to explain many aspects of the omitted stimulus response (Werner et al., 2008). At the cellular level, these two models make quite different predictions, so one expects this question to be resolved soon.

Saccadic vision

In most animals, vision is an intermittent process: Short periods of fixation where the eye is relatively still are interleaved with sudden and rapid reorienting movements, called “saccades” (Land, 1999). Recent work has illuminated the role of the retina in these two dramatically different phases.

Saccadic suppression

During a saccade, the image sweeps across the retina violently for tens of milliseconds, precluding any useful visual processing. In humans, visual perception is largely suppressed during this period (Volkmann, 1986; Burr et al., 1994; Castet and Masson, 2000). The circuits of the retina are at least partly responsible for this suppression: Many types of retinal ganglion cell are strongly inhibited during sweeps of the visual image (Roska and Werblin, 2003). This effect is mediated by spiking, inhibitory amacrine cells, which are themselves excited by the global motion signal. Conceivably, the underlying circuitry resembles the one identified for OMS ganglion cells (Figure 2C). In fact, the OMS cells may be distinct simply by an enhanced sensitivity to the global inhibition, so they are suppressed even by the much smaller eye movements during a fixation.

Latency coding

Once the retinal image comes to rest, the visual system must analyze the new scene within the short period of fixation, a few tenths of a second in humans. Indeed, visual perception can be surprisingly fast. Early psychophysical studies showed that a 100-ms presentation of an image is sufficient to generate an understanding of the image content (Potter, 1976), and more recent experiments revealed that distinct brain signals dependent on image content arise already within 100 ms (Thorpe et al., 1996; Liu et al., 2002). How does the retina provide the relevant information to allow such rapid processing? One possibility is that some of this information is already transmitted with the very first spikes of retinal ganglion cells after the appearance of a new image (Thorpe et al., 2001).

Such a neural code based on first-spike timing has been identified now for a type of ganglion cell in the salamander retina (Gollisch and Meister, 2008a). When a new image appears on the retina, every cell in the population fires a brief burst of spikes, nearly regardless of the image content. However, the responses differ systematically in the timing of the burst: The first spike occurs earlier if the receptive field turns dark and later if it turns bright (Figure 2E). A combination of dark and bright regions in the receptive field produces intermediate spike times. Therefore, the relative latencies among the very first spikes in this ganglion cell population already provide a neural representation of the image. This representation turns out to be contrast-invariant; changing the overall contrast of the image affects the number of spikes in the bursts and their absolute timing, but it preserves the relative latencies of burst onsets.

Pharmacology and modeling studies (Gollisch and Meister, 2008a; Gollisch and Meister, 2008b) have helped elucidate the circuitry that leads to this spike timing code (Figure 2E). The key feature is that the ganglion cell receives excitation from multiple ON and OFF bipolar cells, which have small receptive fields. The signal from each of these subfields is rectified before integration by the ganglion cell. This explains why the ganglion cell fires whenever there is any image change within its receptive field. The latency code arises because the ON bipolar cells respond with slower kinetics than OFF bipolars. Thus a burst caused by brightening is delayed in time relative to one caused by dimming. As in previous examples, the rectification of the signal from each small subunit of the receptive field is a crucial step in this computation. It allows every location within the ganglion cell’s receptive field to contribute to the cell’s activation and prevents cancellation of the contributions from ON and OFF pathways.

A related study focused on retinal processing in the archerfish, an intriguing animal subject that excels in precise, visually guided spitting behavior (Segev et al., 2007). Ganglion cell responses were measured under stimuli that simulated what the fish observed while aiming at targets, including both saccades and fixational eye movements. Again, much of the relevant image information was available already in the burst of spikes immediately following the saccade, and much of it could be extracted by measuring the burst latency. It is unclear, however, how the retina in the archerfish produces these responses; this will require relating the applied stimuli to the cells’ receptive field characteristics.

Clearly it will be important to test whether these findings of latency coding in retinal ganglion cells generalize across species, and if so, whether it is accomplished by the same mechanism of ON/OFF convergence. Virtually all retinas contain ganglion cells with access to both ON and OFF bipolar signals in the inner plexiform layer: these include ganglion cell types with bistratified dendritic trees, those with broad arbors, and even some sharply monostratified cell types (Dacey et al., 2005; Dumitrescu et al., 2009). An interesting alternative circuit would combine excitation from one bipolar pathway with disinhibition via amacrine cells from the other pathway.

Switching circuits

In the preceding examples, we encountered a retinal circuit that gates the transmission of local bipolar cell signals based on motion in the periphery (Figure 2C); another circuit uses the convergence of parallel ON and OFF bipolar pathways (Figure 2E). Combining these features, one can envision a switching circuit: If the gating signal enhances one input pathway and suppresses the other, this would allow a single ganglion cell to switch between representing either of the two inputs (Figure 2F). Surprisingly, such a switching circuit has indeed been observed in the salamander retina.

In amphibian retinas, the OFF pathway tends to dominate. Even the ON-OFF ganglion cells that receive input from both ON and OFF bipolars are strongly biased towards OFF inputs. However, this imbalance can switch rather suddenly (Geffen et al., 2007). In particular, the switch can be triggered by a large peripheral image shift as would happen during a head saccade. For some hundred milliseconds after the shift, the ganglion cell transiently switches from transmitting OFF signals to transmitting ON signals. During this brief interval, excitation of the ganglion cell from the ON bipolar pathway is strengthened, whereas the OFF bipolar pathway is suppressed.

In searching for the underlying mechanisms, a wide-field ON amacrine cell emerged as an interesting suspect. These interneurons respond to the peripheral shift with a strong depolarization. Furthermore, electrical stimulation of such an amacrine cell caused a substantial shift in the ON-OFF balance of nearby ganglion cells. This suggests a circuit model for this switch where the wide-field amacrine cell gates transmission from OFF bipolar cells to the ganglion cell via presynaptic inhibition (Figure 2F). In addition, it inhibits a second type of amacrine cell that itself gates the ON bipolar input. The higher purpose of this switch for visual processing is still unclear. Perhaps the global image shift is not even the relevant trigger event; given that a single amacrine cell produces substantial gating (Geffen et al., 2007), the switching may be initiated by much more subtle image features.

From a broader perspective, such a gated switch is a powerful computational device, as it allows the dynamic routing of information through a circuit. It is impossible to imagine electronic computers without such switches, and they have been postulated to play equally essential roles for brain processing (Anderson and Van Essen, 1987; Olshausen et al., 1995). Thus it is promising to find at least one neural implementation and similar mechanisms may act elsewhere in the nervous system.

Adaptive computation

The visual computations in the retina are not bound to a static set of rules. Instead, their character changes dynamically along with the demands of the visual task. Since the retina receives little efferent input from the brain, these demands cannot be specified from the higher visual centers. Instead, the modulation of retinal function is largely determined by the recent history of the stimulus, which itself is a function of the visual environment as well as the animal’s actions.

Light adaptation is a prominent form of such modulation: Because the ambient light level varies over ~9 orders of magnitude in the course of a day, while spiking neurons have a dynamic range of only ~2 log units, the early visual system must adjust its sensitivity to the prevailing intensities. This adaptation to light level is accomplished by the retina, beginning already in the photoreceptors, and the process is complete before spiking neurons get involved. Over a wide range of intensities, the sensitivity of the retina declines inversely with the average light level. As a result, the ganglion cell signals are more or less independent of the illuminating intensity, but encode the reflectances of objects within the scene, which are the ethologically important variables. The perceptual effects of light adaptation and its basis in the circuitry and cellular mechanisms of the retina have been studied extensively and covered in several excellent reviews (Shapley and Enroth-Cugell, 1984; Hood, 1998; Fain et al., 2001; Rieke and Rudd, 2009).

Contrast and pattern adaptation

In recent years there has been considerable interest in other forms of adaptation. For example, the nature of retinal computations depends strongly on the contrast of the stimulus, namely the range between the low and high intensities in the scene. When the retina is brought from an environment of low contrast to high contrast, its sensitivity declines and its kinetics speed up (Shapley and Victor, 1978). One can roughly distinguish two components: In a rapid initial phase, lasting 0.1 s or less, the sensitivity decreases slightly, accompanied by a substantial change in kinetics; this has generally been called “contrast gain control” (Victor, 1987; Baccus and Meister, 2002). There follows a slow phase, lasting many seconds, during which the kinetics remain unaltered, but the sensitivity continues to decline; this has been called “contrast adaptation” (Smirnakis et al., 1997; Baccus and Meister, 2002; Manookin and Demb, 2006).

An even more intricate form of modulation is “pattern adaptation”. This occurs upon a switch between two environments that may have the same mean intensity and contrast, but differ in the frequency of certain spatio-temporal patterns. For example, in an environment of horizontal bars, the retina’s sensitivity to horizontal patterns declines, while the sensitivity to vertical patterns increases (Hosoya et al., 2005). Similarly, the response of an OMS ganglion cell (see above) undergoes a pronounced decline in sensitivity following the onset of differential motion (Ölveczky et al., 2007). These changes can be observed in individual retinal ganglion cells: The spatio-temporal receptive field becomes modified so as to suppress the dominant patterns in the environment. All these pattern adaptation effects have been of the slow kind, but the methods employed may have missed a substantial fast component.

Circuit mechanisms underlying the gain changes

To identify the neural mechanisms behind contrast adaptation, it helps to trace the emergence of gain changes in various neurons along the retinal pathways. Given the prominent feedback circuits within the retina, this is fraught with some risk, but it has yielded insights nonetheless. For example, following a switch from low to high contrast, the photoreceptor response undergoes no change in gain, quite unlike what happens in light adaptation. Neither is there any change in horizontal cells, suggesting that synaptic release from the photoreceptor is similarly unaffected (Rieke, 2001; Baccus and Meister, 2002; Beaudoin et al., 2007). In some bipolar cells, one begins to see a change in response properties, reflected by both accelerated kinetics and lower sensitivity (Rieke, 2001; Baccus and Meister, 2002). Similar effects are seen in some but not all amacrine cells (Baccus and Meister, 2002). However, the most pronounced effects of contrast gain control and adaptation are measured in the ganglion cells.

To explain the substantial increase in adaptation from bipolar to ganglion cells, three mechanisms have been explored: intrinsic membrane conductances in the ganglion cell; interaction between bipolar and amacrine signals; and adaptation in synaptic transmission from bipolar cells. The ganglion cell itself does indeed display some adaptation to the range of its input (Zaghloul et al., 2005), as confirmed by direct current injection (Kim and Rieke, 2001), but this makes only a partial contribution (Manookin and Demb, 2006). Amacrine cell signals are not required for contrast adaptation, as shown convincingly in experiments with transmitter blockers (Rieke, 2001; Brown and Masland, 2001; Beaudoin et al., 2007). On the other hand, transmission from bipolar cells makes a large contribution to the adaptation. Although transmitter release has not been measured directly, strong evidence comes from recordings of synaptic currents in ganglion cells (Manookin and Demb, 2006), and from experiments that stimulate alternatingly different sets of bipolars connected to the same ganglion cell (Ölveczky et al., 2007). All these studies indicate a slow process of contrast adaptation in bipolar cell synaptic transmission.

We will briefly expand on the process of bipolar cell transmission, because it may well hold broader significance. In a simple view of the events, a high contrast stimulus drives the bipolar cell more strongly, the synaptic terminal releases transmitter at a high rate, which over a few seconds leads to depletion of synaptic vesicles, and in turn decreases the gain of the synapse. Thus the postsynaptic response gradually declines during the period at high contrast. Following a switch to low contrast, transmitter release drops immediately, the vesicles are gradually replenished, and the gain of the synapse recovers. Indeed, continued activation of a bipolar cell terminal does lead to presynaptic depression from vesicle use (Burrone and Lagnado, 2000), and the results can be seen in postsynaptic amacrine cells in the form of paired pulse depression (Singer and Diamond, 2006; Li et al., 2007). Interestingly this depression recovers with a time constant of 5–10 s, similar to the time course of various contrast adaptation phenomena.

If this picture is correct, then each bipolar cell synaptic terminal undergoes contrast adaptation independently. This would be a powerful design. A ganglion cell typically collects excitation from 10 to 100 bipolar cell terminals. Each of these terminals can have different response properties. This is clear if the terminals belong to different bipolar cells. But even for terminals from the same bipolar cell, spatial receptive fields and response kinetics may differ as a result of lateral inhibition from amacrine cells that synapse directly on the terminal (Dowling and Boycott, 1966). Because of this diversity, any given stimulus pattern drives some of these bipolar cell terminals strongly and others weakly (Figure 3). The active terminals will adapt and decline in strength, whereas the silent ones recover. Thus the ganglion cell gradually becomes less sensitive to the prevailing stimulus pattern while it retains sensitivity to other patterns. In this way, the same cellular process of presynaptic depression may explain contrast adaptation as well as the various versions of pattern adaptation (Hosoya et al., 2005; Ölveczky et al., 2007). Note that this hypothesis for pattern adaptation really follows the traditional explanation invoked for such phenomena in the cortex: the fatigue of pattern-selective input units (Graham, 1989). Instead of neurons, here these input units are individual synaptic terminals.

Figure 3. Pattern adaptation based on synaptic depression at bipolar cell terminals.

A: Circuit model for pattern selectivity at bipolar terminals (top) with corresponding ganglion cell receptive fields (bottom). A ganglion cell receives input from multiple bipolar cells. Here bipolar cells B₁ and B₂ have the same receptive field, namely the center square of the lattice below. Their terminals, however, receive presynaptic inhibition from two amacrine cells A₁ and A₂ at different neighboring locations. This interaction leads to different transmitter release rates at the two terminals and makes the terminals preferentially respond to different visual patterns. The B₁ terminal, for example, receives co-occurring inhibition from A₁ for horizontal stripes and therefore responds more strongly to vertical stripes, while the B₂ terminal responds more strongly to horizontal stripes.

B: During stimulation with flickering vertical stripes, bipolar cell B₁ and amacrine cell A₁ are activated out of phase, so there is no presynaptic suppression of release. The synaptic terminal of B₁ is therefore highly active (bold arrow), and over time, synaptic depression reduces the strength of this connection (small circle). Cells B₂ and A₂, on the other hand, are activated in phase. Thus release at the terminal of B₂ is suppressed (thin arrow), vesicles can be replenished, and the connection recovers its strength (large circle). If one subsequently tests the ganglion cell’s receptive field using a mixture of vertical and horizontal stripes (Hosoya et al., 2005), the sensitivity to horizontal stripes, mediated by the strong B₂ terminal, will exceed that to vertical stripes.

C: During stimulation with horizontal stripes, the reverse changes take place. After synaptic depression has taken its course, the ganglion cell becomes more sensitive to vertical stripes.

Functional purpose

What is the role of retinal contrast adaptation for vision? Prolonged viewing of high-contrast patterns leads to clear changes in perception: both a general decline in sensitivity and a specific decline for the adapting pattern (Blakemore and Campbell, 1969; Graham, 1989). The effects arise and decay on the time scale of several to tens of seconds, as for the slow process in the retina. Similar adaptation phenomena have long been reported for neural responses in the visual cortex (Maffei et al., 1973; Movshon and Lennie, 1979). Initially they were attributed to processing within the cortex, but a substantial reappraisal of the evidence has taken place, and certain forms of contrast adaptation observed in cortex are now thought to be inherited from the retina (Baccus and Meister, 2004). One strong indicator was that most of cortical contrast adaptation arises in a purely monocular pathway (Truchard et al., 2000). Definitive evidence resulted from direct comparison of retinal and cortical signals in the same experiment (Solomon et al., 2004).

These developments suggest that contrast adaptation in the retina has a substantial effect on higher visual areas and even human perception. But what may be the ethological purpose of these adaptive dynamics? Here again it is important to distinguish the fast process (contrast gain control) from the slow one (contrast adaptation). As discussed above, human vision is characterized by brief fixations, when the gaze is held still, interrupted by eye saccades. The fast gain control adjusts within tens of milliseconds, considerably less than the length of a fixation (~0.3 s). Therefore, after every saccade of the eye, the retina will operate with somewhat different gain and kinetics depending on the local properties of the scene. Furthermore, this gain control is too fast to accumulate any kind of statistical measure of contrast in the scene, like the time-averaged standard deviation (Bonin et al., 2006). Slow contrast adaptation, on the other hand, occurs over many seconds, and thus may adjust retinal function more properly to the statistics of the environment, averaged over many successive fixations.

Discussion

As we have seen, the retina does not merely convey a pre-filtered pixel image to the brain. Instead, it engages in substantial computations of specific image features, with different ganglion cell types taking on different tasks. Many of these computations can be understood as answers to particular challenges shared by many animals: the need to detect dim lights; the need to detect moving objects and locate them correctly; the struggles with a constantly moving image sensor; and the need to predict the future and adapt to changing conditions.

The case for retinal computation

This view represents a significant departure from the conventional wisdom, and will therefore meet with some skepticism. Here we discuss some of the concerns by answering questions from an imaginary critic, in order of increasing difficulty.

1. All these retinal nonlinearities seem to impose a serious complication for visual coding. How can the brain ever hope to decode the ganglion cell spikes and reconstruct the image on the retina?

The animal as a whole is interested in survival and reproduction, not a veridical reconstruction of what was on the retina. For these goals, its visual system must distill the massive flood of raw visual information at the photoreceptors to just one bit (fight or flight) or maybe a few bits per second (air traffic controller). The retina simply begins this process of data reduction, and sends on just those visual features that are useful for subsequent operations. This process is lossy, and we know well that the brain never recovers the original visual image. The discarding of image information in the retina fundamentally affects our spatial, temporal, and chromatic acuity and underlies many popular optical illusions. In this view, the power of retinal computation should be judged by how much raw information it filters out for discard, not by the amount it preserves.

2. Many of the examples quoted here are from “lower” vertebrates, meaning non-primates. Isn’t it well established that the smarter animals do much less processing with their retina?

This is a persistent myth that deserves to be laid to rest. The anatomical structure of the primate retina is equally as intricate as that of other mammals. Several new ganglion cell types have been discovered recently, now bringing the total to about 17 (Dacey, 2004; Field and Chichilnisky, 2007). Their functions remain mysterious, because virtually all the physiological experiments concern just two classes of ganglion cells, the P and M cells. Although the P cells are by far the most numerous ganglion cell type, this is a simple consequence of their small receptive fields and the need to cover the retina. The newly discovered cell types collectively provide more capacity than the entire cat retina (Field and Chichilnisky, 2007), and thus can offer many additional channels of visual information that may not require a fine-grained representation. Indeed, several of the novel aspects of retinal function have been confirmed in the primate retina (Chander and Chichilnisky, 2001; Uzzell and Chichilnisky, 2004; Pillow et al., 2005; Petrusca et al., 2007) and thus are intimately relevant for an understanding of human vision. That said, the primate fovea is a rather special case. In this small region of the retina, covering 1 degree of visual angle, there is little or no convergence from photoreceptors to bipolars or from bipolars to ganglion cells. Consequently, the circuits of Figures 2 and 3 cannot apply. For a specific understanding of high-acuity human vision, it will indeed be very helpful to learn more about foveal ganglion cells and interneurons.

3. The idea that the retina extracts specific visual features is a blast from the past, to be found in the first anecdotal reports of “bug detectors”. Did we not banish this fuzzy thinking in the 1960s by performing careful parametric measurements of center-surround receptive fields?

Indeed some early studies of retinal function were guided by much more ethological sensitivity and tried to relate ganglion cell responses to specific tasks (Lettvin et al., 1959; Levick, 1967). The next generation overturned this anecdotal thinking and applied the rigorous new engineering tools from systems analysis (Rowe and Stone, 1980). Instead of showing the retina some arbitrary photographs of flies, these workers sampled the stimulus space systematically, feeding the retina with sine waves and white noise to measure its transfer function. Sadly, all the resulting ganglion cell receptive fields looked like Mexican hats with a biphasic time course, and the only possible conclusion was that the interesting visual computations are performed later (Stone, 1983). Now we understand that the problem was not with the ganglion cells, but with the stimuli. For example, the OMS ganglion cells discussed above have perfectly bland-looking center-surround receptive fields when studied with white-noise flicker. Of course the particular condition that reveals their function – differential motion of an image patch and its background – never occurs during white-noise flicker, whereas it represents a common occurrence on the retina in real life.

We think it is safe now to return to a more ethological consideration of the retina for several reasons: First the impoverished traditional view of the retina’s function clashes with its rich synaptic structure. By anatomical and molecular criteria, the mammalian retina has ~15 different ganglion cell types, whereas the parametric functional studies typically reveal around 5, and even those can have similar-looking receptive fields (Stone, 1983; Carcieri et al., 2003; Segev et al., 2006). The anatomical diversity suggests that there is much function left to be discovered and that we probably still have a good distance to go before understanding all the computations performed by the retina. Second we have now far greater technical control over both the light input and the spike output from the retina, allowing a systematic exploration even of complex stimulus conditions like fixational image motion. Third, retinal processing is being studied in at least half a dozen different species, and a comparison among them can distinguish general processing principles from species-specific quirks. Finally, with improved technical access to the interneurons of the circuit, we can now build and test plausible circuit mechanisms to explain the new visual computations (Figure 2).

In many of the studies reviewed here, the ethological perspective served to single out specific tasks that were then studied with specifically designed, artificial stimuli that distilled the task-relevant features of the visual scene. This reductionist approach has proven quite successful for a systematic and quantitative investigation of computational principles. Yet, eventually we need to go a step further and understand retinal function within the rich complexity of natural stimuli. This endeavor will likely provide additional challenges because multiple processes occur simultaneously, for example, when the retina adapts to the large range of intensities and contrasts in a natural scene while at the same time facing different combinations of motion signals. Initial studies of responses to natural stimulus components have already demonstrated the rich dynamics that may arise (van Hateren et al., 2002).

4. What is special about the term “computation”, and why doesn’t it apply for the task of linear filtering that we used to assign to the retina?

This is a more philosophical question, but we suggest that a “computation” should yield an explicit result to a specific question. For example, the question “is there a moving object near this location?” is answered in a simple yes/no manner by the firing of an OMS ganglion cell. No difficult high-order decoding is necessary to extract the desired result, which is an obvious benefit for speedy downstream reactions. The question “where exactly is the moving target right now?” is answered collectively by a population of motion-extrapolating ganglion cells (Figure 4). Now in this example, the coordinates of the target are not provided explicitly in the responses of single neurons but distributed across a population. This requires broadening the concept of “explicit”, and we suggest a criterion that has been used effectively in the past (Marder and Abbott, 1995; Eliasmith and Anderson, 2002; Shamir and Sompolinsky, 2006): In a distributed representation, the answer of interest should be obtainable through linear decoding, namely by a simple weighted summation over the single-neuron activities (Figure 4). Such a decoding could be achieved in a single step by a downstream neuron that samples from the population. By this convention, a retina that merely filters the image on the photoreceptors performs no computation at all, since the desired result can be read via weighted summation directly from the photoreceptors themselves.

Circuit models

As one investigates the computations of a new type of ganglion cell, how should the relationship between stimuli and responses be formulated? Given the vast range of possible stimulus movies one might show the retina, it is impossible to simply deduce the computation from an input-output table of stimuli and responses. One needs to develop some quantitative hypotheses for that relationship and test them in a directed manner. What is a useful format for these hypotheses?

One traditional approach has been to write the response as a Wiener expansion of the stimulus (Sakai and Naka, 1992). This is a highly principled method, with a solid mathematical basis which guarantees that every stimulus-response relationship can be expressed that way. It is particularly effective if the system responds approximately linearly. Unfortunately, the reality is that retinal responses are highly nonlinear under most conditions of practical interest. Estimating all the required Wiener kernels from experiments becomes unrealistic, and this approach has largely fallen out of fashion.

We have had good experiences with a very different formalism based on schematic circuit models (Figure 2). The stimulus is processed by a network of individually simple elements. These components and their interactions are fashioned after retinal neuron types, without simulating the actual biophysics in detail. Each element performs either linear filtering or an instantaneous nonlinear transformation. The list of elements includes: ON and OFF “bipolar cells”, modeled as linear spatio-temporal filters; synaptic transmission, modeled as an instantaneous nonlinearity; pooling over many synaptic inputs, modeled as weighted summation; inhibition, modeled as instantaneous combination of two opposing signals. Others can be added as needed.

This style of modeling offers several benefits. First, it is intrinsically realistic: Any stimulus-response function written this way can be implemented by neural machinery, specifically the interneurons of the retina. Second, the formalism is nonetheless sufficiently broad. Theorems guarantee that one can write an arbitrary spatial transform of the input image this way (Funahashi, 1989; Hornik et al., 1989), though it would be interesting to extend this to spatio-temporal functions. Third, despite their simple graphical nature, these models are nonetheless rigorous. Any such circuit graphic can be translated into a mathematical relationship between input and output of the retina, with parameters that are embodied in the circuit elements. The circuit functions are easily simulated and make quantitative predictions that match actual ganglion cell firing remarkably well (Ölveczky et al., 2003; Baccus et al., 2008; Gollisch and Meister, 2008a). Finally, such a formula for the stimulus-response relationship is at the same time an explicit hypothesis for how the computation is done. It makes predictions about what happens at various stages inside the retina, and these can be tested independently. For example, the spatio-temporal receptive field of a bipolar cell can be measured directly, and that immediately reduces the free parameters of the model (Baccus et al., 2008). In this way, the circuit model for a ganglion cell response serves as the linking hypothesis between a systems level computation and the cellular level of components.

The future: labeled neuron types

Reverse-engineering the connectivity and function in a neural network made of 50 different component types is a daunting challenge. The task would be more plausible if each of the neuron types had a part number stamped on it, much as one finds for components in a radio. Actually, methods to tag specific cell types are now within reach. There has been great interest in the genetic mechanisms by which neuronal cell types are specified during development. As a benefit of this research, a number of cell-specific markers have been identified, and there already exist multiple lines of mice in which specific retinal cell types are labeled fluorescently (Trimarchi et al., 2007; Kim et al., 2008; Siegert et al., 2009; Wässle et al., 2009; Münch et al., 2009). Such type-specific labels are a great boon in studies of circuit function for several practical reasons:

First, a fluorescent marker allows one to focus physiological experiments on that specific type, since electrodes are easily aimed at the fluorescent cells. In this way, even a rare cell type can be studied in a dedicated fashion (Huberman et al., 2008; Kim et al., 2008), and one may hope to systematically address the question whether specific cell types participate in multiple computations. Second, the marking method can immediately reveal the shape of the cell, and thus its likely connectivity within the retina and to projection areas in the brain in the case of ganglion cells. Third, the genetic tag can also be recruited to modify the cell type, so it may be activated or inactivated at will (Lagali et al., 2008). Finally, a genetic marker enables more effective scientific communication: Different laboratories can work on the same cell type simply by using the same mouse line. Previously, the communication of cell types was often hampered by ambiguities in the morphological type definitions, or simply because detailed morphology was not available in physiological studies. One can envision a future in which every retinal cell type comes with a genetic handle, by which it can be visually marked or its function manipulated. Such a state of affairs would greatly enhance our ability to dissect neuronal circuitry, in the retina as elsewhere in the brain.