Throwing a glance at the neural code: rapid information transmission in the visual system

Abstract

Our visual system can operate at fascinating speeds. Psychophysical experiments teach us that the processing of complex natural images and visual object recognition require a mere split second. Even in everyday life, our gaze seldom rests for long on any particular spot of the visual scene before a sudden movement of the eyes or the head shifts it to a new location. These observations challenge our understanding of how neurons in the visual system of the brain represent, process, and transmit the relevant visual information quickly enough. This article argues that the speed of visual processing provides an adjuvant framework for studying the neural code in the visual system. In the retina, which constitutes the first stage of visual processing, recent experiments have highlighted response features that allow for particularly rapid information transmission. This sets the stage for discussing some of the fundamental questions in the research of neural coding. How do downstream brain regions read out signals from the retina and combine them with intrinsic signals that accompany eye movements? And, how do the neural response features ultimately affect perception and behavior?

Picture yourself running late for an appointment. You are all set to go, dressed up, stressed out, knowing that you are already being expected elsewhere. But, alas, your car keys are missing. You know that you placed them somewhere on a surface when you came home a few hours ago. But where, you do not remember. So you search for them, shooting quick glances across the room. Fixating on the coffee table for a split second, you convince yourself that there is no key. A moment later, the window sill is checked, then the kitchen table, the sofa, the bookshelf. Suddenly, you catch a glimpse of the keys, half covered under a newspaper, on a chair. You grab them and storm out of the door without stopping to marvel at the sheer wonders that your visual system performed to guide you towards your object of desire.

Visual perception can function at amazing speeds, and it takes little imagination to appreciate the evolutionary benefit of this swiftness, for example, when we transfer the search for the car keys to one of our ancestors searching for the source of some eerie sound in a primordial forest some million years ago. The speed of the visual system is no small feat, considering the complex signal processing that needs to take place in our nervous system to guide us from the activation of photoreceptors by incoming light to the initiation of an appropriate behavioral response. Several stages in the visual system and billions of nerve cells are involved in the processes that let us arrive at the perception of a meaningful scene, consisting of discrete objects embedded in three-dimensional space with consistent attributes of brightness, contrast, color, and size, nearly regardless of the actual illumination conditions.

Here, we explore how the need for speed in visual processing can guide researchers in studying the features and mechanisms of the visual system. The rapidity of visual processing—manifest in the dynamic nature of vision in everyday life and assessed in psychophysical experiments—provides an expedient framework for connecting behavioral and perceptual features to the physiological characteristics of their biological substrate, the neural circuits in the nervous system. Investigating how the neurons represent, process, and transmit information on short time scales is leading to new insights about the neural code in the visual system. In particular, we will focus here on the first stage of the visual system—the retina—which constitutes a traditional model system regarding questions of neural coding.

NATURAL DYNAMICS AND PSYCHOPHYSICS OF VISION

Investigating how our visual system operates has a long research tradition in various scientific disciplines, ranging from optics, psychology, and neurobiology to engineering and theoretical physics. The interactions between these different approaches allow for new perspectives and inspirations in this field of research. For example, neurobiological research has started to take the statistics of the natural stimulus environment for a given sensory system into account when analyzing how the system operates (Reinagel, 2001; Simoncelli and Olshausen, 2001). For the visual system, besides the particular structures of natural images, this also concerns the dynamical nature of vision. Brief phases of fixation of a particular spot in the visual scene are interspersed with saccades, which are rapid and sudden changes in the direction of gaze, resulting from movements of the eyes, the head, the body, or a combination of these. Saccades help us bring objects of interest into the center of our field of view where our ability to resolve fine details of the scene is highest. Stable fixations, on the other hand, let us acquire image details free of motion blur. But not only humans perform saccades; virtually every animal with a decent sense of vision, from insects to primates, exhibit these sudden shifts in the direction of gaze (Land, 1999). Since the seminal works of Yarbus who described how an observer scans visual objects with saccades (Yarbus, 1967), researchers have characterized many aspects of how the saccadic structure of vision relates the images that enter the eye to visual perception and performance. These investigations range from studies of visual searching (Najemnik and Geisler, 2005) and playing cricket (Land and McLeod, 2000) to everyday tasks (Land et al., 1999), such as preparing a cup of tea [Figs. 1A, 1B].

Figure 1. The speed of vision in everyday life and in psychophysics experiments.

(A) Trajectory of the direction of gaze measured with an eye-tracking device while the subject is engaged in the task of preparing a cup of tea. Saccades are marked by straight lines, fixations by dots. (B) Time course of gaze, eye, and body position during the activity shown in (A). The combination of eye and body movements results in a series of gaze fixations, which often last only few 100 milliseconds. (C) Presentation scheme for a psychophysical investigation of the speed of vision. Each trial begins with subjects looking at a fixation spot. Shortly after the spot disappears, two photographs are shown briefly on the left and right of the spot location. Subsequently, two fixation spots appear, and the subject’s task is to quickly make a saccade towards the side where an animal had been shown in the photograph. (D) Saccade reaction times (SRT) for the task shown in (C). The dark purple trace shows a histogram of reaction times. For comparison, the dark green trace shows reaction times of a similar task where only a single image was shown so that no image processing was necessary to solve the task. The light traces near the bottom of the graph show corresponding error trials. Already with reaction times of little longer than 100 milliseconds, performance is much better than chance level. Furthermore, the task requires only some tens of milliseconds more than the comparative task without image processing. Panels (A) and (B) reprinted with kind permission from Springer Science+Business Media: Journal of Comparative Physiology [A], “Motion and vision: why animals move their eyes,” Vol. 185, pp. 341-352, M.F. Land, Fig. 1. Copyright 1999, Springer-Verlag. Panels (C) and (D) reprinted from Vision Research, “Ultra-rapid object detection with saccadic eye movements: visual processing speed revisited,” Vol. 46, pp. 1762-1776, H. Kirchner and S.J. Thorpe. Copyright 2005, with permission from Elsevier.

On average, we typically make around three to four saccades per second, which leaves little time for the processing of the intermediately fixated images. Indeed, the visual system can handle a series of short image presentations, with each image available for only around 100 milliseconds. This was already demonstrated several decades ago by showing rapid sequences of photographs and querying subjects about the image contents (Potter and Levy, 1969; Potter, 1976). More recently, a binary decision paradigm was applied to test the speed of visual processing at its limits (Thorpe et al., 1996). In this experiment, subjects see two suddenly appearing photographs of different categories, for example, one containing an animal and the other showing a landscape without an animal. The experimenter’s instruction is to identify—for example by pressing a corresponding button—the image that shows the animal, and subjects accurately solve this task within a fraction of a second. In the latest version of this experiment (Kirchner and Thorpe, 2006), subjects signal the detection by a quick movement of their eyes towards the identified image, which brings reaction times down to an amazingly low 120 milliseconds [see Figs. 1C, 1D].

Experiments of this type can also be accompanied by EEG measurements of event-related potentials in the brain. These signals also show differences within the first 150 milliseconds between photographs with or without animals (Thorpe et al., 1996) or between presentations of different emotional facial expressions (Eimer and Holmes, 2007). It is still unclear and a matter of debate what level of processing is represented by these early differences in brain signals (Johnson and Olshausen, 2005). Nonetheless, the short reaction and EEG signal times leave us with a puzzling problem: How is the required visual information represented and transmitted quickly enough through the relevant stages of the visual system? The psychophysical results, together with the short fixation times in saccadic vision, suggest that the visual system is laid out for rapid information processing. This should be reflected in the neural code by which neurons in the visual system represent and transmit information, and it paves the way for new perspectives on studying this code.

THE NEURAL CODE

Neurons process information based on electrical potentials between the cell interior and the exterior space. These potentials can trigger the release of neurotransmitters at synapses that connect neurons and thereby influence the electric potentials in subsequent neurons. Typically, the release of neurotransmitters follows after a stereotypic, sharp depolarization event—an action potential or “spike.” A spike is triggered inside the neuron by voltage-gated ion channels in the cell membrane when the membrane potential has reached a high-enough level. From the stereotypic size and time course of the spike, it follows that the information that is transmitted by the neuron must be contained in the temporal sequence of these spikes, the “spike train.”

The relation between the spike trains of a neuron (or of groups of neurons) and the transmitted information forms the “neural code.” Deciphering this neural code is an important step towards unraveling how nervous systems operate; understanding the language in which neurons communicate with each other will help us understand how they perform computations, how they store and retrieve information, and how their malfunction relates to pathological states. Only if we know the meaning of neuronal spike patterns with respect to an organism’s perception or action can we comprehend what the functions of cellular and synaptic properties are that affect and regulate the spiking activity. The nature of the neural code is therefore among the most fundamental questions of neuroscience.

A range of different paradigms regarding the neural code has been developed over the years (Fig. 2). The oldest and most prominent representative is the so-called “rate code” (Adrian, 1926; Shadlen and Newsome, 1994), which, in its most fundamental formulation, states that single neurons transmit information through the number of spikes produced over an extended temporal period. Often, this encoding type is distinguished from a “temporal code” (Theunissen and Miller, 1995; deCharms and Zador, 2000; Lestienne, 2001; Oram et al., 2002) where the precisely timed patterns of spikes are relevant for information transmission. A particular example for a temporal code is the “latency code,” where the arrival time of the first spike after the onset of a stimulus contains the stimulus information (Thorpe, 1990; Gawne et al., 1996; Johansson and Birznieks, 2004; Gollisch and Meister, 2008). As a reference time for a spike in a temporal code, one typically considers spikes from other neurons within a neuronal ensemble (“relational code”). A specific example for this is the “synchrony code,” according to which stimulus-dependent groups of neurons temporally synchronize their activity (Engel et al., 1992; Abeles et al., 1994).

Figure 2. Schematic depictions of different neural coding schemes.

Hypothetical spike patterns are shown in response to two stimuli A and B. Each mark denotes the time of occurrence of a spike. For the rate code, the temporal code, and the latency code, responses to three repeated presentations of the two stimuli are shown; for the synchrony code, responses from three neurons to a single stimulus presentation are displayed. In a typical rate code, the number of spikes differentiates between the stimuli, but the timing of individual spikes is quite variable. In a timing code, precisely timed and repeatable patterns of spikes mark the stimulus. A latency code is characterized by a precise, stimulus-dependent onset of the activity. In a synchrony code, the stimulus causes different subgroups of neurons to synchronize their activity. Note that variations and mixtures of these simplified coding schemes are to be expected.

At different stages of neuronal processing, different variations and mixtures of these coding schemes are likely to be relevant. Moreover, how much each of these response features contributes to neuronal information transmission will likely depend on the structure and dynamics of the sensory signals. For the visual system, saccades strongly shape the stimulus. Investigating neural coding of brief, saccadelike stimulation is thus an essential ingredient for understanding visual processing in a natural stimulus context. So far, the most intensely studied part of the visual system in this respect is arguably the retina, on which we will focus the following discussion.

THE RETINA

The retina is a neural network at the back of the eyeball and constitutes the first stage of visual processing. It has long been established as one of the most popular model systems for studying neural coding. One primary reason for this is that the input and output of the system are well-defined and can be exquisitely controlled and monitored in experiments. The input is given by the light signals that fall onto the retina and are transduced by photoreceptors into electrical signals. The retina’s output is defined by the patterns of spikes produced by retinal ganglion cells, whose axons form the fibers of the optic nerve. These spike patterns encode all visual information available to the rest of the brain. Between photoreceptors and ganglion cells, a complex network of various neuronal types processes the visual information (Fig. 3) and thus produces the neural code of the retina, which sets the stage for all further visual processing. The neural responses can be recorded by fine electrodes that are brought near or in contact with individual cells. In particular, the use of planar multielectrode arrays, onto which isolated retinas can be placed, has allowed the recording of spikes from many individual ganglion cells simultaneously while stimulating the retina by projecting light patterns onto the photoreceptors (Meister et al., 1994; Segev et al., 2004; Petrusca et al., 2007). The multielectrode arrays not only increase the yield for recording activity of individual neurons, but also provide the means to study concerted neuronal activity by pairs and groups of neurons (Meister et al., 1995; Schneidman et al., 2006; Shlens et al., 2006; Pillow et al., 2008).

Figure 3. Schematic drawing of the retina network.

Photoreceptors take up light stimuli and transduce them into electrical signals. Bipolar cells constitute a feedforward pathway from the photoreceptors to ganglion cells, which form the output layer of the retina. Horizontal and amacrine cells provide a wealth of additional processing capacities, including lateral inhibition, feedback, and long-range connections. Finally, the visual information is encoded into spike patterns of ganglion cells and transmitted along their axons, which form the optic nerve, to different regions of the brain. Note that the drawing simplifies the actual circuitry, which includes various subtypes of each of the depicted neuron types with specific connection patterns (Masland, 2001; Wässle, 2004). Also, the numerous electrical couplings within the network are left out for clarity.

The precise nature of the neural code in the retina is the subject of extensive ongoing research. Several hallmarks of retinal neural responses, however, have been well characterized. One of the most striking features of spike trains recorded from retinal ganglion cells is the high degree of reliability and temporal precision. This is shown in Fig. 4 for a ganglion cell that was recorded under repeated stimulation with the same sequence of spatiotemporal flicker. The responses are characterized by distinct spiking events, typically brief bursts of spikes at short intervals. Over stimulus repeats, the spike bursts are quite reliable in the number of spikes and display temporal precision in their onsets down to a few milliseconds (Berry et al., 1997; Uzzell and Chichilnisky, 2004). A second striking feature of retinal responses is that, when several ganglion cells are recorded simultaneously, one often finds synchronized spiking on various time scales and both in the presence and absence of visual stimulation (Meister et al., 1995; DeVries, 1999). These characteristics illustrate the wide range of possibilities by which retinal ganglion cell spiking could carry visual information: by rate, precise timing, relation to spiking of other cells, or any combination of these. With these general features in mind, let us return to the issues of short fixation periods during saccadic vision and high processing speed in the visual system. One requirement for the retinal neural code here should be that it transmits at least part of the visual information in a rapidly accessible way.

Figure 4. Precision and reliability in retinal ganglion cells.

In response to repeated presentation of a sequence of flickering stripes (left), spikes were recorded from a single retinal ganglion cell from a larval tiger salamander. The raster plot (top) displays individual responses from a subset of the response presentations, the histogram below shows the resulting firing rate obtained from all presentations. The responses are marked by sparse and precisely timed spiking events, underscoring the possibility of temporal neural codes in the retinal signals.

The responses of retinal ganglion cells to saccadelike stimulation have been the subject of a number of recent studies. First, there is the question of how ganglion cells respond to the global motion signals during the saccade itself. By playing natural movies that contain saccadelike sudden shifts of the full image, Roska and Werblin showed that specific types of ganglion cells in the rabbit retina are strongly inhibited during the saccade, but typically respond with bursts of spikes after the saccade is over (Roska and Werblin, 2003). With these bursts, the cells may likely encode the newly encountered image. Other ganglion cell types, on the other hand, were found to respond with increased activity during the saccade, potentially signaling the features of these motion signals themselves. The diversity of responses by different types of ganglion cells was also emphasized by a study that characterized responses in the rabbit retina to sudden changes in mean luminance as can occur as a result of saccades (Amthor et al., 2005). The induced changes in activity by the luminance steps ranged from strong suppression to strong activation, underscoring the complexity of visual information transmission in the presence of such dynamic stimuli.

The effect of saccades, however, goes beyond mere modulations of the level of activity; they can alter fundamental response features of single neurons. This was revealed by a study that showed how saccadelike stimulation can change the classical distinction of neurons into ON and OFF types. Typically, these types denote classes of neurons that primarily respond to onsets and offsets in light intensity, respectively. A study by Geffen et al., however, found certain retinal ganglion cells in the salamander retina that behaved like OFF-type cells without saccadelike stimulation and like ON-type cells right after a simulated saccade (Geffen et al., 2007). Like most neurons in the early visual system, these cells respond primarily to light stimuli within a small spatial region, the neuron’s “receptive field.” The study monitored the response characteristics of these neurons to flickering light within their receptive fields while applying sudden shifts of a striped pattern in the spatial periphery. These stimulus shifts mimicked the global motion signals induced by saccades. In the absence of a saccadelike shift, the investigated neurons responded on average shortly after brief decreases in light intensity, as one would expect from OFF cells. But, for a short period immediately following the shift, these cells switched to responding when the light intensity in the center had shown a brief increase. Consequently, the message conveyed by a spike from such a neuron seems to be different at different times after a saccade. Elucidating the functional significance of this switch in the neural code is among the pending challenges for understanding the operation of the visual system in the presence of saccades.

Does the retina make use of its potential to produce temporally precise spikes for transmitting visual information after a saccade? For a particular subtype of ON-OFF ganglion cells in the turtle retina, this was investigated by stimulation with light intensity steps (Greschner et al., 2006; Thiel et al., 2006). The investigated cells reacted with two precise bursts of spikes whose timing depended on the applied stimulus. Whereas the first burst had a typical dependence on the light intensity step—it came with shorter latency for larger steps—the timing of the second burst, which followed some tens of milliseconds later, showed a surprising nonmonotonic dependence on the step size; minimal latency of the second spiking event was encountered for an intermediate light-intensity step. Moreover, the timing of both spiking events indeed contained substantial stimulus information. This followed from the fact that discrimination of the applied step sizes based on the ganglion cell spikes worked best when the spike counts as well as the latencies of both spiking events were taken into account (Greschner et al., 2006).

Besides a change in light intensity, saccades typically also lead to substantial changes in the spatial pattern of the fixated image. It seems likely that visual systems have evolved to quickly transmit information about the details of the new spatial structure encountered after a saccade. That saccadic stimulation can play an essential role in spatial scene analysis has been demonstrated in a recent study of visual behavior and retinal coding in the archer fish (Segev et al., 2007). These fish can accurately squirt water at insects above the water surface and thereby “shoot them down.” During the localization of the target, the fish’s direction of gaze is shifted by saccades. In recordings from ganglion cells, Segev et al. investigated how well targets of different sizes could be distinguished based on the cells’ spike trains. They found that the best distinction was obtained for the time period immediately following a saccade. The sudden changes in the image that the cells experience after the saccade activate them strongly and reliably, whereas responses during fixation were comparatively weak and unreliable and only slightly improved when small fixational eye movements were included. The authors conclude that saccadic vision provides the opportunity for the visual system to experience extensive global activation, thus acquiring “snapshots” of the environment.

This snapshot concept may be aided by the fact that several ganglion cell types are suppressed during the saccade as discussed above (Roska and Werblin, 2003). The suppression could function as a reset of the message stream sent by the ganglion cells. The reduced activity may mark the beginning of a new signal transmission episode, distinct from previous episodes of the neural code. For reading out the neural code, this has the advantage of providing a specific context for each observed spike as being part of well-demarcated events. For example, this allows for robust signal transmission via the timing of the spike events at the beginning of fixation.

That the arrival times of the first spikes at the onset of a stimulus could form a powerful and rapid neural code for the visual system had been proposed and investigated in a series of theoretical studies (Thorpe and Imbert, 1989; Thorpe, 1990; Gautrais and Thorpe, 1998; VanRullen et al., 1998; Delorme and Thorpe, 2001). These investigations were motivated by the observation that neurons in higher visual areas can show highly selective responses to visual stimuli already around 100 ms after the onset of the stimulus (Perrett et al., 1982). Given that the visual information has to cross several neuronal stages in that time, it was argued that the required information transmission and computations must be based on few spikes and potentially a single spike per neuron at each stage (Thorpe, 1990).

A recent experimental investigation showed that a population of ganglion cells measured in the salamander retina can indeed transmit considerable amounts of information by the latencies of their first spikes (Gollisch and Meister, 2008). This study analyzed how the retina can encode a visual image when it is only briefly displayed. To this end, ganglion-cell spike trains were recorded in response to flashed images. Again, cells responded with bursts of spikes shortly after the onset of the image. The fastest responses came from specific types of ON-OFF cells that also displayed high precision in the timing of the first spike. Most importantly, however, these cells responded with spike bursts of nearly identical numbers of spikes to very different images, whereas the latency of the first spike could differ by about 40 milliseconds. Examples of these responses are shown in Fig. 5. The precise and systematic shifts in spike timing may constitute a powerful channel for information transmission. Indeed, it was shown that, for the vast majority of recorded neurons, the first-spike latency provided considerably more information than the number of elicited spikes. For these cells, a latency code is thus a reliable source of information. Moreover, it also represents a particularly fast code—the information is already available with the first spike, thus providing a potential substrate for rapid image processing.

Figure 5. Ganglion cell responses to flashed stimuli.

Spike trains from two ON-OFF retinal ganglion cells in the larval tiger salamander are shown for repeated presentations of 16 different stimulus patterns (shown on left). Each stimulus consisted of a different subdivision into bright and dark regions and was flashed onto the retina for 150 ms, following an intermediate gray illumination. Both neurons reliably responded to all 16 patterns and displayed a marked dependence of their latencies on the stimulus.

The same study also considered the relative timing of the first spikes from pairs of ganglion cells and found that this provided an even better source of information. Information in relative spike timing could be directly read out by downstream brain regions, for example, through delay lines and coincidence detection. By contrast, information contained in the spike timing of a single cell alone requires an additional reference signal, which—in the case of saccades—could be supplied by a corollary discharge that accompanies the relevant motor commands (Sommer and Wurtz, 2002). It is still not quite clear, however, to what extent the brain uses the corollary discharge of eye movements.

A code based on relative spike timing may have other advantages. In the case of salamander retinal ganglion cells, it was shown that this relative timing is to a large degree independent of visual contrast, thus providing direct information about the image structure independent of illumination conditions (Gollisch and Meister, 2008). Furthermore, the relative timing was surprisingly robust; fluctuations in relative timing, measured over repeated stimulus presentations, were often smaller than for the spike times of the individual cells. In other words, the first-spike times of simultaneously recorded neurons were often strongly correlated; when one of them happened to fire a bit earlier in a given trial, the other tended to follow suit. Different mechanisms can be envisioned that may underlie these correlations. Neighboring ganglion cells can be coupled through electrical gap junctions, and these can have a synchronizing effect in the millisecond range (Brivanlou et al., 1998; Hu and Bloomfield, 2003). Alternatively, shared input into ganglion cells from bipolar or amacrine cells may result in correlated neuronal activity (Mastronarde, 1983; Levine, 1997; Murphy and Rieke, 2008). Regardless of the mechanism, the intercellular correlations in spike timing underscore the potential for temporal coding in the early visual system.

THALAMIC AND CORTICAL PROCESSING DURING SACCADE-LIKE STIMULATION

Given the exquisite sensitivity of thalamic and cortical neurons to precisely timed spike patterns (Usrey et al., 2000; Kara and Reid, 2003), the information contained in temporal spiking patterns of retinal ganglion cells should be well accessible. Whether and how this information is read out or transmitted by these downstream brain areas, however, is an open question and should provide a promising research avenue. Viewing neural coding in the light of brief image presentation, saccades, and rapid object detection may, in fact, provide a particularly suitable context for the challenging question how neuronal messages are read out, transformed, and applied for signal processing in the receiving brain areas. It may be illuminating, for example, to see whether downstream neurons display similar switches in their response properties triggered by peripheral stimulus shifts as found for retinal ganglion cells (Geffen et al., 2007).

For other neuronal dynamics, it has already been possible to compare specific properties of the neural code at different stages of the visual system. One such example regards the dynamics of receptive field size in response to flashed stimulation. Neurons in primary visual cortex exhibit sustained responses to flashed light spots only within a small region of space, whereas early transient responses can be elicited by light spots within a larger area (Wörgötter et al., 1998; Suder et al., 2002). This was interpreted as a shrinkage of the receptive field size over the first few hundred milliseconds, corresponding to a dynamic shift in information transmission from coarse to fine features of the visual image. Similar observations have later been made in thalamic neurons (Ruksenas et al., 2007), suggesting that the effect may be at least partly inherited from previous stages—potentially from the retina, as post-synaptic potentials measured in thalamic neurons show similar features (Ruksenas et al., 2007). A possible explanation for these dynamic changes in receptive field size is the delay in surround inhibition that has been observed in the retina (Werblin and Dowling, 1969).

If the visual stimulus is not flashed onto the eye, but comes into view through a saccade, neuronal processing in higher visual areas is strongly influenced by additional signals. These effects of saccades on neuronal processing and visual perception have long been a focus of visual research. On the perceptual level, the suppression of specific visual aspects—in particular motion perception—during saccades (Burr et al., 1994, 2001) and the distortion of the perception of space and time (Ross et al., 2001; Morrone et al., 2005) are the most prominent findings. In higher cortical areas of visual motion perception, such as medial temporal and medial superior temporal areas, neuronal correlates of suppression (Thiele et al., 2002) and spatial distortion (Krekelberg et al., 2003) have been identified; the activities of specific neuron types in these areas are similarly suppressed and affected in their spatial receptive field properties as the visual percept.

Various effects of saccades on neuronal responses can be found nearly throughout the visual system. Even without visual input, the neural activity is modulated before, during, and after the saccade in different cortical areas and as early as the thalamus (Noda, 1975b, a; Fischer et al., 1998). The most prominent modification appears to be a succession of suppression and facilitation of neuronal activity, observed both in thalamus (Lee and Malpeli, 1998; Reppas et al., 2002) and in primary visual cortex (Vinje and Gallant, 2000; Maldonado and Babul, 2007). The suppression typically starts already before the beginning of the saccade, and the facilitation reaches a maximum about 100 ms after the beginning of fixation. The modulations of neuronal activity with saccades in the absence of visual input are taken as evidence for the influence of extra-retinal signals on visual processing pathways—in the form of corollary discharges that accompany saccade generation (Sommer and Wurtz, 2002).

Corollary discharges are also thought to underlie the remapping of receptive fields in areas V2, V3 (Nakamura and Colby, 2002), and V4 (Tolias et al., 2001) of visual cortex. For some neurons in these areas, the locations in space to which they are sensitive are shifted in the direction of the saccade; in other words, the neurons appear to anticipate the new location in space about which they are to encode visual information. These remapping dynamics may contribute to our ability to form a stable percept of space despite the constant changes in gaze direction (Melcher, 2005, 2007).

Regarding the neural code, a crucial question that awaits answering is how these saccade-related dynamics and modulations interact with the visual signals that are supplied by the retina. Is the succession of suppression and facilitation in thalamus and cortex adjusted to the neural code of the retina? How does the remapping of receptive fields integrate information from different retinal locations? How is activation from the new image distinguished from activation of the previous image (Gawne and Woods, 2003)? Saccadic suppression is generally thought to prevent that our visual system is swamped with self-induced motion signals. In addition, however, it may reinforce the stimulus-induced suppression of ganglion cell activity observed in the retina (Roska and Werblin, 2003) and allow reliable read out of visual information contained in spike latencies (Kupper et al., 2005).

As higher visual areas read out and integrate the messages from the retina and compute different aspects of the visual scene, they themselves must encode visual information. Whether or not cortical areas can use temporal codes with precise spike timing for stimulus encoding has been and will likely continue to be a matter of intense debate (Engel et al., 1992; Shadlen and Newsome, 1994). Thus, it is an interesting open question whether temporally precise signals from the retina are transmitted and potentially recoded by cortical neurons into different attributes of their neuronal activity. Regarding rapid information transmission, it will be particularly interesting to investigate these issues under brief visual stimulation while comparing stimulus onsets from passive viewing to active saccades. With respect to flashed stimulus onsets, first-spike latencies in visual cortex have been associated with the encoding of stimulus contrast (Gawne et al., 1996; Reich et al., 2001). Spike synchrony is also likely to be affected by saccades; in visual cortex, for example, an increase in synchronous activity was observed following the initial saccadic suppression (Maldonado and Babul, 2007).

OUTLOOK

The dynamic structure of natural vision, which is partitioned into short episodes of fixation separated by saccades, provides a particular challenge for the visual system to integrate and transmit information. Psychophysical studies have strikingly reinforced that the visual system is capable of rapid signal processing. Together, these observations provide an exciting framework for investigating neural coding in the visual system in the context of specific relevant features of visual behavior. Regarding the neural code of the retina, the observed precise timing of spikes and the multineuronal spiking patterns suggest that temporal codes play a crucial role for information transmission, particularly in the case of brief exposure to individual images. A neural code based on relative timing of the first spikes after the onset of a new stimulus or of a new fixation period could be a powerful and fast source of information for downstream brain regions (Thorpe, 1990; VanRullen and Thorpe, 2001; Gollisch and Meister, 2008).

Exploring this possibility further requires investigations of how information transmission by retinal spikes is affected by more natural saccadic stimulus shifts. In particular, influences of the previously fixated image and of the motion signals during the saccade must be considered. A better understanding of the neural activity in the retina under natural dynamics of vision will not only deepen our understanding of fundamental neural processes, but may eventually provide essential ingredients for improving artificial vision devices for medical applications (Zrenner, 2002; Wickelgren, 2006).

For brain areas downstream of the retina, it is an open question how information is represented in such a way that it may support the image processing speeds observed in the psychophysical experiments. Probing potential neural codes in experiments with flashed images should thus provide new insights about the processing stream of visual signals. In addition, it will be of particular interest to connect these questions of neural coding and information transmission to the intrinsic activity modulation induced by saccades. Comparing neuronal signals and information transmission for flashing or suddenly shifting stimuli and for actual saccades will be a crucial tool for assessing the role of active stimulus sampling through saccades in neural coding.

A further interesting direction will be to draw connections between the spike timing features of the early visual system and of other sensory systems where similar response characteristics have been observed. In the auditory system, it has long been established that precise spike timing forms the basis of sound localization, see Carr (1993) for a review. Moreover, precise spike timing in the auditory system can also help encode stimulus identity (deCharms and Merzenich, 1996; Chase and Young, 2007). Maybe more surprisingly, sensory cells of the somatosensory system display a pronounced tuning of their first-spike times with respect to the applied touch stimulus, thus suggesting information transmission via a latency code (Johansson and Birznieks, 2004). A particularly pronounced form of a latency code can be observed in some weakly electric fish. In response to the fish’s own electric organ discharge, electrosensory nerve fibers elicit precisely timed spikes whose latencies are determined by the local strength of the electric field (Szabo and Hagiwara, 1967; Bell, 1990; Sawtell et al., 2006).

The discharging of the fish’s electric organ lets the electrosensory receptors take regular snapshots of the fish’s environment and provides a common clock signal for the spike timings of the nerve fibers. It is interesting to compare this stimulus sampling strategy to other sensory systems. In the olfactory system, for example, the rhythm of breathing—or the more active sniffing—brings new samples of odor to the olfactory receptor neurons. This may play the same role as saccades do for the visual system, as has been discussed in Uchida et al., 2006. Indeed, spike timing codes similar to the retinal examples discussed above have been suggested to underlie stimulus discrimination in the olfactory system (Cang and Isaacson, 2003; Margrie and Schaefer, 2003; Schaefer and Margrie, 2007). The parallels in the dynamics of stimulus sampling and in the observed neuronal activity patterns warrant a more detailed comparison of the signal processing in these different sensory systems. This may provide a promising source for investigating general principles regarding temporal coding in nervous systems.

Whether information contained in spike timing is indeed used by the nervous system is a critical question. Most likely, an ultimate answer will only be found through behavioral experiments. This will be among the central goals in research of the neural code in years to come. It will form part of the more general effort aimed at establishing causal relations between specific aspects of neuronal activity and system-level consequences. This constitutes one of the most formidable challenges for neuroscience. With respect to the role of retinal spike timing in visual processing, the ideal experiment would consist of a controlled disruption of the spike timing patterns while an experimental animal is engaged in solving a visual task. Hope of one day doing such experiments has been sparked by the discovery of light-activated ion channels that can be inserted into neurons (Boyden et al., 2005; Han and Boyden, 2007; Zhang et al., 2007). These could be used to subtly alter spiking patterns in neurons through direct light stimulation while observing the effect on the behaving animal. Applying these techniques in the context of visual behavior will bring along additional challenges, but the perspective of directly testing the role of various neural codes for behavior will make the efforts well worth it.