Detection and prediction of periodic patterns by the retina

Abstract

A fundamental task of the brain is detecting patterns in the environment that enable predictions about the future. Here, we show that the salamander and mouse retinas can recognize a wide class of periodic temporal patterns, such that a subset of ganglion cells fire strongly and specifically in response to a violation of the periodicity. This sophisticated retinal processing may provide a substrate for hierarchical pattern detection in subsequent circuits.

Repetitive trains of sensory stimuli have long been known to elicit characteristic patterns of voltage on the scalp in humans when they end. Such an omitted stimulus potential, or mismatch negativity, has been observed in human EEGs in the visual^1,2, auditory³ and somato-sensory⁴ domains, as well as in functional magnetic resonance imaging data⁵, and has generally been interpreted to signal “novelty”^1,6,7. Slower and more cognitive pattern recognition requires attention and is likely to arise in the neocortex⁸, whereas faster patterns may be recognized subcortically⁹. One group has measured an omitted stimulus potential from the optic tectum and optic nerve of fish following sequences of light or dark flashes and concluded that the signal must originate in the retina^10,11. In this study, we measured the omitted stimulus response directly in retinal ganglion cell spike trains, demonstrating that the retina contains temporal expectations about the visual world that have great flexibility and precision.

We presented isolated salamander and mouse retinas with a periodic sequence of spatially uniform dark flashes at a rate of 12 Hz, while recording spikes extracellularly with a multielectrode array¹² (Supplementary Methods online). Most ganglion cells responded very weakly to each flash. But when individual flashes were randomly omitted, a strong burst of spikes resulted (Fig. 1a,b). This response to the omission of a flash from a periodic sequence bears close resemblance to signals previously observed in the local field potential^1,9–11.

Figure 1.

Ganglion cell responses to omitted flashes. (a,b) Firing rate of a single ganglion cell in salamander (a) or mouse (b) during a sequence of 500 dark flashes of 40-ms duration with random omissions at a mean rate of once every 16 flashes (see Methods). Responses are aligned with respect to the time of omitted flashes (dashed line). (c–f) Firing rate of example cells from salamander (left) and mouse (right) during a sequence of 16 flashes, showing start-end (c,d), sustained (e) and end-only (f) response types. Flashes had a duration of 40 ms (salamander) or 24 ms (mouse) and were presented at 12 Hz. Dashed lines show the time of the next expected flash. All cells have an OFF-type receptive field, except for the one in f, which is ON-type.

To characterize the omitted stimulus response (OSR) and determine what kinds of stimuli evoke it, we performed subsequent experiments using short flash sequences. Many ganglion cells fired a volley of spikes at the end of the flash sequence, which was clearly not driven by the last flash. Often this OSR was much stronger than the response to the first flash in the sequence. Within the ganglion cell population, several kinds of responses were observed: ‘start-end’ cells (Fig. 1c,d) fired only at the beginning and end of the sequence, ‘sustained’ cells fired throughout the sequence as well as at the end (Fig. 1e), and ‘end-only’ cells fired only at sequence end (Fig. 1f). All three kinds of responses to the end of a periodic flash sequence were seen in both salamander and mouse retinas.

Flash sequences as slow as 6 Hz can elicit an OSR, and the effect is present up to 20 Hz, the highest frequency tested. The peak firing rate of the OSR tended to increase as the frequency of the flash sequence increased. An OSR was apparent after as few as four flashes (Supplementary Fig. 1 online), and a weak response could sometimes be observed after two flashes. When we used reverse correlation to map ganglion cell receptive fields, we found that a large fraction of both ON cells (40/40 = 100% in salamander; 14/23 = 61% in mouse) and OFF cells (133/317 = 42% in salamander; 29/61 = 48% in mouse) showed an OSR for dark flashes presented at 12 Hz. Although there was great heterogeneity, ON/OFF cells tended to have a stronger OSR than pure OFF cells and medium OFF cells had a stronger OSR than fast OFF or slow OFF cells (Supplementary Figs. 2 and 3 online).

Rather than simply signaling that the flash sequence had ended, the OSR was actually predictive of the time at which the next flash should have occurred. The latency to the time of peak firing depended on the frequency of the preceding stimulus, shifting by over 100 ms with respect to the time of the last flash as the frequency of the sequence varied between 6 and 20 Hz (Fig. 2). When compared to the time when the next flash should have occurred, we found that the average OSR latency in the population was approximately constant for all frequencies tested (Fig. 2b). This OSR latency (80 ± 23 ms, mean ± s.d.; n = 79 cells) was longer than the latency of responses to the first flash (56 ± 18 ms). However, it was roughly the same as the latency for sustained responses during the sequence (73 ± 22 ms). As a result, these ganglion cells seem to signal the precise time at which an expected flash is absent.

Figure 2.

The omitted stimulus response is predictive. (a) Firing rate of a ganglion cell following a 12-Hz (upper) or 20-Hz (lower) sequence of 40-ms dark flashes. Each sequence has 12 flashes; stimuli are aligned so that they begin at different negative times (off the time axis) and have their last flash begin at time zero. Dotted lines indicate the time at which the next flash would have occurred. (b) Response latency of ganglion cells relative to the latency of the response to an isolated flash (56 ± 18 ms; n = 76 cells) plotted over a range of stimulus periods. Bars, s.e.m. Notice that the time between the last real flash and the OSR increases linearly with the stimulus period (solid diamonds); dashed line is a slope of unity, not a curve fit. All data taken from OFF-type salamander ganglion cells.

If the retina is really recognizing a temporal pattern, then its predictive response should not be sensitive to the precise details of the flash sequence. We tested this idea using a variety of periodic flash sequences. When the stimulus was sinusoidal rather than square-wave, the response of ganglion cells at sequence end was nearly identical (Supplementary Fig. 4 online). When we used bright flashes rather than dark, again many ganglion cells showed an OSR (Supplementary Fig. 4), although an individual ganglion cell would typically fire at the end of either a bright or a dark flash sequence, but not both. The OSR for bright flashes shifted predictively as a function of the frequency of the sequence, as for dark flashes. Finally, we found that the OSR was robust with respect to minor changes in the stimulus pattern, being present over the entire range of light levels (1–30 lx) and flash durations (20–45 ms) studied with a cathode-ray tube monitor and light-emitting diode stimulation.

Another requirement for interpreting the OSR as arising from a violation of the retina’s prediction is that this response should be elicited by the periodicity per se. An alternative explanation is that the OSR is simply triggered by an overall change in light level, as the mean light level is lower during the sequence of dark flashes than between trials. To address this possibility, we designed a stimulus in which the light intensity between flashes is higher than the background intensity, such that the mean level remains constant. In this condition, many ganglion cells still showed a sharp peak in firing rate at the same time relative to the end of the sequence, although the response tended to be somewhat more transient than when the light level was not held constant (Supplementary Fig. 4). Together, these data indicate that the retina has a general capability to recognize rapidly repeating temporal patterns.

The visual world contains periodicities in both space and time as well as across multiple frequencies. To begin to explore the limits of retinal pattern detection, we probed with more complex patterns. We found that a subset of ganglion cells fired strongly at the end of two kinds of spatiotemporal patterns—a dark bar that moved back and forth rapidly, and high-frequency drifting gratings that abruptly stopped (Supplementary Fig. 5 online)—in analogy to the OSR for spatially uniform flashes. We also found that some ganglion cells responded selectively to a violation of a sequence having two different flash amplitudes or two different time intervals between flashes. For instance, the cell at the top of Figure 3a did not respond at all to big or small flashes after the two-component sequence started, but when a big flash was followed by second big flash, rather than the expected small flash, this cell fired at over 200 Hz. When stimulated by a pure sequence of big flashes, this same cell had a ‘start-only’ characteristic: firing only to the first flash, not the second or subsequent flashes, nor at sequence end (data not shown). Thus, such a neuron is not simply triggered by two big flashes in a row (‘big/big’), but instead detects a violation in the two-component pattern (‘big/small’). These data show that the retina can recognize patterns with multiple frequencies. They also show that the retina can recognize not just the timing of periodic flashes, but also their amplitude.

Figure 3.

More complex patterns. (a) Upper, light intensity from a flash sequence that alternates between a ‘big’ flash (100% contrast, ‘B’) and a ‘small’ flash (50% contrast, ‘S’), with a single violation consisting of two successive ‘big’ flashes (dashed line). Lower, firing rates of two different ganglion cells. (b) Upper, light intensity from a flash sequence that alternates between big and small flashes with a ‘small/small’ violation (dashed line). Lower, firing rates of two different ganglion cells. (c) Flash sequence that alternates between a long time interval between flashes (100 ms, ‘L’) and a short time interval (71.4 ms, ‘S’) having a ‘long/long’ violation (dashed line); responses of two ganglion cells are shown below. (d) Flash sequence that alternates between long and short time intervals with a ‘short/short’ violation (dashed line); responses of two ganglion cells are shown below. All data are from different OFF-type ganglion cells from the salamander.

The ganglion cell population showed great heterogeneity under these visual conditions. Although 13 of 47 cells fired after a ‘big/big’ violation (Fig. 3a, top), we also found that 17 of 47 cells fired once per pattern and instead selectively skipped the ‘big/big’ violation (Fig. 3a, bottom). Such neurons can serve as positive detectors of the two-component pattern under these circumstances. Other neurons were excited by a ‘small/small’ violation (n = 15/47 cells); curiously, these cells fired after the next big flash, rather than after the small flash that violated the two-component pattern (Fig. 3b, top). Still other neurons were actively inhibited by the ‘small/small’ violation (n = 10/47 cells), such that their response to the next big flash was suppressed (Fig. 3b, bottom). Similar responses were seen during two-component sequences formed from a long time interval between flashes followed by a short time interval (Fig. 3c,d). Again, some ganglion cells were excited by the violation (top panels, n = 13/43, 12/43 cells), whereas other cells showed inhibition or a lack of excitation (bottom panels, n = 10/43, 10/43 cells). Notably, a given ganglion cell tended to react to only one of the four possible violations that we tested, indicating that these responses are not a generic signal of novelty but instead were specific to particular temporal patterns. Together, these results greatly expand the known scope of the retina’s predictive capabilities and broaden the behavioral relevance of the phenomenon.

The fact that deviations from expectation can be explicitly represented in the neural code as early as the retina expands our appreciation of the computational capabilities of local circuits. In particular, there is strong evidence that some ‘novelty’ potentials measured over the human cortex derive from the retina or other subcortical structures^2,13, consistent with the ability of these lower circuits to perform such a computation. Because the cortical column is a circuit with at least as much complexity and computational power as the retina, any computation that the retina can carry out is likely to be available to the cortical column as well. Thus, more complex predictions made by the brain may in some cases result from very similar computations carried out in higher cortical regions, where the input to the circuit represents stimulus features more complex than a simple flash.

Supplementary Material

01

Supplementary Figure 1. Changes in the number of flashes in the sequence. (A) Number of spikes per trial for the OSR in the salamander plotted as a function of the number of dark flashes presented at 12 Hz. Bars are standard error. (B) Example ganglion cell with an OSR after a sequence of only 4 flashes.

Supplementary Figure 2. OSR strength for different cell types. A. Salamander. Bars indicate average values: fast OFF = 0.18, medium OFF = 0.44, slow OFF = 0.06 spikes/trial. B. Mouse: fast OFF = 1.7, medium OFF = 2.6 , slow OFF = 1.7 spikes/trial.

Supplementary Figure 3. OSR strength versus ON-OFF index. A. Salamander ganglion cells. B. Mouse ganglion cells. Dashed lines are a linear regression.

Supplementary Figure 4. Robustness to changes in the stimulus pattern. (A) Firing rate of a salamander ganglion cell during a standard flash sequence (upper, 20 ms flashes) and a sinusoidal variation in light intensity (lower). (B) Firing rate of two salamander ganglion cells (left, right) during a sequence of 20 ms bright flashes with constant mean light level. (C) Firing rate of a salamander ganglion cell during a standard, dark flash sequence (upper, 20 ms flashes) and a flash sequence where the mean light level remains constant (lower).

Supplementary Figure 5. Spatiotemporal patterns. (A) Periodically moving dark bar. Upper: movement of bar as a function of time. Middle: light intensity at the center coordinate of the ganglion cell. Lower: firing rate of a ganglion cell. (B) Schematic depiction of the moving bar stimulus. (C) Drifting sinusoidal grating. Upper: movement of the grating as a function of time. Middle: light intensity at the center coordinate of the ganglion cells. Lower: firing rate of a ganglion cell. (D) Schematic depiction of the drifting grating stimulus.