Evolutionary conservation and neuronal mechanisms of auditory perceptual restoration

Abstract

Auditory perceptual ‘restoration’ occurs when the auditory system restores an occluded or masked sound of interest. Behavioral work on auditory restoration in humans began over 50 years ago using it to model a noisy environmental scene with competing sounds. It has become clear that not only humans experience auditory restoration: restoration has been broadly conserved in many species. Behavioral studies in humans and animals provide a necessary foundation to link the insights being obtained from human EEG and fMRI to those from animal neurophysiology. The aggregate of data resulting from multiple approaches across species has begun to clarify the neuronal bases of auditory restoration. Different types of neural responses supporting restoration have been found, supportive of multiple mechanisms working within a species. Yet a general principle has emerged that responses correlated with restoration mimic the response that would have been given to the uninterrupted sound of interest. Using the same technology to study different species will help us to better harness animal models of ‘auditory scene analysis’ to clarify the conserved neural mechanisms shaping the perceptual organization of sound and to advance strategies to improve hearing in natural environmental settings.

The natural condition of trying to detect a signal in the presence of noise is of fundamental importance. In humans this is often referred to as the cocktail party problem (Cherry, 1953) where the goal is to understand a single person in a crowded room with many people talking. However, from an ecological perspective detecting and tracking a signal in the presence of noise is critical for many animals. For example the inability of an animal to hear a predator encroaching could have deadly consequences. Similarly the inability to identify a vocalization used in mate attraction lowers the chance of successful reproduction. Because of this, animals that can negotiate noisy environments have an improved chance of establishing social interaction, mating and surviving. Here, we review the phenomenon of ‘auditory perceptual restoration’, which has gone by many names since its initial description (Miller and Licklider, 1950). We first introduce the challenge that the auditory system faces in restoring degraded sounds. Then we consider how broadly evolutionarily conserved this phenomenon may be and, upon this basis, what we are beginning to understand of its neuronal bases in humans and other animals.

1. The challenges of restoring masked or obliterated sensory input

Under natural listening conditions, there are many sound producing objects. This interferes with the ability to track sound emanating from a single source and to discriminate and identify features of that sound. Under these conditions we can consider the sound source and features we wish to follow as signal and sound emanating from other sources as ‘noise’. While sound location is one cue that can be used for source segregation, there is abundant evidence that under most conditions non-spatial factors are important for separating signal from noise (Bregman, 1990; Divenyi and Oliver, 1989; Kalikow et al., 1977; Turgeon et al., 2002). Noise originating from nearby sources poses different problems to the auditory system than noise originating from a long distance. Noise arising from a distance can originate from many sound sources because the circumference around the listener increases with distance. Because there are potentially many sources at a distance, and sound attenuates with distance, remote sources tend to blend together into a fairly constant, often low-intensity noise. The auditory system under these conditions extracts the signal based on its ability to hear the signal above the noise. This is a traditional masking problem.

A different scenario arises when nearby objects produce sounds. Because of their closer proximity, nearby sources tend to be fewer and louder than distant sources. Therefore proximal sounds don’t create a smooth low intensity average, but rather tend to produce noise with large amplitude peaks and dips, both temporally and spectrally. Under these conditions the auditory system cannot detect the entire signal; the peaks in noise amplitude are so high that the signal during them is completely masked or obliterated. Here, the auditory system has to estimate what the signal should have been based on what was heard before and immediately after the peak in the noise that interrupted it. This is a model-based reconstruction of the sound. While there is a relatively large literature on listening during dips in noise (Borrill and Moore, 2002; Buus, 1985; Fullgrabe et al., 2006; Hall et al., 1998; Moore et al., 1999), the focus of this article is on efforts to determine the mechanisms underlying what is perceived and actively restored during noise occlusion.

For the auditory system, filling in or restoring the information that is interrupted by noise is conceptually similar to the visual system filling in partially occluded objects. For example, auditory fill-in is in some cases comparable to a dynamic version of the ‘picket-fence’ effect (Fig. 1). Auditory fill-in goes by several names (Bregman, 1990; Miller and Licklider, 1950; Warren, 1970; Warren et al., 1972), being often called ‘perceptual restoration’ because the system is restoring the obliterated sound segment. It is also called ‘auditory induction’, ‘temporal induction’ or ‘continuity’ to emphasize that the segments of sound preceding and following the obliterated segment can be used to induce perceptual restoration of the obliterated segment during the occlusion. The term induction is used to contrast with masking where noise is constant during the entire sound (such as the example of many sources far away), in which case sound segments preceding and following the segment of interest are masked and not available to induce a best-guess percept. It also has been called ‘amodal completion’ to highlight the analogies to perceptual phenomena and illusions in the other sensory systems. In this review, for consistency, we mainly use the term ‘restoration’ to refer to these processes, but, for clarity, we also sometimes use the term ‘continuity’ in the case where this refers to tasks that are based on continuity/discontinuity judgements or the continuation of neuronal responses.

Fig. 1.

Conditions for eliciting auditory perception restoration and analogous visual phenomena. (A) On the left is a drawing of a cat illustrating an intact sensory representation. In the middle we simulate sensory degradation whose source is unclear. Finally on the right the blank areas are filled in identifying the source of the sensory degradation. With this information the visual system can segregate the objects and restore a more global perceptual representation of the ‘cat’. (B) Schematized spectrogram (time-frequency plots) of different auditory stimuli and how they relate to masking and perceptual restoration (continuity). Note how the ‘gapped’ monkey vocalization with interrupting noise in the gap (the condition that evokes fill-in) is similar to the visual picket-fence except the missing segment occurs in time.

The natural condition of a continuous sound being interrupted by loud peaks in noise (interrupting noise) can be exploited to create an illusion. Here a portion of the signal is replaced with silence and noise completely fills that silence (Fig. 1B demonstrates this for a monkey ‘coo’ vocalization). If we keep the loud interrupting noise present we still hear the signal (coo) as if it were continuous, even though we have placed ‘silent’ segments in the signal so that there is no structure associated with the coo during the noise. If we remove the interrupting noise the silent gaps in the coo are obvious. This particular phenomenon has also been called the continuity illusion (Bregman, 1990) which emphasizes the importance of perceived continuity for this illusion (Warren et al., 1972). For the illusion to occur several conditions must be met. First, there must be sufficient signal before and after the noise to induce a filled in percept during the noise. Second the noise must be sufficiently loud so that the gap or transitions into the gap cannot be detected (e.g., Bregman and Dannenbring, 1977). Third the noise must completely fill the gap, both temporally and spectrally; if there is evidence for a gap, no restoration occurs. Fourth the gap cannot be too long. For a general review of the behavioral work see: (Bregman, 1990).

Perceptual restoration of partially obliterated signals is rather universal. It has occurred for every type of signal tested (tones, FM sounds, vocalizations, speech, etc.), and the conditions that are necessary for eliciting perceptual restoration are well established see, e.g., (Bregman, 1990; Darwin, 2005; Lyzenga et al., 2005; Plack and White, 2000; Riecke et al., 2008; Samuel, 1997; Warren et al., 1972). Since speech is a recent evolutionary adaptation, being able to approach the study of auditory restoration with non-speech sounds that range from simple sounds (e.g., pure-tones, AM tones or FM tones) to complex sounds (e.g., species-specific vocalizations or bird song) allows more direct cross-species comparisons to be made.

2. Do non-human animals experience auditory restoration?

What are the evolutionary origins of auditory restoration and exactly which species are susceptible to it? This is important to ask if we are to better understand the neural mechanisms underlying perceptual restoration. It seems obvious that the ability to fill-in incomplete information would be useful to animals, but until recently it was unclear if any animals besides humans experience this phenomenon or whether good animal models exist to study fill-in with techniques that are impractical for study in healthy humans.

There is now mounting evidence that the ability to fill-in interrupted sounds is present in animals. While there is a vast human literature on this perception and its relationship to other aspects of ‘auditory scene analysis’ (for reviews see Alain and Arnott, 2000; Bregman, 1990; Carlyon, 2004; Ciocca, 2008), providing evidence from animals is more difficult because of the subjective nature of reporting illusory perceptions.

Over recent years a growing body of literature supports that illusory restoration is a process that may be present not only in humans, but also in non-human primates (Miller et al., 2001; Petkov et al., 2003), cats (Sugita, 1997) and birds (Braaten and Leary, 1999; Seeba and Klump, 2009). The techniques used range from psychophysical training to natural response paradigms. While each technique has its own advantages and limitations, taken together they complement each other. When taken in totality, the combination of approaches that have supported restoration makes a compelling case that auditory restoration occurs in non-human animals. For example an advantage of applying psychophysical approaches in macaques is that animals can be trained on difficult tasks to obtain thresholds that can be compared to and often directly related to those obtained from humans; however a disadvantage is the process of training the animals takes away the significance of the illusion and may change the way neurons respond. Natural response paradigms can reveal abilities animals already posses and have the advantage of requiring no training, but suffer from not being able to take advantage of many kinds of quantitative comparisons especially across species. Also many natural paradigms do not lend themselves well to the application of neural recordings. Thus any one particular approach is itself inherently limited, but often one technique is strong where another is weak. Furthermore, since auditory restoration is a robust phenomenon that occurs for many types of sounds, studies that have used simple sounds and complex sounds are equally useful as evidence that animals can perceptually restore different types of sounds. To substantiate these points, we now look closely into a few examples from the animal literature.

Sugita (1997) provided early evidence for auditory restoration in cats. The animals were presented with either a continuous or discontinuous exponential frequency-modulated (FM) sweep (a change in tone frequency that is linear in log frequency vs. time plot). The discontinuous FM had a silent gap temporally located in its middle. The cat was rewarded for approaching the audio speaker when a continuous FM sweep was presented. On a small number of trials Sugita introduced a band-passed noise temporally centered in the sweep, although the paper does not provide enough detail on the noise spectrum to determine whether the noise spectrally filled the gap. Sugita reports that humans perceived the FM with a gap temporally filled by noise as continuous at high noise levels. The animal performance curves look similar to those of the humans, which was interpreted as supporting that ’auditory induction’ occurs in cats (Fig. 1 of Sugita, 1997).

The case that animals can without training restore masked parts of complex sounds was supported from a study in a primate – the cotton-top tamarin monkey (Miller et al., 2001). Tamarins often respond to conspecific vocalizations by vocalizing back. This is called ‘antiphonal’ calling, and was exploited to test whether the animals responded to various manipulations of the vocalization as if they were the natural ones. The animals responded to calls with silent gaps in them at a lower rate than to continuous calls. Miller and colleagues showed that complete calls interrupted in the middle by loud noise were responded to as if they were complete vocalizations. This is contrasted with the case where the noise overlapped with the beginning or end of the vocalization (such as in the two panels on the right of Fig. 1B). Under these conditions, which don’t allow restoration because a proper context is not provided, the animals responded behaviorally at the same low rate as they did to discontinuous vocalizations. Besides demonstrating putative correlates of restoration within a natural response paradigm, this also added the important notion that perceived restoration in the animals depends on hearing part of the signal before and after interrupting noise, an essential property for restoration in the human literature (Bregman, 1990; Ciocca and Bregman, 1987).

We wished to determine whether auditory perceptual restoration could be observed in another primate, rhesus macaque monkeys (Petkov et al., 2003). Psychophysical paradigms were used that allow for a more direct comparison to human psychophysical reports of restoration. Our data support the case for restoration (‘temporal induction’ or ‘continuity’) in macaques in two ways: (1) the animals’ response biases to stimuli which caused different perceptions were consistent with the illusion; and (2) thresholds to the different conditions shifted in a manner consistent with human thresholds (e.g., Kluender and Jenison, 1992).

How the macaques’ response biases support continuity is illustrated in Fig. 2. In this task the animals hear two sounds and report by a lever press whether the second sound is continuous or discontinuous. The first sound was always a complete (continuous) signal combined with noise (Fig. 2A). The second sound has both the same noise and segment of signal before and after the noise but the signal is either continuous (Fig. 2A bottom) or discontinuous (Fig. 2A top). The animal’s task was to press the lever to begin a trial and if the second sound differed from the first (had a gap in it) the monkey should release the lever to obtain a reward. If, however, the second sound was the same as the first (continuous signal), then the macaque should continue to hold down the lever for a reward. In other words, the training strongly encouraged the animals to overcome the illusion by rewarding them for correctly detecting the gap in the signal (pure tone, FM or a coo vocalization) during the noise. Within any trial the noise was the same but across trials the noise differed both in type and intensity to match conditions that create different strengths of perceived continuity in humans. The types of noise in this study were interrupting noise (Fig. 1B bottom left and Fig. 2B left) which is thought to cause continuity (restoration), and noise which completely surrounds the signal in time (“surrounding noise”, Fig. 1B bottom right and Fig. 2B right) which causes masking. From human psychophysics it has been established that for interrupting noise at high noise intensities a continuous signal is heard through the noise whether or not the signal is indeed continuous. For masking (‘surrounding’) noise at high noise intensities only noise is heard, and the signal is not perceived (e.g., Kluender and Jenison, 1992).

Fig. 2.

Macaque monkey psychophysics supporting the continuity illusion. (A) the two rows show the two different possible stimulus configurations in the task. In the column, the first set of sounds are always complete, when the second sound has a gap, the macaques must release the lever for a reward. When the second sound is continuous the monkey must not release the lever for reward (otherwise a false alarm, FA, is registered and no reward is given). In total there are four possible responses; two correct and two incorrect for each of these two types of trials. Importantly, the noise is irrelevant because it is the same and equally present in both of the sounds that are heard and the animals are strongly encouraged to detect the discontinuous sound. (B) The different signals used in Petkov et al. (2003) (C) Monkey performance for the two different stimuli for interrupting noise (circles) and surrounding noise (triangles). Hits and false alarms map onto the different conditions shown in (A). See the text for details.

One of the macaque results that supports continuity is that for interrupting noise, which causes the perception of continuity in humans as the intensity gets higher, the animals responded almost always as if the signal were continuous. This is true for trials when the second sound was discontinuous as exemplified by the low hit rate (Fig. 2C, open circles), and for trials when the second sound was continuous as exemplified by the low false alarm rate for this condition (Fig. 2C, filled circles). This means that at the high noise intensities the animals responded as if the physically discontinuous sound was continuous, even though the animal could receive a reward for correctly identifying the discontinuity. A different scenario is seen for surrounding (masking) noise where the percept should be masking (e.g., only hearing the noise and an inability to hear the signal when the noise intensity is high). For surrounding noise, both the hit (Fig. 2C, open triangles) and false alarm (filled triangles) rates go to chance levels at the high noise intensities, suggesting that the animals are adopting a guessing strategy due to uncertainty. In summary, the combination of these two results indicate that 1) the macaques responded consistently as if the signal were complete or continuous for interrupting noise that in humans elicits restoration, and 2) the animals responded as if the signal was undetectable for surrounding noise that results in masking in humans. This result, in terms of concept and logic, is consistent with a finding by Kluender and Jenison (1992) that false alarm rates can provide insight into the subjective experience of perceptual restoration.

The second line of reasoning supporting restoration is how thresholds shift when we manipulate the stimuli in ways that shift continuity perception in humans. One result that can be seen in Fig. 2C is that the threshold for detecting the continuity of a coo vocalization in the interrupting noise is at a lower decibel level than that for surrounding noise. This means that a less intense noise is needed to elicit continuity than to mask the signal from being perceived. This is again consistent with the animals perceiving a continuity illusion and is consistent with human perceptual results that auditory continuity requires a less intense noise than masking (Kluender and Jenison, 1992). For the tone signals, we also used noise with spectral notches and found an amelioration of continuity dependent on the spectral range (center frequency and width) of the notch consistent with the human literature (Warren et al., 1972). Lastly, we measured the macaques’ ability to hear continuity as a function of noise intensity (with gap duration fixed) and as a function of gap duration with noise intensity fixed, e.g., (Petkov et al., 2003; Riecke et al., 2008). For each of these experimental manipulations, our results with macaques are generally consistent with human perceptual reports.

Work in European Starlings (Braaten and Leary, 1999; Seeba and Klump, 2009) provides evidence of auditory restoration in song-birds. In Braaten and Leary (1999), starlings were trained to peck one response key to indicate a starling song and another to indicate a parakeet (Budgerigar) song. On probe trials that were not shaped by reinforcement, the authors observed that when noise filled the starling song gaps the starlings tended to peck the ‘starling-song’ response key more often than in other conditions (e.g., when only noise was presented or the songs contained only silent gaps).

In Seeba and Klump (2009), operant training was also used, but in this case the animals were trained to listen to sound sequences and to fly off the perch if any deviation occurred to get a reward. A complete song, song with gaps and song with noise-filled gaps were individually presented. The latter stimulus in humans should produce illusory restoration. The birds were trained to respond to a deviant stimulus from a sequence. The response latencies to the stimulus that produces illusory restoration were similar to those produced by continuous song, supporting the underpinnings of the illusion in the birds. Of note in the study was that, the effect seemed to be present only for songs familiar to the animals rather than occurring for just any starling song. This observation suggests that restoration is stronger for familiar content, which could relate to the schema-based (Bregman, 1990; Samuel, 1997) and contextual effects (Riecke et al., 2008) that have been shown in humans. These observations may also relate to monkey behavior, in that with macaques we observed that the continuity illusion was easier to induce for species-specific vocalizations than for tones, see (Petkov et al., 2003).

The body of work with human and other animal behavior suggests that auditory restoration might be common to many species. The vastly different behavioral techniques that have all produced results consistent with restoration support the notion that the perception of the continuity illusion is robust in the animal world and not necessarily a result of the particular paradigm that was used to obtain the results. An important way forward would be to more precisely pinpoint the origins of auditory restoration by identifying other species that can rely on restoration and if there are any species that have more limited restorative abilities. Considering that fish and insects have been shown to be capable of certain types of ‘auditory scene analysis’ such as streaming (Hulse, 2002; Schul and Sheridan, 2006), auditory restoration may likewise be an all encompassing phenomenon in animals. Further data is needed to determine the types of animal models that can be developed for neuroscientific study of auditory restoration and whether subtleties exist that make different animal models optimized to study different aspects of restoration.

3. Insights from animal neurophysiology: restoration is related to both increases and decreases in firing rate

As is often the case, knowledge of the neurobiology underlying perception follows a growth in behavioral research, which establishes the necessary foundation for neurobiological study. As such there have been fewer neurophysiological recordings in the animal models of auditory restoration.

Schreiner investigated neuronal responses in the medial geniculate body (MGB) of the auditory thalamus of unanaesthetized guinea pigs while presenting sequentially alternating 100 ms broadband (0.1–22 kHz) noise and 100 ms tones, with the sequence repeating up to 75 times (Schreiner, 1980). This type of alternating stimulus is known to cause a continuity illusion in humans (Houtgast, 1972; Hellstrom, 1989; see also supplementary material). Parameters such as noise level, tone level, and tone duration were varied in a manner that was known to alter the perception of auditory restoration for similar sequences in human psychophysical experiments (Schreiner et al., 1977). Ultimately, if a neuron is mediating the perception of the continuity illusion, its responses to a discontinuous tone that is restored by noise should be indistinguishable from its responses to a continuous tone.Schreiner (1980) found that transient tone onset responses were suppressed during the tone/noise sequence, suggesting that the cell did not readily detect tone onsets during continuity perception. This effect was stronger when the noise was louder, the tone was softer, or the tone/noise durations were shorter, all of which are consistent with how the strength of the perception of restoration varies as these parameters are varied. This result is suggestive of an auditory restorative process, but because most of these neurons responded well to noise, is also consistent with masking and adaptation of the tone onset response (Harris and Dallos, 1979), the latter of which has also been reported during streaming at various stations in the auditory pathway (Fishman et al., 2001; Kanwal et al., 1999; Micheyl et al., 2003, 2005). The sustained responses, however, were not consistent with the hypothesis that these neurons were mediating the perception of the continuity illusion. Sustained responses to a continuous tone need to be maintained during the continuity illusion but Schreiner (1980) found that sustained responses during the noise portions of the sequence were small or completely suppressed. This observation suggests that the process of auditory restoration is not complete at the level of the thalamus. Later we will show that in primary auditory cortex (A1), neurons that respond poorly to noise but have sustained responses to tones do exhibit sustained responses to noise during illusory restoration (Fig. 3B), suggesting that this is an emergent cortical property.

Fig. 3.

Heuristic model and example A1 neuronal responses consistent with perceptual continuity/restoration. (A) Stereotyped sustained (tonic) and phasic responses are commonly observed in auditory cortex and would support continuity in different ways. For brevity we only show phasic onset responses; for other types of phasic responses see (Petkov et al., 2007). Sustained responders decrease their firing during discontinuities whereas phasic responders increase their firing. A lack of phasic responses would be as important as is the maintenance of sustained responses because if brief gaps are detected, the phasic responders can quickly provide information sufficient to reject perceptual continuity (Bregman and Dannenbring 1977). (B) Reponses of a sustained A1 neuron to different stimuli including the one that causes the continuity illusion (lower right in B). Note that the response to the illusory stimulus is similar to the response to a continuous tone (line with arrow). The red (dark in black and white figure) histogram bins show the time periods that were used in the analysis; the temporal window chosen for counting spikes was the window that produced the largest response difference between a tone and a tone with a gap (both conditions without noise). This window then was fixed and used to analyze all conditions. (C) Responses of a phasic neuron to different stimuli including the ones that causes the continuity illusion (lower right of C). Note how the response to the illusory stimulus is similar to the response to a continuous tone (line with arrow).

Sugita (1997), as well as providing possible behavioral support of auditory restoration in cats also described neuronal responses that could be consistent with it. He recorded from A1 of anesthetized cats. In the study once a neuron was shown to respond to a logarithmic FM tone sweep (0.2–15 kHz, 6.34 octaves/s), a 100 ms gap was placed temporally in the center of the sweep. Sugita only recorded from units where the gap seemed to eliminate the neuron’s responsiveness to the FM. Then a 110 ms band-passed noise (center frequency 0.25 or 16 kHz, 1/3 octave band) was placed to fill the gap. Under conditions, that were argued to correspond to perceptual restoration, the neurons’ responsiveness came back as if the FM were continuous. There are a few elements of the experimental design that constrain the interpretation of these results. Namely, the noise does not spectrally match the deleted portion of the FM, suggesting that this might not be a stimulus that causes perceived continuity. Also there was a strong sampling effect. Only neurons that responded to full FMs and did not respond to FMs with gaps or either of the 1/3 octave noises were included, reducing the sample from 326 neurons to 104 of which 65 showed a restoration effect. Therefore only a little more than ½ of the small subset were included limiting the generality of the results. Others have noted such limitations of the study, and have questioned the extent to which the results relate to perceptual restoration (Micheyl et al., 2003).

In primates, we recorded the responses of A1 neurons (Petkov et al., 2007) to stimuli that cause and/or can influence the continuity illusion in the animals (Petkov et al., 2003). By comparing responses to these stimuli we hoped to gain insight into whether cortical responses are consistent with perception. The main observation is that in awake naïve macaques, A1 neurons that respond stronger to tones than noise respond to the stimulus that causes the continuity illusion as if they are representing a continuous tone, but not as if they were representing a discontinuous tone (Petkov et al., 2007). This is consistent with the perception of continuity of the tone.

Motivated by the behavioral results (Petkov et al., 2003), we used both interrupting noise that elicits perceived continuity of the tone and surrounding noise that elicits only a perception of the noise and no audible tone. For the responses to the discontinuous tone with interrupting noise, we predicted that many neurons would respond consistent with the perception of both a continuous tone and the interrupting noise, which are the two sounds heard during auditory perceptual restoration of a tone. For surrounding noise most responses should be consistent with the masking percept (e.g., nearly all neurons should respond as if only the surrounding noise is present). We compared responses to perceptual continuity and masking stimuli with responses to several reference stimuli: (1) continuous and (2) discontinuous tones without noise, and (3) the noises by themselves (Fig. 3B,C). The predictions for individual neurons and for the population are simple. For the masking condition, the response should look more like the response to the noise than it does to any of the tone responses. For interrupting noise that causes the continuity illusion, two conditions should be met: (1) some neurons should represent the noise and some should represent a continuous tone, and (2) those neurons representing the tone should respond as if the tone were continuous (Fig. 3A, first column) and not as if it were discontinuous (Fig. 3A, second column).

A heuristic model of how the responses of A1 neurons that represent the tone would support the continuity illusion (Fig. 3A). For brevity we only show an exemplary sustained and phasic onset response, although other types of phasic responses (e.g., offset, onset and offset) can support continuity, see (Petkov et al., 2007). Matching the model predictions, A1 tone-preferring neurons responded to the perceptual continuity stimulus (Fig. 3B,C, lower right panel) as if a continuous tone were presented (Fig. 3B,C, upper left, also see arrow) and not as if an interrupting noise (Fig. 3B,C, upper right) or discontinuous tone were present (Fig. 3B,C, lower left).

Importantly, different neurons seem to support continuity in different ways. For the neuron in Fig. 3C that responded with brief phasic components, the correlate of continuity of the signal is a decrease in activity: the obliteration of the strong responses to the noise by itself and to the re-onset of the tone after the gap (see the highlighted red bins in Fig. 3C). Both of these are in the stimulus that creates ‘illusory continuity’. Increases in activity are also seen to correlate with continuity. This can be seen in a neuron with a more sustained response to the tone (Fig. 3B). Here, when a gap is introduced into the tonal signal the neuron decreases its response. It is worth noting that this decrease does not look like a mere cessation of activity during the lack of stimulus energy because the gap is 50 ms but the activity is suppressed or inhibited for over 100 ms. The noise by itself suppresses the response of this neuron (Fig. 3B, top right). Surprisingly when this inhibitory noise is added to the discontinuous tone, the activity is restored to the higher level associated with the presentation of only a continuous tone (without gap or noise). This is interesting because the response to both components (noise and gap) on their own was a reduction in activity, but when the two are put together the result is an increase in activity. This differs from Schreiner’s data from the MGB (Schreiner, 1980) where sustained response components did not support restoration. The cortical responses themselves also are quite different than those from the auditory nerve (see Petkov et al., 2007). Also, the dynamic component of auditory restoration adds insights that could not be observed in earlier work with responses to static visual images during visual restoration (Craft et al., 2007; Pessoa and De Weerd, 2003; Ramsden et al., 2001; von der Heydt et al., 1984).

We compared the responses to different stimulus components quantitatively. First we compared the response to the discontinuous tone combined with noise to the response to the tone alone and the response to the noise alone. If the spike counts in the objectively derived analysis window (e.g., the red response bins in Fig. 3B,C, see figure legend) were exactly the same in the combined stimulus as to the tone, the tone-noise similarity index (TNSI) would be +1. If the spike counts in the window were exactly the same in the combined stimulus as to the noise by itself, the TNSI would be −1. In Fig. 4C we can see that the TNSI values to the surrounding noise, which causes masking, were skewed towards negative values. This indicates that with surrounding noise most A1 neurons were signalling that only a noise was present, consistent with masking. When the stimulus with interrupting noise, capable of eliciting restoration, was presented the TNSI distribution was broader, suggesting A1 was signalling the presence of both a continuous tone and a noise. Yet, are the A1 neurons that represent the tone (those with positive TNSI values) signalling a continuous or discontinuous tone? We answered this by computing a tone-gap similarity index (TGSI) for these neurons. Here a value of +1 indicates that the response to the stimulus that causes the continuity illusion is the same as a response to a continuous tone; a value of −1 indicates the response to the stimulus that causes the continuity illusion was the same as a response to a discontinuous tone. The resulting distribution (Fig. 4B) is highly skewed towards positive values indicating that the neurons were encoding a continuous rather than a discontinuous tone. The sum of results is that for stimuli that cause the continuity illusion, A1 neurons send a signal that there is a continuous tone and a noise present (Petkov et al., 2007). This parallels the perception of auditory restoration (Petkov et al., 2003).

Fig. 4.

Population statistics of A1 neuron responses during perceptual continuity and masking. (A) Distributions of tone-noise-similarity-index (TNSI) – which compare whether the response to a discontinuous tone with noise is more like the responses to a complete tone by itself (positive values) or to the noise by itself (negative values). For the interrupting noise that elicits the continuity illusion the neuronal population in A1 represents both the noise (negative TNSI) and the illusory continuous tone (positive TNSI). (B) For neurons representing the tone (negative TNSI values from A) the tone-gap similarity index (TGSI) was calculated and plotted. Positive values are consistent with the neurons responding as if the tone were continuous; negative values are consistent with the neurons responding as if the tone were discontinuous. This plot shows that the neurons that represent tones respond as if the tone were continuous under conditions that elicit illusory continuity. (C) For surrounding noise that elicits masking the neuronal population in A1 responds as if only a noise is detected (negative TNSI). The TNSI values for interrupting noise for the example neurons in Fig. 3 are +0.60 for Fig. 3B and +0.63 for Fig. 3C.

That different response components – onset, offset and sustained –all consistently report a continuous tone in different ways highlights that a temporal code is used across the population of neurons. The time-dependent nature of this code suggests that at the level of A1, spike counts averaged over the entire stimulus cannot account for the percept, but rather the number of spikes, when they occur, and in which neurons are all important.

We tested whether the observed cortical responses were like auditory nerve (AN) responses and whether auditory nerve responses can also support the simple correlate of continuity revealed in A1. To do this, we applied all of our stimuli to an AN model (Patterson et al.,1995; Patterson et al.,1992; Slaney,1993), for details see, (Petkov et al., 2007). One may expect that the AN will respond with excitation to the noise, and that any activity to the combined noise/tone stimulus would be roughly additive, e.g., see (Borg et al., 1988; Feng et al.,1994). This also means that as the noise gets louder the response during the gap will become dominated by the noise (until perhaps saturation occurs), which would look more like masking than responses consistent with continuity or restoration. We applied the model to the stimuli (modeling all of the basic characteristics that were inherent in our A1 neuron sample, such as sample size and best-frequency responses) and found that nearly all of the simulated AN fibers represented only the noise (Fig. 5, bottom) and could not support continuity. Furthermore, the A1 TNSI distribution (Fig. 4A) clearly differed from the AN distribution.

Fig. 5.

The auditory nerve cannot support continuity. Results of model auditory nerve responses using the same stimuli as presented to the A1 neurons. On top are examples of the simulated results for a single neuron, on the bottom is the Tone-Noise Selectivity Index (TNSI) for the population of simulated neurons. Note that the TNSI distribution is highly skewed towards negative values, unlike the distribution for A1 responses in Fig. 4A. The modeled nerve fibers are exquisitely sensitive to spectral and temporal components in the gap and noise, being unable to represent continuity.

It is thus clear that continuity takes form after the AN (Petkov et al., 2007) and by the time primary auditory cortex is reached, responses are not simply excitation from the noise filling the gap, but reflect the more complex interplay of excitation and inhibition that has occurred, perhaps, by interacting with subcortical structures (Las et al., 2005).

At the level of A1 in awake, naïve, non-performing animals the segregation of the two perceived sound ‘objects’ – the interrupting noise and an illusory continuous tone – is incomplete. Since A1 needs to encode many different sounds and sound properties (e.g., Schreiner et al., 2000; Bizley et al., 2009) it makes sense that the representation is distributed among the population of A1 neurons where many neurons code more than one sound type or property. Evidence of this can be found in Fig. 4A where the TNSI distribution is not strongly bimodal. However it is possible that the segregation of responses to tones and noise becomes more complete in behaving animals (Petkov et al., 2003) who actively attend to the tone and/or that further down the cortical hierarchy there is a separation of units responding to noise and other types of sounds of interest (Rauschecker and Tian, 2000; Rauschecker et al., 1995).

It thus seems that the correlate to illusory continuity is unique to the central nervous system, which begins to take form at or beyond the cochlear nucleus. The next section addresses the questions: (1) what correlates of restoration are found in humans and (2) how does this relate to the animal data?

4. Insights from human EEG and fMRI: links from humans to other animals

Working with humans has the advantage that behaviorally humans can be tested much more flexibly. For example humans can readily report a percept and a confidence rating while physiological measurement is made. Also rather than studying small areas of the brain in detail as is usually done with animal electrophysiology, neuroimaging has the potential to show regional brain responses that may be overlooked in animal studies. Because of this it is important to keep in mind that the non-invasive techniques reflect more global aspects of brain processing and mechanisms than are described for the neuronal responses associated with auditory restoration in the animal work. Accordingly the human and animal work are complementary and when taken together provide considerable insight into how the brains of humans and animals participate in perceptual restoration.

In humans Micheyl et al. (2003) found a neurophysiological basis for auditory restoration: the EEG response elicited by illusory continuity stimuli was more like the response elicited by continuous tones than the response elicited by discontinuous tones. As a brain signature related to perceptual restoration/continuity, Micheyl and colleagues used the mismatch negativity (MMN) as a stimulus locked EEG component that can detect oddball or deviant stimuli within a sequence of standard stimuli. There were two types of standard sequences consisting of continuous or discontinuous tones, which alternated with band-passed noise. One of the deviant stimulus conditions used the combined discontinuous tone and noise that elicits illusory continuity (i.e., band-passed noise that temporally and spectrally overlaps the gap in the discontinuous tone). When the illusory continuity stimulus was introduced into a sequence of discontinuous tones alternating with the same type of noise, this resulted in a stronger MMN than when the standards were continuous tones, suggesting that the brain treated the illusory stimulus as if it contained a continuous tone. Because the MMN was temporally early (within 200 ms post tone-gap occurrence) and has been related to fairly automatic ‘pre-attentive’ processes (Naatanen and Alho, 1995; Picton et al., 2000), the authors concluded that A1 (around the human Heschl’s gyrus) supports continuity. In relation to animal single unit recording, the EEG results are consistent with A1 neurons encoding illusory continuity of tones well within a few hundred milliseconds of the restored gap.

Other groups have used EEG to provide additional insights into the brain activity and oscillations that relate to continuity and restoration. Riecke et al. (2009) used amplitude modulated tones and noise with spectral notches (notch noise). Some noises elicited continuity because the notch was small and provided energy close enough in frequency for tone restoration to occur. Other noise did not elicit continuity because the notch was large with little energy near the tone frequency and the subjects could clearly hear when the tone was discontinuous within the noise. The authors analyzed EEG responses to a stimulus that elicited a bistable percept: sometimes it was perceived as continuous and sometimes as discontinuous. The authors analyzed the 2-35 Hz EEG spectrum, noting that strong 3-7 Hz, approx. 5 Hz, mainly theta band oscillations were elicited in response to the interrupting noise within the discontinuous tone. This response was strong when the tone was reported as discontinuous but, critically, during continuity reports, the theta oscillations were significantly weaker (potentially suppressed). Using EEG source localization the effect was localized to around Heschl’s gyrus (potential A1, see Fig. 6). The observations by Riecke et al. (2009), regarding how the theta band oscillations related to auditory restoration are interesting because they show something that is not readily apparent with the single neuron spiking activity recordings in animals (Petkov et al., 2007; Sugita, 1997) but should be apparent in the local-field potential (LFP) recordings. A challenge for future work, given the diversity of neuronal responses supporting auditory continuity (e.g., both increases and decreases in firing rate, Petkov et al., 2007), is resolving the mechanism of how these would translate at the very large population level to theta (or other) oscillation frequencies as indications of neurons synchronizing the phase of their firing rates (Fries et al., 2001).

Fig. 6.

Neural correlates of auditory continuity in human auditory cortex. (A), fMRI group results rendered on an unfolded average cortex representation (adapted from Riecke et al., 2007). Left: Middle superior-temporal gyrus and Heschl’s gyrus (HG) in the right auditory cortex (AC) appear to play a role in the masking of gaps in tones and the hearing of continuity illusions of these tones respectively. Hemodynamic activity in these regions was related to the masking strength (dark blue) or the actual illusion strength (light green). Right: The illusion-related region (green outline) on HG (dotted line) was situated in a medial portion of a mirror-symmetric map of frequency sensitivity suggesting that the underlying primary auditory cortex processes are involved in perceptual restoration. (B) Distributed EEG source modeling group results obtained with the same behavioral paradigm rendered on a folded average cortex representation (adapted from Riecke et al., 2009). Left: Theta oscillations in central portions of the right AC appear to be relevant for hearing continuity illusions (see shading near cross). Right: Theta power in these regions (see cross on the left) was suppressed during and also around the masked gap interval when listeners reported continuity illusions compared to true discontinuity of the same ambiguous stimulus.

Independently of the work by Riecke and colleagues, another group (Vinnik, 2009) used EEG as subjects listened to combined discontinuous tone and band-passed noise stimuli which were perceived part of the time as being continuous and part of the time as discontinuous. Vinnik (2009) also notes the suppression of the theta oscillations associated with stronger perceptual continuity reported by Riecke et al., (2009). Interestingly, Vinnik suggests that this suppression is associated with less delta/theta power in the steady-state brain response preceding the stimulus judged as continuous. A critical link to the human studies can be achieved by pursuing the technically challenging objective of obtaining perceptual reports in animals experiencing bistable continuity perceptions during neuronal (including LFP response) recordings.

The temporal precision of EEG and the opportunity to understand the brain oscillations related to auditory restoration are very informative, but fMRI can better localize the brain regions involved in restoration. Several studies are now available that have used human fMRI during perceived continuity of tones (Riecke et al., 2007), speech like vowels (Heinrich et al., 2008) and speech (Shahin et al., 2009). The overall message from these studies is that the brain regions supporting auditory restoration are not fixed but depend on the nature of the signal being analyzed for restoration. There thus seem to be generic neuronal mechanisms throughout different brain regions that can support restoration regardless of the specific acoustical analyses at hand.

In their fMRI study, Riecke et al. (2007) used amplitude-modulated tones as the restoration signal and directly related fMRI brain activity to perceptual decisions of continuity or discontinuity when the stimulus itself did not change. The brain regions involved in perceiving the continuity of a tone seemed to be centered around the human Heschl’s gyrus (Fig. 6), which relate nicely to the animal recordings from A1 neurons which have also used tonal stimuli as the continuity/restoration signal. In combination with their EEG study the Riecke group interprets their results as indicative of A1 (at least the area around Heschl’s gyrus) in the right hemisphere as being involved in the perceptual restoration of a tone (Riecke et al., 2007, 2009). Serendipitously, our recordings were from the right hemispheres when evaluating A1’s neurons role in continuity (Petkov et al., 2007). However, lateralization effects should be explicitly tested in non-human species, such as by using monkey fMRI (Petkov et al., 2006, 2008) to compare to the human fMRI results (Riecke et al., 2007).

Other studies using more complex sounds suggest that regions outside of A1, in the superior-temporal sulcus (STS) of the temporal lobe are involved in restoration of vowel like sounds (Heinrich et al., 2008) or speech (Shahin et al., 2009). The prediction for animal fMRI is that the use of more complex sounds during restoration would as in humans, involve hierarchically higher stages and presumably different processing pathways (Rauschecker, 1998; Rauschecker and Tian, 2000). So far the discussion has primarily revolved around what areas are involved in illusory continuity, with a focus on what might be considered ‘bottom-up’ contributions (Petkov et al., 2007). While there is a large amount of evidence supporting bottom-up contributions (Bregman, 1990), a broader network is likely involved for more complex sounds (Heinrich et al., 2008; Shahin et al., 2009). This network likely utilizes both bottom-up and top-down mechanisms and an interplay between brain regions sensitive to multisensory influences (Bishop and Miller, 2009). An interesting unresolved question is whether the network for the restoration of speech would differ considerably from that for restoration of species-specific communication signals in other animals?

5. Neuronal principles for perceptual restoration

It seems unlikely that a single neuronal mechanism can support auditory restoration because (1) many different types of sounds need to be restored from many different potential occluders and (2) a diversity of neuronal response types and brain activity are related to restoration (e.g., theta band activity in EEG). Yet, Bregman (1990) predicted that neuronal responsiveness is maintained during continuity, which still resonates as a valid neuronal gestalt for perceptual restoration. Others have related auditory restoration to visual scotomas (blind spots in the visual field) that are filled in by a return to responsiveness of the neurons within the scotoma region (Pessoa and De Weerd, 2003). Moreover, examples from visual fill-in phenomena show sustained responsiveness during illusory fill-in. As examples, neurons respond to illusory contours in stationary visual stimuli as if the contour were present (Peterhans and von der Heydt, 1991; von der Heydt et al., 1984) or respond to occluded motion as if the moving stimulus had not been occluded (Assad and Maunsell, 1995).

Implicit or explicit in the past predictions and observations is that neuronal responsiveness is maintained by sustained neuronal responses. For illusory fill-in with static images or when recording from visual motion areas that encode motion with sustained neural discharge this is reasonable. However with auditory continuity there is a dynamic element of the task that lends itself inherently to temporal coding in the brain. The predictions based on sustained responsiveness do not take this into account. For this filling-in of a temporally interrupted sound we observe neurons with phasic responses that signal continuity by stopping firing, and others have observed decreases in the EEG theta band activity (i.e., the ~5 Hz noise onset response was decreased during continuity, see above section). Both are able to remove evidence of sound discontinuity which if detected is known to preclude perceptual restoration (Bregman and Dannenbring, 1977). Thereby, we can extend Bregman’s pioneering neuronal predictions given the new evidence as follows. A general neuronal principle is that if a neuron is linked to perceptual restoration its response to an occluded sound should be similar to the response to a continuous sound. Different types of responses (e.g., onset vs. sustained) might require different mechanisms to implement this principle. Top-down, ‘schema-based’ or other influences (Bishop and Miller, 2009) could further modulate responses to strengthen the effect using other mechanisms (Herrero et al., 2008).

It is important to emphasize the two key components to the predictions articulated here. First, that masking effects must be overcome such that more than the masker is detected. This cannot be achieved by all parts of the auditory system, as was revealed by our peripheral modeling of eighth nerve neurons which didn’t appear to be able to overcome the noise masking effects (Petkov et al., 2007). Second, the response supporting continuity must be like what would have been given to a continuous sound. We hope that considering neuronal principles for perceptual restoration in many sensory systems will encourage further neurobiological work in different species to better reveal the evolutionarily conserved brain networks that support perceptual restoration and their neuronal mechanisms.

6. A meeting of the human and non-human animal work and the path ahead

In conclusion, despite a long history of human behavior that has enhanced our understanding of perceptual restoration, understanding of the neurobiology underlying it has been slow to come about. Based on the use of natural response and psychophysical training paradigms, it is now clear that the perception of auditory restoration is far-reaching in animals, and that animal models can be used to probe the neurobiological mechanisms responsible for restoration. Single-neuron recordings in animals have revealed neuronal responses that can encode illusory continuity and restoration. Combining modeling with physiology uncovers that perceptual restoration results from central nervous system processing. It is likely that there are complex multiple mechanisms along the pathways from the cochlear nucleus up to and including auditory cortex. This review has allowed us to extend the putative coding principles articulated by Bregman (see above section) beyond the sustained firing of single neurons, to include temporal and population coding principles.

Human imaging studies have revealed brain oscillations, and the brain areas related to restoration. The mesoscopic perspective with the human imaging nicely complements the microscopic perspective obtained in the single neuron recordings. At the nexus, these two perspectives are starting to be tied together.

Exciting new work raises a new set of questions. If data are obtained with the same methods and analyses, general principles that apply across species will be easier to detect and interpret. For instance, to link human and animal behavioral work it was indispensible that at least some studies have used similar stimuli and psychophysical approaches to make direct comparisons between humans and other species. Along these lines conducting monkey or avian fMRI and EEG (some of which is now well feasible) with comparable stimuli and paradigms as those that have been used in humans would allow more precise determination of general properties and mechanisms across species. Moreover, in the animal models, it will then be possible to combine the fMRI and EEG results with those obtained from invasive neuronal recordings or to study the molecular mechanisms and the impact of genetic manipulations (Jarvis, 2004). In this way animal models with different advantages can be fully developed to understand the neuronal responses at a level of detail unavailable for study of the human brain alone.

Auditory restoration, as an example of a complex listening problem, has become an important paradigm. It was argued at a recent Auditory Cortex meeting in Magdeburg, Germany (2009) that how the auditory system analyzes complex auditory scenes and segregates sounds is of fundamental importance for advancing auditory neuroscience and advancing treatments for a host of hearing disorders. We propose that the work on auditory perceptual restoration has set the stage for understanding certain aspects of auditory scene analysis and promising animal models now seem available. Using techniques that can establish direct links between human and animal models would build on the current behavioral work across the species and the budding neuroscientific detail on restoration, which is a clear path to follow. Making this link between humans and animals has tremendous potential, but also requires several heretofore culturally distinct scientific fields within the auditory community to forge a cohesive community working to resolve a fundamental issue in neuroscience.

Abbreviations

EEG

electroencephalography

fMRI

functional magnetic-resonance imaging

FM

frequency modulated

AM

amplitude modulated

MGB

medial geniculate body

A1

primary auditory cortex, field A1

TNSI

tone-noise similarity index

TGSI

tone-gap similarity index

AN

auditory nerve

MMN

mismatch negativity

LFP

local-field potential

HG

Heschl’s gyrus

AC

auditory cortex