Neural mechanisms for illusory filling-in of degraded speech

Abstract

The brain uses context and prior knowledge to repair degraded sensory inputs and improve perception. For example, listeners hear speech continuing uninterrupted through brief noises, even if the speech signal is artificially removed from the noisy epochs. In a functional MRI study, we show that this temporal filling-in process is based on two dissociable neural mechanisms: the subjective experience of illusory continuity, and the sensory repair mechanisms that support it. Areas mediating illusory continuity include the left posterior angular gyrus (AG) and superior temporal sulcus (STS) and the right STS. Unconscious sensory repair occurs in Broca’s area, bilateral anterior insula, and pre-supplementary motor area. The left AG/STS and all the repair regions show evidence for word-level template matching and communicate more when fewer acoustic cues are available. These results support a two-path process where the brain creates coherent perceptual objects by applying prior knowledge and filling-in corrupted sensory information.

INTRODUCTION

Despite interference from background noises, listeners usually perceive speech to be continuous through interruptions. Even if brief segments of the speech signal are artificially removed and replaced by noise, listeners still hear uninterrupted speech provided the noise continues through the missing speech segment. This phenomenon is known as the “continuity illusion”, “phonemic restoration”, or “auditory induction”, and helps maintain robust comprehension in adverse environments (Samuel, 1981; Warren, 1970). The continuity illusion is not unique to speech (Bregman et al., 1999; Micheyl et al., 2003; Petkov et al., 2003, 2007; Riecke et al., 2007) or to humans, as it is also found in cats (Sugita, 1997), non-human primates (Miller et al., 2001; Petkov et al., 2003, 2007), and birds (Braaten RF, 1999), suggesting that its mechanisms are general and conserved across species. Illusory continuity also occurs in other modalities, as illustrated by analogous filling-in phenomena in vision (De Weerd, 2006; Komatsu, 2006; Meng et al., 2005; Ramachandran and Gregory, 1991). For instance, in visual perceptual filling-in, the visual cortex “surface interpolates” missing features, such as color or texture, across a scotoma or blind spot. The auditory continuity illusion may reflect a similar process in which the brain interpolates features of missing speech across frequency and time.

The neural mechanisms mediating the continuity illusion are not well understood. Functional neuroimaging, neurophysiology, and neural network modeling point to low-level cortical processing (Husain et al., 2005; Micheyl et al., 2003; Petkov et al., 2003, 2007; Riecke et al., 2007; Sugita, 1997). For instance, an EEG study using a mismatch negativity design (Micheyl et al., 2003) showed that the continuity illusion can be fulfilled in the absence of focused attention and may occur within 200 ms after the interruption. Also, a macaque study by Petkov et al. (2007) revealed that the continuity illusion is reflected physiologically at the level of primary auditory cortex (A1). They showed that if part of a pure tone is replaced by noise, some A1 neurons continue to fire as if the tone were continuous. Finally, in an fMRI study Riecke et al. (2007) reported that the continuity illusion is specific to a region confined to the middle part of Heschl’s gyrus. While these reports agree that low-level neural mechanisms are involved in illusory continuity, all used simple stimuli (tones) and were only able to localize activity in or near the auditory cortex (Petkov et al., 2007; Riecke et al., 2007). In contrast, one recent whole-brain fMRI study (Heinrich et al., 2008) investigated illusory vowels resulting from perceptual continuity. Though the experiment was designed to characterize the neural bases of vowel perception rather than the mechanisms of illusory continuity, it nonetheless suggested a role for middle temporal gyrus in the illusion. This implies that higher-level regions may be recruited for filling-in spectrotemporally realistic sounds such as speech.

Based on electrophysiological and psychophysical studies, we propose that the continuity illusion relies on at least two dissociable neural pathways (Carlyon et al., 2004; Lyzenga et al., 2005; Repp, 1992; Samuel, 1981; Sivonen et al., 2006): 1) sensory repair, a largely unconscious process, and 2) subjective continuity, or the illusion itself. Repair would entail reconstructing low-level sensory representations where “bottom-up” information is degraded or missing (Petkov et al., 2007). This might include characterizing the errors as well as restoring them, and should recruit brain regions for controlled acoustic sequencing and pattern recognition, such as left inferior frontal gyrus (Burton et al., 2000; Zaehle et al., 2008; Zatorre et al., 1992). Subjective continuity would instead test the overall coherence or plausibility of a stimulus against prior knowledge, which may take several forms, most notably explicit word representations or templates. The role of template-matching to achieve the illusion is supported by evidence showing that words are more readily restored than phonotactically legal pseudowords, and the number of syllables in a word is directly proportional to the strength of the perceived continuity (Samuel, 1981). Subjective continuity would thus depend on a hypothesis test: whether “bottom-up” sensory information matches prior “top-down” expectations. This may recruit regions known to be important for lexical processing, as in the posterior temporal lobe (Binder et al., 2003; Horwitz et al., 1998; Neville et al., 1998; Obleser et al., 2007; Owen et al., 2005).

We identified the neural substrates of the repair and subjective continuity networks of the illusion in a rapid event-related fMRI design. Subjects with normal hearing listened to words with brief noise bursts of varying duration centered about fricatives or affricates. The white noise segment either replaced or was superimposed on the speech sound. Subjects indicated whether they perceived each word as being continuous or interrupted. We evaluated the BOLD (blood oxygenation level dependent) signal based on whether subjects experienced the illusion (interrupted stimuli but perceived continuous, termed illusion), correctly perceived interrupted speech (illusion-failure), or correctly perceived continuous speech (physically and perceptually continuous stimuli, termed natural). This design explicitly dissociates stimulus features from perception, allowing us to identify the repair and continuity pathways independently. Finally, within the repair-continuity network, we compared BOLD responses between words and pseudowords to identify brain areas that apply prior knowledge through high-level template-matching.

MATERIALS AND METHODS

Participants

Twenty-two normal-hearing and right handed subjects were pre-screened on a psychophysical test to determine their suitability for the fMRI experiment. The final pool of subjects participating in the fMRI session consisted of 15 people (Mean age 27, age range 20–48; 10 females). All subjects were native English speakers and had no history of neurological disorders. Participants gave written informed consent in accordance with procedures approved by the University of California and were paid for their participation.

Stimuli

The experimental design consisted of two tasks: a primary task using real words, and a control task using pseudowords. The primary task used 230 English words and the control task used modified versions of 104 words from the primary task, rendered meaningless. The actual words were compiled using the University of Western Australia MRC Psycholinguistic Database (http://www.psy.uwa.edu.au/mrcdatabase/uwa_mrc.htm) and consisted of tri-syllabic English nouns and adjectives that contained between 1 and 3 fricatives/affricates—at least one occurring between the first and last phonemes—with a familiarity rating of 300 – 700. Fricatives/affricates were used as they are well-studied and yield robust phonemic restoration (Samuel, 1981). The words were spoken by a professionally trained female vocalist (fundamental frequency of 203 Hz) and recorded using a Shure KSM studio microphone (http://www.shure.com) at a sampling rate of 48000 Hz, in an acoustically shielded room. The first two adjacent phonemes of each of the 104 words used for the control task were replaced by two different phonemes of the same general class (vowel/consonant) to ensure the transformed pseudowords had similar acoustical properties but were rendered meaningless. For example “diligent” was changed to “saligent”. Transparent near-words were disqualified. These phonotactically legal pseudowords were spoken by the same vocalist under identical recording conditions.

One fricative/affricate ([tʃ], [dʒ], [s], [ʃ], or [z]) present between the first and last phoneme of each word represented the location for the white noise placement. On average the replaced fricative/affricate represented the 5^th phoneme of the word/pseudoword (word: 5.3 ± 1.3; pseudowords: 5.2 ± 1.4). The beginning and end latencies of each fricative or affricate for each word were determined using Adobe Audition 2.0 (http://www.adobe.com/products/audition). The original words were modified into “interrupted” and “continuous” versions. Interrupted words were those with part of the word (centered on the fricative/affricate) completely replaced by white noise. Continuous words were those in which white noise was superimposed (centered on the fricative/affricate). All words were identifiable and unambiguous, even when interrupted, thus minimizing semantic effects. Since ruling out effects due to physical differences of the acoustical stimuli constituted a major goal of the experimental design, we normalized the physical attributes of the stimuli in different conditions, by first matching the loudness of all stimuli based on their A-weighted RMS. Second, the A-weighted loudness of the white noise segment was equalized to the A-weighted RMS of the replaced/superimposed speech plus 3 dB. Since the noise is uncorrelated with the fricative/affricate, the RMS difference between the continuous and interrupted words was negligible—less than 1 dB for all stimuli. Figure 1 shows the time waveforms and spectrograms for an actual word (“aggressor”) with part of the fricative (/ss/) replaced with white noise (middle panel) and with white noise superimposed (right panel). For comparison, the left panel shows the original spectrotemporal representation of the word “aggressor”.

Fig.1. Stimuli.

The time waveforms (top) and corresponding spectrograms (bottom) for an example word (”aggressor”). The middle panel depicts a physically interrupted word, where white noise replaced part of the fricative/ss/. The right panel depicts a physically continuous word, where white noise was superimposed on the fricative. The left panel depicts the original word with no noise, which was not among the experimental stimuli.

It should be noted that the words and pseudowords also differed in their biphone frequency for the initial two phonemes (words = 0.0084, pseudowords = 0.0024; p < 0.001; Vitevitch and Luce, 2004). In principle, initial phoneme frequency could affect fMRI signal. However, if fMRI differences between words and pseudowords were due to biphone frequency then one would expect these differences to be equally exhibited across the three conditions (natural, illusion, illusion-failure). As will be reported in the ANOVAs in Figure 5, this is generally not the case. Moreover, interruptions occurred on average more than two phonemes after the initial biphone, thereby minimizing the influence of the utterance onset. Furthermore, words and pseudowords were perfectly matched after the first biphone. Thus, the interruptions occurred in a local phonetic context with monophone, biphone, and triphone frequencies identical between the words and pseudowords.

Fig. 5. Word-level template-matching in illusory filling-in.

The results of the ANOVAs (3 conditions × 2 tasks) contrasting the BOLD signal intensity for the repair (A) and continuity (B) regions (depicted in figure 4). The p values depict the interaction probability. A significant interaction means that repair or continuity mechanisms distinguish between words and pseudowords. This implies reliance on word-level template matching for filling-in missing sensory information. Regions with significant interactions are denoted with asterisks. Two-tailed standard errors are depicted by the error bars. (Region abbreviations: AG= angular gyrus; STS = superior temporal sulcus; IFG = inferior frontal gyrus; pre-SMA = pre-supplementary motor area; SFS = superior frontal sulcus).

Procedure

Pre-scan

Participants performed a psychophysical test to assess performance 1–2 weeks prior to functional scans. In this test, subjects were presented with the 230 words consecutively and judged whether they perceived each word as continuous or interrupted. The 230 words were presented pseudorandomly every 3 seconds with 50 % of the words physically interrupted and 50 % physically continuous. A two alternative forced choice adaptive procedure was used to match the number of the illusory percepts (illusion) and accurately recognized interrupted words (illusion-failure). At the start of the experiment, a white noise segment spanning a proportion of 100 % of the fricative/affricate duration was used to either replace the fricative/affricate (interrupted word) or add to it (continuous word). If, during the interrupted trials only, subjects identified the word as continuous, the proportion of white noise segment was increased by 15 % of the actual fricative/affricate (7.5 % on either side of the fricative center), but was decreased by the same amount if they identified the word as interrupted. The lengths of white noise segments between different words varied with respect to the proportion of the replaced fricative/affricate rather than a fixed duration of white noise. This allowed evaluation of illusory continuity based on representations contained in speech rather than information contained in fixed temporal values (Bashford et al., 1988). The two alternative forced choice task also served to match the difficulty and the energy of the spectrotemporal information present in the stimuli of the two conditions, by using white noise segments with lengths near the subject’s own illusory threshold. For the physically continuous trials, the white noise proportion on a given trial was the same as the previous trial regardless of subject perception. This equalized the noise proportion distribution between physically continuous and physically interrupted trials. If the physically interrupted word was identified as interrupted the trial was labeled an illusion-failure; if the interrupted word was identified as continuous the trial was labeled an illusion; if the continuous word was identified as continuous, the trial was labeled natural, and finally, if the continuous word was identified as interrupted, the trial was labeled a miss.

By design, all subjects (n = 22) had roughly equal numbers of illusion and illusion-failure but only subjects (n = 21) who had a ratio of natural to miss of 2 or larger were considered to be good responders. This ratio served as an indicator of task performance or discriminability—whether participants were actually doing the task or were guessing on many trials. Out of the 21 good responders, the 15 that could be scheduled first were invited back for the functional scan. The mean and standard deviation of the number of responses out of 230 for each condition across the 15 subjects were as follows (mean ± SD): illusion = 56.5 ± 1.8; illusion-failure = 58 ± 1.55; natural = 100 ± 10.4; Miss = 15.5 ± 10.5. Word presentation and response logging was done in Presentation software (Neurobehavioral Systems, http://www.neurobs.com) interfaced with MATLAB (MathWorks, Natick, MA).

fMRI session

Subjects (n=15), who met the pre-scan test criterion, participated in the fMRI scanning experiment which included six functional EPI pulse sequence sessions lasting ~8.5 minutes each. T2*-weighted EPI images sensitive to BOLD activity were acquired using a 3-Tesla Siemens Trio MRI scanner and an 8-channel head coil with a one-shot EPI sequence (TR 2.1 s; 29 ms echo time; 64×64×34 acquisition matrix, 3.0 mm slice thickness with a 0.42 mm gap between slices, and a 220 mm field of view; bandwidth, 2365 Hz/pixel; flip angle, 84°). Subjects were fitted with E.A.R. TaperFit2 earplugs to attenuate scanner noise. A fixation cross was projected onto a screen at the subject’s feet, viewed through a mirror mounted to the head coil, and remained throughout the duration of the session. Speech stimuli were delivered using an amplifier with MR compatible headphones (mr-confon.com) at a comfortable volume.

The six sessions were randomized across subjects but the presentation order of stimuli and silent interstimulus intervals (ISIs) within each session were the same for all subjects. Each of the six sessions consisted of 148 word stimuli and 93 null trials where no stimulus was presented. The null trials were added to introduce a temporal jitter between stimuli. Each session contained an initial, mid and a final sequence of 10 null TR fixation periods. For the remainder of each session, the order of presentation of words/pseudowords and silent gaps was dictated by a genetic algorithm (Wager and Nichols, 2003) that optimized the contrasts between conditions. Separate genetic algorithm models were used for each session. For five of the sessions, real English words were presented to the subjects and in one control session, randomly ordered, meaningless pseudowords were presented.

Task

While fixating on a cross displayed on a screen, subjects pressed their left index finger when they perceived the stimulus as continuous and their left middle finger when they perceived the stimulus as interrupted. Similar to the pre-scan task, the proportion of the white noise segment was adapted to yield roughly equal numbers of illusion and illusion-failure responses. Based on the pre-scan behavioral measure, in which equal numbers of physically interrupted and physically continuous stimuli were presented, there was a large disparity in the number of physically but perceptually continuous (natural) compared to illusion (physically interrupted but perceptually continuous) or illusion-failure (physically and perceptually interrupted) responses. To minimize this difference and maximize the statistical power of the contrasts, in each of the sessions 58 of the words (40 %) were designated as physically continuous and 90 words (60 %) as physically interrupted. Responses and white noise proportion were monitored and logged via Presentation (Neurobehavioral Systems) software.

Data Analysis

Behavior

Behavioral differences for reaction time or noise proportion (means for each subject) were tested using a 2×3 ANOVA (Statistica v. 6.0, StatSoft, OK) with the first factor being task (words/pseudowords) and the second factor being condition (illusion-failure, illusion, natural).

fMRI pre-processing

fMRI data were processed online with Siemens realignment and distortion correction algorithms and subsequently converted to the Analyze format from DICOM 3.0 using XMedCon. Analyze images were then corrected for slice acquisition time, spatially realigned, and smoothed with an 8 mm³ full-width at half-maximum (FWHM) kernel in SPM2. Within-session linear trends were removed in MATLAB by calculating the mean global signal level over all brain voxels for each image, fitting a line to each session’s mean global estimates, and then dividing these values by the linear fit. T2, EPI, and high resolution MPRAGE anatomical images were co-registered and normalized to an MNI template, and resampled at 2 mm isotropic voxels. Importantly, each subject’s high resolution anatomical image was normalized to a modified MNI template (skull-stripped and faded near the ventral cerebellum using MATLAB and MRIcro) to yield robust normalizations despite signal loss near the ventral cerebellum due to a short radio frequency coil. Except where noted, all statistical results are from random-effects, group tests corrected for multiple comparisons (FDR).

fMRI modeling

Speech identification data were analyzed using a modified general-linear model (GLM) in SPM2 at the subject level. The onset trains (delta functions) for natural, illusion-failure, illusion, and miss trials for both the word/pseudoword conditions were convolved with a hemodynamic response function (HRF) and included in the design matrix. Motion confounds, mean centered within each session, and intersession effects were also included as regressors. Data were pre-whitened to account for temporal auto correlation and allow for valid parametric statistical tests. Contrast images were included in a second order group T-test in SPM. In the main word task, only functional activity at a FDR corrected p-values of 0.05 or better are reported.

Regions of interest analyses (ROIs)

To enhance the signal power of the control task (one session compared to five in the main task), the contrasts between tasks were limited to regions of interest (ROIs). ROIs were determined from the group T-test functional maps of the repair (illusion > natural) and illusion (illusion > illusion-failure) contrasts of the main word task, thresholded at FDR p < 0.05. These ROI masks were applied as spatial filters onto the original illusion, illusion-failure and natural functional maps (conditions contrasted to baseline) of the two tasks. Activity estimates or “betas” within each ROI for each condition were extracted and averaged. The mean betas were contrasted using a 2×3 ANOVA (Statistica 6.0, StatSoft, OK) with variables being task (words/pseudowords) and condition. Post hoc contrasts were made by the Least Square Difference test (LSD test).

Heschl’s gyrus

Activity at Heschl’s gyrus was examined in two ways: 1- A bilateral Heschl’s gyrus mask anatomically defined in MarsBaR (a toolbox of SPM, (Tzourio-Mazoyer et al., 2002)) was created for each subject. Then beta estimates within the Heschl’s gyrus mask for each condition were extracted and averaged. This allowed us to examine illustratively how different conditions affected the entire Heschl’s gyrus. 2-The voxels in Heschl’s gyrus which showed differences between conditions using a whole brain T-test corrected at an FDR p < 0.05 were identified, averaged and plotted for comparison with the results of the entire Heschl’s gyrus. Differences in Heschl’s gyrus were only observed for the illusion-failure > illusion and illusion-failure > natural contrasts and occurred in the middle third of the right Heschl’s gyrus.

Functional connectivity

We used the beta correlation method of Rissman et al. (Rissman et al., 2004) to assess functional connectivity. A second, massive GLM was performed that modeled individual trial beta estimates. In this GLM, each trial of natural, illusion-failure, illusion, and miss for the word task was modeled as a separate covariate in the design matrix. Each covariate was convolved with a single HRF. Motion parameters were included as regressors. Because of computational limits, each session was estimated independently, thus no intersession effects were included in the GLM. The individual beta estimates for all trials in the main experiment (words) were sorted by condition and used to construct connectivity maps (Rissman et al., 2004). A seed or reference region was defined for each subject by finding the 10 most significant contiguous voxels within a functionally defined region of interest (ROI), based on functional maps of the group T-test for the contrast (illusion > illusion-failure). The seed region chosen for this analysis was the left AG/STS, since it reflected most strongly the subjective sense of continuity and was the only continuity region which exhibited lexical specificity. Betas in these 10 voxels for each trial were averaged and combined to create an average “beta series” for each condition representing the reference region (i.e. reference-beta-series). The values of the reference-beta-series were then correlated with the corresponding values of beta-series of every voxel in the functionally defined repair regions (illusion > natural). The Pearson’s correlation coefficients obtained for the correlation analysis for each condition were converted into z-scores by calculating the hyperbolic arctangent of each coefficient and dividing it by the known standard deviation of the hyperbolic arctangent function (1/sqrt(N-3) where N is the number of time points used in the correlation). These normalized correlations were contrasted across conditions and used to create whole brain “contrast” maps. The number of time points (betas) for the natural contrast was equated to the number of time points of the other contrasts by random resampling. These contrast maps were subjected to a group T-test to determine statistical significance. Although the beta correlation method would be unreliable for designs with very short mean ISI (~2 s), it has been validated in event related designs with ISI as short as 4 s (Rissman et al., 2004) and should provide unbiased results given our mean ISI (3.3s) and adequate counterbalancing of trials across conditions.

RESULTS

Behavior

Noise proportion

For acoustically interrupted stimuli, noise bursts replacing a small proportion of the word tended to induce illusory continuity (illusion), and those replacing a large proportion were correctly identified as interrupted (illusion-failure). A key goal of our experimental design was to attribute brain activity to perception while controlling for acoustical differences between conditions. For instance, if conditions illusion and illusion-failure were distinguished not only by percept but also by large differences in noise proportion, we would be unable to claim that fMRI effects are due to perception alone. We thus aimed to equate as closely as possible the noise proportions across different conditions. To achieve this, every trial’s noise proportion with respect to the fricative duration was adaptively roved near each subject’s threshold for illusory continuity, while the subject performed in the scanner (see Materials and Methods). Figure 2 shows the number of trials as a function of noise proportion for illusion-failure, illusion, and natural conditions for the word task. The distributions of white noise proportions among the three conditions overlapped completely, and the maximum mean difference between conditions was only 10.5 % of the mean noise proportion of all conditions, corresponding to approximately 19 ms. Such a small difference between conditions rules out effects based on the physical properties of speech. Furthermore, an ANOVA did not reveal significant differences between white noise proportions of words and pseudowords (F < 1.3) but significant differences occurred among the three conditions (F_{(2, 28)} = 650, p < 0.001; illusion-failure > illusion > natural), despite the overlapping distributions.

Fig. 2. Stimulus properties equalized across conditions.

Number of natural, illusion, and illusion-failure responses according to white noise burst duration. Duration is relative to the distorted speech sound, quantified as the proportion of the entire phoneme interrupted by (illusion and illusion-failure) or added to (natural) white noise. On the x-axis, the white noise proportion for each of the conditions is normalized to the average white noise proportion of the three conditions (142% of the fricative).

Response count

Table 1 shows the behavioral response counts for illusion-failure, illusion, natural, and miss (physically continuous but perceived interrupted) conditions for the word task. Due to the adaptive design, the number of illusion and illusion-failure trials was nearly equal. The natural response count was larger than the miss response count indicating that subjects performed consistently and did not guess. Similar ratios were obtained for the pseudoword task.

Table 1. Response count.

Behavioral response counts for illusion-failure, illusion, natural, and miss (physically continuous but perceived interrupted) conditions for the word task.

	Stimulus Interrupted	Stimulus Continuous
Perceived Interrupted	illusion-failure (195.6 ± 10)	miss (31.2 ± 30)
Perceived Continuous	illusion (200 ± 8.3)	natural (296.3 ± 33.2)

Reaction time

Table 2 shows the reaction time and standard deviation for the illusion, illusion-failure and natural conditions, for the word and pseudoword tasks. There were no differences between tasks (ANOVA F < 1). The reaction time differences between conditions, except for pseudoword illusion-failure, were very small (< 21 ms) and did not differ significantly. An ANOVA revealed a main effect (F_{(2, 28)} = 4.3, p < 0.03) of condition. An interaction between the variables (F_{(2, 28)} = 11.0, p < 0.001) showed that this difference was only due to significant deviation occurring for the illusion-failure condition of the pseudoword task compared to the other conditions (Table 2).

Table 2. Reaction time.

Reaction time and standard deviation for the illusion, illusion-failure and natural conditions, for the word and pseudoword tasks. There were no differences between tasks (ANOVA F < 1).

	Words			Pseudowords
	illusion-failure	illusion	natural	illusion-failure	illusion	natural
MEAN ± SD (ms)	1180 ± 45	1194 ± 43	1173 ± 47	1230 ± 66	1183 ± 49	1184 ± 32

Model

We developed the model in Figure 3 to convey our hypothesis that repair and subjective continuity are neurally dissociable processes. The two possible stimuli (continuous and interrupted) are shown at the bottom, and the two possible percepts (continuous and interrupted) are shown at the top. Low-level spectrotemporal processing occurs in the early auditory cortices. This spectrotemporal representation is conveyed to the continuity network where its initial plausibility or internal coherence is evaluated according to prior expectations (e.g. the lexical or contextual properties of the words). If the initial spectrotemporal representation is a plausible percept—as when the stimuli are physically continuous—the subject will identify the stimulus as continuous (natural). In contrast, when the initial spectrotemporal representation reflects an implausible structure, as when the stimuli are physically interrupted, the internal coherence is poor and the repair mechanism is recruited. Repair entails identifying errors in and reconstructing the low-level representation, even without a subject’s awareness. The repair outcome is then communicated back to the continuity network and low-level sensory regions. If the repair then succeeds and the resulting sensory representation is rendered plausible or coherent, subjects will identify the stimulus as continuous (illusion). If instead a stimulus is inadequately repaired, it will be perceived as interrupted (illusion-failure). In this model, areas mediating subjective continuity are identified by contrasting illusion and illusion-failure conditions (same stimuli, different percept). Similarly, areas mediating unconscious repair are identified by contrasting illusion and natural conditions (same percept, different stimuli). Finally, since the illusion is normally dependent on successful repair, we also hypothesized that the continuity and repair regions would communicate more strongly for successfully repaired percepts than for trials when there is no need to repair.

Fig. 3. Hypothesis.

A schematic depicting the predicted processing paths mediating illusory filling-in of speech. Our design explicitly tests whether the continuity and repair pathways are neurally dissociable. Contrasting BOLD activity for natural > illusion identifies the repair network, and contrasting illusion > illusion-failure identifies the continuity network. See text for full description.

BOLD

Repair (illusion > natural)

To achieve a continuous percept, repair regions should be recruited when relatively less bottom-up sensory information is available. That is, for the same perceptual outcome, greater activity indicates the need for more repair. Figure 4A depicts group functional maps for the repair network (illusion > natural). The repair mechanism elicited greater activity in the left inferior frontal gyrus (Left IFG/Broca’s region; pars opercularis, Brodmann area 44), pre-supplementary motor area (pre-SMA), and bilateral insula.

Fig. 4. Dissociable networks for repair and continuity.

A. Functional maps in axial view of the repair network contrast (illusion > natural), superimposed on the normalized MNI brain of one subject. The repair regions included bilateral insula (with MNI center of activity at x = −34 y = 20 z = 0, p_(FDR) = 0.033 for the left hemisphere and x = 38 y = 22 z = 0, p_(FDR) = 0.007 for the right hemisphere), left IFG (Pars opercularis, Brodmann area 44; x = −52 y = 16 z = 4, p_(FDR) = 0.033) and Pre-SMA (x = −6 y = 18 z = 58, p_(FDR) = 0.032). B. Functional maps in axial view of the continuity network contrast (illusion > illusion-failure) superimposed on the normalized MNI brain of one subject. The continuity regions included left AG/STS (x = −42 y = −82 z = 34, p_(FDR) = 0.042), Precuneus (x = −14 y = −52 z = 16, p_(FDR) = 0.042), right STS (x = 50 y = −56 z = 24, p_(FDR) = 0.042), and bilateral SFS (left: x = −22 y = 28 z = 46, p_(FDR) = 0.042; right: x = 28 y = 32 z = 56, p_(FDR) = 0.042). (Region abbreviations: AG= angular gyrus; STS = superior temporal sulcus; IFG = inferior frontal gyrus; SFS = superior frontal sulcus; pre-SMA = pre-supplementary motor area).

Continuity (illusion > illusion-failure)

For interrupted stimuli, continuity network regions should reflect greater activity during the illusion than when the illusion fails. Notice that if our hypothesis about neurally dissociable processes is false, this comparison would yield the same regions as the repair network. Figure 4B depicts functional maps for the continuity network (illusion > illusion-failure) of the word task. The illusion elicited activity in the left posterior angular gyrus (AG) and superior temporal sulcus (STS), right STS, bilateral superior frontal sulcus (SFS), and precuneus. No brain areas are common to both repair and perceived continuity.

Template-matching

To examine whether the continuity-repair network applies prior knowledge through word-level template matching, the word and pseudoword tasks were compared for the three conditions by ANOVAs. Greater differences in activity between conditions for words than pseudowords indicates lexical template matching while activity common to both tasks (words and pseudowords) favors prior knowledge at a lower representational level.

Figure 5A shows the ANOVAs (2 task × 3 conditions) in the repair regions. The left IFG and insulae displayed similar behavior, namely enhanced activity with increased word degradation, indicative of word-level template matching, while pre-SMA showed only a trend in the same direction. In particular, in left IFG and insulae, ANOVAs revealed no main effect of task but a main effect of condition (left IFG: F_(2,28) = 9.5, p < 0.001; Insulae: F_(2,28) = 39.4, p < 0.0001) and, importantly, an interaction between the variables (left IFG: F_(2,28) = 8.2, p < 0.002; Insulae: F_(2,28) = 14.0, p < 0.0001). Post hoc analysis revealed that the interaction was mainly attributed to enhanced BOLD for the condition order illusion-failure > illusion > natural in the word but not in the pseudoword task. Finally, for the pre-SMA an ANOVA showed a main effect for task (F_(1,14) = 4.6, p < 0.05) and a main effect of condition (F_(2,28) = 29.6, p < 0.0001) with no interaction between variables. The main effect of task was due to enhanced BOLD for the word compared to the pseudoword task and the condition main effect was attributed to enhanced BOLD for the condition order illusion-failure > illusion > natural. Thus the repair regions show strong evidence for word-level template matching because they work harder to repair speech when it is a word rather than a pseudoword. The pre-SMA preference for words over pseudowords across the three conditions also implies that word-level template matching is enhanced regardless of degradation.

Unlike in the repair regions, BOLD values in the continuity regions were negative across all conditions. Figure 5B shows the ANOVAs in the continuity regions. None of the ANOVAs revealed main effects between tasks. However, condition main effects (p < 0.001 or better) occurred for all regions, attributed to enhanced BOLD activity for the natural condition followed by illusion and then illusion-failure. Only the contrast for left AG/STS revealed an interaction between task and condition (F_{(2, 28)} = 4.8, p < 0.02). Post hoc analysis revealed that the interaction was attributed to enhanced BOLD for the condition order natural > illusion > illusion-failure in the word but not the pseudoword task. Thus, among all continuity regions, only left AG/STS showed evidence for applying prior knowledge through word-level template matching.

Heschl’s gyrus

Heschl’s gyrus (A1 and other early auditory cortex) was not identified as a member of our repair-continuity network. We aimed to compare our results to prior work showing that large portions of Heschl’s gyrus are sensitive to acoustical attributes of stimuli, while only the middle part of the gyrus shows sensitivity to perceived continuity (Riecke et al., 2007). Figure 6 shows the mean activity for the three conditions in the entire right Heschl’s gyrus (left bar plot) and its middle portion (right bar plot) (See Methods), which showed functional differences between conditions based on brain-wide t-tests (FDR corrected at p = 0.05). It is notable that the middle region of Heschl’s gyrus exhibited larger mean BOLD intensity compared to the mean across the entire gyrus for all conditions. There were no significant differences between the illusion and natural conditions in this middle portion of the right Heschl’s gyrus. The only differences in this region occurred between the illusion-failure compared to illusion or natural responses. For activity estimated from the average of the entire gyrus, left or right, there were no significant differences between conditions (based on t-tests, p > 0.1), suggesting that most of early auditory cortex treated stimuli across conditions as equivalent, while the right middle portion was sensitive to perceptual rather than physical changes.

Fig. 6. Perceptual sensitivity in early auditory cortex.

Top. Functional map depicting activity in the entire right Heschl’s gyrus (orange) and middle right Heschl’s gyrus (dark brown). Bottom left. Average BOLD intensity (compared to baseline) across the entire right Heschl’s gyrus for the three conditions of the word task. Significance was examined using T-test analysis of the average BOLD intensity between conditions. Bars above columns depict standard deviation. Bottom right. Average BOLD intensity (compared to baseline) in middle right Heschl’s gyrus for the three conditions of the word task. Contrasts with asterisks indicate significant differences for all voxels in the region, based on brain-wide t-test corrected at p (FDR) < 0.05. The middle right Heschl’s gyrus shows significant differences between the illusion-failure and the other two conditions, but not between the illusion and natural conditions. Activity at the entire left Heschl’s gyrus (not shown) were also analyzed and yielded similar results as for the entire right Heschl’s gyrus.

Functional connectivity

According to our hypothesis, repair and continuity regions should communicate more when repair is required for perceived continuity than when there is no need for repair. We used a functional connectivity analysis that correlates trial-by-trial estimates of activity between regions and contrasts these correlations across conditions (Rissman et al., 2004). The first step in the analysis involved identifying a reference or “seed” region. We chose left AG/STS since it reflected most strongly the subjective sense of continuity (region defined as illusion>illusion-failure, FDR p<0.05). We then tested whether the correlation between left AG/STS and the repair regions increased when less bottom-up information was available to achieve subjective continuity (contrast illusion>natural). Since we evaluated correlation in only a small number of regions, the threshold was relaxed to an uncorrected p value of 0.05 (Fig. 7). Given that differences occurred at uncorrected values, the connectivity results must be interpreted with caution and considered supplementary to the main findings. Within each ROI the left angular gyrus (continuity region) communicated more with all three repair regions (bilateral anterior insula, pars opercularis of left IFG and pre-SMA) to achieve continuity when the stimuli were more degraded. These results indicate a functional link between the continuity and repair sub-networks, supporting a two-path process underlying the continuity illusion phenomenon.

Fig. 7. Functional connectivity.

Functional connectivity between the left AG/STS seed (continuity region: green-red gradient) with all repair areas (blue-green gradient). The functional connectivity test was conducted on the repair contrast (illusion > natural). This shows regions with which AG communicates more when unconscious repair succeeds in creating a continuous percept. (Region abbreviations: AG= angular gyrus; STS = superior temporal sulcus; IFG = inferior frontal gyrus; pre-SMA = pre-supplementary motor area).

DISCUSSION

Little is known about the functional networks underlying the auditory continuity illusion, particularly for realistic stimuli such as speech. Here we show that this phenomenon is mediated by two neurally dissociable processes: a repair process which restores the low-level sensory representation of degraded speech, and a subjective continuity process where the structural coherency of the stimulus is evaluated. Figure 8 summarizes these findings in the context of our model. Regions mediating the repair are the bilateral anterior insula, left IFG, and the pre-SMA. The subjective sense of continuity is reflected in left AG/STS, right posterior STS, bilateral SFS, and precuneus. A subset of these regions, in both repair and continuity networks, applies prior knowledge through word-level template matching.

Fig. 8. Model.

A schematic of the processing paths mediating perceptual filling-in of speech. Regions with names in orange show evidence for word-level template matching. (Region abbreviations: AG= angular gyrus; STS = superior temporal sulcus; IFG = inferior frontal gyrus; SFS = superior frontal sulcus; pre-SMA = pre-supplementary motor area).

The network involved in repair includes left IFG (pars opercularis), anterior insulae, and pre-SMA. These areas showed greater activity when less sensory information is available or more repair is required, to achieve a continuous percept. Consistent with a role in repair, insulae and pre-SMA are known to participate in error processing, not limited to speech tasks. Importantly, this participation occurs whether or not subjects are aware of the errors (Hester et al., 2005; Klein et al., 2007), as in our illusion-failure and illusion conditions, respectively. The repair areas are also involved in controlled temporal sequencing, segmentation, and acoustic pattern recognition (Bamiou et al., 2003; Bohland and Guenther, 2006; Burton et al., 2000; Geiser et al., 2008; Zatorre et al., 1992), particularly under conditions where stimulus complexity increases (Sivonen et al., 2006) or expectations are violated (Friederici, 2002; Tillmann et al., 2006). Notice that in the illusion-failure condition, which shows the largest BOLD increase across repair areas, expectations about the acoustic pattern (word) are violated most conspicuously, i.e. with awareness. Furthermore, the insulae and left IFG repair regions have been associated with decision processes (Binder et al., 2004), showing greater activity with more difficult decisions as measured by longer reaction times. However, it is unlikely that our results for left IFG and insulae are mainly related to decision processes, given that the reaction times did not differ across the three conditions. Our results further distinguish left IFG and insulae by their remarkable specificity in repairing words as opposed to pseudowords. These regions therefore restore speech sounds by applying prior expectations through word-level templates. Both have long been implicated in word processing, including lexical retrieval (Bookheimer, 2002), and phonological word recognition (Rumsey et al., 1997). However, they are also known to influence perception within musical and sentential context (Bashford et al., 1996; Obleser et al., 2007; Tillmann et al., 2006), suggesting that word/pseudoword distinction reflects a more general principle. Namely, left IFG and anterior insulae may apply prior knowledge when it exists at an abstract or highly structured representational level.

The network reflecting subjective continuity consisted of left AG/STS, right posterior STS, left precuneus, and bilateral SFS. These areas were more active during the continuity illusion than when the illusion failed. Thus, they reflect the phenomenology of the illusion rather than any stimulus-related processing. According to our model, these regions receive the neural signal representing the acoustical input — corrupt or not —from low-level centers (e.g. Heschl’s gyrus), evaluate the coherency of the signal, and if necessary initiate the repair network. Among the continuity regions, only left AG/STS showed evidence of achieving the illusion by applying prior knowledge through word-level templates. In contrast, the right STS or the SFS and precuneus showed sensitivity to continuity but did not distinguish words and pseudowords. Though our data cannot confirm it, the left and right AG/STS regions could be homotopic, implying a common role for perceived continuity but a left-hemispheric dominance for language. This is consistent with left STS and angular gyrus’ broad role in speech and language tasks in several modalities (speech (Obleser et al., 2007); reading (Horwitz et al., 1998); sign language (Neville et al., 1998)), including semantic decision and verbal working memory (Binder et al., 2003; Owen et al., 2005). Left AG/STS furthermore showed greater functional connectivity with repair regions when less bottom-up information was available. This pattern of connectivity is nearly identical to that seen from left AG to medial and left lateral prefrontal areas when sentence context improves intelligibility of degraded speech (Obleser et al., 2007), and similar to that implicated in abnormal processing in dyslexia (Horwitz et al., 1998). All of these observations support AG’s participation in an internal lexicon. Thus, the left AG/STS plays a central role in the repair-continuity networks for speech by acting as a hub that links the two networks, in part through its explicit word representations.

Most neurophysiological and neuroimaging work on the continuity illusion has focused on early (primary) auditory cortex, including Heschl’s gyrus (Petkov et al., 2007; Riecke et al., 2007) and used pure tones as the foreground stimuli. Our results broadly replicate these results and situate them in the larger repair-continuity network. In our study, activity over the entire Heschl’s gyrus did not distinguish among conditions. We believe this is due to our successful efforts to equalize stimulus attributes across conditions (mean noise duration within ~19 ms). However, the middle portion of the right Heschl’s gyrus was more active for illusion-failure compared to illusion and natural conditions. The particular perceptual sensitivity of middle Heschl’s gyrus corroborates the findings of Riecke et al. (2007). In agreement with Heinrich et al. (2008), we attribute increased Heschl’s gyrus activity during illusion-failure to greater numbers of perceived onsets and offsets, as shown in single-cell work (Qin et al., 2007). Additionally, the statistically indistinguishable activity in the illusion and natural percepts confirms Petkov et al. (2007) findings that many A1 neurons represent illusory continuity and veridical continuity in a similar manner. The results of Heschl’s gyrus suggest that the outcome of continuity-repair networks is communicated back to the low-level auditory sensory processing centers (e.g. A1), where the neural behavior is modified to reflect these stimuli as intact, even though they are not (illusion). The early auditory cortices therefore behave as though the stimuli are continuous. Therefore, while our results generally agree that early auditory cortex can reflect perception, we instead find the causal mechanisms for stimulus repair and perception in higher-order cortices.

We have shown that perception depends on top-down knowledge in the form of word templates. However, several repair-continuity regions did not distinguish words from pseudowords, implying that they use different mechanisms to mediate illusory completion. One could classify prior knowledge for illusory completion into two types: i) Template-matching (or contextual knowledge) to an internal representation, or object-based knowledge, and ii) Gestalt continuity, or rule-based knowledge. Template-matching would fit sensory input to an abstract but detailed representation in memory, but this would occur only for well-learned sounds. For instance in speech, the brain may have templates at both word and sub-word levels. In contrast, a Gestalt mechanism (Wertheimer, 2004) would fill interruptions by following the rule that sounds tend to be spectrotemporally continuous (Bregman, 1990). It would work on any sounds, including unfamiliar ones such as pure tones (Bregman, 1990). These two forms of prior knowledge—object and rule—are not mutually exclusive and could be applied in parallel. For instance, left AG/STS applies lexical templates to complete degraded speech, while right STS may use Gestalt rules for spectrotemporal smoothness. In naturalistic conditions, we would expect a neural hierarchy of repair/continuity depending on the complexity of the stimulus and the type of top-down knowledge applied (Kazanina et al., 2006; Obleser et al., 2007; Samuel, 1981; Shinn-Cunningham and Wang, 2008; Sivonen et al., 2006).

Our results naturally raise the question of whether repair and continuity processes are speech-specific or domain-general (Price et al., 2005). Based on our observation that repair areas tend to be strongly recruited by template-matching, we would propose the following testable hypothesis. Repair areas are domain-general and are recruited for any degraded sound familiar enough to have an internal template or highly structured internal representation. The sound could have any acoustic complexity: speech, environmental noises, even simple tones provided they are highly learned. In the absence of a template, as for unfamiliar sounds, Gestalt processes would suffice for continuity. Continuity processes, on the other hand, should be more closely tied to the level of stimulus representation. So, while repair regions will always be the same (process-determined), continuity will be reflected in cortices with salient representations of the acoustic object. For words, we observe continuity in left AG/STS, a very important region for word-representations. But for degraded environmental sounds, continuity may be reflected in posterior middle temporal gyrus (Lewis et al., 2004), and for degraded pure tones it may be reflected in A1 (Riecke et al., 2007). We would thus expect a hierarchy of continuity that respects the established hierarchy of acoustical processing in the temporal and inferior parietal lobes (Davis and Johnsrude, 2003; Scott and Johnsrude, 2003). For example, when eliciting the continuity illusion with vowel-like non-speech sounds Heinrich et al., (2008) revealed enhanced activity in MTG and middle STS. Functional connectivity between repair and continuity networks should also respect this hierarchy, with left IFG showing maximal correlation with the appropriate continuity regions. Incidentally, for clear speech the current model may reduce to the continuity pathway alone, as there would be no need for repair. However, in realistic situations, interference from background noises is extremely common. Our model thus offers a general mechanism for filling-in of naturalistic sounds.

The negative BOLD values in the continuity regions are noteworthy. In all four areas, activity during the illusion is less negative than during illusion-failure. This activity profile is reminiscent of the default-mode network, which reduces its activity upon commencement of goal-directed, attention-demanding tasks (Greicius et al., 2003; Raichle et al., 2001). The default-mode network consists of numerous regions, including precuneus and medial prefrontal cortex, whose activity negatively correlates with task difficulty. However, here we used an adaptive procedure that incidentally ensured similar difficulty across conditions, as measured by reaction time. Furthermore, the core of our continuity network, left AG/STS, displayed significant specificity for words versus pseudowords. Finally, the functional connectivity analysis showed that left AG/STS interacts with functionally defined areas in the repair network, which are not usually among the default-mode regions. Therefore, our key results cannot be interpreted as general, task-independent modulation of a default network. Nevertheless, as implied by psychophysical work (Lyzenga et al., 2005) our results suggest that perceived continuity could be partly mediated by passive processes, for glossing over interruptions, as much as active ones.

The current study provides the first neural evidence that illusory completion is mediated by at least two dissociable processes: stimulus repair and phenomenological continuity. Using BOLD data, we cannot establish the dynamics of these two pathways but it is likely that the two networks communicate bidirectionally, as well as with the early sensory cortices (e.g. A1) to achieve a coherent percept. Our model provides a framework for testable hypotheses about filling-in and can be extended to include prior knowledge of different forms such as Gestalt rules or sentence context (Sivonen et al., 2006). One could also use this framework to illuminate the neural mechanisms of analogous filling-in illusions in other sensory modalities (Komatsu, 2006). More generally, our findings help us understand how top-down and bottom-up influences interact to improve perception in naturalistic, noisy environments.