Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Jun 1.
Published in final edited form as: J Cogn Neurosci. 2016 Oct 19;29(3):435–447. doi: 10.1162/jocn_a_01052

An association between auditory-visual synchrony processing and reading comprehension: Behavioral and electrophysiological evidence

Julia Mossbridge 1, Jacob Zweig 1, Marcia Grabowecky 1,2, Satoru Suzuki 1,2
PMCID: PMC5300749  NIHMSID: NIHMS826880  PMID: 28129060

Abstract

The perceptual system integrates synchronized auditory-visual signals in part to promote individuation of objects in cluttered environments. The processing of auditory-visual synchrony may more generally contribute to cognition by synchronizing internally generated multimodal signals. Reading is a prime example because the ability to synchronize internal phonological and/or lexical processing with visual orthographic processing may facilitate encoding of words and meanings. Consistent with this possibility, developmental and clinical research has suggested a link between reading performance and the ability to compare visual spatial/temporal patterns with auditory temporal patterns. Here, we provide converging behavioral and electrophysiological evidence suggesting that greater behavioral ability to judge auditory-visual synchrony (Experiment 1) and greater sensitivity of an electrophysiological marker of auditory-visual synchrony processing (Experiment 2) both predict superior reading comprehension performance, accounting for 16% and 25% of the variance, respectively. These results support the idea that the mechanisms that detect auditory-visual synchrony contribute to reading comprehension.

Introduction

The perceptual system integrates synchronized auditory and visual signals (e.g., Driver, 1996; Driver & Spence, 1998; Guzman-Martinez et al., 2012; Iordanescu et al., 2008, 2010, 2011; Molholm et al., 2004; Shimojo & Shams, 2001; Smith et al., 2007; Stein et al., 1989; Van der Burg et al., 2008, 2010), and crossmodal integration is prevalent in early sensory areas as well as in subcortical and higher cortical regions (e.g., Driver & Noesselt, 2008; Ghazanfar & Schroeder, 2006; Stein & Stanford, 2008, for reviews). This is advantageous because synchronized auditory-visual signals typically originate from a single object, so that integrating them into a common source facilitates individuation and selection of objects in a cluttered environment (e.g., Driver, 1996; Van der Burg et al., 2008, 2010).

An intriguing possibility is that auditory-visual synchrony processing mechanisms may also play a general role in cognitive processes by facilitating the synchronization of internally generated crossmodal signals. For instance, while engaging in problem solving, thought processes may be facilitated when internally generated imagery (visual, auditory, tactile, etc.) is appropriately synchronized with a train of verbal thoughts. Although such a general hypothesis is difficult to evaluate, reading provides an interesting test case as prior research has provided indirect evidence suggesting a relationship between auditory-visual synchrony processing and reading.

For example, the ability to crossmodally (in addition to unimodally) compare auditory and visual rhythms accounted for some variability in reading performance in children (Rose et al., 1999, a follow-up study to Birch & Belmont, 1964). Because Rose et al. (1999) used a combination of decoding and comprehension tests to evaluate reading performance, their result suggests that the ability to temporally align the remembered rhythm in one modality with a subsequently presented rhythm in another modality is relevant to the decoding and/or comprehension aspects of reading. Training reading-impaired children to perceptually match an auditory rhythm with a concurrently presented static visual pattern that spatially represented a rhythm from left to right improved word reading in those children (Kujala et al., 2001). This result suggests that the ability to shift visual attention in synchrony with an auditory rhythm contributes to decoding and may also to semantic access because Kujala et al. (2001) used real words as the stimuli. Further, irrelevant auditory stimuli interfere with visual temporal judgments across a longer duration in dyslexic adults than in non-dyslexic adults (Hairston et al., 2005), suggesting that an abnormally wide temporal window of auditory-visual integration, potentially causing less precise processing of auditory-visual synchrony, is related to impaired processing of text in dyslexia.

To which reading-related processes might auditory-visual synchrony processing contribute? A feasible candidate is word decoding, the process of generating phonological representations in synchrony with visual orthographic processing, a process that appears to be impaired in dyslexia (e.g., Adams, 1990; Breznitz, 2002; Breznitz & Misra, 2003; Vellutino et al., 2004; Lehongre et al., 2011). Nevertheless, crossmodal synchrony processing may also contribute to higher-order processes relevant to reading comprehension. For example, crossmodal synchrony processing may facilitate semantic access because the retrieval of word meanings may benefit from the synchronization of sensory processes with dynamic activation of multimodal memory representations (e.g., Federmeier & Laszlo, 2009; Kutas & Federmeier, 2011). Note that reading-related processes are not strictly serial in that word decoding is likely to involve semantic access (e.g., Hoover & Gough, 1990). Reading comprehension ultimately requires the integration of meanings across sentences and paragraphs to understand causal and referential relationships for generating a coherent interpretation of a story (e.g., Gough & Tunmer, 1986; Rapp et al., 2007). Because these integrative processes are limited by attention and working memory (e.g., Rapp et al., 2007; van den Broek et al., 2005; King & Kutas, 1995), crossmodal synchrony processing may facilitate the timely shifting of attention and eye movements across text (e.g., to confirm causal and referential relations) and/or the timely activation of working memory and generation of mental imagery (e.g., to help integrate semantic relationships into a coherent interpretation) in coordination with the progression of phonological, syntactic, and semantic processing. Crossmodal synchrony processing may contribute to these higher-order processes by facilitating the temporal coordination of information flow across the temporo-frontal language circuit (e.g., Friederici, 2012), motor areas, and sensory cortices.

The goal of the current study was to demonstrate a direct relationship between basic sensitivity to auditory-visual synchrony and reading comprehension in typically developing adults. We evaluated sensitivity to auditory-visual synchrony using both behavioral and electrophysiological methods. To measure auditory-visual synchrony sensitivity independently of reading and linguistic abilities, we used non-linguistic stimuli such as flashes, beeps, dynamic visual patterns, and music. To measure reading ability, we used a task that evaluated the ability to comprehend extended text. Because the current study was the first attempt at demonstrating the hypothesized direct association between auditory-visual synchrony sensitivity and reading in typically developing adults, we wished to maximize the chance of detecting such an association by using a task that required multiple processes involved in reading (word decoding, semantic access, and integration), all of which could potentially benefit from crossmodal synchronization (as discussed above). Furthermore, because the typical goal of reading is comprehension of ideas conveyed across multiple paragraphs, we sought to demonstrate the relationship between the basic processing of auditory-visual synchrony and the ability to understand an extended text. We provide converging behavioral and electrophysiological evidence demonstrating that an individual’s ability to judge auditory-visual synchrony and an individual’s left-lateralized electrophysiological sensitivity to auditory-visual synchrony both predict reading comprehension performance.

Experiment 1: The ability to judge auditory-visual synchrony predicts text comprehension performance

We measured the behavioral ability to judge auditory-visual synchrony using a standard synchrony-judgment task (e.g., van Eijk et al., 2008), where a visual flash and an auditory beep were presented with a range of stimulus-onset asynchronies or SOAs (including both auditory-presented-first and visual-presented-first trials) and participants judged whether or not the auditory and visual stimuli were presented simultaneously. Perfect performance would entail responding “yes” on trials where a flash and a beep were presented synchronously and responding “no” on all other trials where a flash and a beep were presented asynchronously. In reality, participants respond “yes” on asynchronous trials when auditory-visual SOAs are small enough that the asynchrony is not detected. The function relating the proportion of “yes” responses to the auditory-visual SOA is well fit by a Gaussian curve (e.g., van Eijk et al., 2008), and the standard deviation of the fitted Gaussian curve provides a measure of the just noticeable difference (JND), with a smaller JND indicating greater perceptual sensitivity to auditory-visual synchrony. Thus, if auditory-visual synchrony processing contributes to reading comprehension, individuals with smaller JNDs should be superior at reading comprehension.

However, even if we found this association, it might reflect individual differences in general factors such as attentiveness, effort, intelligence, and/or in the quality of temporal coding rather than individual differences in the ability to detect auditory-visual synchrony per se. We thus included a control task where participants judged the temporal order between a visual flash and an auditory beep. The stimuli in the temporal-order-judgment task were identical to those in the synchrony-judgment task; both tasks required the processing of auditory-visual timing, except that participants focused on auditory-visual synchrony in the synchrony-judgment task whereas they focused on auditory-visual order in the temporal-order-judgment task.

Importantly, whereas the synchrony-judgment task provides a direct measure of participants’ sensitivity to auditory-visual synchrony, the temporal-order-judgment task forces participants to make order judgments even when the visual and auditory stimuli are perceived to be simultaneous, thus reflecting processes other than auditory-visual synchrony detection (e.g., García-Pérez & Alcalá-Quintana, 2012; van Eijk et al., 2008). If reading comprehension is selectively associated with the sensitivity to auditory-visual synchrony, comprehension performance should be strongly associated with the JND estimated using the auditory-visual synchrony-judgment task, but significantly less associated with the JND estimated using the auditory-visual temporal-order judgment task. Alternatively, because general task demands and required processes are similar for the two tasks (both requiring the processing of auditory and visual timing for the same set of stimuli), if reading comprehension is associated with general factors such as attentiveness, effort, intelligence, and/or the quality of temporal coding, reading comprehension performance should be equivalently associated with the JND measured with either task.

Methods

Participants

Fifty-one Northwestern University undergraduate students (ages 18–22) gave informed consent to participate for partial course credit or monetary compensation. All were right-handed native English speakers with normal hearing and normal or corrected-to-normal vision. They were individually tested in a dimly lit room.

Stimuli and Procedure

Participants performed three tasks in the following order: an auditory-visual (AV) synchrony-judgment task, an AV temporal-order judgment task, and a reading comprehension task. The AV synchrony-judgment task was given first for two reasons. First, had we begun with the AV temporal-order judgment task, as a result of having participated in this task, participants might be led to consider temporal order while judging auditory-visual simultaneity. Second, any incidental correlation (e.g., based on slow fluctuations in arousal) would be more likely to occur between tasks that are performed in closer temporal proximity. Because we hypothesized a selective association between auditory-visual synchrony judgment and reading comprehension, having participants perform the synchrony-judgment task first, the temporal-order-judgment task next, and the reading comprehension task last reduced the probability that the predicted association would be observed as the result of incidental correlation. We will now describe the three tasks in detail.

The auditory-visual (AV) synchrony-judgment task

Participants sat in a comfortable chair and were instructed to look at the center of a dark (4.9 cd/m2) display monitor throughout the experiment. They pressed the space bar on the computer keyboard to start the experiment. Following an interval jittered between 1 and 3 seconds, a bright circle (35 cd/m2 and 5.1° diameter, presented at the center of the screen) and a tone (3500 Hz, 48 dB SPL(A), presented through headphones), each lasting 10 ms, were presented at eight different values of stimulus onset asynchrony (SOA): −250, −187.5, −125, −62.5, +62.5, +125, +187.5, and +250 ms (a range typically used in AV synchrony-judgment and AV temporal-order-judgment tasks; e.g., van Eijk et al., 2008), where the negative values indicate that the auditory tone was presented first and positive values indicate that the visual flash was presented first. Participants responded as to whether or not the visual flash and the auditory beep were simultaneous by pressing a corresponding key in a non-speeded manner (but with a response deadline of 2.5 s). The next auditory-visual stimulus was presented after an interval jittered between 1 and 3 seconds. Each auditory-visual SOA was tested 10 times in a randomized order for a total of 80 trials. Prior to these experimental trials, 10 practice trials were given with SOAs randomly chosen from coarsely sampled intervals, −500, −400, −250, −100, +100, +250, +400, and +500 ms.

The proportion of “synchronous” responses should be maximal at a specific stimulus delay corresponding to the participant’s auditory-visual processing delay, and should monotonically decrease as the auditory-visual SOA is increased in the negative and positive directions. As described earlier, the resultant bell curve is reasonably well fit by a Gaussian function (e.g., van Eijk et al., 2008), and the JND for the auditory-visual synchrony judgment is reflected in the standard deviation of the fitted Gaussian curve, with a smaller standard deviation (i.e., a smaller JND) indicating greater sensitivity to auditory-visual synchrony.

The auditory-visual (AV) temporal-order-judgment task

The stimuli and procedure were identical to those for the AV synchrony-judgment task except that on each trial participants indicated whether the visual flash or the auditory beep was presented first. The proportion of “visual first” responses forms a sigmoidal function of stimulus delay, passing through 50% at a specific stimulus delay corresponding to the participant’s auditory-visual processing delay. The sigmoidal curve is reasonably well fit by a cumulative Gaussian function (e.g., van Eijk et al., 2008), and the JND for auditory-visual temporal-order judgment is reflected in the standard deviation of the fitted cumulative Gaussian curve, with a smaller standard deviation (i.e., a smaller JND) indicating greater sensitivity to auditory-visual temporal order.

The reading-comprehension task

As discussed in the Introduction section, we evaluated comprehension of extended text, the typical goal of reading, which required multiple component processes including word (orthographic-to-phonological) decoding, semantic access, working memory, and the integration of causal and inferential relationships across text. Our extended text consisted of the first 1,182 words of the first chapter of Doctor Pascal by Emile Zola (Lexile score 1,170). We chose this text because it is both conceptually rich and sufficiently unfamiliar; none of our participants had previously read the text. To minimize the potential effects of eye movements on reading (e.g., Henderson & Luke, 2014) and to control for individual differences in reading speed, we presented the extended text one word at a time (with accompanying punctuation marks) at the center of the computer monitor at the rate of 300 ms per word (with a 200-ms inter-word interval), comparable to the average rate of prose reading for college students (e.g., Carver, 1992). The white (76.8 cd/m2) text in Times font was presented against a dark (4.9 cd/m2) background, with vertical visual angles ranging from 0.49° to 0.73° and horizontal visual angles ranging from 0.61° to 3.64°.

The extended text was presented twice, first in a scrambled order and then in the correct order. The primary reason for this manipulation was to verify that participants made an effort to comprehend the story as well as to control for individual differences in the amount of effort, arousal, and carefulness (see the Results section). In both presentation conditions, participants were instructed to press the mouse button as quickly and as accurately as possible every time they saw the word “and.” In the scrambled-order condition, the only task was to respond to “and.” In the correct-order condition, participants had to comprehend the story as well as respond to “and.” Slowing of responses to “and” in the dual-task correct-order condition relative to the scrambled-order condition indicates that participants made an effort to comprehend the story, thus paying less attention to the “and” detection task. A benefit of including the scrambled-order condition prior to the correct-order condition was that it increased the temporal distance between the initial AV synchrony-judgment task and the reading-comprehension task.

Reading comprehension was evaluated at the end of the correct-order reading condition using a modified version of the multiple-choice questions we previously developed (see Mossbridge et al., 2013a for the full extended text and the test questions). The previous set included four questions each accompanied by four answer choices. Participants were told that any number of choices could be correct for each question and that they should circle all choices that they thought were correct. Questions 1 through 3 included two correct choices whereas question 4 included only one correct choice. A stringent scoring method (each question was scored as correct only if all correct choices were circled and none of the incorrect choices were circled) ensured that chance performance would result in a low score; specifically, the probability of getting each question correct by chance was only 6.7% = 1/(24–1), where the −1 accounts for the fact that participants knew that each question had at least one correct answer (thus, not circling any answer was not an option).

We previously used the total score from all four questions as the measure of reading comprehension (Mossbridge et al., 2013a). Here, because we obtained ACT reading scores (American College Testing: http://www.act.org/content/act/en/products-and-services.html) from 25 of the participants, we were able to assess the reliability of the four questions. Questions 1 through 3 were all moderately correlated with ACT reading score (Pearson’s r ranging from 0.20 to 0.27) whereas question 4 was not (r = 0.07). We thus used the average of the scores from questions 1 through 3 as the measure of reading comprehension, which was significantly correlated with ACT reading score, r = 0.44, t(23) = 2.348, p < 0.03.

All visual stimuli were presented on a 21” color CRT monitor (1,024 x 768 pixel resolution) at a 100 Hz refresh rate, and all auditory stimuli were presented through Sennheiser Pro headphones. The experiment was controlled by a PC computer running Windows XP, using MATLAB software (The MathWorks, Inc., Natick, MA) with Psychtoolbox extensions (Version 3.10.11; Brainard, 1997; Kleiner, Brainard, & Pelli, 2007; Pelli, 1997) for the AV synchrony-judgment and AV temporal-order-judgment tasks, and using Presentation software (version 11.0, Build 04.25.07, www.neurobs.com) for the reading-comprehension task. The viewing distance was 123 cm.

Results

The AV synchrony-judgment JND (inversely related to the sensitivity for discriminating between AV synchrony and asynchrony; see the Methods section) was negatively correlated with the comprehension score, r = −0.41, t(48) = 3.14, p < 0.003 (Figure 1; note that one participant whose data point fell outside of the 95% confidence ellipse was removed from the analysis; however, statistical results were unchanged when this outlier was included). This relationship indicates that greater sensitivity for judging AV synchrony is associated with superior reading comprehension.

Figure 1.

Figure 1

Open in a new tab

The comprehension score was negatively correlated with AV synchrony-judgment JND (computed as the standard deviation of fitted Gaussian; see the methods section for details), indicating that greater sensitivity to AV synchrony is associated with superior reading comprehension. The black line represents the linear fit and the ellipse represents the 95% confidence region. The outlier on the extreme left (outside of the confidence ellipse) was removed from analysis. The gray line represents the linear fit without the outlier.

The specificity of this association is suggested by the lack of a similar correlation between the AV temporal-order-judgment JND (inversely related to the sensitivity for judging AV temporal order) and the reading comprehension score, r = −0.13, t(48) = 0.90, p > 0.37. To confirm this specificity, we tested a multiple-regression model with both the AV synchrony-judgment JND and the AV temporal-order-judgment JND as regressors; only the AV synchrony-judgment JND significantly predicted the reading comprehension score (t[47] = −2.95, p < 0.005 for the AV synchrony-judgment JND and t[47] = 0.12, p > 0.90, for the AV temporal-order-judgment JND).

Results from the “and” detection task provided evidence in support of the specific association between the ability to judge auditory-visual synchrony and reading comprehension performance. Overall, responses to the word “and” were significantly worse (longer response times and higher error rates) in the dual-task correct-order (comprehension) condition than in the scrambled-order condition (M = 647 ms [SE = 11] vs. M = 578 ms [SE = 9], t[50] = 7.79, p < 0.0001, for response times, and M = 0.16 [SE = 0.02] vs. M = 0.05 [SE = 0.01], t[50] = 6.06, p < 0.0001, for error rates), confirming that participants made an effort to comprehend the extended text in the correct-order condition.

It is reasonable to assume that participants who made more effort in comprehending the story would have incurred increased response times and error rates in the “and” detection task in the correct-order condition (in which comprehension was required) relative to the scrambled-order condition (in which comprehension was not required). Thus, individual differences in the “and”-detection response times and error rates in the correct-order condition residualized to those in the scrambled-order condition provided a measure of individual differences in the amount of effort participants made to comprehend the story. If some of the individual differences in comprehension performance were accounted for by individual differences in comprehension effort, this would be captured by positive correlations between the residualized response time and/or residualized error rate and the comprehension score. However, the residualized response time was uncorrelated with the comprehension score, r = −0.03 (t[48] = 0.20, p > 0.84), though the residualized error rate was marginally positively correlated with the comprehension score, r = 0.26 (t[48] = 1.86, p < 0.070). These low correlations suggest that the comprehension score primarily reflected an individual’s reading comprehension ability rather than the amount of effort he or she put towards the reading-comprehension task. Importantly, when we included the residualized error rate in addition to the AV synchrony-judgment JND and AV temporal-order-judgment JND as regressors, only the AV synchrony-judgment JND significantly predicted the comprehension score (t[46] = −2.51, p < 0.016, for the AV synchrony-judgment JND, t[46] = 0.23, p > 0.81, for the AV temporal-order-judgment JND, and t[46] = 0.84, p > 0.40, for the residualized error rate). Finally, the “and”-detection response times and error rates in the scrambled-order condition additionally provided measures of general arousal (reflected in faster response times) and carefulness (reflected in reduced error). These measures were uncorrelated with the comprehension score, r = −0.07 (t[48] = 0.51, p > 0.61) for response times, and r = −0.0002 (t[48] = 0.002, p > 0.99) for error rates, suggesting that the individual differences in the comprehension score in this study were unlikely to have been influenced by individual differences in general arousal or carefulness.

Taken together, these results suggest that the ability to judge auditory-visual synchrony is uniquely associated with the ability to comprehend extended text because (1) whereas the ability to judge AV synchrony predicted the comprehension score, the ability to judge auditory-visual temporal order did not, (2) the measures of the effort participants made to comprehend the text, general arousal, and carefulness did not predict the comprehension score, and (3) only the ability to judge auditory-visual synchrony predicted the comprehension score in a multiple regression model that also included the ability to judge auditory-visual temporal order and the accuracy-based measure of comprehension effort that was marginally correlated with the comprehension score. In the next experiment, we aimed to provide converging electrophysiological evidence suggesting that the sensitivity of neural mechanisms that may underlie auditory-visual synchrony detection is associated with reading comprehension.

Experiment 2. The sensitivity of the left-frontal ASSR (auditory steady-state response) to auditory-visual synchrony predicts text comprehension performance

In a previous study, we identified an electroencephalographic (EEG) index that reflected neural sensitivity to auditory-visual synchrony in complex stimuli (Mossbridge et al., 2013b). In that study, we presented classical music with a visualizer that matched the dynamics of the music in terms of changes in luminance, color and motion. The visualizer was presented either synchronously or asynchronously with the music. We amplitude-modulated the music at 40 Hz (which sounded like listening to the music through an electric window fan), and monitored the 40Hz EEG component that was phase-locked to the amplitude modulation, known as auditory steady-state response or ASSR. This 40Hz ASSR, primarily localized in frontal scalp regions, is thought to track the auditory cortical activity in response to amplitude-modulated sounds (e.g., Gutschalk et al., 1999; Herdman et al., 2002; Picton et al., 2003; Ross et al., 2005). Importantly, because the 40Hz ASSR was phase-locked to the music, any modulation of its amplitude by auditory-visual alignment reflected the influences of auditory-visual synchrony processing on auditory-evoked neural activity. Furthermore, because the visualizer display did not contain any dynamics at 40 Hz (Mossbridge et al., 2012b), the 40Hz ASSR was uncontaminated by any visual evoked activity; thus, the magnitude of modulation of 40Hz ASSR by auditory-visual synchrony reflected the sensitivity of crossmodal synchrony processing that influences auditory-evoked activity. We demonstrated that the left-frontal component of the 40Hz ASSR was significantly reduced when the music and the visualizer were played asynchronously relative to when they were played synchronously, suggesting that the left-frontal 40Hz ASSR reflects neural sensitivity to the crossmodal alignment between auditory and visual dynamics (Mossbridge et al., 2013b). If the association between the behavioral ability to judge auditory-visual synchrony and reading comprehension that was demonstrated in Experiment 1 is indicative of a general association between the mechanisms that process auditory-visual synchrony and the mechanisms that enable reading comprehension, the electrophysiological sensitivity to auditory-visual synchrony, as reflected in the left-frontal 40Hz ASSR, should also be associated with reading comprehension. We tested this prediction in a new group of participants. They completed the reading tasks and the comprehension test first. They then experienced the synchronously and asynchronously presented music and visualizer displays, during which their 40Hz ASSR was recorded.

Methods

Participants

Twenty-eight adults (ages 18–29) who responded to a poster on the Northwestern University campus (Evanston, IL, USA) gave informed consent to participate for monetary compensation. All were right-handed native English speakers, and had normal hearing and normal or corrected-to-normal vision. They were individually tested in a dimly lit room that was electrically shielded for EEG recording.

We have previously identified a left-frontal 40Hz ASSR sensitive to auditory-visual synchrony with these same participants (Mossbridge et al., 2013b; also see Figure 3a). In that study, we also administered the two reading tasks, the scrambled-order and correct-order tasks, identical to those used in Experiment 1 as control conditions. We also gave the comprehension test that was identical to that used in Experiment 1. However, because the comprehension test was given primarily as a way to enforce comprehension effort during the correct-order reading task, the scores were not analyzed. The reading tasks and the comprehension test were administered prior to recording 40Hz ASSR evoked by the synchronously and asynchronously presented music and visualizer displays (see below). Here, we analyzed the participants’ comprehension scores as well as the sensitivity of their left-frontal 40Hz ASSR to auditory-visual alignment to determine whether the left-frontal electrophysiological sensitivity to auditory-visual synchrony predicted reading comprehension performance.

Figure 3.

Figure 3

Open in a new tab

Midline: The top figure shows the left-frontal and right-frontal electrode locations within which the CSD-transformed ASSR to the 40Hz amplitude-modulated music was averaged. The lower figures show the scalp topography of the mean 40Hz ASSR amplitudes for the AV-aligned (upper) and AV-misaligned (lower) conditions, with the colored bar indicating scale. (a) The mean left-frontal 40Hz ASSR was significantly higher in the AV-aligned than AV-misaligned condition (frequency specificity shown in upper panel). (b) The mean right-frontal 40Hz ASSR was equivalent in the AV-aligned and AV-misaligned conditions (frequency specificity shown in upper panel). Error bars and line widths represent ±1 standard error of the mean adjusted for within-participants comparisons. See Mossbridge et al., 2013b for details regarding these results. (c) The comprehension score was significantly correlated with the left-frontal ASSR index (AV-aligned condition minus AV-misaligned condition) (same as Figure 2). (e) This correlation persisted after controlling for the right-frontal ASSR index and the ERP index reflecting high-level (post-decoding) comprehension processes. (d) The comprehension score was significantly correlated with the right-frontal ASSR index (AV-aligned condition minus AV-misaligned condition). (f) This correlation disappeared after controlling for the left-frontal ASSR index and the ERP index reflecting high-level (post-decoding) comprehension processes. See text for details.

Stimuli and Procedure

As in Experiment 1, the extended text (identical to that used in Experiment 1) was presented twice, once in a scrambled order and once in the correct order while participants were instructed to respond to the word “and” as quickly and as accurately as possible. Again, this manipulation allowed us to confirm that participants made an effort to comprehend the story in the correct-order condition as well as to reasonably control for individual differences in the amount of effort, arousal, and carefulness. Classical music was played while participants performed these reading conditions for reasons unrelated to the current investigation (see Mossbridge et al., 2013b); however, many people routinely read text with background music, so that the presence of music should not be problematic for assessing reading comprehension. The comprehension test (identical to that used in Experiment 1) was administered following the correct-order condition.

After the two reading conditions (the order was counterbalanced across participants) and the comprehension test, the relevant EEG data were recorded while participants experienced synchronous and asynchronous auditory-visual presentations. The music was Beethoven’s Moonlight Sonata, which was amplitude modulated at 40 Hz (sinusoidal modulation at 100% depth) to evoke an ASSR reflecting auditory sensory activity (see below). We used the iTunes Jelly visualizer, which generated aesthetically pleasing dynamic visual displays by primarily matching changes in the dynamics of the music to changes in the luminance of the visual elements (see Mossbridge et al., 2013b for the verification of this relationship). The visualizer also changed other visual features such as color, motion, and pattern organization in synchrony with changes in the auditory intensity and pitch of the music, but these instances of auditory-visual synchronization were less apparent.

Note that the music and the corresponding visualizer displays included temporal organization across multiple time scales (e.g., rhythmic beats, faster arpeggios, and slower variations such as sweeping crescendos). The use of a complex multi-scale dynamic structure allowed us to increase the chance of identifying an electrophysiological correlate of auditory-visual synchrony sensitivity without a priori knowledge of the temporal scale to which underlying mechanisms might be tuned. It also ensured that any identified electrophysiological correlate of crossmodal synchrony processing would be operative in a complex natural environment where crossmodal dynamics are defined by multiple time scales in the context of multiple concurrently varying features. A drawback of using auditory-visual stimuli with complex dynamics is that future research would be necessary to differentiate the specific contributions from individual time scales and features.

Participants listened to an identical 2-minute portion of the music twice, once while watching the visualizer display presented in synchrony with the music—the AV-aligned condition—and once while watching the visualizer display presented with a 30-s delay—the AV-misaligned condition (the condition order was counterbalanced across participants). The use of a relatively long visual delay in the AV-misaligned condition allowed us to introduce substantial asynchrony across multiple time scales contained in the complex dynamics. To avoid potential stimulus artifacts, we freshly generated a new visualizer display for the AV-aligned and AV-misaligned conditions for each participant. Thus, the visualizer display was different each time in feature-specific details (e.g., in colors, shapes, and motions), but was still dynamically synchronized with the music, so that we were able to identify the scalp pattern of ASSR that was sensitive to auditory-visual synchrony independently of stimulus-specific contributions. Video examples of the AV-aligned and AV-misaligned conditions are available online at https://www.dropbox.com/sh/yrlwfu96qyhum6c/73efhFYfjI. The first two videos provide examples of the AV-aligned and AV-misaligned conditions, respectively.

EEG was recorded using a 64-channel (10–20 configuration) Biosemi system with a nose reference with additional electrodes placed lateral to each eye for recording horizontal electro-oculographic (EOG) activity, and under the left eye for recording vertical EOG activity, including blinks. Data were sampled at 1024 Hz and bandpass-filtered between 0.1 and 100 Hz. The resulting EEG waveforms were segmented into 1-s epochs; epochs with eye blinks and muscle artifacts were manually removed based on vertical EOG activity (generally > 100 μV, but adjusted for some participants as necessary), and epochs with saccades were manually removed based on horizontal EOG activity (> 100 μV, but adjusted as necessary). The first 80 artifact-free epochs from each participant for each condition were transformed into CSD (Current Source Density) maps using CSDtoolbox Version 1.1 (http://psychophysiology.cpmc.columbia.edu/Software/CSDtoolbox) to obtain a reference-free and high-spatial-resolution measure of EEG signals (Tenke & Kayser, 2005).

ASSR amplitude was computed (for each electrode and each participant) by averaging the CSD-transformed EEG waveforms across the 80 epochs, taking a Fast-Fourier Transform (FFT) of the average waveform (using Matlab 7.4.0; Mathworks), and then extracting the amplitude of the Fourier component at 40 Hz (at 1-Hz resolution). Averaging the EEG waveforms across the 80 epochs before taking an FFT reduced any contributions from non-phase-locked responses, thus isolating the stimulus-evoked neural responses. CSD-transformed EEG signals offer a conservative estimate of the locations of the underlying neural generator (Tenke & Kayser, 2012). Lateralized CSD-transformed EEG signals in particular reflect the activity of sources that can be reasonably assumed to be located on the same side of the brain (e.g., Bernier et al., 2009; Carbonnell et al., 2004; Kayser et al., 2006). Further, source localization results from EEG, MEG (magnetoencephalography), and PET (positron emission tomography) studies suggest that ASSR evoked by amplitude modulation at 40 Hz arises from primary auditory cortex with additional contributions from subcortical structures and auditory association areas including the superior temporal plane (e.g., Gutschalk et al., 1999; Herdman et al., 2002; Pastor et al., 2002; Picton et al., 2003; Ross et al., 2005). Because EEG signals from subcortical structures are not lateralized on the scalp, a lateralized modulation of ASSR can be reasonably attributed to a modulation of auditory-evoked cortical activity in the same hemisphere.

We reported (Mossbridge et al., 2013b) that the ASSR to the amplitude-modulated Moonlight Sonata was obtained bilaterally from the frontal scalp regions, consistent with prior results (e.g., Picton et al., 2003 [also using CSD transform]). In particular, the ASSR amplitude from the left-frontal scalp region was selectively reduced in the AV-misaligned condition relative to the AV-aligned condition (Figure 3a). We further showed that neither the degree of ASSR phase-locking to the amplitude modulation of the music (measured as inter-trial phase coherence) nor the stimulus-non-phase-locked oscillatory EEG activity was modulated by auditory-visual synchrony, suggesting that it is the amplitude of the left-lateralized auditory-evoked cortical response rather than its fidelity (phase locking) or other on-going non-sensory oscillatory neural activity that is sensitive to auditory-visual synchrony (Mossbridge et al., 2013b). We thus computed the difference in the left-frontal ASSR amplitude between the AV-aligned and AV-misaligned conditions for each participant as an index of electrophysiological sensitivity to auditory-visual synchrony. We determined whether a larger left-frontal ASSR index, indicative of greater neural sensitivity to auditory-visual synchrony, was associated with a higher reading comprehension score.

The extended text (one word at a time) and the visualizer display were presented on a 21” color CRT monitor (1,024 x 768 pixel resolution) at 60 Hz refresh rate, and the amplitude-modulated music was presented through a pair of Sennheiser Pro headphones at an average level of 70 dB SPL(A). Stimulus presentations and behavioral tasks were controlled using a MacBook Pro laptop computer (OS10.6) with Presentation software (version 11.0, Build 04.25.07, www.neurobs.com). Each participant was seated in a comfortable armchair (to reduce muscle artifacts in the EEG signals) at 120 cm from the display monitor.

Results

The ASSRs obtained were sharply peaked at 40 Hz (the rate of amplitude modulation), indicating an appropriate level of signal-to-noise ratio (Figure 3a, upper panel). Crucially, the magnitude of the left-frontal ASSR index (AV-aligned condition minus AV-misaligned condition), which we had previously identified to reflect the sensitivity of the left-lateralized auditory cortical mechanisms to auditory-visual synchrony (Mossbridge et al., 2013b), was positively correlated with the comprehension score, r = 0.51, t(27) = 3.04, p < 0.006 (Figure 2). This indicates that greater sensitivity of the left-lateralized auditory cortical mechanisms to auditory-visual synchrony is associated with superior reading comprehension.

Figure 2.

Figure 2

Open in a new tab

The comprehension score was positively correlated with the left-frontal ASSR index (AV-aligned condition minus AV-misaligned condition), providing converging electrophysiological evidence suggesting that greater sensitivity to AV synchrony is associated with superior reading comprehension. The line represents the linear fit and the ellipse represents the 95% confidence region.

As in Experiment 1, results from the “and”-detection task provided evidence in support of a specific association between the neural sensitivity to auditory-visual synchrony and reading comprehension performance. To replicate Experiment 1, responses to the word “and” were overall significantly worse (longer response times and higher error rates) in the dual-task correct-order condition than in the scrambled-order condition (M = 637 ms [SE = 17] vs. M=573 ms [SE = 12], t[27] = 5.37, p < 0.0001, for response times, and M = 0.17 [SE = 0.02] vs. M = 0.08 [SE = 0.02]), t[27] = 2.97, p < 0.008, for error rates), confirming that participants made an effort to comprehend the extended text in the correct-order condition (also reported in Mossbridge et al., 2013a). Neither our measures of comprehension effort (the correct-order response time and correct-order error rate residualized to its scrambled-order counterpart), our measure of general arousal (the scrambled-order response time) nor our measure of carefulness (the scrambled-order error rate), was significantly correlated with the comprehension score (r = −0.02, t[24] = 0.12, p > 0.90, for the residualized response time, r = −0.26, t[24] = 1.30, p > 0.20, for the residualized error rate, r = −0.29, t[24] = 1.46, p > 0.15, for the scrambled-order response time, and r = −0.16, t[24] = 0.81, p > 0.42, for the scrambled-order error rate). Thus, the comprehension score in this experiment was unlikely to have been influenced by individual differences in comprehension effort, general arousal, or carefulness. Nevertheless, because some of the non-significant correlation coefficients were moderate, we tested a multiple regression model including all these variables as regressors; only the left-frontal ASSR index significantly predicted the comprehension score (t[20] = 2.58, p < 0.019, for the left-frontal ASSR index, t[20] = 0.76, p > 0.45, for the residualized response time, t[20] = −0.85, p > 0.40, for the residualized error rate, t[20] = −0.09, p > 0.92, for the scrambled-order response time, and t[20] = −0.74, p > 0.46, for the scrambled-order error rate). Note that two outliers were removed in the correlation analyses involving the “and”-detection performance so that all data points for all examined correlations remained within the respective 95% confident ellipses. Nevertheless, even if we included these outliers, the left-frontal ASSR index would still be the only significant predictor of the comprehension score in the multiple regression model.

We focused on the left-frontal ASSR index because our prior study (Mossbridge et al., 2013b) found that the ASSR from the left-frontal scalp region was sensitive to auditory-visual synchrony (Figure 3a) whereas the ASSR from the right-frontal scalp region was not (Figure 3b). Interestingly, the ASSR index from the right-frontal scalp region was also positively correlated with the comprehension score, r = 0.52, t(27) = 3.16, p < 0.004 (Figure 2d). What this means is that the right ASSR index was not sensitive to auditory-visual synchrony in a consistent manner across participants, that is, the right-frontal ASSR was greater in the AV-aligned condition for some participants, but greater in the AV-misaligned condition for others, with the right-frontal ASSR index (AV-aligned condition minus AV-misaligned condition) being largely evenly distributed in the positive and negative directions; nevertheless, individuals with less negative or more positive values of the right-frontal ASSR index yielded higher comprehension scores. This raises the question of whether the left-frontal and right-frontal ASSR indices reflect similar or distinct auditory-visual timing mechanisms relevant to reading comprehension. Although we cannot provide a definitive answer, we present additional analyses that may be informative.

Although the reading tasks and the measurement of the 40Hz ASSR sensitivity to music-visualizer synchrony were conducted at separate times, scalp EEG was recorded while participants performed the reading tasks in the correct-order and scrambled-order conditions. We previously reported the analysis of the event-related potentials (ERPs) recorded during the reading tasks, time-locked to participants’ responses to “and.” Specifically, the difference in a late (400–500 ms post-stimulus onset) ERP component between the scrambled-order and correct-order conditions at a mid-frontal electrode was strongly associated with the comprehension score (Mossbridge et al., 2013a). Because participants had to decode “and” in both the scrambled-order and correct-order conditions, but in the correct-order condition they also had to process “and” in the context of syntactic and semantic analyses for story comprehension, the late mid-frontal ERP difference likely reflected comprehension processes beyond word decoding. Thus, this ERP index allowed us to broadly assess whether the auditory-visual timing mechanisms reflected in the left-frontal and right-frontal ASSR indices are relevant to the process of word decoding (not reflected in the ERP index), the post-word decoding processes reflected in the ERP index, or additional post-word decoding processes not reflected in the ERP index.

We tested a multiple-regression model with the ERP index, the left-frontal ASSR index, and the right-frontal ASSR index as regressors to predict the comprehension score. The model accounted for a large proportion (62%) of variance in the comprehension score, F(3,24) = 15.42, p < 0.0001. Whereas the ERP index and the left-frontal ASSR index made significant contributions to the model (t[24] = 4.68, p < 0.0001, for the ERP index, and t[24] = 2.98, p < 0.007 for the left-frontal ASSR index), the right-frontal ASSR index did not (t[24] = 0.91, p > 0.37). This result is illustrated in Figure 3e and 3f. Whereas the left-frontal ASSR index was strongly correlated with the comprehension score after controlling for the ERP index and the right-frontal ASSR index (Figure 2e), the right-frontal ASSR index was uncorrelated with the comprehension score after controlling for the ERP index and the left-frontal ASSR index (Figure 2f). Taken together, these results suggest that the left-frontal ASSR index, which was consistently sensitive to auditory-visual synchrony across individuals (Figure 3a), is relevant to either the process of word decoding or a component of post-word decoding processes distinct from those reflected in the late mid-frontal ERP index. The right-frontal ASSR index provides only redundant information of lesser reliability.

Discussion

The perceptual system utilizes synchronized auditory-visual signals to integrate auditory and visual information belonging to the same object (e.g., Driver, 1996; Driver & Spence, 1998; Guzman-Martinez et al., 2012; Iordanescu et al., 2008, 2010, 2011; Molholm et al., 2004; Shimojo & Shams, 2001; Smith et al., 2007; Stein et al., 1989; Van der Burg et al., 2008, 2010). We considered the possibility that mechanisms that process crossmodal synchrony may also support cognition by facilitating the dynamic coordination of internal and sensory processes. We investigated reading comprehension as a case study partly because many of the component processes associated with reading (e.g., word decoding, semantic access, working memory, and semantic integration) could potentially benefit from temporally coordinated multimodal processing (see below), and partly because prior research using simple auditory and visual stimuli suggested a relationship between reading performance and the ability to compare or segregate auditory and visual dynamics (e.g., Birch et al., 1964; Hairston et al., 2005; Kujala et al., 2001; Rose et al., 1999; see the introduction section for details).

We used an individual-differences approach to test the hypothesis that auditory-visual synchrony processing may play a role in reading comprehension. Behaviorally, we demonstrated that a greater ability to judge auditory-visual synchrony (but not a greater ability to judge auditory-visual temporal order) predicted superior reading comprehension performance (Figure 1). Electrophysiologically, we demonstrated that greater sensitivity of a left-frontal 40Hz ASSR (Mossbridge et al., 2013b) to auditory-visual synchrony (vs. asynchrony) predicted superior reading comprehension performance (Figure 2). These associations were sizable in that the behavioral ability to judge auditory-visual synchrony accounted for 16%, and the left-frontal ASSR sensitivity to auditory-visual synchrony accounted for 25% of the variance in reading comprehension performance. The fact that robust relationships between auditory-visual synchrony processing and reading comprehension were demonstrated using both simple (flashes and beeps in Experiment 1) and complex (music and visualizer in Experiment 2) non-linguistic stimuli suggests that general auditory-visual synchrony processing mechanisms that operate under diverse circumstances are relevant to reading comprehension.

Future research needs to elucidate how auditory-visual synchrony processing may contribute to reading comprehension. For example, we have demonstrated that both behavioral and electrophysiological sensitivities to auditory-visual synchrony are associated with reading comprehension performance. Although this provides converging evidence for the association between auditory-visual synchrony processing and reading comprehension, because the two experiments involved separate groups of participants, the current results do not inform us as to whether behavioral synchrony judgments and the left-frontal ASSR index account for a common source or distinct sources of variance in reading comprehension performance. An interesting possibility is that these explicit and implicit sensitivities to auditory-visual synchrony may contribute to different processes relevant to reading comprehension (see below).

The current results are also entirely correlational. We are reasonably certain that the obtained association is specifically between auditory-visual synchrony processing and reading comprehension because we have ruled out potential contributions from individual differences in effort, arousal, carefulness (as non-significant covariates), and attention (due to similar attention demands for the synchrony-judgment task that is associated with reading comprehension and the temporal-order-judgment task that is not associated with reading comprehension). However, to demonstrate a causal effect of auditory-visual synchrony processing on reading comprehension, the operation of auditory-visual synchrony processing needs to be experimentally manipulated to see if reading comprehension is altered as predicted. For example, improving auditory-visual synchrony judgment with training (e.g., Powers et al., 2009) should improve reading. Conversely, altering auditory-visual synchrony perception using an adaptation procedure (e.g., Roach et al., 2010) may temporarily interfere with reading. Along these lines, it is interesting to note that, in a longitudinal study, musical training that improves attention to auditory-visual synchrony protected low-income children from the decline in reading performance exhibited by their musically untrained peers (Slater et al, 2014).

To understand the mechanisms through which auditory-visual synchrony processing is associated with reading comprehension, it would be informative to identify the neural sources of the left-frontal 40Hz ASSR sensitive to auditory-visual synchrony. This could be accomplished, for example, with electrocorticography (ECoG) using subdural surface and depth electrodes placed directly on or in the brains of epilepsy patients. ECoG allows recording of electrophysiological signals at high signal-to-noise ratio, 1 ms temporal resolution, and 3–5 mm spatial resolution for localizing neural sources (e.g., Brang et al., 2015). Using this technique, one could identify brain regions that generate 40Hz ASSR that is maximally modulated by auditory-visual synchrony, and then determine in which of those region(s) the modulation is most strongly associated with reading performance. The critical neural source(s) may include left posterior Superior Temporal Sulcus (left pSTS); pSTS activity tends to be modulated by multisensory synchrony (see Keetels & Vroomen, 2012, for a review), and left pSTS activity in particular reflects the reliability of speech-related audiovisual signals (Nath & Beauchamp, 2011) and accounts for individual differences in speech-related auditory-visual integration (Nath & Beauchamp, 2012; Nath et al., 2011). The identified neural sources could also be used as the seed to constrain an EEG source-modeling algorithm (e.g., Scherg, 1990; Oostenveld et al., 2011). Such a model would allow non-invasive assessments of the auditory-visual synchrony mechanisms that are closely associated with reading processes.

The neural sources of the association between auditory-visual synchrony processing and reading comprehension need to be interpreted in conjunction with functional considerations. To this end, it would be informative to investigate the functional sources of the association between auditory-visual synchrony processing and reading comprehension. Reading comprehension is thought to involve multiple perceptual and cognitive processes, including orthographic-to-phonological decoding (word decoding), semantic access, working memory, and higher-order processes that integrate meaning across sentences and paragraphs (e.g., Ferstl et al., 2008; Gough & Tunmer, 1986; Hoover & Gough, 1990; King & Kutas, 1995; Rapp et al., 2007; Friederici, 2012; van den Broek et al., 2005; Thompson-Schill et al., 2005; Martin et al., 1994).

Auditory-visual synchrony processing might primarily facilitate word decoding by appropriately synchronizing the generation of phonological representations with orthographic processing (e.g., Adams, 1990; Breznitz, 2002; Breznitz & Misra, 2003; Vellutino et al., 2004; Lehongre et al., 2011). In fact, the analysis of the current results in conjunction with our prior results has indicated that the left-frontal ASSR sensitivity to auditory-visual synchrony is associated with reading comprehension independently of a late mid-frontal ERP component that reflects comprehension processes beyond word reading (Mossbridge et al., 2013a). This outcome is consistent with the possibility that auditory-visual synchrony processing contributes to orthographic-to-phonetic decoding. Nevertheless, because word decoding likely involves semantic access (e.g., Hoover & Gough, 1990), auditory-visual synchrony processing may also contribute to higher-order reading-related processes distinct from those reflected in the ERP component. In particular, it might contribute to semantic access and/or working memory by providing synchronized multimodal cues that may facilitate access to long-term memory (for semantic retrieval) and/or may enhance activation of working memory mechanisms (e.g., Federmeier & Laszlo, 2009; Kutas & Federmeier, 2011; King & Kutas, 1995).

It would be informative to separately evaluate these component processes, such as evaluating word decoding using the phonetic decoding efficiency subset of the Test of Word Reading Efficiency (Torgensen et al., 1999), semantic access using the N400 ERP component (Kutas & Federmeier, 2011), and semantic working memory using a span task (e.g., Martin et al., 1994), to determine how the sensitivity to auditory-visual synchrony might be associated with these component processes. An interesting possibility is that the explicit crossmodal synchrony processing tapped by our behavioral task and the implicit processing tapped by our left-frontal ASSR index might be differentially associated with word decoding, semantic access, and/or semantic working memory. It is possible that the associations between the behavioral and/or the electrophysiological auditory-visual synchrony sensitivity and extended-text comprehension may persist after controlling for word decoding, semantic access, and semantic working memory. Such an outcome would suggest that crossmodal synchrony processing benefits higher-order processes that integrate causal and referential relationships to construct coherent interpretations (e.g., Gough & Tunmer, 1986; Rapp et al., 2007; van den Broek et al., 2005), potentially through appropriately synchronizing the generation of mental imagery, attention shifts, and/or eye movements with the progression of semantic integration across the extended text.

Whereas the current study focused on reading comprehension, there is evidence suggesting a close relationship between language comprehension and production (e.g., Gennari & MacDonald, 2009; Humphreys & Gennari, 2014). Thus, greater sensitivity to auditory-visual synchrony might benefit language production as well as comprehension. Note that prefrontal cortex, including the Left Inferior Frontal Gyrus (LIFG), appears to be commonly recruited for tasks requiring language comprehension, production, working memory, and inhibitory control (e.g., Thompson-Schill et al., 2005; Humphreys & Gennari, 2014). This raises the intriguing, albeit highly speculative, possibility that the mechanisms underlying the detection of crossmodal synchrony might play a general role in temporally coordinating sensory, attention, working memory, and inhibitory processes.

In summary, the current results provide converging behavioral and electrophysiological evidence suggesting that sensitivity to auditory-visual synchrony (vs. asynchrony) is associated with reading comprehension performance. These results suggest avenues for future research to investigate the causality as well as the neural and functional sources of this association.

References

  1. Adams MJ. Beginning to read: thinking and learning about print. Cambridge, MA: MIT Press; 1990. [Google Scholar]
  2. Bernier PM, Burle B, Hasbroucq T, Blouin J. Spatio-temporal dynamics of reach-related neural activity for visual and somatosensory targets. Neuroimage. 2009;47:1767–1777. doi: 10.1016/j.neuroimage.2009.05.028. [DOI] [PubMed] [Google Scholar]
  3. Birch HG, Belmont L. Auditory-visual integration in normal and retarded readers. American Journal of Orthpsychiatry. 1964;34:852–861. doi: 10.1111/j.1939-0025.1964.tb02240.x. [DOI] [PubMed] [Google Scholar]
  4. Brainard DH. The Psychophysics Toolbox. Spatial Vision. 1997;10:433–436. [PubMed] [Google Scholar]
  5. Brang D, Towle VL, Suzuki S, Hillyard SA, Di Tusa S, Dai Z, Tao J, Wu S, Grabowecky M. Peripheral sounds rapidly activate visual cortex: evidence from electrocorticography. Journal of Neurophysiology. doi: 10.1152/jn.00728.2015. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Breznitz Z. Asynchrony of visual-orthographic and auditory-phonological processes: an underlying factor in dyslexia. Reading and Writing, An Interdisciplinary Journal. 2002;15:15–42. [Google Scholar]
  7. Breznitz Z, Misra M. Speed of processing of the visual-orthographic and auditory-phonological systems in adult dyslexics: the contribution of “asynchrony” to word recognition deficits. Brain and Language. 2003;85:486–502. doi: 10.1016/s0093-934x(03)00071-3. [DOI] [PubMed] [Google Scholar]
  8. Carbonnell L, Hasbroucq T, Grapperon J, Vidal F. Response selection and motor areas: a behavioural and electrophysiological study. Clinical Neurophysiolgy. 2004;115:2164–2174. doi: 10.1016/j.clinph.2004.04.012. [DOI] [PubMed] [Google Scholar]
  9. Carver RP. Reading Rate - Theory, Research, and Practical Implications. Journal of Reading. 1992;36:84–95. [Google Scholar]
  10. Driver J. Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading. Nature. 1996;381:66–68. doi: 10.1038/381066a0. [DOI] [PubMed] [Google Scholar]
  11. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron. 2008;57:11–23. doi: 10.1016/j.neuron.2007.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Driver J, Spence C. Attention and the crossmodal construction of space. Trends in Cognitive Sciences. 1998;2(7):254–262. doi: 10.1016/S1364-6613(98)01188-7. [DOI] [PubMed] [Google Scholar]
  13. Federmeier KD, Laszlo S. Time for meaning: electrophysiology provides insights into the dynamics of representation and processing in semantic memory. In: Ross BH, editor. The Psychology of Learning and Motivation. San Diego, CA: Elsevier; 2009. pp. 1–44. [Google Scholar]
  14. Ferstl EC, Neumann J, Bogler C, von Cramon DY. The extended language network: a meta-analysis of neuroimaging studies on text comprehension. Human Brain Mapping. 2008;29:581–593. doi: 10.1002/hbm.20422. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Friederici AD. The cortical language circuit: from auditory perception to sentence comprehension. Trends in Cognitive Sciences. 2012;16(5):262–268. doi: 10.1016/j.tics.2012.04.001. [DOI] [PubMed] [Google Scholar]
  16. García-Pérez MA, Alcalá-Quintana R. On the discrepant results in synchrony judgment and temporal-order judgment tasks: a quantitative model. Psychonomic Bulletin & Review. 2012;19:820–846. doi: 10.3758/s13423-012-0278-y. [DOI] [PubMed] [Google Scholar]
  17. Gennari SP, MacDonald MC. Linking production and comprehension processes: the case of relative clauses. Cognition. 2009;111:1–23. doi: 10.1016/j.cognition.2008.12.006. [DOI] [PubMed] [Google Scholar]
  18. Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? Trends in Cognitive Sciences. 2006;10(6):278–285. doi: 10.1016/j.tics.2006.04.008. [DOI] [PubMed] [Google Scholar]
  19. Gough P, Tunmer W. Decoding, reading, and reading disability. Remedial and Special Education. 1986;7:6–10. [Google Scholar]
  20. Gutschalk A, Mase R, Roth R, Ille N, Rupp A, et al. Deconvolution of 40 Hz steady-state fields reveals two overlapping source activities of the human auditory cortex. Clinical Neurophysiology. 1999;110:856–868. doi: 10.1016/s1388-2457(99)00019-x. [DOI] [PubMed] [Google Scholar]
  21. Guzman-Martinez E, Ortega L, Grabowecky M, Mossbridge J, Suzuki S. Interactive coding of visual spatial frequency and auditory amplitude-modulation rate. Current Biology. 2012;22(5):383–388. doi: 10.1016/j.cub.2012.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Hairston WD, Burdette JH, Flowers DL, Wood FB, Wallace MT. Altered temporal profile of visual-auditory multisensory interactions in dyslexia. Experimental Brain Research. 2005;166:474–480. doi: 10.1007/s00221-005-2387-6. [DOI] [PubMed] [Google Scholar]
  23. Henderson JM, Luke SG. Stable individual differences in saccadic eye movements during reading, pseudoreading, scene viewing, and scene search. Journal of Experimental Psychology: Human Perception and Performance. 2014;40(4):1390–1400. doi: 10.1037/a0036330. [DOI] [PubMed] [Google Scholar]
  24. Herdman AT, Lins O, Van Roon P, Stapells DR, Scherg M, et al. Intracerebral sources of human auditory steady-state responses. Brain Topography. 2002;15:69–86. doi: 10.1023/a:1021470822922. [DOI] [PubMed] [Google Scholar]
  25. Hoover WA, Gough PB. The simple view of reading. Reading and Writing. 1990;2:127–160. [Google Scholar]
  26. Humphreys GF, Gennari SP. Competitive mechanisms in sentence processing: common and distinct production and reading comprehension networks liked to the prefrontal cortex. NeuroImage. 2014;84:354–366. doi: 10.1016/j.neuroimage.2013.08.059. [DOI] [PubMed] [Google Scholar]
  27. Iordanescu L, Grabowecky M, Franconeri S, Theeuwes J, Suzuki S. Characteristic sounds make you look at target objects more quickly. Attention, Perception & Psychophysics. 2010;72(7):1736–1741. doi: 10.3758/APP.72.7.1736. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Iordanescu L, Grabowecky M, Suzuki S. Object-based auditory facilitation of visual search for pictures and words with frequent and rare targets. Acta Psychologica. 2011;137:252–259. doi: 10.1016/j.actpsy.2010.07.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Iordanescu L, Guzman-Martinez E, Grabowecky M, Suzuki S. Characteristic sounds facilitate visual search. Psychonomic Bulletin & Review. 2008;15(3):548–554. doi: 10.3758/pbr.15.3.548. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Kayser J, Tenke CE, Gates NA, Kroppmann CJ, Gil RB, et al. ERP/CSD indices of impaired verbal working memory subprocesses in schizophrenia. Psychophysiology. 2006;43:237–252. doi: 10.1111/j.1469-8986.2006.00398.x. [DOI] [PubMed] [Google Scholar]
  31. Keetels M, Vroomen J. Perception of Synchrony between the Senses. In: Murray MM, Wallace MT, editors. The Neural Bases of Multisensory Processes. Boca Raton (FL): CRC Press; 2012. [Google Scholar]
  32. King JW, Kutas M. Who did what and when - using word-level and clause-level ERPs to monitor working-memory usage in reading. Journal of Cognitive Neuroscience. 1995;7:376–395. doi: 10.1162/jocn.1995.7.3.376. [DOI] [PubMed] [Google Scholar]
  33. Kleiner M, Brainard D, Pelli D. What’s new in Psychtoolbox-3? Perception 36 ECVP Abstract Supplement 2007 [Google Scholar]
  34. Kujala T, Karma K, Ceponiene R, Belitz S, Turkkila P, Tervaniemi M, Näätänen R. Plastic neural changes and reading improvement caused by audiovisual training in reading-impaired children. Proceedings of the National Academy of Sciences, USA. 2001;98(18):10509–10514. doi: 10.1073/pnas.181589198. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Kutas M, Federmeier KD. Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP) Annual Review of Psychology. 2011;62:621–647. doi: 10.1146/annurev.psych.093008.131123. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Lehongre K, Ramus F, Villiermet N, Schwartz D, Giraud AL. Altered low-gamma sampling in auditory cortex accounts for the three main facets of dyslexia. Neuron. 2011;72:1080–1090. doi: 10.1016/j.neuron.2011.11.002. [DOI] [PubMed] [Google Scholar]
  37. Martin RC, Shelton JR, Yaffee L. Language processing and working memory: neuropsychological evidence for separate phonological and semantic capacities. Journal of Memory and Language. 1994;33:83–111. [Google Scholar]
  38. Molholm S, Ritter W, Javitt DC, Foxe JJ. Multisensory visual-auditory object recognition in humans: a high-density electrical mapping study. Cerebral Cortex. 2004;14(4):452–65. doi: 10.1093/cercor/bhh007. [DOI] [PubMed] [Google Scholar]
  39. Mossbridge JA, Grabowecky M, Paller KA, Suzuki S. Neural activity tied to reading predicts individual differences in extended-text comprehension. Frontiers in Human Neuroscience. 2013a;7:655. doi: 10.3389/fnhum.2013.00655. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Mossbridge JA, Grabowecky M, Suzuki S. Seeing the song: Left auditory structures may track auditory-visual dynamic alignment. PLoS ONE. 2013b;8(10):e77201. doi: 10.1371/journal.pone.0077201. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Nath AR, Beauchamp MS. Dynamic changes in superior temporal sulcus connectivity during perception of noisy audiovisual speech. Journal of Neuroscience. 2011;31(5):1704–1714. doi: 10.1523/JNEUROSCI.4853-10.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Nath AR, Beauchamp MS. A neural basis for interindividual differences in the McGurk effect, a multisensory speech illusion. Neuroimage. 2012;59:781–787. doi: 10.1016/j.neuroimage.2011.07.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Nath AR, Fava EE, Beauchamp MS. Neural correlates of interindividual differences in children’s audiovisual speech perception. Journal of Neuroscience. 2011;31(39):13963–13971. doi: 10.1523/JNEUROSCI.2605-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Oostenveld R, Fries P, Maris E, Schoffelen JM. FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Computational Intelligence and Neuroscience. 2011;2011:1–9. doi: 10.1155/2011/156869. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Pastor MA, Artieda J, Arbizu J, Marti-Climent JM, Penuelas I, et al. Activation of human cerebral and cerebellar cortex by auditory stimulation at 40 Hz. Journal of Neuroscience. 2002;22:10501–10506. doi: 10.1523/JNEUROSCI.22-23-10501.2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Pelli DG. The VideoToolbox software for visual psychophysics: Transforming numbers into movies. Spatial Vision. 1997;10:437–442. [PubMed] [Google Scholar]
  47. Picton TW, John MS, Dimitrijevic A, Purcell D. Human auditory steady-state responses. International Journal of Audiology. 2003;42:177–219. doi: 10.3109/14992020309101316. [DOI] [PubMed] [Google Scholar]
  48. Powers AR, III, Hillock AR, Wallace MT. Perceptual training narrows the temporal window of multisensory binding. Journal of Neuroscience. 2009;29(39):12265–12274. doi: 10.1523/JNEUROSCI.3501-09.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Rapp DN, van den Broek P, McMaster KL, Panayiota K, Espin CA. Higher-order comprehension processes in struggling readers: a perspective for research and intervention. Scientific Studies of Reading. 2007;11:289–312. [Google Scholar]
  50. Roach NW, Heron J, Whitaker D, McGraw PV. Asynchrony adaptation reveals neural population code for audio-visual timing. Proceedings of the Royal Society B. 2010;278:1314–1322. doi: 10.1098/rspb.2010.1737. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Rose SA, Feldman JF, Jankowski JJ, Futterweit LR. Visual and auditory temporal processing, cross-modal tranfer, and reading. Journal of Learning Disabilities. 1999;32:256–266. doi: 10.1177/002221949903200307. [DOI] [PubMed] [Google Scholar]
  52. Ross B, Herdman AT, Pantev C. Right hemispheric laterality of human 40 Hz auditory steady-state responses. Cerebral Cortex. 2005;15:2029–2039. doi: 10.1093/cercor/bhi078. [DOI] [PubMed] [Google Scholar]
  53. Scherg M. Fundamentals of dipole source potential analysis. In: Grandori F, Hoke M, Roamni GL, editors. Auditory Evoked Magnetic Fields and Potentials: Advances in Audiology. Vol. 6. Basel: Karger; 1990. pp. 40–69. [Google Scholar]
  54. Shimojo S, Shams L. Sensory modalities are not separate modalities: plasticity and interactions. Current Opinion in Neurobiology. 2001;11:505–509. doi: 10.1016/s0959-4388(00)00241-5. [DOI] [PubMed] [Google Scholar]
  55. Slater J, Strait DL, Skoe E, O’Connell S, Thompson E, Kraus N. Longitudinal effects of group music instruction on literacy skills in low-income children. PLoS ONE. 2014;9(11):e113383. doi: 10.1371/journal.pone.0113383. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Smith EL, Grabowecky M, Suzuki S. Auditory-visual crossmodal integration in perception of face gender, Current Biology. 2007;17:1680–1685. doi: 10.1016/j.cub.2007.08.043. [DOI] [PubMed] [Google Scholar]
  57. Stein BE, Meredith ME, Huneycutt WS, McDade LW. Behavioral indices of multisensory integration: orientation to visual cues is affected by auditory stimuli. Journal of Cognitive Neuroscience. 1989;1:12–24. doi: 10.1162/jocn.1989.1.1.12. [DOI] [PubMed] [Google Scholar]
  58. Stein BE, Stanford TR. Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience. 2008;9:255–266. doi: 10.1038/nrn2331. [DOI] [PubMed] [Google Scholar]
  59. Tenke CE, Kayser J. Reference-free quantification of EEG spectra: Combining current source density (CSD) and frequency principal components analysis (fPCA) Clinical Neurophysiology. 2005;116:2826–2846. doi: 10.1016/j.clinph.2005.08.007. [DOI] [PubMed] [Google Scholar]
  60. Tenke CE, Kayser J. Generator localization by current source density (CSD): Implications of volume conduction and field closure at intracranial and scalp resolutions. Clinical Neurophysiology. 2012;123(12):2328–2345. doi: 10.1016/j.clinph.2012.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Thompson-Schill SL, Bedny M, Goldberg RF. The frontal lobes and the regulation of mental activity. Current Opinion in Neurobiology. 2005;15:219–224. doi: 10.1016/j.conb.2005.03.006. [DOI] [PubMed] [Google Scholar]
  62. Torgensen J, Wagner R, Rashotte C. Test of word reading efficiency. Austin, TX: Pro-Ed; 1999. [Google Scholar]
  63. Van den Broek P, Rapp DN, Kendeou P. Integrating memory-based and constructionist processes in accounts of reading comprehension. Discourse Processes. 2005;39:299–316. [Google Scholar]
  64. Van der Burg E, Olivers CNL, Bronkhorst AW, Theeuwes J. Pip and pop: Nonspatial auditory signals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance. 2008;34(5):1053–1065. doi: 10.1037/0096-1523.34.5.1053. [DOI] [PubMed] [Google Scholar]
  65. Van der Burg E, Cass J, Olivers CNL, Theeuwes J, Alais D. Efficient visual search from synchronized auditory signals requires transient audiovisual events. PLoS ONE. 2010;5(5):e10664. doi: 10.1371/journal.pone.0010664. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Van Eijk RLJ, Kohlrausch A, Juola JF, van de Par S. Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type. Perception & Psychophysics. 2008;70(6):955–968. doi: 10.3758/pp.70.6.955. [DOI] [PubMed] [Google Scholar]
  67. Vellutino FR, Fletcher JM, Snowling MJ, Scanlon DM. Specific reading disability (dyslexia): what have we learned in the past four decades? Journal of Child Psychology and Psychiatry. 2004;45:2–40. doi: 10.1046/j.0021-9630.2003.00305.x. [DOI] [PubMed] [Google Scholar]

RESOURCES