Impairments of Multisensory Integration and Cross-Sensory Learning as Pathways to Dyslexia

Abstract

Two sensory systems are intrinsic to learning to read. Written words enter the brain through the visual system and associated sounds through the auditory system. The task before the beginning reader is quite basic. She must learn correspondences between orthographic tokens and phonemic utterances, and she must do this to the point that there is seamless automatic ‘connection’ between these sensorially distinct units of language. It is self-evident then that learning to read requires formation of cross-sensory associations to the point that deeply encoded multisensory representations are attained. While the majority of individuals manage this task to a high degree of expertise, some struggle to attain even rudimentary capabilities. Why do dyslexic individuals, who learn well in myriad other domains, fail at this particular task? Here, we examine the literature as it pertains to multisensory processing in dyslexia. We find substantial support for multisensory deficits in dyslexia, and make the case that to fully understand its neurological basis, it will be necessary to thoroughly probe the integrity of auditory-visual integration mechanisms.

1. Introduction

"A primary disturbance in the ability to integrate stimuli from the two critical sense modalities, hearing and vision, may well serve to increase the risk of becoming a poor reader" Birch & Belmont, p. 858, 1963.

Learning to read is one of the great challenges faced by humans over the course of a lifetime of development. Given the enormous complexity of the stimulus set to be mastered, it is perhaps remarkable that the great majority of us manage to acquire this skill to such high degrees of proficiency. In fact, many will learn to do so in more than one language, and not an insignificant sub-population will manage it in many more again, some of which are even acquired in adulthood. This multilingual reading involves the learning of correspondences between entirely novel orthographies and phonologies, which implies a remarkable degree of ongoing plasticity for what is ultimately a large-scale cross-sensory learning task. A significant minority of individuals, however, despite being of normal intelligence and having adequate educational opportunity and ostensibly intact audition and vision, will struggle and ultimately fail to adequately acquire this fundamental skill set. Developmental dyslexia is one of the most common learning disorders, affecting between 5–17% of the population in the United States (S. E. Shaywitz & Shaywitz, 2003; S. E. Shaywitz et al., 1998). A huge puzzle is why individuals who appear to learn perfectly well in other domains, have such difficulties when it comes to the task of matching orthographic tokens (letters and words) with phonological inputs. At its core, efficient reading requires the ability to form reliable cross-sensory associations between speech-sounds and letter combinations. In the initial stages of learning, the child labors to remember the correspondences, but as learning progresses, these correspondences begin to be automated until ultimately, the simple sight of a given orthographic token activates the phonological representation. In this sense, learning to read ultimately relies on the formation of automatic multisensory representations. What is more, we would contend that this relationship persists into adulthood such that the expert reader continues to rely on the engagement of these multisensory representations (Carreiras, Quinones, Hernandez-Cabrera, & Dunabeitia, 2014). It is our view, therefore, that a multisensory approach is essential to understanding the complex process of reading and deficits therein.

Considerable research efforts have been directed at understanding the processing deficits that lead to reading difficulty, and numerous theories regarding core deficits have been proposed (Vellutino, Fletcher, Snowling, & Scanlon, 2004). These include deficits in associative learning (Gascon & Goodglass, 1970), rule learning (Manis et al., 1987), selective attention (Pelham & Ross, 1977), and attention shifting (Hari & Renvall, 2001), as well as processing deficits specific to the auditory or visual sensory systems (higher level of visual processing; visual perception (Morgan, 1896), visual attention, (Valdois, Bosse, & Tainturier, 2004), or low-level sensory processing; magnocellular system (Eden, Stein, Wood, & Wood, 1995; Stein, 2001) or auditory system (Hamalainen, Salminen, & Leppanen, 2012)). One of the more consistent findings is a phonological deficit (Frost, 1998; Kovelman et al.; Ramus & Szenkovits, 2008; M. Snowling, Bishop, & Stothard, 2000; Wagner, 1986), although whether this is best attributed to a breakdown at the level of information storage, representation, or retrieval, and whether it represents a more general deficit or is specific to speech sounds, remains a source of debate (Goswami, Fosker, Huss, Mead, & Szucs, 2011; Leppanen et al., 2012; Ramus & Szenkovits, 2008). While this work has certainly provided valuable knowledge of processing weaknesses that contribute to reading difficulty (for reviews see Hamalainen et al., 2012; Leppanen et al., 2012; Schulte-Korne & Bruder, 2010; Stein, 2001; Vellutino et al., 2004), consistent with the idea that there are many pathways to the dyslexia phenotype (aka “;cognitive subtypes of dyslexia” (Heim et al., 2008)), none have led to a fully satisfactory accounting of the disorder. In turn, despite the fact that reading is arguably a fundamentally multisensory process, our knowledge of multisensory processing in dyslexia is remarkably limited. Here we review this literature, such as it is. Although this literature comprises a still relatively small body of work, it makes a compelling case for the importance of fully understanding the role that multisensory processing plays in reading, and the extent to which its impairment in turn disrupts the acquisition of fluent reading skills. We argue for the need to fully characterize the role of multisensory processing deficits in dyslexia; undoubtedly such information is needed for the development of optimized interventions that 1) enhance audiovisual multisensory processing, and 2) lead to the effortless integration of phonological and orthographic signals that we believe is key to successful reading.

2. Multisensory processing, reading, and dyslexia

2.1. Audiovisual correspondences

The very first published study on audiovisual processing in reading impaired children that we are aware of came from our own institution, the Albert Einstein College of Medicine. In this 1963 monograph Birch and Belmont stated "among several possible causes for subnormality in learning to read could be a primary inadequacy in the ability to integrate auditory and visual stimuli" (p. 853). Extending their work on the typical development of audio-visual processing, they tested the ability of children with and without reading disability to match sequences of tones to dot patterns. Participants judged whether the intervals between tones matched the spacing (short or long) between dots. They reasoned that the ability to map temporally distributed auditory information onto spatially distributed visual information was essential to the acquisition of reading proficiency. An impressive 150 children (boys only) between the ages of 9 and 10 who were reading impaired, and 50 age and gender matched controls, were tested. The reading impaired group was defined as children with an IQ above 80, and raw reading scores in the lowest 10% on at least 3 out of 4 reading tests (sentence reading, word knowledge, word discrimination, and reading). Individuals in the control group had raw reading and IQ scores in the normal range. The reading impaired group was significantly less accurate in matching sequences of dots and sounds, even when children with lower (but still normal) IQ were excluded from the analysis (Birch & Belmont, 1964). Although the contribution of unisensory processing differences was not accounted for in this study, this early finding on audiovisual processing in reading impaired individuals is intriguing (see (Widmann, Schroger, Tervaniemi, Pakarinen, & Kujala, 2012) for a basic replication of this study). The authors argued that the data support the role of audiovisual integration deficits in reading disorders. While explanations of differential performance across groups unrelated to multisensory processing are plausible, the Birch and Belmont study is remarkable for representing an early foray into probing audiovisual processing deficits in dyslexia.

Additional support for difficulties in the mapping of cross-sensory correspondences in dyslexia comes from the observation that early difficultly with naming letters is a strong predictor of dyslexia (Elbro, Borstrom, & Petersen, 1998; Gallagher, Frith, & Snowling, 2000; Lyytinen et al., 2006; Scarborough, 1990). Thus even the very initial stages of learning auditory-visual associations have been linked to reading success (Ehri, 2005).

2.2. Orthographic-speech sound integration

For obvious reasons, the majority of studies on multisensory processing in dyslexic populations have involved letter-speech sound stimuli. One of the earliest researchers on the role of letter-speech sound integration in dyslexia and typical reading was the late Leo Blomert from Maastricht University in the Netherlands. Using electrophysiology, he and his colleagues probed the development of the automatic processing of graphemes-to-phonemes (D. Froyen, Van Atteveldt, Bonte, & Blomert, 2008; D. J. Froyen, Bonte, van Atteveldt, & Blomert, 2009). Novice readers (2nd graders), mid-level readers (5th graders), and reading experts (adults) were repeatedly presented with the same matching letter-speech sound pair, or just speech sounds, while recordings of their brain activity were made. Sometimes the sound changed, resulting in a break in the auditory pattern, which led to a mismatch between the speech sound and the grapheme in the multisensory stimulation condition. In the multisensory condition the visual stimulus could precede the auditory stimulus by 200 ms or be presented simultaneous with the auditory stimulus.

The idea was that a larger mismatch response (MMN for mismatch negativity) to the deviant would be observed in the multisensory condition if integration occurred, since the sound and letter differed, and that the timing of the multisensory influence would vary across age groups due to changes in the automaticity of the interaction of grapheme-phoneme representations. This is precisely what was found in this rather cleverly designed study. Adult readers had a larger AV deviant response in the early timeframe (~175ms) and little evidence for any response in the later timeframe (~650ms); novice readers (grade 2) didn't show any early stage integration, and only showed late integration when the visual stimulus preceded the auditory stimulus by 200 ms. In mid-level readers (5th graders) there was evidence for early integration for the 200 ms offset condition, and indication of late integration in the simultaneous condition. These data demonstrate that there is a prolonged developmental trajectory to automaticity in phoneme-grapheme mapping. Using the same approach, a group of 11 year old dyslexic readers (5th graders) was tested and their data compared with that from the 2009 developmental study (D. Froyen, Willems, & Blomert, 2011). Their responses were much like the novice readers (2nd graders), with integration only evident at the later timeframe, when there was a 200 ms delay between the visual and auditory stimuli. In contrast, they had a very similar MMN evoked by auditory-only speech sounds relative to their typically developing age-matched peers. This finding indicates a delay or an outright roadblock on the road to automaticity of phoneme-grapheme mapping in dyslexia, even when pure auditory processing of the same stimuli is intact. (Mittag, Thesleff, Laasonen, & Kujala, 2013) found similar results in adults who had received a diagnosis of dyslexia in childhood and had significantly poorer reading scores than the control group¹, indicating that this failure of automaticity observed in children with dyslexia persists into adulthood. (Blau et al., 2010) also tested letter-speech sound integration in children with dyslexia, measuring the neural response to congruent versus incongruent letter-speech sound pairs with hemodynamic neuroimaging (fMRI). Whereas differential responses as a function of congruency were observed in a typically developing age-matched control group, no congruency effects were seen for the dyslexia group in the superior temporal sulcus (STS), suggesting impaired integration in dyslexia. The dyslexic group also showed reduced responses to both of the unisensory conditions in auditory (anterior STG) and visual cortices (fusiform gyrus), and no significant difference between the auditory-alone and the multisensory conditions in auditory cortices. A similar lack of a congruency effect was also found in a group of adults with dyslexia, in a study by the same research group (Blau, van Atteveldt, Ekkebus, Goebel, & Blomert, 2009). Thus, remarkably, these studies reveal that there is a lack of differential processing of congruous versus incongruous letter-speech sound pairs in children and adults with dyslexia.

A consequence of the strong and automatic associations that develop between graphemes and phonemes (or spoken and written words) is that presentation of one (a speech sound) can lead to the cross-sensory activation of the representation of the other (a grapheme). In the context of auditory rhyming tasks, a region of visual cortex specifically engaged during visual-orthographic processing (the visual word form area; VWFA) has been shown to also respond when only spoken language stimuli are presented (Cone, Burman, Bitan, Bolger, & Booth, 2008). That is, under these circumstances this visual word area responds to spoken words. (Desroches et al., 2010) found that activation of the VWFA (in the left fusiform gyrus) during an auditory rhyme judgment task (e.g. gate-hate vs. press-list) was impaired in a cohort of 9 to 15 year-old children with dyslexia compared to typically developing age-matched controls. It is worth highlighting that activation of the superior temporal gyrus (STG), which is associated with phonological processing, did not differ between the groups. These findings indicate that the typically strong association between phonological and orthographic representations is weakened or absent in dyslexia, even when auditory processing regions are activated in a typical manner.

This as yet small literature reveals that a synesthetic relationship between phonemes and corresponding graphemes evolves over the schooling years during typical development, but that this is not achieved in individuals with dyslexia. Indeed, further support is found in the fact that reading impaired individuals are worse at recognizing mismatches between written and spoken words or pseudowords (words that meet the phonological criteria of a certain language without being an actual word in the language) than typically developing controls (Fox, 1994; M. J. Snowling, 1980). A key question is just how domain-specific multisensory processing deficits in dyslexia are. The above data could be accounted for by an insult to the regions involved in the processing and integration of graphemes and phonemes (e.g., VWFA, pSTS, or parietal regions), or the pathways connecting them. Alternatively they may represent a more general deficit in the integration of multisensory information. If the latter is the case, then impaired multisensory processing in dyslexia should also be observed for different multisensory configurations. Ideally, simple and semantically unrelated stimuli would be used to test this hypothesis (Brandwein et al., 2013; Brandwein et al., 2011; Molholm, Ritter, Javitt, & Foxe, 2002).

Although almost all studies directly testing the integrity of multisensory integration in dyslexia have involved speech sound-grapheme combinations or audio-visual speech, there is one exception that we are aware of. (Harrar et al., 2014) looked at multisensory facilitation of reaction-times in college students with a childhood diagnosis of dyslexia. The reading impaired participants had reading scores that were lower than those of the typically developing control group, but that still fell within the normal (unimpaired) range. Using "miller's race model"² to test the extent to which responses to multisensory stimuli (a white noise burst and a Gabor patch) were speeded compared to responses to unisensory stimuli (the white noise burst or the Gabor patch), they found reduced multisensory speeding in the remediated dyslexia group. Further, across all participants, extent of multisensory facilitation was found to correlate with "discrepancy" scores (the discrepancy between literacy scores and scores on a non-verbal test (Matrices)). Thus even in this remediated dyslexia group of adults, there was evidence for impaired multisensory integration of simple non-linguistic audio-visual stimuli, and across the study cohort there was indication that multisensory integration was related to reading ability. We suspect that probes of basic multisensory processing in children with dyslexia, prior to extensive intervention, would reveal even greater multisensory integration deficits for simple audio-visual stimuli.

2.3. Audiovisual speech processing

In studies of audiovisual speech integration, the focus is on an individual’s tendency to integrate the redundant speech cues that are contained in speech sounds and in the visual gestures made in producing these sounds, so-called articulatory gestures. Similar to speech sounds and letters, optimized AV speech integration develops over the course of many years (McGurk & MacDonald, 1976; Ross et al., 2011; Sekiyama & Burnham, 2008). However, this process begins much earlier than reading, with some level of sensitivity to the congruency between the sounds of certain vowels and their corresponding articulations already present in infants as young as 2 months (Patterson & Werker, 2003) and even, it has been argued, in newborns (Aldridge, Braga, Walton, & Bower, 1999). We turn now to this literature, while keeping in mind that the acquisition of these multisensory associations might be considerably easier because 1) the speech sounds and mouth gestures are causally related, 2) audio-visual speech is encountered beginning in infancy and on a very regular basis, and 3) the learning of these relationships is largely implicit rather than explicit.

Initial interest in AV speech processing in dyslexia came from the notion that since the majority of individuals with dyslexia have phonological processing deficits, they might actually rely to a greater extent on visual-articulatory cues during AV speech processing. De Gelder and Vroomen (de Gelder & Vroomen, 1998) tested 9-to-14 year-old poor readers and separate age and reading level matched TD control groups in an AV speech perception task. Poor readers were defined as children with normal intelligence (IQ>85) whose performance on a standardized reading test lagged 1–5 years behind their peers. Two sublexical speech syllables, ba and da, served as the end points of a 9-point continuum in which the second resonant frequency of the stimulus systematically varied from one end point to the other ('ba' to 'da'). These could be accompanied by visual articulations of /ba/ or /da/. Participants made a forced-choice decision of “ba” or “da”. Despite poor readers being less accurate at identifying the visual-alone articulations, the influence of visual articulations on the identification function did not differ across groups (indexed by measuring the difference between the auditory-alone function and the audio-visual function). Thus, by this metric, audiovisual multisensory speech processing appeared relatively intact in dyslexia. Likewise, Baart and colleagues (Baart, de Boer-Schellekens, & Vroomen, 2012) found that reading impaired adults were as susceptible to the influence of a visual speech adapter (a video of someone saying "aba" presented in the context of an ambiguous speech token falling between "aba" and "ada") on identification of auditory stimuli from a 9-point /aba/ to /ada/continuum as were non-reading impaired controls, showing an influence of visual speech articulations on auditory speech perception in reading impaired adults.

Using fMRI, (Pekkola et al., 2006) compared brain activity in response to congruent and incongruent audio-visual speech (e.g. the sound /a/and a video of someone saying /y/) between adults with dyslexia (with a childhood diagnosis, and neuropsychological test scores falling at least 1 standard deviation below normal on at least 3 out of 4 reading related tasks) and typically developing controls. To ensure that the stimuli were attended, participants were instructed to indicate when the stimulation condition (congruent, incongruent, or no stimulation) switched. Individuals with dyslexia showed increased activation relative to controls for both matching and conflicting stimuli in a number of brain areas including the motor speech regions and the left inferior parietal lobule. Comparison of congruency effects between the groups further revealed a number of differences in cortical activation patterns. Individuals with dyslexia had larger responses than controls to congruent pairs in bilateral anterior and middle auditory regions (superior and middle temporal gyri) and right inferior frontal cortex (among other regions), and larger responses than controls to incongruent pairs in ventral visual cortex, supplementary motor area, and anterior cingulate cortex. Further, the level of activation within visual and auditory cortices and Broca’s area (a region associated with the processing and production of speech) correlated with dyslexic adults’ phonological processing abilities. However, the study sample was very small (10 versus 10) for group comparisons and thus one must be cautious in extending these findings to the dyslexia population at large. Further, with no unisensory control conditions or related behavioral data, it is very difficult to speculate on the underlying causes or implications of these differential activation patterns.

Individuals with dyslexia have particular difficulty processing auditory speech when it is presented in a noisy background (Bradlow, Kraus, & Hayes, 2003; Cunningham, Nicol, Zecker, Bradlow, & Kraus, 2001; Dole, Hoen, & Meunier, 2012; C. King, Warrier, Hayes, & Kraus, 2002; Ziegler, Pech-Georgel, George, & Lorenzi, 2009). Typically, visual-articulatory information is effectively used to recover speech information from a noisy auditory signal (Foxe et al., 2013; Ma, Zhou, Ross, Foxe, & Parra, 2009; Ross et al., 2011; Ross, Saint-Amour, Leavitt, Javitt, & Foxe, 2007; Ross, Saint-Amour, Leavitt, Molholm et al., 2007). Several studies have examined the ability of individuals with dyslexia to similarly benefit from these multisensory cues. Ramirez and Mann (Ramirez & Mann, 2005) presented short syllables (e.g. ra, ka, wa) in different signal-to-noise levels and their corresponding visual articulations in unisensory and multisensory configurations to adults with and without dyslexia (diagnosed in childhood, and with low reading scores compared to overall IQ in adulthood). Individuals with dyslexia did not benefit from the visual cues as effectively as controls, a finding moderated however by the fact that they were also poorer at accurately identifying the visual cues when presented in isolation. Another study that focused on adult dyslexics (with a childhood diagnosis but whose reading scores fell within the normal range), failed to find differences in the use of visual-articulatory cues during a speech-in-noise task (Megnin-Viggars & Goswami, 2013) in which participants performed a 2-alternative forced-choice word identification task. However, both control and dyslexic groups had near ceiling performance in the AV condition, suggesting that the task became trivial when both auditory and visual information was available. It is possible that a more demanding task would have revealed group differences. Thus, unfortunately, for methodological reasons, we are reluctant to draw strong conclusions regarding the integration of visual articulatory and auditory speech information during speech-in-noise tasks from either of these studies.

To assess multisensory integration of audio-visual speech cues in reading impaired children, Hayes and colleagues investigated the so-called McGurk illusion in quiet and noise (Hayes, Tiippana, Nicol, Sams, & Kraus, 2003). Children in the reading impaired group had reading/spelling performance that was at least 1 standard deviation lower than their overall IQ. The McGurk illusion occurs when participants see and hear incongruent speech (/pa/ and /ka/), and report hearing a third speech segment (/ta/), a fusion (pka), or a speech sound that more closely matches the visual than the auditory input (McGurk & MacDonald, 1976; Saint-Amour, De Sanctis, Molholm, Ritter, & Foxe, 2007). The control group identified the visual-alone stimuli more accurately than did the reading impaired children, and there was no group difference in identification of the auditory-alone stimuli. Of most interest for the purpose of evaluating the integrity of multisensory processing in dyslexia, however, are the types of responses made for the bisensory incongruent (McGurk) conditions; that is, whether these identifications matched the auditory, the visual, or some combination of the auditory and visual speech stimuli, and how these patterns differed across groups. For no-noise and low-noise conditions, there were no group differences. For the high-noise level, visual based identifications were significantly higher in the reading impaired compared to the TD group (~74 versus 63%, estimated from the graphic representation in Figure 1 of the paper), whereas audiovisual combination/fusion identifications were significantly greater in the TD compared to the reading impaired group. Overall the responses to the McGurk stimuli revealed that higher noise levels lead to a greater reliance on visual cues in both groups (despite poorer visual-alone performance in the reading impaired group), and that reading impaired individuals had less of a tendency to actually integrate the audio-visual speech cues under noisy conditions.

As can be seen from the above, the evidence is equivocal when it comes to the integrity of audio-visual speech integration in dyslexia. Although some data are consistent with a deficit in multisensory speech processing, interpretation is often clouded by the possibility of unisensory explanations. In the Hayes study described above (Hayes et al., 2003), however, the decrease in audiovisual fusions compared to typical readers could not be accounted for by unisensory differences. This study, we believe, provides the most direct evidence for reduced audiovisual integration of speech in a reading impaired group. Additional investigation is nonetheless necessary to fully characterize the extent of audio-visual speech processing impairment in dyslexia.

3. The impact of multisensory training on reading in individuals with dyslexia

A promising set of studies suggests that multisensory training significantly improves reading in individuals with dyslexia (Ecalle, Magnan, Bouchafa, & Gombert, 2009; Kast, Baschera, Gross, Jancke, & Meyer, 2011; Veuillet, Magnan, Ecalle, Thai-Van, & Collet, 2007). One study in particular stands out for using non-linguistic stimuli and thus deconfounding the training from experience with the stimuli of interest. (Kujala et al., 2001) enrolled 48 7-year-olds with dyslexia (determined by school referral) into their training study. Half were trained to follow and categorize audiovisual sequences and the other half served as non-treatment controls. Children were presented with sequences of 3 to 15 sounds, and their corresponding graphic representation. Features of sounds such as pitch, duration and intensity were represented visually by the vertical position, lengths and thickness of rectangles. Over the course of 7 weeks each participant was presented with two 10-minute interactive computer based training sessions per week. For the first four training sessions, participants made a forced choice as to which of two visual patterns matched an auditory sequence. For the remaining 10, participants were presented the visual pattern and instructed to follow it from left to right as a sequence of sounds was being played, and to indicate when there was a mismatch. Pre and post testing revealed significantly greater improvement of reading-skills (e.g., correct reading of words; reading speed) for the training group. Further, evidence for the impact of the training was seen in ERP recordings made from a subset of the participants before and after the training period. Before training, MMNs elicited by infrequent order-reversals of tone pairs were highly similar between the two dyslexia groups. After training, however, MMN amplitude increased in the training group, leading to a significant group difference. Thus the audiovisual training was also found to impact auditory neural representations. These data are consistent with other training programs that have used linguistic stimuli and similarly shown improvement in reading or other skills that are impaired in dyslexia (phonological awareness; spelling). The most successful of these seem to direct attention toward the phonological feature level of speech sounds and their corresponding graphemes, usually focusing on voicing contrasts such as 'pa' versus 'ba' (Ecalle et al., 2009; Veuillet et al., 2007). For example, Ecalle and colleagues compared the effects of two training programs on reading performance in children with dyslexia (13 in each group, second graders). Audio-visual training involved matching of spoken to written syllables, and stressed discrimination of voicing pairs (ba versus pa). A control training program involved reading single words on a computer screen, with computer generated speech feedback. Training was performed over a period of 5 weeks, with half an hour of training 4 times a week. The audio-visual program led to greater improvement on tests of reading. While the control condition also appeared to involve multisensory elements, the audio-visual training program more specifically targeted direct speech sound to grapheme matching; the authors hypothesized that the requirement to process the auditory and visual representations of a specific phonetic feature lead to improved speech sound to grapheme binding. It is notable that in both the Kujala and Ecalle studies, the training programs focused attention on relatively small units of input, and involved the simultaneous presentation of the auditory and visual stimuli, both of which might be key to a successful multisensory training program. Lending further support for a key role for multisensory training in remediation of reading, a report from the National Reading Panel (NRP, 2000) concluded that teaching phonemic awareness and/or explicitly teaching letter-sound correspondences ("phonics") leads to significant improvements in reading skills. Further, they reported that even phonemic awareness instruction, which on the face of it is purely auditory, was most effective when the letters were present as well.

Thus, there is compelling evidence for the efficacy of multisensory training programs in remediating reading scores in dyslexia. At the same time, in order to confidently demonstrate a role for multisensory training on reading it will be necessary to parse the respective roles of multisensory aspects of such training programs versus other more general features, such as performing a challenging matching task or narrowing the focus of attention. Acquiring objective metrics of multisensory processing before and after training will also be key to drawing conclusions regarding whether changes in multisensory processing play a role in remediating reading deficits, and to understanding the mechanism of action of treatment.

4. The multisensory temporal window of integration

Learning to read is a temporally constrained process that requires matching dynamic and rapid speech sounds with the correct orthography. Across a number of sensory/perceptual processing domains, it is commonly assumed that there is a temporal window of integration (TWIN) within which inputs tend to be integrated into one perceptual unit (Winkler, Czigler, Jaramillo, Paavilainen, & Naatanen, 1998; Yabe, Tervaniemi, Reinikainen, & Naatanen, 1997; Yabe et al., 1998; Zwislocki, 1960), and it is further assumed that this principle extends to multisensory inputs (A. J. King & Palmer, 1985; Meredith, Nemitz, & Stein, 1987; Spence & Squire, 2003). This multisensory TWIN is clearly important for the proper binding of speech sounds to print in learning to read. It has been hypothesized that anomalies in the size of the TWIN contribute to reading difficulty. That is, that a too large or overly variable window might lead to greater errors in the accurate pairing of orthography and speech sounds.

The size of the TWIN is typically tested using a temporal order judgment (TOJ) task, in which participants are presented with pairs of stimuli and judge which came first (Megevand, Molholm, Nayak, & Foxe, 2013). The onset asynchrony necessary to make reliable judgments at above chance levels is considered to represent the TWIN (although this might be a conservative judgment since the task directs participants to specifically not integrate the inputs). (Hairston, Burdette, Flowers, Wood, & Wallace, 2005) presented simple lights and sounds to typical and dyslexic adults and had them perform a TOJ task for stimulus pairs presented at stimulus onset asynchronies (SOAs) ranging from 0 to 350 ms. Individuals with dyslexia required a larger SOA to reach a 74% accuracy criterion than did typically developing controls (63 versus 47 ms). It should be mentioned that though IQ was considered in the 'normal' range for all participants (IQ>70), since it was not specifically matched between groups, it is possible that IQ differences partially account for differences in the TWIN in Hairston and colleagues. (Laasonen, Service, & Virsu, 2002) also looked at the multisensory TWIN in dyslexic adults and age and IQ matched controls using a TOJ task (they set criterion at 84% accuracy). Whereas audio-tactile temporal order judgments required a longer SOA for the dyslexic group (187 versus 130 ms), in contrast to Hairston et al., they did not find evidence for a statistically different TWIN for auditory and visual stimuli. However, the audiovisual TOJ data were in the direction of larger TOJs for dyslexic adults. Further, with only 16 adults with dyslexia (versus 36 adults in Hairston et al), it is possible that larger participant numbers would have revealed statistical differences. Thus, although the data thus far on the audiovisual TWIN in dyslexia are equivocal, there is indication that the multisensory TWIN differs in dyslexia, and this could well contribute to difficulties in learning of grapheme-phoneme associations.

5. Assessing functional activation and connectivity of the reading network in dyslexia

Reading requires effective neural communication across a network of spatially segregated brain regions. Regions strongly implicated in reading include left inferior frontal cortex, and left temporo-parietal (dorsal) and occipito-temporal (ventral) cortices. The inferior frontal region (i.e. Broca’s area) is associated with sequencing, articulation, silent reading and naming (Fiez & Petersen, 1998; Murphy et al., 1997). The dorsal circuit, which includes the posterior superior temporal gyrus, inferior parietal lobe, the angular gyrus and the supramarginal gyrus, has been associated with the integration of orthography and phonology (Schlaggar & McCandliss, 2007; Simos et al., 2002). And the left occipito-temporal sulcus of the ventral visual pathway, the aforementioned visual word form area (VWFA), is central to the rapid processing of written words and letter strings (Allison, Puce, Spencer, & McCarthy, 1999; Cohen & Dehaene, 2004; Dehaene & Cohen, 2011; Dehaene, Le Clec, Poline, Le Bihan, & Cohen, 2002).

In dyslexia, left posterior regions tend to be underactivated in comparison to typical readers (Brunswick, McCrory, Price, Frith, & Frith, 1999; Landi, Frost, Mencl, Sandak, & Pugh, 2013; Langer, Benjamin, Minas, & Gaab, 2013; Richlan, Kronbichler, & Wimmer, 2009, 2011; B. A. Shaywitz et al., 2002), whereas some studies find that left frontal regions (the IFG) show hyper-activation (Pugh, Mencl, Shaywitz et al., 2000) and others hypo-activation (Norton et al., 2014) as a function of reading disability. There is also some evidence for compensatory involvement of right hemisphere homologues of the left posterior reading network (Pugh, Mencl, Jenner et al., 2000; Pugh, Mencl, Shaywitz et al., 2000), although this finding is somewhat controversial (Richlan et al., 2009).

Advances in structural neuroimaging methods, such as diffusion tensor imaging (DTI), have enabled examination of the integrity of anatomical connectivity in dyslexia. A comprehensive review of DTI studies on dyslexia by Vandermosten and colleagues (Vandermosten, Boets, Wouters, & Ghesquiere, 2012) indicates that the most reliable finding to date is of lower values of white matter integrity (functional anisotropy (FA) values) within left temporo-parietal and left inferior frontal areas.. Further, lower FA values in left temporo-parietal and frontal areas correlate with poorer reading in both individuals with and without dyslexia. Although precise fiber tracts can be difficult to identify due to their overlapping structure, the arcuate fasciculus, a large white matter tract that connects posterior STS and the IFG (Yeatman et al., 2011) has been most strongly implicated by these findings (for a review see Vandermosten, Boets, Wouters et al., 2012). The partially overlapping corona radiata has also been identified as showing lower FA values in dyslexia compared to TD controls. The inferior fronto-occipital fasciculus has also been implicated, which projects from visual regions including the VWFA to the IFG (Vandermosten, Boets, Poelmans et al., 2012; Yeatman, Rauschecker, & Wandell, 2013). These data reveal potential weaknesses in key nodes of the reading network in dyslexia and the pathways that connect them. The temporo-parietal focus of reduced connectivity is consistent with deficits in the neural architecture involved in integrating multisensory inputs. However, work specifically testing the relationship between performance on multisensory tasks and the integrity of this pathway is needed to confirm such a role.

A highly informative approach to understanding the multisensory reading network will be to use a multimodal approach that combines EEG, fMRI, and DTI measures. With these complementary techniques it will be possible to thoroughly map the flow of information through the reading network, in order to identify both when and where processing weaknesses emerge. We believe that a key component here will be to examine the functional network as individuals learn cross-modal associations, and to determine the impact of this learning on multisensory integrative processes.

6. Consideration of subphenotypes and comorbidities within dyslexia

Dyslexia is a heterogeneous disorder with multifactorial etiology (Pennington, 2006). The field is increasingly aware that there are likely many sub-phenotypes within the dyslexia spectrum (Heim et al., 2008), and that comorbidity with developmental disorders such as dyscalculia, specific language impairment or ADHD need to be taken into account when considering the neurobiological pathways to this condition (Reid, Szczerbinski, Iskierka-Kasperek, & Hansen, 2007; van Bergen, van der Leij, & de Jong, 2014). Unfortunately this is easier said than done. For example, estimates of co-morbidity of dyslexia and ADHD range from 25 to 40% (August & Garfinkel, 1990; Dykman & Ackerman, 1991; Gayan et al., 2005; Gilger, Pennington, & DeFries, 1992; Willcutt & Pennington, 2000), and yet the majority of dyslexia studies do not consider ADHD status. In general, the few studies that have directly compared “pure” dyslexia and dyslexia with ADHD report additional deficits in executive function, mostly inhibition, and more severe symptoms of dyslexia in the comorbid ADHD group (Nigg, Hinshaw, Carte, & Treuting, 1998; Shanahan et al., 2006; Willcutt, Pennington, & DeFries, 2000; Willcutt, Pennington, Olson, Chhabildas, & Hulslander, 2005). Intriguingly, preliminary data from our lab suggests clear segregation in performance on a multisensory task as a function of whether or not dyslexic individuals have a comorbid diagnosis of ADHD. More specifically, we find that children with pure dyslexia are severely impaired in their ability to benefit from multisensory cues on a multisensory speech-in-noise word identification task (see Foxe et al., 2013 for a full description of the paradigm), whereas children with co-morbid ADHD are not. What is more, studies using functional and anatomical MRI suggest that the affected neural regions differ between individuals with dyslexia subgrouped as a function of their profile of cognitive deficits (Jednorog, Gawron, Marchewka, Heim, & Grabowska, 2013; van Ermingen-Marbach, Grande, Pape-Neumann, Sass, & Heim, 2013). Thus it is clear that is necessary to thoroughly characterize dyslexia participants and consider the relationship between specific co-morbidities/cognitive deficits (e.g., poor working memory) and the underlying neurobiological pathway.

7. Summary and conclusions

There are likely to be many routes to the behavioral phenotype that defines Dyslexia. One highly plausible pathway is impaired multisensory integration. This could be a very specific impairment resulting from impaired connectivity between phonological and graphemic representations. For example, the vertical occipital fasciculus connecting the VWFA in the ventral visual stream and the angular gyrus in parietal cortex (Yeatman et al., 2013) could be selectively damaged. Alternately, and consistent with findings from DTI studies, the posterior arcuate fasciculus connecting phonemic processing regions (and more general auditory processing regions) with multisensory integration regions in parietal cortex could be selectively damaged (Vandermosten, Boets, Wouters et al., 2012). Indeed, there is now highly compelling evidence for reduced integration of grapheme-phoneme inputs in dyslexia. Given the evidence to date, however, it is not possible to conclude if reduced grapheme-phoneme integration represents just such a specific deficit, or if it is the consequence of a more fundamental, and thus far reaching, impairment in audiovisual multisensory integration. It is conceivable that the alterations in neurodevelopmental processes that lead to the learning difficulties associated with dyslexia have widespread impact on sensory integration; in our own work we have observed broadly impaired multisensory integration in other developmental disorders such as autism (Brandwein et al., 2013; Foxe et al., 2013; Russo et al., 2010). Audiovisual multisensory remediation training programs that have required participants to report on auditory to visual correspondences, with the use of simple nonlinguistic stimuli, have led to improved reading in individuals with dyslexia. Reading improvement following this nonlinguistic training program suggests a role for a more general multisensory processing deficit in dyslexia, at least when it comes to the acquisition of multisensory correspondences (although non multisensory aspects of the training could also account for some of the improvement). In the reviewed literature we see a promising start to characterizing the role of deficits in multisensory processing in dyslexia. But clearly much works remains to: 1) Determine whether this represents a general versus a specific deficit. It is our contention that the use of simple and semantically unrelated stimuli is necessary to understand the fundamentals of multisensory processing in dyslexia (Brandwein et al., 2013; Brandwein et al., 2011; Molholm et al., 2002), a notion that receives some support from the recent finding of Harrar and colleagues (2014). And, 2) unpack the respective roles of impaired learning of multisensory associations versus multisensory integrative processes. In turn, as we parse these roles, we can expect to develop biomarkers of multisensory dysfunction in dyslexia, so that individual remediation programs can be titrated to best suit the individual.

Highlights.

• learning to read relies on the formation of automatic multisensory representations

• knowledge of multisensory integration in dyslexia is remarkably limited

• extant literature is consistent impaired multisensory processing in dyslexia

• need to probe if this is specific to speech and graphemes or a more general deficit