Unification of Sentence Processing via Ear and Eye: An fMRI Study

Abstract

We present new evidence based on fMRI for the existence and neural architecture of an abstract supramodal language system that can integrate linguistic inputs arising from different modalities such that speech and print each activate a common code. Working with sentence material, our aim was to find out where the putative supramodal system is located and how it responds to comprehension challenges. To probe these questions we examined BOLD activity in experienced readers while they performed a semantic categorization task with matched written or spoken sentences that were either well-formed or contained anomalies of syntactic form or pragmatic content. On whole-brain scans, both anomalies increased net activity over non-anomalous baseline sentences, chiefly at left frontal and temporal regions of heteromodal cortex. The anomaly sensitive sites correspond approximately to those that previous studies (Constable et al., 2004, Michael et al., 2001) have found to be sensitive to other differences in sentence complexity (object relative minus subject relative). Regions of interest (ROIs) were defined by peak response to anomaly averaging over modality conditions. Each anomaly-sensitive ROI showed the same pattern of response across sentence types in each modality. Voxel-by-voxel exploration over the whole brain based on a cosine similarity measure of common function confirmed the specificity of supramodal zones.

Supramodal potential, or the ability to glean similar information from spoken and written forms of a message, is an essential characteristic of the language brain, making it possible to convey linguistic messages by writing as well as speech. Of course, many perceptual abilities, such as object and person perception, also exhibit supramodal potential in the sense that people can recognize the same entity on the basis of sensory input in various modalities. Supramodality in language is special in requiring for its realization a long period of learning and instruction. Clearly, the human brain is not automatically adapted for reading and writing as soon as a person is able to speak and understand speech. Yet, only in recent years has the neural architecture of the supramodal language system and its development been an object of study in its own right. In undertaking this research, we exploit functional neuroimaging (fMRI) to study the cerebral interrelations between speech and print modalities in persons who have learned to read fluently to determine how the cerebral network that supports reading comprehension is aligned with the network for comprehension of spoken language. We adopted a strategy motivated by a number of previous studies: we compare difficult language processing to easy language processing with closely matched materials and seek the areas where speech and print both produce the same type of response in brain. If difficult language processing causes extra effort to be expended in regions of brain whose job it is to perform the computations associated with the language processing task, then the comparison of difficult to easy will pinpoint loci of relevant processing. We can then look for regions in which speech and print produce similar deflections under conditions of stress in order to identify tissue that may be part of a supramodal system.

In the next two sections, we review prior work that motivates our approach. A variety of evidence can now be cited pertaining to convergence of activity stemming from speech and print inputs on common populations of neurons. Earlier research with neuroimaging tools has examined the neural response to spoken and printed material of widely varying sorts. Since our concern is with comprehension at the sentence level, our review will be selective, focusing on studies exploiting sentence material, especially those incorporating specific challenges to comprehension, especially due to syntactic complexity, such as posed by the contrast between object relative clauses and subject relatives (Constable et al., 2004; Michael, Keller, Carpenter, and Just, 2001).¹ In undertaking the present study, we aimed to extend and generalize the findings of these studies to another type of challenge to comprehension, sentences containing anomalies of morpho-syntax and of pragmatics (Caplan, 2004; Ni et al., 2000; Carpentier et al., 2001; Homae et al., 2002).

Motivation from Comparative Anatomy and Physiology

We use the term heteromodal association cortex to refer to the probable loci of convergence in humans of neural signals coming from different sensory regions as indicated by neurophysiological studies on nonhuman primates. Because the neurophysiological work is not based on humans, the boundaries of heteromodal cortex in the human brain are only approximately known. Regions that have undergone expansion in the human brain relative to other primates, such as the posterior parietal region and prefrontal region, comprise zones that are certainly hetermodal (Geschwind, 1965; Mesulam, 1998). It is generally considered that heteromodal cortex includes portions of temporal cortex, including Wernicke’s area, portions of the inferior parietal lobe, as well as portions of prefrontal cortex, including Broca’s area (Poremba et al., 2003; Seltzer and Pandya, 1994). Heteromodal regions are obvious candidates for participation in the supramodal language network. Here, we adopt the assumption that delineation of an actual supramodal network, including the specifics of the alignment of the neural substrate for printed language with the substrate for spoken language, while helpfully guided by data from other primates, can best be inferred from examination of correlations between specific language behaviors and appropriate indicators of brain activity.

The cerebral basis of reading in relation to the spoken language system cannot be fixed and immutable because the relation changes over the course of a person’s lifetime. Therefore, the network for reading is likely to be fluid to some degree, reflecting in part the biological commitments of primary language, but reflecting also the variables of type of orthography, education, and experience that may loom especially large in the configuration of the supramodal brain. In this research, we have studied young adults who are experienced readers (of English), and can be expected to have achieved at least a moderately high degree of synergy in dealing with language inputs in each modality (see also Shankweiler et al., (2008), which addresses the influence of differences in reading skill on BOLD responses during sentence processing).

Direct comparisons of brain activity elicited by speech and print have more often been directed to individual words and their parts than to connected text (see, for example Bookheimer, Zeffiro, Blaxton, Gaillard, and Theodore, 1995; Booth et al., 2002; Chee, O’Craven, Bergida, Rosen, and Savoy, 1999; Howard et al., 1992; Shaywitz et al., 2001). Imaging studies of orthographic skills supporting printed word recognition have reliably implicated temporo-parietal and temporo-occipital regions together with the inferior frontal region (Pugh, Mencl, Jenner et al., 2000; Rumsey et al., 1997), all arguably, portions of heteromodal cortex. Word-level neuroimaging studies, valuable though they have been in supplementing evidence from clinical studies on delineating the reading brain, are limited in the information they can provide about the cerebral basis of reading syntactically structured groups of words, i.e., sentences. In the present study, we chose sentence material as the stimuli because sentences are necessary to convey propositions while admitting of variations in their processing difficulty. At the same time, sentence stimuli can be compact enough to allow the temporal pattern of brain activity to be correlated with specific events during the course of a stimulus trial.

Motivation from Functional Evidence based on Neuroimaging

There is evidence dating from previous neuroimaging studies of sentence processing that easy or predictable material, or passive listening tasks, do not reliably produce activation of all portions of the language-relevant cortex (e.g., Crinion et al., 2003). Accordingly, in our earlier work (Constable et al., 2004) we introduced differences in complexity, contrasting subject-gapped and object-gapped relative clauses, in the context of a comprehension task. Particularly relevant to our present concerns are findings by Carpentier et al., (2001), Michael et al. (2001), Constable et al. (2004), Spitsyna et al. (2006), Homae et al., (2002), and Lindenberg and Scheef, (2007). Each of these studies was designed in part to compare the neural architecture of sentence processing in speech and print modes. In each study, experienced adult readers read or heard sentences that varied in the kinds of processing demands they posed, either because the materials varied or the task varied. Cortical regions engaged by processes involved in the apprehension of sentence meaning were mapped, using fMRI or PET, under experimental conditions that allowed the influences of input modality (speech vs. print) and type of stimulus material to be assessed.

Constable et al. (2004) and Michael et al. (2001) each exploited similar, widely-used manipulations of syntactic complexity, contrasting object-gapped relative clauses with simpler subject-gapped relative clauses or semantically similar conjoined clauses. Thus, these studies included explicit contrasts of text difficulty. A common goal of these studies was to determine the location and extent of cortical sites that respond to sentence material irrespective of modality and that also register differences in difficulty. Michael et al. (2001) contrasted sentences containing object-gapped relative clauses with less complex conjoined active clauses. The participants’ explicit task was to read or listen to alternating blocks of print and speech sentences and for each sentence to press a button indicating TRUE or FALSE in response to a probe question. Activity associated with sentences in each modality was assessed relative to a common rest/fixation baseline at anatomically-defined ROIs. Perisylvian language areas in inferior frontal, and superior and middle temporal lobes responded to sentence material in both auditory and visual presentation. In the main, these areas also showed a significant increase in activity with increases in sentence complexity. Increased activation for spoken sentences, relative to printed ones, was seen at anterior temporal sites and in the anterior portion of the inferior frontal gyrus (IFG). Printed sentences showed significant increases, relative to spoken ones, in the visual extrastriate region. A limitation of this study is that the rest/fixation baseline does not separate task-related auditory and visual processes inherent in each task from activity specific to linguistic processing of speech and print. Among other things, this could distort estimates of the overlap in activity associated with the reading and listening tasks.

In contrast, the study by Constable et al. (2004) incorporated modality-specific baseline conditions. Constable et al. contrasted sentences containing object-gapped relative clauses with less complex subject-gapped relative clauses. Participants made an acceptability judgment by button press for each sentence. Activity elicited by sentences in each modality was measured against a nonlinguistic baseline task matched for sensory channel (tone similarity or line-orientation judgments). As in Michael et al., sentences in each modality activated the inferior frontal region and the posterior temporal region bilaterally, with the response to print more strongly left-lateralized. Auditory presentation was associated with activity in proximity to primary auditory cortex and across the middle and anterior portions of the superior temporal gyrus. Printed sentences, by contrast, evoked major activity at posterior temporal and inferior parietal sites. Speech-print overlapping sites were largely within the left hemisphere and included the occipito-temporal region (OT), middle temporal and superior temporal gyri (MTG, STG), and IFG. The more complex object-relative sentences evoked heightened responses (in comparison to subject-relative sentences) most strongly in left IFG but also in STG. Despite differences in stimulus materials, and baseline condition, Michael et al. (2001) and Constable et al. (2004) confirm sentence complexity effects at inferior frontal and temporal locations previously shown to express them. (See meta-analysis by Indefrey, in press). Further, in reporting sites where activation is modulated by complexity independent of input modality, the studies lend support to our hypothesis that, for syntactic and semantic processing necessary to apprehension of propositional meaning, print processing merges with speech processing in particular regions of the brain.

The two studies just discussed, Michael et al. (2001) and Constable et al. (2004), are closest in design to the study we report. In particular, each of these studies compared two sentence conditions in which, according to many theories of sentence parsing, stimulus sentence types contrasted with respect to how severely they taxed the parsing system (see, for example, Gibson, 1998; Lewis and Vasishth, 2005; MacDonald and Pearlmutter, 1995; Van Dyke and Lewis, 2003; van der Velde and Kamp, 2006; Vosse and Kempen, 2000). Moreover, the in-magnet task was designed to ensure that the participants attempt to interpret each sentence, thus lending support to the assumption that the parsing system was involved.

Four additional studies report locations of speech-print convergence under different stimulus and task conditions and baselines, but none of these studies included a minimal contrast in sentence processing conditions motivated by a strong theory of sentence processing. Three of them also employed passive listening or reading in the magnet without an explicit comprehension probe, so we are less confident that the critical language systems were strongly engaged. In the first, by Spitsyna et al. (2006), the task was passive viewing or listening to narrative passages against a baseline of unintelligible spectrally rotated speech or false font, or alternatively, an odd-even number discrimination task presented as speech. Sites of modality convergence were noted at anterior and posterior temporal regions, the fusiform gyrus and the parieto-temporal-occipital junction (for the number task), but not in prefrontal regions. The second study, by Lindenberg and Scheef (2007), also with narrative material, required no in-magnet response and employed either nonlinguistic baselines (fixation cross, or alternatively, white noise) or low-level semi-linguistic baselines consisting of letter strings and pseudowords, or alternatively, backwards speech. Sites of modality convergence with the corresponding nonlinguistic baseline yielded both temporal and inferior frontal sites of activation, whereas compared against language-like stimuli, convergent activity was focused on MTG and the posterior parietal lobule. In a third study (Jobard et al., 2007) participants attended passively to word lists, sentences and brief texts. Clusters located within temporal and frontal cortex were jointly activated by reading and listening. Left IFG, the left precentral region and posterior STG were recruited more by sentence or text conditions than by word lists. None of the preceding three studies incorporated a text difficulty contrast in their stimulus materials, making it difficult to compare their results to the previously discussed Constable et al. (2004) and Michael et al. (2001).

In the fourth additional study, by Homae et al. (2002), participants had to indicate whether each successive block of connected sentence material contained an anomaly of pragmatic content. A modality-matched control task tested for recognition of pseudowords within randomly ordered phrases. The conjunction of sentence comprehension-related activity with modality isolated a ventral portion of left IFG. This region was selectively activated by discourse comprehension and was independent of whether the input was spoken material or printed text. Like our study, Homae et al. used a task that required sentence comprehension in the magnet, but the contrast between conditions (coherent sentences vs. randomly ordered phrases) is a complex relation for which current theories of processing do not give us much guidance.

Encouragingly, each of the studies we have reviewed identified one or more zones of modality overlap within temporal and/or frontal regions. However, there is a good deal of variation of implicated areas across the studies. In the current work, we followed the example of Michael et al. (2001) and Constable et al. (2004) in adopting a design that was motivated by processing theory and a task that encouraged sentence comprehension. We believe designs with these properties are best suited to identify regions of convergent processing across modalities because the crucial processing is occurring at a level of linguistic abstraction that is modality-independent.

To introduce the stimulus materials and task, we describe another study from our laboratory by Ni et al. (2000). This study showed that sentences incorporating anomalies of morpho-syntax and of pragmatic content could yield similar effects on the distribution of brain activity as the complexities involving relative clause syntax studied by Michael et al. (2001) and Constable et al. (2004). Ni et al. (2000) investigated the cortical responses of experienced readers to spoken sentences containing anomalies of verbal morpho-syntax, trees can grew, and pragmatic content, trees can eat. In one study (Ni et al., Experiment 1), the task was to make acceptability judgments for these kinds of sentences, while a tone-pitch judgment task provided a baseline. The results implicated frontal and temporal regions for each anomaly type. Moreover, sites activated by sentences representing each anomaly condition minus the non-linguistic baseline were largely the same. Response to anomaly was found in both hemispheres in frontal, temporal and temporo-parietal regions at or near sites that showed the influence of relative clause complexity in Constable et al. (2004) and Michael et al. (2001). Using identical sentence materials as Ni et al., Carpentier et al. (2001) arrived at a similar result.²

Inconsistencies among the putative sites of modality convergence, especially in regard to IFG, likely reflect variations among the studies in task and choice of baseline. In particular, the use of sentence acceptability judgments may have contributed to this situation. Acceptability judgments introduce incommensurability by requiring different overt responses to anomalous and non-anomalous items, and they focus participants’ attention directly on the anomaly manipulation, posing an artificial, metalinguistic task. Such incidental task factors may modulate frontal activations, as several have suggested (Dapretto and Bookheimer, 1999; Love et al., 2006; Meyer, Friederici, and von Cramon, 2000; Ni et al., 2000; Stromswold, Caplan, Alpert, and Rauch, 1996). Carpentier et al. (2001), Ni et al. (2000), Exp. 1, and Homae et al. (2002) are all open to that criticism. In choosing materials and task for the present study, we built on the work of Ni et al. (2000), Experiment 2, which studied neural responses to the same morpho-syntactic and pragmatic anomalies with the same participants as their Experiment 1 summarized above, but discarding explicit acceptability judgments in favor of judgments about the presence of a target semantic category (animate beings) which, by design, was orthogonal and irrelevant to the presence of anomaly. In Ni et al. Experiment 2 the anomaly-irrelevant semantic identification task was mated with an event-related oddball design, embedding the anomalous sentences (presented by ear) in a matrix of more frequently occurring non-anomalous sentences. Due to differences in design and task, caution must be used in comparing the results of the two Ni et al. experiments. However, their Experiment 2, again, implicated inferior frontal as well as temporal regions, suggesting that IFG involvement is not an artifact of the task elicited by metalinguistic judgments.

In the present study, we asked where parallel streams originating from speech and print merge for sentence processing. We adopt a methodology based on the considerations just discussed. We manipulated morpho-syntactic and pragmatic anomalies of main verbs in simple sentence materials, and presented them in parallel speech and print conditions, with a sentence-level linguistic baseline consisting of matched non-anomalous sentences. As noted, our choice of task, a semantic category identification that is irrelevant to the anomalies, avoids requiring participants to attend to anomaly or to make judgments pertaining to anomaly. An oddball design, interspersing anomalous sentences among a much larger set of non-anomalous control sentences, further works against the formation of response sets that could distort the results. Thus, our goal was to delineate the supramodal language comprehension system by examining the joint effects on brain activity of input modality and sentence characteristics during sentence comprehension by adults who are experienced readers. We anticipated that this would allow us to identify cortical sites that are responsive to anomaly-based challenges to comprehension independent of input modality, and compare them with sites differentially engaged (in other studies) by object- and subject-relative clauses. Furthermore, we go a step beyond prior studies by exploiting the structure of our three-level anomaly factor (non-anomaly vs. pragmatic anomaly vs. syntactic anomaly) to discern regions of parallel patterning more specific than simple activation in common. We adopt a cosine similarity measure which allows us to apply this more refined approach systematically to the whole brain.

Method

Participants

Neurocognitive data reported here are derived from a study of 44 young adults who represented a wide range of reading ability. Individuals were recruited primarily from adult education centers and community colleges. Criteria for inclusion in that study were: age from 16 through 24 years; no uncorrected hearing or visual impairments; full-scale IQ via the Wechsler Abbreviated Scale of Intelligence, (The Psychological Corporation, 1999) of at least 80 (mean = 104.9; sd = 12.8); native speaker of English; no history of neurological problems; ability to read and comprehend simple expository material. The performance of many of the individuals in this sample on reading ability tasks was far below the levels typically seen in samples from university students. Shankweiler et al. (2008) reported on the effect of individual skill differences on modality relations using the data from this participant pool.

In the present study our interest was to develop a new analytic method to assess supramodal language areas. Accordingly, we considered it most appropriate to apply this new tool initially to a more homogeneous population of skilled readers. To this end, we selected young adults from our sample presenting levels of literacy in keeping with university students who were the participants in the existing body of research to which we wished to compare our results. Eighteen participants of the aforementioned group of forty-four contributed to the present report. Identification of the subset proceeded as follows. First, it was determined that thirty-six of the forty-four yielded analyzable data on the fMRI exam, scans from the remaining eight being contaminated with excessive motion artifact. Then, eighteen from the upper quantile of a median split on a composite measure of reading comprehension were selected for the present report (12 female; mean age 21.7 years, sd = 2.2). The reading comprehension composite has two components. These were adapted from the sentence comprehension subtest of the Peabody Individual Achievement Test -- Revised (Markwardt, 1998) and the Gray Oral Reading Test, 4th edition (Wiederholt and Bryant, 2001). See Braze et al. (2007) for details. Thus, participant selection made it possible to achieve comparability with findings based on university students. Participants gave informed consent and were paid $80 for completing the protocols reported here, as well as an extensive battery of behavioral protocols reported elsewhere (Braze et al., 2007). Protocols were approved by the Yale University Human Investigation Committee.

Materials

Stimuli consisted of short sentences, varying in length from 4 to 8 words (mean = 6.06, sd = .77) and generated from common vocabulary. Three sentence types are represented, shown in Table 1. Non-anomalous sentences, which were grammatically well-formed and expressed a conventional meaning, were the most numerous. In addition, there were two types of anomalous sentences which had most of their words in common: a morpho-syntactic anomaly type and a pragmatic anomaly type. Each anomalous sentence was generated from a non-anomalous base sentence by substitution of the subject noun (pragmatic anomaly) or the auxiliary verb (mropho-syntactic anomaly). Base sentences, like the bracketed examples in Table 1, were not actually presented to the subjects. They were never seen or heard. Non-anomalous test sentences that were included in the test were created anew, observing the same constraints on length and vocabulary. Thus, they were similar, but unrelated, to the pairs of anomalous sentences.

Table 1.

Examples of sentence anomaly types used in the present study, without (A) and with (B) the target semantic category (a plant, part of a plant, or something made from a plant).

A	New cars might shrink when washed.	Pragmatic Anomaly
	New shirts have shrink when washed.	Syntactic Anomaly
	[New shirts might shrink when washed.]	Non-anomalous
B	Paint can attack wooden fences.	Pragmatic Anomaly
	Termites are attack wooden fences.	Syntactic Anomaly
	[Termites can attack wooden fences.]	Non-anomalous

Like Ni et al. (2000), the morpho-syntactic anomaly used in the present study is created through a violation of the normal dependency between main verbs and modals or auxiliary verbs (See Table 1). These sentences were meaningful, but contained violations of verbal morphology. This type of anomaly presents challenges to the hierarchical structure building aspect of language processing (Friederici, 1995) and may serve to trigger mechanisms of structural reanalysis (Fodor, Ni, Crain, and Shankweiler, 1996). The morpho-syntactically anomalous items were crafted to avoid pragmatically odd combinations of content words. We anticipated that cortical activations to this type of anomaly would be similar to those elicited by complex non-anomalous sentences, specifically the object-relative sentences of Constable et al. (2004) and Michael et al. (2001). This expectation is based on the similarities in the spatial and temporal distributions of event-related brain potentials of object-relative clauses (e.g., King and Kutas, 1995; Kluender and Kutas, 1993) and morpho-syntactic anomalies (e.g., Friederici, 1995; Friederici, Pfeifer, and Hahne, 1993). In each case the characteristic event-related potential is a P600, a left-anterior negativity, or both (also see Kaan, Harris, Gibson, and Holcomb, 2000). In the pragmatic anomaly condition, sentences were grammatical, but expressed meanings that were unusual or inconsistent with world knowledge. Note that for each anomaly type, the anomaly becomes apparent at the main verb.

The test protocol consisted of 560 sentences, half presented in the print modality and half in speech. Modality was counter-balanced across subjects. There were 28 occurrences of each anomaly type in each modality, giving a total of 112 anomalous sentences on the model of Table 1 and 448 unrelated non-anomalous sentences. Consequently, 80% of trials contained non-anomalous sentences and the remaining 20% were evenly divided between pragmatic anomalies and syntactic anomalies.

Within the stimulus set, counterbalancing ensured that each participant read or heard one member of each anomaly pair. So, for each such pair, a participant would be presented with one member of that pair in either print or speech. Non-anomalous sentences were also counterbalanced across modality. Counterbalancing resulted in 4 presentation lists, with participants randomly assigned to lists.

Additionally, 25% of sentences, evenly distributed across Anomaly and Modality, included mention of a specific semantic category (a plant, part of a plant, or something made from a plant). The other 75% of sentences did not contain an item from this category. When a sentence contained mention of the target category, that mention always occurred after the main verb. See Table 1B.

Procedure

The 560 trials of each presentation list were evenly divided into 14 blocks; each block was presented during a single functional activation imaging run. See Image acquisition and analysis, below. Blocks contained 40 sentence trials: 32 non-anomalous sentences and 4 trials each of pragmatically anomalous and syntactically anomalous sentences. Thus, anomalous sentences occurred infrequently in the stimulus sequence, distributed pseudo-randomly among the non-anomalous trials. Sentence order within each block was fixed. Number and proportions of items in each condition are shown in Table 2.

Table 2.

Number and proportion of sentences in each stimulus condition. Note that 25% of the items in each condition require an affirmative response in the in-magnet behavioral task.

	No Anomaly	Syntactic Anomaly	Pragmatic Anomaly
Print	224	28	28
	40%	5%	5%
Speech	224	28	28
	40%	5%	5%

Each participant’s 14 image activation runs were evenly divided between print and speech blocks, 7 of each. For all participants, odd numbered runs contained speech stimulus blocks, while even numbered runs contained print stimulus blocks. Block order was randomized within each modality. The inter-trial interval for both modalities was 5 seconds. At least 2 non-anomalous sentences occurred between each pair of anomalous sentences. So, a minimum of 15 seconds intervened between occurrences of anomalous verbs, allowing the associated hemodynamic response to relax between anomaly trials. Print stimuli were presented word by word in a rapid serial visual presentation (RSVP) format at a rate of 2Hz. Thus, print sentences lasted between 2 and 4 seconds. Spoken sentences were presented at a moderate speaking rate and ranged from 1.51 to 3.40 seconds in duration. Participants were instructed to monitor sentences for mention of the target semantic category (a plant or plant product) and, at the end of each sentence, to press a button indicating whether or not the category was mentioned. Button presses were made with the first and second fingers of the right hand, corresponding to YES and NO, respectively. Expected YES and NO responses were evenly distributed across all anomaly and modality conditions. Participants were given practice on the task in both modalities before entering the scanner. Stimulus presentation and response collection was controlled by an Apple PowerPC computer running Psyscope software (Cohen, MacWhinney, Flatt, and Provost, 1993). Speech stimuli were presented through MR compatible headphones and print stimuli were displayed through an LCD projector onto a back-projection screen located at the foot of the MR scanner table. Participants viewed the screen through a mirror situated inside the scanner bore. A fiber optic button box was used to collect participants’ responses.

Image acquisition and analysis

Imaging was performed on a Siemens 1.5T Sonata MR system. Sagittal localizer images (TE, 14 ms; TR, 500 ms; FOV, 24×24 cm; matrix, 256×192; 5mm slice thickness, no skip; 1 NEX) were acquired to identify the inter-commissural line. Subsequently, 20 axial-oblique anatomic images, parallel to the AC-PC line, were acquired (TE, 11 ms; TR, 420 ms; FOV, 20×20 cm; matrix, 256×256; 6mm slice thickness, no skip; 1 NEX). Functional activation images were collected at the same slice locations using single shot, gradient echo-planar sequencing (flip angle, 80 degrees; TE, 50 ms; TR, 2000 ms; FOV, 20×20 cm; matrix 64×64; 6mm slice thickness, no skip; 1 NEX, resulting in voxel dimensions 3.125 × 3.125 × 6 mm). Fourteen runs of functional activation images were acquired. Each run provided 103 full volume images, for a total of 1442 per participant.

Each participant’s functional images were first corrected for slice acquisition time, then motion-corrected (Friston et al., 1995) and spatially smoothed using a gaussian filter of 3.125mm full-width at half-maximum. For each participant, an affine transformation to Montreal Neurologic Institute (MNI) standard space was obtained using the intensity-only module of the algorithm described in Papademetris et al. (2004), mapping between the subject-space anatomic image and the MNI-space “Colin” brain (available at http://www.bic.mni.mcgill.ca). Prior to across-subjects analysis, this transformation is applied to the single-subject activation maps, with trilinear interpolation, into 2 mm isotropic MNI space.

Map-based data analysis was performed using in-house software within MATLAB (Matlab, 2001). Multiple regression was used for single-subject, event-related analyses. At each voxel, signal intensity over time was modeled with a set of synthetic hemodynamic response functions (HRF) created from a gamma variate (tau: .9, n: 5, onset delay: 1.4, time to peak: 5s). The specific time-to-peak was chosen based on previous work from our lab using similar materials and protocols (Ni et al., 2000). The onset of the synthetic HRF was temporally aligned to the onset of the critical word (the verb) in each stimulus sentence. See Figure 1. Individual regressors were employed for each of 12 sentence conditions (3 anomaly [non-, syntactic, pragmatic] × 2 modality [print, speech] × 2 target category [yes, no]). A simultaneous multiple regression was employed with these 12 predictors of interest, and additional regressors to account for run-to-run mean offsets in signal intensity. The resulting regression parameters (B-weights) were converted to standardized activation scores by scaling them against the square root of the error mean-square for the model (Frost et al., 2005). These standardized activation scores are best viewed as effect size scores scaled in standard deviation units of the background noise for the voxel.³ Standardized activation maps were transformed into MNI space for subsequent analysis. Across-subjects, at each voxel, an ANOVA was employed with stimulus condition as a within-subjects variable, implementing a mixed-model or repeated measures ANOVA (Kirk, 1982). Planned comparisons were applied within this model to address hypotheses of interest. Resulting activation and contrast maps were corrected for multiple comparisons using the False Discovery Rate method (Genovese et al., 2002).

Figure 1.

The synthetic hemodynamic response function and its relationship to words in stimulus sentence. For both print and speech stimuli, the synthetic HRF peaks 5 seconds after the onset of the verb. Annotation reflects the two word per second presentation rate for print stimuli.

Two additional treatments of whole-brain data were implemented. In the first, across-subject contrast maps were used to create images showing both the logical intersection and the union of activations for two contributing contrasts (Hadjikhani and Roland, 1998; Ledberg, O’Sullivan, Kinomura, and Roland, 1995). For example, consider two contrasts, A and B, each evaluated at threshold of p < .05. The probability of a chance activation at any given voxel by either A or B, but not both (symmetric difference) is then .0975, and the probability that a voxel will be activated by both contrasts (intersection) is .0025. While, for expository purposes, we will show both the symmetric differences and intersections on such maps, our chief interest lies with the intersections.

A second, novel, whole-brain analysis was designed to illuminate brain regions that showed a similar pattern of response to our three sentence anomaly conditions, regardless of modality. So, for each modality separately, we extracted the 3-vector of response levels (activation values) under the three conditions of our experiment: non-anomalous, pragmatically anomalous, and syntactically anomalous. This vector characterizes the “shape” or signature of activation for the three sentence types within a modality. We then compare the shape of this response vector for speech versus print using a cosine-based metric (cosine of the angle between the speech 3-vector (s⃗) and the print 3-vector (p⃗):

$$cos(\overrightarrow{s},\overrightarrow{p})=\frac{\overrightarrow{s}·\overrightarrow{p}}{\left|\right|\overrightarrow{s}\left|\right|\left|\right|\overrightarrow{p}\left|\right|}$$

The cosine measure is commonly used as an unbiased index of similarity that is sensitive to consonance of patterning independent of vector magnitude (for example, see Hinton and Shallice, 1991 for an application to modeling of lexical recognition by artificial neural networks; Mitchell et al., 2008 for a different application to neurological data.). To provide an initial test-bed of the cosine metric, we identified six left hemisphere regions whose response to sentence materials, in previous studies, had proved to be contingent upon sentence difficulty: two frontal sites, three temporal sites in the posterior superior temporal, the posterior middle temporal and the occipito-temporal regions, and one site in the inferior parietal region. Within these regions, centers of 6 spherical ROIs (6mm radii) were established at the maximally activated voxel for the contrast between anomalous sentences with non-anomalous sentences, collapsing the two anomaly types and averaging over modality.

Results

Semantic category identification task

Response rates to the category identification task (plant judgments) were high overall, with 16 of 18 participants responding to more than 98% of the 560 trials (a response failure arises when the trial times out before a response is made). Each of the two exceptions responded to about 85% of trials. Failures to respond were similarly distributed across modality and sentence type [all Fs ≈1]. Category identification accuracy, considering only trials on which responses were made, was high, as shown in Table 3. There is a marginal effect of modality on accuracy [F(1,17) = 3.60, p = .08, MSe = .310], as well as a reliable effect of anomaly [F(2,34) = 5.01, p=.01, MSe = .046]. The modality by anomaly interaction is marginal [F(2,34) = 2.61, p = .09, MSe = .040]. The anomaly effect is due to accuracies in the syntactic and pragmatic conditions being slightly lower than in the non-anomaly condition [F(1,17) = 10.49, p = .005, MSe = .035; F(1,17) = 7.42, p = .01, MSe = .058], but not differing from each other.

Table 3.

Mean percent accuracy (sd) for category identification, by Anomaly and Modality.

	No Anomaly	Syntactic Anomaly	Pragmatic Anomaly
Print	97% (16)	96% (19)	97% (18)
Speech	97% (17)	94% (25)	94% (24)

Activation in relation to sentence anomaly

Non-anomalous sentences provide a linguistic baseline for evaluating the effects of modality. Figure 2 shows the spatial distribution of evoked responses to speech and print versions of the non-anomalous sentences and their intersection (Ledberg et al., 1995) averaged over the 18 participants. Voxels that exceed threshold (p < .00001, uncorrected) for spoken sentences are represented by green, for printed sentences by red, with zones that are above threshold in both listening and reading coded as yellow. Thus, the conjoint probability for the intersection is p < 1.0e-10. Regions of co-activation of both modalities occur in large portions of left IFG, as well as dorsolateral frontal cortex, bilateral MTG and STG, OT, and occipital (lingual) areas. Sites more active on listening trials are distributed throughout the IFG, insula, inferior temporal (IT), MTG, and STG, with smaller areas of activity in occipital cortex. Temporal and occipital activations in response to speech are predominantly bilateral, but IFG activations are primarily in the left hemisphere. Sites activated more by reading are found in both right and left IFG. Reading-related activity is also evident in each temporal lobe, albeit more sparsely than listening activity. By contrast, reading-related activation in the occipital and parietal lobes is more extensive than that for speech perception.

Figure 2.

Intersection and differences in activation for evoked responses to non-anomalous sentences in each modality (p < .00001, uncorrected). Areas coded in green are activated above threshold in speech only, areas in red are activated in print only, and yellow indicates areas that are active in both modalities (conjoint probability < 1.0e-10). Montreal Neurologic Institute (MNI) z-coordinate is indicated on each slice.

Against the background of evoked activity to non-anomalous sentences, we investigated how the distribution of activity is modulated by sentence anomalies that pose challenges to syntactic or pragmatic processing. The difference between activity levels for anomalous sentences and non-anomalous ones isolates regions activated specifically by sentence processing from regions activated by lexical processing and characteristics of the task. Figure 3 is a composite map for speech and print combined, also collapsing across anomaly type, showing locations of suprathreshold (p < .05, FDR corrected) contrast in activation between non-anomalous and anomalous sentences (yellow for regions activated more by anomaly than non-anomaly; blue for the reverse). The figure shows the spatial distribution of cortical responses to anomalous sentences in the context of an overt task (“plant” judgments) that is both irrelevant to the anomaly and orthogonal to it. Thus, while not the target of the semantic judgments, anomaly can be seen to activate some regions more strongly than the non-anomalous control sentences. The anomaly-sensitive sites (like their complexity-sensitive counterparts in Constable et al., 2004) are located chiefly in the left hemisphere, anteriorly within IFG and adjacent insula, the precentral region, and the middle frontal gyrus (MFG). Posteriorly, they are located in STG, the inferior OT region (fusiform gyrus), and inferior parietal lobule (IPL). There are a few sites that were more responsive to non-anomalous sentences: Inferior temporal, medial frontal, and posterior cingulate.

Figure 3.

Map of standardized activations for the contrast between anomalous with non-anomalous sentences, collapsing across anomaly types and averaging over modality (p < .05, FDR corrected). Hot colors indicate regions more active for anomalous sentences; cold colors show areas more active for non-anomalous sentences. MNI z-coordinate is indicated on each slice.

Whereas the main-effect analysis shown in Figure 3 identified a set of areas that show an overall anomaly effect, the next step was to isolate effects specific to each type of anomaly in conjunction with modality. Accordingly, the maps in Figure 4 depict the conjunction of speech and print modalities for anomaly minus non-anomaly, indicating areas more responsive to syntactically anomalous sentences than to non-anomalous sentences (Figure 4A), or to pragmatically anomalous than to non-anomalous sentences (Figure 4B). Voxels activated above threshold for the given anomaly type in speech are shown in green and similarly for print in red (p < .05, FDR corrected). Voxels activated above threshold in both modalities are shown in yellow. The maps for syntactic anomaly (Figure 4A) show modality overlap anteriorly in IFG and MFG, the precentral gyrus, and posteriorly, in STG, and lingual gyrus. The maps for sentences containing pragmatic anomalies (Figure 4B) show fewer regions of supra-threshold activity. But as was the case with syntactic anomaly, there is speech-print overlap, mainly at IFG, with a smaller overlapping site located inferiorly in the occipito-temporal region. For each anomaly type, sites showing modality overlap are mainly within the left hemisphere.

Figure 4.

Conjunction map indicating areas more responsive (p < .05, uncorrected) to (A) syntactically anomalous sentences than to non-anomalous sentences or to (B) pragmatically anomalous sentences than to non-anomalous sentences, in speech (green), or print (red), or both (yellow; conjoint probability < .0025). MNI z-coordinate is indicated on each slice.

Concordant responses to anomaly, across modality

To further examine the joint influence of modality and sentence characteristics and to connect them with benchmarks in the neuroimaging literature, we identified left hemisphere regions of interest which in previous studies had shown effects contingent on sentence difficulty (Caplan, 2004; Indefrey, in press). Within each of these areas we located the voxel most responsive to sentence anomalies in our own data; these locations were the centers of 6 spherical ROIs. These comprised two frontal sites, three temporal sites in the posterior superior temporal, the posterior middle temporal and the occipito-temporal regions, and one site in the inferior parietal region. The MNI coordinates corresponding to the centers of these ROIs are given in the caption to Figure 5. See Methods for details of ROI selection. For each designated ROI we measured activity elicited by each sentence type in speech and print. Figure 5 shows mean activity levels for each of the six combinations of modality and anomaly for each ROI. It should be appreciated that, by design, each ROI is differentially sensitive to anomaly as contrasted with non-anomalous control sentences. Our goal was to consider whether putative supramodal language regions are similarly sensitive to anomaly regardless of modality.

Figure 5.

Mean standardized activation scores (±sem) for each of 6 sentence types in 6 empirically derived regions of interest (6mm radius spheres, encompassing 123 voxels each). Note that y-axes are all on the same scale, although differently centered. ROIs are centered on the focal activation within anatomic regions known to respond to linguistically challenging material. ROI centers in MNI coordinates are: dIFG (−46, 16, 24); vIFG (−44, 26, 12); STG (−54, −36, 8); IPL (−58, −38, 24); MTG (−66, −10, −22); OT (−32, −40, −16).

It is apparent that in each of the 6 ROIs, the pattern of activation across the three sentence types is qualitatively very similar in both modalities. However, in order to assess supramodal potential objectively across the entire brain, we implemented a novel method for determining which regions show a similar pattern of response to the various sentence types, regardless of input modality. For this purpose we computed the cosine of the angle between the print 3-vector and the speech 3-vector for each subject as a function of the activation levels within each ROI, a value which can vary from −1 to 1. See Method for details of the cosine metric. The mean cosine value based on evoked responses averaged within ROI and across subjects is high in five of the six regions, excepting MTG, consistent with the visually apparent similarity in BOLD activity level across modalities in each ROI shown in Figure 5 (cosine in vIFG = .69; dIFG = .55; STG = .77; IPL = .69; MTG = .35; OT = .61). Regional cosine values computed individually for subjects and then averaged are shown in Table 4.

Table 4.

Peak and Mean cosine values for the 6 ROIs whose activation patterns are depicted in Figure 5. MNI: Montreal Neurologic Institute.

		MNI coordinate (center)			Cosine
	name	x	y	z	Mean	Peak
1	vIFG	−44	26	12	.52	.64
2	dIFG	−46	16	24	.56	.65
3	STG	−54	−36	8	.59	.69
4	IPL	−58	−38	24	.46	.67
5	MTG	−66	−10	−22	.12	.36
6	OT	−32	−40	−15	.36	.51

A central goal of this study was to explore the boundaries of the supramodal language system. Therefore, we wished to identify all brain regions in both hemispheres that showed a similar pattern of response to our three anomaly conditions, regardless of modality. The cosine similarity metric was employed for this purpose. In order to identify regions of highest similarity across the entire brain, we calculated the cosine value individually for each voxel in each subject, creating whole-brain subject maps of the similarity metric. We then averaged these subject maps to identify voxels that tend to show supramodal function across subjects. Figure 6 shows a composite map of averaged cosine values across the brain, where the cosine measure exceeded .6. It is noteworthy that the areas of convergence identified by this measure overlap considerably with the 6 left hemisphere ROIs just described. Right hemisphere cosine similarity clusters are also apparent. Table 5 lists all the areas in which the cosine measure was consistently greater than .6 over a volume of at least 80 mm³.

Figure 6.

Results of applying the cosine similarity metric to the whole brain. Colored voxels indicate high similarity as determined by high cosine values (≥ .6) for the angle between the speech vector, specified by values of evoked responses to three sentence conditions (Pragmatic Anomaly, Syntactic Anomaly, and No Anomaly), and the corresponding print vector.

Table 5.

Regions of cosine maxima (cosine ≥ .6) based on whole-brain analysis. MNI: Montreal Neurologic Institute.

				MNI coordinate (peak)
	Hemi.	name	Volume mm3	x	y	z	Peak Cosine
1	L	Inferior Frontal Gyrus	1008	−48	18	14	.73
2	L & R	Medial Frontal Gyrus	392	0	8	50	.69
3	L & R	Medial Frontal Gyrus	1152	2	−2	64	.74
4	L	Precentral Gyrus	176	−48	−6	46	.70
5	R	Precentral Gyrus	288	58	−8	38	.72
6	L	Superior Temporal Gyrus	6584	−44	−40	8	.78
7	R	Superior Temporal Gyrus	3392	58	−30	8	.83
8	R	Middle Temporal Gyrus	104	62	−2	−4	.66
9	L	Postcentral Gyrus	80	−42	−24	52	.64
10	L	Fusiform Gyrus	80	−34	−54	−16	.62
11	R	Fusiform Gyrus	648	38	−62	−14	.68
12	L	Lingual Gyrus	392	−20	−84	−8	.68
13	R	Lingual Gyrus	104	20	−84	−16	.67
14	L	Inferior Occipital Gyrus	88	−34	−80	−6	.63
15	L	Inferior Occipital Gyrus	80	−12	−88	−12	.63
16	R	Middle Occipital Gyrus	88	34	−84	4	.63
17	R	Culmen (cerebellum)	248	10	−68	−10	.69

The whole brain survey based on the cosine vector analysis, like the activation maps shown in Figures 3 and 4, shows a preponderance of left hemisphere sites, although there are also reversals in some posterior regions. As may be seen from Table 5, a prominent single site that emerged only in the left hemisphere is in IFG. It includes portions of Brodmann areas 44 and 45, a region noted for its participation in language function, including syntactic processes (see meta-analysis by Vigneau et al., 2006). Although the size of this region is relatively small in each of the x-y planes shown in Figure 6, it spans the inferior-superior axis contiguously across the IFG from z=+8 to z=+26, and was the fourth largest region identified by this analysis. We also observe large areas of high similarity in both right and left hemisphere posterior STG; convergence of speech and print activations are especially strong in left STG, a region that has been found to be increasingly engaged as reading skill develops (Gabrielli, 2009). Notably, convergent portions of STG listed in Table 5 (both hemispheres) exclude the Transverse Temporal Gyri containing primary auditory cortex. There is a small activation in left IPL at slice z=24 in Figure 6, which is not listed in the table as it falls below the 80 mm³ cutoff for inclusion; nonetheless, it is wholly contained within the spherical ROI whose activation pattern is depicted in the lower left panel of Figure 5; functional differences in this region have been implicated in specific reading disability (e.g., Pugh, Mencl, Shaywitz et al., 2000). We also find concordant patterns of activation in the occipito-temporal regions, bilaterally. In left hemisphere OT, this site of concordance approximates the so-called visual-word form area (VWFA; see Cohen and Dehaene, 2004). Activation 10 in Table 5 (not readily apparent in Figure 6 due to slice selection) is slightly posterior to the OT ROI in Figure 5 and, as noted, in close proximity to the conventionally acknowledged location of the VWFA. Although the lingual gyrus shows a larger convergent area on the left, the right fusiform gyrus shows a larger area of high cosine values than does the left. We also note high cosine values bilaterally in portions of extra-striate cortex, as well as the medial frontal and pre-central gyri and a large portion of right cerebellum.

In summary, we employed a novel application of cosine similarity to create unbiased whole-brain maps depicting the relative concordance of response to 3 sentence types, regardless of modality. Results arising from this analysis largely confirm the locations of regions presumed on other grounds to engage in supramodal language function.

Discussion

We have supposed that the language processing system of the human brain is essentially a supramodal system and that an important function of the system, in people who know how to read, is to establish a common currency across speech and print modalities. Accordingly, our goal in this project was to identify those cortical zones that prove sensitive to sentence comprehension challenges independent of input modality. We have proposed that these characteristics are integral features of the language comprehension network (see also Constable et al., 2004).

As a way to probe this hypothesis, we designed sentence materials that presented greater or lesser challenges to comprehension, enabling us to compare patterns of response to these differing sentence types across modalities. Accordingly, our materials included syntactically and pragmatically anomalous sentences as well as a non-anomalous baseline condition. We incorporated a neutral semantic category monitoring task because we wanted to ensure an active perceiver. At the same time, the monitoring task was designed to be irrelevant to the source of the comprehension challenge because we wanted to tap into preattentive, automatic processing of the anomalies, avoiding the metalinguistic mode of processing elicited by anomaly detection tasks. Analysis of the behavioral responses to the semantic judgment task showed that performance did not differ according to type of anomaly, and thus was neutral with respect to the anomalies, as we intended.

The imaging findings in their entirety, including evoked responses to non-anomalous sentences, show large zones of regional overlap in response to spoken and printed sentences as well as modality-specific sites. The evoked responses to the non-anomalous sentences in each modality, mapped in Figure 2, show activity in perisylvian and extrasylvian cortex, as well as visual and auditory sensory regions. Areas of overlap that reflect activity common to spoken language perception and reading are found most densely represented in frontal and temporal regions bilaterally, including Broca’s and Wernicke’s regions and their homologues. Responses to simple non-anomalous sentences in each modality provide a standard against which to gauge the effects of the challenging sentence anomaly conditions.

Subtraction of anomaly from non-anomaly isolates activity specific to sentence processing from activity associated with the task that is common to both sentence types (Figure 3). This includes lower-level sensory activity stemming from visual and auditory stimulus conditions, motor-related activity elicited by the manual button press response, and activity evoked by the neutral, semantic monitoring task, all of which are balanced across anomaly conditions. When we contrasted activations due to non-anomalous sentences with those evoked by anomalous sentences, collapsing across anomaly type and modality, we found that the presence of anomaly heightens activity within some frontal and temporal regions, and that, in agreement with Ni et al. (2000, Experiment 2), most of the specifically anomaly-sensitive sites are left-lateralized. Thus, despite the fact that the participants were not asked to attend to the anomaly, this manipulation of sentence type clearly produced an effect on brain activity that is consistent with prior work on language processing in the brain. Residual activity associated with anomaly was detected at regions previously identified as portions of the language brain. Moreover, the anomaly-sensitive sites correspond approximately to the complexity-based contrasts reported by Michael et al. (2001), Carpentier et al. (2001), and Constable et al. (2004). Prominent sites are seen in perisylvian ventral IFG, STG, and extra-sylvian dorsal IFG, MTG, and OT. Despite differences in method, materials and task, we corroborate the indications of previous studies, which used relative clause complexity manipulations, regarding the location of sites sensitive to comprehension challenges in either modality.

In short, our findings parallel results reported by Michael et al. (2001), Carpentier (2001), and Constable et al. (2004), who examined the relation between sentence complexity and modality with quite sentence types (relative clauses). There is good agreement between our new findings with sentence anomaly and these earlier studies. In each case the effects are seemingly supramodal, implicating mainly inferior frontal and temporal regions.

Supramodal potential

Moving beyond simple identification of regions that respond to inputs in both modalities, a central goal of our research is to investigate the mechanism of the supramodal potential of the language brain. To this end, we implemented a novel application of a cosine similarity metric to identify areas in which the structure of processing is similar across modalities. As a preliminary evaluation of the metric, we chose six left-hemisphere areas, based on a review of the neurolinguistic literature, that are most frequently active during complex sentence processing tasks.⁴ At each ROI, the cosine metric showed that, regardless of whether sentences were presented in printed or spoken form, these regions responded in a similar way to the comprehension challenges posed by morpho-syntactic and pragmatic anomalies. We then extended our approach based on the cosine vector analysis to evaluate the amodality of response to sentence anomaly across the entire brain. This unbiased whole brain survey shows sites in both hemispheres, but with a preponderance of left hemisphere sites and nearly double the total volume of supramodal cortex on the left than on the right (see Table 5). This is true in most frontal and temporal regions. Some posterior regions display greater bilaterality or, in the case of the fusiform gyrus, a larger region on the right. Overall, these results are in line with the general finding in neuroimaging studies that language tasks generate activity in both hemispheres, greater on the left. In terms of function, the principle of left hemisphere language dominance is firmly established in the clinical literature, although there is also evidence for bilaterality in at least some language functions. In the case of reading, our findings are in keeping with evidence that typical reading development is characterized by decreased right hemisphere engagement and increased left hemisphere engagement (Pugh, 2001; Gabrielli, 2009).

We acknowledge that there is not complete agreement among the studies we reviewed regarding the locations of modality convergent sites. In view of the variety of tasks and stimulus materials employed in these studies, this is hardly surprising. For example, some studies have reported temporal and parietal sites, but no sites within frontal or prefrontal areas. Earlier we considered how task factors may have influenced these outcomes. First we noted that not all studies that addressed the issue of the supramodal language brain manipulated sentence difficulty and not all tested for comprehension during the neuroimaging session. Studies by Spitsyna et al. (2006) and Lindenberg and Scheef (2007) did not. These studies, which employed similar methodology using narrative material with passive listening or viewing, reported activity at temporal and temporo-parietal sites, but neither reported activity at frontal sites. It has long been observed that passive listening or reading tasks do not reliably engage inferior frontal sites (See Crinion et al., 2003). In contrast, Constable et al. (2004), Carpentier et al. (2001), Michael et al. (2001), and Homae et al. (2002) each report prominent inferior frontal sites among the modality convergent zones. Each of these studies required explicit judgments about the grammatical or semantic acceptability of the test sentences, raising the possibility that recruitment of the inferior frontal region reflects the special demands of this task, not ordinary comprehension processes.

Our method, though incorporating anomalous sentence materials, did not solicit acceptability judgments, while the semantic category judgments required were balanced across anomaly condition. Thus, our findings imply that inclusion of IFG in the supramodal system is not necessarily a consequence of acceptability judgments. Another spurious factor that could lead to the participation of IFG and adjacent motor cortex is the manual button press response (see Crinion et al. 2003; Love et al. 2006). The present study required a button press, as did several of the studies we reviewed, except for Spitsyna et al. (2006) and Lindenberg and Scheef (2007). However, in the present study, as in Homae et al. (2002), button press is common both to the experimental condition and the baseline condition, hence its effects are removed from the contrasts of interest. Our findings and those of Homae point away from the engagement of IFG being an artifact of method assignable to acceptability judgments or to use of a manual response. On the contrary, IFG seems to be an integral part of the supramodal language system.⁵

It is apparent from Table 5 and Figure 6 that modality convergence is found at other cortical regions than those we selected as ROIs. The whole brain analysis revealed modality independent sites in left IFG, left and right posterior STG, as well as the left occipito-temporal region. Each of the left hemisphere areas has been implicated in some aspect of language processing. IFG has confirmed relevance to syntactic processes (Cooke et al., 2006; Vigneau et al., 2006) and, arguably, to unification of meaning more generally (Hagoort, 2005). Left posterior STG (Wernicke’s Area) demonstrated especially strong and wide-spread convergence between speech and print activations (also noted by Jobard et al., 2007). Notably, the primary locus of convergence in STG did not include primary auditory cortex. In fact, Cooke et al. (2006) and Friederici et al. (2003) have each proposed wide-scale cortical networks with fronto-temporal components involved in the syntactic and semantic aspects of language processing. Right STG has been implicated in processing prosodic aspects of language (Friederici and Alter, 2004; for discussion see Bookheimer, 2002). The fact that our cosine metric indicates a high degree of concordance between print and speech activations in prosody-implicated right hemisphere regions is especially intriguing in light of recent indications from behavioral reading research that the apprehension of print may well engage prosodic processes (Ashby, 2006). Alignment of response to anomaly across modality in occipito-temporal regions is consistent with the view that these regions subserve the interface between orthographic and phonological processes (Pugh, Mencl, Shaywitz et al., 2000; Cohen and Dehaene, 2004), on the assumption that neural pathways activated by phonological (speech) inputs can feed back into regions that are putatively involved in orthographic-to-phonological mapping. Finally, we note that our cosine metric indicates concordance of response to anomaly regardless of modality in medial frontal and pre-central gyri. The first of these corresponds to the supplementary motor area, and the second with primary motor cortex. These areas have established connections to speech production, but there are also some indications in the literature that they, particularly supplementary motor area, may be involved in language comprehension (Rieker et al., 2005; Calvert and Campbell, 2003; Murphy et al., 1997; Zatorre et al., 1996).

Though the discussion to this point has focused on the architecture of speech-print convergence, our findings, of course, also reveal modality-specific patterns of activity. As is apparent from the conjunction maps in Figure 4, broken out by anomaly type, and confirmed by ROI analyses presented in Figure 5, some sites activated by sentence anomalies also responded more vigorously to one or the other modality. Specifically, STG was slightly more responsive to speech than to print (as Spitsyna et al., 2006 also observed), while the OT site responded with greater vigor to printed sentences than to spoken ones and the more dorsal of the inferior frontal sites slightly so. These findings, particularly with respect to IFG and OT, are consistent with other indications in the literature (Constable et al., 2004; Poldrack et al., 1999).

Conclusion

A rigorous test of the hypothesis that the essential language brain is supramodal is that it responds similarly to differing sentence characteristics irrespective of whether the input was speech or print. New findings are presented delineating the neural architecture of the supramodal system for comprehension of language. This research examined the BOLD responses of young adults to spoken and printed sentences incorporating anomalies of morpho-syntax and pragmatics thereby extending to different kinds of linguistic material earlier findings based on relative clauses. The implicit effects of anomaly on the BOLD signal in the present study largely mirror the effects due to syntactic complexity manipulations used in other studies. Both kinds of comprehension challenges engage inferior frontal as well as posterior temporal and parietal sites. We identified six left-hemisphere ROIs, including both Broca’s and Wernicke’s regions, sensitive to comprehension challenges engendered by sentence anomalies. Across ROIs the pattern of response to sentence type is highly similar for each modality. Further, an unbiased survey of the whole brain identified regions of high similarity in both hemispheres, but predominantly in the left, confirming that the supramodal language system is a distributed, asymmetrically organized system. Through our choice of method and baseline, we are able to shed light on a major inconsistency in earlier findings with respect to the contribution of the inferior frontal region. Our findings suggest that IFG is integral to the supramodal language system.

The cosine similarity measure that we used to identify regions of convergence across the whole brain can be extended to arbitrary numbers of conditions (that is, more than the three that we employ here). The analysis may be thought of as a way of asking about two regions, X and Y of brain, not only, “Do they constitute part of a subsystem with a particular functional purpose?” but also “Do they function in the same way within that subsystem?” This refinement of hypothesizing may help us move closer to the goal of using brain imaging data to address questions about process as well as questions about localization.