Neuroimaging Studies of Language Production and Comprehension

Abstract

The 1990s were dubbed the “Decade of the Brain.” During this time there was a marked increase in the amount of neuroimaging work observing how the brain accomplishes many tasks, including the processing of language. In this chapter we review the past 15 years of neuroimaging research on language production and comprehension. The findings of these studies indicate that the processing involved in language use occurs in diffuse brain regions. These regions include Broca’s and Wernicke’s areas, primary auditory and visual cortex, and frontal regions in the left hemisphere, as well as in the right hemisphere homologues to these regions. We conclude the chapter by discussing the future of neuroimaging research into language production and comprehension.

INTRODUCTION

For cognitive neuroscientists the Holy Grail of research is to ascertain how the brain accomplishes its work. Throughout the history of psycholinguistics this goal could only be had through indirect means: Hints about the brain regions involved in language production and comprehension were provided by the vast and growing neuropsychological literature, documenting the task performance of patients who had suffered lesions to particular brain regions (e.g., Zurif & Swinney 1994, Obler & Gjerlow 1999).

In the decades following World War II, however, technology was developed that allowed researchers to exploit the changes in blood flow that accompany neural activity to study the brain at work. Use of this neuroimaging technology was in its infancy in 1994, when the first Handbook of Psycholinguistics appeared in print. Whereas a few pioneering neuroimaging studies had been performed incorporating language tasks (e.g., Petersen et al. 1988, 1989, 1990), research of this sort was scant. Indeed, there is hardly a reference to neuroimaging work in the pages of the Handbook, though there were chapters on neuropsychology. In the decade since then the situation has changed. There now exists a small but growing literature exploring the means by which the working brain of healthy individuals processes language, covering most of the major areas of psycholinguistics.

In this chapter we review the past 15 years of neuroimaging work on language processing. We take the term “neuroimaging” to refer to research using functional magnetic resonance imaging (fMRI) and positron emissions tomography (PET) methods. We do not attempt to distinguish between these imaging modalities with respect to the findings generated within each, given that both imaging methods will yield similar patterns of activation when participants engage in similar experimental tasks (e.g., Xiong et al. 1998). Additionally, our exclusion of event related potentials (ERP) studies (and studies using similar methodologies) reflects not a belief that these are not “true” imaging techniques, but rather a desire to focus on a body of literature developed using a limited set of techniques. First, however, we briefly review the logic and mechanics of neuroimaging studies.

The Logic of Neuroimaging Studies

PET and fMRI methodologies are based on the notion that increases in the neural activity in one particular brain region are accompanied by increases of the blood flow to that region (for a detailed technical explanation of how MRI works, see Turner & Jezzard 1994 and Cohen et al. 1993; the latter also explains the technical details of PET techniques). The increase in blood flow is delayed a few seconds from the initial increase in neural activity. Because of this delay, and because of the relatively slow rise and fall of the change in blood flow, the temporal resolution of neuroimaging studies is somewhat limited. A typical neuroimaging study relates stimulus- and task-related changes to changes in neural activity in an attempt to discern what brain regions underlie a particular type of processing and how those regions go about their work.

For example, suppose a researcher is interested in studying the detection of syntactic anomalies in sentences. One means of doing this would be to present participants with a set of sentences that contained anomalies and a set that did not. The researcher could then look at the task-related changes in blood flow (i.e., those changes that correlate with a particular event in the experiment) to determine which (if any) brain regions appear to be particularly active when anomalies are detected and which regions are not active at this time. These task-related changes have been examined using two primary techniques. First, one can compare the activity present in one task (e.g., normal sentence processing) with activity in another task (e.g., processing of sentences with anomalies). From this method, called the subtraction method, one can infer which brain regions are particularly important for detection of anomalies by noting the brain regions that show activity when processing sentences with anomalies that is over and above the activity shown in the same regions when processing normal sentences. To do this, the researcher subtracts the activation present in the “anomaly” condition from the activation present in the “normal” condition. Any regions that show activation following the subtraction are taken to be specifically involved in the processing of the anomalous sentences.

A second means of correlating brain activity with task performance is the event-related approach to neuroimaging. Similar to the use of ERPs, the event-related approach operates by correlating particular components of the changes in blood flow observed during the experiment with particular task-related events (e.g., the presentation of a syntactic anomaly in a sentence). By time-locking task-related and neural events, the researcher can determine which brain regions were particularly active in the processing associated with any given task component. The examples presented here obviously oversimplify the complexity of experimental design that is required to perform an interpretable neuroimaging study; for a more complete discussion, see Postle et al. (2000) and Sarter et al. (1996).

Imaging Studies of Language Processing

This review covers the results of neuroimaging studies on different aspects of language processing: sub–word-level processing, word processing, sentence-level processing, discourse processing, and issues of the neural architecture of language processing that have been addressed using imaging techniques. These findings complement and extend the traditional neuropsychological model of language (discussed in Caplan 1994, Zurif & Swinney 1994): Wernicke’s area is involved in semantic processing, Broca’s area is involved in syntax and production, and other distributed (mainly) left-hemisphere regions process language components such as phonology or orthography. To avoid presenting a confusing jumble of brain regions and Broadmann’s areas (BA), we report the findings in this section using general descriptors (e.g., “inferior frontal gyrus”) to locate brain activity. In the summary we incorporate Broadmann’s areas to be more exact with regard to where the activity is occurring.

Processing Below the Word: Phonology and Orthography

Low-level linguistic processing involves a number of components, some of which are perceptual prerequisites for linguistic processing (e.g., primitive auditory and visual analysis of language forms) and some of which are language specific (e.g., phonological processing).

Speech processing begins bilaterally in the superior temporal gyrus, which includes the primary and secondary auditory processing regions. Imaging studies have demonstrated activation of this region in the processing of both linguistic and nonlinguistic sounds (Alavi et al. 1981; Petersen et al. 1988, 1989; Frith et al. 1991; Mazoyer et al. 1993). Many of these studies involve a subtraction between auditory stimulation and a passive resting state. Mazoyer et al. (1993) report that this region was the only active cortical region during the processing of speech sounds both from a familiar language (in this case French), and a language unfamiliar to the participants (Tamil). These authors also report that activation was greater in the left hemisphere than the right when the participants were processing their native language. In the same study it was observed that phonological processing was specific to the left middle-temporal gyrus.

These findings have been corroborated in a number of studies showing superior and middle temporal activation in phonological processing, as well as activation in the planum temporale (Binder et al. 1996a, Binder 1997). Additionally, the right temporal region (particularly the anterior regions) has been implicated in the processing of prosodic aspects of the speech signal (Buchanan et al. 2000). Other research involving the maintenance of phonological information (e.g., for deciding if two words rhyme or for a memory task) has implicated the inferior frontal gyrus and adjacent areas in phonological processing (Petersen & Fiez 1993, Shaywitz et al. 1995, Gabrieli et al. 1998, Wagner et al. 2001, Xu et al. 2001). Posterior temporal regions, including Wernicke’s area and the supramarginal gyrus (Petersen et al. 1989, Demonet et al. 1994, Zatorre et al. 1996) have also been implicated in phonological processing. These regions may underlie the translation of orthographic symbols to phonemic representations (e.g., Xu et al. 2001), but there is evidence that they play a role in phonological processing that is independent of this translation process (e.g., Demonet et al. 1994).

The role of frontal regions in phonological processing has been debated. Evidence from rhyme judgment or generation tasks (Cuenod et al. 1995, Xu et al. 2001) indicates that the inferior frontal gyrus is involved in processing that is phonological in nature, although it appears that this processing is secondary to the initial temporal lobe processing of phonological information from the speech signal. Thus, the processing of phonological information in frontal regions may reflect a maintenance function (the phonemic information is maintained while the experimental task is performed). In addition, neighboring cortical regions (e.g., Broca’s area) likely play a role in the production of speech sounds (Cuenod et al. 1995), and neighboring premotor areas may play a role in processing that is phonology dependent (e.g., Price et al. 1994). Buchanan et al. (2000) have suggested that the inferior frontal region in the right hemisphere is involved in detecting the emotional content of prosody. When participants in this experiment were asked to discriminate words based on their sounds, left prefrontal regions were active; however, when participants were asked to discriminate words based on the emotional tone conveyed by the prosody of the speaker, right frontal regions showed activation.

The processing of visual linguistic stimuli is slightly more involved than speech processing, owing to the translation between orthography and phonology believed to occur (Pugh et al. 1996, Xu et al. 2001). In Pugh et al.’s experiment participants performed four tasks for which they made same/different judgments, two of which are important for low-level word processing: (a) a line judgment task (same pattern of orientations in two sets of lines?) and (b) a letter case judgment task (same pattern of case alternation in two sets of consonant strings?). Consistent with lesion studies (Henderson 1986) and previous PET studies (e.g. Petersen et al. 1989), lateral extrastriate regions were activated during orthographic processing. In addition, Fujimaki et al. (1999) have shown that complex visual forms such as letters and pseudocharacters activate the occipital-temporal sulcus and the posterior inferior-temporal region bilaterally. These regions thus play a role in the early processing of visual linguistic stimuli.

Beyond orthographic processing, it has been proposed that visual linguistic input is translated into phonological form (e.g., Xu et al. 2001 and references therein). Regions in and around Wernicke’s area, including the supramarginal gyrus and the angular gyrus, have been implicated in this process, both with regard to being “word-form centers” (especially regions in and near the angular gyrus; Small et al. 1996) and/or being responsible for the actual translation process. There remains some debate on the former issue. A series of conflicting studies of visual word processing (Petersen et al. 1990, Price et al. 1994, Pugh et al. 1996), suggest that whereas the brain regions involved in processing particular aspects of the linguistic stimulus have been identified, there does not yet appear to be any evidence of a specific word-form processing region. More recently, however, Polk & Farah (2002) conducted an experiment in which participants read alternating case words and pseudowords. These stimuli produced similar patterns of activation in the left-ventral visual cortex, suggesting the presence of a word form area that pays attention to abstract orthographic patterns rather than strictly perceptual components of visually presented language.

Xu et al. (2001) performed a PET study showing that whereas phonological processing of words and pseudowords shared regions of common cortical activation, processing pseudowords (but not words) activated the supramarginal gyrus and neighboring regions. This suggests that lexical items and pseudowords (and perhaps low-frequency lexical items) may undergo different routes in the visual-to-phonological translation, with the word forms of lexical items directly accessing phonological representations, and pseudowords needing to have a phonological code assembled in a more brute-force fashion, performed in the surpramarginal gyrus and neighboring regions. If it turns out that low-frequency lexical items are processed in a manner similar to that of pseudowords, the presence of frequency effects in a variety of cognitive tasks could be (at least in part) a function of the additional time required to access the phonological code of the low-frequency words.

As discussed in the next section, there appears to be a good deal of overlap between many aspects of auditory and visual word processing. The preliminary processes, however, differ across modalities. Initial speech processing appears to occur in the bilateral temporal lobes, with left lateralized processing occurring with higher levels of processing (i.e., phonological processing). Visual language processing begins in the occipital cortex and involves several regions near the boundary of the occipital and temporal lobes (e.g., left angular gyrus, bilateral fusiform gyrus, posterior inferior-temporal regions, occipital temporal sulcus, supramarginal gyrus, Wernicke’s area). The putative function of these regions is to identify orthographic forms and to translate these forms into a phonological representation.

Word-Level Processing

The earliest imaging studies of language processing centered around the processing of words (e.g., Petersen et al. 1988). The authors of these early studies reported some degree of surprise at finding activation in brain regions not traditionally believed to be implicated in language processing (such as inferior frontal regions; Raichle 1996). Since then, these findings have been replicated and extended in a number of studies, identifying a wide range of regions of activation during word processing.

One set of tasks used to investigate word processing requires the participant to make a semantic judgment. In some cases this judgment is relatively simple: Is the word abstract or concrete (e.g., Demb et al. 1995, Desmond et al. 1995, Gabrieli et al. 1996)? In other cases (e.g., Frost et al. 1999) the judgment is more challenging: Is this an animal that lives in the United States and is used by humans? One such study (Gabrieli et al. 1996) involved comparing the abstract/concrete task with a perceptual judgment (are the letters in the words in upper or lower case?). The critical subtraction (semantic task–perceptual task) revealed activation in several left hemisphere regions: the inferior frontal gyrus, the cingulate cortex, and the superior frontal region.

These findings have been replicated with similar experimental tasks and with paradigms using slightly different subtractions (Desmond et al. 1995, Binder 1995, Binder et al. 1996b). For instance, Demb et al. (1995) have shown left inferior-frontal activation when the abstract/concrete task is contrasted to both easy non-semantic tasks (upper- versus lower-case judgments) and more difficult tasks (are the letters in the sequences ascending or descending alphabetically?). An interesting component of this study was the inclusion of a repeat condition, in which the stimulus materials were presented to the participants twice. In the repetition phase of the study a decrease in activation was noted in the inferior frontal regions. A similar decrease in activation was noted in the abstract/concrete-hard nonsemantic task subtraction, relative to the same subtraction using the easy nonsemantic task. This suggests that the frontal activation observed in studies of word processing may be the result of both semantic processing of some sort and the increased task difficulty of the semantic judgments relative to the baseline tasks.

Another task used to explore word processing is the word generation task. Participants are asked to generate words, aloud or silently, given a particular cue. A popular variant of this methodology is the verbal fluency task, in which participants generate words that start with a letter provided by the experimenter (e.g., generate words that start with “F”) (e.g., Cuenod et al. 1995, Phelps et al. 1997, Schlosser et al. 1998). When compared to a baseline condition of rest (Cuenod et al. 1995), this task produces left inferior-frontal activity as well as activation in and around Wernicke’s area and the superior temporal gyrus. In comparison to other baseline tasks (such as word repetition), the verbal fluency task has also produced activation in the superior frontal regions and the right cerebellum (Phelps et al. 1997, Schlosser et al. 1998). Similar results have been obtained in experiments in which participants are asked to generate nouns and/or verbs, although verb generation typically results in stronger activation in and around Broca’s area (e.g., Weiller et al. 1995).

Other generation experiments involve sequence production (e.g., de Zubicaray et al. 1998, Wildgruber et al. 1999) and word stem completion (e.g., Desmond et al. 1998). Wildgruber et al. (1999) asked participants to silently recite the months of the year either forward or in reverse order. Subtracting forward recitation from reverse, one sees activation in bilateral middle and inferior frontal regions, in the parietal cortex, and in the anterior cingluate in the left hemisphere. Widespread activation was associated with purposeful recitation (i.e., reciting the months in reverse order) (similar results were found by de Zubicaray et al. 1998) as opposed to rote production (forward recitation). Word stem completion (given the stem “mat–”, generate the first word that comes to mind) also shows inferior frontal activation, as well as activation in the supplementary motor area and the right cerebellum (Ojemann et al. 1998, Desmond et al. 1998, Chee et al. 1999b).

From the aforementioned studies there appears to be some regularity in the areas in which word-level information is accessed and processed, mostly centering around the inferior frontal region and posterior temporal regions near Wernicke’s area, depending on the task demands and subtractions employed. Other studies have looked for finer distinctions in the ways particular types of words are processed. Kiehl et al. (1999) have explored the processing of abstract and concrete words. Comparing word processing to the processing of pseudowords, they found activation in superior parietal regions, anterior cingulate, left inferior-frontal regions, the left middle temporal gyrus, and the right superior temporal gyrus. Compared with concrete words, abstract words produced more activation in right temporal and frontal regions. These findings are consistent with literature that implicates the right hemisphere in abstract language processing (Bottini et al. 1994, Beauregard et al. 1997).

Chee et al. (1999a) have looked for differences in the processing of verbally and visually presented words. Participants engaged in an abstract/concrete task for words in both modalities. As a baseline task for visual words they performed a case judgment task (upper versus lower case); for spoken words they responded to whether the stimulus word had one or many syllables. Predictably, initial processing of visual and verbal stimuli involved different brain regions. As found in other studies (Pugh et al. 1996, Petersen et al. 1989), auditory language processing activated mainly left temporal regions, whereas visual word processing involved the posterior left superior–temporal gyrus, the left supramarginal gyrus, and the cerebellum. Nonetheless, a similar pattern of activation was observed for the semantic task performed in both modalities. This common pattern of activation involved the inferior frontal region, the anterior prefrontal region (bilaterally), the left premotor region, and the cerebellum.

These findings are consistent with studies that argue for a common semantic system across modalities. Similarities have been noted in the semantic processing of verbal and visual words (Chee et al. 1999a), words and pictures (Vandenberghe et al. 1996, Federmeier & Kutas 2001), and between faces and proper names (Gorno-Tempini et al. 1997). Whereas the specific modality and task demands associated with these individual comparisons can lead to variation in the exact regions of brain activation observed, the fact that verbal and visual language processing overlap so greatly may be due to the fact visually presented language is translated into phonological form at an early stage of processing (see Xu et al. 2001 for a discussion).

The distinction between regular verbs (those whose past tense is created by adding “-ed” to the verb stem: kill, killed) and irregular verbs (those whose past tense is formed by an irregular pattern: make, made; take, took) has also been explored (Marlsen-Wilson & Tyler 1997, Pinker 1997). Neuropsychological case studies (Marlsen-Wilson & Tyler 1997, Ullman et al. 1997) and neuroimaging experiments (Jaeger et al. 1996, Indefrey et al. 1997) have shown that irregular and regular verbs activate different cortical regions during processing. Jaeger et al. (1996) report that producing the past tense of regular verbs activates inferior frontal regions, whereas producing the past tense of irregular verbs activates middle temporal regions. The inferior frontal activation is argued to arise from the assembly of regular past tense forms by combining a verb stem (kill) and a morphological inflection (-ed). Irregular verbs activate temporal regions because they need to be recalled verbatim from the lexicon. Pinker (1997) concludes that this is evidence for the “words-and-rules” theory of lexical structure. Without a finer understanding of the processing that occurs in these cortical regions, however, it may be premature to rule out alternative theories.

Price et al. (1994) manipulated the duration with which words were displayed to participants. They report that many word processing regions were suppressed (i.e., showed decreased activation) once the word form had been processed (as evidenced by changes in activation in short versus long stimulus display times). This suggests an attentional component (or some other type of control component) in cognitive processing, which prevents the processing of redundant information in circumstances such as this (i.e., the word is not processed repeatedly while it is still displayed). This issue will require further investigation, and it appears that temporal factors need to be examined closely as research on lexical processing proceeds.

An interesting addendum to this literature is provided by Pulvermuller (1999). He presented a theory of word structure in the brain in which it is asserted that words are Hebbian cell assemblies that link orthographic, phonological, and meaning-specific information. For instance, Pulvermuller proposes that “vision words” have distributed representations that include temporal regions (phonological word-form representation) as well as areas in the occipital lobe (semantic information), whereas motor words (such as verbs) have representations that span temporal regions as well as frontal areas in and around the motor cortices. Evidence from ERP studies, neuropsychological case studies, and behavioral studies are cited in support of this hypothesis (e.g., Preissl et al. 1995, Pulvermuller 1996). In the main, activity associated with the semantics of particular words or word classes was absent from the studies reported here. Whereas some verb generation studies did show activation in and around Broca’s area and the supplementary motor region (Weiller et al. 1995), this activation may have been the result of the language production demands of the experiment, not the firing of cell assemblies.

Across many studies, a general map of word processing has been generated. The left inferior-frontal gyrus and adjacent supplementary- and pre-motor areas are involved in semantic processing, phonological processing, and perhaps word-form access and production. Middle, superior, and anterior frontal regions have been associated with semantic processing. These regions are also implicated in the processing of semantic memory (see Cabeza & Nyberg 2000 for a review of some of this evidence). It is likely that frontal activation in word processing is due to some combination of several factors: short-term storage of phonological information, lexical access, semantic processing, and task difficulty. Similar right hemisphere regions appear to be active during the processing of abstract nouns. In addition, the superior temporal regions of both hemispheres have shown activation related to the processing of the semantics related to word forms.

Posterior temporal regions (in and around Wernicke’s area) have been implicated in word-form access and production, semantic processing, and in some phonological tasks. Small et al. (1996) argue that regions near or in the left angular gyrus may serve as a word-form center, though there is debate on this issue (Price et al. 1992, Howard et al. 1992, Fujimaki et al. 1999). Other regions believed to be involved in word processing are the fusiform gyrus (bilaterally) and the left supramarginal gyrus. Finally, studies of word-level processing have revealed the surprising role of the cerebellum in cognitive tasks.

The cerebellum has long been thought to underlie motor-coordination functions. It was not until whole-brain scans of cognitive tasks were performed that it was discovered that the cerebellum was activated during higher-level tasks. Desmond et al. (1998) performed an experiment in which participants were asked to complete word stems that were either easy (many possible completions) or difficult (few possible completions). In other experiments of this sort it had been noted that there was a correlation between left frontal activation and right cerebellar activation (e.g., Ojemann et al. 1998), where left frontal activation appeared to be associated with right cerebellar activity. Desmond et al. (1998) qualified this relationship by noting a dissociation between activity in the frontal and cerebellar regions. Whereas frontal activation was associated with task performance when there were many possible stem completions, cerebellar activation was associated with performance when there were few possible stem completions. Desmond et al. (1998) hypothesize that the cerebellum may play a role in maintaining effortful cognitive activity, though the mechanism through which this is done remains obscure (see Leiner et al. 1995 for a discussion of the role of the cerebellum in cognitive processing).

Sentence Processing

Whereas many studies have explored issues related to word-level and sublexical processing, fewer studies have explored sentence processing. Indeed, most of the studies reported here treat sentence processing as a relatively monolithic construct and do not make an effort to tease apart the activation associated with specific components of the task.

The studies that have been performed reveal that sentence processing involves brain regions known as the “classical language areas” (e.g., Broca’s and Wernicke’s areas) and surrounding regions. Bavelier et al. (1997) report a sentence-reading study in which participants either read short, declarative sentences or viewed consonant strings in the experimental condition, and in the control condition viewed sentences or nonlinguistic strings in American Sign Language (ASL), a language that was unfamiliar to all of the participants. Activation unique to sentence processing was seen in and around both Broca’s and Wernicke’s areas (including the supramarginal gyrus), in the superior and middle temporal gyri in the left hemisphere, in the superior temporal gyrus in the right hemisphere, in the left inferior-frontal gyrus, and in the angular gyrus.

The activation in the angular gyrus is likely due to word processing, consistent with the neuropsychological literature (Bavelier et al. 1997). Activation in Broca’s area suggests that this region plays a role in syntactic processing (e.g., Zurif & Swinney 1994, Ni et al. 2000). Wernicke’s area activation is most likely the result of phonological processing (translating written language to a phonological code) or word-form processing (Demonet et al. 1992, Mazoyer et al. 1993). Hypotheses concerning the superior temporal gyrus and the inferior frontal gyrus are less clear. Both of these regions have been implicated in the maintenance of verbal information (see Milner 1971, Frisk & Milner 1990 for evidence based on lesion data) and in the processing of speech sounds. Additionally, the frontal regions have been associated with word-level processing, whereas the superior temporal regions have been associated with syntactic and semantic processing, as well as lexical processing (Damasio & Damasio 1992).

Similar patterns of activation have been observed in other studies in which sentence-level processing has been isolated (e.g., Robertson et al. 2000). These findings suggest that the temporal lobe is more active in sentence processing than had been assumed by neuropsychological models. Nonetheless, the temporal resolution of fMRI techniques makes it difficult to tease apart the components of sentence processing, given that most psycholinguistic research assumes that the processes of interest, to the degree that they are able to be differentiated, operate on the order of milliseconds, rather than seconds.

Ni et al. (2000) made a noteworthy attempt to separate the processes involved in comprehending sentences. In two experiments the researchers attempted to localize functions associated with syntactic and semantic processing. Unlike Bavelier et al.’s (1997) study, Ni et al.’s stimuli were presented verbally. In the first experiment participants engaged in an anomaly detection task. They heard blocks of (a) nonanomalous, syntactically anomalous sentences (e.g., “trees can grew”), (b) nonanomalous, semantically anomalous sentences (e.g., “trees can eat”), and (c) tone pairs that were the same or different in pitch. Relative to tone discrimination, the sentence judgment task activated regions in the inferior, middle, and superior frontal cortex bilaterally, as well as the superior and middle temporal gyri bilaterally. These regions of activation match closely with those reported by Bavelier et al. (1997), although this experiment featured much more frontal activation and a much more significant degree of bilateral activation. This may be due to the attentional or strategic components of the task, or to the use of auditory (as opposed to visual) stimuli (see Gabrieli et al. 1996, de Zubicaray et al. 1998 for examples in which frontal activation in both hemispheres accompanies increases in task difficulty).

In the second experiment participants heard the sentences from Experiment 1 but were not asked to detect anomalies. Rather, they were asked to determine if the sentence contained an animate being or not. The question of interest is what regions would show changes in activation to the syntactic and semantic anomalies. The event-related design allowed for the observation of changes in three time windows relative to the presentation of the anomaly (called early, middle, and late). Syntactically anomalous sentences produced no changes in early activation but caused activation in the left inferior and middle frontal regions in the middle and late periods. In the early period semantic anomalies caused activation in bilateral frontal regions (superior and middle frontal gyri). These regions remained active through the middle and late periods and were accompanied by activation in the left superior and middle temporal gyri. Thus, syntactic anomalies produced strongly lateralized activation in the inferior frontal regions in and around Broca’s area, whereas semantic anomalies produced bilateral activation in frontal areas and in the left posterior temporal lobe.

These results suggest that there are separable regions that attempt to deal with sentence anomalies of different sorts. Given the role that “world knowledge” is assumed to play in semantics, it is perhaps not surprising that semantic anomalies produce widespread activation. This may represent efforts to recruit appropriate information to make a sentence like “Trees can eat” acceptable (e.g., information that would allow a figurative interpretation of the sentence), as well as the increased difficulty of processing such sentences. It is not as clear what activation associated with the syntactic anomalies represents, given that these sentences do not seem to require reanalysis to recover from errors. Since many of the sentences that occur in natural language use are ungrammatical in some way, the activation may represent an attempt to normalize the sentence for further processing. The Broca’s area activation associated with syntactic anomalies may reflect the detection of sequencing anomalies. Similar regions were activated in a study in which participants heard a harmonic progression that ended with an anomalous (i.e., chromatic) chord (Maess et al. 2001). A more detailed understanding of what the language processing system attempts to do when it encounters this kind of syntactic anomaly is needed to investigate this matter further.

Studies have also assessed how changes in syntactic complexity alter brain activity. Stromswald et al. (1996) conducted a PET study in which participants read center-embedded sentences (“The car that the officer drove crashed into the wall”) versus right-branching sentences (“The officer drove the car that crashed into the wall”). They found that the more difficult center-embedded sentences produced activation in inferior frontal regions. Their results were inconclusive, however, given the lack of a nonlinguistic baseline task and the lack of appropriate controls for sentence difficulty, memory demands, and the like, all of which can influence sentence processing (Just & Carpenter 1992, MacDonald et al. 1994).

A related study by Just et al. (1996) compared reading of simple conjoined sentences (“The reporter attacked the senator and admitted the error”), subject-relative sentences (“The reporter that attacked the senator admitted the error”) and object-relative sentences (“The reporter that the senator attacked admitted the error”). Sentence reading was compared with a control task of reading consonant strings. They reported left hemisphere activation comparable to the studies discussed above for the sentence reading task. In addition, they found that activation in the right hemisphere homologues to these regions increased as sentence difficulty increased. Just et al. attributed this increased activation to the more resource-intensive nature of the processing of the object-relative and subject-relative sentences, in keeping with their capacity-based theory of language processing (Just & Carpenter 1992). Nonetheless, the increased activation need not signify only the recruitment of working memory resources. Bader (1998) argued that garden-path effects arise from sentences whose reanalysis requires particular types of changes in the prosodic contour of the sentence. Given the role of the right hemisphere in prosodic processing (Buchanan et al. 2000), the increased right hemisphere activation might reflect (in part) increased access to prosodic information rather than an increase in processing capacity per se. The activation may also reflect a lexical ambiguity resolution process (cf. MacDonald et al. 1994).

The processing of sentences thus involves Wernicke’s area (word/phonological processing), superior and middle temporal regions (phonology/lexical/semantic processing), Broca’s area (production/syntactic analysis), inferior frontal gyrus (phonological/syntactic/semantic processes), middle and superior frontal regions (semantics), and the right hemisphere homologues to these regions. The degree of right hemisphere activation found in studies of sentence processing varies with the nature of the task and the controls used in subtraction. The right hemisphere appears to be involved in lexical (e.g., Damasio & Damasio 1992) and prosodic processing, and activation in these regions may also reflect the recruitment of resources for task performance.

Discourse Processing

Studies of discourse processing have been of two types. The first type is concerned with the general processing that goes into comprehending connected discourse (e.g., St. George et al. 1994, 1999; Robertson et al. 2000). The second type is concerned with the processing of specific aspects of discourse, such as apprehending the moral or theme of a text (e.g., Nichelli et al. 1995). Each is discussed in turn.

Tzourio et al. (1998) had participants passively listen to stories. The listening task was contrasted with a resting condition. Story listening activated the temporal poles bilaterally, the superior temporal gyrus bilaterally, and the left middle temporal gyrus. The lack of a tight control task (i.e., a control task that matches the experimental task on several processing components, allowing one to determine more exactly which brain regions underlie specific subcomponents of the main task being performed) is due to the fact that the researchers were primarily interested in exploring the relationship between particular anatomical features of the brain and language lateralization. This design thus does not allow a detailed analysis of the circuitry associated with story processing.

Gernsbacher, Robertson, and colleagues (Robertson et al. 2000) have explored discourse processing in a series of experiments. In one experiment participants read sentences connected to a discourse and unrelated sentences and viewed non-alphabetic character strings. The creation of the “discourse” and “disconnected sentences” conditions was achieved by using the definite article “the” in one set of sentences and indefinite articles (“a” or “an”) in another set. This manipulation produced no change in left hemisphere activation, but activation specific to connected sentences was found in middle and superior frontal regions of the right hemisphere.

Another narrative processing experiment from the same laboratory (Robertson 2000) showed that narrative processing increases in right hemisphere activation in more distributed regions: the precuneus, cuneus, posterior cingulate, parieto-temporo-occipital regions (bilaterally), the frontal poles, and a stretch of cortex extending along the right superior temporal sulcus to the right temporal pole. Interestingly, many of these same areas are activated when participants are asked to process picture stories in which a succession of pictures is presented such that a simple narrative like those used in the reading task is constructed. This suggests that these regions of activation are not specialized language centers but represent more general processes involved in story comprehension (e.g., a memory retrieval process).

St. George et al. (1999) asked participants to read both titled and untitled stories. They found patterns of activation nearly identical to those reported by Robertson et al. (2000). However, the right hemisphere activation was found to be stronger when the stories were untitled. This may indicate a greater degree of effort being expended to create a coherent episode out of the untitled stories relative to the titled stories.

Interestingly, Mazoyer et al. (1993) report a study in which participants listened passively to stories, and none of the right frontal regions reported by Robertson et al. (2000) or St. George et al. (1999) showed activation. The lack of right frontal activation was also observed by Tzourio et al. (1998). This suggests that the right hemisphere activation may only arise when the participant is building a story representation, rather than passively processing speech sounds.

The general finding that discourse processing involves extensive right hemisphere regions is in keeping with other studies noting the role of the right hemisphere in different aspects of language processing: processing prosody (Buchanan et al. 2000), comprehending irony and metaphor (Bottini et al. 1994), and processing words (Damasio & Damasio 1992), particularly abstract words (Kiehl et al. 1999). What we should make of this activation is not entirely clear. Studies indicate that increasing task difficulty can lead to the activation of more diffuse brain regions, and Just et al. (1996), among others, have noted that processing difficult sentences can involve the right hemisphere. That the right hemisphere activation noted in relation to sentence or lexical processing represents the same thing as the activation observed in the studies reported here is unlikely, given the subtractions that were performed (discourse–unrelated sentences). Indeed, the activation of the temporal poles during discourse comprehension appears to be unique to this task (relative to other language tasks), indicating that there are specific processes above and beyond those required for sentence processing involved in discourse processing. These findings argue that the right hemisphere activation observed in these experiments is not simply the recruitment of more processing resources. The activation noted in the precuneus (a region with extensive links to other cortical regions and implicated in memory processes) suggests that the right frontal activation associated with discourse processing might be the result of particular memory processes needed to create a coherent story representation.

Studies into more specific aspects of discourse processing have also been done. Nichelli et al. (1995) performed a PET study in which participants were presented with selections from Aesop’s fables. When activation associated with a “semantic” task (e.g., answering whether or not the fable had a character with a particular characteristic) was subtracted from activation associated with a “moral” task (e.g., answering whether the fable contained a given moral), activation was observed in the right inferior-frontal gyrus and the right middle temporal gyrus.

Fletcher et al. (1995) asked participants to read three types of texts: theory-of-mind stories (which require the reader to make inferences about the internal mental states of the characters), physical stories (which are about physical events and do not require reasoning about the mental processes of the characters in the story), and collections of unrelated sentences (see Happe et al. 1996 for a similar study). Here again, story processing produced bilateral activation of the temporal poles, as well as in the left superior-temporal gyrus and the posterior cingulate cortex. Relative to the physical stories, theory-of-mind stories activated the posterior cingulate and the left middle-frontal gyrus. The posterior cingulate has connections throughout the cortex, including prefrontal and middle temporal regions. This region has been associated with the encoding of episodic memory (Grasby et al. 1993). Its activation may reflect the fact that the theory-of-mind stories are processed into a coherent episode, although this does not explain the difference in activation seen between the physical stories and theory-of-mind stories.

Brockway (1999, Brockway et al. 1998) noted that hippocampal structures are important to the role of memory in story processing and storage. In a study of patients with temporal lobe epilepsy, they demonstrated that damage to the left hippocampus impaired long-term memory for connected discourse (~35 min separated reading of the texts and recall). Based on the results of this and other memory tests, the authors concluded that the left temporal region (particularly, but perhaps not exclusively, the hippocampus) is of critical importance for maintaining the coherence of incoming information such as that presented by discourse.

The processing of discourse therefore appears to involve a distributed network of brain regions. These include the areas involved in lower levels of language processing (words, sentences, etc.), as well as areas specific to discourse: right temporal and frontal regions (important for the integrative aspects of discourse processing), as well as both temporal poles. The exact function of these regions is not yet known. A current hypothesis is that the discourse-specific activation seen in these studies is a function of memory processes that maintain the coherence of the discourse across sentences.

Particularly intriguing in this area of research is the activity of the temporal poles. This is a cortical region about which relatively little is known. The poles have been implicated in discourse processing (see above), and it is believed that they may serve an integrative function, sharing connections with the temporal and frontal lobes. Damasio et al. (1996) have also demonstrated (using patients with lesions in and around the temporal poles and controls with intact temporal regions) that the temporal poles are involved in lexical retrieval. Because so little is known about the temporal poles, the role of this region in language processing marks an important direction for future exploration.

Neuroimaging and Language Organization

In addition to exploring issues of where particular aspects of language are processed in the brain, neuroimaging experiments have been performed to explore issues related to the differences in language organization across different populations (e.g., males versus females, different categories of bilinguals).

Shaywitz et al. (1995) reported an experiment that looked for differences in activation for males and females in letter recognition, rhyme, and semantic tasks. Differences were found in the activation associated with the rhyme task (subtracting the letter processing task from the rhyme task): Whereas males showed activation that was strongly left lateralized in the inferior frontal region, females showed bilateral activation in this region.

Contrary to these findings, Frost et al. (1999) argued that there are no substantive differences in the organization of language in males and females. In this experiment participants alternated between a tone monitoring task (respond to sequences containing two “high” tones) and a semantic classification task (“Is this animal found in the United States and used by humans?”). As in the Shaywitz et al. (1995) study, no difference was found in the semantic processing of males and females. The authors assert that gender differences in processing may exist at the level of the microstructure of the brain but are absent from the macrostructure: For both sexes, language is strongly left lateralized. However, they did not replicate the phonological processing task of Shaywitz et al. (1995), leaving open the possibility that there is a genuine contrast between males and females with regard to that type of processing.

A similar debate over language organization has centered around bilinguals. The question is whether the native language of the bilingual (L1) and the second language (L2) share the same cortical space or whether they occupy different space. In a number of studies differences in where L1 and L2 are localized (Kim et al. 1997) and processed (Perani et al. 1996) were noted.

Chee et al. (1999b) conducted a word processing study with bilingual speakers of Mandarin and English. They collected data from bilinguals who were exposed to both languages before age 6 and bilinguals who learned Mandarin at an early age but were not exposed to English until after age 12. Participants were given word-stem completion tasks in both languages. The results of the study suggested no differences in the cortical organization or processing of language in this task, comparing across both L1 and L2, and across early and late acquisition of L2. This is in contrast to the findings of earlier studies suggesting differences in organization between L1 and L2. For example, Dehaene et al. (1997) report differences in sentence processing between L1 and L2 English-French bilinguals, with L2 processing requiring significantly more right hemisphere activation.

A possible way to reconcile the data observed in exploring L1-L2 differences is to note that in certain cases participants may use different strategies to perform the experimental task with L1 and L2. For instance, bilinguals may need to use a different strategy when processing L2 sentences than when they are processing L1 sentences. Or, in keeping with Just et al. (1996) it may be that task performance with L2 requires more processing resources. One way to clarify this issue is to design studies in which strategies or processing difficulty are controlled and can be ruled out as causal factors.

Another strategy might be to explore monolingual processing in each language as a preliminary step in the research process. In the case of visual processing of language, languages might differ in the complexity of the characters in the alphabet or in the degree to which particular letters can be translated into phonological forms (e.g., Japanese character systems, in which one system of characters have phonological counterparts and the other system has only semantic counterparts) (Kamada et al. 1998). To illustrate, it has been shown that American Sign Language (ASL) has a different cortical organization than English, involving a more widely distributed network of structures in both the right and left hemisphere (Neville et al. 1998). This is presumably due to the greater extent to which spatial and motor information is required to both produce and comprehend utterances in ASL. The pattern of activation on a given task in monolinguals can be an important baseline against which to judge activation in bilinguals. If the two languages have different processing requirements (like English and ASL), one might expect to see different areas of the cortex occupied with L1 and L2 in bilinguals regardless of whether L1 and L2 actually require different cortical space. However, if the processing requirements of the languages are similar, then one might propose, a priori, that there ought to be no substantial differences in the organization of L1 and L2 unless something about the acquisition of a second language requires this to be the case. In these circumstances, and to the degree that strategies, etc. can be ruled out as causal factors, one can interpret language-related differences in cortical activation of bilinguals more strongly.

Summary

Across the studies reviewed above the following conclusions about neural activation in language production and comprehension can be drawn:

FRONTAL REGIONS

The inferior frontal regions, including parts of Broadmann’s areas (BA’s) 45, 46, and 47, show activation during tasks involving phonological processing, semantic decision tasks, and sentence- and discourse-level processing. These regions are also believed to be involved in short-term memory storage and/or maintenance. The middle and superior frontal regions show activation mainly during semantic decision tasks (BA 6, 8, 9) and have also been implicated in semantic memory tasks. These regions also showed activation in tasks that required the processing of theory of mind information within a story. The supplementary motor area (BA 6, 44) and Broca’s area (BA 6, 44) have been implicated in producing verbal and nonverbal motor responses to tasks, in the maintenance of phonological representations, and in the production of subvocalizations (Hinke et al. 1993). Broca’s area has also shown activation during syntactic processing, as well as during music perception tasks (Maess et al. 2001).

Similar regions of activation occur in the right hemisphere. The inferior frontal region is active during processing of abstract words, sentence processing, discourse processing, detection of emotional content in speech, and phonological processing in limited cases. The middle and superior frontal regions have shown activation during semantic decision tasks and have been implicated in the integrative aspects of discourse processing.

TEMPORAL AND POSTERIOR REGIONS

The superior temporal region (BA 22) has shown activation during auditory processing of speech sounds, during semantic processing, and during syntactic processing (mostly in anterior regions). The middle temporal region (BA 21) has been implicated in both phonological and semantic processing. The temporal pole (BA 38) has been implicated in discourse-level processing. Wernicke’s area (BA 40) as well as the supramarginal gyrus (BA 40) have been suggested to play a role in semantic processing and certain aspects of phonological processing, such as translating written words into phonological representations. Posterior inferior-temporal regions, as well as the occipito-temporal sulcus (BA 42, 37) have been implicated in the early visual processing of words, as have the angular gyrus (BA 39/40), the fusiform gyrus (BA 37), and the lateral extrastriate regions (BA 18/19). Superior parietal regions (BA 7, 19) have shown activation in the processing of discourse. The cerebellum has shown activation during cognitive search, decision tasks, and naming tasks.

In the right hemisphere many of these same regions show activation. The superior parietal regions show activation during discourse processing, as does the temporal pole. Superior temporal regions in the right hemisphere show activation during early speech processing, processing of difficult sentences, and discourse processing. Many of the visual processing regions are activated bilaterally in the early stages of word processing. Right temporal regions are also associated with the processing of prosody.

Two things need to be kept in mind when faced with this many-to-many mapping of structure and function. First, the labels attached to these tasks (e.g., “semantic decision”) are probably best considered a convenient means of categorization. The labels are often so broad as to be next to useless in making generalizations across experiments. Second, the activation displayed in imaging studies is observed across trials and across participants. Thus, what is shown as active in a particular experiment is what is common to task performance across the experiment. This issue was discussed above when we considered word processing. A semantic decision task may activate frontal regions, but that does not preclude the possibility that these regions are accessing information from other regions of the cortex to actually perform the task. This additional activation for individual trials may not necessarily register as activity associated with the task. This failure to register might be because (a) the additional information is located in diffuse cortical regions and these cancel out across trials or (b) the circuitry involved in accessing this information does not produce a sufficiently strong signal (relative to baseline) to show up as significant activation.

THE FUTURE OF NEUROIMAGING WORK ON LANGUAGE PRODUCTION AND COMPREHENSION

The work presented in this review constitutes a groundbreaking effort by physicists, neuroscientists, radiologists, and psychologists to harness neuroimaging techniques in an attempt to observe the brain at work. In many ways the rules of the game are still being developed. Statistical analysis, experimental methodology, and imaging technology are rapidly advancing to allow more precise and veridical information about neural processes to be obtained. Indeed, studies using magnetoencephalography (MEG), an imaging technology with better spatial and temporal resolution than fMRI, have already appeared (e.g., Kamada et al. 1998, Patel & Balaban 2000).

The previous section summarized the state of the art regarding what is known from imaging studies about the neural circuitry that underlies language processing. This literature mainly focuses on answering the “where” question of neural processing. Across the body of studies reviewed, one can get a reasonable sense of the cortical regions that process language: The left hemisphere, superior, middle, and inferior temporal regions, as well as the temporal poles and superior, middle, and inferior frontal regions are implicated. In the right hemisphere, superior temporal regions, frontal regions, and homologues to left hemisphere language areas are implicated. Of course, many of these findings need to be replicated across a wider range of experimental paradigms before more stock is placed in them.

As we become more certain about the “where” questions in language processing, we must begin to consider the important question of “how?” As the temporal and spatial quality of imaging techniques improves, we can begin to look at the microstructure of the cortex to ascertain exactly how processing of a given type is carried out. The mapping work been done to this point will be an invaluable guide in this respect. Having some sense of where to look for a given type of process will greatly simplify the quest for the fine details of how the brain works. There is no guarantee that more precise observation of the brain will yield valuable information, but there is every hope that more precise observation, in combination with clear thinking about how the methods developed can best be applied to the task at hand, will shed light on the neural processing underlying language.

That said, there are a number of realms of psycholinguistics that have yet to be explored in any detail in neuroimaging studies. The processing of pragmatic information in language comprehension, as well as conversational aspects of language use, are an example of such an area of research that needs to be addressed (but see e.g., Caplan et al. 2000). Investigation can also be done to ascertain if there are any anatomical properties of “language areas” that cause linguistic processing to localize there regularly across the population (in keeping with the proposal of Jacobs’s 1997 “mixture of experts” theory). As we noted in the previous section, the accessing of particular “semantic” information in task performance has not been adequately addressed and may be explored as described in that section. Exploration of all of these areas of language use may reveal important information about the neural processing the brain carries out.

None of this will be easy. An obstacle to understanding neural processing may arise from the field’s co-opting of cognitive psychological methods and metaphors. The information-processing model of cognition, which underlies much work in psycholinguistics, arose at a time when the neural operations involved in cognition could mainly be discussed metaphorically. This framework has proven useful in yielding experimental paradigms and task analyses of the participants’ performance in different experimental settings. Nonetheless, there is reason to suspect that nature has not cooperated by designing the brain to match our information processing intuitions. This likely explains why we observe the many-to-many mapping of structures and putative processes across imaging studies.

As Kosslyn (1999) argues, the results of neuroimaging studies are of interest only to the degree that they are theoretically informative; scanning participants while they perform tasks and interpreting the areas of activation post hoc is not in and of itself an interesting research enterprise. Cognitive theories allow the neuroscientist to avoid this by developing tasks and hypotheses that are potentially informative. Nonetheless, it may be that making fundamental progress in understanding how the brain processes language will require the exploration of neural processing in a way that does not rely as heavily on the theoretical baggage of cognitive psychology.

Three years into the new millennium, we can reflect on the Decade of the Brain (as the 1990s were dubbed) as an important time in cognitive neuroscience. Technological developments merged with an expansion of researcher interest to help the field of neuroimaging blossom into a viable young science. Whereas much work remains to be done in the quest for cognitive neuroscience’s Holy Grail, the research described herein constitutes a good start along the path.