Neural Aspects of Sentence Comprehension: Syntactic Complexity, Reversibility, and Reanalysis

Abstract

Broca's area is preferentially activated by reversible sentences with complex syntax, but various linguistic factors may be responsible for this finding, including syntactic movement, working-memory demands, and post hoc reanalysis. To distinguish between these, we tested the interaction of syntactic complexity and semantic reversibility in a functional magnetic resonance imaging study of sentence–picture matching. During auditory comprehension, semantic reversibility induced selective activation throughout the left perisylvian language network. In contrast, syntactic complexity (object-embedded vs. subject-embedded relative clauses) within reversible sentences engaged only the left inferior frontal gyrus (LIFG) and left precentral gyrus. Within irreversible sentences, only the LIFG was sensitive to syntactic complexity, confirming a unique role for this region in syntactic processing. Nonetheless, larger effects of reversibility itself occurred in the same regions, suggesting that full syntactic parsing may be a nonautomatic process applied as needed. Complex reversible sentences also induced enhanced signals in LIFG and left precentral regions on subsequent picture selection, but with additional recruitment of the right hemisphere homolog area (right inferior frontal gyrus) as well, suggesting that post hoc reanalysis of sentence structure, compared with initial comprehension, engages an overlapping but larger network of brain regions. These dissociable effects may offer a basis for studying the reorganization of receptive language function after brain damage.

Introduction

A key topic in the study of sentence comprehension is the determination of thematic roles, that is, “who is acting on whom.” As lexical recognition of the component words of a sentence does not suffice to determine this, languages depend on syntactic information such as word order and case marking to convey the information. Thematic role comprehension is often assessed clinically with a sentence picture–matching task, in which subjects hear a sentence and must then select a matching picture from a field that may include a syntactic foil picture, in which the thematic roles are reversed (e.g., a boy tickling a girl vs. a girl tickling a boy). This task has played a major role in neurolinguistic research since the seminal study of Caramazza and Zurif (1976), which examined syntactic comprehension deficits in Broca's aphasics. Patients exhibited chance comprehension performance on semantically reversible sentences containing noncanonical object-embedded relative clauses, such as “The girl that the boy is tickling is happy.” However, comprehension was largely spared on sentences with a simpler syntactic structure (a subject-embedded relative clause), such as “The girl that is tickling the boy is happy.” Additionally, performance was intact on irreversible sentences, in which the meanings of the words strongly constrained the possible thematic roles, for example, “The apple that the boy is eating is red,” as assessed with lexical foils not involving role reversal.

Consistent with the finding that damage to Broca's area impairs comprehension of syntactically complex reversible sentences, numerous neuroimaging studies have demonstrated selective activation in the left inferior frontal gyrus (LIFG, aka Broca's area) for these sentences (Just et al. 1996; Stromswold et al. 1996; Caplan et al. 1998, 1999; Ben-Shachar et al. 2003, 2004), although other studies using similar contrasts have only found effects in other regions (Caplan 2001; Caplan et al. 2002; Yokoyama et al. 2007). However, the interpretation of the activation in LIFG remains subject to vigorous debate, paralleled by a similar debate in the lesion literature. Some researchers have asserted (Grodzinsky 1995, 2000; Beretta et al. 1999) that specific computations related to processing syntactic movement (a feature of noncanonical sentences) are localized to LIFG. Other researchers have attributed syntactic comprehension deficits not to a loss of grammatical knowledge (Linebarger et al. 1983) but instead to the fact that comprehension of complex reversible sentences depends on more general cognitive resources such as working memory (Carpenter et al. 1995), although whether this is a specific form of “syntactic” working memory is also under debate (Caplan and Waters 1999). A special role of working memory in complex sentence comprehension is supported by findings that noncanonical sentences are somewhat difficult for neurologically intact subjects to process as well (Dick et al. 2001; Traxler et al. 2002) and hence may place increased demands on general cognitive resources that are likely to be impaired in patients with any kind of frontal damage (Haarmann et al. 1997; Caplan 2006).

The debate over the interpretation of comprehension deficits in Broca's aphasia has been accompanied by considerable empirical debate over the true prevalence and specificity of the deficit. Although a selective deficit for syntactically complex sentences has been statistically linked with damage to Broca's region in a large sample of patients (Drai and Grodzinsky 2006), other studies have questioned the significance of this finding, emphasizing high individual variability in comprehension performance across patients diagnosed with “agrammatic aphasia” (Berndt et al. 1996; Caplan et al. 2007; Johnson and Cannizzaro 2009). The association of syntactic comprehension deficits with a variety of lesion sizes and locations suggests that multiple factors play a role in rendering reversible complex sentences vulnerable to comprehension failure.

The goal of the present study is to elucidate the nature of selective activations to reversible object-embedded clauses by distinguishing the effects of 2 different factors that make these sentences hard to process. We employed an auditory sentence picture-matching task, in which subjects first heard a sentence and then selected a matching picture from a field of 2. A 2 × 2 factorial design crossed complexity (object-embedded vs. subject-embedded relative clauses) with reversibility. The first factor is commonly referred to as either canonicity or syntactic complexity. In sentences containing an object-embedded clause, the “patient” of the sentence is mentioned before the “agent.” Although these sentences are grammatically correct, they violate a strong “agent-first” bias in English (and many other languages), in which the performer of an action is mentioned first in the vast majority of sentences. In this respect, object-embedded clauses, along with some other structures, are considered to be “noncanonical.” Because some theories of syntax derive these sentences through constituent movement that is more complex than the movement involved in subject-embedded sentences (Grodzinsky 1995, 2000), syntactic complexity is a common term for this factor, and we will use it in this paper. Notably, other syntactic structures that violate the agent-first bias also elicit comprehension impairments in Broca's aphasics, including passive voice (Luzzatti et al. 2000) and scrambling (Beretta et al. 2001).

A second factor is semantic reversibility. Behavioral experiments have suggested that both word order and semantic constraints influence sentence interpretation in parallel (Bates et al. 1982) and that the syntactic cues available to listeners are not always fully processed in the course of normal language comprehension, as subjects may rely on a simpler heuristic strategy based primarily on word meaning (Ferreira et al. 2002; Sanford and Sturt 2002; Ferreira 2003). In the aphasia literature, it has been suggested that such heuristics underlie the preserved comprehension abilities of patients for noncomplex sentences (Caramazza and Zurif 1976; Grodzinsky 1995, 2000; Beretta et al. 1999). In the context of a sentence picture–matching task, it is expected that reversible sentences should elicit more intensive syntactic processing for normal subjects than irreversible sentences, because explicit consideration of word order is necessary for task completion only in the reversible sentences. The existence of specialized mechanisms for processing word-order information is bolstered by findings of patients exhibiting chance comprehension on reversible sentences in general, regardless of complexity (Davis et al. 2008; Miozzo et al. 2008). Therefore, we examined the effect of reversibility in the present study, asking to what extent the areas responsive to this factor overlap with areas responsive to syntactic complexity. The results of this comparison inform the debate on whether reversibility and syntactic complexity tap the same underlying cognitive mechanisms.

Crucially, the interaction between the 2 factors serves to elucidate whether LIFG activation for reversible complex sentences is related to the automatic processing of syntactic movement, or rather to more general cognitive demands induced by these sentences, such as working memory. Some imaging studies have combined comparisons of syntactic structure with other manipulations of working memory load, such as the distance between a moved constituent and its trace position. These studies have found that long-distance dependencies preferentially activate Broca's area, suggesting that this brain region may be selectively activated by syntactic working memory rather than by particular syntactic structures (Cooke et al. 2002; Fiebach et al. 2005; Santi and Grodzinsky 2007). In the present study, however, we keep the distance of the dependency constant, and instead compare the effect of complexity within reversible and irreversible sentences. Example sentences from all conditions are shown in Table 1.

Table 1.

Example sentences

Code	Reversibility	Syntactic complexity	Example
RSS	Reversible	Subject-embedded clause	The boy who is tripping the girl hopes to win the race.
RSO	Reversible	Object-embedded clause	The boy who the girl is tripping hopes to win the race.
RAC	Reversible	Simple active	The boy is tripping the girl in order to win the race.
ISS	Irreversible	Subject-embedded clause	The boy who is burning the paper gets in trouble a lot.
ISO	Irreversible	Object-embedded clause	The paper that the girl is burning is an old telephone bill.
IAC	Irreversible	Simple active	The boy is burning the paper with a new lighter.

In Table 2, we enumerate the predictions of specific contrasts tested in this experiment, according to 2 alternative positions, which we refer to as a “syntactic account” and a “cognitive account.” In either case, reversible object-embedded clauses are expected to activate LIFG, as has been amply demonstrated. According to the syntactic position, LIFG is sensitive to the presence of object-embedded clauses due to the long-distance dependency in them and should therefore be selectively activated by the complex sentences regardless of reversibility. Reversibility itself may also activate the same area but should not interact with complexity. According to the cognitive position, however, the enhanced signal for complex sentences should occur only within the reversible category. Only the reversible sentences present a challenge to comprehension, as evidenced by increased reaction times (RTs) and error rates, and increased signal reflects the extra mental effort needed to determine thematic roles in this case. Therefore, there should be no general effect of complexity, but instead a complexity by reversibility interaction, driven by the increased signal specific to the reversible object-embedded clauses. Note that these 2 positions are not mutually exclusive for the whole brain—there may be some areas that respond chiefly to the cognitive demands, whereas others exhibit a specific sensitivity to syntactic structure.

Table 2.

Predictions of 2 accounts of LIFG function

Effect description	Conditions contrasted	Syntactic prediction	Cognitive prediction
Complexity within reversible	RSO–RSS	Yes	Yes
General effect of complexity	(RSO + ISO) − (RSS–ISS)	Yes	No
Complexity by reversibility interaction	(RSO–RSS) − (ISO–ISS)	No	Yes
Complexity within irreversible	ISO–ISS	Yes	No
General effect of reversibility	(RSS + RSO) − (ISS + ISO)	?	?
Reversibility within noncomplex	RSS–ISS	?	?

In addition to the goals listed above, the design of the present experiment also allows for an additional comparison of key interest for theories of syntactic comprehension. Some authors have suggested that comprehension of complex syntax involves cognitive processes extending in time well beyond the presentation of the sentence, known commonly as “reanalysis” (Caplan and Waters 1999). When presented with a difficult sentence, a subject may rethink the sentence as it is held in working memory over several seconds, and this process may involve different mechanisms than normal online comprehension. In reading experiments, this may be seen as an increase in eye movements back to the relative clause (Traxler et al. 2002). Reanalysis processes may play a role in producing selective activation to complex sentences, as increased activation in LIFG has been observed in studies that manipulated the difficulty of a postsentence comprehension probe (Love et al. 2006; Caplan, Chen, et al. 2008).

In the present experiment, a jittered rapid event-related functional magnetic resonance imaging (fMRI) design was used to distinguish between activity attributable to auditory sentence processing and to subsequent picture selection. Syntactic complexity effects detected in the hemodynamic responses to auditory sentence presentation will reflect online comprehension processes, although a component of reanalysis may also be present. However, complexity effects on the responses to subsequent picture presentation can be interpreted as primarily reflecting processes of effortful post hoc reanalysis, as they are temporally decoupled from the presentation of the actual sentence. Thus, the design of this experiment can identify dissociable effects of syntactic complexity at the stages of online comprehension and post hoc reanalysis, testing whether they rely on the same brain areas.

Materials and Methods

Subjects

Twenty-four healthy volunteers (12 female, age 22–37) were recruited from the NIH community. All were right-handed monolingual native speakers of English. Subjects gave informed consent (NIH protocol 92-DC-0178) and were financially compensated. All subjects participated in 2 experimental sessions. In the first session, electroencephalography (EEG) and magnetoencephalography (MEG) data were acquired simultaneously. These data were intended to explore the time course of syntactic comprehension in greater temporal detail and will be reported separately. However, the first session was also used to gather more detailed behavioral information, relevant for interpretation of the fMRI data. Behavioral data from the EEG/MEG session are therefore reported in supplementary information.

Materials

Five hundred and forty sentences were composed for this experiment, in 6 categories, for a 2 × 3 factorial design (semantic reversibility × syntactic complexity). Examples of the 6 conditions are shown in Table 1, and a more detailed description of the sentences is given in supplementary information. All sentences involved one or 2 of 4 possible people, namely, “the boy, the girl, the man, and the woman.” Reversible sentences (R) involved a human as both subject and object and were constructed to avoid plausibility biases. Irreversible sentences (I) involved 1 human and 1 inanimate object. Three levels of syntactic complexity were employed: simple actives (AC), subject-embedded relative clauses (SS), and object-embedded relative clauses (SO). The abbreviations (RSS, RSO, etc.) will be used throughout the paper to denote the 6 conditions, in accordance with previous studies of relative clause processing (e.g., Caplan, Stanczak, et al. 2008). The primary contrast of interest is between the 2 types of embedded clause, but the active condition was included as “filler” to reduce subjects’ habituation to the relative clause structure and also included in analyses of behavioral data. The 540 sentences were sorted into 15 runs, each run containing 36 sentences, 6 of each condition. Seven runs were used in the MEG experiment, and 7 in the fMRI experiment, with the remaining run reserved for practice.

FMRI Task

The fMRI task comprised a sentence picture-matching paradigm, in which a subject first heard a spoken sentence and then viewed 2 pictures, selecting the matching picture via a button press. Registration of the subject's choice was confirmed by highlighting the selected picture in a green box, but no accuracy feedback was given. A jittered event-related design was used, in order to distinguish between activity related to auditory sentence perception and picture selection. Despite the fact that picture selection always followed sentence presentation, it was possible to disentangle the hemodynamic responses related to the 2 task stages using linear regression (Miezin et al. 2000), along with 2 techniques that served to reduce the correlation between the hemodynamic responses of the 2 stages: temporal jitter and partial trials (Ollinger et al. 2001). Subjects were informed that a random subset of the sentences would be followed by a picture-matching trial, and instructed to attend to each sentence in preparation for a possible response. Subjects were informed that they could forget about the proceeding sentence as soon as a new one began. Accordingly, only 50% of the sentences were followed by a picture-matching trial. The delay between each stimulus event, either sentence or picture, was jittered as 6, 8, or 10 s.

The experimental design is illustrated in Figure 1A. The “partial trial” method, combined with hemodynamic deconvolution, allows all sentences in a given condition to be treated identically in the statistical analysis, regardless of whether or not they were followed by a picture, as the picture events were modeled separately. Mathematical simulations were carried out prior to data collection to ensure the statistical adequacy of the experimental design.

Figure 1.

Task design. (A) Trial structure for the fMRI experiment, in which both “partial trials” (sentence only) and “full trials” (sentence and pictures) were presented, in order to disambiguate hemodynamic responses for the 2 events. (B) A sample picture set for the reversible sentence “The woman who the man is teaching is very tired right now.” The target shows the correct arrangement. A syntactic foil has the thematic roles of the 2 named actors switched, whereas a lexical foil (not used in the fMRI experiment) substitutes one of the actors. (C) A sample picture set for the irreversible sentence “The glass that the man is washing has a small chip in it.”

For reversible sentences, the 2 pictures featured 1 correct depiction and 1 syntactic foil in which the roles of the 2 people are reversed (Fig. 1B). In the MEG pretest session, we also included some trials with lexical foils, in which a different person is depicted other than the 2 mentioned in the sentence, thus allowing the subject to determine the correct answer by lexical information alone. However, these were not used in the fMRI experiment (see supplementary data for the behavioral pretest results). For irreversible trials, the foil picture randomly substituted either the agent (the person performing the action) or the patient (the inanimate object acted upon).

FMRI Acquisition

Whole-brain gradient-echo echo-planar imaging (EPI) data were acquired on a 3-T GE Signa scanner with an 8-channel head coil (repetition time [TR] = 2000 ms, echo time = 30 ms, flip angle = 90°, 64 × 64 matrix, field of view 224 mm, 38 slices, 3.5mm thick, obliquely aligned to the plane between the anterior and posterior commissures). A 1-mm isotropic magnetization prepared rapid acquisition gradient echo (MPRAGE) image was also acquired. Two hundred and twenty-six volumes were acquired in each run (preceded by dummy scans to achieve stead-state magnetization), with 7 runs total. Auditory stimuli were presented through pneumatic headphones (Avotec, Inc., Stuart, FL) at an individually adjusted volume level. Blood oxygen level–dependent (BOLD) images were preprocessed in AFNI software with standard steps, including brain extraction, motion correction, spatial smoothing (8 mm full width at half maximum), and voxelwise time course normalization to percent of mean signal level. Deconvolution of hemodynamic responses was performed on individual subjects in their native brain space, and the results were transformed into Talairach space using a 6-parameter rigid transformation from the EPI image to the MPRAGE and nonlinear grid-based deformation (Papademetris et al. 2004) to the “colin27” brain in Talairach space. Coregistration and warping were done using the program BioImage Suite (http://bioimagesuite.org/). Warped statistical maps were interpolated to isotropic 2-mm voxels.

Voxelwise Statistical Analysis

BOLD runs were analyzed with a general linear model approach, using a series of 7 lagged “tent” basis functions for each condition in order to generate an empirical estimate of the hemodynamic response, rather than a single standardized function. Additionally, a set of third-order Legendre polynomials was included in the regression model to account for slow signal drift, along with estimates of motion parameters for each volume to reduce the influence of motion-induced signal changes. Six different conditions of sentence presentation were modeled separately, along with 6 different conditions for the picture-matching events. When errors occurred (incorrect or absent responses), the corresponding sentence and picture events were removed from the condition-specific trial regressors and modeled as a separate condition, which was not analyzed further. The resulting basis coefficients for each condition were integrated into an estimated time course of the hemodynamic response, with 13 time points covering multiples of the TR from 0 to 24 s relative to stimulus onset. Estimates of hemodynamic response (HR) magnitude for second-level statistical analysis were generated by averaging the time course estimates from 2 to 12 s poststimulus (time points 2–7), as visual inspection of averaged time courses demonstrated that this range was sufficient to capture the entire positive peak of the BOLD response without including poststimulus undershoot.

HR magnitude estimates were entered into a voxelwise repeated-measures analysis of variance (ANOVA), implemented in AFNI (http://afni.nimh.nih.gov/sscc/gangc/ANOVA.html), with subject as a random factor and reversibility and syntactic complexity as within-subject factors. Separate ANOVAs were computed for sentence and picture effects. As the simple active sentence structure is not central to the hypotheses of the experiment, only the conditions involving a relative clause structure (SS and SO) were included in the ANOVA, resulting in a 2 × 2 factorial design. Although the ANOVA did produce F-tests, these tests are not directional and may yield a mixture of qualitatively different effects across brain regions. Given the directional hypotheses of the study, we used one-tailed t-tests as planned contrasts within the ANOVA, to test for general effects of the 2 factors reversibility and complexity (collapsing across levels of the other factor), interactions, and when indicated, post hoc tests of direct contrasts between 2 individual conditions. See Table 2 and the Results section for the specific tests run and their interpretations. Correction for multiple comparisons in whole-brain maps was achieved through setting a cluster-size criterion, combined with a voxelwise threshold of P < 0.01. Monte Carlo simulations with the AFNI program “Alphasim” were used to set a cluster criterion of 220 contiguous voxels (1.76 mL), for a whole-brain family-wise error of P < 0.05. In one case, a smaller search volume was used, as indicated in the Results section. For display of time courses, spherical regions of interest (ROIs) of 6-mm radius were placed at the center of mass of activations of interest, and the event-related time courses (constructed by integrating the basis function coefficients) were averaged across voxels in the ROIs.

Results

Behavioral

RT and accuracy were recorded during MRI scanning and yielded similar effects as seen in the behavioral data from the MEG–EEG experimental session that was conducted prior to the MRI scans on the same subjects. Results from the MEG session, presented in the supplementary information, are a more definitive characterization of the different conditions, as every sentence was followed by a picture-matching event, with a consistent delay between the end of the sentence and the picture onset. For the MRI session, only 50% of the trials had a picture-matching event, and the delay time between sentence and picture was variable. Also, the MRI session used only syntactic foils in reversible trials, which were found to produce longer RTs and more errors than lexical foils in the pretest data.

RTs during MRI for all 6 sentence conditions are shown in Figure 2A. RTs were averaged within subject for each condition and submitted to a 2 × 3 repeated-measures ANOVA, with reversibility (R = reversible and I = irreversible) and syntactic structure (SS = subject-embedded, SO = object-embedded, and AC = simple active) as within-subject factors. There was a highly significant effect of reversibility (F(1,23) = 14.43, P < 0.0001), indicating that subjects were slower to respond to the reversible pictures involving 2 people than to the irreversible pictures involving 1 person and an object, regardless of the grammatical structure of the sentence. There was a main effect of syntactic structure (F(2,46) = 14.23, P < 0.0001), and also a strong interaction between reversibility and syntax (F(2,46) = 17.58, P < 0.0001). Inspection of the condition-specific RTs reveals that the interaction is driven by an elevated RT specifically to reversible object-embedded clauses (the RSO condition). This was expected, given that these sentences have previously been shown to be somewhat difficult to process even in neurologically intact individuals (Dick et al. 2001; Traxler et al. 2002).

Figure 2.

Behavioral results. (A) RT to picture-matching trials during the fMRI experiment, across the 6 conditions of sentence type. (B) Error rate (% incorrect or no response) across conditions in the fMRI experiment.

Error rates are presented in Figure 2B. As with RTs, error rates within each condition and subject were submitted to a 2 × 3 repeated-measures ANOVA. There was no main effect of reversibility (F(1,23) = 2.34, P = 0.14), but there was a main effect of syntactic structure (F(2,46) = 5.23, P = 0.009) and a much larger interaction effect between reversibility and syntax (F(2,46) = 11.53, P = <0.0001). The interaction is driven by an elevated error rate in the RSO condition, about 8% on average. This is still well above chance performance but approximately twice as many errors as most of the other conditions. Along with the RT data, the error rate indicates that RSO sentences present a special processing challenge to the listener compared with other sentences used in this study. Overall, error rates in the fMRI experiment, although quite low, were slightly higher than those seen in the MEG experiment, possibly reflecting the increased difficulty of speech perception in the presence of scanner noise.

fMRI Effects of Syntactic Complexity and Reversibility on Sentence Comprehension

We report here on differential responses between contrasting sentence conditions. In order to ensure that these effects are not attributable to any low-level differences between sentences, such as length, volume, and pitch, we also examined the hemodynamic responses to sentences and pictures in early sensory areas. Responses in auditory and visual areas showed equivalence between the different conditions (Fig. S2C,D), demonstrating that the observed differences reported below are attributable to higher cognitive factors.

General Effect of Reversibility

Figure 3A presents the effect of reversibility in general, in the form of a directional contrast between both the reversible relative clause conditions (RSS and RSO) and the corresponding irreversible conditions (ISS and ISO). We found that reversible sentences produced greater activation in almost all of the left-hemisphere brain areas that are thought to be core areas in auditory language comprehension, with the exception of primary auditory cortex, where responses were equivalent across all conditions. Areas with increased responses to reversible sentences include LIFG (or Broca's area), left precentral gyrus (including premotor cortex), anterior temporal cortex, posterior middle temporal gyrus, posterior superior temporal gyrus (Wernicke's area), and a portion of the inferior parietal lobe corresponding to the angular and submarginal gyri. Smaller activations in homologous regions in the right hemisphere were also detected. Due to the extensive activation seen in this contrast, it was difficult to cluster the activations for the purposes of reporting useful Talairach coordinates for points of activation. However, the simple contrast of RSS–ISS is a more “pure” test of the reversibility factor (as explained below) and yields essentially similar activations with smaller cluster sizes. The Talairach coordinates for that contrast are reported in Table 3C.

Figure 3.

Effects of grammatical structure on auditory sentence comprehension. (A) General effect of reversible versus irreversible sentences. Axial slices at Talairach z = 2, 14, 26, 38, and 50. The left side of the image is the left side of the brain. (B) Response to auditory sentence events in left middle temporal cortex (Talairach −55, −52, +11), an area showing an effect of reversibility but not complexity. (C) General effect of complexity, object-embedded versus subject-embedded, regardless of reversibility. Axial slices at z = 0, 5, and 10. (D) Time course in LIFG, showing effect of complexity (Talairach −46, 12, 12). (E) Complexity by reversibility interaction: Object-embedded minus subject-embedded sentence effects, in reversible versus irreversible sentences. This contrast reveals areas activated for grammatical structure within reversible sentences but not within irreversible sentences. Axial slice at z = 43. (F) Time courses of the response to auditory sentences in left dorsal premotor cortex (Talairach coordinates −44,−2, +45). (G): Time courses of the response to auditory sentences in the supplementary motor area (Talairach −7, +2, +50).

Table 3.

FMRI activation clusters

Descriptive name	BA	Volume	x	y	z
A: Sentence (RSO + ISO) − (RSS + ISS)
Ventral LIFG	44,45	1024	−45	12	13
B: Sentence (RSO–RSS) − (ISO–ISS)
L SMA	6	905	−11	−1	52
L AG	39	736	−47	−56	21
L Precuneus	31	523	−6	−57	28
L Pmd	6	478	−40	−5	45
C: Sentence (RSS–ISS)
L LPmd, SMAa	6	1844	—	—	—
L IFG, Insula	44,45,13	858	−42	18	8
L MTG, STG	22,21	706	−53	−46	11
L AG, SMG	39,40	556	−41	−57	40
RAG, SMG	39,40	423	42	−56	39
L MFG	10	369	−36	44	17
R Insula	13	352	35	21	4
L Precuneus	7	223	−10	−63	37
D: Sentence (RSO–RSS)
Ventral L IFG, insula	44,45,6,13	1086	−46	12	11
L SMA	6	511	−6	3	49
L PMd	44,6	487	−43	−3	45
E: Sentence (ISO–ISS)
Ventral LIFGb	44	48	−43	13	7
F: Picture (RSO–RSS)
L IFG SFG, insula	44,45,6,9, 13	1370	−40	−7	26
R IFG, insula	44,45,13	759	42	18	6
R Fusiform gyrus, cerebellum	37	679	41	−60	−21
L SMA	6	640	−2	7	50
G: Picture (RSS–ISS)
R MTG, STG	22,21	1720	55	−61	0
L MTG, STG	22,21	1121	−49	−773	4
Bilateral precuneus	7	432	−3	−61	54
R Cerebellum		352	25	−67	−50
L MFG	6	283	−20	−9	47
L Middle cingulate/SMA	24,31,6	277	−7	−3	44

Abbreviations: L = left, R = right, IFG = inferior frontal gyrus, SMA = supplementary motor area, PMd = dorsal premotor cortex, AG = angular gyrus, MTG = middle temporal gyrus, STG = superior temporal gyrus, SMG = supramarginal gyrus, MFG = middle frontal gyrus.

Note: Descriptive names are based on visual examination of the extent of clusters and consultation of multiple atlases. Brodmann areas listed are those into which the cluster extends. Volumes are in voxels, which are 2 mm isotropic, thus 8 mm³ in volume. Coordinates are in Talairach atlas space.

^aThis cluster is large, encompassing the separate activations detected in LPMd and SMA reported in other contrasts, including Table 3B. Therefore, the center of mass coordinates are not given, as they are located between these 2 areas of strong activation.

^bThis cluster was detected using a hypothesis-driven small-volume correction (see Results), whereas all other clusters were detected using a whole-brain correction.

Although reversible sentences induced larger signals than irreversible ones in much of the language network, effects of syntactic structure were much more limited, being mainly confined to frontal areas (see below). In Figure 3B, we present time courses from a typical posterior region, the left middle temporal gyrus. These time courses illustrate that reversible sentences induced a signal almost twice as large as irreversible sentences but that no significant differences were seen between object-embedded and subject-embedded relative clauses, in either condition of reversibility.

General Effect of Syntactic Complexity

Figure 3C presents the general effect of syntactic complexity, in the form of a directional contrast between object-embedded clauses (RSO and ISO) and subject-embedded clauses (RSS and ISS). Only one significant cluster was detected, located in LIFG (coordinates in Table 3A), consistent with previous reports. Time courses from this region are shown in Figure 3D. This area exhibits not only a large effect of reversibility but also a parallel effect of syntactic structure, with both the RSO and ISO conditions elevated over their subject-embedded counterparts. Therefore, this region seems to be truly sensitive to syntactic structure, even in the absence of a difficulty effect between the ISS and ISO conditions.

Interaction Effect between Syntactic Complexity and Reversibility

The behavioral data indicated that the RSO condition was particularly challenging, in contrast to the ISO condition that was similar to all other conditions. We used an interaction contrast (RSO–RSS) − (ISO + ISS) to identify areas in which the effect of object-embedded clauses within reversibles exceeds the effect within irreversibles; in other words, regions in which the hemodynamic response mirrors the specially increased behavioral challenge of the RSO condition. This analysis yielded 4 clusters in the brain (Table 3B). Two prominent clusters were located in the precentral gyrus, where effects of reversibility were also seen. Time courses from these clusters, the left dorsal premotor cortex (LPMd) and left supplementary motor area (LSMA) are shown in Figure 3F and G, respectively. These areas show a specific elevation to RSO sentences beyond the elevation already seen to reversible over irreversible sentences. In other words, the effect of syntactic complexity was selective for reversible sentences only. In contrast, no such interaction effect was found within LIFG, as seen in Figure 3D. That region instead exhibited 2 parallel effects of reversibility and complexity, without an interaction between these factors. This dissociation suggests that LPMd and LSMA are sensitive to the heightened cognitive demand of RSO sentences, but only LIFG is also sensitive to the syntactic contrast in the irreversible context.

Simple Contrast: RSS–ISS

Interaction effects in both the behavioral data and the hemodynamic responses in certain regions indicate that the RSO condition comprises a special challenge to the listener, beyond that expected from the addition of reversibility and complexity factors alone. Therefore, the general effect of reversibility reported in Figure 3A may include a large contribution from the RSO condition. A more pure test of reversibility alone is to contrast the 2 subject-embedded conditions, RSS versus ISS. The results of this contrast are plotted in Figure 4A and Table 3C. This contrast identified essentially the same regions as the general effect comparison but with smaller clusters. This indicates that the reversibility effects seen throughout the language network are not simply attributable to elevated signal in the RSO condition.

Figure 4.

Individual condition contrasts. (A) Reversible subject embedded versus irreversible subject embedded. (B) Reversible object embedded versus irreversible object embedded. (C) Conjunction of the above 2 contrasts, showing overlap in frontal regions. (D) Irreversible object embedded versus irreversible subject embedded, detected with small-volume correction. (E) Average time courses of the response to sentence presentation in a 3-mm radius spherical ROI centered on the LIFG cluster showing the ISO-ISS effect (Talairach −44, +13, +7).

Simple Contrast: RSO–RSS

Both the general effect of syntactic complexity and the interaction effect identify areas in which RSO induces an enhanced response over other conditions. To see these areas together, we applied the direct contrast of RSO–RSS, presented in Figure 4B and Table 3D. As expected, this yielded 3 clusters, in LIFG, LPMd, and LSMA. Notably, these areas seemed to overlap almost perfectly with the frontal regions sensitive to reversibility. To assess this overlap formally, we conducted a conjunction analysis between the “pure reversibility” contrast (RSS–ISS) and the “complexity within reversibles” contrast (RSO–RSS). The conjunction simply identifies voxels that appear in both corrected maps and preserves the bulk of all 3 clusters (plotted in Fig. 4C). These results indicate a dissociation between posterior and anterior language areas. Although the posterior areas (middle temporal, superior temporal, and inferior parietal) have greater responses to reversible sentences, there is no additional effect of the challenging RSO structure. In the anterior areas (LIFG, LPMd, and LSMA), syntactic complexity leads to additional activation beyond that attributable to reversibility.

Simple Contrast: ISO–ISS

The significant general effect of complexity in LIFG indicated that even irreversible object-embedded sentences induce an enhanced signal relative to irreversible subject-embedded sentences. However, the general effect includes a contribution from the RSO condition, which may be enough to drive the effect. A more pure contrast for sensitivity to syntactic structure would be to contrast ISO and ISS directly. This contrast did not yield any significant clusters when corrected for multiple comparisons across the entire brain. Nonetheless, the general effect in LIFG, as seen in the time courses in Figure 3D, suggests that an elevated signal to ISO sentences does exist within LIFG. Therefore, as a more sensitive test of this effect, we used a small-volume correction rather than a whole-brain correction, using the corrected activation map from the RSO–RSS contrast as a mask. Notably, the 2 contrasts are orthogonal, being computed from entirely different trials. As we would expect any area with a sensitivity to syntax in irreversible sentences to also show it for reversible sentences, the use of this empirical mask is based on the a priori hypotheses of the study. Alternatively, we could have specified an anatomical mask of Broca's area, but we also wished to confirm that the areas showing an interaction effect (LPMD and LSMA) had no significant activation for ISO–ISS, rather than the interaction effect being driven simply by a greatly enhanced effect for RSO–RSS. Using the empirical small-volume mask, a cluster size of 45 voxels (0.36 mL) was required. One significant cluster was detected, located within the LIFG, on the borderline of Brodmann areas 44 and 45 (Fig. 4D, Table 3E, time courses shown in 4E).

fMRI Effects of Syntactic Structure on Subsequent Picture Response

Even though the forced-choice picture-matching events in this task occur several seconds after the auditory presentation of the sentence, the behavioral data show that the syntactic structure of the sentence influences the process of picture selection. Specifically, there is an increased processing cost for the RSO sentences, likely due to the necessity of syntactic reanalysis of the sentence held in working memory, in order to determine the thematic roles of the named actors. Brain regions involved in this reanalysis process are revealed by the contrast (RSO–RSS) for the picture-matching events. The results of such a contrast are shown in Figure 5A and Table 3F, showing selective activation in essentially the same frontal regions as the same contrast conducted on the sentence responses (confirmed by conjunction analysis, not shown), but with the additional inclusion of the right hemisphere IFG. Also, activations for picture responses in left and right IFG extend more medially into the insula than the corresponding activations for sentence responses. We also tested for an effect of reversibility alone (RSS–ISS) on picture response, shown in Figure 5B and Table 3G. Reversibility strongly affected the response to pictures in temporal and parietal regions, but not frontal regions, which exhibited an elevated response specifically to the RSO condition alone.

Figure 5.

Effects of grammatical structure and reversibility on subsequent picture matching. (A) Responses to picture-matching events following reversible object-embedded versus reversible subject-embedded sentences. Axial slices for panels (A,B) are z = 6, 16, 26, 36, and 46. (B) Responses to picture-matching events following reversible subject-embedded versus irreversible subject-embedded sentences. (C) Time course of response to “auditory sentence events” in LIFG (Talairach −48, +13, +4). (D) Response to “visual picture-matching events” in the same area, LIFG. (E) Time course following sentences in the right hemisphere homolog of Broca's area (Talairach +48, +13, +4). No auditory response is apparent. (F) Response to pictures in the same area, RIFG.

To demonstrate the partial dissociation between left and right IFG in their response to sentences and picture selection, time courses were extracted from 2 spherical ROIs. We show here time courses from both sentence-listening and picture-matching events, so that the activity of the same brain region for these 2 very different events may be compared. Figure 5C shows auditory sentence time courses from a region in left BA 44/45 that was maximally activated for the general effect of syntactic structure on sentence comprehension and was also activated for the picture contrast (RSO–RSS). This was shown in Figure 3D but is repeated here for comparison with the picture response.

A modest effect of (SO–SS) can be seen for both R and I sentences. Upon picture presentation, however, the same region displays an elevated signal change only for the RSO condition (Fig. 5D). Figure 5E displays auditory sentence time courses for the exact same location in Talairach space but reflected to the right hemisphere. Here, in the right hemisphere homolog of Broca's area, there is no significant effect of sentence structure on the auditory response; indeed, there is barely any appreciable event-related response. However, this right hemisphere region does respond to the visual picture-matching event (Fig. 5F), with an elevated response in the RSO condition, similar to that seen in its left-hemisphere counterpart.

Discussion

This experiment examined the effects of syntactic complexity and reversibility on auditory sentence comprehension and also on reanalysis associated with subsequent picture matching. We will discuss the auditory responses first. We detected regions that were selectively activated by reversibility in general, syntactic complexity in general, and by syntactic complexity only within reversible sentences. Visual inspection, confirmed by formal conjunction analysis, revealed that these different effects were nested within a common set of areas, rather than a different set of regions for each effect.

The factor with the largest influence on the brain's response to auditory sentence presentation was reversibility. Reversible sentences induced signal changes in several regions that were nearly twice as large as the responses to irreversible sentences. Regions affected by this factor included LIFG, LPMd, LSMA, left posterior STG, left anterior and middle temporal gyri, and the angular and supramarginal gyri. Not only does this list include all regions affected by syntactic complexity in this experiment, but it also includes virtually all regions thought to be specifically involved in speech comprehension in recent models (e.g., Hickok and Poeppel 2007).

The increased activation seen in this study to reversible sentences overlaps with activations reported for semantically unconstrained sentences in 2 other recent studies (Caplan, Stanczak, et al. 2008; Richardson et al. 2009), but those studies did not observe reversibility effects throughout the entire language network as we did. Unlike those studies, which employed a plausibility judgment and passive presentation, respectively, our study employed a demanding sentence picture-matching task. The larger signals evoked by reversible sentences suggests that the presence of 2 animate nouns that are eligible to swap thematic roles results in a greater engagement of cognitive processes related to syntactic information. Thus, proper comprehension of the sentence “The boy is chasing the girl” requires the explicit consideration of syntactic information beyond simply activating lexical representations for the nouns and verbs that are mentioned. Given the necessity in this task of determining thematic roles in reversible sentences from syntactic information alone, subjects may engage in a more detailed analysis of the sentence structure, as reflected by increased activation in fMRI. For irreversible sentences, subjects may have no need to construct a full syntactic parse and may instead rely on a simpler heuristic strategy, resulting in less activation throughout the language network.

In contrast to the widespread effects of reversibility, syntactic complexity induced selective activations only in 3 left frontal clusters, in LIFG (Broca's area), LPMd, and the LSMA. Of these 3 regions, the latter 2 proved to be sensitive to the object-embedded versus subject-embedded distinction only within the reversible sentences. This is consistent with the hypothesis that these 2 regions are driven by the increased cognitive demands associated with processing the difficult object-embedded reversible sentences. These regions exhibited increased activity to processing the syntactically complex sentences only when a full syntactic parse was necessary for task completion, that is, when the sentences were semantically reversible. Thus, the syntactic engagement of LPMd and LSMA reflect the engagement of cognitive processes that may be less automatic, under executive control.

The activation in pattern in LIFG was distinctly different. Like other regions, reversibility induced the largest effect in this region, but a general effect of syntactic complexity was also seen, for both reversible and irreversible sentences. Thus, our results confirm a special role of LIFG in processing sentences with noncanonical word order, although it clearly responds to other factors as well. The region of maximal effect for the ISO–ISS contrast, the “purest” test of syntactic sensitivity, was in the inner lip of the LIFG, somewhat medial to the external surface of the brain. The Talairach coordinates of this activation are close to activations reported in other studies that have sought to identify the neural substrates of syntactic processing in an abstract sense (Friederici, Bahlmann, et al. 2006; Friederici, Fiebach, et al. 2006). Thus, these findings are consistent with theories that postulate a special role for Broca's area in the processing of syntactic movement. However, they may also be compatible with theories not based on movement, which may recognize canonicity in word order through other means. Other recent fMRI studies have indicated that departures from the canonical animate-agent-first word order activate Broca's area, even when not based on movement (Bornkessel et al. 2005; Grewe et al. 2006, 2007). Thus, the available evidence from brain imaging support the idea that comprehension of noncanonical sentences engages an extra processing load in Broca's area, regardless of the theoretical derivation of the word order.

Some authors have suggested that syntactically complex reversible sentences induce a greater processing load not upon first-pass comprehension but rather in a stage of post hoc reanalysis after the sentence has already been heard. Note, by “post hoc reanalysis,” we are referring to effortful rethinking of the sentence in working memory, extending up to several seconds from the time it is heard (Caplan and Waters 1999). This is somewhat distinct from other, more automatic, processes that are also described as post hoc, such as P600 responses following garden-path resolutions (Kaan and Swaab 2003; Bornkessel and Schlesewsky 2006).

In this study, we attempted to tease apart first-pass and post hoc processes by examining the effect of syntactic structure at 2 distinct stages. The first stage was the response to the auditory presentation of the sentence, discussed above. Due to the poor temporal resolution of hemodynamic responses in fMRI, the response to auditory sentences may include processes of both first-pass comprehension and post hoc reanalysis. As noted, the selective activation for reversible object-embedded sentences observed in left premotor and supplementary motor areas suggests a process under some degree of executive control, which may involve reanalysis. The second stage was the response to visual picture-matching trials that followed 50% of the sentences. Due to the temporal segregation between sentence and picture events, any effects of syntactic structure on the subsequent picture response must be solely due to post hoc (not first-pass) processes. Thus, the hemodynamic deconvolution of these 2 phases offers at least a partial dissociation.

We found that picture selection following a reversible object-embedded sentence induced a larger signal change in essentially the same left-hemisphere regions that exhibit a selective response to that condition upon picture presentation, including left IFG, PMd, and SMA. This suggests that reanalysis of difficult syntactic structures in working memory relies mainly on the same regions involved in first-pass processing. The finding of activation for syntactic working memory demands in this set of regions is in close agreement with other recent studies of noncanonical sentence processing in fMRI (Kinno et al. 2008) and lesion analysis (Amici et al. 2007; Kinno et al. 2009). These regions are among the many frontal regions implicated in various kinds of working memory, but the dorsal activations (LPMd and SMA) correspond most closely to regions involved in working memory for temporal order (Wager and Smith 2003), consistent with a role in syntactic reanalysis. The consistency of these activations at both temporal stages, sentence and picture, suggests that reanalysis does not depend on qualitatively different mechanisms of working memory than those involved in ordinary comprehension.

Despite the overall similarity between the syntactic sensitivity of sentence and picture responses, one striking dissociation was observed between them, in the right IFG. Although LIFG was preferentially activated by syntactic complexity at both the initial stage (auditory presentation) and the reanalysis stage (picture matching), right inferior frontal gyrus (RIFG) exhibited an appreciable hemodynamic response only to the pictures, and this response was selectively augmented in the RSO condition. This suggests that RIFG may be commonly recruited in effortful reanalysis of sentences but not in ordinary first-pass comprehension. This finding has implications for studies of poststroke aphasia, in which increased right-hemisphere activation is commonly observed during language tasks (Crinion and Price 2005; Raboyeau et al. 2008). Studies of aphasic comprehension have indicated that some syntactic effects are often delayed in aphasia, rather than absent entirely (Burkhardt et al. 2003; Love et al. 2008). This may relate to a shift in aphasic patients toward reliance on post hoc reanalysis, as ordinary mechanisms of language comprehension that operate more instantaneously may be unavailable.

We have discussed above the similarity of selective activations for reversible complex sentences at both the sentence and picture stages, as indicating that reanalysis depends upon similar mechanisms as initial comprehension. However, we must acknowledge a limitation of the fMRI technique for distinguishing between immediate and post hoc comprehension processes, both of which may play a role in generating the hemodynamic response to the sentences. Because the hemodynamic response integrates neural activity over several seconds, an element of reanalysis may be present even at the sentence stage. Therefore, another possible interpretation of our findings is that RSO sentences induce reanalysis that persists over several seconds, spanning the interval from the sentence to the picture. In order to investigate that possibility further, greater temporal resolution is necessary. Analysis of event-related and oscillatory activity in EEG and MEG data may help to distinguish between these possible interpretations of the fMRI results.

In summary, we have found that both factors of reversibility and syntactic complexity produce robust, dissociable patterns of activation upon auditory sentence presentation, and additionally upon subsequent picture matching. Upon auditory sentence presentation, reversibility and syntactic complexity contrasts both produce strongly left-lateralized patterns of activation for sentence comprehension in young, healthy subjects, although right hemisphere involvement is somewhat evident in the case of reversibility. Our results identify Broca's area as a region uniquely sensitive to the syntactic structure of sentences, whereas a wider network of left prefrontal structures is activated by the increased cognitive demands that are specific to reversible sentences with noncanonical word order. Processing of reversible, compared with irreversible sentences, not only engages essentially the same left prefrontal structures that are engaged by syntactic complexity but also recruits portions of temporal and parietal cortex that may contribute to thematic role assignment, even for syntactically simple sentences. Reanalysis of complex sentences, measured at the time of picture selection, activates essentially the same network of left frontal regions but not the classical posterior language areas in the superior temporal and inferior parietal lobes. However, reanalysis processes do activate the right-hemisphere homolog of Broca's area more strongly than instantaneous comprehension does. This rich palate of sentence content effects on brain responses may prove useful for the evaluation of neural plasticity of language networks in brain-damaged individuals.

Supplementary Material

Supplementary material can be found at: http://www.cercor.oxfordjournals.org/.

Funding

Intramural Research Program, National Institute on Deafness and Other Communication Disorders, National Institutes of Health.