The neural response to changing semantic and perceptual complexity during language processing

Abstract

Speech comprehension involves processing at different levels of analysis, such as acoustic, phonetic, and lexical. We investigated neural responses to manipulating the difficulty of processing at two of these levels. Twelve subjects underwent positron emission tomographic scanning while making decisions based upon the semantic relatedness between heard nouns. We manipulated perceptual difficulty by presenting either clear or acoustically degraded speech, and semantic difficulty by varying the degree of semantic relatedness between words. Increasing perceptual difficulty was associated with greater activation of the left superior temporal gyrus, an auditory‐perceptual region involved in speech processing. Increasing semantic difficulty was associated with reduced activity in both superior temporal gyri and increased activity within the left angular gyrus, a heteromodal region involved in accessing word meaning. Comparing across all the conditions, we also observed increased activation within the left inferior prefrontal cortex as the complexity of language processing increased. These results demonstrate a flexible system for language processing, where activity within distinct parts of the network is modulated as processing demands change. Hum Brain Mapp, 2010. © 2009 Wiley‐Liss, Inc.

INTRODUCTION

There are several levels of analysis that occur prior to understanding a spoken word. These include acoustic/phonetic processing, lexical‐semantic processing, and word order processes such as syntax [Bishop, 1997]. Introducing difficulty at any one of these levels can make speech harder to understand. Thus, both speech distorted by a bad phone connection, or conversation that is very abstract in its content can be hard to follow, albeit for very different linguistic reasons. Functional imaging has informed our understanding of the neural basis of the acoustic/phonetic and semantic processing of speech [Scott and Johnsrude, 2003; Thompson‐Schill, 2003] and has emphasized that a widely distributed neural network is recruited. The aim of the current study is, for the first time, to identify explicitly how activity within the neural systems supporting speech processing is modulated by introducing either acoustic/phonetic or semantic difficulty to language processing. This allows us to test the hypothesis that the presence of different types of processing complexity influences different parts of the language network in distinct ways. This would provide evidence for a flexible language network, in contrast to one in which the elements have a more modular organization.

The acoustic/phonetic processing of speech is supported by the superior temporal regions [e.g., Davis and Johnsrude, 2003; Scott et al., 2000]. Auditory processing is organized hierarchically, with responses to sounds of increasing intelligibility and complexity progressing from the superior temporal plane into lateral superior temporal gyri and sulci [Belin et al., 2000; Davis and Johnsrude, 2003; Scott et al., 2000; Spitsyna et al., 2006]. During speech comprehension, the outputs of this perceptual processing are mapped to stores of distributed semantic knowledge where the activation of specific semantic properties allows meaning to be accessed. These semantic processes involve the inferior temporal cortices (IT), the temporal poles, the left angular gyrus (AG), and regions within the left prefrontal cortex (PFC) [Binder, 2002; Martin and Chao, 2001; McClelland and Rogers, 2003; Sharp et al., 2006; Spitsyna et al., 2006; Vandenberghe et al., 1996]. An open question is to what extent different “nodes” in this network are differentially and flexibly recruited, depending on the perceptual and semantic properties being processed in spoken language.

The AG appears to be a particularly important “node” in the left hemisphere language network. The region contains high‐order association cortex that is directly connected to visual, somatosensory, and auditory association cortex [Geschwind, 1965], as well as to prefrontal cortical regions [Catani et al., 2005]. One suggestion is that the left AG is involved in mapping perceptual input to distributed semantic knowledge [Binder, 2002; Geschwind, 1965]. Alternatively, this region may be better thought of as providing higher level control for semantic processing [Jefferies and Lambon Ralph, 2006].

The left inferior PFC is a second brain region that appears to be central thought not exclusive to language processing. Acting in conjunction with more posterior brain regions it is particularly important for the control of verbal processing, with activation reflecting both semantic and phonological processing demands [Devlin et al., 2003; Gold and Buckner, 2002; Poldrack et al., 1999; Sharp et al., 2004b, 2005; Thompson‐Schill et al., 1999; Wagner et al., 2001] and neuropsychological evidence suggests that this region is involved in the control of semantic processing [Jefferies and Lambon Ralph, 2006].

In this study, we compared the effects of manipulating the acoustic/phonetic and the semantic difficulty of language processing. In contrast to most previous research in this field we varied the properties of the speech itself, keeping the task constant. Subjects made decisions based on the semantic relatedness between heard words. In a control condition, subjects were presented with clear speech, in which the words were clearly semantically related (Speech Contr). Semantic difficulty was varied in another condition by increasing the semantic distance between the words, such that they were not closely related (Sem Diff). Acoustic/phonetic difficulty was varied in a third condition by presenting the words as acoustically degraded speech (Ac/Phon Diff). To control for low‐level sensory and motor activation, as well as nonsemantic cognitive processing, a baseline task consisting of sound‐based decisions made on rotated speech stimuli was also employed (Rot).

We predicted that increasing the difficulty of language processing, either as the result of semantic or acoustic/phonetic factors, would result in increased activation of the left inferior PFC, in keeping with its role in controlled processing. We expected these common prefrontal changes to be associated with distinct task‐dependent changes in posterior cortex. Thus, acoustic/phonetic difficulty would be associated with greater activation in auditory‐perceptual cortex, whilst increased semantic difficulty would be associated with greater activation in regions involved in accessing and using stored semantic memory, such as IT and the left AG.

MATERIALS AND METHODS

Subjects

Twelve right‐handed healthy volunteers (7 males) aged between 35 and 61 years (median age = 49.5) participated in the study after giving informed written consent. They were native English speakers with no history of neurological disorders and received no financial reward other than reimbursement of travel expenses. Permission to administer radioisotopes was given by the Department of Health, UK, and the local research ethics committee of the Hammersmith Hospital approved the project. All subjects performed within the normal range on the following neuropsychological tests: digit span, the National Adult Reading Test [Nelson, 1982], the Pyramids and Palm Trees Test [Howard and Patterson, 1992], and the Mini Mental State Examination [Folstein et al., 1975].

Study Design

The three semantic conditions (Speech Contr, Sem Diff, Ac/Phon Diff) all involved the same task: hearing a series of three words, then making a decision based on the degree of their semantic relatedness. Word triplets were presented in an “AB–X” form, with the target word, “X,” preceded by cue and distracter words. Cue words were presented at either position “A” or position “B,” with presentation order pseudo‐randomized across triplets. A standard question was asked once at the onset of each scanning block (“Which of the first two words has most in common with the last word e.g., frost, pear, cherry?”). Subjects signaled their response with a button press. Within each triplet, onsets of stimuli “A,” “B,” and “X” were fixed at 0, 1,200, and 2,400 ms relative to the start of each trial (see Fig. 1). Reaction times were taken from the onset of the target word “X.” Stimulus pacing was fixed, occurring every 6 s. Stimuli were digitally recorded in an anechoic chamber and spoken by a female British English speaker.

Figure 1.

Stimulus timings. Word triplets consist of cue, distracter (“a” and “b”), and target words (X). Timings are in milliseconds.

Semantic conditions

Manipulation of semantic difficulty (Fig. 2)

Figure 2.

The experimental manipulations used to generate variable semantic and acoustic/phonological (Ac/Phon) processing difficulty.

The stimuli were nouns, mostly taken from a computational model of semantic memory [Vigliocco et al., 2004]. The model provides a measure of the “semantic featural distance” between words. Pairs of words are grouped into those that are “very close,” “close,” “medium,” or “far” in terms of semantic distance. Words that are “very close” in semantic space share many features, while words that are “far” share few features. Semantic difficulty was manipulated by varying the semantic distance between the cue, distracter, and target words (see Supporting Information Appendix 1 for stimuli list). In the control speech condition (Speech) and the condition with high acoustic/phonological difficulty (Ac/Phon Diff), the semantic distance between cue and target words was classified as “very close,” while the distance between the cue and distracter words was classified as “far” (e.g., lion, tennis, leopard). In the difficult semantic condition (Sem Diff), the semantic distance between cue and target words was classified as either “close” or “medium,” while the link between cue and distracter words was classified as either “medium” or “far” (e.g., pepper, pencil, axe). Measurements taken from the computational model confirm distinct semantic difficulties in the two conditions. In the high semantic difficulty condition, the cue‐target distance was 6.6 and the cue‐distracter distance was 11.1. For the low semantic difficulty condition, the cue‐target distance was 2.7 and the cue‐distracter distance was 23.0 (in the model semantic distances of greater than 18 units characterize semantically unrelated pairs). Thus, selection of the correct response in the condition with high semantic difficulty was more difficult on semantic grounds.

Word triplets were matched across the high and low semantic difficulty conditions for their frequency and familiarity using available scores from the MRC psycholinguistic database (http://www.psy.uwa.edu.au/mrcdatabase/uwa_mrc.htm). The average Kucera and Francis log frequency scores (±SD) were 0.98 (±0.055) and 1.01 (±0.041) for words used in the high and low semantic difficulty conditions. Average familiarity ratings were 531.2 (±50.7) and 528.8 (±54.2) for these conditions. The two conditions also consisted of words that were equally noun like. Ratings taken from CELEX were used to calculate average noun ratings of 95.8 (±14.6) and 94.6 (±16.7) for the high and low semantic difficulty conditions. There were also no significant differences between average scores of these variables for cue, target, and distracter words compared separately across the two sets of triplets.

Manipulation of acoustic/phonetic difficulty

Acoustic/phonetic difficulty was manipulated by acoustically degrading the speech stimuli. In this condition (Ac/Phon Diff), highly semantically related stimuli were presented as eight‐channel noise‐vocoded speech (8‐VoCo), constructed according to a method described previously [Scott et al., 2001; Shannon et al., 1995]. 8‐VoCo sounds like a harsh whisper, but can be understood after a brief period of training. Stimuli presented as either clear or degraded speech were pseudo‐randomized across subjects.

Piloting

A behavioral pilot study was carried out using 10 subjects who did not take part in the subsequent positron emission tomographic (PET) study. The main aim of the piloting was to increase the behavioral difficulty of the Sem Diff and Ac/Phon Diff conditions relative to semantic control condition, and balance the difficulty of Sem Diff and Ac/Phon Diff conditions as closely as possible. Stimuli were selected using an item analysis to attempt to construct conditions with the correct behavioral characteristics (in terms of reaction times and error rates). For example, in the difficult semantic condition a mixture of triplets where the link between cue and the two distracter words was either close/medium or medium/far were chosen to construct stimulus lists that would be more difficult than the Speech Contr condition and match the performance as closely as possible on triplet decisions from eight‐channel noise‐vocoded speech that we had observed in our previous work [Sharp et al., 2004a].

Rotated speech baseline task

The baseline condition utilized the same basic format as the semantic conditions (decision‐making on triplets of heard stimuli), but involved an acoustic rather than a semantic decision. Stimuli in the baseline condition (Rot) consisted of word triplets that had been spectrally rotated (inverted), according to a method described previously [Blesser, 1972; Scott et al., 2000]. Spectral rotation renders speech stimuli unintelligible, but retains the spectro‐temporal structure of words, including their syllabic structure [Blesser, 1972]. The timing of sound onsets within the rotated speech triplets and the intertrial interval were identical to those of the word triplets in the semantic conditions. Subjects were told that these sounds were not words, and were instructed to make decisions about the structure of the sound, that is, “Which of the first two sounds has the same number of beats as the third sound?” This provided a nonsemantic task that followed the same basic structure as the semantic task.

Training

Training consisted of a single session held immediately prior to scanning. Subjects were trained on examples of all four conditions, as well as being trained in the comprehension of eight‐channel vocoded speech. Stimuli used in piloting were not subsequently used during scanning. Toward the end of the training period, subjects heard 54 single words and were scored on the accuracy of their repetition of these words. The period of training was adjusted until each subject was repeating with >50% accuracy. Mean accuracy was 75% with a range of 61–87%.

PET Scanning

Subjects were scanned on a Siemens HR++ (966) PET camera [Spinks et al., 2000]. Water, labeled with a positron‐emitting isotope of oxygen (H${}_{\text{2}}^{\text{15}}$O), was used as the tracer to demonstrate changes in regional cerebral blood flow, equivalent to changes in tissue concentration of H${}_{\text{2}}^{\text{15}}$O. Positron emissions are integrated over the course of each scan to provide a single measure of regional cerebral blood flow for each voxel from each scan. Analysis involved relating changes in local tissue activity (normalized for global changes in activity between scans) to the behavioral task.

Sixteen scans were performed on each subject with the room darkened and the subjects' eyes closed. Each of the four conditions was repeated four times, with the order of conditions pseudo‐randomized across subjects. Scans were repeated at 6‐min intervals. This interscan interval allows positron emission to decay to background levels by the start of the next scan. Thirteen stimuli were presented in each block, as described above. The block was timed to start 15 s before the arrival of radiolabeled water in the brain. After measured attenuation correction, images were reconstructed by filtered back projection (Hanning filter, cut‐off frequency 0.5 Hz).

Data Analysis

SPM99 software (Wellcome Department of Cognitive Neurology, Queen Square, London: http://www.fil.ion.ucl.ac.uk/spm) was used to realign the individual PET scans. These were then spatially transformed (normalized) into standard MNI (Montreal Neurological Institute) stereotactic space [Evans et al., 1993]. This transformation allowed comparisons to be made across individuals. The scan data were then smoothed using an isotropic 16 mm, full width half‐maximum Gaussian kernel to account for individual variation in gyral anatomy and to improve the signal‐to‐noise ratio. Specific effects were investigated using appropriately weighted linear contrasts and covariates to create SPMs of the T‐statistic. We used a blocked ANCOVA with global counts as confound to remove the effect of global changes in perfusion across scans. Scan order was entered as a nuisance variable.

The neural system involved in processing word meaning was identified by contrasting the semantic conditions with the rotated speech baseline condition (i.e., [(Speech Contr + Sem Diff + Ac/Phon Diff) − Rot]). To identify brain regions that responded to changing semantic or perceptual difficulty, decision‐making with either high semantic (Sem Diff) or high perceptual difficulty (Ac/Phon Diff) was contrasted with the low semantic and low perceptual difficulty condition (Speech Contr). In addition Sem Diff and Ac/Phon Diff were compared directly. This allowed the comparison of conditions matched for reaction time, demonstrating brain regions that differentially responded to high semantic or perceptual difficulty independent of reaction time. Second‐level random effects analyses with a threshold of P < 0.05 corrected for multiple comparisons across the brain volume were employed, except for activations that fell within the semantic or perceptual system, about which we had a priori hypotheses. These areas consisted of left and right superior temporal gyri, the left AG, left IT, the temporal poles and the left inferior PFC and the left rostral PFC [Binder et al., 1997; Scott et al., 2000; Sharp et al., 2004b; Wagner et al., 2001]. For these activations, we employed a threshold of P < 0.001 uncorrected.

Regions involved in semantic processing were investigated further by employing two separate region of interest (ROI) analyses of the semantic and auditory‐perceptual systems. ROIs were constructed around peaks of activation taken from the overall semantic contrast [(Speech Contr + Sem Diff + Ac/Phon Diff) − Rot] using a sphere centered on the peak of activation with a diameter of 5 mm. This resulted in regions within the left AG, the left fusiform gyrus/parahippocampal gyrus, the left lateral OFG, and the left superior frontal gyrus. A further ROI analysis was performed to investigate differences in activation within the superior temporal gyri. These superior temporal regions were not apparent in the main semantic contrast, as the rotated speech baseline also activated unimodal auditory areas. Therefore, to allow unbiased sampling two anatomical ROIs were used, which encompassed the central part of the STG (midSTG) in both hemispheres taken from a probabilistic atlas [Hammers et al., 2003]. The region extended from Y −30 to +3 and contained Heschl's gyrus and parts of planum temporale posteriorly and planum polare anteriorly. Subject‐specific mean activation values from these regions were obtained by using an ROI analysis toolbox implemented within SPM99 [Brett et al., 2002]. For all ROIs, we derived estimates of mean activation for the contrasts of Speech Contr, Sem Diff, and Ac/Phon Diff against baseline. The mean activation values for these contrasts were entered as the dependent variable into stepwise multiple regression analyses.

RESULTS

Behavioral Performance Within the Scanner (Table I)

Table I.

Behavioral performance

Condition	% Acc (SEM)	RT ms (SEM)
Speech Contr	95.30 (0.91)	1606.63 (68.04)
Sem Diff	74.80 (2.10)	2252.38 (82.76)
Ac/Phon Diff	82.90 (1.50)	2035.60 (71.56)
Rot	79.50 (1.90)	1621.83 (66.81)

Table of behavioral performance in the scanner: Percentage accuracy (% Acc) and reaction times (RT).

The Sem Diff and Ac/Phon Diff conditions were behaviorally more difficult than the semantic control and rotated speech conditions. Responses were significantly slower for Sem Diff compared to Speech Contr (T = 11.913 (df 10), P < 0.0005) and Rot conditions (T = 6.302 (df 10), P < 0.0005). Accuracy was significantly less for Sem Diff compared to Speech Contr (T = −9.896 (df 10), P < 0.0005) and Rot conditions (T = −7.326 (df 10), P < 0.0005). Likewise responses were significantly slower for Ac/Phon Diff compared to Speech Contr (T = −7.983 (df 10), P < 0.0005) and Rot conditions (T = 5.493 (df 10), P = 0.001). Accuracy was significantly less for Ac/Phon Diff compared to Speech Contr (T = −9.06 (df 10), P < 0.0005), but not Rot conditions. In addition, responses were significantly slower for Sem Diff than Ac/Phon Diff (T = 2.61 (df 10), P = 0.026) and less accurate (T = −2.805 (df 10), P = 0.019). Accuracy was significantly less for Rot compared to Speech Contr conditions (T = −7.326 (df 10), P < 0.0005). There was no significant difference in accuracy between Sem Diff and Rot.

Brain Imaging Results

Whole brain analysis (Figs. 3, 4, and 5)

Figure 3.

Contrasting semantic processing with the nonlexical baseline. (i) Regions where activation was greater during semantic processing (Speech Contr, Sem Diff, and Ac/Phon Diff) than the nonlexical baseline (Rot), rendered onto coronal slices from the group MRI template. The threshold was set at 0.001 uncorrected, with a cluster threshold of 10 voxels. T‐statistic values are represented in the color scale. X and Y values refer to MNI co‐ordinates. (ii) Plots of effect sizes for the three semantic conditions relative to the nonlexical baseline task, taken from the peaks of activation illustrated in (i): (a) left superior frontal gyrus (L SFG), (b) left lateral orbitofrontal gyrus (Lat OFG), (c) left fusiform gyrus/parahippocampal gyrus (L FG/PHG), and (d) left angular gyrus (L AG). Units of effect size are relative to whole brain mean activity values.

Figure 4.

The effects of varying semantic and acoustic/phonological difficulty. (i) (1) High semantic difficulty versus the control speech condition in yellow (Sem Diff > Speech Contr). (2) High acoustic/phonological difficulty versus the control speech condition in green (Ac/Phon Diff > Speech Contr). (3) High semantic versus high acoustic/phonological difficulty conditions in red (Sem Diff > Ac/Phon Diff). The overlap of (1) and (2) displayed in blue and of (1) and (3) in purple. Contrasts are rendered onto sagittal (a–c) and coronal (d–f) slices from the group MRI template. (ii) A display of the statistical parametric map showing sagittal and coronal projections of the overall contrast of semantic processing with the nonlexical baseline. (a–f) represent the locations of sagittal and coronal slices shown in (i). Thresholding and labeling as in Figure 3.

Figure 5.

The effects of changing semantic and acoustic/phonological difficulty in the superior temporal lobes. (i) Regions where activation was greater during semantic processing with high acoustic/phonological difficulty (Ac/Phon Diff) than high semantic difficulty (Sem Diff), rendered on to coronal and axial slices from the group MRI template. Thresholding and labeling as in Figure 3. (ii) Plots of effect size for the three semantic conditions relative to the nonlexical baseline task, taken from the mid superior temporal gyrus (STG) ROI in left (a) and right (b) hemispheres. Units as in Figure 3.

All semantic decision‐making conditions against baseline

The contrast of all semantic decision‐making conditions (Speech Contr, Sem Diff, and Ac/Phon Diff) with the rotated speech baseline (Rot) was used to demonstrate brain regions involved in semantic processing. This contrast produced a predominantly left lateralized system with peaks of activation within left IT, the left parietal lobe, and the left PFC (Fig. 3 and Table II). Within IT, peaks of activation were located within the left fusiform gyrus, the left parahippocampal gyrus, the left inferior temporal gyrus, and the left hippocampus. Within the parietal lobe, there were two peaks, one in the left AG and a second at the boundary of the left AG and the lateral bank of the intraparietal sulcus. Within the PFC, there were peaks in the left lateral orbitofrontal gyus (lateral OFG) and the superior and middle frontal gyri. Activation was also observed in the right cerebellum.

Table II.

Regions of significant activation for each analysis

Analysis	Region	Brodmann area	MNI co‐ordinates			T‐score
			x	y	z
Overall semantic contrast	L Hippocampus	35	−28	−36	−8	4.74
	L FG/PHG	20/36	−36	−28	−24	3.93
	L FG	20	−42	−12	−36	3.65
	L ITG	20	−38	−8	−36	3.48
	L lateral OFG	47/12	−38	36	−16	3.82
	L SFG	9	−12	50	50	3.77
	L MFG	6/8	−32	12	46	3.67
	L AG/IPS	40/7	−32	−78	48	4.83
	L AG	40	−38	−74	24	4.25
	R cerebellum		38	−74	−40	4.15
SHPL vs. SLPL	L MFG	8/9	−42	24	49	4.61
	L IFG	47/12	−44	28	−4	4.43
	L lateral OFG	47/12	−38	38	−24	4.02
	L AG	40	−30	−64	46	4.10
SLPH vs. SLPL	L IFG	45	−42	32	10	3.78
	L MFG	9	−46	24	36	3.66
SHPL vs. SLPH	L SFG	8	−6	44	50	4.45
	L lateral OFG	47/12	−44	40	−12	4.04
	L IFG	47	−50	48	−2	3.56
	L AG/IPS	39/7	−30	−78	32	4.07
	R IPL	40/7	40	−58	52	3.38
SLPH vs. SHPL	L STG (PT)	22	−44	−32	14	4.87
	L Par Op	—	−48	−16	28	4.67
	L Temp Op	21	−46	−12	−2	4.41
	L MTG	21	−54	−10	−4	4.31
	L STS	21/22	−58	−54	8	4.25
	L STG (PT)	22	−62	−18	10	4.15
	L Par Op	40	−56	−38	28	3.87
	R IPL	40	64	−28	28	4.53
	R IFG	44	58	14	10	3.75
	R IFG	44	54	10	6	3.72
	R MFG	10	36	62	−2	4.33
	R Cerebell	—	22	−62	−26	3.99
	R Ins	—	−40	8	−4	4.51

Approximate Brodmann's areas are taken from the Talairach and Tournoux Atlas after transforming MNI co‐ordinates into Talairach space (http://imaging.mrc-cbu.cam.ac.uk/imaging/MniTalairach). (Hammers et al., 2003) (Chiavaras et al., 2001; Rushworth et al., 2005) used for additional anatomical localization. Overall semantic contrast [(SLPL + SHPL + SLPH) – Rot]. Left (L), right (R), fusiform gyrus (FG), inferior temporal gyrus (ITG), superior temporal gyrus (STG), lateral orbitofrontal gyrus (lateral OFG), inferior frontal gyrus (IFG), middle frontal gyrus (MFG), superior frontal gyrus (SFG), superior temporal gyrus (STG), superior temporal sulcus (STS), middle temporal gyrus (MTG), planum temporale (PT), temporal operculum (Temp Op), parietal operculum (Par Op), insula (Ins) angular gyrus (AG), inferior parietal lobe (IPL), and intraparietal sulcus (IPS).

The effects of semantic and acoustic/phonetic difficulty (Figs. 3, 4, and 5)

The effects of varying semantic and acoustic/phonetic difficulty were first investigated by directly comparing conditions with high semantic or high perceptual difficulty (Sem Diff and Ac/Phon Diff) with uncomplicated semantic decision‐making consisting of decisions with a low semantic and low perceptual difficulty (Speech Contr) (see Fig. 4).

Increased semantic difficulty (Sem Diff vs. Speech Contr) was associated with greater activation within the left PFC and the left AG. Peaks of differential activation in the left frontal lobe were observed within the inferior frontal gyrus, the middle frontal gyrus, and the lateral OFG. The reverse contrast (Speech Contr vs. Sem Diff) was associated with greater activation for semantic decisions with a low semantic difficulty within the planum temporale bilaterally. Greater activation for Speech Contr was also observed in the left STG anterior to planum temporale and in the left inferior parietal lobe where activation extended superiorly from the planum temporale.

Increased perceptual difficulty (Ac/Phon Diff vs. Speech Contr) was also associated with greater activation within the left PFC. In general, more posterior and superior parts of the left PFC were activated in this contrast when compared to the pattern of activation observed when decisions were made with an increased semantic difficulty (see Fig. 4). Peaks of activation difference for Ac/Phon Diff versus Speech Contr were observed within the left inferior frontal gyrus (BA 45) and the left middle frontal gyrus (BA 9). The reverse contrast (Speech Contr vs. Ac/Phon Diff) showed no regions where activation was greater for perceptually simple semantic decisions.

Contrasting decisions with a high semantic or high perceptual difficulty (Sem Diff and Ac/Phon Diff) allowed the comparison of two conditions with different sources of decision‐making complexity. The comparison of decisions made with a high semantic difficulty against those with a high perceptual difficulty (Sem Diff vs. Ac/Phon Diff) showed greater activation of the left PFC when decisions were semantically complex (see Fig. 4). Peak differences in activation were observed within the anterior part of the left inferior frontal cortex with a peak within the left lateral OFG and the left superior frontal gyrus. Greater activation for this contrast was also observed in both left and right parietal lobes for Sem Diff, with peaks at the boundary of the left AG and the intraparietal sulcus, and within the right AG.

The reverse contrast (Ac/Phon Diff vs. Sem Diff) demonstrated areas of greater activation when speech comprehension was perceptually complex, correcting for whole brain comparisons (see Fig. 5). Extensive parts of the mid and posterior STG in the left hemisphere were activated, extending superiorly into the inferior parietal lobe and inferiorly into the superior temporal sulcus and middle temporal gyrus. Peaks of differential activation included left planum temporale, the left superior temporal sulcus, the temporal operculum, the inferior parietal lobe, and the parietal operculum. In the right hemisphere, one activation peak lay within the left inferior parietal lobe with activation extending inferiorly into the STG. In addition, Ac/Phon Diff was associated with greater activity in the right dorsolateral PFC and the right cerebellum.

ROI analyses (Figs. 3 and 5)

Two additional ROI analyses were performed to further investigate the semantic and auditory‐perceptual systems. This allowed us to analyze the relationship between the three semantic conditions and also to test explicitly for the presence of region × condition interactions. Regions within the semantic system were defined from the contrast of all the semantic conditions against the nonspeech baseline. Regions within the auditory‐perceptual system were defined from a probabilistic atlas of the temporal lobe [Hammers et al., 2003]. ROIs were defined both functionally and anatomically, because regions involved in the auditory‐perceptual analysis of speech would not be expected to be seen in a contrast of semantic processing against a nonsemantic baseline matched for acoustic complexity. This approach allowed unbiased sampling of activation from the auditory‐perceptual and semantic systems. Plots of activation from these ROIs are shown in Figures 3 and 5.

ROI analysis of the semantic system

Four semantic ROIs were constructed centered on the left AG, the left fusiform gyrus/parahippocampal gyrus, the left lateral OFG, and the left superior frontal gyrus. Activation for the three semantic tasks was significantly greater than for the nonspeech baseline in all regions except for Ac/Phon Diff in the left superior frontal gyrus.

Across this semantic system, repeated measures ANOVA revealed significant effects of condition (F (2,22) = 10.17, P = 0.001) and region (F(3,33) = 9.185, P < 0.0005) but no condition × region interaction. The condition effect resulted from greater activation when decisions had a high semantic difficulty: Sem Diff was associated with more activation than both Speech Contr (F(1,11) = 11.65, P = 0.006) and Ac/Phon Diff (F(1,11) = 13.37, P = 0.003). The effect of region resulted from the combination of greater activation during semantic decision‐making in the left lateral OFG than either the left parietal (F(1,11) = 5.67, P = 0.044) or left superior frontal gyrus (F(1,11) = 10.05, P = 0.009) and, similarly, greater activation in the left fusiform gyrus/parahippocampal gyrus than either the left parietal (F(1,11) = 13.07, P = 0.004) or the left superior frontal gyrus (F(1,11) = 20.37, P = 0.001).

ROI analysis of the auditory‐perceptual system

In the auditory‐perceptual system, an ROI analysis was carried out on anatomical regions encompassing the central part of the STG in both hemispheres. The right and left STG showed differential activation for decisions made with high semantic or high perceptual difficulties. The semantic tasks were compared with the unintelligible rotated‐speech baseline, which would be expected to activate similar auditory‐perceptual regions.

A repeated measures ANOVA revealed significant hemispheric differences in STG activation. A hemisphere by condition interaction was present (F(1,22) = 9.11, P = 0.002), the result of a distinct effect of perceptual complexity in the two hemispheres. Essentially, relative to the rotated speech baseline, a high perceptual difficulty is associated with increased activation in the left STG both relative to the baseline condition and to activity in the right STG. In contrast, a high semantic difficulty is associated with reduced activation in both left and right superior temporal gyri. In the left STG, there was a main effect of condition (F(2,22) = 20.84, P < 0.0005). Activation associated with Ac/Phon Diff was greater than both Speech Contr (T = 2.92 (df 11), P = 0.014) and Sem Diff (T = 6.75 (df 11), P < 0.0004). Activation associated with Speech Contr was also greater than Sem Diff (T = 3.6 (df 11), P = 0.004). In the right hemisphere there was also a significant main effect of condition (F(2,22) = 11.75, P < 0.0005), but this resulted from less activation for Sem Diff than both Speech Contr (T = −5.36 (df 11), P < 0.005) and Ac/Phon Diff (T = 2.61 (df 11), P = 0.024), but no difference in activation between Speech Contr and Ac/Phon Diff. Comparing across the hemispheres there was a significantly greater activation for difficult acoustic/phonological decision‐making in the left than right hemisphere (T = 2.65 (df 11), P = 0.023), but no other interhemispheric differences in the other condition.

Comparing each of the semantic conditions to the baseline condition showed that simple semantic decisions made on clear speech stimuli produced similar levels of activation to the baseline task in both hemispheres. However, in the left STG increasing the perceptual or semantic difficulty of decision‐making was associated with changing levels of activation; Ac/Phon Diff was associated with increased activation (T = 2.45 (df 11), P = 0.032) and Sem Diff with reduced activation (T = −3.43 (df 11), P = 0.006). In the right STG, only Sem Diff showed a significantly different level of activation to the baseline, being associated with reduced activation in the right midSTG (T = −4.98 (df 11), P < 0.0005).

DISCUSSION

This study compares for the first time the neural response to changes in semantic and in acoustic/phonetic difficulty during language processing. Increasing perceptual demands result in greater involvement of the left STG, a change consistent with attentional support for this area during effortful perceptual processing. In contrast, increasing semantic demands result in greater involvement of the left AG, an area central to the semantic processing of speech, and a reduced activation in auditory‐perceptual regions of the superior temporal cortex bilaterally.

The variability of activation we observed within the STG cannot simply be explained by bottom–up stimulus‐driven factors. First, low‐level acoustic differences between the stimuli do not explain the reduction in STG activation as semantic difficulty increases, as clear (i.e., undistorted) speech was used in both the Sem Diff and Speech Contr conditions. In addition, although the high perceptual difficulty and the baseline conditions both used acoustically degraded speech, previous studies have demonstrated that, in the absence of explicit task demands, listening to clear and degraded speech stimuli of comparable intelligibility results in similar activation of early unimodal auditory cortex in mid STG [Crinion et al., 2003; Narain et al., 2003; Scott et al., 2000]. In particular, one study from our group used the same PET scanner and showed similar levels of activation within the STG when subjects listened passively to either noise vocoded speech or clear speech stimuli [Scott et al., 2000]. Finally, although activation of parts of the left STG has previously been shown to increase with greater speech intelligibility [Davis and Johnsrude, 2003; Narain et al., 2003; Scott et al., 2000], this effect cannot account for the changes observed in our study. Despite the noise vocoded stimuli being slightly less well understood, the effects in STG are greater for noise vocoded stimuli than for the clear speech.

Increased activation of the left STG has been observed during the perception of various types of degraded speech stimuli when subjects were required to make explicit decisions about the stimuli [Davis and Johnsrude, 2003]. Electrophysiological investigations of speech and auditory processing have also shown that activation in secondary auditory areas can be enhanced by attentional demands [Obleser et al., 2004; Poeppel et al., 1996]. Together with our results, this suggests increases in perceptual processing that occurs specifically when acoustic input is poor. For example, an attentional mechanism might enhance the perceptual analysis of acoustic features, improving the extraction of phonetic information from degraded speech to allow more accurate word identification. At eight channels, noise vocoded speech has just enough acoustic information for single words to be understood, and lacks a sense of pitch and speech melody. Furthermore, it is hard to hear any individual speaker characteristics at eight channels. This might make the system more reliant on acoustic/phonetic representations in left STG to process noise vocoded speech, since supporting information about speech melody and speaker identity are not available.

In contrast, we observed that greater semantic processing demands were associated with less activation of the STG bilaterally. This change may reflect reallocation of attentional resources from regions involved in perceptual processing to those involved in semantic processing. This type of flexible attentional mechanism has been demonstrated in the visual system. Electrophysiological studies provide evidence for a limited capacity attentional system that supports the discrimination of visual stimuli [Hillyard et al., 1998], and within this system high level control signals have been shown to either enhance or reduce the level of activation associated with visual processing depending on the task demands [Gazzaley et al., 2005]. Although there is less evidence to support this type of effect in the auditory system, right hemisphere evoked responses have been shown to reduce when attention is focused on speech sounds [Poeppel et al., 1996], and a previous functional imaging study from our laboratory has shown that highly demanding postperceptual processing of heard speech results in lower activation in right STG than when hearing and repeating a familiar word [Sharp et al., 2005]. The reduced activation we observed within the STG could therefore reflect an active inhibition of perceptual processing, leading to direct suppression of neuronal activity. Alternatively, the removal of attentional support from the STG may necessarily result in reduced enhancement of activation, and hence produce a fall in neural activation without the need to invoke an active inhibitory signal.

Distinct hemispheric effects were observed in the superior temporal gyri. Activity reduced bilaterally in the superior temporal gyri during demanding semantic processing. In contrast, activation was greater in the left than the right STG during demanding acoustic/phonological processing. The presence of increased activation within the left STG during speech perception is in keeping with the results of Davis and Johnsrude [ 2003]. This pattern suggests that the enhancement of early auditory speech processing is left lateralized.

Increasing semantic difficulty was associated with increased activation in the inferior parietal lobes, in keeping with the importance of the AG as a node in the language processing network. The left AG has previously been shown to be activated during the semantic processing of stimuli across modalities [e.g., Hoenig and Scheef, 2009; Noppeney and Price, 2003; Noppeney et al., 2008; Sharp et al., 2004a; Vandenberghe et al., 1996]; however, its role in the semantic processing of speech remains uncertain. The white matter connections of the AG position it to act as an interface between auditory‐perceptual, memory, and executive control systems. Diffusion tensor imaging work has demonstrated connections from the left inferior parietal lobe to the posterior part of the left STG via the posterior segment of the arcuate fasciculus and to the left inferior PFC via the anterior segment of the arcuate fasciculus [Catani et al., 2005]. In addition, a pathway similar to the inferior longitudinal fascicle described in macaque monkeys links the inferior parietal lobe to parts of IT, including the fusiform gyrus and more anterior temporal lobe regions [Rushworth et al., 2006]. These connections allow bottom–up auditory input to be received from parts of the superior temporal lobe involved in the acoustic and phonological analysis of speech [Warren et al., 2005], top–down control signals to be received through links to the PFC and semantic processing to be influenced through links to the inferior/anterior temporal areas involved in accessing distributed semantic memory.

Geschwind proposed a model for word recognition where involvement of the AG generated the cross modal associations needed to link an activated lexical representation to its distributed semantic correlates [Geschwind, 1965], an idea that has since been developed by others [Binder, 2002; Mesulam, 1998; Warrington and Shallice, 1984]. However, this model would predict that the AG was strongly activated for passive speech perception, whereas few speech perception studies report AG activation to easily intelligible speech [Crinion et al., 2003; Scott et al., 2000, 2006]. An alternative proposal is that the AG forms part of a left lateralized fronto‐parietal network that is engaged to enhance the extraction of meaning from speech by increasing the top–down control of semantic processing in other regions [Jefferies and Lambon Ralph, 2006]. Consistent with this proposal, left AG activation is seen in passive speech perception when the signal is degraded and high levels of linguistic predictability improve intelligibility [Obleser et al., 2007].

The left AG may play a specific role in processing lexical‐semantic information within working memory. In conditions where speech comprehension is difficult, this process becomes increasingly important as it allows coherent meaning to be extracted [Martin and Romani, 1994]. A large amount of evidence supports the proposal that the supramarginal gyrus within the inferior parietal lobe forms part of a fronto‐parietal circuit supporting the encoding, maintenance, and selection of phonological information held in working memory [Braver et al., 1997; Paulesu et al., 1993; Shallice and Vallar, 1990; Zatorre et al., 1996]. Recently, it has been proposed that the AG plays a specific role in inhibiting irrelevant semantic information when word meaning is ambiguous [Hoenig and Scheef, 2009], and this may operate as a selection mechanism to support the representation of semantic information held in working memory.

Our study also provides some evidence for an anterior/posterior distinction in how the inferior parietal lobe is engaged by different types of linguistic difficulty. Semantic difficulty was associated with AG activation in the posterior part of the inferior parietal lobe whereas acoustic/phonetic difficulty was associated with activation within the inferior parietal lobes more anteriorly in keeping with an anterior–posterior distinction within the inferior parietal lobe for processing of phonological and semantic information. However, the role of the left AG seems unlikely to be limited to processing semantic information. Activation of the left AG can also be observed when stimuli from nonlanguage modalities are processed. For example, greater activation of the left AG has recently been reported for processing unpredictable compared to predictable spatial stimuli [Hahn et al., 2007]. Therefore, the relevance of AG activation during language processing is likely to depend on its interactions with connected parts of the semantic network. Functional imaging is beginning to inform the nature of this interaction, with both bottom–up and top–down inputs to the left AG modulated by distinct aspects of language processing [Noppeney et al., 2008; Obleser et al., 2007]. For example, when increased top–down control is needed to facilitate difficult speech comprehension, interaction between the left AG and the PFC increases [Obleser et al., 2007].

The left inferior PFC also increased with the difficulty of semantic processing. This was observed when decisions on word triplets with a weaker semantic connection were compared with either the control condition or the Ac/Phon condition. As semantic difficulty increased, greater left lateral OFG activation was observed. This confirms the observation made previously that the left anterior/inferior PFC is sensitive to the semantic relatedness between cue and target during semantic decision‐making [Wagner et al., 2001]. Our study also provides some evidence that increasing perceptual difficulty recruits more posterior regions within the inferior PFC, as the peak of activation for the contrast of Ac/Phon Diff with Speech Contr was located within the IFG around BA 45. This is in keeping with claims of a segregation of processing within the inferior PFC for semantic and phonological processing [e.g., Poldrack et al., 1999]. However, a large amount of evidence also suggests that this region is organized in a more general process‐specific way, for example, supporting the selection of relevant semantic information held in working memory [e.g., Gold and Buckner, 2002; Thompson‐Schill et al., 1997, 1999]. It has proved difficult to decide between these alternative theoretical positions [Thompson‐Schill, 2003], and one needs to be cautious before interpreting our results as support for a domain‐specific organization. The direct contrast between Ac/Phon Diff with Sem Diff, where perceptual difficulty did not show greater activation within the posterior part of the IFG for difficult acoustic/phonological processing, making it harder to interpret the anterior–posterior differences we observe. In addition, increased semantic difficulty in the context of this task is likely to be related to a greater competition between activated semantic information, so selection and retrieval demands cannot easily be disentangled.

Reaction time and accuracy rate differences were present between some of the conditions (Table I). As expected, the Ac/Phon and Sem Diff conditions both led to longer reaction times than the Speech control and Rot baselines. However, relative to the high accuracy of the Speech Control condition (95%), accuracy rates were low for the Rot baseline (79.5%) as well as Ac/Phon (83%) and Sem Diff (75%) conditions. This profile makes a simple effect of accuracy on the specific semantic activation for either the Ac/Phon or Sem Diff conditions unlikely. For example, the AG and the left lateral OFC (BA 47/12) were activated across all three semantic conditions relative to the Rot baseline (see Fig. 3), despite the baseline having a high error rate. The behavioral differences also cannot explain the differential activations we observed in the superior temporal lobes, although it is possible that the small RT difference (around 200 ms) between Sem Diff and Ac/Phon Diff could explain some of the differential activation we observed within the left AG and lateral PFC. However, as this increase in RT is likely to reflect more prolonged semantic processing during complex semantic decisions it does not detract from the general conclusion that these regions are important nodes in a network that supports semantic processing.

In summary, our results provide evidence for a flexible neural system in which different patterns of cortical activation are seen when subjects perform the same decision‐making task on speech that is either semantically or perceptually hard to process. This system involves interactions between prefrontal and temporo‐parietal cortices, and a variable involvement of the left auditory association cortex. These results are likely to reflect a flexible system of attentional control that allocates resources across auditory‐perceptual and semantic cortical regions, depending on the current demands of language processing.

Supporting information

Additional Supporting Information may be found in the online version of this article.

Supplementary Material